imlucas/mongodb-datasets — pull request #2 by TZ2016 (2014-06-16 21:36:50+00:00)
Title: Working skeleton code for random data generator

@imlucas
The generator now is working, with some colorful cli output that records each non-trivial action! Highlights are:
1. supports user defined schema as long as there is no embedding and the random data type is supported by Chance.js
2. impose max number of threads (default is 6 but fully configurable) to prevent database from crashing for too many connections
3. use "bulk insert" (the size is set to 10 for ease of debugging) to save unnecessary round trip time(RTT)
4. all data is generated on the fly, to avoid unnecessary buffer in memory

Please use the command "node index.js --size=" to run the program, and feel free to change the supposedly user-entered schema in index.js.

Next steps:
1. support embedded schema
2. encapsulate Schema class and add input validation
3. wrap up the interface of Inserter and DataStream, delete useless methods and/or add useful ones
4. dynamically adjust the max number of threads and the bulk size according to database performance
5. get started on "importer.js" that implements a DataStream object of external sources, to support replicating datasets using the existing skeleton

Thanks!