Package: pymongo-schema
pymongo-schemaA schema analyser for MongoDB, written in Python.This tool allows you toextract your application's schema, directly from your MongoDB data. It comes withpowerful schema manipulation and export functionalities.It will be particularly useful when you inherit a data dump, and want to quickly learn how the data is structured.pymongo-schema allows to map your MongoDB data model to a relational (SQL) data model. This greatly helps to configuremongo-connector-postgresql, a tool to synchronize data from MongoDB to a target PostgreSQL database.It also helps you tocompare different versions of your data model.This tools is inspired byvariety, with the following enhancementextract thehierarchical structureof the schemaversatile output options : json, yaml, tsv, markdown or htlmfiner grained types. ex: INTEGER, DOUBLE rather than NUMBERfilteringof the output schema, using anamespaceas defined bymongo-connectormapping to a relational schemacomparisonof successive schemaInstallYou can install latest stable version PyPi :pipinstall--upgradepymongo-schemaOr directly from github :pipinstall--upgradegit+https://github.com/pajachiet/pymongo-schemaUsageCommand linepython-mpymongo_schema-h
usage:[-h][--quiet]{extract,transform,tosql,compare}...

commands:{extract,transform,tosql,compare}extractExtractschemafromaMongoDBinstancetransformTransformajsonschematoanotherformat,potentiallyfilteringorchangingcolumnsoutputstosqlCreateamappingfrommongoschematorelationalschema(jsoninputandoutput)compareComparetwoschemas

optionalarguments:-h,--helpshowthishelpmessageandexit--quietRemoveloggingonstandardoutput

Usage:python-mpymongo_schemaextract-husage:[-h][-f[FORMATS[FORMATS...]]][-oOUTPUT][--portPORT][--hostHOST][-d[DATABASES[DATABASES...]]][-c[COLLECTIONS[COLLECTIONS...]]][--columnsCOLUMNS[COLUMNS...]][--sizeSIZE][--without-counts]python-mpymongo_schematransform-husage:[-h][-f[FORMATS[FORMATS...]]][-oOUTPUT][--categoryCATEGORY][-nFILTER][--columnsCOLUMNS[COLUMNS...]][--without-counts][input]python-mpymongo_schematosql-husage:[-h][-f[FORMATS[FORMATS...]]][--columnsCOLUMNS[COLUMNS...]][--without-counts][-oOUTPUT][input]python-mpymongo_schemacompare-husage:[-h][-f[FORMATS[FORMATS...]]][-oOUTPUT][input][--columnsCOLUMNS[COLUMNS...]][--without-counts][--detailed_diff]prev_schema[new_schema]To display full usage, with options description, run:pymongo-schema<command>-hPython packagepymongo_schema modules can also be imported to be used directly inside python code :frompymongo_schema.compareimportcompare_schemas_basesfrompymongo_schema.exportimporttransform_data_to_filefrompymongo_schema.extractimportextract_pymongo_client_schemafrompymongo_schema.filterimportfilter_mongo_schema_namespacesfrompymongo_schema.tosqlimportmongo_schema_to_mappingFore more details, refer to modules and functions docstrings.ExamplesFirst, lets populate a collection in test database from mongo shelldb.users.insert({name: "Tom", bio: "A nice guy.", pets: ["monkey", "fish"], someWeirdLegacyKey: "I like Ike!"})
db.users.insert({name: "Dick", bio: "I swordfight.", birthday: new Date("1974/03/14")})
db.users.insert({name: "Harry", pets: "egret", birthday: new Date("1984/03/14"), location:{country:"France", city: "Lyon"}})
db.users.insert({name: "Geneviève", bio: "Ça va?", location:{country:"France", city: "Nantes"}})
db.users.insert({name: "MadJacques", location:{country:"France", city: "Paris"}})Bash api examplesEasy examplesExtract the schema from this database, with a json format on standard output$ python -m pymongo_schema extract --database test
=== Start MongoDB schema analysis
Extract schema of database test
...collection users
   scanned 5 documents out of 5 (100.00 %)
--- MongoDB schema analysis took 0.00 s
=== Write output

{"test": {
    "users": {
        "object": {"_id": {"prop_in_object": 1.0, "count": 5, "type": "oid", "types_count": {"oid": 5}},
                   "pets": {"array_types_count": {"string": 2}, "prop_in_object": 0.4, "count": 2, "array_type": "string", "type": "ARRAY", "types_count": {"string": 1, "ARRAY": 1}},
                   "birthday": {"prop_in_object": 0.4, "count": 2, "type": "date", "types_count": {"date": 2}},
                   "name": {"prop_in_object": 1.0, "count": 5, "type": "string", "types_count": {"string": 5}},
                   "bio": {"prop_in_object": 0.6, "count": 3, "type": "string", "types_count": {"string": 3}},
                   "someWeirdLegacyKey": {"prop_in_object": 0.2, "count": 1, "type": "string", "types_count": {"string": 1}},
                   "location": {"object": {"country": {"prop_in_object": 1.0, "count": 3, "type": "string", "types_count": {"string": 3}},
                                           "city": {"prop_in_object": 1.0, "count": 3, "type": "string", "types_count": {"string": 3}}},
                                "types_count": {"OBJECT": 3}, "prop_in_object": 0.6, "type": "OBJECT", "count": 3}},
        "count": 5}}}Extract the same schema in md format.$ python -m pymongo_schema extract --database test --format md
=== Start MongoDB schema analysis
Extract schema of database test
...collection users
   scanned 5 documents out of 5 (100.00 %)
--- MongoDB schema analysis took 0.00 s
=== Write output

### Database: test
#### Collection: users 
|Field_compact_name     |Field_name             |Count     |Percentage     |Types_count                           |
|-----------------------|-----------------------|----------|---------------|--------------------------------------|
|_id                    |_id                    |5         |100.0          |oid : 5                               |
|name                   |name                   |5         |100.0          |string : 5                            |
|bio                    |bio                    |3         |60.0           |string : 3                            |
|location               |location               |3         |60.0           |OBJECT : 3                            |
| . city                |city                   |3         |100.0          |string : 3                            |
| . country             |country                |3         |100.0          |string : 3                            |
|birthday               |birthday               |2         |40.0           |date : 2                              |
|pets                   |pets                   |2         |40.0           |ARRAY(string : 2) : 1, string : 1     |
|someWeirdLegacyKey     |someWeirdLegacyKey     |1         |20.0           |string : 1                            |Map this schema to a relational mapping$ python -m pymongo_schema extract --database test | python -m pymongo_schema tosql
=== Start MongoDB schema analysis
Extract schema of database test
...collection users
   scanned 5 documents out of 5 (100.00 %)
--- MongoDB schema analysis took 0.00 s
=== Write output
=== Generate mapping from mongo to sql
=== Write output

{"test":
 {"users":
      {"_id": {"type": "TEXT", "dest": "_id"},
       "pets": {"valueField": "pets", "fk": "id_users", "type": "_ARRAY_OF_SCALARS", "dest": "users__pets"},
       "location.city": {"type": "TEXT", "dest": "location__city"},
       "name": {"type": "TEXT", "dest": "name"},
       "someWeirdLegacyKey": {"type": "TEXT", "dest": "someWeirdLegacyKey"},
       "pk": "_id",
       "bio": {"type": "TEXT", "dest": "bio"},
       "birthday": {"type": "TIMESTAMP", "dest": "birthday"},
       "location.country": {"type": "TEXT", "dest": "location__country"}},
  "users__pets": {"id_users": {"type": "TEXT"},
                  "pets": {"type": "TEXT", "dest": "pets"},
                  "pk": "_id_postgres"}}}Other examplesextract:Extract the schema for collectionstest_collection_1andtest_collection_2fromtest_dband write it intomongo_schema.htmlandmongo_schema.jsonfilespython-mpymongo_schemaextract--databasestest_db--collectionstest_collection_1test_collection_2--outputmongo_schema--formathtmljsonextract:Extract the schema for collectiontest_collection_1with only 1000 random rows scanned and write it intomongo_schema.htmlfilespython-mpymongo_schemaextract--collectionstest_collection_1--size1000--outputmongo_schema--formathtmltransform:Filter extracted schema (mongo_schema.json) usingnamespace.jsonfile and write output intomongo_schema_filtered.html,mongo_schema_filtered.csvandmongo_schema_filtered.jsonfilespython-mpymongo_schematransformmongo_schema.json--filternamespace.json--outputmongo_schema_filtered--formathtmlcsvjsontosql:Create mapping file based onmongo_schema_filtered.jsonpython-mpymongo_schematosqlmongo_schema_filtered.json--outputmapping.jsonPython api examplesExtract the schemas of all collections and all databases in a MongoDB instance:importpymongofrompymongo_schema.extractimportextract_pymongo_client_schemawithpymongo.MongoClient()asclient:schema=extract_pymongo_client_schema(client)Arguments can be specified to extract only some databases and some collections. See code documentation for more details.Filter extract schema with anamespace:importjsonfrompymongo_schema.filterimportfilter_mongo_schema_namespaces# assuming a namespace is defined in a file named namespace.jsonwithopen("namespace.json")asf:namespace=json.load(f)schema_filtered=filter_mongo_schema_namespaces(schema,namespace)Save filtered_schema (could be used for schema) to file in json and md formats in adocsdirectory:frompymongo_schema.exportimporttransform_data_to_filetransform_data_to_file(schema_filtered,['json','md'],output='docs/schema_filtered')Compare filtered_schema (could be used for schema) to another (previous for example) schema:frompymongo_schema.compareimportcompare_schemas_bases# assuming a namespace is defined in a file named namespace.jsonwithopen("old_schema_filtered.json")asf:old_schema_filtered=json.load(f)differences=compare_schemas_bases(old_schema_filtered,schema_filtered)Save differences to file in json and md formats in adocsdirectory:transform_data_to_file(differences,['json','md'],output='docs/diff',category='diff')Transform filtered_schema to a relational mapping:frompymongo_schema.tosqlimportmongo_schema_to_mappingmapping=mongo_schema_to_mapping(schema_filtered)Save mapping to file in json and md formats in adocsdirectory:transform_data_to_file(mapping,['json','md'],output='docs/mapping',category='mapping')SchemaWe define 'schema' as a dictionary describing the structure of MongoDB component, being either a MongoDB instances, a database, a collection, an objects or a field.Schema are hierarchically nested, with the following structure :# mongo_schema : A MongoDB instance contains databases{"database_name_1":{},#database_schema,"database_name_2":# A database contains collections{"collection_name_1":{},# collection_schema,"collection_name_2":# A collection maintains a 'count' and contains 1 object{"count":int,"object":# object_schema : An object contains fields.{"field_name_1":{},# field_schema,"field_name_2":# A field maintains 'types_count_information# An optional 'array_types_count' field maintains 'types_count' information for values encountered in arrays# An 'OBJECT' or 'ARRAY(OBJECT)' field recursively contains 1 'object'{'count':int,'prop_in_object':float,'type':'type_str','types_count':{# count for each encountered type'type_str':13,'Null':3},'array_type':'type_str','array_types_count':{# (optional) count for each type encountered  in arrays'type_str':7,'Null':3},'object':{},# (optional) object_schema}}}}}Contributing - Limitations - TODOThe code base should be easy to read and improve upon. Contributions are welcomed.Mixed types handlingpymongo-schema handles mixed types by looking for the lowest common parent type in the following tree.If a field contains both arrays and scalars, it is considered as an array. The 'array_type' is defined as the common parent type of scalars and array_types encountered in this field.TODOImprove mapping from Python type to name (TYPE_TO_STR dict)see documentation:bson-types,specCheck a mongo scheme for compatibility to an sql mappingHandle incompatibilitiesSupport Python 3 versionfix encoding issues when exporting manually added non-ascii charactersDiff between schemasA way to compare the schema dictionaries and highlights the differences.Test if a mongo schema can be mapped tosqltest for the presence of mongo types in the mappinglook for mixes of list and scalar, that are currently not supported by mongo-connector-postgresqllook for the presence of an '_id'=> It may be donne directly in mongo-connector-postgresql doc_managerAdding fields in json/yaml outputsfor example to add commentsOther option to sort text outputsIt is currently based on counts and then alphabetically.Tackle bigger databasesThis code has been only used on a relatively small sized Mongo database, on which it was faster than Variety.To tackle bigger databases, it certainly would be usefull to implement the following variety's features :Analyze subsets of documents, most recent documents, or documents to a maximum depth.TestsThe codebase is still under development. It should not be trusted blindly.