Package: docdb-import-export
Document DB import exportA simple utility package to import json files to document db and export data from document db collections.https://github.com/msankhala/docdb-import-exportSponsorCrohn's & Colitis Foundation-https://www.crohnscolitisfoundation.orgRoadmap[x] Provider importer script to import data from json files to document db.[x] Provider a simple python api to extend the functionality of the package.[ ] Provider exporter script to export data from document db collections to json files.[ ] Provider exporter script to export all data from document db collections to json files in a given directory.SetupCreate a EC2 instance in the same VPC as your document dbConnecting to an Amazon DocumentDB Cluster from Outside an Amazon VPCRun SSH tunnel to the EC2 instancessh-i<path/to/ec2-private-key.pem>-L27017:<DOCUMENT-DB-SERVER-HOSTNAME>:27017ec2-user@EC2-INSTANCE-DNS-ENDPOINT-Nkeep this command running in a separate terminal window.Create.envfile with the following variables and set the values.DOCDB_HOST="YOUR_DOCUMENT_DB_HOSTNAME"
DOCDB_PORT=YOUR_DOCUMENT_DB_PORT
DOCDB_USERNAME="YOUR_DOCUMENT_DB_USERNAME"
DOCDB_PASSWORD="YOUR_DOCUMENT_DB_PASSWORD"
DOCDB_REPLICA_SET="rs0"
DOCDB_READ_PREFERENCE="secondaryPreferred"
DOCDB_RETRY_WRITES="false"
DOCDB_DBNAME="dbname"
DOCDB_IS_TLS_CONNECTION="false"
DOCDB_TLS_CA_FILE_PATH="aws/aws-documentdb-ca-global-bundle.pem"
DOCDB_TLS_ALLOW_INVALID_HOSTNAMES="false"
DOCDB_DIRECT_CONNECTION="false"
COLLECTION_NAME=recipe
USER_COLLECTION_NAME=userUsesImport data from a json file to document dbpython-mdocdb_import_exportimport\--env-file=/path/to/.env\--fromjson=../my-data-folder/my.json\--db=test\--collection=temp\--dropImport data from a json file to document db using custom importer classpython-mdocdb_import_exportimport\--env-file=/path/to/.env\--fromjson=../my-data-folder/my.json\--db=test\--collection=temp\--import-class=some-dir/MyCustomImporter.py\--dropThe importer class filename and classname should be same and importer class should be a subclass ofDocDbDefaultJsonImporterclass and should implement all abstract methods.Import data from a directory to document dbpython-mdocdb_import_exportimport\--env-file=/path/to/.env\--fromjsondir=../my-data-folder/\--db=test\--collection=temp\--dropImport data from a directory to document db using custom importer classpython-mdocdb_import_exportimport\--env-file=/path/to/.env\--fromjsondir=../my-data-folder/\--db=test\--collection=temp\--import-class=some-dir/MyCustomImporter.py\--dropThe importer class filename and classname should be same and importer class should be a subclass ofDocDbDefaultJsonImporterclass and should implement all abstract methods.Providing your own custom importer classCreate a custom importer class that extendsDocDbDefaultJsonImporterclass and implement all abstract methods.src/some-path/MyCustomImporter.pyimportjsonfromdotenvimportload_dotenvfromdocdb_import_export.docdb_clientimportDocDbClientfromdocdb_import_export.docdb_json_importerimportDocDbDefaultJsonImporterload_dotenv()classMyCustomImporter(DocDbDefaultJsonImporter):def__init__(self,source_json_file_path,db_name,collection_name,drop_collection,update):super().__init__(source_json_file_path,db_name,collection_name,drop_collection,update)defimport_json(self):# Only add if you want to add support for --drop option.self.delete_collection()# Read the json data from the file.withopen(self.source_json_file_path)asf:json_list=json.load(f)items=[]forindexinjson_list:# Call the transform_item method to transform the json data.items.append(self.transform_item(json_list[index]))# Insert the items into DocumentDB.self.docdb[self.db][self.collection].insert_many(items)print("Successfully imported json file: "+self.source_json_file_path)# This method allows you to transform the json data so that you can add or# remove the fields from the json data.deftransform_item(self,item):item["_id"]=item["id"]delitem["id"]# Add more transformations here if you want to.returnitemExample usage:python-mdocdb_import_exportimport--env-file=src/docdb_import_export/.env--fromjson=../recipe-finder-data/ccf.json--db=test--collection=recipe--import-class=docdb-migration/RecipeImporter.py--drop
Thiswillimporttheprovidedjsonfiletothe"test"databaseand"recipe"collectionusingthecustomimportclass"docdb-migration/RecipeImporter.py".Areyousureyouwanttocontinue?[y/N]:y
Importingjsonfile:../recipe-finder-data/ccf.json
Successfullyimportedjsonfile:../recipe-finder-data/ccf.json