Package: dsw2to3
dsw2to3CLI tool to support data migration from DSW 2.14 to DSW 3.0UsagePrerequisitesDSW 3.0MongoDB (with DSW 2.14 data)PostgreSQL (with initial DSW 3.0 structure)S3 storage (e.g.Minio)Python 3.6+ (recommended to usevirtual environment)postgresql-devel(libpq-devin Debian/Ubuntu,libpq-develon others)The machine where you are going to execute the migration tool must have access to MongoDB, PostgreSQL, and S3 storage. Seeexamples/docker-compose.ymlfor reference.You need to run DSW 3.0 at least once before the data migration, so it initializes your PostgreSQL database (it will create tables and initial data). You can try to log-in with the default user to check if it is initialized correctly.Don't hesitate to consult with us if unclear.InstallationYou can install the tool using PyPI:$python-mvenvenv
$.env/bin/activate(env)$pipinstallwheel(env)$pipinstalldsw2to3
...(env)$dsw2to3--helpOr using this repository:$gitclonehttps://github.com/ds-wizard/dsw2to3.git
$python-mvenvenv
$.env/bin/activate(env)$pipinstallwheel(env)$pipinstall.
...(env)$dsw2to3--helpImportant notesMigration tool must have access to MongoDB database (data source), PostgreSQL database and S3 storage (target). It needs to be configured inconfig.yml. During the migration (e.g. from DSW or other tool), the data must not change to avoid inconsistency.Migration tool does not make any changes in MongoDB, it only reads data from there.Migration tool checks if target PostgreSQL database is in expected state (after fresh installation of DSW 3.0).Migration tool initially deletes all data from PostgreSQL database before migrating to avoid duplication and inconsistency (for regular use it just removes the default data, e.g., default users).Migration tool initially deletes all objects in configured S3 bucket. If the bucket does not exist, it tries to create a new one.Migration tool migrates data from MongoDB to PostgreSQL in expected way for DSW as well as from MongoDB (GridFS) to S3 storage.You can run the tool with--dry-runto check what it will do. During dry run, nothing is deleted, changed, or added (no SQL transactions are committed).It may happen that your MongoDB database contains inconsistent data (violating integrity). With--fix-integrityyou can fix that by skipping data. You should first check what the data are, and then decide if you will fix it manually in MongoDB or migrate without them.This tool may improve based on feedback, check new version and update usingpip install -U dsw2to3if needed.StepsPrepareconfig.ymlfor the migration based on your setup (seeconfig.example.yml)Stop DSW in order to prevent changes in data during the migrationArchive data from MongoDB (e.g. usingmongodump)Rundsw2to3 -c path/to/config.yml --dry-runto see how it will work with your configurationRundsw2to3 -c path/to/config.yml(seedsw2to3 --helpfor more options)After migration, run DSW 3.0 and check the migrated dataClean up your deployment (get rid of unused services and configuration files)In case of error during the migration, follow the details from logs. You can run it with--best-effortflag that will skip errors (just log them out).Questions and DiscussionIf anything is unclear, or you need help, let us know via issue in this repository.LicenseThis project is licensed under the Apache License v2.0 - see theLICENSEfile for more details.