Package: opensearch-reindexer
opensearch-reindexeropensearch-reindexeris a Python library that serves to help streamline reindexing data from one OpenSearch
index to another using either the native OpenSearch Reindex API or Python, the OpenSearch Scroll API and Bulk inserts.FeaturesNative OpenSearch Reindex API and Python based reindexing using OpenSearch Scroll APIMigrate data from one index to another in the same clusterMigrate data from one index to another in a different clusterMigration historyRun multiple migrations one after anotherTransform documents using native OpenSearch Reindex API or Python using Scoll API and Bulk insertsSource indices/data is never modified or removedGetting started1. Install opensearch-reindexerpip install opensearch-reindexerorpoetry add opensearch-reindexer2. Initialize projectreindexer init3. Configure your source_client in./migrations/env.pyYou only need to configuredestination_clientif you are migrating data from one cluster to another.4. Createreindexer_versionindexreindexer init-indexThis will use yoursource_clientto create a new index named 'reindexer_version' and insert a new document specifying the revision version.{"versionNum": 0}.reindexer_versionis used to keep track of which revisions have been run.When reindexing from one cluster to another, migrations should be run first (step 8) before initializing the destination cluster with:reindexer init-index5. Create revision (repeat if you have multiple indices)Two revision types are supported,painlesswhich uses the native OpenSearch Reindex API, andpythonwhich using
the OpenSearch Scroll API and Bulk inserts.painlessrevisions are recommended as they are more performant thanpythonrevisions. You don't have to use one or the other;./migrations/versions/can contain a combination of
bothpainlessandpythonrevisions.To create apainlessrevision run:reindexer revision 'my revision name'To create apythonrevision run:reindexer revision 'my revision name' --language pythonThis will create a new revision file in./migrations/versions.Note:revision files should not be removed and their names should not be changed once created../migration/migration_template_painless.pyand./migration/migration_template_python.pyare referenced for each revision.
You can modify them if you find yourself making the same changes to revision files.6. Modify your revision fileNavigate to your revision file./migrations/versions/1_my_revision_name.pyPainlessModifysourceanddestinationinREINDEX_BODY, you can optionally setDESTINATION_MAPPINGS.Note: If you only want to create the index, set the source index toNonee.g."source": {"index": "reindexer_revision_1"},To transform data as data is reindexed, you can use
painless scripts. For example, the following will convert data in field "c" from an object to a JSON string
before inserting it into indexdestination.REINDEX_BODY={"source":{"index":"reindexer_revision_1"},"dest":{"index":"reindexer_revision_2"},"script":{"lang":"painless","source":"""def jsonString = '{';int counter = 1;int size = ctx._source.c.size();for (def entry : ctx._source.c.entrySet()) {jsonString += '"'+entry.getKey()+'":'+'"'+entry.getValue()+'"';if (counter != size) {jsonString += ',';}counter++;}jsonString += '}';ctx._source.c = jsonString;"""}}For more information onREINDEX_BODYseehttps://opensearch.org/docs/latest/opensearch/reindex-data/PythonModifySOURCE_INDEXandDESTINATION_INDEX, you can optionally setDESTINATION_MAPPINGS.Note: If you only want to create the index, set the source index toNonee.g."source": {"index": "reindexer_revision_1"},To modify documents as they are being re-indexed to the destination index, updatedef transform_document. For example:classMigration(BaseMigration):deftransform_document(self,doc:dict)->dict:# Modify this method to transform each document before being inserted into destination index.importjsondoc['c']=json.dumps(doc['c'])returndoc7. See an ordered list of revisions that have not be executedreindexer list8. Run your migrationsreindexer runNote: Whenreindexer runis executed, it will compare revision versions in./migrations/versions/...to the version number inreindexer_versionindex of the source cluster.
All revisions that have not been run will be run one after another.FAQ 💬 🙋How do I start usingOpenSearch reindexerin a new project?To start usingOpenSearch reindexer, simply follow the steps outlined in the getting started guide.What happens if multiple revisions need to be executed?OpenSearch reindexercompares the remote version in thereindexer_versionindex on your OpenSearch cluster to your local version.
Any versions that have not been executed will be executed one after another.How to handle multiple indices?Create a revision for each index and follow the same steps as you would for a single index.How do I migrate from another schema management tool toOpenSearch reindexer?To migrate toOpenSearch reindexer, follow steps 1-6 in the getting started guide, repeating steps 5 and 6 for each
index. Set the source index to None during step 6 to create the destination index if it doesn't exist, or if it already
exists, proceed to the next revision.Downloading a project that usesOpenSearch reindexerIf thereindexer_versionindex on the OpenSearch cluster is up-to-date, runningreindexer runwon't do anything.
However, if the OpenSearch cluster hasn't been initialized, runreindexer init-indexfollowed byreindexer runto
create and initialize thereindexer_versionindex and run all migrations.Reindexing data from one OpenSearch cluster to anotherFollow the same steps for reindexing data to the same cluster, but update the "destination_client" in./migrations/env.py.Once you have reindexed all indices from one cluster to another, update the source and destination clients.