Package: django-elastic-migrations
django-elastic-migrationsis a Django app for creating, indexing and changing schemas of Elasticsearch indexes.OverviewElastic has given us basic python tools for working with its search indexes:elasticsearch-py, a python interface to elasticsearch’s REST APIelasticsearch-dsl-py, a Django-esque way of declaring Elasticsearch schemas,
built uponelasticsearch-pyDjango Elastic Migrations adapts these tools into a Django app which also:Provides Django management commands forlisting indexes, as well as performingcreate,update,activateanddropactions on themImplements concurrent bulk indexing powered by pythonmultiprocessingGives Django test hooks for ElasticsearchRecords a history of all actions that change Elasticsearch indexesSupports AWS Elasticsearch 6.0, 6.1 (6.2 TBD; see#3 support elasticsearch-dsl 6.2)Enables having two or more servers share the same Elasticsearch clusterModelsDjango Elastic Migrations provides comes with three Django models:Index,IndexVersion, andIndexAction:Index- a logical reference to an Elasticsearch index.
EachIndexpoints to multipleIndexVersions, each of which contains
a snapshot of thatIndexschema at a particular time. EachIndexhas anactiveIndexVersionto which all actions are directed.IndexVersion- a snapshot of an ElasticsearchIndexschema at a particular
point in time. The Elasticsearch index name is the name of theIndexplus the
primary key id of theIndexVersionmodel, e.g.movies-1. When the schema is
changed, a newIndexVersionis added with namemovies-2, etc.IndexAction- a record of a change that impacts anIndex, such as updating
the index or changing whichIndexVersionis active in anIndex.Management CommandsUse./manage.py es--helpto see the list of all of these commands.Read Only Commands./manage.py es_listhelp: For eachIndex, list activation status and doc
count for each of itsIndexVersionsusage:./manage.py es_listAction CommandsThese management commands add an Action record in the database,
so that the history of eachIndexis recorded../manage.py es_create- create a new index../manage.py es_activate-activatea newIndexVersion. all
updates and reads for thatIndexby will then go to that version../manage.py es_update- update the documents in the index../manage.py es_clear- remove the documents from an index../manage.py es_drop- drop an index../manage.py es_dangerous_reset- erase elasticsearch and reset the
Django Elastic Migrations models.For each of these, use--helpto see the details.UsageInstallationpip installdjango-elastic-migrations; seedjango-elastic-migrationson PyPIPut a reference to this package in yourrequirements.txtEnsure that a validelasticsearch-dsl-pyversion is accessible, and configure
the path to your configured Elasticsearch singleton client in your django settings:DJANGO_ELASTIC_MIGRATIONS_ES_CLIENT = "tests.es_config.ES_CLIENT".
There should only be oneES_CLIENTinstantiated in your application.Adddjango_elastic_migrationstoINSTALLED_APPSin your Django
settings fileAdd the following information to your Django settings file:DJANGO_ELASTIC_MIGRATIONS_ES_CLIENT = "path.to.your.singleton.ES_CLIENT"
# optional, any unique number for your releases to associate with indexes
DJANGO_ELASTIC_MIGRATIONS_GET_CODEBASE_ID = subprocess.check_output(['git', 'describe', "--tags"]).strip()
# optional, can be used to have multiple servers share the same
# elasticsearch instance without conflicting
DJANGO_ELASTIC_MIGRATIONS_ENVIRONMENT_PREFIX = "qa1_"Create thedjango_elastic_migrationstables by running./manage.py migrateCreate anDEMIndex:from django_elastic_migrations.indexes import DEMIndex, DEMDocType
from .models import Movie
from elasticsearch_dsl import Text

MoviesIndex = DEMIndex('movies')


@MoviesIndex.doc_type
class MovieSearchDoc(DEMDocType):
    text = TEXT_COMPLEX_ENGLISH_NGRAM_METAPHONE

    @classmethod
    def get_queryset(self, last_updated_datetime=None):
        """
        return a queryset or a sliceable list of items to pass to
        get_reindex_iterator
        """
        qs = Movie.objects.all()
        if last_updated_datetime:
            qs.filter(last_modified__gt=last_updated_datetime)
        return qs

    @classmethod
    def get_reindex_iterator(self, queryset):
        return [
            MovieSearchDoc(
                text="a little sample text").to_dict(
                include_meta=True) for g in queryset]Add your new index to DJANGO_ELASTIC_MIGRATIONS_INDEXES in settings/common.pyRun./manage.py es_listto see the index as available:./manage.py es_list

Available Index Definitions:
+----------------------+-------------------------------------+---------+--------+-------+-----------+
|   Index Base Name    |         Index Version Name          | Created | Active | Docs  |    Tag    |
+======================+=====================================+=========+========+=======+===========+
| movies               |                                     | 0       | 0      | 0     | Current   |
|                      |                                     |         |        |       | (not      |
|                      |                                     |         |        |       | created)  |
+----------------------+-------------------------------------+---------+--------+-------+-----------+
Reminder: an index version name looks like 'my_index-4', and its base index name
looks like 'my_index'. Most Django Elastic Migrations management commands
take the base name (in which case the activated version is used)
or the specific index version name.Create themoviesindex in elasticsearch with./manage.py es_create movies:$> ./manage.py es_create movies
The doc type for index 'movies' changed; created a new index version
'movies-1' in elasticsearch.
$> ./manage.py es_list

Available Index Definitions:
+----------------------+-------------------------------------+---------+--------+-------+-----------+
|   Index Base Name    |         Index Version Name          | Created | Active | Docs  |    Tag    |
+======================+=====================================+=========+========+=======+===========+
| movies               | movies-1                            | 1       | 0      | 0     | 07.11.005 |
|                      |                                     |         |        |       | -93-gd101 |
|                      |                                     |         |        |       | a1f       |
+----------------------+-------------------------------------+---------+--------+-------+-----------+

Reminder: an index version name looks like 'my_index-4', and its base index name
looks like 'my_index'. Most Django Elastic Migrations management commands
take the base name (in which case the activated version is used)
or the specific index version name.Activate themovies-1index version, so all updates and reads go to it../manage.py es_activate movies
For index 'movies', activating 'movies-1' because you said so.Assuming you have implementedget_reindex_iterator, you can call./manage.py es_updateto update the index.$> ./manage.py es_update movies

Handling update of index 'movies' using its active index version 'movies-1'
Checking the last time update was called:
 - index version: movies-1
 - update date: never
Getting Reindex Iterator...
Completed with indexing movies-1

$> ./manage.py es_list

Available Index Definitions:
+----------------------+-------------------------------------+---------+--------+-------+-----------+
|   Index Base Name    |         Index Version Name          | Created | Active | Docs  |    Tag    |
+======================+=====================================+=========+========+=======+===========+
| movies               | movies-1                            | 1       | 1      | 3     | 07.11.005 |
|                      |                                     |         |        |       | -93-gd101 |
|                      |                                     |         |        |       | a1f       |
+----------------------+-------------------------------------+---------+--------+-------+-----------+DeploymentCreating and updating a new index schema can happen before you deploy.
For example, if your app servers are running with themovies-1index activated, and you
have a new version of the schema you’d like to pre-index, then log into another
server and run./manage.py es_create moviesfollowed by./manage.py es_update movies--newer. This will update documents in allmoviesindexes that are newer than the active one.After deploying, you can run./manage.py es_activate moviesto activate the latest version. Be sure to cycle your
gunicorn workers to ensure the change is caught by your app servers.During deployment, ifget_reindex_iteratoris implemented in such a way as to respond
to the datetime of the last reindex date, then you can call./manage.py es_update movies--resume, and it will indexonly those documents that have
changed since the last reindexing. This way you can do most of the indexing ahead of time,
and only reindex a portion at the time of the deployment.Django Testing(optional) updateDJANGO_ELASTIC_MIGRATIONS_ENVIRONMENT_PREFIXin
your Django settings. The default test prefix istest_.  Every
test will create its own indexes.if 'test' in sys.argv:
    DJANGO_ELASTIC_MIGRATIONS_ENVIRONMENT_PREFIX = 'test_'Override TestCase -test_utilities.pyfrom django_elastic_migrations import DEMIndexManager

class MyTestCase(TestCase):

    def _pre_setup(self):
        DEMIndexManager.test_pre_setup()
        super(MyTestCase, self)._pre_setup()

    def _post_teardown(self):
        DEMIndexManager.test_post_teardown()
        super(MyTestCase, self)._post_teardown()Excluding from Django’sdumpdatacommandWhen callingdjango’s dumpdata command,
you likely will want to exclude the database tables used in this app:from django.core.management import call_command
params = {
    'database': 'default',
    'exclude': [
        # we don't want to include django_elastic_migrations in dumpdata,
        # because it's environment specific
        'django_elastic_migrations.index',
        'django_elastic_migrations.indexversion',
        'django_elastic_migrations.indexaction'
    ],
    'indent': 3,
    'output': 'path/to/my/file.json'
}
call_command('dumpdata', **params)An example of this is included with themoviegen management command.Tuning Bulk Indexing ParametersBy default,/.manage.py es_updatewill divide the result ofDEMDocType.get_queryset()into batches of sizeDocType.BATCH_SIZE.
Override this number to change the batch size.There are many configurable paramters to Elasticsearch’sbulk updater.
To provide a custom value, overrideDEMDocType.get_bulk_indexing_kwargs()and return the kwargs you would like to customize.DevelopmentThis project usesmaketo manage the build process. Typemake helpto see the availablemaketargets.Elasticsearch Docker Composedocker-compose-flocal.yml upSee docs/docker_setup for more infoRequirementsThis project usespip-tools. Therequirements.txtfiles are generated
and pinned to latest versions withmake upgrade:runmake requirementsto run the pip install.runmake upgradeto upgrade the dependencies of the requirements to the latest
versions. This process also excludesdjangoandelasticsearch-dslfrom therequirements/test.txtso they can be injected with different
versions by tox during matrix testing.Populating Localtests_moviesDatabase Table With DataIt may be helpful for you to populate a local database with Movies test
data to experiment with usingdjango-elastic-migrations. First,
migrate the database:./manage.py migrate--run-syncdb--settings=test_settingsNext, load the basic fixtures:./manage.py loaddata tests/100films.jsonYou may wish to add more movies to the database. A management command
has been created for this purpose. Get aFree OMDB API key here,
then run a query like this (replaceMYAPIKEYwith yours):$> ./manage.py moviegen --title="Inception" --api-key="MYAPIKEY"
{'actors': 'Leonardo DiCaprio, Joseph Gordon-Levitt, Ellen Page, Tom Hardy',
 'awards': 'Won 4 Oscars. Another 152 wins & 204 nominations.',
 'boxoffice': '$292,568,851',
 'country': 'USA, UK',
 'director': 'Christopher Nolan',
 'dvd': '07 Dec 2010',
 'genre': 'Action, Adventure, Sci-Fi',
 'imdbid': 'tt1375666',
 'imdbrating': '8.8',
 'imdbvotes': '1,721,888',
 'language': 'English, Japanese, French',
 'metascore': '74',
 'plot': 'A thief, who steals corporate secrets through the use of '
         'dream-sharing technology, is given the inverse task of planting an '
         'idea into the mind of a CEO.',
 'poster': 'https://m.media-amazon.com/images/M/MV5BMjAxMzY3NjcxNF5BMl5BanBnXkFtZTcwNTI5OTM0Mw@@._V1_SX300.jpg',
 'production': 'Warner Bros. Pictures',
 'rated': 'PG-13',
 'ratings': [{'Source': 'Internet Movie Database', 'Value': '8.8/10'},
             {'Source': 'Rotten Tomatoes', 'Value': '86%'},
             {'Source': 'Metacritic', 'Value': '74/100'}],
 'released': '16 Jul 2010',
 'response': 'True',
 'runtime': 148,
 'title': 'Inception',
 'type': 'movie',
 'website': 'http://inceptionmovie.warnerbros.com/',
 'writer': 'Christopher Nolan',
 'year': '2010'}To save the movie to the database, use the--saveflag. Also useful is
the--noprintoption, to suppress json. Also, if you addOMDB_API_KEY=MYAPIKEYto your environment variables, you don’t have
to specify it each time:$ ./manage.py moviegen --title "Closer" --noprint --save
Saved 1 new movie(s) to the database: CloserNow that it’s been saved to the database, you may want to create a fixture,
so you can get back to this state in the future.$ ./manage.py moviegen --makefixture=tests/myfixture.json
dumping fixture data to tests/myfixture.json ...
[...........................................................................]Later, you can restore this database with the regularloaddatacommand:$ ./manage.py loaddata tests/myfixture.json
Installed 101 object(s) from 1 fixture(s)There are already 100 films available usingloaddataas follows:$ ./manage.py loaddata tests/100films.jsonRunning Tests LocallyRunmake test. To run all tests and quality checks locally,
runmaketest-all.To just run linting,make quality. Please note that if any of the
linters return a nonzero code, it will give anInvocationErrorerror
at the end. Seetox’s documentation for InvocationErrorfor more information.We useedx_lintto compilepylintrc. To update the rules,
changepylintrc_tweaksand runmake pylintrc.Cutting a New Versionoptional: runmake updateto update dependenciesbump version indjango_elastic_migrations/__init__.py.updateCHANGELOG.rst.make cleanpython3 setup.py sdist bdist_wheeltwine checkdist/django-elastic-migrations-*.tar.gzto see if there are any syntax mistakes before taggingsubmit PR bumping the versionensure test matrix is passing on travis and merge PRpull changes to mastermake cleanpython3 setup.py sdist bdist_wheeltwine checkdist/django-elastic-migrations-*.tar.gzto see if there are any syntax mistakes before taggingtwine upload-rtestpypidist/django-elastic-migrations-*.tar.gzCheck it at https://test.pypi.org/project/django-elastic-migrations/python3 setup.py tagto tag the new versiontwine upload-rpypidist/django-elastic-migrations-*.tar.gzUpdate new release at https://github.com/HBS-HBX/django-elastic-migrations/releasesChangelog0.8.2 (2018-11-20)fix#59 twine check error in 0.8.10.8.1 (2018-11-19)fix#50 add test coverage for es_listfix#58 ignore indexes with a dot in name in es_list –es-only and es_dangerous_reset0.8.0 (2018-11-13)fix#6 support Django 2fix#43 remove es_deactivatefix#44 add django 1.10 and 1.11 to test matrixfix#45 remove support for python 2In practice, Python 2 may work, but it is removed from the test matrix and won’t be updated0.7.8 (2018-11-13)fix#7 Convert Readme to rst for pypifirst release on PyPIupdate project dependencies0.7.7 (2018-09-17)fix#41 stack trace when indexing in py30.7.6 (2018-09-11)fix#36 es_update –start flag broken0.7.5 (2018-08-20)fix#35 open multiprocessing log in context handler0.7.4 (2018-08-15)fix#33 error when nothing to resume using –resume0.7.3 (2018-08-14)fix #31 es_update movies –newer –workers does not store worker information0.7.2 (2018-08-13)fix #21 wrong batch update total using multiprocessing in 0.7.1fix #23 KeyError _index_version_name in es_update –neweraddress #25 use pks for queryset inside workers #290.7.1 (2018-08-07)fixed gh #8 es_dangerous_reset –es-only to sync database to ESfixed gh #17 make es_dangerous_reset remove dem modelsimproved test coverageadded tests fores_create--es-onlyaddedIndexVersion.hard_delete()(not called by default)addedhard_deleteflag toDropIndexActionaddedhard_deleteflag toDEMIndexManager.test_post_teardown()updated__str__()ofIndexActionto be more descriptive0.7.0 (2018-08-06)fixed gh #5: “add python 3 support and tests”0.6.1 (2018-08-03)fixed gh #9: “using elasticsearch-dsl 6.1, TypeError in DEMIndex.save”0.6.0 (2018-08-01)Added test structure for py2 - GH #2Renamed default log handler fromdjango-elastic-migrationstodjango_elastic_migrations0.5.3 (2018-07-23)First basic release