openstate/open-cultuur-data — pull request #67 by bartdegoede (2014-11-17 15:25:24+00:00)
Title: Backup and restore

Backup and restore
===============
This pull request contains changes to the API and ETL pipeline for supporting creating and restoring backups (also referred to as "dumps") of data collections, regardless of source and target. We can make dumps available as download, which can then easily be ingested into local API instances. For instance, it does not matter if the dump originates from another OCD API instance than your own, which makes it quicker for developers to get started.

CLI
---
A command line interface is included for creating dumps and loading dumps into the ETL-pipeline. There is support for downloading (for other API instances), creating (dumping your own collections), listing available collections, and loading data. For a more detailed usage explanation, please refer to the [included documentation](https://github.com/bartdegoede/open-cultuur-data/blob/backup-and-restore/docs/backup_and_restore.rst).

ETL-pipeline
----------------
The ETL-pipeline now provides [an extractor for dumps](https://github.com/bartdegoede/open-cultuur-data/blob/backup-and-restore/ocd_backend/extractors/staticfile.py#L98-122). It uses the default transformer, as most of the functionality of marshalling items is contained within Item classes anyway. Keeping with this pattern, there is a [`LocalDumpItem`](https://github.com/bartdegoede/open-cultuur-data/blob/backup-and-restore/ocd_backend/items/__init__.py#L237-303) that takes care of extracting the relevant fields from dumps. At this point, the Item is the same as all other items, so the same `ElasticsearchLoader` is used.

*N.B.:* In order to support the creation of the `combined_index` from collection-specific dumps, I had to include an extra field in each document containing the data from the combined index as string (which is [not indexed](https://github.com/bartdegoede/open-cultuur-data/commit/d26adb6414d55c8e55ef0ba1ad3f474180113e12?diff=unified)). Unfortunately, this means that before we can run this "in production", we have to reindex all data once this pull request is merged.

Documentation
--------------------
As briefly mentioned above, this pull request also contains [usage documentation](https://github.com/bartdegoede/open-cultuur-data/blob/backup-and-restore/docs/backup_and_restore.rst), as well as [API documentation](https://github.com/bartdegoede/open-cultuur-data/blob/backup-and-restore/docs/dev/api.rst) for the CLI.

Any comments, questions and improvements are appreciated :-)

Tests, tests and some more tests
=========================
@breyten, @justinvw: I took the liberty to provide a test-suite layout within the context of this pull request. It can be activated using the script @breyten initially used (thus not breaking Travis). The general idea is that there is a directory for the main packages (`ocd_backend` and `ocd_frontend`), which contains directories which contain test cases per module. In general, each module's `__init__.py` contains a base testcase, which can be extended for each submodule that is written. For example, in `tests.ocd_backend.items.__init__` a [`ItemTestCase`](https://github.com/bartdegoede/open-cultuur-data/blob/backup-and-restore/tests/ocd_backend/items/__init__.py#L5-18) is defined, which is subsequently extended in [`tests.ocd_backend.items.localdump`](https://github.com/bartdegoede/open-cultuur-data/blob/backup-and-restore/tests/ocd_backend/items/localdump.py#L9-83), providing item-specific tests.

I slightly modified the `run_tests.sh` script, in order to include all Python files, not only the ones that are touched during testing.

Testcases should be created for every Item that is already available in the repository, and I would even go as far as proposing to only accept pull-requests which provide testcases. Using this layout, it should be relatively easy to spot missing tests, and maintain existing tests, as the layout of the tests mirrors the project layout.