Package: tableschema-elasticsearch
tableschema-elasticsearch-pyGenerate and load ElasticSearch indexes based onTable Schemadescriptors.Featuresimplementstableschema.StorageinterfaceContentsGetting StartedInstallationDocumentationUsage overviewMappingsCustom mappingsAPI ReferenceStorageContributingChangelogGetting StartedInstallationThe package use semantic versioning. It means that major versions  could include breaking changes. It's highly recommended to specifypackageversion range in yoursetup/requirementsfile e.g.package>=1.0,<2.0.pipinstalltableschema-elasticsearchDocumentationUsage overviewimportelasticsearchimportjsontableschema_esINDEX_NAME='testing_index'# Connect to Elasticsearch instance running on localhostes=elasticsearch.Elasticsearch()storage=jsontableschema_es.Storage(es)# List all indexesprint(list(storage.buckets))# Create a new indexstorage.create('test',{'fields':[{'name':'num','type':'number'}]})# Write data to indexl=list(storage.write(INDEX_NAME,({'num':i}foriinrange(1000)),['num']))print(len(l))print(l[:10],'...')l=list(storage.write(INDEX_NAME,({'num':i}foriinrange(500,1500)),['num']))print(len(l))print(l[:10],'...')# Read all data from indexstorage=jsontableschema_es.Storage(es)print(list(storage.buckets))l=list(storage.read(INDEX_NAME))print(len(l))print(l[:10])In this driverelasticsearchis used as the db wrapper. We can get storage this way:fromelasticsearchimportElasticsearchfromjsontableschema_elasticsearchimportStorageengine=Elasticsearch()storage=Storage(engine)Then we could interact with storage ('buckets' are ElasticSearch indexes in this context):storage.buckets# iterator over bucket namesstorage.create('bucket',descriptor,reindex=False,always_recreate=False,mapping_generator_cls=None)# reindex will copy existing documents from an existing index with the same name (in case of a mapping conflict)# always_recreate will always recreate an index, even if it already exists. default is to update mappings only.# mapping_generator_cls allows customization of the generated mappingstorage.delete('bucket')storage.describe('bucket')# return descriptor, not implemented yetstorage.iter('bucket')# yield rowsstorage.read('bucket')# return rowsstorage.write('bucket',rows,primary_key,as_generator=False)# primary_key is a list of field names which will be used to generate document idsWhen creating indexes, we always create an index with a semi-random name and a matching alias that points to it. This allows us to decide whether to re-index documents whenever we're re-creating an index, or to discard the existing records.MappingsWhen creating indexes, the tableschema types are converted to ES types and a mapping is generated for the index.Some special properties in the schema provide extra information for generating the mapping:arraytypes need also to have thees:itemTypeproperty which specifies the inner data type of array items.objecttypes need also to have thees:schemaproperty which provides a tableschema for the inner document contained in that object (or havees:enabled=falseto disable indexing of that field).Example:{"fields":[{"name":"my-number","type":"number"},{"name":"my-array-of-dates","type":"array","es:itemType":"date"},{"name":"my-person-object","type":"object","es:schema":{"fields":[{"name":"name","type":"string"},{"name":"surname","type":"string"},{"name":"age","type":"integer"},{"name":"date-of-birth","type":"date","format":"%Y-%m-%d"}]}},{"name":"my-library","type":"array","es:itemType":"object","es:schema":{"fields":[{"name":"title","type":"string"},{"name":"isbn","type":"string"},{"name":"num-of-pages","type":"integer"}]}},{"name":"my-user-provded-object","type":"object","es:enabled":false}]}Custom mappingsBy providing a custom mapping generator class (viamapping_generator_cls), inheriting from the MappingGenerator class you should be ableAPI ReferenceStorageStorage(self,es=None)Elasticsearch Tabular Storage.Package implementsTabular Storageinterface (see full documentation on the link):Only additional API is documentedArgumentses (object): ElasticSearch instancestorage.createstorage.create(self,bucket,descriptor,reindex=False,always_recreate=False,mapping_generator_cls=None,index_settings=None)Create index with mapping by schema.Argumentsbucket(str):
Name of index to be createddescriptor:
dDscriptor of index to be createdalways_recreate:
Delete index if already exists (otherwise just update mapping)reindex:
On mapping mismath, automatically create
new index and migrate existing indexes to itmapping_generator_cls:
subclass of MappingGeneratorindex_settings:
settings which will be used in index creationstorage.deletestorage.delete(self,bucket=None)Delete index with mapping by schema.Argumentsbucket(str): Name of index to deleteContributingThe project follows theOpen Knowledge International coding standards.Recommended way to get started is to create and activate a project virtual environment.
To install package and development dependencies into active environment:$makeinstallTo run tests with linting and coverage:$maketestChangelogHere described only breaking and the most important changes. The full changelog and documentation for all released versions could be found in nicely formattedcommit history.v1.0Initial driver implementation