Package: elastic-wikidata
Elastic WikidataSimple CLI tools to load a subset of Wikidata into Elasticsearch. Part of theHeritage Connectorproject.Why?InstallationSetupUsageLoading from Wikidata dump (.ndjson)Loading from SPARQL queryTemporary side effectsWhy?Running text search programmatically on Wikidata means using the MediaWiki query API, eitherdirectlyorthrough the Wikidata query service/SPARQL.There are a couple of reasons you may not want to do this when running searches programmatically:time constraints/large volumes:APIs are rate-limited, and you can only do one text search per SPARQL querybetter search:using Elasticsearch allows for more flexible and powerful text search capabilities.*We're using our own Elasticsearch instance to do nearest neighbour search on embeddings, too.*CirrusSearchis a Wikidata extension that enables direct search on Wikidata using Elasticsearch, if you require powerful search and are happy with the rate limit.Installationfrom pypi:pip install elastic_wikidatafrom repo:Downloadcdinto rootpip install -e .Setupelastic-wikidata needs the Elasticsearch credentialsELASTICSEARCH_CLUSTER,ELASTICSEARCH_USERandELASTICSEARCH_PASSWORDto connect to your ES instance. You can set these in one of three ways:Using environment variables:export ELASTICSEARCH_CLUSTER=https://...etcUsing config.ini: pass the-cparameter followed by a path to an ini file containing your Elasticsearch credentials.Example here.Pass each variable in at runtime using options--cluster/-c,--user/-u,--password/-p.UsageOnce installed the package is accessible through the keywordew. A call is structured as follows:ew<task><options>Taskis either:dump:load data from Wikidata JSON dump, orquery:load data from SPARQL query.A full list of options can be found withew --help, but the following are likely to be useful:--index/-i: the index name to push to. If not specified at runtime, elastic-wikidata will prompt for it--limit/-l: limit the number of records pushed into ES. You might want to use this for a small trial run before importing the whole thing.--properties/-prop: a whitespace-separated list of properties to include in the ES index e.g.'p31 p21', or the path to a text file containing newline-separated properties e.g.this one.--language/-lang:Wikimedia language code. Only one supported at this time.Loading from Wikidata dump (.ndjson)ewdump-p<path_to_json><other_options>This is useful if you want to create one or more large subsets of Wikidata in different Elasticsearch indexes (millions of entities).Time estimate:Loading all ~8million humans into an AWS Elasticsearch index took me about 20 minutes. Creating the humans subset usingwikibase-dump-filtertook about 3 hours using itsinstructions for parallelising.Download the complete Wikidata dump (latest-all.json.gz fromhere). This is alargefile: 87GB on 07/2020.Usemaxlath'swikibase-dump-filterto create a subset of the Wikidata dump.Note: don't use the--simplifyflag when running the dump. elastic-wikidata will take care of simplification.Runew dumpwith flag-ppointing to the JSON subset. You might want to test it with a limit (using the-lflag) first.Loading from SPARQL queryewquery-p<path_to_sparql_query><other_options>For smaller collections of Wikidata entities it might be easier to populate an Elasticsearch index directly from a SPARQL query rather than downloading the whole Wikidata dump to take a subset.ew queryautomatically paginates SPARQL queriesso that a heavy query like'return all the humans'doesn't result in a timeout error.Time estimate:Loading 10,000 entities into Wikidata into an AWS hosted Elasticsearch index took me about 6 minutes.Write a SPARQL query and save it to a text/.rq file. Seeexample.Runew querywith the-poption pointing to the file containing the SPARQL query. Optionally add a--page_sizefor the SPARQL query.Temporary side effectsAs of version0.3.1refreshing the search index is disabled for the duration of load by default, asrecommended by ElasticSearch. Refresh is re-enabled to the default interval of1safter load is complete. To disable this behaviour use the flag--no_disable_refresh/-ndr.