Package: collective.elastic.ingest
Ingestion service queue runner between Plone RestAPI and ElasticSearch 8+ or OpenSearch 2+.
Provides Celery-tasks to asynchronous index Plone content.auto-create Open-/ElasticSearch…indexmapping from Plone schema using a flexible conversions file (JSON),ingest-attachment pipelines using (same as above) file.task toindex a content object with all data given plus allowedRolesAndUsers and section (primary path)unindex an content objectconfigure from environment variables:celery,elasticsearch or opensearchsentry logging (optional)Table of ContentsInstallationConfigurationEnvironmentUpgradeJSON-Filesmappings.jsonpreprocessings.jsonanalysis.jsonStart upOCI Image usageExamplesOpenSearch with Docker ComposeElasticSearch with Docker ComposeLocal/ DevelopmentComplex Mapping With German Text AnalysisText AnalysisSource CodeInstallation for developmentLicenseInstallationWe recommended to use a Python virtual environment, create one withpython3-mvenv venv, and activate it in the current terminal session withsource venv/bin/activate.Installcollective.elastic.ingestready to use with redis and opensearch:pip install collective.elastic.ingest[redis,opensearch]Depending on the queue server and index server used, the extra requirements vary:queue server:redisorrabbitmq.index server:opensearchorelasticsearch.ConfigurationConfiguration is done via environment variables and JSON files.EnvironmentEnvironment variables are:INDEX_OPENSEARCHWhether to use OpenSearch or ElasticSearch.Default: 1INDEX_SERVERThe URL of the ElasticSearch or OpenSearch server.Default: localhost:9200INDEX_USE_SSLWhether to use a secure TLS connection or not.Default: 0INDEX_VERIFY_CERTSWhether to verify TLS certificates on secure connection or not.Default: 0INDEX_SSL_SHOW_WARNWhether to warn for unverified TLS request is made or not.Default: 0INDEX_SSL_ASSERT_HOSTNAMEWhether to assert the hostname in TLS request or not.Default: 0INDEX_LOGINUsername for the ElasticSearch 8+ or OpenSearch server.Default: adminINDEX_PASSWORDPassword for the ElasticSearch 8+ or OpenSearch server.Default: adminCELERY_BROKERThe broker URL for Celery.
Seedocs.celeryq.devfor details.Default:redis://localhost:6379/0PLONE_SERVICEBase URL of the Plone ServerDefault:http://localhost:8080PLONE_SITE_PREFIX_PATHPath to the site to index at the Plone Server.Default:PlonePLONE_SITE_PREFIX_METHODWether to keep the prefix path while requesting the content from the Plone Server or not.
Allowed values:strip,keepOnkeep, the prefix path is kept in the index/schema path.
If thePLONE_SERVICEruns underhttp://localhost:8080and thePLONE_SITE_PREFIX_PATHisPlone,
the index/schema base path where the ingest service fetches its data and schema from ishttp://localhost:8080/Plone.Onstrip, the prefix path is stripped from the index/schema path.
If thePLONE_SERVICEruns underhttps://www.mydomain.tldand thePLONE_SITE_PREFIX_PATHisPlone,
the index/schema base path ishttps://www.mydomain.tld.Default:keepPLONE_USERUsername for the Plone Server, needs to have at least Site Administrator role.Default: adminPLONE_PASSWORDPassword for the Plone Server.Default: adminMAPPINGS_FILEAbsolute path to the mappings configuration file.
Configures field mappings from Plone schema to ElasticSearch.No default, must be given.PREPROCESSINGS_FILEConfigures preprocessing of field values before indexing.Default: Uses a defaults file of this package.ANALYSIS_FILE(optional) Absolute path to the analysis configuration file.SENTRY_DSN(optional) Sentry DSN for error reporting.Default: disabledSENTRY_INGEST(optional) Enable sentry reporting in Celery.
Reason behind this is, SENTRY_DSN_DSN is possibly provided in a Plone environment when this package is used as a library.
To not override any existing sentry-sdk initialization, this flag is used to enable sentry reporting specifically in ingest mode.
Allowed values: true, falseDefault: falseUpgradeComing from version 1.x of this package, in 2.x you need to change some names of the environment variables.ELASTICSEARCH_INGEST_*toINDEX_*OPENSEARCH*toINDEX_OPENSEARCHPLONE_PATHtoPLONE_SITE_PREFIX_PATH.
Additional a new optionPLONE_SITE_PREFIX_METHOD=stripto strip the path prefix from the index/schema path is available.
See above for details.If you use Sentry, additionalSENTRY_INGEST=trueis needed.JSON-Filesmappings.jsonThe mappings file is a JSON file with the following structure:First level:Key: ValuePairsThe key is
- either a fully qualified field name (path) to the field in the schema (behaviors/...ortypes/...), likebehaviors/plone.basic/title.
- or the dotted name of a zope.schema based field type, likeplone.namedfile.field.NamedBlobImage.The value is an instruction how to map this specific field or field type to OpenSearch or ElasticSearch.
The actual mapping send to the index server is generated from this instruction and the full schema fetched from Plone.
At generation time, the process iterates over the full schema and applies the mapping instructions to each field.At first the instruction lookup is done by the fully qualified field name.
If no instruction is found, the dotted name of the field type is used.There are two types of instructions: Simple ones and complex ones.Thesimple instructionhas thetypedefined as a top level key.
The type is the mapping type defined by the index server for the mapping, liketextorboolean.
For some types this is enough, others take additional keys.
Thenestedtype is such a type.
Here the keyspropertiesanddynamicare required.
Those keys are provided on top level.Thecomplex instructionhas thetypedefined in thedefinitionkey.
Thedefinitionkey is a mapping with thetypekey and the same additional keys for the definition of the field type as for the simple one.
There are two other possible top-level keys for complex instructions:detectionsandpipelines.Adetectionis a method to do something based on the schemas field parameters.
At the moment this is only used to detect avalue_typeof a Plone list field or similar.
This detector is registered asreplace.Apipelineis a method to add a processing pipeline to the field.
Those are used to ingest binary data like images or PDFs, but any other pipeline can be configured.
The pipeline is registered and executed.
The configuration of a pipeline consists of asource, atarget,typeas above for defining the target data,processors, and anexpansion.source is the field name with the input data for the pipeline.target is the field name for the output data of the pipeline.type is the definition of the target field.processors are a list of processors to apply to the data.expansion not directly mapping related, but configured here as it defines where in a postprocessing step the data is fetched from.
Binary data is not provided in the content data, only a link where to download.preprocessings.jsonPre-processings are steps done before anything else is processed.
They run on the raw data from the Plone REST API, the full schema fetched from the Plone backend, and the full content object fetched from the Plone backend.
Each preprocessing is a function that takes the data and modifies the full schema or full content.The pre-processings-file consists of list a processing instructions records.Each record is a mapping  with amatch, anactionand aconfiguration.The match call an function that returns a boolean value.
If the value is true, the action is executed, otherwise skipped.There are two matches availablealwaysAlways matches.Example configuration{"match": {"type":"always"},...}This is the default if no match is given.content_existsMatches if the fieldconfiguration["path"]is present in the content data.
Path can point to a fieldfooor check for its sub entries likefoo/bar/baz.Example configuration{"match": {"type": "content_exists", "path":"foo"},...}The action is a function that takes the full schema and content data, the configuration, and then modifies the full schema or full content.These actions ar available:additional_schemaAdds an additional schema to the full schema.
The configuration must a valid schema, i.e. with name and field, to add.
In the resulting mapping the schema is added to a (new) sectionadditionalunder the subsectionpreprocessed.rewriteMoves content data from one position in the field-tree to another.
The configuration must be a mapping withsourceandtargetkeys.
The value ofsourceis the path to the data to move.
The value oftargetis the path to the new location of the data (missing containers are created).
The value ofenforceis a boolean value (default: False). If True, the source must exist, otherwise an error is raised.Example:"configuration": {"source": "@components/collectiveelastic/blocks_plaintext",  "target": "blocks_plaintext", "enforce": false}removeDeletes a field or sub-field from the content data.
The value oftargetis the path to the data to delete.field_removeDeletes a field from the full schema and its field value from the content.
The value ofsectionis the section (one ofbehaviorsortypes)
The value ofnameis the name of the behavior or type.
The value offieldis the name of the field to delete.full_removeDeletes a full behavior or type with all its fields from the full schema and its fields values from the content.
The value ofsectionis the section (one ofbehaviorsortypes)
The value ofnameis the name of the behavior or type.remove_emptyDeletes all empty fields from the content data.
A field is considered empty if it isNone,[],{}or""analysis.jsonThis file provides the index with analyzers to be used in themappings.jsonfiles different definition sections (top-level, nested, complex or pipeline target).Read more on this topic in the dedicated section below.Start upRun celery worker:celery -A collective.elastic.ingest.celery.app worker -c 1 -l infoOr with debug information:celery -A collective.elastic.ingest.celery.app worker -c 1 -l debugThe number is the concurrency of the worker.
For production use, it should be set to the number of Plone backends available for indexing load.OCI Image usageFor use in Docker, Podman, Kubernetes, …, an OCI image is provided at theGithub Container Registry.The environment variables above are used as configuration.Additional the following environment variables are used:CELERY_CONCURRENCYThe number of concurrent tasks to run.Default: 1CELERY_LOGLEVELThe log level for celery.Default: infoTheMAPPINGS_FILEvariable defaults to/configuration/mappings.json.
By default no file is present.When a mount is provided to/configuration, the mappings file can be placed there.
Alternatively, the mappings file can be provided as aconfigs element in docker composeor as aconfigmapin Kubernetes.ExamplesExample configuration files are provided in the./examplesdirectory.OpenSearch with Docker ComposeLocation:examples/docker-os/*A docker-compose filedocker-compose.ymland aDockerfileto start an Ingest, Redis and an OpenSearch server with dashboard is provided.Precondition:Docker and docker-compose are installed.Max virtual memory map needs increase to run this:sudo sysctl-wvm.max_map_count=262144(not permanent,see StackOverflow post).enter the directorycd examples/dockerSteps to start the example OpenSearch Server with theingest-attachmentplugin installed:locally build the custom OpenSearch Docker image enriched with the plugin using:docker buildx use default
docker buildx build --tag opensearch-ingest-attachment:latest Dockerfilestart the cluster withdocker-composeup.Now you have an OpenSearch server running onhttp://localhost:9200and an OpenSearch Dashboard running onhttp://localhost:5601(user/pass: admin/admin).
The OpenSearch server has theingest-attachmentplugin installed.
The plugin enables OpenSearch to extract text from binary files like PDFs.A Redis server is running onlocalhost:6379.Additional the ingest worker runs and is ready to index content from a Plone backend.Open another terminal.In another terminal windowrun a Plone backendathttp://localhost:8080/Plonewith the add-oncollective.elastic.ploneinstalled.
There, create an item or modify an existing one.
You should see the indexing task in the celery worker terminal window.For production use, pleasecheck that the port 9200 is not exposed to the internet.
For a good measure block it with a firewall rule.ElasticSearch with Docker ComposeLocation:examples/docker-es/*A docker-compose filedocker-compose.ymlto start an Ingest, Redis and an ElasticSearch server with Dejavu dashboard is provided.Precondition:Docker and docker-compose are installed.Max virtual memory map needs increase to run this:sudo sysctl-wvSITE_PREFIXm.max_map_count=262144(not permanent,see StackOverflow post).enter the directorycdexamples/docker-esRun the cluster with:source .env
docker compose upFirst you need to set the passwords for the ElasticSearch, execute the following command and note the passwords printed in a new terminal:docker exec -it docker-es-elasticsearch-1 /usr/share/elasticsearch/bin/elasticsearch-setup-passwords autoFind the password for the userelasticand set it in the environment variableINDEX_PASSWORDin the.envfile.
Stop the cluster (Ctrl-C),source .envwith the new settings and start it again (as above).Now you have an ElasticSearch server running onhttp://localhost:9200and an Dejavu Dashboard running onhttp://localhost:1358.
(The ElasticSearch server has theingest-attachmentplugin installed by default).A Redis server is running onlocalhost:6379.Additional the ingest worker runs and is ready to index content from a Plone backend.Open another terminal.In another terminal windowrun a Plone backendathttp://localhost:8080/Plonewith the add-oncollective.elastic.ploneinstalled.
There, create an item or modify an existing one.
You should see the indexing task in the celery worker terminal window.For production use, pleasecheck that the port 9200 is not exposed to the internet.
For a good measure block it with a firewall rule.Local/ DevelopmentLocation:examples/docker/local/*A very basic mappings fileexamples/docker/local/mappings.jsonis provided.
To use it setMAPPINGS_FILE=examples/mappings-basic.jsonand then start the celery worker.
An environment fileexamples/docker/local/.envis provided with the environment variables ready to use for local startup.Runsourceexamples/.envto load the environment variables.
Then start the celery worker withcelery-Acollective.elastic.ingest.celery.app worker-ldebug.Complex Mapping With German Text AnalysisLocation:examples/docker/analysis/*A complex mappings file with german text analysis configured,mappings-german-analysis.jsonis provided.
It comes together with the matching analysis configuration fileanalysis-german.jsonand a stub lexicon fileelasticsearch-lexicon-german.txt.
Read the next section for more information about text analysis.Text AnalysisTest analysis is optional.
Skip this on a first installation.Search results can be enhanced with a tailored text analysis.
The simple fuzzy search, which can be used without any analysis configuration, has its limits.
This is even more true in complex languages like German.This is an advanced topic.You can find detailed information abouttext analysis in the ElasticSearch documentation.
We provide an example analysis configuration for a better search for German compounded words.Example: A document with the string ‘Lehrstellenbörse’ can be found by querying ‘Lehrstelle’.
It shall be found too by querying ‘Börse’ using adecompounderwith a word list ‘Lehrstelle, Börse’ and an additionalstemmer.
The example analyzer configuration applies astemmer, which can handle inflections of words.
This is an important enhancement for better search results.The analysis configuration is a configuration of analyzers.
The example provided here uses two of them:german_analyzerandgerman_exact.The first decompounds words according the word list inlexicon.txt.
Astemmeris added.The second one is to allow also exact queries with a quoted search string.These two analyzers are to be applied to fields.
You can apply them in your mapping.Example:"behaviors/plone.basic/title": {
    "type": "text",
    "analyzer": "german_analyzer",
    "fields": {
        "exact": {
            "type": "text",
            "analyzer": "german_exact_analyzer"
        }
    }
},Check your configured analysis with:POST {{elasticsearchserver}}/_analyze

{
    "text": "Lehrstellenbörse",
    "tokenizer": "standard",
    "filter": [
        "lowercase",
        "custom_dictionary_decompounder",
        "light_german_stemmer",
        "unique"
    ]
}The response delivers the tokens for the analyzed text ‘Lehrstellenbörse’.Note: The fileelasticsearch-lexicon.txtwith the word list used by thedecompounderof the sample analysis configuration inanalysis.json.examplehas to be located in the configuration directory of your elasticsearch server.Source CodeThe sources are in a GIT DVCS with its main branches atgithub.
There you can report issues too.We’d be happy to see many forks and pull-requests to make this addon even better.Maintainers areJens Klein,Katja Suessand the BlueDynamics Alliance developer team.
We appreciate any contribution and if a release is needed to be done on PyPI, please just contact one of us.
We also offer commercial support if any training, coaching, integration or adaptions are needed.Installation for developmentclone source code repository,enter repository directoryrecommended: create a Virtualenvpython-mvenvenvdevelopment install./bin/env/pip install-e.[test,redis,opensearch]load environment configurationsourceexamples/.env.LicenseThe project is licensed under the GPLv2.