Package: ckanext-resource-indexer
ckanext-resource_indexerDiscover more results in the Dataset search by searching through the content of resources.This extension indexes the content of files attached to resources. In this way
user has more chances to find the relevant results when using site search.The process if indexation can be customized for each file format viaresource
indexers. The following formats are supported out of the box:Plain textPDFJSONStructureInstallationConfigurationIndexersRegister own indexerBuilt-in indexersInstallationInstall the package as a CKAN extensionpipinstallckanext-resource-indexerAddresource_indexerto the list of enabled pluginsOptional. Enable built-in indexers by adding the following items to the list of enabled plugins:plain_resource_indexerpdf_resource_indexerjson_resource_indexerConfiguration# Make an attempt to index remote files(fetch into tmp folder# using URL)# (optional, default: false).ckanext.resource_indexer.allow_remote=1# Tiemeout for the attempt to download remote file# (optional, default: 2).ckanext.resource_indexer.remote_timeout=10# The size treshold(MB) for remote resources# (optional, default: 4).ckanext.resource_indexer.max_remote_size=4# List of resource formats(lowercase) that should be# indexed.# (optional, default: None)ckanext.resource_indexer.indexable_formats=txt pdf# Store the data extracted from resource inside specified field in the index.# If empty, store data inside the general-purpose `text` field.# (optional, default: text)ckanext.resoruce_indexer.index_field=extras_res_attachment# Boost matches by resource's content. Set values greater that 1 in order# to promote such matches and value between 0 and 1 in order to put such# matches further in search results. Works only when using custom index# field(`ckanext.resoruce_indexer.index_field`)# (optional, default: 1)ckanext.resoruce_indexer.search_boost=0.5##### Indexer specific option ################## Plain# Space-separated list of formats that can be indexed as a plain text# (optional, default: txt csv json yaml yml html)ckanext.resource_indexer.plain.indexable_formats=xml txt csv### PDF# Change a text from a single page before it added to the index# (optional, default: builtins:str)ckanext.resoruce_indexer.pdf.page_processor=custom.module:value_processor### JSON# Index JSON files as plain text(in addition to indexing as mapping)# (optional, default: false)ckanext.resoruce_indexer.json.add_as_plain=true# Change a key before it's used for patching the package dictionary# (optional, default: builtins:str)ckanext.resoruce_indexer.json.key_processor=custom.module:key_processor# Change a value before it's used for patching the package dictionary# (optional, default: builtins:str)ckanext.resoruce_indexer.json.value_processor=custom.module:value_processorIndexersIn order to extract the data from resources, this extension usesIndexers. These are CKAN plugins implementingIResourceIndexerinterface.For every resource with the format specified byckanext.resource_indexer.indexable_formatsconfig option, an appropriate
indexer is searched. If no indexers were found(or resource format is missing
from theckanext.resource_indexer.indexable_formatsconfig option), the
resource is skipped.:information_source: Indexation can be temporarily disabled using one of the
following approaches:Set environment variableCKANEXT_RESOURCE_INDEXER_BYPASS(any non-empty
value), and the plugin won't interfer into standard dataset indexation
process.Useckanext.resource_indexer.utils.disabled_indexationcontext manager:withdisabled_indexation():here_indexation_does_not_happen()here_indexation_happens()Every indexer has weight(priority). Indexer with the highest weight will be
used to index the resource.Indexation consists of two steps:meaningful data segments extracted from the resourcethese data segments are merged into the package dictionary consumed by the
search engine(Solr) for indexingIt means, that the format of extracted segments must be compatible with the
merging logic from the second step. But other than that, there are no
particular requirements for the format of extracted data.Data extraction happens locally. If the resource was uploaded to the local
filesystem, data is extracted directly from the resource's file. If the
resource is stored remotely(either uploaded to the cloud or linked via remote
URL), it can be temporarily downloaded to the local filesystem and removed
after processing. By default, non-local resources are ignored, but this can be
changed viackanext.resource_indexer.allow_remoteconfig option.Register own indexerImplementckanext.resource_indexer.interface.IResourceIndexerby providing following methods:classCustomIndexerPlugin(plugins.SingletonPlugin):plugins.implements(IResourceIndexer)defget_resource_indexer_weight(self,resource:dict[str,Any])->int:"""Define priority of the indexerArgs:resource: resource's detailsReturns:the weight of the indexerExpected values:0: skip handler10: use handler if no other handlers found20: use handler as a default one for the resource30: use handler as an optimal one for the resource40: use handler as a special-case handler for the resource50: ignore all the other handlers and use this one instead"""returnWeight.fallbackdefextract_indexable_chunks(self,path:str)->Any:"""Extract indexable data from the resourceThe result can have any form as long as it can be merged into thepackage dictionary by implementation of `merge_chunk_into_index`.Args:path: path to resource fileReturns:all meaningfuld pieces of data with no type assumption"""return[]defmerge_chunks_into_index(self,pkg_dict:dict[str,Any],chunks:Any):"""Merge data into the package dictionary.Args:pkg_dict: package that is going to be indexedchunks: collection of data fragments extracted from resourceReturns:all meaningfuld pieces of data with no type assumption"""passBuilt-in indexersPlain indexerIndex formats specified byckanext.resource_indexer.indexable_formatsif they fall into the value ofckanext.resource_indexer.plain.indexable_formatsconfig option, unless other handler with a non-fallback weight(>10) found.Resources are indexed as-is. File is read and sent to the index without any additional changes.Enable it by addingplain_resource_indexerto the list of enabled plugins.PDF indexerExtract and index text from the PDF file.In order to enable it:install current extension with thepdfextra:pipinstall'ckanext-resource-indexer[pdf]'addpdf_resource_indexerto the list of enabled plugins andinstall system packages for PDF processing. This will be different depending on your system. Examples:CentOSyuminstall-ypulseaudio-libs-devel\gcc-c++pkgconfig\python3-devel\libxml2-devellibxslt-devel\popplerpoppler-utilspoppler-cpp-develDebianaptinstallbuild-essentiallibpoppler-cpp-devpkg-configpython3-devmacOSbrewinstallpkg-configpopplerpythonIf PDF content requires preprocessing, specify function that converts text from
a every separate as ackanext.resoruce_indexer.pdf.page_processor. It uses
standard import-string format:module.import.path:functionJSON indexerRead a dictionary from the JSON file, convert all non-string values into
strings(i.e, no nested values allowed), and apply it as a patch to the indexed
dataset.Optionally, ifckanext.resoruce_indexer.json.add_as_plainflag enabled, index
the content of the file as a plain-text(similar to theplain
indexer)If key or value requires preprocessing, specify function that converts data as
ackanext.resoruce_indexer.json.key_processororckanext.resoruce_indexer.json.value_processor. It uses standard import-string
format:module.import.path:functionEnable it by addingjson_resource_indexerto the list of enabled plugins.