Package: oarepo-oai-pmh-harvester
OARepo OAI-PMH harvesterAn OAI-PMH harvesing library for Invenio 3.5+. The library provides initial transformation
of OAI-PMH payload to an intermediary json representation which is later on transformed by
a specific transformer to the format of invenio records.Due to their generic nature, these transformers are not part of this library but have to be
provided by an application.The progress and transformation errors are captured within the database.For now, the library does not provide error notifications, but these will be added. Sentry might
be used for the logging & reporting.Installationpipinstalloarepo-oaipmh-harvester
pipinstall<yourtransformerlibrary>ConfigurationAll configuration is inside theOAIHarvesterRecordrecord.
There is a command-line tool to add a new record:inveniooarepooaiharvesteraddnusl\--name"NUSL harvester"\--urlhttp://invenio.nusl.cz/oai2d/\--setglobal\--prefixmarcxml\--loadersickle\--transformermarcxml\--transformernusl\--writer'service{service=nr_documents}'This will register an oai-pmh harvester with code "nusl",
its url, oai set and metadata prefix. Records from this
harvester will be loaded with the sickle loader (default loader
if not specified) and at first transformed from marcxml to
json format and subsequently by NUSL transformer to get
nr_documents compatible json.The json is then used by service writer to create/update
the target record.UsageCommand-lineOn command line, invokeinveniooarepooaiharvesterrunnusl<optionallistofoaiidentifierstoharvest>Options:--all-records      Re-harvest all records, not from the last timestamp
  --on-background    Run harvest on background (via celery task)
  --identifier       Harvest the passed identifier/sYou can also pass arguments for "havester add" to override the defaults from the configuration.Harvest statusHarvester uses 3 additional types of record:Run represents a single run of the harvesterBatch - the run consists of multiple batches, that is records, that are processed together.
Each such batch is represented in thebatchrecordOAI record is a link between harvested record and its original. If there are errors harvesting
the record, it is still created and itserrorsfield is filled with the errorCustom parsers and transformersTransformerA piece of code that gets a StreamEntry (or a StreamBatch) instance, processes it and returns modified
StreamEntry. An example is a MarcXML transformer that takes the string with xml representation of the
entry and transforms it into simple json representation{abcxy: value(s)}, where abcxy is marc field code.Seeoarepo_runtime.datastream.transformerspackage for StreamEntry/StreamBatch interfaces.The transformer needs to be registered:# mypkg.transformersfrom.implimportMyTransformermy_transformer={"class":MyTransformer,"params":{# default parameters that will go to the MyTransformer constructor}}And setup.cfg:# setup.cfg[options.entry_points]oarepo.oaipmh.transformers=my_transformer=mypkg.transformers:my_transformerThen you can usemy_transformerwhen creating your harvester.ReaderA reader is responsible for fetching records and creating a stream of StreamEntry items.
Seeoarepo_runtime.datastreams.readersfor details. Then register the reader intooarepo.oaipmh.readersentry point with the same syntax as above.