Package: oaiharvest
ContentsDescriptionAuthor(s)Latest VersionDocumentationRequirements / DependenciesInstallationBugs, Feature requests etc.Copyright And LicensingExamplesDescriptionA harvester to collect records from an OAI-PMH enabled provider.The harvester can be used to carry out one-time harvesting of all
records from a particular OAI-PMH provider by giving its base URL. It
can also be used for selective harvesting, e.g. to harvest only records
updated after, or before specified dates.To assist in regular harvesting from one or more OAI-PMH providers,
there’s a provider registry. It is possible to associate a short
memorable name for a provider with its base URLs, destination directory
for harvested records, and the format (metadataPrefix) in which records
should be harvested. The registry will also record the date and time of
the most recent harvest, and automatically add this to subsequent
requests in order to avoid repeatedly harvesting unmodified records.This could be used in conjunction with a scheduler (e.g. CRON) to
maintain a reasonably up-to-date copy of the record in one or more
providers.Examplesof how to accomplish these tasks are available
below.Author(s)John Harrison <john.harrison@liv.ac.uk> at theUniversity of LiverpoolLatest VersionThe latest release version is available in the Python Packages Index:https://pypi.python.org/pypi/oaiharvestSource code is under version control and available from:http://github.com/bloomonkey/oai-harvestDocumentationAll executable commands are self documenting, i.e. you can get help on
how to use them with the-hor--helpoption.At this time the only additional documentation that exists can be found
in this README file!Requirements / DependenciesPython>= 2.7 or Python 3.xpyoailxmlsqlite3Note that Python 3.x support requires pyoai 2.4.6+.As this release is not yet available on PyPI, usepip3 installgit+https://github.com/infrae/pyoai.gitPython3 support is still in beta and might have some bugs.InstallationUserspip installgit+http://github.com/bloomonkey/oai-harvest.git#egg=oaiharvestDevelopersI recommend that you usevirtualenvto isolate your development
environment from systemPythonand any packages that may be installed
there.InGitHub, fork the repositoryClone your fork:git clone git@github.com:<username>/oai-harvest.gitSetup development virtualenv using tox:pip install tox
tox -e devActivate development virtualenv:-nix:source env/bin/activateWindows:env\Scripts\activateBugs, Feature requests etc.Bug reports and feature requests can be submitted to the GitHub issue
tracker:http://github.com/bloomonkey/oai-harvest/issuesIf you’d like to contribute code, patches etc. please email the author,
or submit a pull request on GitHub.Copyright And LicensingCopyright (c)University of Liverpool, 2013-2014SeeLICENSE.rstfor licensing details.ExamplesHarvesting records from an OAI-PMH provider URLAll recordsoai-harvest http://example.com/oaiRecords modified since a certain dateoai-harvest --from 2013-01-01 http://example.com/oaiRecords from a named setoai-harvest --set "some:set" http://example.com/oaiLimiting the number of records to harvestoai-harvest --limit 50 http://example.com/oaiGetting help on all available optionsoai-harvest --helpOAI-PMH Provider RegistryAdding a provideroai-reg add provider1 http://example.com/oai/1If you don’t supply--metadataPrefixand--directoryoptions,
you will be interactively prompted to supply alternatives, or accept
the defaults.Removing an existing provideroai-reg rm provider1 [provider2]Listing existing providersoai-reg listHarvesting from OAI-PMH providers in the registryYou can harvest from one or more providers in the registry using the
short names that they were registered with:oai-harvest provider1 [provider2]By default, this will harvest all records modified since the last
harvest from each provider. You can over-ride this behavior using the--fromand--untiloptions.You can also harvest from all providers in the registry:oai-harvest allScheduling Regular HarvestingIn order to maintain a reasonably up-to-date copy of all the the
records held by those providers, one could configure a scheduler to
periodically harvest from all registered providers. e.g. to tell CRON
to harvest all at 2am every day, one might add the following to
crontab:0 2 * * * oai-harvest all