inveniosoftware/invenio — pull request #2601 by aw-bib (2014-12-01 16:18:39+00:00)
Title: Fix oai harvesting for name space qualified xml

OpenAIRE exposes the XML files with full name spaces. This breaks invenios harvesting routines which rely on regexp matching of the returned answers.

Two issues arise:
1. `oai_harvest_getter.py` checks, if any records are returned by searching for a `` line. This fails, if this line reads `` instead. The _fix_ uses the `noRecordsMatch` error return in OAI in case no records match the harvesting criteria. (cf. http://www.openarchives.org/OAI/openarchivesprotocol.html#ErrorConditions)
2. `oai_harvest_utils.py` uses a function to dedupe returned records by means of their OAI-ID. This also lives on regexping. In the OpenAIRE usecase this results in completely invalid XML, as only the first `len()` chars of the header get written at all (`header_index_end = data.find("") + len("")`. Secondly, all subsequent regexps fail, as they never match due to the namespace qualifier. The latter can in principle be arbitrary. The _fix_ introduces `lxml.etree` parsing of the document to extract the OAI-IDs by means of `XPath` statements. Additionally, only thos records get written to the result file that were not already done. Searching for records is also implemented by `XPath`, building of the final element uses `lxml` factories.

Combined they should address and fix #2300 also for harvesting from OpenAIRE.

@jlavik and @kaplun for review. Backported code 1.1 maint is available as well.