inveniosoftware/invenio — pull request #2021 by fpoli (2014-08-01 14:18:48+00:00)
Title: BibIndex: refactoring and author parent tokenizer

* Refactoring of BibIndex:
implements inveniosoftware#1710

* Adds a generic parent tokenizer, useful for the dataset records:
implements inveniosoftware#1885

* Adds 4 demo records for testing purpose

* Adds (and fixes) tests

More details:

* Refactoring of `task_run_core` in `bibindex_engine.py`: easier to
follow, moves the decision of which records need to be reindexed to
the tokenizers

* Fixes bug: the command `bibindex -w [index name] -m [date range]`
previously was updating the `last_updated` field of the index

* Moves many of the index specific special cases from
`bibindex_engine.py` to the tokenizers
(e.g. `if index_name == "author": # add special tokens`)

* Moves some utility functions from `bibindex_engine.py` to
`bibindex_engine_utils.py`

* Adds tokenizers `BibIndexAbstractParentTokenizer`,
`BibIndexCanonicalAuthorTokenizer`,
`BibIndexExactParentAuthorTokenizer`,
`BibIndexParentAuthorTokenizer`,
`BibIndexParentCanonicalAuthorTokenizer`

* Adds indexes `simpleauthor`, `canonicalauthor`, `parentauthor`,
`parentcanonicalauthor`, `exactsimpleauthor`, `exactparentauthor`

* Adds fields `simpleauthor`, `exactsimpleauthor`

* Adds upgrade script for the database changes
(uses function upgrade_atlantis)

* Transforms `author` and `exactauthor` indexes from regular to
virtual index

* Adds file `bibindex_engine_tokenizer_regression_tests.py`

* Adds tests for tokenizers, for bibindex CLI, for bibindex utils

* Closes inveniosoftware#1662
(BibIndex: --reindex should not zero last_updated)

* Edits the commands run by `--load-demo-records` in order to
reindex the bibauthorid-dependant indexes after the `bibauthorid`
task (for example `canonicalauthor` and `parentcanonicalauthor`)