BibIndex: move helper functions to utils file

* Moves some of the helper functions from bibindex_engine
  to bibindex_engine_utils file.

Signed-off-by: <grzegorz.szpura@cern.ch>
Tested-by: Tibor Simko <tibor.simko@cern.ch>
BibIndex: WordTable API changes

* Removes 'fields_to_index' argument from WordTable's argument
  list.

Signed-off-by: <grzegorz.szpura@cern.ch>
Tested-by: Tibor Simko <tibor.simko@cern.ch>
BibIndex: abstraction layer for terms retrieval

* Introduces new TermCollector class which
  handles terms retrieval from different
  part of the system.

Signed-off-by: Grzegorz Szpura <grzegorz.szpura@cern.ch>
Tested-by: Tibor Simko <tibor.simko@cern.ch>
BibIndex: indexing non-MARC standards

* Introduces NonMarcTermCollector for acquiring data
  from non-MARC records.

* Adds new column to tag table and inserts
  corresponding RecJson fields as another
  non-MARC value for 'tag'.

* Enhances test suite acordingly.

Signed-off-by: Grzegorz Szpura <grzegorz.szpura@cern.ch>
Tested-by: Tibor Simko <tibor.simko@cern.ch>
BibIndex: recjson fields in admin interface

* Admin interface shows recjson fields
  in logical fields overview.

* Fields and tags are shown directly in index
  overview.

* Edition of tag enhanced with option to
  change value for recjson field.

Signed-off-by: Grzegorz Szpura <grzegorz.szpura@cern.ch>
Tested-by: Tibor Simko <tibor.simko@cern.ch>
Merge branch 'master' into next

* master:
  BibIndex: recjson fields in admin interface
  BibIndex: indexing non-MARC standards
  BibIndex: abstraction layer for terms retrieval
  BibIndex: WordTable API changes
  BibIndex: move helper functions to utils file

* Merge note: creation of fixtures and  movement of new files will be
  done in the next commits.

Conflicts:
	invenio/legacy/bibindex/adminlib.py
	invenio/legacy/bibindex/engine.py
	invenio/legacy/bibindex/engine_utils.py
	invenio/legacy/search_engine/__init__.py
	invenio/modules/indexer/testsuite/test_indexer_engine.py
	invenio/modules/indexer/tokenizers/BibIndexAuthorCountTokenizer.py
	invenio/modules/indexer/tokenizers/BibIndexDOITokenizer.py
	invenio/modules/indexer/tokenizers/BibIndexJournalTokenizer.py
	invenio/modules/records/testsuite/fields/atlantis.cfg
	invenio/modules/upgrader/upgrades/invenio_2013_03_20_idxINDEX_synonym_kb.py
	modules/bibindex/lib/Makefile.am
	modules/bibindex/lib/bibindex_regression_tests.py
	modules/bibupload/lib/bibupload_regression_tests.py
	modules/websearch/lib/websearch_regression_tests.py

Signed-off-by: Esteban J. G. Gabancho <esteban.gabancho@gmail.com>
bibindex: move new files to legacy and fix imports

Signed-off-by: Esteban J. G. Gabancho <esteban.gabancho@gmail.com>
records: move new recordext function to records

Signed-off-by: Esteban J. G. Gabancho <esteban.gabancho@gmail.com>
search: update of fixtures and models

* Updates `Tag` fixtures and models accordingly to the new indexer
  capabilities to index non-marc records.

Signed-off-by: Esteban J. G. Gabancho <esteban.gabancho@gmail.com>
BibField: fix copyright field names

* Fixes a typo for one of the copyright fields and also changes its name
  to `copyright_status` to avoid duplicate names.  (closes #1933)

Signed-off-by: Esteban J. G. Gabancho <esteban.gabancho@gmail.com>
Reviewed-by: Tibor Simko <tibor.simko@cern.ch>
Merge branch 'master' into next

* master:
  BibField: fix copyright field names
deposit: fieldlist adaptation for import

Signed-off-by: Kamil Neczaj <kamil.neczaj@cern.ch>
deposit: removal of alert attribute

* Removes useless alert attribute
  appended to the class.

Signed-off-by: Pedro Gaudêncio <pedro.gaudencio@cern.ch>
Reviewed-by: Javier Martin Montull <javier.martin.montull@cern.ch>
deposit: fix of field messages

* Fixes field info/warning/error/success
  messages, according to Bootstrap's newest
  documentation; also, adds a variable that
  prepares for future changes on the 'error'
  field name.

* Updates to $.map() for better browser
  integration.

Signed-off-by: Pedro Gaudêncio <pedro.gaudencio@cern.ch>
Reviewed-by: Javier Martin Montull <javier.martin.montull@cern.ch>
utils: fix of arxiv validator function

* The same regex was being verified twice,
  instead of checking the pre and post 2007,
  only the post one was being checked.

Signed-off-by: Pedro Gaudêncio <pedro.gaudencio@cern.ch>
Reviewed-by: Javier Martin Montull <javier.martin.montull@cern.ch>
utils: addition of DOI json data fetch

* Adds JSON data response for a given DOI.

* Adds etree to dictionary translation on
  processor utils.

* Corrects get_marcxml_for_doi() misuse of
  plain text params in the url.

* Needs the user's email as a configuration
  variable.

Signed-off-by: Pedro Gaudêncio <pedro.gaudencio@cern.ch>
Reviewed-by: Javier Martin Montull <javier.martin.montull@cern.ch>
deposit: addition of holding pen

* This task adds the deposition to the
  Holding Pen so it gets approved or
  rejected by an administrator.

Signed-off-by: Pedro Gaudêncio <pedro.gaudencio@cern.ch>
Reviewed-by: Javier Martin Montull <javier.martin.montull@cern.ch>
deposit: API for Crossref DOI fetch

* Searches the DOI on crossref and returns the result as JSON format,
  also handles if the DOI isn't found or it's malformed.

Signed-off-by: Pedro Gaudêncio <pedro.gaudencio@cern.ch>
Reviewed-by: Javier Martin Montull <javier.martin.montull@cern.ch>
deposit: addition of block tags to template

* Adds block tags to run_base.html template
  to improve costumization.

Signed-off-by: Pedro Gaudêncio <pedro.gaudencio@cern.ch>
Reviewed-by: Javier Martin Montull <javier.martin.montull@cern.ch>
deposit: PEP8 fixes

Signed-off-by: Pedro Gaudêncio <pedro.gaudencio@cern.ch>
Reviewed-by: Javier Martin Montull <javier.martin.montull@cern.ch>