BibIndex: move helper functions to utils file

* Moves some of the helper functions from bibindex_engine
  to bibindex_engine_utils file.

Signed-off-by: <grzegorz.szpura@cern.ch>
Tested-by: Tibor Simko <tibor.simko@cern.ch>
BibIndex: WordTable API changes

* Removes 'fields_to_index' argument from WordTable's argument
  list.

Signed-off-by: <grzegorz.szpura@cern.ch>
Tested-by: Tibor Simko <tibor.simko@cern.ch>
BibIndex: abstraction layer for terms retrieval

* Introduces new TermCollector class which
  handles terms retrieval from different
  part of the system.

Signed-off-by: Grzegorz Szpura <grzegorz.szpura@cern.ch>
Tested-by: Tibor Simko <tibor.simko@cern.ch>
BibIndex: indexing non-MARC standards

* Introduces NonMarcTermCollector for acquiring data
  from non-MARC records.

* Adds new column to tag table and inserts
  corresponding RecJson fields as another
  non-MARC value for 'tag'.

* Enhances test suite acordingly.

Signed-off-by: Grzegorz Szpura <grzegorz.szpura@cern.ch>
Tested-by: Tibor Simko <tibor.simko@cern.ch>
BibIndex: recjson fields in admin interface

* Admin interface shows recjson fields
  in logical fields overview.

* Fields and tags are shown directly in index
  overview.

* Edition of tag enhanced with option to
  change value for recjson field.

Signed-off-by: Grzegorz Szpura <grzegorz.szpura@cern.ch>
Tested-by: Tibor Simko <tibor.simko@cern.ch>
DocExtract: preload kbs on wsgi load

Co-authored-by: Alessio Deiana <alessio.deiana@cern.ch>
Signed-off-by: Alessio Deiana <alessio.deiana@cern.ch>
PdfChecker: skip records without unique ids in 037 tags
WebSearch: add more info on verbose output
BibIndex: special journal page handling

* Improves the standard Journal tokenizer to keep page-end
  when indexing, as well as adding new journal,volume pairs.

* Amends the search_engine to specially handle searches in
  the journal field with page ranges to make use of the new
  tokenized strings. This fixes issues seen in BibRank and
  other search engine clients that passes a valid full page
  range and obtains no result (because it was not indexed!).

Co-authored-by: Alessio Deiana <alessio.deiana@cern.ch>
WebSearch: special journal search fallback

* For certain journal search queries with INSPIRE style page
  ranges the user is now presented with an alternative
  result without the page-end part of the query, should the
  original query return no results.

Co-authored-by: Alessio Deiana <alessio.deiana@cern.ch>
BibCheck: improvements in DOI checks

* Sends only start page to crossref, not the range
BibAuthorID: hotfix in author-hepnames connection

* Session is now correctly initialized before use.
BibAuthorID: fix in WAPAFF format

* Now handles correctly repeatable affiliations
BibCheck: add $$9 bibcheck to dois added
BibAuthorID: javascript ticket system optimization

* Optimised ticketing system sync, significantly reduces server polling load.
datacher: minimum caching time

* Introduces a new parameter in the datacacher that defaults to 30s.
  This is the mininum time the data is cached. That means that calling
  recreate_cache_if_needed() does nothing if it was called less than 30s ago.
BibRank: details citations diff

* When bibrank -w citation fails because we lost too many citations,
  displays in the log for each record how many citations they lost.
BibIndex: bibauthorid performance improvements

* Re-implements get_author_canonical_ids_for_recid() to bypass
  complex BibAuthorID API usage and directly retrieve data
  from the DB for a gain of speed of ~800x.

Signed-off-by: Samuele Kaplun <samuele.kaplun@cern.ch>
Co-authored-by: Alessio Deiana <alessio.deiana@cern.ch>
Co-authored-by: Samuele Carli <samuele.carli@cern.ch>
BibEdit: holdingPen fix

*  When merging a record with a template, if the record is a hp record
   make all its subfields volatile.

Reviewed-by: Javier Martin Montull <javier.martin.montull@cern.ch>
BibAuthorId: Fixed bibref selection dropdown bug in claiming system.