BibRank: better Solr regression tests

* Fixes regression tests for richer indexes.

* Adds tests for indexing of logical fields.

Signed-off-by: Patrick Glauner <patrick.oliver.glauner@cern.ch>
Tested-by: Tibor Simko <tibor.simko@cern.ch>
BibFormat: fix for snippet generation

* Uses CFG_WEBSEARCH_FULLTEXT_SNIPPETS, the correct config
  variable, for maximum amount of snippets.

Signed-off-by: Patrick Glauner <patrick.oliver.glauner@cern.ch>
Tested-by: Tibor Simko <tibor.simko@cern.ch>
SolrUtils: faster snippet factility

* Makes use of FastVectorHighlighter.

* Backward-compatible to old indexes.

Signed-off-by: Patrick Glauner <patrick.oliver.glauner@cern.ch>
Tested-by: Tibor Simko <tibor.simko@cern.ch>
SolrUtils: better exception handling in indexers

* Improves exception handling for add and commit calls.

* Makes use of autocommit in case commit could not be
  performed.

Signed-off-by: Patrick Glauner <patrick.oliver.glauner@cern.ch>
Tested-by: Tibor Simko <tibor.simko@cern.ch>
Merge branch 'master' into next

* master:
  SolrUtils: better exception handling in indexers
  SolrUtils: faster snippet factility
  BibFormat: fix for snippet generation
  BibRank: better Solr regression tests
  BibRank: faster Solr indexer
  BibRank: new multi-tag Solr indexer
  BibRank: index latest records first in Solr
  SolrUtils: support entire full-text indexing
  SolrUtils: cleaner schema.xml
  SolrUtils: support high count of logical clauses
  WebSearch: fix missing cc info in req object
  WebJournal: new image template
  OAIRepository: optional support for --notimechange
  BibDocFile: RHEL6 magic bindings support

Conflicts:
	modules/websearch/lib/search_engine.py

Signed-off-by: Tibor Simko <tibor.simko@cern.ch>
BibCatalog: new email ticketing backend

* Implements simple email ticketing backend.  Adds corresponding
  unit tests.  (addresses #872)

Reviewed-by: Joe Blaylock <jrbl@slac.stanford.edu>
Tested-by: Tibor Simko <tibor.simko@cern.ch>
installation: Hogan prerequisite documentation

Signed-off-by: Tibor Simko <tibor.simko@cern.ch>
BibCatalog: default email backend

* Makes simple ticketing email backend to be the default ticketing system.
  (closes #872)

* Amends unit test case email address.

* Removes trailing whitespace.

Signed-off-by: Tibor Simko <tibor.simko@cern.ch>
installation: help for BibAuthorID email settings

Signed-off-by: Tibor Simko <tibor.simko@cern.ch>
BibSched: enhanced write_message()

* Enhances write_message() function to allow for lazy printing of messages.
  When calling write_message(msg), msg can now be a callable, that is
  executed only if the necessary verbosity level is reached (thus
  allowing printing detailed information without performance penalties).

Signed-off-by: Samuele Kaplun <samuele.kaplun@cern.ch>
Reviewed-by: Tibor Simko <tibor.simko@cern.ch>
BibRecord: new API identical_records()

* Adds a new identical_records() API function, that allow to
  compare for identity two records, without considering the 005
  MARC tag (i.e. their modification timestamp).

Signed-off-by: Samuele Kaplun <samuele.kaplun@cern.ch>
Reviewed-by: Tibor Simko <tibor.simko@cern.ch>
BibUpload: smart record uploader

* Adds bibupload_revisionverifier submodule to compare the submitted record
  revision information (stored in MARC tag 005) against the list of
  known archived revisions of the record, in order to generate a record patch.
  (Useful when cataloguer modifies an older version of a record while another
  cataloguer touches the record in question, because revision verification
  step will now merge the two wanted changes together).  As a side effect,
  the --replace mode of bibupload will be always changed to --correct mode with
  the wanted record patch. In case of conflicting needs (when two users
  modified the same fields), the bibupload ends with an error and submits the
  given modifications to the holding pen.
  (closes #816) (closes #864)

* Implements field deletion in bibupload based on DELETE_CODE and DELETE_VALUE
  set for the field.
  (closes #897)

* Updates DB only for those elements that are modified in the
  submitted version.

* Drops support for not documented (and no longer used) FMT tags.

* Deprecates support for --reference/-z in favor of --correct/-c.

Co-authored-by: Samuele Kaplun <samuele.kaplun@cern.ch>
Signed-off-by: Samuele Kaplun <samuele.kaplun@cern.ch>
Tested-by: Tibor Simko <tibor.simko@cern.ch>
BibUpload: bibrec timestamp bug

* In 70f76183f018c1a3bb7313c39b6a4ce8e5e908a4 bibupload was modified
  to store as record modification time the initial timestamp of when the
  modification began, rather than the latest timestamp of when modification
  finished (as it was before this patch).
  This might lead to all sorts of malfunctions if tasks such as bibindex,
  bibrank, refextract, bibauthorid, etc. are run at the same time of
  a bibupload. This commit partially reverts this behaviour by indeed
  storing only the very latest timestamp into the bibrec table (so
  that other bibtasks are not missing the update).
  (fixes #1431)

Signed-off-by: Samuele Kaplun <samuele.kaplun@cern.ch>
Tested-by: Tibor Simko <tibor.simko@cern.ch>
BibUpload: sensible history and other goodies

* Drops support for not documented and unbound option opt_tag.

* Drops support for unused (and complicating) CLI --stage option.

* Stores history only if a record is really modified. Includes
  a migration kit to remove useless history entries.
  (references #498) (closes #1250)

* Take use of new write_message lazy mode, that allows computing
  strings to print, only if verbosity level is reached.

* Introduce new CFG_BIBUPLOAD_DISABLE_RECORD_REVISIONS flag
  to control whether it is desired to optionally disable keep
  track of record revisions in a history table (e.g. in case
  record updates are all coming from an external system which
  already keeps track of history).
  (closes #871)

* Minimize that handling of fulltextes to when the incoming record
  refers to 8564 tags or contain FFT tags.

* Since the 005 timestamp granularity is of the order of seconds,
  properly introduces sleeping points of 1 second, in case a record
  is modified a second time within the same second.

* Removes try..except: clauses that were capturing all exception
  and just logged them as warning.

* Additionally takes care of properly update record timestamp only
  if there was really a metadata update or at least one FFT tag.

Signed-off-by: Samuele Kaplun <samuele.kaplun@cern.ch>
Tested-by: Tibor Simko <tibor.simko@cern.ch>
BibUpload: BibCatalog connection

* When BibUpload can't automatically resolve a given revision conflict
  it is putting the modification in the Holding Pen. At the same time
  now a ticket is created in BibCatalog to be able to refer to it.

Signed-off-by: Samuele Kaplun <samuele.kaplun@cern.ch>
Tested-by: Tibor Simko <tibor.simko@cern.ch>
BibUpload: smart record uploader fixes

* Improves identical_records() flexibility.

* Introduce several bugfixes to the new bibupload revision
  verifier for some corner cases.

* Extends bibupload regression tests by always check for consistency
  among bibfmt, bibxxx and hstRECORD tables after any bibupload.

Signed-off-by: Samuele Kaplun <samuele.kaplun@cern.ch>
Tested-by: Tibor Simko <tibor.simko@cern.ch>
BibRecord: new API records_identical()

* Renames identical_records() to records_identical() for better consistency
  with the other BibRecord APIs.

* Amends all clients accordingly.

Signed-off-by: Tibor Simko <tibor.simko@cern.ch>
BibUpload: conflicting revision ticket queue

* Amends configuration for confliction revision ticket queue to be
  consistent with other usage of ticket queues in other modules.

Signed-off-by: Tibor Simko <tibor.simko@cern.ch>
WebSearch: bug fix for SPIRES date math search

* Fixes find last week when executed on a Saturday. It was giving a
  too long time range.

Reviewed-by: Tibor Simko <tibor.simko@cern.ch>
WebSearch: asciification of `oe` grapheme

* Properly asciifies `oe` grapheme.  (closes #1582)

* NOTE Please reindex if your records may contain `oe` UTF-8 graphemes.

Signed-off-by: Tibor Simko <tibor.simko@cern.ch>