BibCatalog: new email ticketing backend

* Implements simple email ticketing backend.  Adds corresponding
  unit tests.  (addresses #872)

Reviewed-by: Joe Blaylock <jrbl@slac.stanford.edu>
Tested-by: Tibor Simko <tibor.simko@cern.ch>
BibCatalog: default email backend

* Makes simple ticketing email backend to be the default ticketing system.
  (closes #872)

* Amends unit test case email address.

* Removes trailing whitespace.

Signed-off-by: Tibor Simko <tibor.simko@cern.ch>
installation: help for BibAuthorID email settings

Signed-off-by: Tibor Simko <tibor.simko@cern.ch>
BibSched: enhanced write_message()

* Enhances write_message() function to allow for lazy printing of messages.
  When calling write_message(msg), msg can now be a callable, that is
  executed only if the necessary verbosity level is reached (thus
  allowing printing detailed information without performance penalties).

Signed-off-by: Samuele Kaplun <samuele.kaplun@cern.ch>
Reviewed-by: Tibor Simko <tibor.simko@cern.ch>
BibRecord: new API identical_records()

* Adds a new identical_records() API function, that allow to
  compare for identity two records, without considering the 005
  MARC tag (i.e. their modification timestamp).

Signed-off-by: Samuele Kaplun <samuele.kaplun@cern.ch>
Reviewed-by: Tibor Simko <tibor.simko@cern.ch>
BibUpload: smart record uploader

* Adds bibupload_revisionverifier submodule to compare the submitted record
  revision information (stored in MARC tag 005) against the list of
  known archived revisions of the record, in order to generate a record patch.
  (Useful when cataloguer modifies an older version of a record while another
  cataloguer touches the record in question, because revision verification
  step will now merge the two wanted changes together).  As a side effect,
  the --replace mode of bibupload will be always changed to --correct mode with
  the wanted record patch. In case of conflicting needs (when two users
  modified the same fields), the bibupload ends with an error and submits the
  given modifications to the holding pen.
  (closes #816) (closes #864)

* Implements field deletion in bibupload based on DELETE_CODE and DELETE_VALUE
  set for the field.
  (closes #897)

* Updates DB only for those elements that are modified in the
  submitted version.

* Drops support for not documented (and no longer used) FMT tags.

* Deprecates support for --reference/-z in favor of --correct/-c.

Co-authored-by: Samuele Kaplun <samuele.kaplun@cern.ch>
Signed-off-by: Samuele Kaplun <samuele.kaplun@cern.ch>
Tested-by: Tibor Simko <tibor.simko@cern.ch>
BibUpload: bibrec timestamp bug

* In 70f76183f018c1a3bb7313c39b6a4ce8e5e908a4 bibupload was modified
  to store as record modification time the initial timestamp of when the
  modification began, rather than the latest timestamp of when modification
  finished (as it was before this patch).
  This might lead to all sorts of malfunctions if tasks such as bibindex,
  bibrank, refextract, bibauthorid, etc. are run at the same time of
  a bibupload. This commit partially reverts this behaviour by indeed
  storing only the very latest timestamp into the bibrec table (so
  that other bibtasks are not missing the update).
  (fixes #1431)

Signed-off-by: Samuele Kaplun <samuele.kaplun@cern.ch>
Tested-by: Tibor Simko <tibor.simko@cern.ch>
BibUpload: sensible history and other goodies

* Drops support for not documented and unbound option opt_tag.

* Drops support for unused (and complicating) CLI --stage option.

* Stores history only if a record is really modified. Includes
  a migration kit to remove useless history entries.
  (references #498) (closes #1250)

* Take use of new write_message lazy mode, that allows computing
  strings to print, only if verbosity level is reached.

* Introduce new CFG_BIBUPLOAD_DISABLE_RECORD_REVISIONS flag
  to control whether it is desired to optionally disable keep
  track of record revisions in a history table (e.g. in case
  record updates are all coming from an external system which
  already keeps track of history).
  (closes #871)

* Minimize that handling of fulltextes to when the incoming record
  refers to 8564 tags or contain FFT tags.

* Since the 005 timestamp granularity is of the order of seconds,
  properly introduces sleeping points of 1 second, in case a record
  is modified a second time within the same second.

* Removes try..except: clauses that were capturing all exception
  and just logged them as warning.

* Additionally takes care of properly update record timestamp only
  if there was really a metadata update or at least one FFT tag.

Signed-off-by: Samuele Kaplun <samuele.kaplun@cern.ch>
Tested-by: Tibor Simko <tibor.simko@cern.ch>
BibUpload: BibCatalog connection

* When BibUpload can't automatically resolve a given revision conflict
  it is putting the modification in the Holding Pen. At the same time
  now a ticket is created in BibCatalog to be able to refer to it.

Signed-off-by: Samuele Kaplun <samuele.kaplun@cern.ch>
Tested-by: Tibor Simko <tibor.simko@cern.ch>
BibUpload: smart record uploader fixes

* Improves identical_records() flexibility.

* Introduce several bugfixes to the new bibupload revision
  verifier for some corner cases.

* Extends bibupload regression tests by always check for consistency
  among bibfmt, bibxxx and hstRECORD tables after any bibupload.

Signed-off-by: Samuele Kaplun <samuele.kaplun@cern.ch>
Tested-by: Tibor Simko <tibor.simko@cern.ch>
BibRecord: new API records_identical()

* Renames identical_records() to records_identical() for better consistency
  with the other BibRecord APIs.

* Amends all clients accordingly.

Signed-off-by: Tibor Simko <tibor.simko@cern.ch>
BibUpload: conflicting revision ticket queue

* Amends configuration for confliction revision ticket queue to be
  consistent with other usage of ticket queues in other modules.

Signed-off-by: Tibor Simko <tibor.simko@cern.ch>
WebSearch: bug fix for SPIRES date math search

* Fixes find last week when executed on a Saturday. It was giving a
  too long time range.

Reviewed-by: Tibor Simko <tibor.simko@cern.ch>
WebSearch: asciification of `oe` grapheme

* Properly asciifies `oe` grapheme.  (closes #1582)

* NOTE Please reindex if your records may contain `oe` UTF-8 graphemes.

Signed-off-by: Tibor Simko <tibor.simko@cern.ch>
Merge branch 'maint-1.0' into maint-1.1

* maint-1.0:
  WebSearch: asciification of `oe` grapheme

Conflicts:
	RELEASE-NOTES
	modules/websearch/lib/search_engine.py
	modules/websearch/lib/search_engine_unit_tests.py

Signed-off-by: Tibor Simko <tibor.simko@cern.ch>
Merge branch 'maint-1.1'

* maint-1.1:
  WebSearch: asciification of `oe` grapheme
  WebSearch: bug fix for SPIRES date math search
Merge branch 'master' into next

* master:
  WebSearch: asciification of `oe` grapheme
  WebSearch: bug fix for SPIRES date math search
  BibUpload: conflicting revision ticket queue
  BibRecord: new API records_identical()
  BibUpload: smart record uploader fixes
  BibUpload: BibCatalog connection
  BibUpload: sensible history and other goodies
  BibUpload: bibrec timestamp bug
  BibUpload: smart record uploader
  BibRecord: new API identical_records()
  BibSched: enhanced write_message()
  installation: help for BibAuthorID email settings
  BibCatalog: default email backend
  BibCatalog: new email ticketing backend

Conflicts:
	modules/bibupload/lib/bibupload_regression_tests.py

* NOTE Amended lazy import in some regression tests during merge.

Signed-off-by: Tibor Simko <tibor.simko@cern.ch>
WebSearch: fix for facet builder return type

* Addresses problem with function returning list instead of intbitset.

Reported-by: Jaime Garcia Llopis <garciallopis@iloguest.org>
Signed-off-by: Jiri Kuncar <jiri.kuncar@cern.ch>
BibCatalog: system email unit tests fix

Signed-off-by: Jiri Kuncar <jiri.kuncar@cern.ch>
testutils: fix for testing page with 401 error

Signed-off-by: Jiri Kuncar <jiri.kuncar@cern.ch>