WebSearch: correct record sums from hosted colls

* The number of records in a collections (nbrecs) is now correctly calculated
  including records coming from descendant hosted collections.  (closes #1651)

Reported-by: Theodoropoulos Theodoros <theod@lib.auth.gr>
Signed-off-by: Nikolaos Kasioumis <nikolaos.kasioumis@cern.ch>
Reviewed-by: Tibor Simko <tibor.simko@cern.ch>
WebAccess: remove Facebook testing credentials

* Removes Facebook credentials that were left in
  the code by mistake.
WebSearch: CV output formats

* introduce hcv, htcv, and tlcv formats
* based on SPIRES html, text, and LaTeX formats (fixes #314)
* alphabetized Makefile.am

Reviewed-by: Samuele Kaplun <samuele.kaplun@cern.ch>
WebSearch: find doi in search engine

* Handles find doi in search engine. (addresses #1051)

Reviewed-by: Samuele Kaplun <samuele.kaplun@cern.ch>
BibSched: refactoring and improvements

* Moves the bibsched monitor code to bibsched_monitor.py.

* Order node_relevant_bibupload_tasks by status first then by id
  so that we wake up slept bibuploads first.

* The row limit was affecting the running tasks instead of the
  archived ones

* Remove bibupload from monotasks for INSPIRE.

* Handles specific sleeping of bibupload tasks:
  If a bibupload comes with higher priority than a bibupload that is
  already in the queue, it sleeps the existing one and executes.
  (closes #1274, closes #1275)

* Related to this behavior, on INSPIRE we do not need to change the
  priority of bibuploads anymore so bibupload priority is never
  changed.

* Prevents tasks that are still in SCHEDULED status to be asked
  to go to sleep. The task expects to be in SCHEDULED status when
  starting.

* When the email-logs option is set in a task via --email-log-to
  we shall not send emergency messages to admins. The error is already
  reported to the submitter via email.
  (closes #1449)

* When the a motd line was longer than the window width,
  curses would crash and burn.
  We now wrap the motd to the screen width.

* Moves motd to CFG_TMPSHAREDDIR, so that it is visible on all
  servers that can run bibsched.

* Add timeout to javascript code, so that ping stops after some
  inactivity time.

* BibSched was not checking properly the number of tasks running.
  It was counting tasks that were running on the same host too.

* Reworks tasks list (running, sleeping, etc.) so that we handle
  them in a cleaner way and with less bugs.

* Adds a debug parameter to Log() in order to avoid having to
  write ifs protecting debug output.

* Sets the task to ERROR status after it fails to start properly
  instead of stopping the queue.

* Introduces a new config variable that allows us 2 tasks that
  should not run concurrently.

* Also introduces a new config variable that allows us to configurare
  incompatible tasks.
  The difference is the non concurrent tasks can run while the other
  task is sleeping. Incomptible tasks can never run even if the other
  task is sleeping.

* Selected line keeps to the same task even if the task list changes
  It matches the tasks by id

* task_sleep_now_if_required would raise a SystemExit which was
  interpreted as an exception. We now ignore if the code is 0.

* Optimizes update_rows() to use "select...in" instead of
  "select...not in"

* Prevents waking up tasks that are sleeping on other nodes.
  (it is not possible and the task would be marked as failed)

* If a monotask wants to run on node A,
  the tasks on node B can still wake up from sleep,
  but the monotask sleeps them again right after they have woken,
  so they enter an infinite cycle.

* The automatic does not kill the daemon anymore.
  Also to enter the automatic mode from the bibsched monitor
  you need to select a duration and enter a motd if this duration
  is more than 5 minutes.

* Handles bibindex:author, specifically the name given after :
  in tasks specified in CFG_BIBSCHED_NON_CONCURRENT_TASKS.

* Fixes a bug that would allow 2 non concurrent tasks to run at the
  same time.

* Adds a new configurable in invenio.conf, CFG_BIBSCHED_LOGDIR,
  where the location of bibsched task logs of type
  bibsched_task_%d.err|log will be stored.

  The default log location will still be CFG_LOGDIR.

* Amends all references to the bibsched logs to use the new
  variable.

* Adds --host which allows to force the execution of a task to a
  certain host

Co-authored-by: Samuele Kaplun <samuele.kaplun@cern.ch>
Reviewed-by: Samuele Kaplun <samuele.kaplun@cern.ch>
WebStyle: /info pages using webdoc infrastructure

* Adds new admin interface to manage the /info space allowing a user
  to see and modify the contents of that space

* Adds jQuery File Tree plugin

* Adds new action 'runinfomanager'

* If the URL requested has an extension, look for the file on disk
  and send it back to the user with the correct mime type

* Adds a dictionary of mappings that allow us to bind any /info url
  to a dynamic page
  e.g. /info/hello can be mapped to HelloWebInterface()
  This dictionary is overridable by placing additional mappings
  in webdoc_info_mappings.py

* Uses bibdocfile stream_file function to use xsendfile when
  available.

Co-authored-by: Alessio Deiana <alessio.deiana@cern.ch>
Tested-by: Samuele Kaplun <samuele.kaplun@cern.ch>
WebSearch: INSPIRE fulltext warning update

* Amend INSPIRE fulltext warning.

Reviewed-by: Samuele Kaplun <samuele.kaplun@cern.ch>
BibFormat: progress display improvement

* Displays ids which are being processing in bibreformat.
inveniogc: add new session deletion mode

* Adds a new option, -S --sessions, that only deletes expired
  sessions from the database session table. Useful in large
  production environments where the standard --guests option
  does not scale well.
HepData: new HepData module

* Created basis for integration of Invenio with data coming from HEPData.

Co-authored-by: Laura Rueda <laura.rueda@cern.ch>
Tested-by: Samuele Kaplun <samuele.kaplun@cern.ch>
RefExtract: improves docextract, refextract

* Bumps version to 1.5.32
* Adds "0" subfield that associates recids to refs.
* Improves arxiv rn parsing.
* More permissive url, authors and doi regexps.
* Reworks pagination removal.
* Reworks journals identification.
* Reworks references splitting.
* Reworks refs section detection.
* Handles authors with weird characters.
* Fixes -c option.
* Prevents have 2 "a" (author) subfields.
* Improves handling of PoS.
* Removes "xbook" subfield.
* Adds "c" (collaboration) subfield.
* Only accept 1 as first ref.
* Only output o subfield if marker is present.
* Tweaks find numeration.
* Only create tickets for records that are less than 2 years old
* Only accept 1 as first ref.
* Only output o subfield if marker is present.
* Fixes marker detection.
* Allows : and - in line markers.
* Tweaks find numeration.
* Fixes find rawref
  The tag number for converting to the journal format was hardcoded
  This is not the case anymore.
* Handles ATL-CONF:
  Hande several formats for atlas conferences used in references
  ATL-CONF-99, ATLAS-CONF-2010-001, ...
* Fixes missing lines when a line marker with a number that.
  does not match the sequence (1, 2, 3, etc.) is detected.
* We do not strip pagination anymore. I am keeping this test
  in case we re-enable it.
* Fixes bug that would remove letter from o subfield
* arXiv report numbers with a category are not found by the search
  engine so we remove the category before searching.
* Script to convert journals abbreviations to short form
* New record model similar to bibrecord but easier to use
* Switches to the new bibrecord model to generate xml.
  This way we can generate it more easily and we can ensure
  it will always be valid.
* Encoding tweaking: we now return bytes instead of unicode
  for functions that provide
  a serialization format namely xml
* If a reference has 2 different DOI, we split it
  We also remove duplicate doi tags.
* Prevent authors with "paper" as name
* Handles 1.1 numeration
* When unidecode produced a string with a different length than
  the original, the replace would be off overwriting text to the
  right. We know pass the output through unidecode in that case.

Signed-off-by: Alessio Deiana <alessio.deiana@cern.ch>
BibFormat: empty record check

* Modifies the empty record check to allow records with a single
  tag code (e.g. 999) to be formatted by bibformat.
DocExtract: several improvements

* Ability to load journals from journal records database

* Moves html generation of /textmining to a dedicated template file
  so that it can be overwritten by INPSIRE

* Reworks the removal of [] and () around report numbers
* Which fixes a bug that would add extra characters in the misc
  string
* Only unidecode strings when authors are present

* Replaced the [A-Z] with a list of uppercase characters that is
  computed at load.

* Adds handling for caret in pdf2text results

* We can know check for existing tickets for a given recid.
  Which means we can always create tickets for extracted references
  by making sure there is not existing ticket.

* Adds a function to know if we can safely extract references
  from a record.

* Submits a single bibupload task, for a refextract task run instead
  of one per record.

Signed-off-by: Alessio Deiana <alessio.deiana@cern.ch>
Tested-by: Samuele Kaplun <samuele.kaplun@cern.ch>
PdfChecker: new module for arxiv pdf checker

* Introduces new PDF Checker module specialized
  in re-harvesting from arXiv missing PDFs and
  the corresponding latest versions.

Signed-off-by: Alessio Deiana <alessio.deiana@cern.ch>
BibRank: optimized cited by sorting

* We already have the presorted citations count in memory.
  This switches the sort to use the presorted list from memory.

Signed-off-by: Alessio Deiana <alessio.deiana@cern.ch>
BibRank: handle records with mulitple journals

* Citations to the second journal were missed
  (fixes #1394)

Signed-off-by: Alessio Deiana <alessio.deiana@cern.ch>
BibRank: citesummary optimizations

* Use the key parameter for sort instead of the cmp parameter
  to sort by citations faster in the citesummary
* Still in the citesummary, instead of appending to lists, uses
  list comprehensions
* Instead of storing the whole citation dict in memory,
  we now only store citations counts in memory.
* Stores citation dict in the database and removes it from memory.
* Improves code quality in bibrank_citation_indexer.py.
* In citesummary, sort the citations count first, so that we only
  have to do it once.
* Use 2 different methods for sorting citations
  counts in citesummary:
    - for less than 20000 records, we use the counts dictionary
      and access by key
    - for more than 20000 records, we use the presorted citations
      counts list
  (closes #1481)
* Split stats computation and html rendering in the citesummary
  in 2 different functions.
* With that change, we can now iterate only once through, the
  citations counts and compute all the stats.
  (closes #1217)
* When citesummary was computed for less than 20000 records,
  we would display the number of self-citations instead of the
  number of citations.

Signed-off-by: Alessio Deiana <alessio.deiana@cern.ch>
WebSearch: spires syntax and quotes

* Ignores text in "" when looking for SPIRES keywords
BibRank: store selfcites in a table

* This mirror the way of doing of citations.
  We store the self-cites counts in exactly the same way.
  They are serialized in a dictionary in rnkMETHODATA.
  We store the self-cites in table in exactly the same way.
  They are stored in the table rnkSELFCITEDICT.
  (closes #1417)

* Fixes a bug when using -R, we would not update the fast access
  dictionary. It would therefore be empty or outdated. We now store
  the updated data there

* Allows to specify ids to recompute on the command line via:
  bibrank -w selfcites --id 1,3-5

* Selfcites was failing to take into account
  the right dates ranges and missing some records in the process.
  It is now using index/bibauthorid last update.

Signed-off-by: Alessio Deiana <alessio.deiana@cern.ch>
DocExtract: help messages & compatibility warnings

* Refextract api has changed and we now give help messages if the
  renamed options are used. (closes #1220)
* Updated help messages.