BibSched: adjust runtimes when using sequence-id

* Whithin a set of tasks having the same sequence-id, when, according to
  its runtime value, a task is scheduled to be executed after another task
  which has higher task_id, the other task's runtime is adjusted
  accordingly to be executed after the earlier task.

* When a task is postponed because it happens to be executed outside of
  its time limit, all the other tasks in the same sequence id, with an
  earlier runtime but a later task_id are also postponed automatically.
  (closes #1145)
urlutils: MS Office redirection workaround

* When a URL pointing e.g. to some restricted content is followed from
  a MS Office document, MS Office tries to be smart and already prefetch
  the content, and somehow passes the result to the browser. This
  might lead to lost session e.g. with SSO. A workaround is here
  implemented that provides MS Office with a simple HTML page and a
  200 OK result, instead of the required redirection. The browser will
  then correctly follow the redirection.  See
  <http://support.microsoft.com/kb/899927>.
Merge branch 'maint-1.0'

* maint-1.0:
  urlutils: MS Office redirection workaround
WebApiKey: initial release

* Initial release of the user-signed Web API key facility.
  (closes #806)

* Adds an admin interface to view/edit user's Web API keys.
BatchUploader: more RESTful interface

* Introduce a more RESTful interface to batchuploader/robotupload
  allowing to PUT/POST MARCXML as the body of the HTTP request.
  (closes #976)

* Introduce a new nonce argument to the bibupload/robotupload
  stack, in order to allow clients to uniquely identify
  multiple feedbacks POSTed to the callback_url.
  (closes #977)

* Introduce new make_user_agent_string function to urlutils
  to use a uniform User-Agent string accross Invenio HTTP
  requests.
  (addresses #997)

* Drop the comma-separated list based CFG_BATCHUPLOADER_WEB_ROBOT_AGENT
  invenio.conf variable in favour of regexp-based
  CFG_BATCHUPLOADER_WEB_ROBOT_AGENTS variable.
global: uniformize user-agent header usage

* Use make_invenio_opener/make_user_agent_string every time
  Invenio performs an HTTP request as a client.
  (closes #997)
WebSession: enforce external user identifiers

* When using external authentication systems, the email address
  (which is internally used in Invenio as identifier) might not
  correspond with what is the external identifier (and externally
  the user might change his email address while still being
  identified by the same arbitrary identifier). External
  identifiers are now enforced and smart algorithms are in place
  to solve potential conflicts when a user is using more than
  one authentication system or when he changes his email
  address. (closes #424)

* Changes external_authentication plugin API to support external
  identifiers.

* Supports merging two user accounts that happened to be distinguished
  but are discovered to be corresponding to the same user because
  of an identical email address being associated to both.
BibAuthorID: adds aidUSERINPUTLOG userid column
bibdocfile: bibdocfsinfo update fix

* When hard-deleting a file (a.k.a. calling BibDoc.delete_file()),
  properly update the bibdocfsinfo table if needed.
BibSched: same-name concurrent task fix

* Prevents two tasks with the same name to run concurrently.
BibSched: try harder to keep bibupload order

* When a task is SLEEPING or RUNNING, a bibupload task priority
  is increased to match the max of the WAITING tasks priorities.
  This behavior already existed but it did only change the priority
  of WAITING tasks but not the SLEEPING and RUNNNING ones.
BibSched: scheduling improvements

* A monotask would be previously blocked on ABOUT TO SLEEP tasks
  by trying to sleep them repetitively, thus interrupting the
  scheduler cycle.
* Adds checks to make sure a monotask is never run when other
  tasks are running.
* Adds checks to make sure we never start more tasks
  than CFG_BIBSCHED_MAX_NUMBER_CONCURRENT_TASKS.
* The max procs computation was done on a partial list which would
  sometimes lead to behaviours like having enough resources even
  though we did not and the task would fail to execute later.
* node_relevant_bibupload_tasks would be missing the sleeping tasks
  making it so a bibupload could never be woken up.
* Check for available spots in the queue before trying to sleep
  tasks.
  (closes #1114)
BatchUploader: use localhost in robotupload tests

* In robotupload regression tests, assume the requests are coming
  from localhost (i.e. either 127.0.0.1 or 127.0.1.0). If this is
  not the case, than silently disable the tests.
BibAuthorID: new major release

* Tortoise: The clustering is separated from the matrix
  creation which results into huge scheduler improvement.

* Tortoise: The matrix cache is no longer stored on the
  database but on the file system. Save and load methods
  are much faster and reliable.

* Rabbit: The algorithm can now optionally store parts
  of the database in the memory for a performance boost.

* Rabbit: INSPIRE ids are included in the decisions.

* A lot of interface improvements and fixes.

Co-authored-by: Samuele Carli <samuele.carli@cern.ch>
WebSubmit: configurable Document File Manager

* Moves configuration of the Document File Manager into
  invenio.conf, so that admins can customize e.g. the list
  of available doctypes and restrictions.

* Provides transparent fallback to default values for those
  v1.0.x versions that do not have these variables in
  invenio.conf.
Merge branch 'maint-1.0'

* maint-1.0:
  WebSubmit: configurable Document File Manager

Conflicts:
	modules/miscutil/lib/inveniocfg.py
	modules/websubmit/lib/websubmit_webinterface.py
Merge branch 'master' into next

* master:
  WebSubmit: configurable Document File Manager
  BibAuthorID: new major release
  BatchUploader: use localhost in robotupload tests
  BibSched: scheduling improvements
  BibSched: try harder to keep bibupload order
  BibSched: same-name concurrent task fix
  bibdocfile: bibdocfsinfo update fix
  BibAuthorID: adds aidUSERINPUTLOG userid column
  WebSession: enforce external user identifiers
  global: uniformize user-agent header usage
  BatchUploader: more RESTful interface
  WebApiKey: initial release
  urlutils: MS Office redirection workaround
  BibSched: adjust runtimes when using sequence-id

Conflicts:
	modules/bibcirculation/lib/bibcirculation_utils.py
	modules/bibcirculation/lib/bibcirculationadminlib.py
	modules/miscutil/lib/urlutils.py
	modules/miscutil/sql/tabcreate.sql
	modules/miscutil/sql/tabdrop.sql
	modules/webaccess/lib/access_control_config.py