celery: msgpack serialization usage

* Changes default celery configuration to use msgpack instead of
  pickle and use rabbitmq instead of redis broker, after some
  problems under heavy load.

Tested-by: Jan Aage Lavik <jan.age.lavik@cern.ch>
search: change union_update to union

* Due to changes in the behaviour of intbitset union_update function,
  union should be used if a return result is expected.

Reported-by: Kamil Neczaj <kamil.neczaj@cern.ch>
Signed-off-by: Javier Martin Montull <javier.martin.montull@cern.ch>
base: Apache configuration template fix

* Fix newline issue when CFG_WEBSTYLE_HTTP_USE_COMPRESSION is missing.

Signed-off-by: Adrian-Tudor Panescu <adrian.tudor.panescu@cern.ch>
celery: make Redis default broker

* Makes Redis the default broker for Celery.

Signed-off-by: Jan Aage Lavik <jan.age.lavik@cern.ch>
Acked-by: Jiri Kuncar <jiri.kuncar@cern.ch>
BibClassify: new API

* Adds a new API module for use by workflow tasks.

* Several updates and fixes to the module to improve output.

* Moves a function for easier future optimization,
  simplifying some operations.

Tested-by: Jan Aage Lavik <jan.age.lavik@cern.ch>
pidstore: name conflict fix

* Fixes a name conflict between a class defined in the file
  and a class imported in the same file.

Signed-off-by: Lars Holm Nielsen <lars.holm.nielsen@cern.ch>
legacy: tasklets configuration and loading

* Adds configuration variable CFG_BIBSCHED_TASKLET_PACKAGES, which
  defines which packages to load tasklets from.

Signed-off-by: Lars Holm Nielsen <lars.holm.nielsen@cern.ch>
workflows: engine branch halt testcase

* Adds test case for halting workflow execution in workflows
  with branches.

* Adds temporary fix to be superseeded by larger refactoring
  of workflows.

Signed-off-by: Lars Holm Nielsen <lars.holm.nielsen@cern.ch>
deposit: default deposition fix

* Fixes issues for default deposition types.

* Removes debugging code.

Signed-off-by: Lars Holm Nielsen <lars.holm.nielsen@cern.ch>
deposit: jsonalchemy refactoring fix

* Fixes deposit API to work with latest jsonalchemy updates.

* Removes conditional loading of pidstore.

* Fixes API error reporting after upgrade to Cerberus 0.5.

Co-authored-by: Esteban J. G. Gabancho <esteban.gabancho@cern.ch>
Signed-off-by: Lars Holm Nielsen <lars.holm.nielsen@cern.ch>
base: sticky footer fix

* Fix sticky footer issue on pages with content longer than 100% window
  size, by updating to the reference sticky footer implementation for
  Bootstrap3.

Signed-off-by: Adrian-Tudor Panescu <adrian.tudor.panescu@cern.ch>
accounts: gettext import fix

Signed-off-by: Adrian-Tudor Panescu <adrian.tudor.panescu@cern.ch>
deposit: fix file size length

Signed-off-by: Jiri Kuncar <jiri.kuncar@cern.ch>
installation: typeahead Grunt fix

* Fix typeahead path for v0.10+.

* Minor lint fixes.

Signed-off-by: Adrian-Tudor Panescu <adrian.tudor.panescu@cern.ch>
grunt: fix Prism configuration

* Fixes an issue with the Prism plugin having outdated
  configuration in grunt/bower.

Signed-off-by: Jan Aage Lavik <jan.age.lavik@cern.ch>
OAIHarvest: move to workflows

* Changes OAI Harvest daemon logic to use the new workflows
  module, with many changes overall.

* Refactors DB-layer for oaiharvest part. Uses the new
  sqlalchemy models.

* Changes OaiHARVEST table schema to add workflow options and
  removes frequency parameter. BibSched is the master of
  task frequency.

* Allows new CLI parameters --workflow or -r/--repository,
  --workflow or just only -r/--repository.

* Adds support to harvest with all combinations. It means
  one repository or one workflow, one repository N workflows
  N repositories one workflow or N repositories N workflows.

* Optimized request in daemon to mimize the number of
  workflows to start to complete the work asked.

* Improves task_run with better error reporting.

* Cleans the code of several functions, simplifies the
  harvesting a bit. Changes the way the updating of
  the last_time update.

* Updates OaiHarvest model in oaiharvester.

Reviewed-by: Jan Aage Lavik <jan.age.lavik@cern.ch>
oaiharvester: update model with defaults

* Adds better defaults for the oaiharvester model.

* Changes the argument column to MarshalBinary and
  removes getter get_arguments().

Signed-off-by: Jan Aage Lavik <jan.age.lavik@cern.ch>
workflows: update calls to OaiHarvest model

* Updates the calls to get the arguments dict from the
  OaiHarvest model to adhere to recent API changes.

Signed-off-by: Jan Aage Lavik <jan.age.lavik@cern.ch>