deposit: fix fields argument on post processors

* Updates sherpa romeo methods to receive fields argument.

Signed-off-by: Pedro Gaudêncio <pedro.gaudencio@cern.ch>
Reviewed-by: Javier Martin Montull <javier.martin.montull@cern.ch>
Reviewed-by: Lars Holm Nielsen <lars.holm.nielsen@cern.ch>
Reviewed-by: Jiri Kuncar <jiri.kuncar@cern.ch>
BibDocFile: model and API expunge fix

Signed-off-by: Jiri Kuncar <jiri.kuncar@cern.ch>
BibIndex: fix for regression tests

* Fixes regression tests for 'itemcount' index.

Signed-off-by: Grzegorz Szpura <grzegorz.szpura@cern.ch>
Reviewed-by: Jiri Kuncar <jiri.kuncar@cern.ch>
WebSession: fix for translatable strings

Signed-off-by: Jiri Kuncar <jiri.kuncar@cern.ch>
workflows: add OAIHarvest integration

* Refactors log messages in generic harvesting workflow tasks.

* Adds a new task to convert records with information directly from
  OAI repository models.

* Generic workflow now takes parameter of the given repository.

* Cleans the code of several functions, simplifies the
  harvesting a bit. Changes the way the updating of
  the last_time update.

Signed-off-by: Jan Aage Lavik <jan.age.lavik@cern.ch>
workflows: refactor holding pen widgets

* Adds mini approval widget.

* Removes every hardcoded url in .js files.

* Adds small helper function in maintable.js

* Adds buttons for multiple record approval.

* Adds new row_formatter to format datatables rows.

* Adds checkbox functionality.

* Fixes breadcrumbs in Holding Pen pages.

* Upgrades templates to bootstrap 3.

* Adds a navbar above Holding Pen table and updates layouts
  across the details page and approval widget page.

* Adds a new tag based UI element for filtering objects in the
  Holding Pen.

Signed-off-by: Jan Aage Lavik <jan.age.lavik@cern.ch>
workflows: new API in engine and model

* Updates the BibWorkflowObject with new API functions to add
  widgets to trigger an holding pen action.

* Adds a function to BibWorkflowObject to add task results
  using a newly created class WorkflowTaskResult.

* Changes the data preview in Holding Pen back to using
  the previously added get_formatted_data function.

* Widgets and tasks now use these new functions.

* Enriches sample workflow to showcase more tasks.

Signed-off-by: Jan Aage Lavik <jan.age.lavik@cern.ch>
workflows: update workflow tasks

* Adds new filtering task for record categories, allowing to
  specify which categories to trigger an action in holding pen.

* Adds task results to objects in various tasks.

* Updates the sample harvesting workflow to use new bibclassify
  API.

* Several other updates and improvements to workflow tasks.

* Updates the category filtering task to allow regular expression
  syntax when defining which categories to halt, continue or stop
  the workflow.

* Adds new data type attribute to workflows, by Jan Aage Lavik.

* Updates the exception handling to better handle exceptions in
  workflows when running a workflow in a BibSched task.

* Improvements to category filter.

* Adds options for the foreach loop, we can now cache the
  list to go through it will then allow to five the choice
  to the user if he wants to priviledge CPU vs Memory,
  cache uses few CPU but a lot of memory and doesn't provide
  a dynamic list. Without the cache option, we recompute the
  list each time so we use much more CPU but we keep just
  the data needed so few memory, and the list is still
  refresh for each loop.

* Refactors use of extra_data in the engine.

* Changes the task foreach so that obj.data takes the value
  it has before the foreach. In the previous version we were
  keeping the last value of the foreach.

* Adds the function filtering_oai_pmh_identifier which
  allows you to filter the identifier. Mainly used to
  avoid processing already processed one.

* Adds features to  inspire_filter_category, now parameter
  can be passed throught the extra_data of the engine.

* Fixes some bugs with regex in inspire_filter_category

* Adds the function update_last_update which allows you to
  update the last_update time of a repository. It is
  important because the user can choose exactly where in
  the workflows he considers that a crash is not a problem
  for harvested documents.

* Adds the different features needed by the change from
  pickle to message pack in Celery, to the container for
  BibWorkflowObject exchange.

* Adds the inspire_filter_custom task which allows to
  filter record by examinating some field of its bibfield
  It is a generalization of inspire_filter_category.

* Reenables all the functions in full_doc_process
  workflow.

* Adds some control over the number of workflows generated
  it is to avoid the memory saturation or the messaging
  queue limit.

* Adds some protected name for workflows to avoid possible
  conflict with new task an new user data etc.

* Add some functions to control workflows states.

* Fixes PEP8 and clean code

* Deletes non-informative logs.

* Modifies the different workflows to take into account the
  previous modification made.

Co-authored-by: Jan Aage Lavik <jan.age.lavik@cern.ch>
Signed-off-by: Jan Aage Lavik <jan.age.lavik@cern.ch>
workflows: Holding Pen update

* Removes some uninteresting/unused columns in the Holding Pen
  main table and adds a new columns called categories containing
  any record subjects found, if applicable.

* Cleans some blueprint code and routes in Holding Pen.

* Refactors the tag system used in the main Holding Pen display. Now
  using a new JS package called bootstrap-tagsinput.

* Changes approval widget design and layout.

* Improves detailed display of workflow definition.

* Updates in harvesting workflow tasks.

* Adds a new template for detailed layout.

* Fixes a problem with hotkeys in the main table display.

* Adds a new bootstrap-tagsinput package for use in the holding pen
  display.

* Removes the current config.py and moves the enumerations
  into engine.py and models.py where appropriate and changing the
  naming from CFG_OBJECT_VERSION to ObjectVersion.

* Updates deposit models to use the new ObjectVersion.

* Removes the DATA_TYPE enumeration. This will now be dynamically
  loaded from the database in the holding pen.

* Initial implementation of a new Edit Record widget which
  will be used for (limited) record editing from Holding Pen.
  Still WIP.

* Adds templating blocks for Holding Pen navigation bar and
  record preview.

* Creates a new field rendering macro for better WTForm rendering.

Co-authored-by: Jan Aage Lavik <jan.age.lavik@cern.ch>
Signed-off-by: Jan Aage Lavik <jan.age.lavik@cern.ch>
workflows: admin JS updates

* Updates the JavaScript and templates for the workflows admin
  pages to implement the module pattern.

* Refactors some blueprint handlers for the workflows admin
  pages.
workflows: delete old CSS
WIP workflows: tmp approval fix
workflows: fix test_halt
workflows: re-fix get taskname
workflows: refactor approval.js
workflows: remove style.css refs
workflows: re-structure JS files

* Refactors structure of JS files using module pattern.
workflows: url passing
workflows: tmp updates to JS