workflows: add OAIHarvest integration

* Refactors log messages in generic harvesting workflow tasks.

* Adds a new task to convert records with information directly from
  OAI repository models.

* Generic workflow now takes parameter of the given repository.

* Cleans the code of several functions, simplifies the
  harvesting a bit. Changes the way the updating of
  the last_time update.

Signed-off-by: Jan Aage Lavik <jan.age.lavik@cern.ch>
workflows: refactor holding pen widgets

* Adds mini approval widget.

* Removes every hardcoded url in .js files.

* Adds small helper function in maintable.js

* Adds buttons for multiple record approval.

* Adds new row_formatter to format datatables rows.

* Adds checkbox functionality.

* Fixes breadcrumbs in Holding Pen pages.

* Upgrades templates to bootstrap 3.

* Adds a navbar above Holding Pen table and updates layouts
  across the details page and approval widget page.

* Adds a new tag based UI element for filtering objects in the
  Holding Pen.

Signed-off-by: Jan Aage Lavik <jan.age.lavik@cern.ch>
workflows: new API in engine and model

* Updates the BibWorkflowObject with new API functions to add
  widgets to trigger an holding pen action.

* Adds a function to BibWorkflowObject to add task results
  using a newly created class WorkflowTaskResult.

* Changes the data preview in Holding Pen back to using
  the previously added get_formatted_data function.

* Widgets and tasks now use these new functions.

* Enriches sample workflow to showcase more tasks.

Signed-off-by: Jan Aage Lavik <jan.age.lavik@cern.ch>
workflows: update workflow tasks

* Adds new filtering task for record categories, allowing to
  specify which categories to trigger an action in holding pen.

* Adds task results to objects in various tasks.

* Updates the sample harvesting workflow to use new bibclassify
  API.

* Several other updates and improvements to workflow tasks.

* Updates the category filtering task to allow regular expression
  syntax when defining which categories to halt, continue or stop
  the workflow.

* Adds new data type attribute to workflows, by Jan Aage Lavik.

* Updates the exception handling to better handle exceptions in
  workflows when running a workflow in a BibSched task.

* Improvements to category filter.

* Adds options for the foreach loop, we can now cache the
  list to go through it will then allow to five the choice
  to the user if he wants to priviledge CPU vs Memory,
  cache uses few CPU but a lot of memory and doesn't provide
  a dynamic list. Without the cache option, we recompute the
  list each time so we use much more CPU but we keep just
  the data needed so few memory, and the list is still
  refresh for each loop.

* Refactors use of extra_data in the engine.

* Changes the task foreach so that obj.data takes the value
  it has before the foreach. In the previous version we were
  keeping the last value of the foreach.

* Adds the function filtering_oai_pmh_identifier which
  allows you to filter the identifier. Mainly used to
  avoid processing already processed one.

* Adds features to  inspire_filter_category, now parameter
  can be passed throught the extra_data of the engine.

* Fixes some bugs with regex in inspire_filter_category

* Adds the function update_last_update which allows you to
  update the last_update time of a repository. It is
  important because the user can choose exactly where in
  the workflows he considers that a crash is not a problem
  for harvested documents.

* Adds the different features needed by the change from
  pickle to message pack in Celery, to the container for
  BibWorkflowObject exchange.

* Adds the inspire_filter_custom task which allows to
  filter record by examinating some field of its bibfield
  It is a generalization of inspire_filter_category.

* Reenables all the functions in full_doc_process
  workflow.

* Adds some control over the number of workflows generated
  it is to avoid the memory saturation or the messaging
  queue limit.

* Adds some protected name for workflows to avoid possible
  conflict with new task an new user data etc.

* Add some functions to control workflows states.

* Fixes PEP8 and clean code

* Deletes non-informative logs.

* Modifies the different workflows to take into account the
  previous modification made.

Co-authored-by: Jan Aage Lavik <jan.age.lavik@cern.ch>
workflows: Holding Pen update

* Removes some uninteresting/unused columns in the
  Holding Pen main table and adds a new columns called
  categories containing any record subjects found,
  if applicable.

* Cleans some blueprint code and routes in Holding Pen.

* Refactors the tag system used in the main Holding Pen
  display. Now using a new JS package called bootstrap-tagsinput.

* Changes approval widget design and layout.

* Improves detailed display of workflow definition.

* Updates in harvesting workflow tasks.

* Adds a new template for detailed layout.

* Fixes a problem with hotkeys in the main table display.

* Adds a new bootstrap-tagsinput package for use in the
  Holding Pen display.

* Removes the current config.py and moves the enumerations
  into engine.py and models.py where appropriate and changing
  the naming from CFG_OBJECT_VERSION to ObjectVersion.

* Updates deposit models to use the new ObjectVersion.

* Removes the DATA_TYPE enumeration. This will now be
  dynamically loaded from the database in the holding pen.

* Initial implementation of a new Edit Record widget which
  will be used for (limited) record editing from Holding Pen.
  Still WIP.

* Adds templating blocks for Holding Pen navigation bar and
  record preview.

* Creates a new field rendering macro for better WTForm
  rendering.

Co-authored-by: Jan Aage Lavik <jan.age.lavik@cern.ch>
workflows: update static files on admin pages

* Updates the JavaScript and templates for the workflows admin
  pages and the approval widget to implement the module pattern.

* Refactors some blueprint handlers for the workflows admin
  pages.

* Removes obsolete CSS files.

Signed-off-by: Jan Aage Lavik <jan.age.lavik@cern.ch>
workflows: refactor JavaScript files

* Completely revamps the JavaScript files used in the
  Holding Pen to more closely follow the module pattern.

* Splits the main JavaScript file into smaller pieces.

* Adds get() API to BibWorkflowObject.

Signed-off-by: Jan Aage Lavik <jan.age.lavik@cern.ch>
workflows: fix get_current_task API

* Update the get_current_task member function of
  BibWorkflowObject to work better with workflow logic
  tasks such as IF_ELSE.

Signed-off-by: Jan Aage Lavik <jan.age.lavik@cern.ch>
workflows: further update JavaScript

* Refactors the way URLs are passed to JavaScript, adding
  a new context object used when initializing the JavaScript files.

* JavaScript files for widgets are now dynamically loading, instead
  of being hard-coded in templates. These files are now required
  to be listed in the widget WTForm definition.

* Removes unused Blueprint code and removes no longer
  needed containers.py file, moving it's content inside
  Holding Pen blueprint.

Co-authored-by: Dinos Kousidis <konstantinos.kousidis@cern.ch>
Signed-off-by: Jan Aage Lavik <jan.age.lavik@cern.ch>
workflows: add missing license

* Adds missing license information in templates and workflow
  tasks and also correcting license years.

Signed-off-by: Jan Aage Lavik <jan.age.lavik@cern.ch>
workflows: update HTML attribute

* Updates the HTML attribute used in TD elements in
  row_formatter to conform with standards.

Signed-off-by: Jan Aage Lavik <jan.age.lavik@cern.ch>
workflows: clean up templates

* Reworks and simplifies workflow admin area.

* Cleans up and restructures templates, blueprints and
  static files to better conform with standard structures.

* Removes out-dated templates and JS files and combines
  related templates together.

* Updates unit-tests based on newest changes.

Signed-off-by: Jan Aage Lavik <jan.age.lavik@cern.ch>
workflows: Python 3 compatibility

* Does a pass over the workflows module to replace iteritems,
  unicode, basestring references for Python 3 compatibility.

Signed-off-by: Jan Aage Lavik <jan.age.lavik@cern.ch>
workflows: PEP8 fixes

Signed-off-by: Jan Aage Lavik <jan.age.lavik@cern.ch>
records: aggregation field definitions fix

Signed-off-by: Esteban J. G. Gabancho <egabancho@gmail.com>
workflows: move persistent_ids to extra_data

* Moves the persistent_ids column into an extra_data field.

* Changes tasks appropriately.

Signed-off-by: Jan Aage Lavik <jan.age.lavik@cern.ch>
Data Analysis: Core drafts done
Data Anlysis: Radio fileds fixed