workflows: abstraction layer for workers

* Adds an abstract class to be used like a base for
  worker wrapper.

* Implements a worker wrapper for abstraction for
  celery worker.

* Starts the implementation of a worker wrapper for
  redis worker.

* Fixes PEP8 and sphinx documentation specification.

* Fixes mistake about iteritems with six.

Signed-off-by: Guillaume Lastecoueres <guillaume.lastecoueres@cern.ch>
Reviewed-by: Jan Aage Lavik <jan.age.lavik@cern.ch>
workflows: more unit tests

* Adds more unit tests to the workflows module, including
  new test workflows. Coverage should now be about 50%.

* Modifies API internals for unit tests conformation.

* Adds one file for tests which are using asynchronous
  worker to compute.

* Corrects unit tests.

Signed-off-by: Guillaume Lastecoueres <guillaume.lastecoueres@cern.ch>
Reviewed-by: Jan Aage Lavik <jan.age.lavik@cern.ch>
workflows: global improvement

* Updates workflow to comply with the modification in
  tasks libraries.

* Fixes an error with the new ObjectVersion type with
  python3.

* Fixes sphinx mistakes.

* Removes useless regex.

* Fixes PEP8 mistakes.

* Adds more documentation to some functions.

* Removes useless and obsolete code, factorize some parts.

* Changes the start workflow function to give the
  possibility to start from a new clean BibWorkflowObject.

* Adds the refresh frequency as a parameter for workflows
  synchronization.

Signed-off-by: Guillaume Lastecoueres <guillaume.lastecoueres@cern.ch>
Reviewed-by: Jan Aage Lavik <jan.age.lavik@cern.ch>
oaiharvest: PEP8 and errors category

* Fixes PEP8 errors in a strict way.

* Allows log of errors of loglevel=40 in oaiharvest.

Signed-off-by: Guillaume Lastecoueres <guillaume.lastecoueres@cern.ch>
Reviewed-by: Jan Aage Lavik <jan.age.lavik@cern.ch>
workflows: overhaul update

* Adds more documentation.

* Fixes some sphinx errors in documentation.

* Changes imports to relives one when possible.

* Adds a new way to handle errors, to allow easy reading.

* Removes obfuscations of exceptions.

* Clarifies exceptions.

* Updates the way exception unicodes are handling.

* Fixes the behavior of stop_on_error.

* Adds Python 3 compatibility where possible.

* Adds a method to clean extra_data in a Workflow Engine object.

* Fixes problems with Python 2.6 loggers.

* Adds reliable way to handle types for BibWorkflowObject.

* Adds a method to get widget message easily.

* Adds an abstraction method for jsonalchemy conversion in
  workflow tasks.

* Adds more tests.

* Adds start test procedure.

* Fixes bad sorting instructions in Holding Pen.

* Improves the way of restarting workflows.

* Simplifies celery worker.

* Reorganizes tree of the module.

* Tasks cleaning and optimizations.

Signed-off-by: Guillaume Lastecoueres <guillaume.lastecoueres@cern.ch>
Reviewed-by: Jan Aage Lavik <jan.age.lavik@cern.ch>
workflows: Holding Pen improvement

* Removes selection of row on link click.

* Changes the way of update data in approval widget.

* Changes the way of handling widget in pages.

* Changes the way of managing tags.

* Adds retro-compatibility for _widget field in extra_data.

* Simplifies the handling of informations in the HTTP
  request.

* Adds new way to extract data from Workflow object.

Signed-off-by: Guillaume Lastecoueres <guillaume.lastecoueres@cern.ch>
Reviewed-by: Jan Aage Lavik <jan.age.lavik@cern.ch>
workflows: str() addition to calls

* Fixes an issue in the MARCXML tasks that caused legacy
  create_record to fail when a string is passed as unicode.

Signed-off-by: Jan Aage Lavik <jan.age.lavik@cern.ch>
workflows: Holding Pen display changes

* Replaces the 'source' column with a more useful
  'identifier' column in the main HP display.

* Tries to factor out some of the data extraction
  happening when displaying object data in Holding Pen.
  Will be revisited in the near future.

Signed-off-by: Jan Aage Lavik <jan.age.lavik@cern.ch>
workflows: Holding Pen stabilization

* Adds search feature through tags.

* Simplifies the sort code server side.

* Gets request parameters before application ones.

* Adds data extraction methods.

* Changes the way we are creating object to removes
  cyclic dependencies.

Signed-off-by: Guillaume Lastecoueres <guillaume.lastecoueres@cern.ch>
Reviewed-by: Jan Aage Lavik <jan.age.lavik@cern.ch>
workflows: Holding Pen object history

* Adds the history of an object on the details page.

Signed-off-by: Guillaume Lastecoueres <guillaume.lastecoueres@cern.ch>
Reviewed-by: Jan Aage Lavik <jan.age.lavik@cern.ch>
workflows: merging correction

* Fixes import format.

* Fixes tests presentation.

* Fixes unit tests.

* Fixes PEP257.

* Fixes useless UTF-8 conversion.

* Fixes _task_result type modification.

* Fixes Python 3 compatibility.

* Reformats the JavaScript code.

Signed-off-by: Guillaume Lastecoueres <guillaume.lastecoueres@cern.ch>
Reviewed-by: Jan Aage Lavik <jan.age.lavik@cern.ch>
workflows: test cleanup

* Makes sure that all objects are cleaned up after running
  tests.

Signed-off-by: Jan Aage Lavik <jan.age.lavik@cern.ch>
workflows: INSPIRE task removal

* Removes specific tasks created for the INSPIRE site.

Signed-off-by: Jan Aage Lavik <jan.age.lavik@cern.ch>
workflows: better widget and filtering

* Changes the way we do the filtering.

* Reduces number of requests.

* Fixes workflows problem with persistent data.

* Fixes approval widget.

Signed-off-by: Guillaume Lastecoueres <guillaume.lastecoueres@cern.ch>
Reviewed-by: Jan Aage Lavik <jan.age.lavik@cern.ch>
workflows: Holding Pen UI/UX update

* Fixes a bug about the creation date of snapshot.

* Fixes not taking snapshot when continuing a workflow.

* Fixes the order in the history.

* Removes details column.

* Adds details when clicking on the title.

* Hides search field of the data-table.

* Add a mapping to convert Object version to their correct
  printable text.

Signed-off-by: Guillaume Lastecoueres <guillaume.lastecoueres@cern.ch>
Reviewed-by: Jan Aage Lavik <jan.age.lavik@cern.ch>
workflows: missing RQ worker code

* Adds missing RQ worker code for .get() and .status().

Signed-off-by: Jan Aage Lavik <jan.age.lavik@cern.ch>
deposit: addition of dynamic KB autocomplete func

Signed-off-by: Javier Martin Montull <javier.martin.montull@cern.ch>
Reviewed-by: Jiri Kuncar <jiri.kuncar@cern.ch>
deposit: PEP8 and PEP257 improvements

* FIXME there are still many docstrings that do not comply PEP257.

Signed-off-by: Jiri Kuncar <jiri.kuncar@cern.ch>