global: source code reorganization

Signed-off-by: Jiri Kuncar <jiri.kuncar@cern.ch>
Merge branch 'pu' into next

* pu: (141 commits)
  JSONAlchemy: bug fixes and tests improvements
  upgrader: fix auto-generation of upgrades
  upgrader: add auto-generation of upgrades
  deposit: plupload filetable fix
  restful: fix extension initialization
  Grunt: typeahead.js fix
  requirements: qrcode removal
  upgrader: change to module-aware engine
  base: add inveniomanage database diff command
  comments: bind modal on record tab change
  deposit: static file fixes
  comments: fix for JavaScript in record tab
  requirements: better separation
  installation: MANIFEST.in file recursive-include fix
  installation: version modification to 1.9999
  deposit: migrate workflows and fix test
  JSONAlchemy: bug fixes
  JSONAlchemy: allow `extend` on parser extension
  travis: pip --upgrade removal
  requirements: dictdiffer egg fix
  ...

Signed-off-by: Jiri Kuncar <jiri.kuncar@cern.ch>

Conflicts:
	invenio.wsgi
	invenio/celery/config.py
	invenio/ext/cache/__init__.py
	invenio/ext/email/backends/smtp_adminonly.py
	invenio/ext/logging/models.py
	invenio/intbitset.c
	invenio/intbitset.h
	invenio/intbitset.pyx
	invenio/intbitset_helper.py
	invenio/intbitset_impl.c
	invenio/legacy/bibconvert/scripts/bibconvert.py
	invenio/legacy/oaiharvest/daemon.py
	invenio/legacy/websession/password_migration_kit.py
	invenio/modules/cloudconnector/user_settings.py
	invenio/modules/comments/templates/comments/add.html
	invenio/modules/comments/templates/comments/add_review.html
	invenio/modules/deposit/fields/__init__.py
	invenio/modules/deposit/forms/__init__.py
	invenio/modules/deposit/templates/deposit/run.html
	invenio/modules/deposit/templates/deposit/run_action_bar.html
	invenio/modules/deposit/templates/deposit/run_group_start.html
	invenio/modules/deposit/views/api.py
	invenio/modules/error_log/models.py
	invenio/modules/rank/fixtures.py
	invenio/modules/rank/models.py
	invenio/modules/rank/rankext/configuration/bibrankgkb_jif_conv.kb
	invenio/modules/rank/rankext/configuration/bibrankgkb_jif_example.kb
	invenio/modules/rank/rankext/configuration/citation.cfg
	invenio/modules/rank/rankext/configuration/citerank_citation_t.cfg
	invenio/modules/rank/rankext/configuration/citerank_pagerank_c.cfg
	invenio/modules/rank/rankext/configuration/citerank_pagerank_t.cfg
	invenio/modules/rank/rankext/configuration/demo_jif.kb
	invenio/modules/rank/rankext/configuration/gkb.cfg
	invenio/modules/rank/rankext/configuration/selfcites.cfg
	invenio/modules/rank/rankext/configuration/stopwords.de.kb
	invenio/modules/rank/rankext/configuration/stopwords.el.kb
	invenio/modules/rank/rankext/configuration/stopwords.kb
	invenio/modules/rank/rankext/configuration/template_citation.cfg
	invenio/modules/rank/rankext/configuration/template_combine_method.cfg
	invenio/modules/rank/rankext/configuration/template_download_similarity.cfg
	invenio/modules/rank/rankext/configuration/template_download_total.cfg
	invenio/modules/rank/rankext/configuration/template_download_users.cfg
	invenio/modules/rank/rankext/configuration/template_index_term_count.cfg
	invenio/modules/rank/rankext/configuration/template_single_tag_rank_method.cfg
	invenio/modules/rank/rankext/configuration/template_word_similarity.cfg
	invenio/modules/rank/rankext/configuration/template_word_similarity_solr.cfg
	invenio/modules/rank/rankext/configuration/template_word_similarity_xapian.cfg
	invenio/modules/rank/rankext/configuration/wrd.cfg
	invenio/modules/rank/testsuite/test_citation_searcher.py
	invenio/modules/rank/testsuite/test_citerank_indexer.py
	invenio/modules/rank/testsuite/test_donwloads_indexer.py
	invenio/modules/rank/testsuite/test_record_sorter.py
	invenio/modules/rank/testsuite/test_selfcites.py
	invenio/modules/rank/testsuite/test_tag_based_indexer.py
	invenio/modules/ranker/fixtures.py
	invenio/modules/ranker/models.py
	invenio/modules/ranker/rankext/configuration/bibrankgkb_jif_conv.kb
	invenio/modules/ranker/rankext/configuration/bibrankgkb_jif_example.kb
	invenio/modules/ranker/rankext/configuration/citation.cfg
	invenio/modules/ranker/rankext/configuration/citerank_citation_t.cfg
	invenio/modules/ranker/rankext/configuration/citerank_pagerank_c.cfg
	invenio/modules/ranker/rankext/configuration/citerank_pagerank_t.cfg
	invenio/modules/ranker/rankext/configuration/demo_jif.kb
	invenio/modules/ranker/rankext/configuration/gkb.cfg
	invenio/modules/ranker/rankext/configuration/selfcites.cfg
	invenio/modules/ranker/rankext/configuration/stopwords.de.kb
	invenio/modules/ranker/rankext/configuration/stopwords.el.kb
	invenio/modules/ranker/rankext/configuration/stopwords.kb
	invenio/modules/ranker/rankext/configuration/template_citation.cfg
	invenio/modules/ranker/rankext/configuration/template_combine_method.cfg
	invenio/modules/ranker/rankext/configuration/template_download_similarity.cfg
	invenio/modules/ranker/rankext/configuration/template_download_total.cfg
	invenio/modules/ranker/rankext/configuration/template_download_users.cfg
	invenio/modules/ranker/rankext/configuration/template_index_term_count.cfg
	invenio/modules/ranker/rankext/configuration/template_single_tag_rank_method.cfg
	invenio/modules/ranker/rankext/configuration/template_word_similarity.cfg
	invenio/modules/ranker/rankext/configuration/template_word_similarity_solr.cfg
	invenio/modules/ranker/rankext/configuration/template_word_similarity_xapian.cfg
	invenio/modules/ranker/rankext/configuration/wrd.cfg
	invenio/modules/ranker/testsuite/test_citation_searcher.py
	invenio/modules/ranker/testsuite/test_citerank_indexer.py
	invenio/modules/ranker/testsuite/test_donwloads_indexer.py
	invenio/modules/ranker/testsuite/test_record_sorter.py
	invenio/modules/ranker/testsuite/test_selfcites.py
	invenio/modules/ranker/testsuite/test_tag_based_indexer.py
	invenio/modules/records/testsuite/deprecated_test_bibfield_config_engine.py
	invenio/modules/records/testsuite/deprecated_test_bibfield_utils.py
	invenio/modules/search/static/js/search/typeahead.js
	invenio/modules/search/templates/search/collection.html
	invenio/modules/tags/__init__.py
	invenio/modules/tags/template_context_functions/tfn_webtag_record_tags.py
	invenio/modules/tags/templates/tags/display_list.html
	invenio/modules/tags/templates/tags/editor.html
	invenio/modules/tags/templates/tags/record_tags.html
	invenio/modules/tags/validators.py
	invenio/modules/textminer/testsuite/test_textminer_documents.py
	invenio/modules/workflows/__init__.py
	invenio/modules/workflows/static/css/workflows/DT_bootstrap.css
	invenio/modules/workflows/static/js/workflows/DT_bootstrap.js
	invenio/modules/workflows/static/js/workflows/entry_details.js
	invenio/modules/workflows/static/js/workflows/hp_details.js
	invenio/modules/workflows/static/js/workflows/hp_index.js
	invenio/modules/workflows/static/js/workflows/index.js
	invenio/modules/workflows/static/js/workflows/widgets/approval.js
	invenio/modules/workflows/tasks/marcxml_tasks.py
	invenio/modules/workflows/templates/workflows/hp_approval_widget.html
	invenio/modules/workflows/views/holdingpen.py
	invenio/modules/workflows/widgets/approval_widget.py
	invenio/testsuite/intbitset_example.int
	invenio/testsuite/test_datastructures.py
	invenio/testsuite/test_intbitset.py
	invenio/testsuite/test_legacy_elmsubmit.py
	invenio/testsuite/test_utils_datastructures.py
	modules/bibrank/etc/bibrankgkb.cfg.in
	modules/bibrank/etc/bibrankgkb_jif_conv.kb
	modules/bibrank/etc/bibrankgkb_jif_example.kb
	modules/bibrank/etc/citation.cfg
	modules/bibrank/etc/citerank_citation_t.cfg
	modules/bibrank/etc/citerank_pagerank_c.cfg
	modules/bibrank/etc/citerank_pagerank_t.cfg
	modules/bibrank/etc/demo_jif.kb
	modules/bibrank/etc/selfcites.cfg
	modules/bibrank/etc/stopwords.de.kb
	modules/bibrank/etc/stopwords.el.kb
	modules/bibrank/etc/stopwords.kb
	modules/bibrank/etc/template_citation.cfg
	modules/bibrank/etc/template_combine_method.cfg
	modules/bibrank/etc/template_download_similarity.cfg
	modules/bibrank/etc/template_download_total.cfg
	modules/bibrank/etc/template_download_users.cfg
	modules/bibrank/etc/template_index_term_count.cfg
	modules/bibrank/etc/template_single_tag_rank_method.cfg.in
	modules/bibrank/etc/template_word_similarity.cfg
	modules/bibrank/etc/template_word_similarity_solr.cfg
	modules/bibrank/etc/template_word_similarity_xapian.cfg
	modules/bibrank/etc/wrd.cfg
	modules/bibrank/lib/bibrank_citation_searcher_unit_tests.py
	modules/bibrank/lib/bibrank_citerank_indexer_unit_tests.py
	modules/bibrank/lib/bibrank_downloads_indexer_unit_tests.py
	modules/bibrank/lib/bibrank_fixtures.py
	modules/bibrank/lib/bibrank_model.py
	modules/bibrank/lib/bibrank_record_sorter_unit_tests.py
	modules/bibrank/lib/bibrank_selfcites_unit_tests.py
	modules/bibrank/lib/bibrank_tag_based_indexer_unit_tests.py
	modules/bibworkflow/lib/widgets/approval_widget.py
	modules/miscutil/bin/inveniomanage.in
	modules/miscutil/lib/celery_config.py
	modules/miscutil/lib/datastructures_unit_tests.py
	modules/miscutil/lib/errorlib_model.py
	modules/webcomment/etc/templates/webcomment_add.html
	modules/webcomment/etc/templates/webcomment_add_review.html
	modules/websearch/etc/templates/websearch_collection.html
	modules/websearch/lib/facets/Makefile.am
	modules/websession/lib/password_migration_kit.py
	modules/webtag/etc/templates/webtag_display.html
	modules/webtag/etc/templates/webtag_editor.html
	modules/webtag/lib/template_context_functions/tfn_webtag_record_tags.py
	scripts/password_migration_kit.py
classifer: fix classifer registry name

* Fixes the name of the classifer registry which accidentially
  was discovered by bibconvert registry due to name clash.

Signed-off-by: Lars Holm Nielsen <lars.holm.nielsen@cern.ch>
Tested-by: Jiri Kuncar <jiri.kuncar@cern.ch>
BibDocFile: hotfix plugins loading

Signed-off-by: Jiri Kuncar <jiri.kuncar@cern.ch>
oairepository: automatically compute model field

* Connects OaiREPOSITORY model to SQLAlchemy events to ensure field
  setDefinition is properly computed on db insert/update when using
  the SQLAlchemy model.
oairepository: include restricted records

* Adds configuration variable CFG_OAI_FILTER_RESTRICTED_RECORDS
  to include restricted records in OAI-PMH output. Useful if a
  record is part of both public and restricted collections.
comments: remove unused property

* Removes an unused property on the CmtSUBSCRIPTION model, that also
  caused an initialization error due to the user of current_user
  proxy outside of the Flask application context.

Co-authored-by: Jiri Kuncar <jiri.kuncar@cern.ch>
Signed-off-by: Lars Holm Nielsen <lars.holm.nielsen@cern.ch>
formatter: fix for xml record formatting

* Fixes issue in formatting of XML records.
formatter: fix unicode decoding error in rss

* Fixes a unicode decoding issue in the RSS feed.
celery: fix email reporting

* Fixes an issue preventing error emails from being sent.

Signed-off-by: Lars Holm Nielsen <lars.holm.nielsen@cern.ch>
access: improve login performance

* Adds argument batch mode in acc_authorize_action to lower number of
  database queries when testing many arguments against the same action
  and user.

* Improves login performance by using the acc_authorize_action
  argument batch mode to calculate list of permitted collections for
  a user.
persistentid: add function to create url

* Adds a function that for a given persistent identifier will
  try to create a URL to a resolver.
persistentid: fix ISSN validation issue

* Fixes an issue where an 8-character non-ISSN string would cause an
  exception.

Signed-off-by: Lars Holm Nielsen <lars.holm.nielsen@cern.ch>
bower: upgrade ckeditor to version 4

* Upgrades CKEditor to version 4 in bower requirements.
apikeys: add option to disable signing

* Adds option to disable need for signing an API request.
legacy: fix issue with undefined variables

* Fixes issue with undefined variables.
records: add configurable breadcrumb title

* Fixes an issue where the breadcrumb did not display the record title,
  due to using a wrong jsonalchemy field.
searchengine: fix restricted collection search

* Fixes an issue where a guest user could not search for records
  in a restricted collection authorized for to the guest user.
base: add pre-template-render signal

* Adds signal that can be sent from views prior to
  a template being rendered. This allows simple customizations
  of the template context.
global: translation fixes

* Adds configuration file for message extraction.

* Extracts messages from source code files.

* Usage: python setup.py extract_messages -k '_(' \
  -F mapping.cfg -w 100 --input-dirs invenio \
  -o invenio/base/translations/invenio.pot --sort-output

Signed-off-by: Eirini Psallida <eirini.psallida@cern.ch>
Reviewed-by: Jiri Kuncar <jiri.kuncar@cern.ch>