RefExtract: hdl handling

Conflicts:
	modules/docextract/lib/refextract_engine.py
	modules/docextract/lib/refextract_regression_tests.py
BibField: clean up of atlantis.cfg demo file

* Fixes field names and legacy rules to match demo site tag setup.
 (closes #1557)

Signed-off-by: Esteban J. G. Gabancho <esteban.gabancho@cern.ch>
Tested-by: Tibor Simko <tibor.simko@cern.ch>
BibField: new producer section added to config

* New `producer` section inside the configuration file that allows
  to produce different low level outputs for a given field.

* First release of two producers, one for marc and one for dublin
  core.

* New functions `guess_legacy_field_names` that, using the legacy
  rules from the configuration files try to find which is the field
  name for a given tag from a given master format.

* Better legacy export as marc that uses the new producer capability.

Signed-off-by: Esteban J. G. Gabancho <esteban.gabancho@cern.ch>
Co-authored-by: Lars Holm Nielsen <lars.holm.nielsen@cern.ch>
Tested-by: Tibor Simko <tibor.simko@cern.ch>
BibField: fix for misbehavior when parsing rules

Signed-off-by: Esteban J. G. Gabancho <esteban.gabancho@cern.ch>
Co-authored-by: Lars Holm Nielsen <lars.holm.nielsen@cern.ch>
Tested-by: Tibor Simko <tibor.simko@cern.ch>
BibField: better handling of calculated fields

* Adds new internal key `__calculated_functions` to store how to
  update the cached value of the field.

* All the fields marked with `@do_not_cache` are not longer stored
  stored inside rec_json, only the key is reserved.

Signed-off-by: Esteban J. G. Gabancho <esteban.gabancho@gmail.com>
Tested-by: Tibor Simko <tibor.simko@cern.ch>
BibField: new decorator @only_if_master_format

* New decorator @only_if_master_value, similar to @only_if but in
  this case we can create a boolean expression based on `value`,
  which is coming from the master format before evaluation.

Signed-off-by: Esteban J. G. Gabancho <esteban.gabancho@cern.ch>
Tested-by: Tibor Simko <tibor.simko@cern.ch>
BibField: bibdoc integration

* Adds two new calculated fields to interact with files from
  BibDocFile, one `files` which provides a dict containing some
  useful information about the files and `bibdocs` which contains
  the BibDoc object itself.

Signed-off-by: Esteban J. Gabancho <esteban.gabancho@cern.ch>
Tested-by: Tibor Simko <tibor.simko@cern.ch>
BibField: continuable vs fatal errors

Signed-off-by: Esteban J. G. Gabancho <esteban.gabancho@cern.ch>
Tested-by: Tibor Simko <tibor.simko@cern.ch>
BibField: fix for `is_type_isbn`

Signed-off-by: Esteban J. G. Gabancho <esteban.gabancho@cern.ch>
Tested-by: Tibor Simko <tibor.simko@cern.ch>
BibField: fix for '__eq__' to better compare recs

Signed-off-by: Esteban J. G. Gabancho <esteban.gabancho@cern.ch>
Tested-by: Tibor Simko <tibor.simko@cern.ch>
BibField: `is_empty` update

Signed-off-by: Esteban J. G. Gabancho <esteban.gabancho@cern.ch>
Tested-by: Tibor Simko <tibor.simko@cern.ch>
BibField: `schema` in `split_blob`

* When splitting the blob it might be the case that the schema is
  needed, moreover if the reader accepts several schemas for the
  same input format.

Signed-off-by: Esteban J. G. Gabancho <esteban.gabancho@cern.ch>
Tested-by: Tibor Simko <tibor.simko@cern.ch>
BibField: elimination of None values in recjson

Signed-off-by: Esteban J. G. Gabancho <esteban.gabancho@cern.ch>
Tested-by: Tibor Simko <tibor.simko@cern.ch>
BibField: new field filtering for `get_record`

* Modified `__repr__` to only display relevant keys and hide keys
  of the form `__key`.

Signed-off-by: Esteban J. G. Gabancho <esteban.gabancho@cern.ch>
Tested-by: Tibor Simko <tibor.simko@cern.ch>
BibField: fix number_of_copies field

* Fixes problem with numbers_of_copies field definition
  which fixes also recjson cache recreation.  (closes #1625)

Reported-by: Theodoropoulos Theodoros <theod@lib.auth.gr>
Reviewed-by: Tibor Simko <tibor.simko@cern.ch>
BibField: improvements backported from next

Co-authored-by: Tibor Simko <tibor.simko@cern.ch>
BibField: change recid field type to integer

* Fixes problems with fields like `number_of_citations` where
  an integer was needed instead of a string.  (closes #1633)

Signed-off-by: Esteban J. G. Gabancho <esteban.gabancho@cern.ch>
Reviewed-by: Tibor Simko <tibor.simko@cern.ch>
BibField: no caching of calculated fields

* Introduces no caching of calculated fields as a workaround for issues
  related to cache invalidation.  (E.g. when the citation indexer runs
  and updates citation counts, the recjson cache is not invalidated,
  either directly or via some time expiry mechanism, so that the number
  of citations would remain invalid until some process would touch
  record metadata.)

* NOTE The cache invalidation issue will be properly addressed later.

Signed-off-by: Tibor Simko <tibor.simko@cern.ch>
Co-authored-by: Esteban J. G. Gabancho <egabancho@gmail.com>
BibField: backported improvements from pu

* Major update of BibField module with some speed improvements and
  enhancements from pu branch.

* New functional test to verify that the speed of recjson is below two
  times of that of recstruct.  (closes #1687)

Signed-off-by: Esteban J. G. Gabancho <esteban.gabancho@gmail.com>
Tested-by: Tibor Simko <tibor.simko@cern.ch>

Conflicts:
	modules/bibfield/etc/atlantis.cfg
	modules/bibfield/lib/bibfield_marcreader_regression_tests.py
	modules/bibfield/lib/bibfield_regression_tests.py
	modules/bibfield/lib/bibfield_utils_unit_tests.py
	requirements.txt
OPS: backported travis from master