BibFormat: allow PoS fulltext links

* Modifies bfe_INSPIRE_links format element to also allow PDFs
  with the "PoS" doctype to be rendered.

Signed-off-by: Jan Aage Lavik <jan.age.lavik@cern.ch>
BibExport: disable fulltext export in sitemap
BibRank: citation_loss_per_record_limit in citation.cfg
Bibharvest: new filter for PoS

* Downloads fulltexts and adds the proper FFT datafield

* creates 3 files:
  append.xml for already existing records
  insert.xml for non existing records
  not_found.xml when fulltext harvest fails

* Adds beatifulsoup4 requirement

* Amends bibharvest/Makefile to setup the new filter

Signed-off-by: Georgios Papoutsakis <georgios.papoutsakis@cern.ch>
BibTasklets: update APS harvest conversion

* Updates the APS harvest tasklet to convert using the new
  ApsPackage available from Harvesting Kit.

* Refactors the threshold date checking as APS has changed
  their format.

* Updates harvesting functions as APS has updated their REST
  API.

Signed-off-by: Jan Aage Lavik <jan.age.lavik@cern.ch>
BibFormat: stricter input data in ref correction form

* Makes sure record_id is an integer in the references correction form.
BibTasklets: various fixes for consyn

* Adds 2 new execution modes to exctract and convert
  files from a zip package or a list of packages.

* Appends a message with the summary of the harvest
  on the end of the bibsched log.

* Reports a summary of the harvest to the desired
  receipient (CFG_CONSYNHARVEST_EMAIL).

* Fixes imports.

* Updates the task to automatically delete
  the CONSYN atom feed file from disk.

Signed-off-by: Georgios Papoutsakis <georgios.papoutsakis@cern.ch>
Reviewed-by: Jan Aage Lavik <jan.age.lavik@cern.ch>
miscutils: obsolete files cleanup

* Removes dateutils files.
BibTasklets: consyn ignores more index pages

* Ignores pages with title "Preface", "List of participants"
  and pages with titles starting with "Papers"

Signed-off-by: Jan Aage Lavik <jan.age.lavik@cern.ch>
BibHarvest: zenodo2inspire filter

* Introduces a new filter for OAI harvesting of Data records
  from Zenodo.

* The script tries to link with articles using DOIs and arXiv IDs.

Co-authored-by: Jan Aage Lavik <jan.age.lavik@cern.ch>
BibHarvest: fix matching in Zenodo filter

* Amends the collection name from DATA to Data when matching
  against existing Zenodo data records.

Signed-off-by: Jan Aage Lavik <jan.age.lavik@cern.ch>
BibEdit: ticket templates for new RT ticket

* Adds plugin templates that are used when creating new tickets
  from within BibEdit.

Reviewed-by: Javier Martin Montull <javier.martin.montull@cern.ch>
BibHarvest: enchancements to PoS filter

* The output files now have the input file path as a prefix.

* InvenioConnector now connects to CFG_SITE_URL

* The fulltext are now stored in a separated folder for each run.

Signed-off by: Georgios Papoutsakis <georgios.papoutsakis@cern.ch>
BibHarvest: use search engine API in PoS filter

* Changes the method used in the PoS filter to query the database
  to use the search engine API instead of InvenioConnector.

Signed-off-by: Jan Aage Lavik <jan.age.lavik@cern.ch>
BibTasklets: use bs4 instead of lxml

* Changes the way the fulltext XML is parsed from lxml.etree
  to bs4.BeautifulSoup to avoid errors with HTML5 entities that
  etree currently do not support.

* Improves submission feedback when running in devmode.

Signed-off-by: Jan Aage Lavik <jan.age.lavik@cern.ch>
BibEdit: add KBs

* Adds new KBs used for BibEdit autocomplete

* New KBs for degree and HEP collections

Reviewed-by: Javier Martin Montull <javier.martin.montull@cern.ch>
kbs: change formatting of degree and HEP kb
bst_fermilab: new bibtasklet for Fermilab

* Introduces official bibtasklet to generate Fermilab
  institutional HTML reports such as:
  <http://bss.fnal.gov/techpubs/fermilab-reports-preprints.html>

* Was based on original Rob's cronjob.

Signed-off-by: Samuele Kaplun <samuele.kaplun@cern.ch>
Co-authored-by: Samuele Kaplun <samuele.kaplun@cern.ch>
bst_fermilab: use pytz for Chicago time

Signed-off-by: Samuele Kaplun <samuele.kaplun@cern.ch>
BibHarvest: fix KeyError exception

* Fixes an issue that in some cases threw a KeyError when
  converting records from CDS to INSPIRE MARCXML.

Signed-off-by: Jan Aage Lavik <jan.age.lavik@cern.ch>