Package: invenio-classifier
Invenio module for record classification.Free software: GPLv2 licenseDocumentation:https://pythonhosted.org/invenio-classifierFeaturesClassifier automatically extracts keywords from fulltext documents. The
automatic assignment of keywords to textual documents has clear benefits
in the digital library environment as it aids catalogization,
classification and retrieval of documents.Keyword extraction is simpleNoteClassifier requires PythonRDFLibin order
to process the RDF/SKOS taxonomy.In order to extract relevant keywords from a documentfulltext.pdfbased on a controlled vocabularythesaurus.rdf, you would run
Classifier as follows:${INVENIO_WEB_INSTANCE}classifierextract-kthesaurus.rdf-ffulltext.pdfLaunching${INVENIO_WEB_INSTANCE} classifier--helpshows the options available.As an example, running classifier on documentnucl-th/0204033using the
high-energy physics RDF/SKOS taxonomy (HEP.rdf) would yield the
following results (based on the HEP taxonomy from October 10th 2008):Input file: 0204033.pdf

Author keywords:
Dense matter
Saturation
Unstable nuclei

Composite keywords:
10  nucleus: stability [36, 14]
6  saturation: density [25, 31]
6  energy: symmetry [35, 11]
4  nucleon: density [13, 31]
3  energy: Coulomb [35, 3]
2  energy: density [35, 31]
2  nuclear matter: asymmetry [21, 2]
1  n: matter [54, 36]
1  n: density [54, 31]
1  n: mass [54, 16]

Single keywords:
61  K0
23  equation of state
12  slope
4  mass number
4  nuclide
3  nuclear model
3  mass formula
2  charge distribution
2  elastic scattering
2  binding energyThesaurusClassifier performs an extraction of keywords based on the recurrence
of specific terms, taken from a controlled vocabulary. A controlled
vocabulary is a thesaurus of all the terms that are relevant in a
specific context. When a context is defined by a discipline or branch of
knowledge then the vocabulary is said to be asubject thesaurus.
Various existing subject thesauri can be foundhere.A subject thesaurus can be expressed in several different formats.
Different institutions/disciplines have developed different ways of
representing their vocabulary systems. The taxonomy used by classifier
is expressed in RDF/SKOS. It allows not only to list keywords but to
specify relations between the keywords and alternative ways to represent
the same keyword.<Conceptrdf:about="http://cern.ch/thesauri/HEP.rdf#scalar"><compositerdf:resource="http://cern.ch/thesauri/HEP.rdf#Composite.fieldtheoryscalar"/><prefLabelxml:lang="en">scalar</prefLabel><notexml:lang="en">nostandalone</note></Concept><Conceptrdf:about="http://cern.ch/thesauri/HEP.rdf#fieldtheory"><compositerdf:resource="http://cern.ch/thesauri/HEP.rdf#Composite.fieldtheoryscalar"/><prefLabelxml:lang="en">fieldtheory</prefLabel><altLabelxml:lang="en">QFT</altLabel><hiddenLabelxml:lang="en">/fieldtheor\w*/</hiddenLabel><notexml:lang="en">nostandalone</note></Concept><Conceptrdf:about="http://cern.ch/thesauri/HEP.rdf#Composite.fieldtheoryscalar"><compositeOfrdf:resource="http://cern.ch/thesauri/HEP.rdf#scalar"/><compositeOfrdf:resource="http://cern.ch/thesauri/HEP.rdf#fieldtheory"/><prefLabelxml:lang="en">fieldtheory:scalar</prefLabel><altLabelxml:lang="en">scalarfield</altLabel></Concept>In RDF/SKOS, every keyword is wrapped around aconceptwhich
encapsulates the full semantics and hierarchical status of a term -
including synonyms, alternative forms, broader concepts, notes and so on
- rather than just a plain keyword.The specification of the SKOS language andvarious
manualsthat aid the building of a semantic thesaurus can be found at theSKOS
W3C
website.
Furthermore, Classifier can function on top of an extended version of
SKOS, which includes special elements such as key chains, composite
keywords and special annotations.Keyword extractionClassifier computes the keywords of a fulltext document based on the
frequency of thesaurus terms in it. In other words, it calculates how
many times a thesaurus keyword (and its alternative and hidden labels,
defined in the taxonomy) appears in a text and it ranks the results.
Unlike other similar systems, Classifier does not use any machine
learning or AI methodologies - a just plain phrase matching usingregular expressions: it
exploits the conformation and richness of the thesaurus to produce
accurate results. It is then clear that Classifier performs best on top
of rich, well-structured, subject thesauri expressed in the RDF/SKOS
language.Happy hacking and thanks for flying Invenio-Classifier.Inspirehep Development TeamEmail:admin@inspirehep.netTwitter:http://twitter.com/inveniosoftwareGitHub:https://github.com/inveniosoftware-contrib/invenio-classifierURL:http://inveniosoftware.orgChangesVersion 1.3.2 (release 2018-03-12)Bug fixesPrevent the “unprintable KeyError” problem from happening.Version 1.3.1 (release 2017-06-23)Bug fixesReverse sort core keywords.Version 1.3.0 (release 2017-06-22)Incompatible changesExports list of objects rather than list of tuples, to help ES
directly indexing the content.Version 1.2.0 (release 2017-06-21)Incompatible changesDo not use keywords as dictionary keys, rather as elements in a list.Version 1.1.2 (release 2017-05-22)Bug fixesSupports ‘·’ author separatorSupport utf8 author-keywordsVersion 1.1.1 (release 2017-05-19)Bug fixesEnforce utf8 also for non PDF files in extractor.Version 1.1.0 (release 2017-05-17)Incompatible changesChanges dict export format for author keywords, into an improved and semantic
way.Renames keys in dict export to be lower case and separated by_.Bug fixesDrop trailing dots in author keywords.Version 1.0.1 (release 2017-01-11)Incompatible changesChanges module to be compatible with Invenio 3.Bug fixesFixes a crash when trying to discover a taxonomy when
CLASSIFIER_WORKDIR is set to None.Updates minimum dependencies of Invenio packages to newer versions.Removes a bug in bibclassify_keyword_analyzer.py. If a combination
is found via a synonym or regexp it is no longer thrown away just
because the components of the combination are not found in the text.Adds missinginvenio_basedependency.Version 0.1.0 (release 2015-08-19)Initial public release.