Package: econll
eCoNLL: Extended CoNLL Utilities for Shallow ParsingShallow ParsingSequence Labeling and ClassificationClassificationis the problem of identifying to which of a set of categories (subpopulations) a new observation belongs,
on the basis of a training set of data containing observations (or instances) whose category membership is known.Sequence Labelingis a type of pattern recognition task that involves the algorithmic assignment of a categorical label to each member
of a sequence of observed values.
It is a subclass ofstructured (output) learning,
since we are predicting asequenceobject rather than a discrete or real value predicted in classification problems.Sequence Labeling and Shallow ParsingShallow Parsingis a kind of Sequence Labeling.
The main difference from Sequence Labeling task,
such asPart-of-Speech Tagging,
where there is an output label (tag) per token;
Shallow Parsing additionally performschunking-- segmentation of input sequence into constituents.
Chunking is required to identify categories (or types) ofmulti-word expressions.In other words, we want to be able to capture information that expressions like"New York"that consist of 2 tokens,
constitute a single unit.What this means in practice is that Shallow Parsing performsjointly(or not) 2 tasks:Segmentationof input into constituents (spans)Classification(Categorization, Labeling) of these constituents into predefined set of labels (types)CoNLL Corpus FormatCorpus inCoNLLformat consists of series of sentences, separated by blank lines.
Each sentence is encoded using a table (or "grid") of values, where each line corresponds to a single word,
and each column corresponds to an annotation type (such as various token-level features & labels).The set of columns used by CoNLL-style files can vary from corpus to corpus.Since a line in a data can correspond to any token (word or not), it is referred to by a more general termtoken.
Similarly, since a data can be composed of units more or less than a sentence,
a new line separated unit is referred to asblock.Encoding Segmentation InformationIOB SchemeThe notation scheme is used to labelmulti-wordspans in token-per-line format,
e.g. thatNew Yorkis aLOCATIONconcept that has 2 tokens.As such a token-leveltagconsists of anaffixthat encodes segmentation information
andlabelthat encodes type information.
Consequently, the corpustagsetconsists of all possibleaffixandlabelcombinations.A segment encoded withaffixes and assigned alabelis referred to aschunk.Both, prefix and suffix notations are commons:prefix:B-LOCsuffix:LOC-BMeaning of AffixesIfor Inside of spanOfor Outside of span (no prefix or suffix, justO)Bfor Beginning of spanAlternative SchemesNo affix (useful when there are no multi-word concepts)IO: deficient withoutBIOB: see aboveIOBE:Efor End of span (LinBILOUfor Last)IOBES:Sfor Singleton (UinBILOUfor Unit)EvaluationThere are several methods to evaluate performance of shallow parsing models.
They can be evaluated attoken-level and atchunk-level.Token-Level EvaluationThe unit of evaluation in this case is atagof atoken,
and what is evaluated is how accurately a model assigns tags to tokens.
Consequently, thetoken(ortag)accuracymeasures the amount of correctly predictedtags.Since atagconsists of anaffix-labelpair,
it is additionally possible to separately computeaffixandlabelperformances.Chunk-Level EvaluationThe unit of evaluation in this case is achunk, and the evaluation is "joint";
in the sense that it jointly evaluates segmentation and labeling.
That is, atrueunit is the one that has correctlabelandspan.Similar to token-level evaluation, it is possible to evaluate segmentation independently of labeling.
This is achieved ignoring thechunklabel, e.g. by converting all of them to a single label.WhyeCoNLL?Token-level evaluation is readily available from a number of packages,
and can be easily computed usingscikit-learn'sclassification_report, for instance.Chunk-level evaluation was originally provided byconllevalperl scriptwithin CoNLL Shared Tasks.
However, the one limitation ofconllevalis that it does not supportIOBESorBILOUschemes.Theconllevalscript was ported to python numerous times, and these ports have various functionalities.
One notable port isseqeval,
which is also included in Hugging Face'sevaluatepackage.InstallationTo installeconllrun:pip install econllUsageIt is possible to runeconllfrom command-line, as well as to import the methods.Command-Line Usageusage: PROG [-h] -d DATA [-r REFS] 
            [--separator SEPARATOR] [--boundary BOUNDARY] [--docstart DOCSTART] 
            [--kind {prefix,suffix}] [--glue GLUE] [--otag OTAG]
            [-f {conll,parse,mdown}] [-o OUTS]
            [{eval,conv}]

eCoNLL: Extended CoNLL Utilities

positional arguments:
  {eval,conv}        task to perform

options:
  -h, --help            show this help message and exit

I/O Arguments:
  -d DATA, --data DATA  path to data/hypothesis file
  -r REFS, --refs REFS  path to references file

Data Format Arguments:
  --separator SEPARATOR
                        field separator string
  --boundary BOUNDARY   block separator string
  --docstart DOCSTART   doc start string

Tag Format Arguments:
  --kind {prefix,suffix}
                        tag order
  --glue GLUE           tag separator
  --otag OTAG           outside tag

Data Conversion Arguments:
  -f {conll,parse,mdown}, --form {conll,parse,mdown}
                        output format (kind)
  -o OUTS, --outs OUTS  path to output fileEvaluationpython -m econll -d DATA
python -m econll eval -d DATA
python -m econll eval -d DATA -r REFSConversionpython -m econll conv -d DATA -f FORMAT -o PATHVersioningThis project adheres toSemantic Versioning.