Package: piereling
Pierelingis a webservice and web-application to convert between a
variety of document formats and to and from theFormat for Linguistic
Annotation(FoLiA). It is intended
to be used in Natural Language Processing pipelines. Piereling itself
does not actually implement the convertors but relies on numerous other
specialised conversion tools in combination with notable third-party
tools such aspandocto accomplish its goals.Pierelingis the word for earthworm in Limburgish dialect. Data
conversion forms thegroundworkfor linguistic annotation, and thse
little worms, eating the input file on one end and secreting a
conversion on its outer end, perform that job.We useFoLiAas our pivot format
so you will mostly encounter conversions from or to FoLiA. FoLiA is a
format for Linguistic Annotation that also incorporates elaborate
document structure and mark-up facilities. Another important
intermediate format used in many of our conversions through pandoc isReStructuredText, a
lightweight markup format. Although, Pandoc support a huge number of
conversions between all its supported document formats, it is beyond the
scope of his project to offer those in the webservice.Available ConversionsConversions to FoLiAFrom Document and Markup Formatsfromplain text; usestxt2foliafromFoLiA-Tools.In addition to an attempted extraction of text structure
(paragraphs) by detecting blank lines, it also supports
one-sentence-per-line and one-paragraph-per-line styles.If you can deliver your input as ReStructuredText or Markdown then
this is is strongly preferred if you want to preserve structure
and markup, as these formats resolve a lot of ambiguity inherent
in unspecified plain text.Information loss: NonefromReStructuredText; usingrst2foliafromFoLiA-Tools.Information loss: Minimal to None (almost all rst structures are
supported)fromMarkdown; via ReStructuredText usingpandocand thenrst2foliafromFoLiA-Tools.Information loss: Minimal to None (most markdown structures are
supported; exceptions are mathematical formula)fromHTML; via ReStructuredText usingpandocand thenrst2foliafromFoLiA-Tools.Information loss: Some; complex layout, complex tables, and
imagery will generally get lost. Should only be used for
semantically clean and simple HTML rather than complex
presentational HTML from the web.fromWord(Office Open XML, docx); via ReStructuredText usingpandocand thenrst2foliafromFoLiA-Tools.Information loss: Some; complex layout, complex tables, and
imagery will generally get lost.Note that the Word 2007 DOC format from up until 2007 is not
supported, only the modern DOCX variant.fromOpenDocument Text(LibreOffice, odt); via ReStructuredText
usingpandocand thenrst2foliafromFoLiA-Tools.Information loss: Some; complex layout, complex tables, and
imagery will generally get lost.fromEPUB; via ReStructuredText usingpandocand thenrst2foliafromFoLiA-Tools.Information loss: Some; complex layout, complex tables, and
imagery will generally get lost.fromLaTeX; via ReStructuredText usingpandocand thenrst2foliafromFoLiA-Tools.Information loss: Some to considerable; complex layout, complex
tables, custom packages, math, and imagery will generally get
lost.fromMediaWiki(as used by Wikipedia); via ReStructuredText usingpandocand thenrst2foliafromFoLiA-Tools.Information loss: Some; complex layout, complex tables. Wikipedia
specific elements.fromDocBook; via ReStructuredText usingpandocand thenrst2foliafromFoLiA-Tools.Information loss: UnknownfromTEI P5 XML(Text Encoding Initiative); usestei2foliafromFoLiA-Tools.TEI is a very extensive and flexible format with many different
formsInformation loss: Our converter will only work for a certain
subset of TEI, mostly equivalent to TEI Lite, and may fail on
others. Though we support a lot of TEI elements, there is also
still a lot that is not covered by the converter. There will be
comments in the output for anything that could not be converted
properly.fromPDF; usespdftotextfromPopplerand thentxt2foliafrom FoLiA-tools.Only works for PDFs with embedded text, not for imagery which
would require OCR!Information loss:Considerable!PDF conversion is notoriously
difficult, the layout of the document will most probably get lost
in the conversion (especially in case of multi-columned output).
The markup will get lost too.Structural conversion is very inaccurate (i.e. paragraphs will not
be nicely mapped) and produces ugly FoLiA.Always avoid this conversion if you can!fromhOCR; usesFoLiA-hocrfromfoliautils.hOCR is a standard format outputted by OCR systems such asTesseract.Information loss: UnknownfromALTO; usesFoLiA-altofromfoliautils.ALTO is a standard format for the description of text OCR and
layout information of pages for digitized material.Information loss: UnknownFrom other Linguistic Annotation FormatsfromNAF(NLP Annotation Format) to FoLiA; usesnaf2foliafromNAFFoLiAPy.This converter is still in an early and experimental stage.Information loss: Not all annotation layers are implemented yet.
Those that are should suffer minimal to no information loss. See
thewebsitefor details.fromCONLL-U; usesconllu2foliafromFoLiA-Tools.Information loss: NonefromAlpino XML; usesalpino2foliafromFoLiA-Tools.Information loss: Minimal to NoneConversions from FoLiAtoplain text, usesfolia2txtfromFoLiA-Tools.Information loss: Considerable, as only the text will be outputted
and any annotations, most structure, and all markup will be lost.
The text itself, however, will be very accurately converted, in
either tokenised (if available) or untokenised form.toHTML; this conversion is offered through the default viewer in
the web-interface.Information loss: Minimal, but information is represented purely
for presentational purposes rather than focussing on semantics.toReStructuredText, usesfolia2rstfromFoLiA-Tools.Information loss: Structure and mark-up will be preserved, but
annotations will be lost!Validation & UpgradeFoLiA validation; usingfoliavalidatorfromFoLiA-Tools.FoLiA upgrade; upgrades an older FoLiA version to a newer one (mostly
inteneded for FoLiA v1 to FoLiA v2); usesfoliaupgradefromFoLiA-Tools.InstallationInstall using pip (preferably in a Python virtual environment):pip install pierelingPiereling is supplied as part of ourLaMachinedistribution, which
includes all dependencies out of the box. If you don’t use this, you
will need to take care of installing certain dependencies yourself in
order for all convertors to work, this includes:pandocfoliautilspoppler-utilsFor production use, we recommend using uwsgi in combination with a
webserver such as Apache (with mod_uwsgi_proxy), or nginx. A uwsgi
configuration has been generated (piereling.example.ini); it is
specific to the host you deploy the webservice on. This in turn loads
the wsgi script (piereling.wsgi), which loads your webservice.Sample configurations for nginx and Apache have been generated as a
starting point, add these to your server and then use the./startserver_production.shscript to launch CLAM using uwsgi. If
you use LaMachine, all this has already been set up for you.UsageRunclamservice piereling.pierelingto start thedevelopmentserver and then navigate your browser to the address printed.WebPiereling is a RESTful webservice and also provides a web-interface for
human end users (powered byCLAM).
If you instead seek to do conversions locally on the command line then
you have no need for Piereling and should simply invoke the
aforementioned conversion tools directly.A public instance of this webservice is available athttps://webservices-lst.science.ru.nl/piereling, register for a
free account athttps://webservices-lst.science.ru.nlfirst.Related ToolsIf you want to convert to TEI, or use TEI as a pivot format for
conversions, then you can look atOxGarage(source) andOpenConvert.