Package: inception2corpus
inception2corpus-CLIA CLI for retrieving a corpus annotated with named entities from INCEpTION instance to an archived and reusable corpus in context of any NER project.This tool was created in the context of the NER4Archives project (INRIA/Archives nationales); it is adaptable and reusable for any other project under the terms of theMIT license.The CLI launches a linear process, called a "pipeline", which executes the components in the following order:Fetch curated documents (XMI format) from an INCEpTION instance (check state of document in Inception > "Monitoring" window);preprocessing curated documents (retokenize, remove unprintable characters etc.);Convert XMI to CONLL files (inception2corpus usexmi2conll clias a module);Merge CONLL files in one;Provides a report containing statistics and metadata about the corpus;Reduce (get only sentences annotated and reject other) and serialize dataset in 2 (train/dev) and 3 sets (train/dev/test) according to a ratio defined by the userAt the end of the execution of the program, anoutput_annotated_corpus folder/is provided at the root working directory, for more details see thissection.🛠️ Installation (easy way)You need Python 3.7 or higher installed (if not, install ithere).First, create a new directory and set up a code environment with virtualenv and correct Python version, follow these steps (depending on your OS):MacOSx / Linuxvirtualenv--python=/usr/bin/python3.7venvthen, activate this new code environment with:sourcevenv/bin/activateWindowspy-mvenvvenvthen, activate this new code environment with:.\venv\Scripts\activateFinally, installinception2corpusCLI via pip with:pipinstallinception2corpus🛠️ Installation (for developers only)# 1. clone git repositorygitclonehttps://github.com/NER4Archives-project/inception2corpus-CLI.git# 2. Go to repository and create a new virtual env (follow steps in easy way installation)# 3. install packages# (on MACOSx/LINUX):pipinstall-rrequirements.txt# (on Windows):pipinstall-r.\requirements.txt▶️ Usageinception2corpusCLI use a YAML file as argument to specify INCEpTION HOST information,
corpus metadata, conll format, serialization options etc.
You can use and update the template hereUSER_VAR_ENV.yml.When configuration YAML file is completed use this command:inception2corpus./USER_VAR_ENV.ymlAt the end of this process, a new output directory is created at the root of working
directory (./output_annotated_corpus folder/) that contains your final corpus, ready to train.
Also, a newtemp_files/folder is created at the root, leave it or delete it as you want.📁 Full output folder description./output_annotated_corpus folder/
 |
 |- output_annotated_corpus folder.zip/
 |           |
 |           |- data_split_n2/ : The all_reduced.conll divided into 2 sets (train, dev)
 |           |
 |           |- data_split_n3/ : The all_reduced.conll divided into 3 sets (train, dev, test)
 |           |
 |           |- data_split_n3_idx/ : The all_reduced.conll divided into 3 sets (train, dev, test) with sentences ID
 |           |
 |           |- data_split_n2_idx/ : The all_reduced.conll divided into 2 sets (train, dev) with sentences ID
 |           |
 |           |- XMI_curated/ : Original XMI to import into INCEpTION
 |           |
 |           |- all.conll : All documents in CONLL format
 |           |- all_reduced.conll : All documents in CONLL format reduced to only annotated sentences
 |
 |- meta_corpus.json : corpus metadata and statistics