Package: igannotator
Institutional Grammar 2.0 annotatorAboutPython tool for processing and tagging sentences withIG 2.0 syntaxwith additional tools for text cleaning, preprocessing and postprocessing.ManualInstallationCreate a virtual environment:python -m venv .envActivate the virtual environment:source .env/bin/activateInstall packagepython -m pip install --upgrade pip
python -m pip install igannotator
ig-cliChain of command-line toolsig-cliPossible tasks are executed as shell commands on files:ig-cli <task_type> <input_file_path> -o output_file_path --some-additional-optionHelpTo show information about possible commands, arguments and options execute:ig-cli -hSplit text document into sentencesInput:Plain.txtfile with text.Output:Plain.txtfile with sentences separated by new empty lines or.tsvfile with ['sentence no.', 'sentence_type', 'text'] columns
(with optional parameter--format=tsv)Command:ig-cli atomize input_text.txt
ig-cli atomize input_text.txt -o sentences.txt --split_type ml
ig-cli atomize input_text.txt --format txtOptional parameters--format (txt/tsv)--output_file_path--split_type (ml/rule_based)About:Complex sentences with enumerations are splitted into atomic sentences when it is possible. (xxx xxx (a) ccc, (b) vvv” -> “xxx xxx ccc”, “xxx xxx vvv”).Split type possible values: ‘ml’, ‘rule_based’. ML variant uses  a special tool (Spacy library) for recognizing the beginnings and ends of sentences in text. Rule-based variant uses simple matching based on capital letter and period at the end of the sentence (regular expressions).These two are different approaches and can give different results. The basic option is rule_based, ml can do better with lower quality text because of considering whole sentence structure (not only dots and capital letters).Both splits recognize enumeration based on a, b, c… or 1, 2, 3… to split bigger sentences into smaller ones. Which is implemented as matching such expressions (xxx xxx (a) ccc, (b) vvv”) in the sentence, then splitting and constructing new sentences from extracted parts (“xxx xxx ccc”, “xxx xxx vvv”).For example:The employee is subject to  (1) a Federal quarantine order related to COVID-19 (2) a Federal isolation order related to COVID-19.The employee is subject to a Federal quarantine order related to COVID-19.The employee is subject to a Federal isolation order related to COVID-19.Sentences 2-3 are extracted from sentence 1 based on(1) (2)pattern.Assign sentence typeInput:Plain.txtfile with sentences separated by new lines or.tsvfile with 3 columns ['sentence no.', 'sentence_type', 'text']. (Based on file extension)Output:.tsvfile with 3 columns: ['sentence no.', 'sentence_type', 'text'].Command:ig-cli classify sentences.tsvOptional parameters--output_file_pathAbout:Sentences are classified as regulative (r) or constitutive (c). For this purpose, simple ML model is prepared trained on a small annotated dataset. The output file should be reviewed and corrected manually.IG tagging:Input:.tsv file with 3 columns ['sentence no.', text, 'sentence_type'] compatible with results ofclassifycommand.Output:.tsv file with tagged sentencesCommand:ig-cli tag classified_sentences.tsv -o tagged_sentences.tsvOptional parameters--output_file_pathAbout:Tagging is based on natural language processing with linguistic features recognition
and rules constructed for mapping linguistic features to Institutional Grammar tags.
Every sentence is analysed accordingly then results are saved with tags corresponding to each word token.Conversion to horizontal Excel format of IG document  (in the future)Input:Output:Command:About:Comparison of resultsComparison between files (e.g. for quality/error assessment) is possible via other tools such as (diff- command line tool (usediff -hfor detailed instruction),diffchecker- web tool)Technical informationUpdate of modelsSentence type classification-The ML modelcan be changed/retrained as a new file with serialized Python object with.predict(self, sentences: List[str]) -> List[bool]method and returns True for regulative sentences. Corrected files can be gathered for building better classifier.Programming interfaceThe package can be used withinimport igannotatorwith object-oriented operations included inigannotator.backendand file operations included inigannotator.frontend.from igannotator import backend
backend.get_annotated_sentences(df)ContributionsThe tool is based on the results of previous work on Institutional Grammar annotation:Group project for the previous version of IG syntax and Polish language -linkWork by Aleksandra Wichrowska on developing rules for English language and new IG 2.0 syntax -linkWork by Karolina Seweryn on ML models: constitutive/regulative classification