Package: quickner
Quickner ⚡A simple, fast, and easy to use NER annotator for PythonQuickner is a new tool to quickly annotate texts for NER (Named Entity Recognition). It is written in Rust and accessible through a Python API.Quickner is blazing fast, simple to use, and easy to configure using a TOML file.Installation# Create a virtual environmentpython3-mvenvenvsourceenv/bin/activate# Install quicknerpipinstallquickner# or pip3 install quicknerUsageUsing the config filefromquicknerimportQuickner,Configconfig=Config(path="config.toml")# or Config() if the config file is in the current directory# Initialize the annotatorquick=Quickner(config=config)# Annotate the texts using the config filequick.process()# or annotator.process(True) to save the annotated data to a fileUsing DocumentsfromquicknerimportQuickner,Document# Create documentsrust=Document("rust is made by Mozilla")python=Document("Python was created by Guido van Rossum")java=Document("Java was created by James Gosling")# Documents can be added to a listdocuments=[rust,python,java]# Initialize the annotatorquick=Quickner(documents=documents)quick>>>Entities:0|Documents:3|Annotations:>>>quick.documents[Document(id="87e03d58b1ba4d72",text=rustismadebyMozilla,label=[]),Document(id="f1da5d23ef88f3dc",text=PythonwascreatedbyGuidovanRossum,label=[]),Document(id="e4324f9818e7e598",text=JavawascreatedbyJamesGosling,label=[])]>>>quick.entities[]Using Documents and EntitiesfromquicknerimportQuickner,Document,Entity# Create documents from textstexts=("rust is made by Mozilla","Python was created by Guido van Rossum","Java was created by James Gosling at Sun Microsystems","Swift was created by Chris Lattner and Apple",)documents=[Document(text)fortextintexts]# Create entitiesentities=(("Rust","PL"),("Python","PL"),("Java","PL"),("Swift","PL"),("Mozilla","ORG"),("Apple","ORG"),("Sun Microsystems","ORG"),("Guido van Rossum","PERSON"),("James Gosling","PERSON"),("Chris Lattner","PERSON"),)entities=[Entity(*(entity))forentityinentities]# Initialize the annotatorquick=Quickner(documents=documents,entities=entities)quick.process()>>>quickEntities:6|Documents:3|Annotations:PERSON:2,PL:3,ORG:1>>>quick.documents[Document(id=87e03d58b1ba4d72,text=rustismadebyMozilla,label=[(0,4,PL),(16,23,ORG)]),Document(id=f1da5d23ef88f3dc,text=PythonwascreatedbyGuidovanRossum,label=[(0,6,PL),(22,38,PERSON)]),Document(id=e4324f9818e7e598,text=JavawascreatedbyJamesGosling,label=[(0,4,PL),(20,33,PERSON)])]Find documents by label or entityWhen you have annotated your documents, you can use thefind_documents_by_labelandfind_documents_by_entitymethods to find documents by label or entity.Both methods return a list of documents, and are not case sensitive.Example:# Find documents by label>>>quick.find_documents_by_label("PERSON")[Document(id=f1da5d23ef88f3dc,text=PythonwascreatedbyGuidovanRossum,label=[(0,6,PL),(22,38,PERSON)]),Document(id=e4324f9818e7e598,text=JavawascreatedbyJamesGosling,label=[(0,4,PL),(20,33,PERSON)])]# Find documents by entity>>>quick.find_documents_by_entity("Guido van Rossum")[Document(id=f1da5d23ef88f3dc,text=PythonwascreatedbyGuidovanRossum,label=[(0,6,PL),(22,38,PERSON)])]>>>quick.find_documents_by_entity("rust")[Document(id=87e03d58b1ba4d72,text=rustismadebyMozilla,label=[(0,4,PL),(16,23,ORG)])]>>>quick.find_documents_by_entity("Chris Lattner")[Document(id=3b0b3b5b0b5b0b5b,text=SwiftwascreatedbyChrisLattnerandApple,label=[(0,5,PL),(21,35,PERSON),(40,45,ORG)])]Get a Spacy Compatible Generator ObjectYou can use thespacymethod to get a spacy compatible generator object.The generator object can be used to feed a spacy model with the annotated data, you still need to convert the data into DocBin format.Example:# Get a spacy compatible generator object>>>quick.spacy()<builtins.SpacyGeneratorobjectat0x102311440># Divide the documents into chunks>>>chunks=quick.spacy(chunks=2)>>>forchunkinchunks:...print(chunk)...[('rust is made by Mozilla',{'entitiy':[(0,4,'PL'),(16,23,'ORG')]}),('Python was created by Guido van Rossum',{'entitiy':[(0,6,'PL'),(22,38,'PERSON')]})][('Java was created by James Gosling at Sun Microsystems',{'entitiy':[(0,4,'PL'),(20,33,'PERSON'),(37,53,'ORG')]}),('Swift was created by Chris Lattner and Apple',{'entitiy':[(0,5,'PL'),(21,34,'PERSON'),(39,44,'ORG')]})]Single document annotationYou can also annotate a single document with a list of entities.This is useful when you want to annotate a document with a list of entities is not in the list of entities of the Quickner object.Example:fromquicknerimportDocument,Entity# Create a document from a string# Method 1rust=Document.from_string("rust is made by Mozilla")# Method 2rust=Document("rust is made by Mozilla")# Create a list of entitiesentities=[Entity("Rust","PL"),Entity("Mozilla","ORG")]# Annotate the document with the entities, case_sensitive is set to False by default>>>rust.annotate(entities,case_sensitive=True)>>>rustDocument(id="87e03d58b1ba4d72",text=rustismadebyMozilla,label=[(16,23,ORG)])>>>rust.annotate(entities,case_sensitive=False)>>>rustDocument(id="87e03d58b1ba4d72",text=rustismadebyMozilla,label=[(16,23,ORG),(0,4,PL)])Load from fileInitialize the Quickner object from a file containing existing annotations.Quickner.from_jsonlandQuickner.from_spacyare class methods that return a Quickner object and are able to parse the annotations and entities from a jsonl or spaCy file.fromquicknerimportQuicknerquick=Quickner.from_jsonl("annotations.jsonl")# load the annotations from a jsonl filequick=Quickner.from_spacy("annotations.json")# load the annotations from a spaCy fileConfigurationThe configuration file is a TOML file with the following structure:# Configuration file for the NER tool[general]# Mode to run the tool, modes are:# Annotation from the start# Annotation from already annotated texts# Load annotations and add new entities[logging]level="debug"# level of logging (debug, info, warning, error, fatal)[texts][texts.input]filter=false# if true, only texts in the filter list will be usedpath="texts.csv"# path to the texts file[texts.filters]accept_special_characters=".,-"# list of special characters to accept in the text (if special_characters is true)alphanumeric=false# if true, only strictly alphanumeric texts will be usedcase_sensitive=false# if true, case sensitive search will be usedmax_length=1024# maximum length of the textmin_length=0# minimum length of the textnumbers=false# if true, texts with numbers will not be usedpunctuation=false# if true, texts with punctuation will not be usedspecial_characters=false# if true, texts with special characters will not be used[annotations]format="spacy"# format of the output file (jsonl, spaCy, brat, conll)[annotations.output]path="annotations.jsonl"# path to the output file[entities][entities.input]filter=true# if true, only entities in the filter list will be usedpath="entities.csv"# path to the entities filesave=true# if true, the entities found will be saved in the output file[entities.filters]accept_special_characters=".-"# list of special characters to accept in the entity (if special_characters is true)alphanumeric=false# if true, only strictly alphanumeric entities will be usedcase_sensitive=false# if true, case sensitive search will be usedmax_length=20# maximum length of the entitymin_length=0# minimum length of the entitynumbers=false# if true, entities with numbers will not be usedpunctuation=false# if true, entities with punctuation will not be usedspecial_characters=true# if true, entities with special characters will not be used[entities.excludes]# path = "excludes.csv" # path to entities to exclude from the searchFeatures Roadmap and TODOAdd support for spaCy formatAdd support for brat formatAdd support for conll formatAdd support for jsonl formatAdd support for loading annotations from a json spaCy fileAdd support for loading annotations from a jsonl fileFind documents with a specific entity/entities and return the documentsAdd support for loading annotations from a brat fileSubstring search for entities in the text (case sensitive and insensitive)Partial match for entities, e.g. "Rust" will match "Rustlang"Pattern/regex based entites, e.g. "Rustlang" will match "Rustlang 1.0"Fuzzy match for entities with levenstein distance, e.g. "Rustlang" will match "Rust"Add support for jupyter notebookLicenseMOZILLA PUBLIC LICENSE Version 2.0ContributingPull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.Please make sure to update tests as appropriate.Authors[Omar MHAIMDAT]