Package: pylighter
PyLighter: Annotation tool for NER tasksPyLighter is a tool that allows data scientists to annotate a corpus of documents directly on Jupyter for NER (Named Entity Recognition) tasks.ContentsInstallationBasic usageAdvanced usageUsing an already annotated corpusChanging labels namesDocument stylingAdding additional informationAdding additional outputsUsing keyboard shortcutsContributingTestingLicenseInstallationFrom Pypi:https://pypi.org/project/pylighter/pip install pylighter
jupyter nbextension enable --py widgetsnbextensionFrom Github:https://github.com/PayLead/PyLightergit clone git@github.com:PayLead/PyLighter.git
cd PyLighter
python setup.py install
jupyter nbextension enable --py widgetsnbextensionDemosThedemofolder contains working examples of PyLighter in use. To view them, open any of the ipynb files in Jupyter.Basic usageThe use case of PyLighter is to easily annotate a corpus in Jupyter. So let's first define a corpus for this example:corpus=["PyLighter is an annotation tool for NER tasks directly on Jupyter. "+"It aims on helping data scientists easily and quickly annotate datasets. "+"This tool was developed by Paylead.","PayLead is a fintech company specializing in transaction data analysis. "+"Paylead brings retail and banking together, so customers get rewarded when they buy. "+"Welcome to the data-for-value economy."]Now let's start annotating !frompylighterimportAnnotationannotation=Annotation(corpus)Running that cell gives you the following output:You can know start annotating entities using the predefined labelsl1,l2, etc.When your annotation is finished, you can either click on the save button or retrieve the results in the current Notebook.The save button will save the results in a csv file namedannotation.csvwith two columns: the documents and the labels.You can access the labels of your annotations inannotation.labelsNote: The given labels are in IOB2 format.Advanced usageThe above example works just fine but PyLighter can be customized to best fit your specific use case.Using an already annotated corpusIn most cases, you want to use an already annotated corpus or simply continue your annotation.To this, you can use the argument namedlabelswith the labels of the corpus. Moreover, if you stopped at the ithdocument, you can directly get back to where you stopped withstart_index=i.You can see more on that withthisdemo.Changing labels namesPyLighter usesl1,l2, ....,l7as default labels names, but in most cases, you want to have explicit labels such asNoun,Verb, etc.You can define your own labels names with the argumentlabels_names. You can also define your own colors for your labels with the argumentlabels_colorsin HEX format.You can see more on that withthisdemo.Document stylingYou can adjust the font size, the minimal distance between two characters and the size of spaces with the argumentchar_params.Default value for char_params is:# Each field expects css value as a string (ex:"10px", "1em", "large", etc.)char_params={"font_size":"medium","width_white_space":"1Opx","min_width_between_chars":"4px",}Adding additional informationIn some cases, you may want to know additional information about the current document, such as the source of it.To do this, you can use the argumentadditional_infos. This argument must be a pandas DataFrame of shape (size of the corpus,number of additional information). The ithrow of the DataFrame will be associated with the ithelement of the corpus.The elements of the given DataFrame need to have a proper string representation to be correctly displayed.For instance, to add the source to each element of the corpus:importpandasaspd# define corpus of size 2additional_infos=pd.DataFrame({"source":["Github","Paylead.fr"]})annotation=Annotation(corpus,additional_infos=additional_infos)The result will be:You can see more on that withthisdemo.Adding additional outputsIn some cases, you want to flag a document as difficult to annotate, or spot as wrong, or give a value that estimates your confidence in your annotation, etc. In short, you need to return additional information.To do this, you can use the argument:additional_outputs_elements. This argument expects a list ofpylighter.AdditionalOutputElement.Apylighter.AdditionalOutputElementis defined like this:frompyligherimportAdditionalOutputElementAdditionalOutputElement(name="name_of_my_element",display_type="type_of_display"# checkbox, int_text, float_text, text, text_areadescription="Description of the element to display",default_value="Default value for the element")Here is an example:Note: Additional outputs will be added to the save file. But you can also retrieve them withannotation.additional_outputs_values. You can also use previously returned additional outputs values with the argument:additional_outputs_values(same as the label).You can see more on that withthisdemo.Using keyboard shortcutsAnnotation tasks are pretty boring. Thus you may want to use keyboard shortcuts to easily change documents or to select an other label.By default, there are only a few shortcuts defined:next:Alt + nprevious:Alt + pskip:Alt + ssave:Shift + Alt + sHowever, you can fully customize them with the arguments:standard_shortcutsandlabels_shorcuts. Thestandard_shortcutsargument is used to redefined shortcuts for the standard buttons such as the next button whereas theA shortcut is defined like this:frompylighterimportShortcutShortcut(name="skip",# Name of the button to bind on (ex: "next", "skip") or name of the label (ex: "l1", "l2", or one you defined)key="Ò",# Usually represents the character that is displayed.code="KeyS",# Usually represents the key that is pressed.shift_key=False,# Wether the shift key is pressedalt_key=True,ctrl_key=False)It is pretty hard to know what is the value for thekeyand the value for thecode. It depends on a lot of different factors such as your keyboard, your browser, etc.Thus, you can use theShortcutHelperto pick the right shortcut. Here is an example of it.frompylighterimportShortcutHelperShortcutHelper()You can see more on that withthisdemo.ContributingTestingPyLighter usespytest. Thus, tests can be run with:make testPyLighter usesflake8,isortandcheck-manifestto control the quality of the code. You can test the quality of the code with:make test-qualityIf you wish to test everything including the packaging, you can run:make test-allLicenseMIT License