Package: pigeonxt-jupyter
🐦 pigeonXT - Quickly annotate data in Jupyter LabPigeonXT is an extention to the originalPigeon, created byAnastasis Germanidis.
PigeonXT is a simple widget that lets you quickly annotate a dataset of
unlabeled examples from the comfort of your Jupyter notebook.PigeonXT currently support the following annotation tasks:binary / multi-class classificationmulti-label classificationregression taskscaptioning tasksAnything that can be displayed on Jupyter
(text, images, audio, graphs, etc.) can be displayed by pigeon
by providing the appropriatedisplay_fnargument.Additionally, custom hooks can be attached to each row update (example_process_fn),
or when the annotating task is complete(final_process_fn).There is a full blog post on the usage of PigeonXT onTowards Data Science.ContributorsAnastasis GermanidisDennis BakhuisRitesh AgrawalDeepak TunuguntlaBram van EsInstallationPigeonXT obviously needs a Jupyter Lab environment. Futhermore, it requires ipywidgets.
The widget itself can be installed using pip:pipinstallpigeonXT-jupyterCurrently, it is much easier to install due to Jupyterlab 3:
To run the provided examples in a new environment using Conda:condacreate--namepigeonpython=3.9condaactivatepigeonpipinstallnumpypandasjupyterlabipywidgetspigeonXT-jupyterFor an older Jupyterlab or any other trouble, please try the old method:condacreate--namepigeonpython=3.7condaactivatepigeoncondainstallnodejspipinstallnumpypandasjupyterlabipywidgetsjupyternbextensionenable--pywidgetsnbextensionjupyterlabextensioninstall@jupyter-widgets/jupyterlab-managerpipinstallpigeonXT-jupyterStarting Jupyter Lab environment:jupyterlabDevelopment environmentI have moved the development environment to Poetry. To create an identical environment use:condaenvcreate-fenvironment.yml
condaactivatepigeonxt
poetryinstall
pre-commitinstallExamplesExamples are also provided in the accompanying notebook.Binary or multi-class text classificationCode:importpandasaspdimportpigeonXTaspixtannotations=pixt.annotate(['I love this movie','I was really disappointed by the book'],options=['positive','negative','inbetween'])Preview:Multi-label text classificationCode:importpandasaspdimportpigeonXTaspixtdf=pd.DataFrame([{'example':'Star wars'},{'example':'The Positively True Adventures of the Alleged Texas Cheerleader-Murdering Mom'},{'example':'Eternal Sunshine of the Spotless Mind'},{'example':'Dr. Strangelove or: How I Learned to Stop Worrying and Love the Bomb'},{'example':'Killer klowns from outer space'},])labels=['Adventure','Romance','Fantasy','Science fiction','Horror','Thriller']annotations=pixt.annotate(df,options=labels,task_type='multilabel-classification',buttons_in_a_row=3,reset_buttons_after_click=True,include_next=True,include_back=True,)Preview:Image classificationCode:importpandasaspdimportpigeonXTaspixtfromIPython.displayimportdisplay,Imageannotations=pixt.annotate(['assets/img_example1.jpg','assets/img_example2.jpg'],options=['cat','dog','horse'],display_fn=lambdafilename:display(Image(filename)))Preview:Audio classificationCode:importpandasaspdimportpigeonXTaspixtfromIPython.displayimportAudioannotations=pixt.annotate(['assets/audio_1.mp3','assets/audio_2.mp3'],task_type='regression',options=(1,5,1),display_fn=lambdafilename:display(Audio(filename,autoplay=True)))annotationsPreview:multi-label text classification with custom hooksCode:importpandasaspdimportnumpyasnpfrompathlibimportPathfrompigeonXTimportannotatedf=pd.DataFrame([{'example':'Star wars'},{'example':'The Positively True Adventures of the Alleged Texas Cheerleader-Murdering Mom'},{'example':'Eternal Sunshine of the Spotless Mind'},{'example':'Dr. Strangelove or: How I Learned to Stop Worrying and Love the Bomb'},{'example':'Killer klowns from outer space'},])labels=['Adventure','Romance','Fantasy','Science fiction','Horror','Thriller']shortLabels=['A','R','F','SF','H','T']df.to_csv('inputtestdata.csv',index=False)defsetLabels(labels,numClasses):row=np.zeros([numClasses],dtype=np.uint8)row[labels]=1returnrowdeflabelPortion(inputFile,labels=['yes','no'],outputFile='output.csv',portionSize=2,textColumn='example',shortLabels=None,):ifshortLabels==None:shortLabels=labelsout=Path(outputFile)ifout.exists():outdf=pd.read_csv(out)currentId=outdf.index.max()+1else:currentId=0indf=pd.read_csv(inputFile)examplesInFile=len(indf)indf=indf.loc[currentId:currentId+portionSize-1]actualPortionSize=len(indf)print(f'{currentId+1}-{currentId+actualPortionSize}of{examplesInFile}')sentences=indf[textColumn].tolist()forlabelinshortLabels:indf[label]=NonedefupdateRow(example,selectedLabels):print(example,selectedLabels)labs=setLabels([labels.index(y)foryinselectedLabels],len(labels))indf.loc[indf[textColumn]==example,shortLabels]=labsdeffinalProcessing(annotations):ifout.exists():prevdata=pd.read_csv(out)outdata=pd.concat([prevdata,indf]).reset_index(drop=True)else:outdata=indf.copy()outdata.to_csv(out,index=False)annotated=annotate(sentences,options=labels,task_type='multilabel-classification',buttons_in_a_row=3,reset_buttons_after_click=True,include_next=False,example_process_fn=updateRow,final_process_fn=finalProcessing)returnindfdefgetAnnotationsCountPerlabel(annotations,shortLabels):countPerLabel=pd.DataFrame(columns=shortLabels,index=['count'])forlabelinshortLabels:countPerLabel.loc['count',label]=len(annotations.loc[annotations[label]==1.0])returncountPerLabeldefgetAnnotationsCountPerlabel(annotations,shortLabels):countPerLabel=pd.DataFrame(columns=shortLabels,index=['count'])forlabelinshortLabels:countPerLabel.loc['count',label]=len(annotations.loc[annotations[label]==1.0])returncountPerLabelannotations=labelPortion('inputtestdata.csv',labels=labels,shortLabels=shortLabels)# counts per labelgetAnnotationsCountPerlabel(annotations,shortLabels)Preview:The complete and runnable examples are available in the provided Notebook.