Package: potato-annotation
🥔Potato: the POrtable Text Annotation TOol📖Documentation|🍎Feature hightlights|🛫️Quick Start|🌰Example projects (project hub)|🔥Design Team and Support|💰License|🍞Cite usPotato is an easy-to-use  web-based annotation tool accepted by EMNLP 2022 DEMO track. Potato allows you to quickly mock-up and deploy a variety of text annotation tasks. Potato works in the back-end as a web server that you can launch locally and then annotators use the web-based front-end to work through data. Our goal is to allow folks to quickly and easily annotate text data by themselves or in small teams—going from zero to annotating in a matter of a few lines of configuration.Potato is driven by a single configuration file that specifies the type of task and data you want to use. Potato does not require any coding to get up and running. For most tasks, no additional web design is needed, though Potato is easily customizable so you can tweak the interface and elements your annotators see.Please check out ourofficial documentationfor detailed instructions.Jiaxin Pei, Aparna Ananthasubramaniam, Xingyao Wang, Naitian Zhou, Jackson Sargent, Apostolos Dedeloudis and David Jurgens. 🥔Potato: the POrtable Text Annotation TOol. In Proceedings of the 2022 Conference on Empirical Methods on Natural Language Processing (EMNLP'22 demo)Recent Updates1.2.1.6 Supports different HTML templates for surveyflow and annotation pages (link)1.2.1.5 Supports disallowing copy-pasting into textboxes (link)Feature hightlightsPotato supports a wide range of features that can make your data annotation easier:Easy setup and flexible for diverse needsPotato can be easily set up with simply editing a configuration file. You don't need to write any codes to set up your annotation webpage. Potato also comes with a series of features for diverse needs.Built-in schemas and templates: Potato supports a wide range of annotation schemas including radio, likert, checkbox, textbox, span, pairwise comparison, best-worst-scaling, image/video-as-label, etc. All these schemas can beFlexible data types: Potato supports displaying short documents, long documents, dialogue, comparisons, etc..Multi-task setup: NLP researchers may need to set up a series of similar but different tasks (e.g. multilingual annotation). Potato allows you to easily generate configuration files for all the tasks with minimum configurations and has supported theSemEval 2023 Task 9: Multilingual Tweet Intimacy AnalysisImproving Annotator ProductivityPotato is carefully desinged with a series of features that can make your annotators experience better and help you get your annotations faster. You can easily set upKeyboard Shortcuts: Annotators can direcly type in their answers with keyboardsDynamic Highlighting: For tasks that have a lot of labels or super long documents, you can setup dynamic highlighting which will smartly highlight the potential association between labels and keywords in the document (as defined by you).Label Tooltips: When you have a lot of labels (e.g. 30 labels in 4 categories), it'd be extremely hard for annotators to remember all the detailed descriptions of each of them. Potato allows you to set up label tooltips and annotators can hover the mouse over labels to view the description.Knowing better about your annotatorsPotato allows a series of features that can help you to better understand the background of annotators and identify potential data biases in your data.Pre and Post screening questions: Potato allows you to easily set up prescreening and postscreening questions and can help you to better understand the backgrounds of your annotators. Potato comes with a seires of question templates that allows you to easily setup common prescreening questions likedemographics.Better quality controlPotato comes with features that allows you to collect more reliable annotations and identify potential spammers.Attention Test: Potato allows you to easily set up attention test questions and will randomly insert them into the annotation queue, allowing you to better identify potential spammers.Qualification Test: Potato allows you to easily set up qualification test before the full data labeling and allows you to easily identify disqualified annotators.Built-in time check: Potato automatically keeps track of the time annotators spend on each instance and allows you to better analyze annotator behaviors.Quick startinstall potatopypi packagepip install potato-annotationCheck all the available project templatespotato list allGet one from the project hubpotato get sentiment_analysisStart the projectpotato start sentiment_analysisStart directly from the github repoClone the github repo to your computergit clone https://github.com/davidjurgens/potato.gitInstall all the required dependenciespip install -r requirements.txtTo run a simple check-box style annotation on text data, runpython potato/flask_server.py start project-hub/simple_examples/configs/simple-check-box.yaml -p 8000This will launch the webserver on port 8000 which can be accessed athttp://localhost:8000.Clicking "Submit" will autoadvance to the next instance and you can navigate between items using the arrow keys.Theproject-hub/simple_examples/configsfolder contains example.yamlconfiguration files that match many common simple use-cases. See the fulldocumentationfor all configuration options.Baked potatoesPotato aims to improve the replicability of data annotation and reduce the cost for researchers to set up new annotation tasks. Therefore, Potato comes with a list of predefined example projects, and welcome public contribution to the project hub. If you have used potato for your own annotation, you are encouraged to create a pull request and release your annotation setup.Potato currently include the following example projects:simple_schema_examplesdialogue_analysisempathygif_replyimmigration_framingmatch_findingmatch_finding_with_prestudysentiment_analysissummarization_evaluationtextual_uncertaintyquestion_answeringPlease check full list ofbaked potatoesfor more details!Design Team and SupportPotato is run by a small and engergetic team of academics doing the best they can. For support, please leave a issue on this git repo. Feature requests and issues are both welcomed!
If you have any questions or want to collaborate on this project, please emailpedropei@umich.eduorjurgens@umich.eduLicensePotato is dual-licensed. All use cases are covered by Polyform Shield but a commercial license is available for those use cases not allowed by Polyform Shield. Please contact us for details on commercial licensing.FAQ:If I am an open-source developer, can I fork potato and work on it separately?Yes, this is allowed with the licenseIf I am an open-source developer, can I fork potato and publicly release a new version with my own features?No, this is not allowed with the license; such a product would be considered as a “competitor” (see the license for details)If I am working for a company, can I use potato to annotate my data?Yes, this is allowed with the licenseIf I am working for a company, can I use potato within my company’s pipelines for data annotation (e.g., integrate potato within my company’s internal infrastructure)?Yes, this is allowed with the license—we’d love to hear about these to advertise, so please contact us atjurgens@umich.edu.Can I integrate potato within a larger annotation pipeline and release that pipeline as an open-source library or service for others to use?Yes, this is allowed with the license—we’d love to hear about these to advertise, so please contact usCan I integrate potato within a larger annotation pipeline and release that publicly as commercial software/service/resource for others to use?No, this is not allowed by Polyform Shield but commercial licensing of potato for this purpose is available. Please reach out to us atjurgens@umich.edufor details.I am working for a crowdsourcing platform, can I combine potato in our platform to provide better service for my customers?No, this is not allowed by Polyform Shield but commercial licensing of potato for this purpose is available. Please reach out to us atjurgens@umich.edufor details.Have a question or case not covered by the above? Please reach out to us and we’ll add it to the list!Cite usPlease use the following bibtex when referencing this work:@inproceedings{pei2022potato,
  title={POTATO: The Portable Text Annotation Tool},
  author={Pei, Jiaxin and Ananthasubramaniam, Aparna and Wang, Xingyao and Zhou, Naitian and Dedeloudis, Apostolos and Sargent, Jackson and Jurgens, David},
  booktitle={Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations},
  year={2022}
}