Package: nessie
nessieis a package for annotation error detection. It can be used to automatically detect errors in annotated
corpora so that human annotators can concentrate on a subset to correct, instead of needing to look at each
and every instance.💡Please also refer to ouradditional documentation! It contains detailed explanations and code examples!Contact person:Jan-Christoph Kliehttps://www.ukp.tu-darmstadt.dehttps://www.tu-darmstadt.deDon't hesitate to report an issue if something is broken (and it shouldn't be) or if you have further questions.⚠️This repository contains experimental software and is published for the sole purpose of giving additional background details on the respective publication.Please use the following citation when using our software:@misc{https://doi.org/10.48550/arxiv.2206.02280,
  doi = {10.48550/ARXIV.2206.02280},  
  url = {https://arxiv.org/abs/2206.02280},  
  author = {Klie, Jan-Christoph and Webber, Bonnie and Gurevych, Iryna},  
  title = {Annotation Error Detection: Analyzing the Past and Present for a More Coherent Future},  
  publisher = {arXiv},  
  year = {2022}
}Installationpip install nessieThis installs the package with default dependencies and PyTorch with only CPU support.
If you want to use your own PyTorch version (e.g., with CUDA enabled), you need to install it afterwards manually.
If you needfaiss-gpu, then you should also install that manually afterwards.Basic UsageGiven annotated data, this package can be used to find potential errors. For instance, usingRetag, that is,
training a model, letting it predict on your data and then flagging instances where model predictions disagree
with the given labels can be done as:fromnessie.dataloaderimportload_example_text_classification_datafromnessie.helperimportCrossValidationHelperfromnessie.models.textimportDummyTextClassifierfromnessie.detectorsimportRetagtext_data=load_example_text_classification_data().subset(100)cv=CrossValidationHelper(n_splits=10)tc_result=cv.run(text_data.texts,text_data.noisy_labels,DummyTextClassifier())detector=Retag()flags=detector.score(text_data.noisy_labels,tc_result.predictions)💡Please also refer to ouradditional documentation! It contains detailed explanations and code examples!MethodsWe implement a wide range of annotation error detection methods. These are divided in two categories,flaggersandscorers.Flaggersgive a binary judgement whether an instance is considered wrong,Scorersgive a certainty
estimate how likely it is that an instance is wrong.FlaggerAbbreviationMethodTextTokenSpanProposed byCLConfident Learning✓✓✓Northcutt (2021)CSCurriculum Spotter✓Amiri (2018)DEDiverse Ensemble✓✓✓Loftsson (2009)IRTItem Response Theory✓✓✓Rodriguez (2021)LALabel Aggregation✓✓✓Amiri (2018)LSLeitner Spotter✓Amiri (2018)PEProjection Ensemble✓✓✓Reiss (2020)RERetag✓✓✓van Halteren (2000)VNVariation n-Grams✓✓Dickinson (2003)ScorerAbbreviationMethodTextTokenSpanProposed byBCBorda Count✓✓✓Larson (2020)CUClassification Uncertainty✓✓✓Hendrycks (2017)DMData Map Confidence✓✓✓Swayamdipta (2020)DUDropout Uncertainty✓✓✓Amiri (2018)KNNk-Nearest Neighbor Entropy✓✓✓Grivas (2020)LELabel Entropy✓✓Hollenstein (2016)MDMean Distance✓✓✓Larson (2019)PMPrediction Margin✓✓✓Dligach (2011)WDWeighted Discrepancy✓✓Hollenstein (2016)ModelsModel-based annotation detection methods need trained models to obtain predictions or probabilities.
We already implemented the most common models for you to be ready to use.
You can add your own models by implementing the respective abstract class forTextClassifierorSequenceTagger.
We provide the following models:Text classificationClass nameDescriptionFastTextTextClassifierFasttextFlairTextClassifierFlairLgbmTextClassifierLightGBM with handcrafted featuresLgbmTextClassifierLightGBM with S-BERT featuresMaxEntTextClassifierLogistic Regression with handcrafted featuresMaxEntTextClassifierLogistic with S-BERT featuresTransformerTextClassifierTransformersYou can easily add your own sklearn classifiers by subclassingSklearnTextClassifierlike the following:class MaxEntTextClassifier(SklearnTextClassifier):
    def __init__(self, embedder: SentenceEmbedder, max_iter=10000):
        super().__init__(lambda: LogisticRegression(max_iter=max_iter, random_state=RANDOM_STATE), embedder)Sequence ClassificationClass nameDescriptionFlairSequenceTaggerFlairCrfSequenceTaggerCRF with handcrafted featuresMaxEntSequenceTaggerMaxent sequence taggerTransformerSequenceTaggerTransformerDevelopmentWe useflitfor dependency management and packaging.
Follow their documentation to install it. Then you can runflit install -sto download the dependencies and install in its own environment.
In order to install your own PyTorch with CUDA, you can runmake force-cuda113or install it manually in the poetry environment. You can format the code viamake formatwhich should be run before every commit.BibliographyAmiri, Hadi, Timothy Miller, and Guergana Savova. 2018."Spotting Spurious Data with Neural Networks."Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), 2006-16. New Orleans, Louisiana.Dligach, Dmitriy, and Martha Palmer. 2011."Reducing the Need for Double Annotation."Proceedings of the 5th Linguistic Annotation Workshop, 65-73. Portland, Oregon, USA.Grivas, Andreas, Beatrice Alex, Claire Grover, Richard Tobin, and William Whiteley. 2020."Not a Cute Stroke: Analysis of Rule- and Neural Network-based Information Extraction Systems for Brain Radiology Reports."Proceedings of the 11th International Workshop on Health Text Mining and Information Analysis, 24-37. Online.Hendrycks, Dan, and Kevin Gimpel. 2017."A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks."Proceedings of International Conference on Learning Representations, 1-12.Hollenstein, Nora, Nathan Schneider, and Bonnie Webber. 2016."Inconsistency Detection in Semantic Annotation."Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), 3986-90. Portorož, Slovenia.Larson, Stefan, Anish Mahendran, Andrew Lee, Jonathan K. Kummerfeld, Parker Hill, Michael A. Laurenzano, Johann Hauswald, Lingjia Tang, and Jason Mars. 2019."Outlier Detection for Improved Data Quality and Diversity in Dialog Systems."Proceedings of the 2019 Conference of the North, 517-27. Minneapolis, Minnesota.Loftsson, Hrafn. 2009."Correcting a POS-Tagged Corpus Using Three Complementary Methods."Proceedings of the 12th Conference of the European Chapter of the ACL (EACL 2009), 523-31. Athens, Greece.Northcutt, Curtis, Lu Jiang, and Isaac Chuang. 2021."Confident Learning: Estimating Uncertainty in Dataset Labels."Journal of Artificial Intelligence Research 70 (April): 1373-1411.Reiss, Frederick, Hong Xu, Bryan Cutler, Karthik Muthuraman, and Zachary Eichenberger. 2020."Identifying Incorrect Labels in the CoNLL-2003 Corpus."Proceedings of the 24th Conference on Computational Natural Language Learning, 215-26. Online.Rodriguez, Pedro, Joe Barrow, Alexander Miserlis Hoyle, John P. Lalor, Robin Jia, and Jordan Boyd-Graber. 2021."Evaluation Examples Are Not Equally Informative: How Should That Change NLP Leaderboards?"Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 4486-4503. Online.Swayamdipta, Swabha, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A. Smith, and Yejin Choi. 2020."Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics."Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 9275-93. Online.van Halteren, Hans. 2000."The Detection of Inconsistency in Manually Tagged Text."Proceedings of the COLING-2000 Workshop on Linguistically Interpreted Corpora, 48-55. Luxembourg.