Package: interpret-eval
ExplainaBoard: An Explainable Leaderboard for NLPIntroduction|Web Tool|API Tool|Download|Paper|Video|BibIntroductionExplainaBoard is an interpretable, interactive and reliable leaderboard with seven (so far) new features (F) compared with generic leaderboard.F1:Single-system Analysis: What is a system good or bad at?F2:Pairwise Analysis: Where is one system better (worse) than another?F3:Data Bias Analysis: What are the characteristics of different evaluated datasets?F5:Common errors: What are common mistakes that top-5 systems made?F6:Fine-grained errors: where will errors occur?F7:System Combination: Is there potential complementarity between different systems?UsageWe not only provide a Web-based Interactive Toolkit but also release an API that users can flexible evaluate their systems offline, which
means, you can play with ExplainaBoard at following levels:U1:Just playing with it: You can walk around, track NLP progress, understand relative merits of different top-performing systems.U2:We help you analyze your model: You submit your model outputs and deploy them into online ExplainaBoardU3:Do it by yourself: You can process your model outputs by yourself using our API.API-based Toolkit: Quick InstallationMethod 1: Simple installation from PyPI (Python 3 only)pip install interpret-evalMethod 2: Install from the source and develop locally (Python 3 only)# Clone current repogitclonehttps://github.com/neulab/ExplainaBoard.gitcdExplainaBoard# Requirementspipinstall-rrequirements.txt# Install the packagepythonsetup.pyinstallThen, you can run following examples via bashinterpret-eval--taskchunk--systems./interpret_eval/example/test-conll00.tsv--outputout.jsonwheretest-conll00.tsvdenotes your system output file whose format depends on different tasks.
For each task we have provided oneexample output fileto show how they are formated.
The above command will generate a detailed report (saved inout.json) for your input system (test-conll00.tsv).
Specifically, following statistics are included:fine-grained performanceConfidence intervalError CaseWeb-based Toolkit: Quick LearningWe deploy ExplainaBoard as aWeb toolkit, which includes 9 NLP tasks,
40 datasets and 300 systems. Detailed information is as follows.So far, ExplainaBoard covers following  tasksTaskSub-taskDatasetModelAttributeSentiment8402Text ClassificationTopics4182Intention132Text-Span ClassificationAspect Sentiment4204Text pair ClassificationNLI267NER3749Sequence LabelingPOS3144Chunking3149CWS7647Structure PredictionSemantic Parsing4124Text GenerationSummarization2367Submit Your ResultsYou can submit your system's output by thisformfollowing the formatdescription.Download System OutputsWe haven't released datasets or corresponding system outputs that require licenses. But If you have licenses please fill in thisformand we will send them to you privately. (Description of output's format can referhereIf these system outputs are useful for you, you cancite our work.AcknowledgementWe thanks all authors who share their system outputs with us: Ikuya Yamada, Stefan Schweter,
Colin Raffel, Yang Liu, Li Dong. We also thank
Vijay Viswanathan, Yiran Chen, Hiroaki Hayashi for useful discussion and feedback about ExplainaBoard.