Package: errudite
ErruditeThis opensourcing is work-in-progress!Errudite is an interactive tool for scalable, reproducible, and counterfactual error analysis.
Errudite provides an expressive domain-specific language for extracting relevant features of
linguistic data, which allows users to visualize data attributes, group relevant instances,
and perform counterfactual analysis across all available validation data.Getting StartedWatchthis video demothat contains the highlights of Errudite's functions & use casesGetset upquicklyTryErrudite's user interfaceon machine comprehensionTry thetutorials on JupyterLab notebooksRead thedocumentationCitationIf you are interested in this work, please see ourACL 2019 research paperand consider citing our work:@inproceedings{2019-errudite,
    title = {Errudite: Scalable, Reproducible, and Testable Error Analysis},
    author = {Wu, Tongshuang and Ribeiro, Marco Tulio and Heer, Jeffrey and Weld Daniel S.},
  booktitle={the 57th Annual Meeting of the Association for Computational Linguistics (ACL 2019)},
    year = {2019},
    url = {https://homes.cs.washington.edu/~wtshuang/files/acl2019_errudite.pdf},
}Quick StartInstallationPIPErrudite requires Python 3.6.x. The package is avaiable throughpip:
Just install it in your Python environment and you're good to go!# create the virtual environmentvirtualenv--no-site-packages-ppython3.6venv# activate venvsourcevenv/bin/activate# install erruditepipinstallerruditeInstall from sourceYou can also install Errudite by cloning our git repository:gitclonehttps://github.com/uwdata/erruditeCreate a Python 3.6 virtual environment, and install Errudite ineditablemode by running:pipinstall--editable.This will makeerruditeavailable on your system but it will use the sources from the local clone
you made of the source repository.GUI ServerErrudite has a UI wrapped for Machine Comprehension and Visual Question Answering tasks.
The interface integrates all the key analysis functions (e.g., inspecting instance attributes,
grouping similar instances, rewriting instances), It also provides exploration
support such as visualizing data distributions, suggesting potential queries, and presenting the
grouping and rewriting results. While not strictly necessary, it makes their application much
more straightforward.To get a taste of GUI for the machine comprehension task, you should first download a cache folder
for preprocessedSQuADinstances, which will help you
skip the process of running your own preprocessing:python -m errudite.download

Commands:
    cache_folder_name
                A folder name. Currently, we allow downloading the following:
                squad-100, squad-10570.
    cache_path  A local path where you want to save the cache folder to.Then, we need to start the server:# the model relies on Allennlp, so make sure you install that first.pipinstallallennlp==0.8.4sourcevenv/bin/activate
python-merrudite.server

Commands:config_fileAyamlconfigfilepath.The config file looks like the following (or inconfig.yml):task: qa # the task, should be "qa" and "vqa".
cache_path: {cache_path}/{cache_folder_name}/ # the cached folder.
model_metas: # a model.
- name: bidaf
  model_class: bidaf # an implemented model class
  model_path: # a local model file path
  # an online path to an Allennlp model
  model_online_path: https://s3-us-west-2.amazonaws.com/allennlp/models/bidaf-model-2017.09.15-charpad.tar.gz
  description: Pretrained model from Allennlp, for the BiDAF model (QA)
attr_file_name: null # It set, to load previously saved analysis.
group_file_name: null
rewrite_file_name: nullThen visithttp://localhost:5000/in your web browser.JupyterLab TutorialBesides used in a GUI, errudite also serves as a general python package. The tutorial goes
through:Preprocessing the data, and extending Errudite to different tasks & predictorsCreating data attributes and data groups with a domain specific language (or your customized functions).Creating rewrite rules with the domain specific language (or your customized functions).To go through the tutorial, do the following steps:# clone the repogitclonehttps://github.com/uwdata/errudite# initial folder: errudite/# create the virtual environmentvirtualenv--no-site-packages-ppython3.6venv# activate venvsourcevenv/bin/activate# run the default setup scriptpipinstall--editable.# get to the tutorial folder, and start!cdtutorials
pipinstall-rrequirements_tutorials.txt
jupyterlab