Package: corrupted-text
Corrupted-Text: Realistic Out-of-Distribution TextsA python library to generate out-of-distribution text datasets.
Specifically, the library appliesmodel-independent,commonplace corruptions(not model-specific, worst-case adversarial corruptions).
We thus aim to allow benchmark-studies regarding robustness againstrealistic outliers.Implemented CorruptionsMost corruptions are based on a set ofcommon words, to which a corruptor is fitted. Thesecommon wordsmay be
domain specific and thus, the corruptor can be fitted with abase datasetfrom which the most common words are
extracted.Then, the following corruptions are randomly applied on a per-word basis:Bad AutocorrectionWords are replaced with another, common word to which it has a small levenshtein distance. This mimicks wrong
autocorrection, as for example done by "intelligent" mobile phone keyboards.Bad AutocompletionWords are replaced with another, common word with the same starting letters. This mimicks wrong autocompletion. If no
common word with at least 3 common start letters is found, a bad autocorrection is attempted instead.Bad SynonymWords are replaced with a synonym, accoring to a naive, flat mapping extracted
fromWordNet, ignoring the context. This mimicks dictionary based translations,
which are often wrong. This assumes that you are using an english-language dataset.TypoA single letter is replaced with another, randomly chosen letter.To any word, at most one corruption is applied, i.e., corruptions are not applied on top of each other.The severity]0, 1]is a parameter to steer how many corruptions should be applied. It roughly corresponds to the
percentage of words that should be corrupted
(onlyrouglyas not all bad autocompletion attempts are successful, and as sometimes, the bad synonyms consist of
multiple words, thus extending the number of words in the text).Optionally, users can define weights to each corruption type, steering how often they should be applied.AccuraciesThe following shows the accuracy of a regular, simple transformer model on the imdb sentiment classification dataset.
Clearly, the higher the chosen corruption severity, the lower the model accuracy.Severity0 (*)0.10.30.50.70.91 (max!)Accuracy.87.81.78.75.710.660.64(*) No corruption, original test set.InstallationIt's as simple aspip install corrupted-text.You'll need python >= 3.7UsageUsage is very straigthforward.
The following shows an example on how to corrupt the imdb sentiment classification dataset.You can also run the example in colab:importcorrupted_text# pip install corrupted-textimportloggingfromdatasetsimportload_dataset# pip install datasets# Enable Detailed Logginglogging.basicConfig(level=logging.INFO)# Load the dataset (we use huggingface-datasets, but any list of strings is fine).nominal_train=load_dataset("imdb",split="train")["text"]nominal_test=load_dataset("imdb",split="test")["text"]# Fit a corruptor (we fit on the training and test set,#   but as this takes a while, you'd want to choose a smaller subset for larger datasets)corruptor=corrupted_text.TextCorruptor(base_dataset=nominal_test+nominal_train,cache_dir=".mycache")# Corrupt the test set with severity 0.5. The result is again a list of corrupted strings.imdb_corrupted=corruptor.corrupt(nominal_test,severity=0.5,seed=1)Citation@inproceedings{Weiss2022SimpleTip, 
  title={Simple Techniques Work Surprisingly Well for Neural Network Test Prioritization and Active Learning (Replication Paper)},
  author={Weiss, Michael and Paolo, Tonella}, 
  booktitle={Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis},
  year={2022}
}Other Corrupted DatasetsMNIST-Cby Mu and GilmerCIFAR-10-Cby Hendrycks and DietterichImagenet-Cby Hendrycks and DietterichFashion-MNIST-Cby Weiss and Tonella (i.e., same ascorrupted-text)