Package: text-correction-benchmarks
Text correction benchmarksBenchmarks and baselines for various text correction tasks. Light-weight and
easy to use.Installationpip install text-correction-benchmarksorgit clone https://github.com/ad-freiburg/text-correction-benchmarks
cd text-correction-benchmarks && pip install .After installation you will have two commands available to you:tcb.evaluatefor evaluating model predictions on benchmarkstcb.baselinefor running baselines on benchmarksUsageThis repository containsbenchmarksfor text correction tasks such asWhitespace correction (wsc)Spelling error correction (sec)Word-level spelling error detection (sedw)Sequence-level spelling error detection (seds)in the following simple text file format:Whitespace correctioncorrupt.txt: Input text with whitespace errors (can also contains spelling errors, but they remain uncorrected in the groundtruth)correct.txt: Groundtruth text without whitespace errors"Th isis a tset." > "This is a tset."Spelling error correctioncorrupt.txt: Input text with spelling errors (can also contain whitespace errors to make the task harder)correct.txt: Groundtruth text without whitespace and spelling errors"Th isis a tset." > "This is a test."Word-level spelling error detection:corrupt.txt: Input text with spelling errors (should not contain whitespace errors, since we assume they are already fixed)correct.txt: Groundtruth label for each word in the input (split by whitespace), indicating whether a word contains a spelling error (1)
or not (0)"This is a tset." > "0 0 0 1"Sequence-level spelling error detection:corrupt.txt: Input text with spelling errors (can also contain whitespace errors)correct.txt: Groundtruth label for each sequence in the input, indicating whether the sequence contains a spelling error (1) or not (0)"this is a tset." > "1"For each format one line corresponds to one benchmark sample.Note that for some benchmarks we also provide versions other thantest, e.g.devortuning, which can be used to assess performance
of your method during developement. Final evaluations should always be done on thetestsplit.To evaluate predictions on a benchmark usingtcb.evaluate,
the following procedure is recommended:Run your model onbenchmarks/<split>/<task>/<benchmark>/corrupt.txtSave your predictions in the expected format for the benchmark under
inbenchmarks/<split>/<task>/<benchmark>/predictions/<model_name>.txtEvaluate your predictions on a benchmark usingtcb.evaluate:# evaluate your predictions on the benchmarktcb.evaluate-bbenchmarks/<split>/<task>/<benchmark># optionally sort by some metric and highlight the best predictionstcb.evaluate-bbenchmarks/<split>/<task>/<benchmark>--sort"<metric>"--highlightYou can also evaluate across multiple benchmarks like so:# when evaluating across multiple benchmarks you always need to specify a metric,# otherwise you will get an error# listing multiple benchmarkstcb.evaluate-bbenchmarks/<split>/<task>/<benchmark1>\benchmarks/<split>/<task>/<benchmark2>...-m"<metric>"# using glob patterntcb.evaluate-bbenchmarks/<split>/<task>/<gl*ob_patte*rn>-m"<metric>"# you can highlight the best predictions per benchmarktcb.evaluate-bbenchmarks/<split>/<task>/<gl*ob_patte*rn>-m"<metric>"--highlightDepending on the task the following metrics are calculated:Whitespace correctionF1 (micro-averaged)F1 (sequence-averaged)Sequence accuracySpelling error correctionF1 (micro-averaged)F1 (sequence-averaged)Sequence accuracyWord-level spelling error detectionWord accuracyBinary F1 (micro-averaged)Sequence-level spelling error detectionBinary F1Sequence accuracyBaselinesWe also provide baselines for each task:Whitespace correction:Dummy (wsc_dummy)Spelling error correction:Dummy (sec_dummy)Close to dictionary (sec_ctd)Norvig(sec_norvig)Aspell(sec_aspell)Hunspell(sec_hunspell)Jamspell(sec_jamspell)Neuspell (BERT)(sec_neuspell_bert)Word-level spelling error detection:Dummy (sedw_dummy)Out of dictionary (sedw_ood)From spelling error correction1(sedw_from_sec)Sequence-level spelling error detection:Dummy (seds_dummy)Out of dictionary (seds_ood)From spelling error correction1(seds_from_sec)The dummy baselines produce the predictions one gets by leaving the inputs unchanged.1We can reuse spelling error correction baselines to detect spelling errors both on
a word and sequence level. For the word level we simply predict that all words changed
by a spelling corrector contain a spelling error. For the sequence level we
predict that a sequence contains a spelling error if it is changed by a spelling corrector.
All spelling error correction baselines or prediction files
can be used as underlying spelling correctors for this purpose.You can run a baseline usingtcb.baseline:# run baseline on stdin and output to stdouttcb.baseline<baseline_name># run baseline on file and output to stdouttcb.baseline<baseline_name>-f<input_file># run baseline on file and write predictions to filetcb.baseline<baseline_name>-f<input_file>-o<output_file># some baselines require you to pass additional arguments,# you will get error messages if you dont# e.g. all dictionary based baselines like the out of dictionary baseline# for word-level spelling error detection need the path to a dictionary# as additional argumenttcb.baselinesedw_ood-f<input_file>--dictionary<dictionary_file>Dictionaries can be foundhere.Predictions of the baselines and other models from the literature
can be found in a subdirectorypredictionsin each benchmark,
see e.g.here.This repository is backed by thetext-correction-utilspackage.