Package: texterrors
texterrorsFor calculating WER, CER, other metrics, getting detailed statistics and comparing outputs.Meant to replace older tools likescliteby being easy to use, modify and extend.Features:Character aware, standard (default) and ctm based alignmentMetrics by group (for example speaker)Comparing two hypothesis files to referenceOracle WERSorting most common errors by frequency or countMeasuring performance on keywordsMeasuring OOV-CER (seehttps://arxiv.org/abs/2107.08091)Colored output to inspect errorsExample of colored output below (use-cflag). Read the white and green words to read the reference. Read the white and red words to read the hypothesis.See here forbackground motivation.InstallingRequires minimum python 3.6!pip install texterrorsThe package will be installed astexterrorsand there will be atexterrorsscript in your path.ExampleThe-soption means there will be no detailed output. Belowrefandhypare files with the first field equalling the utterance ID (therefore theisarkflag).$ texterrors -isark -s ref hyp  
WER: 83.33 (ins 1, del 1, sub 3 / 6)You can specify an output file to save the results, probably what you want if you are getting detailed output (not using-s).
Here we are also calculating the CER, the OOV-CER to measure the performance on the OOV words inside theoov_listfile, and using
colored output (therefore the-cflag).$ texterrors -c -isark -cer -oov-list-f oov_list ref hyp detailed_wer_outputUseless -Rto view the colored output. Skip the-cflag to not use color.Checktexterrors/__init__.pyto see functions that you may be interested in using from python.Options you might want to useCalltexterrors -hto see all options.-cer,-isctm- Calculate CER, Use ctms for alignment-utt-group-map- Should be a file which maps uttids to group, WER will be output per group (could useto get per speaker WER for example).-second-hyp-f- Use to compare the outputs of two different models to the reference.-freq-sort- Sort errors by frequency rather than count-oov-list-f- The CER between words aligned to the OOV words will be calculated (the OOV-CER).-keywords-list-f- Will calculate precision & recall of words in the file.-oracle-wer- Hypothesis file should have multiple entries for each utterance, oracle WER will be calculated.Why is the WER slightly higher than in kaldi if I use-use_chardiff?You can make it equal by not using the-use_chardiffargument.This difference is because this tool can do character aware alignment. Across a normal sized test set this should result in a small difference.In the below example a normal WER calculation would do a one-to-one mapping and arrive at a WER of 66.67%.testsentenceokaywordsendingnowtestasentencokendinnowBut character aware alignment would result in the following alignment:test-sentenceokaywordsendingnowtestasentencok-endinnowThis results in a WER of 83.3% because of the extra insertion and deletion. And I think one could argue this is the actually correct WER.ChangelogRecent changes:22.06.22 refactored internals to make them simpler, character aware alignment is off by default, added more explanations20.05.22 fixed bug missing regex dependency16.05.22 fixed bug causing wrong detailed output when there is utterance with empty reference, and utts with empty reference are not ignored21.04.22 insertion errors on lower line and switching colors so green is reference27.01.22 oracle WER and small bug fixes26.01.22 fixed bug causing OOV-CER feature to not work22.11.21 new feature to compare two outputs to reference; lots of small changes04.10.21 fixed bug, nocolor option, refactoring, keywords feature works properly, updated README22.08.21 added oracle wer feature, cost matrix creation returns cost now16.07.21 improves alignment based on ctms (much stricter now).TODO: use nanobind