SlowGuess/NAACL_2022_OCR huggingface dataset:
Fig. 8 illustrates via a flowchart our process of assigning the tuples in ground-truth and predicted outputs into different error categories. Tab. 7 presents examples for the four categories discussed in Sec 6.4. These are examples from the dev set, using the Distill-Pipeline E2E model trained on 100 hours of data.
<sup>7</sup>https://asappresearch.github.io/slue-toolkit/how-to-submit.html
Error category	Outputs from E2E model
GT	Predicted
Correct ASR, over detection	and this means that you look and tell us honestly what does it mean if you start @ three years later
[('WHEN', 'three years')]	this means that you look and tell us honestly what does it mean if you start @ three years later ]
[('WHEN', 'three years later')]
Correct ASR, missed detection	the situation in the % drc ] is indeed terrible and it has been this way for quite a while and i am deeply concerned about the handling of the current issue with regard to the % kasai ] province
[('PLACE', 'drc'), ('PLACE', 'kasai')]	the situation in the drc is indeed terrible and it has been this way for quite a while and i am deeply concerned about the handling of the current issue with regard to the a province
[]
Correct ASR, false detection	and yet @ one month ] after we adopted our compromise the council did not put it on the agenda did not even present it i used this time to talk to the member states and the presidencies
[('WHEN', 'one month')]	still @ one month ] after we voted a compromise the ‘council’ did not put it on the agenda did not even present i use this time to talk with the member states and the presidency
[('WHEN', 'one month'), ('ORG', 'council')]
Incorrect ASR, false detection	it has nothing to do with religion but it has all to do with patriarchy
[]	it has nothing to do with religion but it has all to do with % turkey ]
[('PLACE', 'turkey')]
Table 7: Qualitative examples for different error categories from the output of the E2E model using 100 hours of unlabeled speech (Distill-Pipeline).
Figure 8: Illustration of algorithm for obtaining error category types for each (entity phrase, entity tag) tuple in ground-truth and predicted outputs.
1: Remove corresponding tuples from the GT and predicted lists
2: Evaluated based on word overlap between the two phrases, after filtering out function words
3: Checked using a threshold on the character edit distance between GT and predicted phrase. For a particular pair to classify as misspelling, edit distance should be less than 0.4.
Condition A: GT entity phrase correctly decoded?
Condition B: Predicted entity phrase correctly decoded?
Condition C: Does the predicted tag match?
s2t : speech-to-text conversion