# Fit for What Purpose? NER Certification of Automatic Captions in English and Spanish
## 6 Discussion  
### This Study  
The average AR obtained by [PERSON]'s captions is 98.56% in English (6.5/10, indicating good quality) and 98.26% in Spanish (5.5/10, average quality). This is similar to the 98.3% (6/10) obtained by the 78,000 captions produced by UK respekers between 2013 and 2015 and analysed in a study run by the UK's governmental regulator Ofcom [20]. More relevantly, when compared to the most recent study on live captioning quality in the US and Canada [7], the average AR obtained by [PERSON]'s captions is higher than the 98% yielded by the best automatic captions analysed and lower than the 98.9-99.4% obtained by human captions. The results obtained in this analysis show that the primary challenges for [PERSON]'s automatic captions differ from those faced by human-made captions, such as specialised vocabulary or a fast speech rate by a single speaker; instead, the main issues are speaker ID and punctuation. [PERSON] finds it difficult to tell the difference between speakers, especially in those videos featuring several speakers and quick or overlapping exchanges. When this happens, almost half of the errors are caused by incorrect speaker IDs. We have classified most of these errors as minor, but many of them could have been scored as standard as well, which would have resulted in even lower ARs. Until this is fixed, automatic captions, as analysed here, can only be considered good when produced for content with no or few speaker changes. Once speaker ID is not (or is less of) a problem, [PERSON]'s accuracy can be closer to the results showed at the beginning of this report, which did not take into account speaker ID errors. The average AR obtained by [PERSON]'s English captions without considering speaker ID errors is 99.03% (7.5 out of 10, very good quality), with a score of 99.23%, 99.13% and 98.82% for Levels 1, 2 and 3. This is higher than the 98.3% (6/10) obtained by UK respekers in the Ofcom study [20], higher than the 98% recently yielded by the best automatic captions in the US and Canada [7]. and at a par with the 98.9-99.4% obtained by human captions in the same study. The second main issue is punctuation, which causes 65% of all the errors in [PERSON]'s captions. Most punctuation errors are minor and are caused by missing question marks, missing commas in repetitions, hesitations and asides, incorrect use of commas and question marks with vocatives and missing colons and missing punctuation when reporting someone's speech. Misrecognitions, which were once the most frequent type of error in live captions produced through speech recognition, are now a minority, although they are the ones that cause standard and serious errors. Finally, it is important to note that whether or not speaker ID errors are considered, [PERSON]'s English and Spanish captions have not managed to reach the NER threshold of 98% with content featuring colloquial and fast exchanges involving several people, as is the case in podcasts and similar online content. This is the case in videos 3 EXTRA_EN, 3B_ES, 3C_ES and 3 EXTRA_ES, none of which reached 98%, as shown in Figure 2. This shows that, even if speaker ID issues are solved, [PERSON]'s captions still need to improve in order to guarantee the quality required by the users for this challenging (but frequent) content. As for the first question posed at the beginning of this paper (Do [PERSON]'s captions pass the LiRICS certification? ), [PERSON]'s English captions have passed all levels of the LiRICS test with a score (98.56%) that shows that these are good quality captions. More specifically, they are very good quality captions for Levels 1 and 2 and average quality captions for Level 3, where they do not reach the required threshold in the video featuring colloquial and fast exchanges. [PERSON]'s Spanish captions have passed Levels 1 and 2 of the LiRICS test with an AR of 99.23% and 99.06%, respectively, which shows that these are captions of very good quality.