# Fit for What Purpose? NER Certification of Automatic Captions in English and Spanish
## 2 Literature Review  
Quality, and more specifically accuracy, is one of the most debated topics in live captioning and one in which the requirements of the industry and the users seem to have met the interest of many researchers in the field. Live captioning accuracy has been studied by academics [10; 11; 12; 13], user associations [14], broadcasters [15] and regulators [16]. Different models have been put forward to assess the accuracy of live captions, whether based on captioning theory [17], the everyday practice of live captioning [18] or the automatization of quality assessment [14].  
In Canada, the Canadian Radio-television and Telecommunications Commission (CRTC) set up in 2012 a two-year project requiring broadcasters to analyse the accuracy of their human live captions (produced through respeaking and stenotyping) in 265 programmes using the so-called Verbatim Test [15], which regards accuracy as the extent to which the captions match word for word the audio of a programme. Only 19% of the programmes monitored met the 95% threshold established by the Verbatim Test, which was criticised by the broadcasters for not being a reliable determinant of accuracy and for punishing captioners who achieve better quality through adept editing than through verbatim captions that may be too fast for the viewers to read. With a view to accounting for the possibility of correct and incorrect editing and for the presence of different types of errors in live captioning, the NER model [19] was created in 2012. Since then, it has been adopted by universities, broadcasters, access service providers and regulators in countries such as the UK, Spain, France, Switzerland, Italy, South Africa and Australia. In the UK, Ofcom adopted the NER model to set up the largest study conducted so far on the quality of live captioning, analysing the accuracy, delay, speed and edition rate of 78,000 human captions from all terrestrial TV channels in the country between 2013 and 2015 [20]. The results show an overall accuracy rate of 98.4%, above the 98% threshold set by the NER model. More recently, [PERSON]. [7] used the NER model to compare the accuracy of 17,000 English human and automatic live captions produced in the UK, the US and Canada from 2018 to 2022, which included 388 min of human (respoken and steno-made) captions and 410 min of automatic captions. With a few exceptions, the accuracy of human captions was very good, with averages between 98.9% and 99.4% (7-8.5/10), higher than the 98.4% obtained in the Ofcom study. The accuracy of fully automatic captions improved significantly over the course of the study, from initial averages of 95.7% and 96.3% in 2018 and 2019, to some broadcasters consistently reaching the 98% threshold at the end of 2021.  
In Poland, [PERSON]. [21] analysed the accuracy of 13,620 live captions in Polish from 96 TV programmes, mostly produced through respeaking. The average NER score for all the samples in the study was 97.55%, thus falling slightly below the acceptable threshold. In Spain, Ref. [22] used the NER model to compare the accuracy of 10,387 human (respoken) Spanish captions and 4353 automatic Spanish captions. The average accuracy of human captions was 98.15%, just above the required threshold, whereas the average accuracy of automatic captions fell slightly below at 97.27%. Still, the accuracy of automatic captions seems to keep improving, as some samples were shown to exceed the 98% NER threshold with average accuracy rates of 98.22% and 98.39%.