KEBAP: Korean Error Explainable Benchmark Dataset for ASR and Post-processing
Authors: Seonmin Koo
Chanjun Park
Jinsung Kim
Jaehyung Seo
Sugyeong Eo
Hyeonseok Moon
Heuiseok Lim
Venue: EMNLP 2023 Main (2023)
Keywords: Automatic Speech Recognition (ASR)
Error Explainable Benchmark
Post-procssing
Recognition Accuracy
User Readability

Abstract: Automatic Speech Recognition (ASR) systems are instrumental across various applications, with their performance being critically tied to user satisfaction. Conventional evaluation metrics for ASR systems produce a singular aggregate score, which is insufficient for understanding specific system vulnerabilities. Therefore, we aim to address the limitations of the previous ASR evaluation methods by introducing the Korean Error Explainable Benchmark Dataset for ASR and Post-processing (KEBAP). KEBAP enables comprehensive analysis of ASR systems at both speech- and text levels, thereby facilitating a more balanced assessment encompassing speech recognition accuracy and user readability. KEBAP provides 37 newly defined speech-level resources incorporating diverse noise environments and speaker characteristics categories, also presenting 13 distinct text-level error types. This paper demonstrates detailed statistical analyses of colloquial noise categories and textual error types. Furthermore, we conduct extensive validation and analysis on commercially deployed ASR systems, providing valuable insights into their performance. As a more fine-grained and real-world-centric evaluation method, KEBAP contributes to identifying and mitigating potential weaknesses in ASR systems.

КЕBAP: Korean Error Explainable Benchmark Dataset for ASR and Post-processing
Seonmin Koo'r, Chanjun Park Jinsung Kim', Jachyung Seo
Sugyeong Eo', Hyeonseok Moon', Heuiseok Lim
Korea University, Department of Computer Science and Engineering
Upstagge Al
(fhdahd, jin62304, seojae777, djtnrud, glee889, 1imhseok]@korea..eo c parkēupstage. ai
Abstract
Conventional (WER CER)
Automatic Speech Recognition (ASR) systems are instrumental across various applications, with their performance being critically tied to user satisfaction. Conventional evaluation met- rics for ASR systems produce singular aggre- gate score, which insufficient for understand- ing specific system vulnerabilities. Therefore, we aim to address the limitations of the previ- ASR evaluation methods by introducing the Korean Error Explainable Benchmark Dataset for ASR and Post-procet Post-prucsssing (KEBAP). KE- BAP enables comprehensive analysis of ASR systems at both speech- and text levels, thereby facilitating more balanced assessment encom- passing speech recognition accuracy and user readability. KEBAP provides 37 newly defined speech-level resources incorporating diverse noise environments and speaker characteris- tics categories, also presenting 13 distinct text- level error types. This paper demonstrates de- tailed statistical analyses of colloquial noise categories and textual error types. Furthermore, we conduct extensive validation and analysis on commercially deployed ASR systems. pro- viding valuable insights into their performance. As fine-grained and real-world-centric evaluation method, KEBAP contributes to iden- tifying and mitigating potential weaknesses in ASR systems.
model
+ 0.45
KEBAP
model
Error types
Explainability
Figure I: An example of using KEBAP to evalu- ate weaknesses in ASR results. The above approach with conventional evaluation methods lacks explana- tory power. Evaluation using KEBAP below enables detailed description the weaknesses.
the ASR model accurately recognizes the input voice, the user's 's readability may decrease. This is because humans do not always utter perfect sen- tences in the real world (e.g., incomplete utterances, sighs, etc.). To achieve balanced ASR results in this trade-off situation, it is required consider both recognition accuracy and user readability.
In terms of recognition accuracy, various ASR evaluation metrics such as word error rate (WER) (Woodard and Nelson) and character er- rate (CER) (Morris al., 2004) have been prevalent. Also, readability is considered in ASR post- processogsssing (ASRP) tasks, where the goal improve the clarity and comprehension of speech recognition outputs without modifying the under- lying model architecture (Mani et al., 2020b; Liao et al., 2020; Leng et al., 2021). The ASRP task relies on quantitative metrics, such as BLEU (Pa- pineni et al., 2002) and GLEU (Napoles al., 2015), similar to the ASR evaluation using WER and CER.
1 Introduction
Automatic speech recognition (ASR) is task that recognizes speech and converts into text, and it is getting more and more attention with the de- velopment of voice interface applications and de- vices such as Alexa, Siri, and Cortana (Williams and Young, 2007; Wang et al., 2018, 2020). In the real world, the ASR result has trade-off between recognition accuracy and user readability. Even
Equally contributed, Corresponding author RRecognition accuracy is the measure of accurately per- cciving phonemes as they are externaly expressed, regardless of user input quality (Lian et ul., 2022).
However, is crucial to recognize that even if

---

quantitative evaluation scores are similar, the quali- tative aspects of the ASR results may not necessar- ily align. Conventional research methods, which fo- cus on accuracy or user readability, compute quan- titative scores based on the degree of alignment between inputs and outputs. This approach falls short in classifying potential error types or pinpoint- ing the model's specific weaknesses, thus lacking explanatory power for real-world ASR model out- puts. This deficiency hinders the establishment of clear directions for model improvement. To this end, datasets that aim to enhance the explanatory power of ASR evaluations by considering noisy environments or speaker characteristics have been published recently (Sikasote and Anastasopoulos, 2022; Lakomkin et al., 2019; Gong et al., 2022; Dai et al., 2022). However, these datasets still focus on accuracy and provide a limited set of error types, thereby leading to insufficiency in diagnosing spe- cific weaknesses within ASR models.
world-oriented evaluations of ASR systems and can contribute the advancements in this domain.
2 KEBAP
2.1
Why KEBAP?
In the real-world scenario, mitigating the trade- off between recognition accuracy and user read- ability is erucial. To address this, we propose KE- BAP, emphasizing the importance of considering both aspects. A detailed explanation is as follows. Firstly, in real-world speech recognition, it is essen- tial to consider the accuracy of model and end-user satisfaction simultaneously. To facilitate this, we propose to map the accuracy of the ASR model to 'speech-level noises' and user readability "text-level errors' mitigate this inherent trade- off. From the perspective of the accuracy of the ASR model, it should output the recognition re- sults 'as heard,' regardless of the quality of the user-provided input. Conversely, from the stand- point of the end-user receiving the result, satisfac- tion increases when the output is presented in a refined state, despite any errors in the initial input. For instance, if speaker stammers during their speech, the ASR model would likely deem its out- put more accurate if it recognizes and outputs all the words uttered. However, this would likely result in lower readability from the user's perspective. In addition, previous research lacks an adequate number of error types for detailed diagnosis. Since benchmarks measure performance with quan- titative metrics, it is crucial subdivide character- istics for a more detailed diagnosis. In industry contexts, communication between model and ser- vice teams is critical. When there's an issue with the model, clear criteria for the data flywheel sig- nificantly facilitate communication. That is, distin- guishing the error type criteria for speech- and text- level aids in detailed diagnosis for model improve- ment. However, conventional benchmark datasets lack sufficient error types for detailed model anal- ysis, leading to extensive usage of human evalua- tion in real-world settings. Humans can cope using commonsense, even if the criteria are unclear, but existing benchmarks with limited eттor types fall short. Hence, to solve the explainability issue, we must define error type criteria that consider both the speech- and text-level and create benchmarks to achieve human-level explainability.
Therefore, we introduce the novel Korean Error Explainable Benchmark Dataset for ASR and Post- processing (KEBAP)². It encompasses speech- level distraction- resources and text-level er- ror types relevant to real-world ASR applications. These diverse error types of KEBAP can lead to improved explanatory capability compared to the previous examination methods, as illustrated in Figure 1. In particular, speech-level noise types are bifurcated into categories: noisy environ- ments and speaker characteristics, comprising 37 distinct types. Additionally, KEBAP includes 13 types of textual errors pertinent in ASR contexts. The dataset stands out in its authenticity since all speech samples are recorded by human speakers, and background noises are derived from real-world environments. Also, We annotate the difficulty lev- cls to all types, enhancing the interpretability of the ASR model.
We employ KEBAP to conduct an empirical analysis of the correlations between speech-level noise types and textual error types. Moreover, lever- aging ChatGPT (OpenAI-Blog, 2022), we explore the potential of language models in discovering the vulnerabilities of ASR models. Our observations highlight KEBAP's significant interpretability of ASR model diagnostics and shed light on the press- ing need for research on diagnostic tasks for ASR systems. Our work sets the stage for more real-
To enhance the explanatory power of the vali- dation process for ASR models, we define errors
"Our KEBAP dataset is publicly available at ht github, www.uu..com//eeee wtt.////////a/eeee coom/seonmindi Aok.o//AAAn

---

Noise Type
Description
Washenkiryer muchine
Home appliances
Difficulty recognition ambient electrical appliance
Vacusmn cleaner
Mokorcylk
Individual transportation
Siren
Difficulty recognition due suroondinng individual transportation noise.
Honk
Road side
Street
Difliculty in recognition due the suroonding street notoc.
Crowd
Conversation
Caferrostauran
Challenges pereeption due to the noise in cafeo/restaurants.
Non-oxwersation
Traditional market
Markeeshupping mail
Difficulties perception caused by the neise in murpptssshupping malls.
Shopping mall
Subway platform
Noiay environment
Inside the subway
trnsportazion
Inside the train (STR/KTX) Difficulty recognition sumounndnng public transportaion
Inside the bus
Train terminal wailing room
Temiinal
Challenges perception due to the noise at terminals.
Bas temminal waiting
Construction site
Outdoce consiruction site
Difficulties perception caused by the neise at constraction sites.
Indoor constraction
Factory
prucessing process
Difficulties perception caused el by the noise in factocies.
Assembly
Sound of zain
Nature ambient
Challenges perception due to natural ambient noise.
Sound the
In cases where external noise is present, although not falling imto the aforementinee categories.
Ric.
Artifiiaal mechanical sound
When there is presence of pauses between syllables in spooch that has
Pause (vilent)
Hilled pause
When habitual sounds are inserted during moments of silence bweak time.
longer segments inserted regardless their relevance to the intended content being es pressed.
When
Interjaction
When grammatically acceptable
inserted without conveyiing
Parenhetical
spceifie meaning or significane.
Unifnished interlocutor
sperch terminatedd
concluding the
Word copetition
Ropeating the same word or phrase in sucecasion during speech.
Syllable repetision
Ropeating same syllable sucuession during speech.
Phoneme repetition
Ropcating the same phoneme in succession during speech.
Charatteristics of interlocutor
Sustained
When elongating certain parts
within sentence during speech. When excessively verbose speech is employed.
Hyperflueccy
muttering an unclear demeanor.
When syliahic intonation inupproprate for the intended speech purpose
Dynamic
difficult human-love comprehemvion.
When speech ralc is excessively tast, making it cifficuli for human- level comprehensšon.
Speaking
Table I: Proposed novel speech-leve noise type classification criteria for KEBAP
at speech- and text-level and propose KEBAP, a benchmark that considers various potential error- prone environments in real-world scenarios
and 13 for speaker characteristics.
Considering environments inundated with noise, does not represent a quiet recording situation but rather a condition intertwined with noise. Real- world scenarios frequently involve inputs replete with ambient noise (Sikasote and Anastasopou- los, 2022). Reflecting on these practical situations where voice interface applications and devices are deployed,
2.2 Speech-Level Noise Type
Error types the speech-level refer factors that trigger inaccuracies in speech recognition sit- uations. For example, identical utterances may be challenging to recognize due to background noise (Sikasote and Anastasopoulos, 2022). Ad- ditionally, even in quiet environments, individu- als do not consistently articulate perfect sentences and each speaker has unique characteristics that may negatively influence speech recognition (Gong al., 2022).
propose an enhanced categorization scheme that elosely follows the classification in the AI-HUB's noisy environment speech recognition dataset 3 which are representative Korean data plat- form. We divide the noisy environment errors into 11 nuanced subcategories, including home appli- ances, where recognition is impaired due to sur- rounding appliance noise; individual transporta- tion, which includes instances with ambient trans- portation noise; street, covering situations with disruptive street noise; cafe/restaurant, address- ing cases with the cafe or restaurant ambient noise;
Table 1 illustrates the speech-level error type classification criteria considering these characteris- tics. The speech-level error types allow the classifi- cation of two main categories (noisy environment and characteristics of interlocutor) and more de- tailed error types, with 24 sub-types for noise error
Phttps: /www. ai aub... or...rr

---

Category
Description
Level A
Level
Level
Spacing Punctuation
Violating the spacing rules.
Punctuation marks allached in Koreun
atluched in the wrong.
Cardinal number indicating quantity and ordinal number indicating the order error
Namerical
Remove
Some words
recognized, endings suffixes ommited.
Addition
Same word repeated, or unused postposition ending is added.
Replace
Word is replaced by another word.
Spelling
Separation
Sepuraiing comsonants and vowels in characters.
Instances of incorrect conversion of syllables between English and Korean, as well as
and
Foreign word conversion
writing spellings according to promunciation, have been observed.
Grammatical
G2P
Writing spellings according to promunciation.
Spelling
CVC
Spelling error in non-speaking alphabet units.
Post-position
Instances of inconsistent missing post positiosioon usage targel utterances.
Synlax
grammatically accurate yet interprelatively amhiguous meanings.
Iostances of the discrepancy between larget and its similarity meuning, pronunciation, and absence Korean lexicon.
Neologism
Table 2: Proposed text-level error type classification criteria for KEBAP. G2P and CVC indicate Grapheme-to- phoneme and Consonant vowel conversion, respectivity
market/shopping mall, indicating instances with market or shopping mall noise; public transporta- tion, comprising cases with subway or bus noise; terminal, reflecting instances with terminal noise; construction site, for cases hindered by construc- tion site noise; factory, indicating instances with factory noise; nature ambient, for cases disturbed by natural sounds. Lastly, we include an etc. cat- egory for instances where recognition affected by external noise types not encompassed in the previous categories.
configure the Korean GEC dataset, K-NCT, to suit speech recognition situations. The existing K-NCT dataset includes errors that only occur at the text- level and not in speech situations (Koo et al., 2022). Hence, errors that do not have vocal characteristics removed.
Table 2 illustrates the text-level error type clas- sification criteria considering speech recognition situations, including 13 text-level errors that can occur in speech recognition situations. Detailed explanations for each text-level error type can be found in Appendix C
Considering speaker characteristics, recognition can be hampered due to the individual traits of the recorder. Inspired by studies idiolectal elements in the field of psycholinguistics (Ha and Sim, 2008; Shin et al., 2005), we propose nuanced categoriza- tion comprising detailed subcategories. The de- tails description of the subcategories are described in Appendix B.
2.4 KEBAP Construction Process
In this work, we propose a comprehensive data con- struction guideline for the ASR and ASRP dataset. grounded in the application of a GEC dataset. Our methodology encompasses build text-level error corpus, speech recording, noise synthesis, and diffi- culty annotation. For the efficiency of the task, we choose the 'consensus labeling' method (Tang and Lease, 2011) 1), in which human overseer, who pos- sesses an elevated degree of task completion, serves as a quality controller. During the progression of the task, any outcomes that do not conform to the established guidelines are promptly dismissed and subsequently reconstructed.
2.3 Text-Level Error Type
Text-level error types refer to issues that emerge in speech recognition results and must be addressed by post-procsssing Since the output of the speech recognizer serves the input for downstream tasks, it is one of the most significant factors influenc- ing end-user satisfaction. By improving the perfor- mance of downstream tasks through quality input and diagnosing the performance of post- post-processing models through detailed error types, is possible to enhance end-user satisfaction.
Step 1: Build Text-Level Error Corpus In this study, we employ human-curated GEC dataset, which encompasses various text-level error types (Koo et al., 2022). Considering the inapplica- bility of the standard GEC benchmark dataset in a speech recognition setting, we selectively compose a text-level error types dataset by human evaluation.
Existing datasets that detail error types, such as grammatical error correction (GEC) datasets, do not consider speech recognition situations (Koo et al., 2022; Yoon et al., 2022). Therefore, we re-

---

We assess whether the given error types are valid or invalid in the context of speech recognition situa- tions by human evaluators. Invalid types are filtered out, and the type structure is reconfigured In par- ticular, we extract 13 categories that resonate with speech recognition scenarios (c.g., honorific collo- quial expression) and reorganize their hierarchy for ease of labeling. Consequently, our refined dataset includes data reflecting 13 error types relevant to speech recognition contexts.
ror in the proposed speech-level. The background noise used for this integration derived directly from recordings of the identified environments. We ensure that the collected noise spans a duration longer than that of the recording file, fostering noise diversity. To mimic real-worid situations, we con- duct both single and multiple noise syntheses while filtering out instances that are unlikely to co-occur. During noise synthesis, the noise is integrated though it is ambient background noise, designed to be audible the onset of the voice file. Noise is composited into the recording by randomly excis- ing sections, thus ensuring variation within sounds, even when they are categorized under the same noise type.
Subsequently, we authenticate the quality of the filtered dataset focusing on the alignment between labels and text, and the inclusion of text-level errors with specific consideration of the speech recogni- tion context. Validation processes proceed with human supervisor, priorily trained with each error type. Evaluators are presented with an erroncous sentence, its correct counterpart, and a specified error type with the corresponding error span indi- cated. They are then tasked with assessing whether the sentence contains the presented error types. Sen- tences deemed to be incorrect are appropriately amended. This procedural framework ensures the generation of high-quality dataset.
Step 4: Difficulty Annotation Difficult data for ASR models refers to data that is not frequently encountered in the training data and is imbalanced, varying depending on the user (Aleksic et al.. 201 5a). Therefore, annotate the difficulty of the data to enable detailed assessment of the model's coverage ability. To this end, we employ a frame- work that distinguishes between utterances consid- cred casier for ASR and those deemed harder or more noisy for ASR (Breiner et al., 2022). We ex- tend this framework to include the tagging of diffi- culty using Likert scale by human annotators. Hu- mans listen to audio file and select score based evaluation criteria. We ask humans, 'How difficult is it to recognize the presented speech accurately as the same as the transcript?' Scores range from (very easy) to (very difficult). Three evaluators assess each audio file, and the average score is se- lected as the difficulty level of the data. This allows for a detailed unalysis of the model's performance. The details of construction process described in Appendix D.
Step 2: Speech Recording In the second phase, we request that recording participants incorpo- rate characteristics of interlocutor errors into their recordings by presenting them with speech-level errors and transcription relevant to the respective error types. At most 3 error types are presented, which could include an instance of 'no етrоr type', indicating clean data. The placement of the error within the sentence is non-specific, with the en- surance that it includes only the errors specified. The recording environment should be ensured to be quiet without background noise. Each recorder is instructed to speak as naturally as possible, emu- lating their speech patterns when interacting with a voice interface application in real-world scenarios. After completing the recording, participants have the opportunity to listen to their own voice, and if they determine that the speech does not meet the criteria, they can re-record it. Participants are re- quired go through the process of listening their recorded speech in order to complete the recording task. The detailed information about the workers can be found in Appendix D.
3 KEBAP Analysis
3.1 Text-level Distribution
We filter out cases that cannot occur in speech recognition situations, such as typing language er- rors caused by keyboard language switching, in the GEC dataset. After the filtering process, the text- level distribution shown in Table 3. It includes 2,478 instances of errors and correct sentences, in-
Step 3: Synthesis of Background Noise In the next stage, we incorporate background noise into the recording to reflect the noise environment er-
cluding text-level errors. The statistical information for the text is provided in Table 4

---

Category
S1D
24Clesto
Count (%)
Level A
Level B
Level
Specing Punctuation
514 (14.62)
505 (14.37)
Numerrcall
500 (14,22)
Remove
122 (3,47)
Addition
104 (2.96)
Replace
483 (13.74)
Spelling
Separation
94 (2.67)
and
Foreign word conversion
193 (5,49)
Grammatical
G2P
195 (5.55)
601
Spelling
CVC
534 (15.19)
Post-position
90 (2.56)
Figure 2: Data distribution for speech-level. (a) indicates distribution for the speech file length. (b) indicates the difficulty distribution of human evaluation.
Syntax
71 (2.02)
Nenlagism
110 (3.13)
Total
3515 (100)
Table 3: Statistics of labels in text level category of KE- BAP. Here, the lowest level of data granularity is the category attribute in Level C. G2P and CVC indi- cated as seopheme-to-phoneme and consonant vowel conversion, respectively.
shown in Figure 2-(b). To more accurately diagnose the model's s capability, we enhance its interpretabil- ity by tagging the diffcultty level the data. Since the perceived difficulty the same data may vary among individuals, we determine the difficulty of each data based on the average difficulty annotation provided by three evaluators. The difficulty ratings for the data are generally concentrated between and 2, but there also a significant presence of ratings This indicates that the dataset includes a range of difficulty levels, which we believe will be beneficial for assessing the performance ASR models.
KEBAP
Test
Error sentence
Correct sentence
of sents
2,478
2,478
of tokens
107,411
107,209
# of words
25,772
26,250
avg of SL
43.35
43.26
avg of WS avg SS
10.40
10.59
9.40
9.59
Table 4: Statistics our KEBAP dataset. of sents/tokenswwords: number of sentences/rokens/word:: A avg of SL/WS/SS: average of sentence length/words/spaces per sentence.
3.4 Category Distribution
Each data includes single multiple speech-level characteristics. Figure 3 shows the distribution of each category of speech-level. The speech-level errors can be broadly classified into two main categories: noisy environment and characteristics of the interlocutor. Our dataset encompasses var- ious combinations of characteristics within each category and also includes cross coos-ccategory combi- nations, providing a diverse range of eтror types. Noisy enviromment and characteristies of the inter- locutor represent mutually exclusive types, while co-occurrence indicates cases where character- istics occur simultaneously. When considering only noisy environments, and 2 characteristics account for 40.02% each, and characteristics account for 19.96%.
3.2 Speech-level Distribution
KEBAP consists of a total of 2,478 speech files, transcriptions, and speech-level noise types. Fig- ure 2-(a) illustrates the data distribution for speech- level. KEBAP is composed of total of 24,021.82 seconds of speech. It includes an average speech duration of 9.69 seconds, with the shortest file be- ing 3.8 seconds and the longest file being 27.68 seconds. Although transcription sentences are com- posed of single sentences, their lengths can vary de- pending on speaker characteristics, such as pauses (silent) or mutters, even for sentences of the same length. This allows for the inclusion of speech files of varying durations, covering the characteristics of diverse users who ASR systems.
of inrrrtstoo
3.3 Difficulty Distribution
13%
Overall, the average Krippendorff's a for inter- annotator agreement of each annotation level is 0.476. The label distribution of the collected data is
Figure 3: Distribution of each category of speech-level.

---

Noisy environments that cannot occur simultane- ously are not included. When considering only the characteristics of the interlocutor, 1 and 2 charac- teristics account for 39.98% each, and 3 character- istics account for 20.04%. Co-occurrence occurs in 49.88 99.8 of cases for 2 characteristics and 50.12% of cases for 3 characteristics. This demonstrates the presence of diverse range of error levels, both in terms of types and quantities. Actual workers recorded the data, and the background noise was collected directly from real-world environments, ensuring high quality. There is no synthetic audio involved the recordings.
weaknesses of the model, hindering the establish- ment of directions for model improvement. To en- hance interpretability, we analyze the tendency of text-level error propagation at the speech level for ASR model. To clearly understand the impact of each speech-level category on the text-level, we sample data that includes single speech-level noise type. The results of the ASR model are la- beled by humans trained in explanations and exam- ples of text-level errors.
WER
CER
Easy
Medium
Hard
Easy
Medium
Hard
Google ASR
0.47
0.63
0.93
0.21
0.34
0.69
sere aphesoo-
Clova ASR
0,53
0.67
0.94
0,2
0.35
0.73
Whisper
0.48
0.67
0,92
0.23
0.35
0.65
Table 5: Evaluation results of ASR commercialization systems and publicly available model (Radford et al., 2023). Word Error Rate (WER) and Character Error Rate (CER) indicate better performance as their values decrease.
E25
Figure 4: Correlation distribution between speech-level: noisy environment and text-level in Google(a) and Clova(b). Spa/Punc/Num represent spacing, punctua- tion, and numerical, respectively. Rem/Add//eepSeep indicate remove, addition, replace, and separation, respectively. FWC/G2P/CVC correspond to foreign word conversion, graphermetto-hhoneeee, and conso- nant vowel conversion, respectively, PP/SYn/Neo signify post-position. syntax, and neologism. respectively.
4 Efficacy Validation for KEBAP
In this section, we assess the specific capabilities of commercialized ASR models using KEBAP. To achieve this, we conduct detailed correlation anal- ysis of commercialized systems such as Google Cloud Speech-to-TTex (Aleksic et al., 2015b) and CLOVA Speech (Chung, 2019). We examine the correlation between speech-level noise types and text-level errors, aiming for a granular understand- ing. We comprehensively validate the model's capa- bilities by considering both speech- and text-level aspects. We verify whether the LLM possesses the necessary qualities diagnostic model through the error type classification task.
Ch
Sprcch lendl thannee sereoertiiio of ratooter
Pous
Poec
7382521134
4.1 Analysis of Correlation in ASR models
Figure 5: Correlation distribution between speech-leveli characteristics of interlocutor and text-level in Google(a) and Clova(b).
Table 5 shows the evaluation results of ASR mod- els. Based on conventional evaluation metrics such as WER (Woodard and Nelson) and CER (Morris et al., 2004), we observe similar performance be- tween the two ASR models. However, even though the quantitative evaluation results may be similar, it does not necessarily mean that the qualitative as- pects of the ASR model's outputs are also similar. This makes it challenging to identify the specific
Figure 4 illustrates the correlation between the noisy environment at the speech level and text- level errors. In the speech-level category, we have grouped similar types together, excluding miscella- neous types. In the case of text-level punctuation errors, we consider only those instances where peri-

---

ods (:)) are missing in all sentences, leading the omission of other punctuation marks such ques- tion marks (?²') or exclamation points (*!"). This specific condition allows us to focus on scenarios where the absence of periods directly affects the presence of other punctuation marks in the tran- scriptions.
WER
CER
Clean
Noise
Clean
Noise
Google ASR
0.49
0.51
0.23
0.25
Clova ASR
0.53
0.57
0.19
0.24
Table 6: Performance of commercialize systems hased on the presence or absence of noise synthesis. 'Clean' and "Noise" represent the settings before and after noise synthesis, respectively.
Both the Google and Naver ASR systems exhibit significant error propagation in the domain of pub- lie transportation. Specifically, for Google, there is a high correlation between speech-level errors and text-level errors punctuation, spacing, and replace. On the other hand, Clova shows strong correlation between speech-level errors and text- level errors in punctuation, spacing, and addition. Furthermore, google showed robustness in the ture ambient, but Clova showed relatively more text errors.
while the WER after noise synthesis is 0.71 and the CER is 0.43. These results are interpretable in that the resources provide are high-quality and helpful.
4.3 Examination of ASR models through LLM
With recent advancements in Large Language Model (LLM) development, most tasks are con- verging towards LLM based upproaches. this study, we explore the potential of using Chat- GPT (OpenAI-Blog, 2022) as diagnostic tool for ASR results. Understanding error types is essential for verifying the models, and to measure this under- standing, we perform error type classification task. ChatGPT utilized to classify text-level error types based on provided sentences in a few-shot setup. The specific prompt used for this experiment is listed in Appendix F.
Figure 5 shows the correlation between speech- level characteristics of interlocutor and text-level errors. For Google, the presence of pause (silent) in speech had significant impact on the occurrence of remove errors in the transcriptions, while word repetition contributed to the occurrence of addition errors. In the case of Clova, overall, a higher num- ber of errors were observed compared to Google. Particularly, hyperfluency had the most significant impact on the occurrence of addition errors in the transcriptions.
We task ChatGPT with classifying all text-level errors occurring in the ASR results. However, as seen in the examples (please refer to Appendix F.2), it is evident that ChatGPT not only misclassifies text-level errors but also struggles more when mul- tiple errors are present within sentence. Although LLMs are converging towards covering various tasks, they exhibit limitations in performing diag- nostic tasks for commercial systems. This indicates that while various tasks may converge with LLM, the diagnostic domain for the proposed model is far from convergence with LLMs, highlighting the need for further research.
This analysis provides valuable insights into the correlation between speech- noise, particularly noisy environments, and text-level errors in the Google and Clova ASR systems. The varying im- pact of different types of speech-level characteris- tics on text-level errors highlights the need for fur- ther granularity in categorizing these types. Even if models demonstrate similar performance, the in- dividual capabilities of each model can differ. This demonstrates that KEBAP helps enhance the inter- pretability of ASR model verification 4.
4.2 Adequacy of Synthesized Noise
Table shows the performance before and after noise synthesis. Experimental results show that for Google, the WER is 0.49, the CER is 0.23 before noise synthesis, and the WER is 0.68 and the CER is 0.41 after noise synthesis. For Clova, the WER before noise synthesis 0.53, and the CER 0.19,
Conclusion
In the real-worid, ASR results involve trade-off between recognition accuracy and user readability, thus requiring a balanced consideration of these fac- tors. To provide guidance for improving model per- formance, it is necessary to enhance interpretability, which entails considering both speech-level accu- racy and text-level user readability. To this end, we propose Korean Error Explainable Benchmark
'Additionally, KEBAP provides difficulty information, en- abling even more fine-grained analysis. Appendix E includes the detailed analysis results based on diverse levels of diffi- culty.

---

Dataset for ASR and Post-processing (KEBAP) for diagnosing and validating models by segment- ing error types while considering both speech- and text-level. To facilitate the construction process, we utilize a GEC dataset that includes text-level errors and structure the process into validation, recording, synthesis of background noise, and difficulty tag- ging stages, employing consensus labeling within each stage to enhance the efficiency and quality of the task. We performed a detailed diagnostic analysis of the commercialization systems using KEBAP. Furthermore, the proposed task falls into a domain that is challenging for ChatGPT to cover, and it indicates the need for further research to achieve closer approximation to real-world diag- nostics. We demonstrated that KEBAP contributes to enhancing the interpretability of the model's s weaknesses.
Evaluation). This work was supported by Institute of Information & communications Technology Planning & Evaluation(IITP) grant funded by the Korea gMV)) (No. 2020-0-00368, A Neural-Symbolic Model for Knowledge Acquisition and Inference Techniques). This work was supported by Institute for Information & communications Technology Planning & (Xaluation(IITPP) grant funded by the Korea govemmeen(MSIT) (No. 2022-0-00369, (Part 4) Development of AT Technology to support Expert Decision-making that can Explain the Reasons/Grounds for Judgment Results based on Expert Knowledge)
References
Petar Alcksic, Cyril Allauzen, David Elson, Aleksan- dar Kracun, Diego Melendo Casado, and Pedro J Moreno. 2015a. Improved recognition of contact names in voice commands. 2015 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5172-5175. IEEE.
Limitations
This study has the limitation of only building data for the Korean language. Additionally, as this paper proposes new task, it was not able to conduct ex- tensive quantitative analyses by comparing it with existing models, which remains limitation. How- ever, this paper made a contribution by proposing new data and tasks and making them publicly avail-
Petar Aleksic, Mohammadreza Ghodsi, Assaf Michaely, Cyril Allauzen, Keith Hall, Brian Roark, David Ry- bach, and Pedro Moreno. 2015b. Bringing contextual information to google speech recognition.
able.
Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler. Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis Tyers, and Gre- gor Weber. 2020. Common voice: A massively- multilingual speech corpus. In Proceedings of the Twelfth Language Resources and Evaluation Confer- ence, pages 4218-4222, Marseille, France. European Language Resources Association.
Ethics Statement
We discuss the main ethical considerations of KE- BAP benchmark we presented: (1) Privacy. KE- BAP benchmark constructed to acquire factual dataset, and does not contain privacy issues. (2) Hu- man evaluation. During data evaluation process, we paid human workers the legal wage determined by the average time of evaluation and local labor com- pensation standards. We also guided them to take a rest when they are in state of fatigue during work. (3) Potential problems. While principled measures are taken to ensure the quality of the dataset, there might still be potential problems with the dataset quality.
Youssef Bassil and Paul Semaan. 2012. Ast context- sensitive error correction based on microsoft n- n-gram dataset. arXiv preprint arXiv: 120..5262.
Theresa Breiner, Swaroop Ramaswamy, Ehsan Variani. Shefali Garg, Rajiv Mathews, Khe Chai Sim, Kilol Gupta, Mingqing Chen, and Lara MeConnaughey. 2022. Userlibri: A dataset for asr personalization using only text. arXiv preprint arXiv:2007.00706
Hui Bu, Jiayu Du Xingyu Na, Bengu Wu, and Hao Zheng. 2017. Aishell-1: An open-source mandarin speech corpus and speech recognition baseline. In 2017 20th conference of the oriental chapter of the international coordinating committee on speech databases and speech 1/O systems and assessment (O-COCOSDA), pages 1-5. IEEE.
Acknowledgements
This research was supported by the MSIT(Ministry of Science and ICT), Korea, under the ITRC(Information Technology Research Center) 20102200 supervised by the ITP((nstitute for Information & Communications Technology Planning &
support
Joon Son Chung. 2019. Naver at activitynet challenge 2019-task active speaker detection (ava). arXiv preprini arXiv: 1906. 10555.

---

Gonçalo M Correia and André FT Martins. 2019, simple and effective appreach to automatic post- editing with transfer learning. arXiv preprint
Egor Lakomkin, Sven Magg, Cornelius Weber, and Ste- fan Wermter. 2019. Kt-speech-crawler, Automatic dataset construction for speech recognition from youtube videos. arXiv preprint
Wenliang Dai, Samuel Cahyawijaya, Tiezheng Yu, El- ham J Barezi, Peng Xu, Cheuk Tung Yiu, Rita Frieske, Holy Lovenia, Genta Winata, Qifeng Chen, et al. 2022. Ci-avsr: A cantonese audio-visual speech datasetfor in-car command recognition. In Proceed- ings of the Thirteenth Language Resources and Eval- uation Conference, pages 0786-67933
Myunghoon Lee, Hyeonho Shin, Dabin Lee, and Sung- Pil Choi. 2021. Korean grammatical error correction based on transformer with copying mechanisms and grammatical noise implantation methods. Sensors, 21(8):2658.
Yichong Leng. Xu Tan, Linchen Zhu, Jin Xu, Renqian Luo, Linquan Liu, Tao Qin, Xiangyang Li, Edward Lin, and Tic- Yan Liu. 2021. Fastcorrect: Fast error correction with edit alignment for automatic speech recognition. Advances in Neural Information Pro- cessing Systems, 34:21 34:21708-21719)
John Evershed and Kent Fitch. 2014, Correcting noisy ocr: Context beats confusion. In Proceedings of the First International Conference on Digital Access Textual Cultural Heritage, pages 45-51.
Jinjuan Feng and Andrew Sears. 2004. Using confi- dence scores to improve hands-free speech based nav- igation in continuous dictation systems. ACM Trans- actions on Compuler-Haman Interaction (TOCHI), 11(4)3229335
Junwei Liao, Sefik Emre Eskimez, Liyang Lui, Yu Shi, Ming Gong, Linjun Shou, Hong Qu, and Michael Zeng. 2020. Improving readability for automatic speech recognition transcription. Transactions on Asian and Res-urs Language Information Pro- cessing.
Zorik Gekhman, Dina Zverinski, Jonathan Mallinson, and Genady Beryozkin. 2022a. RED-ACE: Robust error detection for ASR using confidence embed- dings. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2800-2808, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Junwei Liao, Yu Shi, and Yong Xu. 2022. Automatic speech recognition post-proesssing for readability: Task, dataset and a two-stage pre- rre-traieed approach. IEEE Access, 10:117053-11700
Zorik Gekhman, Dina Zverinski, Jonathan Mallinson, and Genady Beryozkin, 2022b. Red-ace: Robust error detection for ast using confidence embeddings. arXiv preprint arXiv:2203.07172.
Anirudh Mani, Shruti Palaskar, and Sandeep Konam. 2020a. Towards understanding asr error correction for medical conversations. In Proceedings of the first workshop on natural language processing for medical conversations, pages 7-11.
Yuan Gong, Jin Yu, and James Glass. 2022. Vocal- sound: A dataset for improving human vocal sounds recognition. In ICASSP 2022-2022 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 151-155. IEEE.
Anirudh Mani, Shruti Palaskar, Nimshi Venkat Meripo, Sandeep Konam, and Florian Meize. 2020b. Asr er- ror correction and domain adaptation using machine translation. In ICASSP 2020-2020 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6344-6348. IEEE.
Jinxi Guo, Tara N Sainath, and Ron J Weiss. 2019. A spelling correction model for end-to-end speech recognition. In ICASSP 2019-2019 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5651 1-655, IEEE
Andrew Morris, Viktoria Maier, and Phil Green. 2004. From and ril mer and wil: improved evaluation measures for connected speech recognition.
Ji- Wi- Ha and Hyun Sub Sim. 2008 A comparison study of interjectional characteristics between people who stutter and people who do not stutter. Communi- cation Sciences and Disorders, 13(3):338-453
Courtney Napoles, Keisuke Sakaguchi, Matt Post, and Joel Tetreault. 2015. Ground truth for grammatical ervor correction metrics. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Confer- ence on Natural Language Processing (Volume 2: Short Papers), pages 588-593
Oleksii Hrinchuk, Mariya Popova, and Boris Ginsburg. 2020. Correction of automatic speech recognition with transformer sepuence-to-squenee
model. In Icassp 2020-2020 ieee international conference on acoustics, speech and signal processing (icassp). pages 7074-7078. IEEE.
Thi- Hai Nguyen, Mickael Coustaty, Antoine Doucet, Adam Jatowt, and Nhu-Van Nguyen. 2018. Adaptive edit-distance and regression approach for post-ocr text correction. In Maturity and Innova- tion in Digital Libraries: 20th International Confer- ence Asia-Pacific Digital Libraries, ICADL 2018, Hamilton, New Zealund, November 19-22, 2018, Pro- ceedings 20, pages 278-289. Springer.
Sconmin Koo. Chanjun Park, Jachyung Sco. Seungjun Lee, Hyeonseok Moon, Jungseob Lee, and Heuiseok Lim. 2022. K-net: Korean neural grammatical emror correction gold-standard test set using novel error type classification criteria. IEFE Access, 10: 10:118167- 118175

---

Thi Tuyet Hai Nguyen, Adam Jatowt, Nhu-Van Nguyen, Mickael Coustaty, and Antoine Doucet. 2020. Neural machine translation with bert for post-ocr errrr detec- tion and correction. Proceedings of the ACM/IEEE joint conference on digital libraries in 2020, pages 333-336
robust speech recognition errors. arXiv preprint arXiv:2006.05633
Sihui Wang, Tom Gunter, and David VanDyke. 2018. On modelling uncertainty in neural language gen- cration for policy optimisation dialog assistants. In 2nd Workshop on Conversa- tional AI: Today's 's Practice and Tomorrow's 's Poten- tial, NeurIPS.
voice-triggered
OpenAI-Blog- 2022. Chatgpt: Optimizing language models for dialogue.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: an asr cor- pus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5206-52111 IEEE.
Jason D Williams and Steve Young. 2007. Partially ob- servable markov decision processes for spoken dialog systems. Computer Speech & Language, 21(2):393- 422.
J.P. Woodard and year 1982 journal = Workshop on standardisation for speech I/O technology. Naval Air Development Center, Warminster, PA title An in- formation theoretic measure speech recognition performance Nelson, J.T.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: method for automatic evalu- ation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computa- tional Linguisties, pages 311-318.
Soyoung Yoon, Sungjoon Park, Gyuwan Kim. Junhee Cho, Kihyo Park, Gyu Tae Kim, Minjoon Seo, and Alice Oh. 2022. Towards standardizing korean gram- matical error correction; Datasets and annotation. arXiv preprint 2rXiv22200.43899
Chanjun Park. Jachyung Seo. Seolhwa Lee, Chanhee Lee, Hyeonseok Moon, Sugyeong Eo, and Heui-Seok Lim. 2021. Bis: Back transcription for speech-to-ten post-processor using text-to-spp to-speech-to-tere In Pro- ceedings of the 8th Workshop on Asian Translation (WA72021), pages 106-116.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brock- man, Christine McLeavey, and Ilya Sutskever. 2023, Robust speech recognition via large-scale weak su- pervision. In International Conference Machine Learning. pages 28492 28518. PMLR.
Yongmei Shi and Lina Zhou. 2011. Supporting dicta- tion speech recognition error correction: the impact of external information. Behaviour & Information Technology, 3066)7761-774.
Myung-Suue Shin, Jong-Bot Ahn, Hyun-Wook Nam, and Do-Ha Kwon. 2005. A study of dysfluency char- acteristics in normal adults and children in mono- logue. Speech Sciences, 12(3):49-57.
Claytone Sikasote and Antonios Anastasopoulos. 2022. BembaSpeech: A speech recognition corpus for the Bemba language. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 7277-7283, Marseille, France. European Lan- guage Resources Association.
Bernhard Suhm, Brad Myers, and Alex Waibel. 2001. Multimodal error correction for speech user inter- faces. ACM transactions on computer-human inter- action (TOCHI), 8(1):60-98.
Wei Tang and Matthew Lease. 2011. Semi-supervised consensus labeling for crowdsourcing. In SIGIR 2011 workshop on crowdsourcing for information retrieval (CIR), pages 1-6.
Longshaokan Wang. Maryam Fazel-Zarandi, Aditya Ti- wari, Spyros Matsoukas, and Lazaros Polymenakos. 2020. Data augmentation for training dialog models

---

A Related Works and Background
and accurate context, is provided to assist in post-processing for error correction (Shi and Zhou, 2011). In particular, Bassil and Semaan (2012) use the N-gram dataset for ASR errors to detect and correct errors automatically. Models such as LSTM-based or Transformer-base sequence-to-sequence architectures are adopted to correct the speech recognition results while considering the semanties and spelling (Guo al., 2019; Hrinchuk al., 2020).
Post-processing Model Post-processing serves an important role in quality enhancement across various fields by modifying the distorted output into appropriate statements. For instance, in the field of optical character recognition (OCR), con- ventional approaches such as manual, lexical, and statistical methods have been used (Evershed and Fitch, 2014; Nguyen et al., 2018). More recently, language models like BERT have been employed for eπror detection in tasks like named entity recog- nition (NER) and are performed through character- level machine translation (Nguyen et al., 2020).
Recent studies strive to improve ASRP perfor- mance by utilizing the results derived from ASR. Gekhman et al. (2022a) introduce the ASR confi- dence embedding (ACE) layer to the encoder of the ASR model to jointly encode the confidence scores and transcribed text into contextualized represen- tation. To mitigate the time and cost-related chal- lenges associated with the parallel data required for training, Park et al. (2021) employ Text-to-speech speech (TTS) and Speech-to-text (STT) technologies to construct parallel data.
As another field, machine translation (MT) often utilizes the following methods. Post-processing re- search is being carried out in automatic post-editing (APE) to improve translation quality by adopt- ing transfer learning (Correia and Martins, 2019). Concurrently, in the grammatical error correction (GEC) field, transformers and the copy mechanism are used to correct spelling and grammatical errors MT results (Lee et al., 2021). Studies that de- fine error types to construct test sets or utilize an automatic grammatical error annotation system to create datasets also exist to improve Korean GEC studies (Koo et al., 2022; Yoon et al., 2022). Like- wise, the study on post-processing is actively ex- plored in a wide range of fields and holds signifi- cance in terms of enhancing the quality of output results. This can also be of significant importance in the field of Automatic speech recognition (ASR), which is discussed in the following section.
ASR dataset The availability of suitable datasets is imperative for the active progression of ASRP. Previously, post-processing studies have been con- ducted with ASR datasets. Panayotov al. (2015) organize the two labels in the ASR dataset that de- note the quality of speech recognition, classified into 'clean' and 'other' categorics, providing valu- able assistance in the analysis. Ardila et al. (2020) constuct comprehensive ASR dataset that includes demographic metadata such as age, sex, and accent to provide wider representation.
Transcription hypotheses obtained by decoding audio data using an ASR model are used to align hypothesis words with the reference (correct) tran- scription. The process of labeling errors and non- errors is facilitated by employing the minimum edit distance (Gekhman et al., 2022b). In the context of Chinese language datasets, significant dataset is available for speech recognition systems, labeled with audio devices and recording environments (Bu et al., 2017). Gekhman et al. (2022b) build dataset by aligning hypothesis words with the reference (correct) transcription through a transcription hy- pothesis obtained by decoding audio data with an ASR model and labeling errors and nonerrors using minimum edit distance. In the context of Chinese, large-scale dataset is available for speech recogni- tion systems labeled with audio device information and recording enviromments (Bu al., 2017).
ASR Post-Processing Model ASR post- processing (ASRP) involves the detection and correction of errors in the output of an ASR, distinguishing it from simple error correction in that it considers user-friendliness aspect. This approach can improve the final quality of statements without modifying the ASR system structure. For instance, in specialized fields like the medical domain, attempts have been made to eliminate punctuation errors in ASR through prot-psrcessiig (Mani et al., 2020a). Prior research has primarily focused on providing information that allows humans to manually rectify erroneous segments, proposing alternative words for correction or creating an environment conducive to modification (Suhm et al., 2001; Feng and Sears, 2004). External information, such as word alternative hypothesis, noisy context,
additional
To mitigate the problem of insufficient training

---

data, methodologies that synthesize data via data augmentation methods have been proposed (Liao et al., 2022). However, the overall quality of the data is more crucial than the size. Specifically, the detailed datasets that consider both speech- and text-level like the real world are absent. Con- sequently, we aim to construct the ASR Post- Processing dataset, which contemplates audio- and text-level for the first time.
C Description of Text-Level Error Type
Spacing encapsulates instances contravening stan- dard spacing conventions. Punctuation entails cases where punctuation omitted or misapplied in Korean sentences for instance, when 'Can teach?' is interpreted as 'Can I teach.' Numerical encompasses cases where number conversion fails, such as when 'Ahead of the three-month schedule' is interpreted as 'Bill 2, 3-month schedule'.
B Description of Speech-Level Noise Type
Spelling and Grammar consists of ten detailed subcategories. Remove designates cases where some word components are not recognized, or end- ings or particles are missing example, when 'The champion is in the final' is misinterpreted as 'Champion final'. Addition involves cases where the same word is repeated or unutilized particles or endings are appended. For instance, when 'World's fruits, fish, and meat' is interpreted 'World's world's fruits, fish, and meat'. Replace refers instances where one word is substituted with an- other-for example, when 'Apply the filter.' is in- terpreted as 'Wear the pizza'. Separation refers to instances where consonants and vowels the target utterance are separated, exemplified when "The discount applies as it is.' is interpreted as 'Dis- count app lise it is.'. Foreign word conversion refers to cases where words deviate from standard foreign word pronunciation or some syllables are incorrectly converted from English to Korean or vice versa. For example, when 'Brazil's Samba Fes- tival' is interpreted as Brazil's SsamBap Festival,' or 'I prefer to use ATM.' is interpreted as 'I prefer to use hm.'.
Pause (silent) category captures instances where silence intervenes mid-utterance before comple- instance, when 'I am cating' is articu- lated as 'I am... eating'. . Filled pause represents cases characterized by the habitual insertion of filler sounds during pauses, as in utterances supple- mented by sounds such 'um... uh... so I'. Inter- jection category encompasses instances where one more words or phrases irrelevant the intended message are interjected, evident in utterances like "Okay I see, but you know'. Parenthetical category includes instances where grammatically correct, but semantically neutral phrases are inserted-for instance, utterances incorporating phrases such as 'you know' and 'I mean'. Unfinished interlocu- tor category denotes cases where the utterance concludes prematurely- instance, when eating' truncated 'I am.... Word repetition category signifies instances where the same word is iterated, as in saying 'Hello' as 'Hello Hello', Syllable repetition category characterizes cases where the same syllable is iterated-for instance, when 'Hello' is articulated 'He-hello'. Phoneme repetition category encapsulates instances where the same phoneme is repeated, such saying 'Hello' 'Hel-llo'. Sustained category accounts for instances where part of an utterance is elon- gated, exemplified in 'Is that so-right?'. Hyper- fluency category represents instances of excessive verbosity. Mutter category includes cases where utterances are murmured in an indistinct manner, as in 'That.. is.. like that.... Dynamic error category encompasses instances where syllable articulation strength is incongruous with the intended utterance, instances that are challenging to comprehend at the human-level. Finally, speaking rate category accounts for instances where rapid speech pace hinders comprehension at a hemen-level.
Spelling is bifurcated into two types: Grapheme- (o-Phoneme (G2P) and Consonant vowel conver- sion. G2P pertains to instances where a character is recognized per its pronunciation. Consonant vowel conversion refers to instances where phone- mic units are incorrectly spelled. Post-position refers to cases where different particles are used or omitted for example, when 'Ordinary high school students' is interpreted as 'Ordinary at high school students." Syntax involves cases where the gram- matical interpretation remains valid, but the seman- tic interpretation varies. Finally, neologism refers to cases where the target word and its meaning and pronunciation are dissimilar and are not included in Korean vocabulary.

---

D Human Annotation
D.1 Crowd-soureing and Compensation
We recruited individuals who are native speakers of Korean and selectively hired candidates suit- able for the task through validation questions. Ev- ery employee has been fairly remunerated least a rate of 140 KRW per task. is expected that each worker will complete 2-3 questions within a minute, guaranteeing a minimum compensation of 16,800 KRW per hour. Comparatively, the mini- mum hourly wage in South Korea for 2023 is 9,620 KRW. The guidelines for annotation and the user interface illustrated in Figure 6 and Figure 7.
4-1 OH. 100
6:00 ++
D.2 Annotation Guidelines and Interface
200 034
Speech-level enorr
Speech level error
w8th
Speoch-level
8/
A
प|l
0
2
Lelh
2
THE BEB EUR ESHN S1 RNETA
199 HRI H8t 318 C 033
#3554 -2--000 BUP EAR 6-211
3
#2202 9/1 -00- B GHB
Transcription
Figure 7: Difficulty annotation setup. Q: Please evalu- the level difficulty accurately transcribing the speech to match the given transcript.
2025-05-00 18:8 Tot R.by 8 738 442-48
M2)
75
Figure 6: Speech recording setup.

---

D.3 Annotation Demographies
E Analysis of Correlation by Difficulty Levels
The detailed demographic information is provided
in Table
We believe that providing difficulty information fa- cilitates the analysis of weaknesses in ASR models. We extracted an equal number of samples for each difficulty level and analyzed them. Figure Fig- ure 9, and Figure 10 show the correlation between the noisy environment at the speech level and text- level errors in diverse difficulty settings. Figure Figure 12, and Figure 13 illustrate the correlation between speech-level characteristics of interlocutor and text-level errors in diverse difficulty settings. Spa/Punc/Num represent spacing, punctuation, and numerical, respectively. Rem/Add//Rep/Sey indi- remove, addition, replace, and separation, spectively. FWC/G2P/CVC correspond to foreign word conversion, grapheme-to-phoneme and con- sonant vowel conversion, respectivcly. PP/Syn/Nco signify post-position, syntax, and neologism, re- spectively.
Gender
Male
186 (7.50%) 2292 (92.49%)
Female
Age
20-29
479 (19.33%)
30-39
1318 (53. (53.
40-49
681 (27.48%)
Nationality
Korea 2478 (100%)
Table 7: Demographics of the crowd workers involved in the composition of the data.
Analyzing the details based on different diffi- culty levels can be employed to enhance the inter- pretability of the ASR model. For example, in the case of Google, experimental results show that the correlation from "Terminal' speech-level type 'Punctuation' text-level type is strong for easy level. 'Construction site' speech-level type to 'Addition' text-level type for medium level, and 'Terminal' speech-level type to 'Replace' or 'Remove' text- level type for hard level. For Clova, the tendency of 'Replace" text-level type in 'Individual transporta- tion' speech-level type is strongest easy level, and it is strongly related 'Syntax' and 'Replace' text- type at medium level. At the hard level, has a strong tendency to 'Remove' and 'Syntax' text-level types.
(8)
Figure 8: Correlation distribution between speech-level: noisy environment and text-level in Google(a) and Clova(b), in easy-level difficulty setting.

---

20
30
Poun
Open
Figure 9: Correlation distribution between speech-level: noisy environment and text-level in Google(a) and Clova(b), in medium-level difficulty setting.
Figure 12: Correlation distribution between speech- level: characteristics interlocutor and text-level Google(a) and Clova(b), in medium-level difficulty ting.
16
Figure Correlation distribution between speech- level: noisy environment and text-level in Google(a) and Clova(b), hard-level difficulty setting.
(0)
Pott 2009
Pert ptm
Jubala
Figure 13: Correlation distribution between speech- level: characteristics of interlocutor and text-level in Google(a) and Clova(b), in hard-level difficulty setting.
Figure 11: Correlation distribution between speech- level: characteristics of interlocutor and text-level in Google(a) and Clova(b), in easy-level difficulty setting.

---

F LLM for Validation
F.1 Prompts of ChatGPT
Task Description
Your task to classify any spelling or grammar errors within a sentence,
Always answer in Korean.
Error Types Description
The definition and examples are as follows: o] ø*) : [] 87]
a 인 s 8
Example: a _ a
> J _ oo HR
Result: =|0] 2
#*#
Examples
Examples of classifying multiple grammatical error types are as follows: {{examples)}
Input
Referring to the definition and example, classify grammatical error type that fit the given sentences.
Example: {(input sentence))
Result:
Figure 14: Error type classification prompt.

---

F.2
Examples of Classified Types
Input Sentence
STT Result
Predict Types
Target Types
4-2l 2/0/7} 3 127l 2/17 7h.
npst 7
Euror
Punctuation
(Shall we wait until a seat hecomes available for us)
(Shall we wait untill a seat hecomes available for us?)
> aoot 4| ghd $1 4:44 4:
[ Alfo # 4 ghd 4
AAE. []
g. *o044.
Addition, Spacing, Punctuation
Remove, Spacing
only instruct you
only
you
The homework homework
done by yourself)
homework must done by yourself.)
2 at & & oif E/g of 440t
Eng at 12 of t- all [el..
Syntax, Replace, Punctnation
(When students wear school uniforms, what does
(When students wear school uniforms, the students
Post positionitonn
'neatness' mean)
appear neat.)
187 / 4i Rojs iI $11. police officer)
!/ 2 Bo Moig? a police officer?) (What would have asked
Spacing
Punctuation
(What would have asked
2- $4 J U 2 + gojiz A
4 3 Al 14452 4l Ale A you want import our company" new nutritional supplement products.)
Punctuation, Spacing,
Panctuation
heard you want to import our compuny's new
Remove
nutritional supplement products.)
Table 8: Example of text-level types classification in ChatGPT. STT result refers to the speech-to-text result.


Reviews:
[{"review_id": "KGKgmouJCJ", "reviewer": "Reviewer_chwF", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Thanks for your comments!\n\nThank you for your constructive comments and suggestions; they are exceedingly helpful in improving our paper. Our point-to-point responses to your comments are given below. \n\n> While the paper provides the performance of Google and Clova networks, it will also be interesting to report and benchmark on some common existing networks that encourage academia to find a baseline/benchmark for their future research.\n\n- We carried out experiments using the most representative and effective commercialized systems in the Korean language, aiming to demonstrate the practicality of our dataset from a real-world-centric perspective. Moreover, as you have recommended, we intend to conduct supplementary experiments encompassing the Whisper model and other potential alternatives.\n- For this purpose, we first provide a comparison with Whisper. Both the code and models of Whisper are publicly available to researchers, facilitating their utilization.\n\n\n|||WER|||CER||\n|--- |:---: |:---: |:---: |:---: |:---: |:---: |\n||Easy|Medium|Hard|Easy|Medium|Hard|\n|Google ASR|0.47|0.63|0.93|0.21|0.34|0.69|\n|Clova ASR|0.53|0.67|0.94|0.2|0.35|0.73|\n|Whisper|0.48|0.67|0.92|0.23|0.35|0.65|\n\n- In the final version, we will incorporate the analysis between other models' speech- and text levels as a guide for future researchers.\n- We aim to expand our research scope to encompass a broader range of models. Given the practical perspective of this paper, we kindly request your consideration of our 1) key results centered on commercial systems, 2) motivation, and 3) releasing our dataset.\n- Furthermore, KEBAP emphasizes the balance between recognition accuracy and user readability. To achieve this balance, we have meticulously categorized noise types, enabling a thorough correlation analysis between speech and text levels. While some studies focus on either recognition accuracy or user readability individually, no study investigates a real-world-centric situation where the balance between the two sides is essential, like ours.\n- For example, some previous research has categorized noise types to improve recognition accuracy. However, our study has at least 3.7 times as many defined types, resulting in the diagnosis of fine-grained error types. We will include the statistical comparisons in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The dataset provides many additional domain information that increase the diversity of the corpus. The new corpus also contains many additional evaluative metric that helps to assess the ASR system in more complete angles. As such, it is distinctive to existing dataset and adds value to the community when the corpus is available open-sourced. Besides, the work provides many interesting evaluations from the dataset.", "reasons_to_reject": "While the paper provides the performance of Google and Clova networks, it will also be interesting to report and benchmark on some common existing networks that encourage academia to find a baseline/benchmark for their future research. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a new dataset called Korean Error Explainable Benchmark Dataset for ASR and Post-processing (KEBAP). The proposed dataset addresses the limitations of the previous existing corpora by providing the readability component to assess the quality of the predictions. KEBAP provides diverse utterances design with different noise corruption and speaker characteristics categories. It presents 13 distinct text level error types and it appears that the dataset is more fine-grained and real-world-centric to evaluate the performance of the ASR systems.  ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "OYn6orINow", "reviewer": "Reviewer_L3vD", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Thanks for your comments!\n\nThanks for your constructive comments and encouraging words, and we sincerely appreciate your time in reading the paper. In the following, your comments are first stated and then followed by our point-by-point responses.\n\n> 1. Line 97 – clarify more explicitly how you annotate difficulty levels.\n\n- We asked the question, \"How difficult is it to recognize the presented speech accurately as the same as the transcript?\" to three human evaluators and measured their responses using a Likert scale. Specifically, we asked people to answer the question after listening to the recorded voice without looking at the transcript. Additionally, to mitigate subjectivity in scoring, we accompanied the scoring system with clear explanations of the intended significance of each score. For example, a score of 1 (very easy) signified that the speech could be recognized entirely (100%) identical to the transcript. Conversely, a score of 5 (very difficult) indicated that only 60% or less of the speech could be recognized identically. At the end of the evaluation, we adopted people's average score as the difficulty level. This method enhances the scoring process's objectivity and ensures a comprehensive understanding of the perceived difficulty in accurately recognizing speech.\n\n\n> 2. Any bias to be addressed in human speaker recording and background noise addition?\n\n- To address bias, we engaged a dedicated expert in the process who played a role in validation. During the stages of speaker recording and background noise addition, the expert ensured the accurate inclusion of provided sentences and noises and scrutinized any potential biases that could be socially problematic.  \n- First, a debiasing step was conducted for the transcript validation process  (Section 2.4 - Step 1). Following the guidelines provided to our workers, any content that could raise societal concerns was subjected to rejection through consultation with the expert. We will explain this in more detail in the final copy.\n- In addition, Annotation Demographics are presented in Appendix D.3. KEBAP consists of recorders ranging in age from 20s to 40s. We also used more female recorders to include data of varying difficulty, as ASR AI models [1][2].\n\n\n[1] Investigating the Impact of Gender Representation in ASR Training Data:\na Case Study on Librispeech (https://aclanthology.org/2021.gebnlp-1.10.pdf)\n\n[2] Effects of Talker Dialect, Gender & Race on Accuracy of Bing Speech and YouTube Automatic Captions (https://www.isca-speech.org/archive_v0/Interspeech_2017/pdfs/1746.PDF)\n\n> 3. Missing point of view on Scaling to other languags.\n\n- Our dataset is the first paper considering both recognition accuracy and user readability, and the constructed dataset is planned to be released. While we have focused on Korean due to cost considerations, our dataset's approach of fine-grained types is easily extendable to other languages using the same methodology. We are devoted to expanding our dataset to encompass other languages in future work. \n\n\n> 4. Could benefit from possible Ablation study in the setup.\n\n- Our main contribution lies in proposing and releasing a novel type of dataset from a practical perspective. For effective validation of KEBAP, we analyzed tendencies between speech- and text-level. Since we annotated the difficulty level of the data, we believe adding ablation studies based on the difficulty also would benefit. For this purpose, we extracted an equal number of samples for each difficulty level and analyzed them.\n- For example, in the case of Google, experimental results show that the correlation from ‘Terminal’ speech-level type to ‘Punctuation’ text-level type is the strongest for easy level, ‘Construction site’ speech-level type to ‘Addition’ text-level type for medium level, and ‘Terminal’ speech-level type to ‘Replace’ text-level type for hard level. We will include the entire results for the other models in the final copy.\n- In addition, we tried to measure the performance before and after noise synthesis. Experimental results show that for Google, the WER is 0.49, the CER is 0.23 before noise synthesis, and the WER is 0.68 and the CER is 0.41 after noise synthesis. For Clova, the WER before noise synthesis is 0.53, and the CER is 0.19, while the WER after noise synthesis is 0.71 and the CER is 0.43. These results are interpretable in that the resources we provide are high-quality and helpful.\n\n\n> 5. Missing comparison with other models and bench-marking.\n\n- We conducted experiments with the most typical and high-performing commercial systems in the Korean language to demonstrate the utility of our dataset from a real-world perspective. Additionally, we greatly value your constructive review, and in consideration of future research, we plan to perform additional experiments, including the Whisper model and other alternatives, as you have suggested.\n- For this purpose, we would like to share our comparison with Whisper, as the code and models of Whisper are publicly available for researchers.\n\n|||WER|||CER||\n|--- |:---: |:---: |:---: |:---: |:---: |:---: |\n||Easy|Medium|Hard|Easy|Medium|Hard|\n|Google ASR|0.47|0.63|0.93|0.21|0.34|0.69|\n|Clova ASR|0.53|0.67|0.94|0.2|0.35|0.73|\n|Whisper|0.48|0.67|0.92|0.23|0.35|0.65|\n\n- We will include the analysis of trends between speech- and text-level in the final copy so that it can be an indicator for future researchers. \n- We will try to extend our work to a wider variety of models in the future. As this paper is approached from a practical point of view, we hope you will consider our 1) key findings of focusing on the commercial systems, 2) motivation, and 3) data release.\n- Furthermore, KEBAP emphasizes the balance between recognition accuracy and user readability. To this end, we have fine-grained noise types, enabling a detailed correlation analysis between speech- and text-level. While existing studies consider either recognition accuracy or user readability, no studies investigate a real-world-centric situation where the balance between the two aspects is necessary, like our study. For instance, while existing studies have categorized noise types for recognition accuracy, the number of defined types in our research is at least 3.7 times greater, resulting in the diagnosis of subdivided error types. We will enhance our paper by including statistical comparisons in the final copy.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tNew benchmark dataset for ASR and post-processing evaluation in the Korean language and making it public for further research.\n2.\tWide range of speech-level noise type classes.\n3.\tBrings in much needed explainability in ASR to forefront.\n4.\tWell written paper with technical soundness.\n", "reasons_to_reject": "1.\tLine 97 – clarify more explicitly how you annotate difficulty levels.\n2.\tAny bias to be addressed in human speaker recording and background noise addition?\n3.\tMissing point of view on Scaling to other languages.\n4.\tCould benefit from possible Ablation study in the setup.\n5.\tMissing comparison with other models and bench-marking.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The author through this paper propose Korean Error Explainable Benchmark Dataset for ASR and Post-processing (KEBAP) as a new ASR evaluation methods. KEBAP enables comprehensive analysis of ASR systems at both speech and text levels, facilitating a more balanced assessment encompassing speech recognition accuracy and user readability. As part of the proposal KEBAP presents 37 newly defined speech-level error types, incorporates diverse noise environments, speaker characteristics categories and it also presents 13 distinct text-level error types. It ends with detailed analysis of noise categories and error types and also uses LLM for classifying error type.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "6ifbAvR5He", "reviewer": "Reviewer_yM4D", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Thanks for your comments!\n\nWe truly appreciate your considerate understanding of our contribution and your high praise. We felt our work was worthwhile because of your thoughtful review. It motivates us to continue striving for excellence in our contributions to the academic community. Thank you again for taking the time to review my work and for providing such encouraging feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Introducing a new and novel evaluation method for identifying and mitigating potential weakness in ASR systems.\n2. Clarity in research problem, hypothesis and analysis of how the proposed method could benefit the community. \n3. Easy to read and understand.\n4. The proposed method is useful, as it helps identifies potential errors associated with models outside impressive scores on known metrics. It is interesting to analyse the models performance on these fine-grained error levels. ", "reasons_to_reject": "Null.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The manuscript introduces a novel benchmark dataset for assessing and identifying weakness and vulnerabilities associated with ASR systems at both speech and text level. The study identifies a key research problem in  current evaluation methods, which fails to provide information on the exact weakness of an ASR model . The proposed method introduces explainability through speech and text level errors. The speech level comprise of errors on noisy environments and speaker characteristics in 37 distinct error types, and 13 distinct text-level error types. Furthermore, the authors applied this proposed evaluation method to commercially deployed ASR systems such as Google  cloud ASR and CLOVA. The authors concluded by presenting a comparison analysis on errors for  each of these commercial systems and discussed possible limitations of their work. ", "excitement": 5, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: **Originality**

This paper is very original and proposes evaluating ASR systems in fairly novel way. It enables this evaluation by presenting a dataset with fine-grained annotations for noise types (both background and "speaker" noise).

**Significance** 

The topic of explainability in ASR, which this paper is among the first to address, is very important. The resource that the authors have created can easily be used in conjunction with deletion, insertion and substitution errors to help diagnose ASR errors in the presence of specific source noises or speech characteristics. However, it does not appear that automatic classification of text errors using ChatGPT is feasible, so there may be limits the extent to which this error analysis can scale to other ASR systems.

**Clarity**

The paper is generally well structured, and most details are explained. Most reviewers agreed the paper was well-written, however, as the meta-reviewer, I did want to inform the authors of a number of grammatical, spelling or other errors that I found while reading the paper, and which I found made it slightly difficult follow at times, but which are easily fixable.

(1). "Our motivation for proposing a KEBAP, which contains both aspects, is detailed below." -- Which two aspects? It was not clear.

(2) "Since benchmarks measure performance with quantitative metrics, it is crucial **to fine-grain** characteristics for a more detailed diagnosis" -- Perhaps there was a missing word here? it is crucial to **examine**?

(3) "Hence, to solve the explainability issue, we must define **the** error type criteria that consider both the speech- and text-level and create benchmarks to achieve human-level explainability."  -- remove **the**

(4) GEC acronym in 2.3 is not defined until the subsequent section 

(4) "we selectively compose **a** text-level error types dataset by human evaluation." 

Step 1: Build Text-Level Error Corpus (An example would be nice for clarification)

(5) "we request the recording participants to incorporate characteristics" --> "we request that recording participants incorporate characteristics" 

(6) The x-axes in figures 4. and 5. use a different order of text-level errors. It makes it hard to interpret those plots. Furthermore, they are called correlation distributions, but I think they are just 2-d histograms with counts of errors. The text should be updated to reflect this.  

**Pros:**
   - An exciting new way to evaluate and compare ASR systems
   - A new data resource to enable ASR model comparison using different noise types (background and speech)
   - Generally well-written save for the items mentioned in the previous section
   - The noise portion of the dataset is easily applicable in the analysis of new models, as demonstrated by the rapid feedback from the authors.

**Cons:**
   - Almost none
   - The proposed method currently relies on human annotation of text errors as far as I can tell which limit applicability of the text errors to other models.
Decision: Accept-Main