New Datasets and Controllable Iterative Data Augmentation Method for Code-switching ASR Error Correction
Authors: Zhaohong Wan
Xiaojun Wan
Wei Peng
Rongjun Li
Venue: EMNLP 2023 Findings (2023)
Keywords: ASR Error Correction
Code Switching
Data Augmentation

Abstract: With the wide use of automatic speech recognition(ASR) systems, researchers pay more attention to the ASR error correction task to improve the quality of recognition results. In particular, ASR in bilingual or multilingual settings, namely code-switching ASR, has greater challenges and research value. In this paper, we first present code-switching ASR correction datasets obtained from solid ASR systems and automatic annotators. The datasets contain Chinese-English code-switching dialogues of bilingual speakers in Singapore, Malaysia, and Hong Kong. Based on this task, we propose a controllable iterative (CI) data augmentation method for improving the performance of mainstream ASR error correction systems. With a small amount of training data, our proposed method has the ability to iteratively produce abundant pseudo parallel data from the monolingual corpus for Chinese-English code-switching ASR correction. Results of experiments show that our method achieves the best performance compared with the rule-based, back-translation-based data augmentation methods and large language model ChatGPT.

New Datasets and Controllable Iterative Data Augmentation Method for Code-switching ASR Error Correction
Zhaohong Wan 1.2 and Xiaojun Wan and Wei Peng' and Rongjun Li !Wanxxuan Institute of Computer Technology, Peking University The MOE Key Laboratory of Computational Linguistics, Peking University Artificial Intelligence Application Research Center, Huawei Technologies { {xmwzh, wanxiaojun jekku edu.
Abstract
Error Type
Redundant
12, S start on XT a essay
Source
(1 want to start on my a essay.)
With the wide use of automatic speech recog- (itioo(ASR)) systems, researchers pay more at- tention to the ASR emrrr correction task to im- prove the quality of recognition results. In par- ticular, ASR in bilingual multilingual tings, namely code-switchig ASR, has greater challenges and research value. this paper, we first present code-switching ASR correc- tion datasets obtained from solid ASR systems and automatic annotators. The datasets con- tain Chinese-Englishe code-switching dialogues bilingual speakers in Singapore, Malaysia, and Hong Kong. Based on this task, we pro- pose controllable iterative (CI) data augmen- tation method for improving the performance of mainstream ASR error correction systems. With a small amount of training data, our proposed method has the ability to iteratively produce abundant pseudo parallel data from the monolingual corpus for Chinese- Englishh code-switching ASR correction. Results of experiments show that our method achieves the best performance compared with the rule- based, sacc--ranslation-basee data augmenta- tion methods and large language model Chat- GPT.
# T2 start on 38B) essay
Target
(1 want start my essay.)
Table 1: Examples of Shiinss-Engllsee code-switching ASR error correction in datasets. The contents in brackets are the translations for non-native Chinese speakers.
language pairs have been introduced (Dowlagar and Mamidi, 2023; Seddah et al., 2020), including the code-switching ASR task.
Employing multilingual ASR systems solve code-switching ASR is proposed in recent work (Lovenia al., 2021). However, the performance of systems in many cases still far from satisfac- tory. Errors introduced by ASR systems usually affect the performance of downstream tasks. To improve the recognition performance, the code- switching ASR error correction task is necessary.
In this paper, we focus on code-switching ASR error correction between Chinese and English, which is common in the Chinese communities of Asia area. To enable the research and development of code-switching ASR error correction, we pro- pose two datasets SEAME-C and ASCEND-C that are constructed from two Chinese-English code- switching ASR datasets SEAME (Lyu et al., 2010) and ASCEND (Lovenia et al., 2021). In Table 1, we present typical examples of Chinese-Englist code-switching ASR error correction.
1 Introduction
Recently, automatic speech recognition (ASR) sys- tem has achieved impressive success with the adop- tion of advanced deep neural network (Moritz et al.. 2020). Learning from training data, the ASR sys- tem can match human transcribers for some critical tasks. It has been widely applied several areas, such as live captioning, intelligent appliance con- trol (Aliprandi et al., 2014).
A challenge for the code-switching ASR error correction is the lack of a large scale of training data. Moreover, we can not even collect sufficient high-quality Chinese- Considering these challenges, we propose control- lable iterative (CI) data augmentation method that can iteratively generate abundant code-switching ASR error correction instances from the monolin- gual corpus with small-scale labeled training data.
In some bilingual and multilingual societies, people often speak more than one language in conversation. The speech phenomenon of switch- ing language among sentences is called code- switching. To explore the code-switching phc- nomenon, code-switching tasks in many different
code-switching data.

---

To verify the effectiveness of our proposed method, we we implement two data augmentation base- lines and evaluate the performance of these meth- ods on proposed datasets at first. With a similar scale of augmented data, our proposed controllable iterative method achieves the best performance in both MaxMatch (M2) scorer (Dahlmeier and Ng. 2012) and MER metries SEAME-C and ASCEND-C datasets. It performs well for both mT5 (Xue al., 2021) and mBART (Tang al., 2020) correction methods. Besides, considering the strong ability of large language models (LLM), we compare the performance of our method with the most powerful LLM ChatGPT. Experiments show that our method outdistances the ChatGPT method, which shows the effectiveness of our method in the Chinese-Englinn English code-switching ASR error correc- tion task. Besides, we find that this task is chal- lenging, and LLM method is far from achieving satisfactory results at now.
error correction has become a prevalent task in recent years. The idea of regarding the ASR er- ror correction as the machine translation and ap- plied advanced neural machine translation models is adopted in several previous approaches(Mani et al., 2020; Wang et al., 2020). D'Haro and Banchs (2016) first introduce machine translation to im- prove automatic transcription. Considering the ma- chine translation models require only parallel text pairs, Guo et al. (2019) propose an approach to uti- lizing text-only data by training correction model to explicitly correct the errors. The joint modeling method shows great performance in this task Wang et al.. 2023). Re-ranking ASR n-best hypotheses is also a popular topic for error recovery (Corona et al., 2017: Jonson, 2006).
To address the problem lack of training data, data augmentation method widely adopted in text error correction area. Many works (Grund- kiewicz et al., 2019; Lichtarge et al., 2019) adopt pre-defined rules to generate sentences with errors. Inspired by back-translation procedure for machine translation (Senmrich al., 2016), works (Xic al., 2018; Kiyono et al., 2019)are proposed to train model to generate erroneous sentences. Recently, researchers focus on the exposure bias problem of data augmentation method and proposed meth- ods to utilize the augmented data in more effective ways (Solyman et al., 2023; Cao et al., 2023).
Our contributions are summarized as follows:
1. We propose two datasets SEAME-C and ASCEND-C for the challenging Chinese-Englin
code-switching ASR enror correction task.
2. To address the problem of lacking sufficient training data, we propose the controllable itera- tive data augmentation method that can iteratively generate abundant code-switching ASR error cor- rection instances from the monolingual corpus. 3. Extensive experiments show the superiority of our proposed controllable iterative method. More- over, combining the pseudo data produced by the rule-based and controllable iterative methods can further improve the performance of error correction models.
Code-switching ASR Error Correction Datasets
In this section, we provide details about our two datasets SEAME-C and ASCEND-C. Specifically, we give the construction process of our datasets at first. Then, we do an analysis of our datasets and show some characteristics of them. In this work, we adopt the same corpus splitting as the original ASR datasets. addition, we remove some bad cases that are too short and climinate monolingual sentences in the validation and test sets to ensure the quality of evaluation. Our datasets provide quality resources for the research and development of switcshit ASR error correction. Table 2 describes the statisties of our proposed datasets.
The resources are publicly available'.
2 Related Work
Code-switching ASR, the task aiming to recog- nize the speech that contains more than one lan- guage within the sentences, has been inereasingly reported in speech technology and linguistic stud- ies recently (Su, 2001; Auer, 2013). However, to our knowledge, there is no research on the code- switching ASR ситоr correction task at present. In this section, we introduce two relevant fields: ASR error correction and data augmentation method in text error correction.
3.1 Data Construction
In some Asia's multilingual countries and regions, speaking a mixture of Chinese and English within a sentence is common. We collect data from two Chinese- -ngglish code-switching ASR datasets with
With the successes of ASR systems, the ASR
http://hhhhb..ommmmapppoccccccnn Emor-CCrrecction

---

SEAME-C
ASCEND-C
Train
Test
Total
Test
Dialogues
297
Sentences
48,405
6,293
54,698
9,869
213
10,455
Tokens
801,548
99,837
901,385
122,348
3,385
5,538
131,271
Avg. sentences in a dialogue
181.3
209.8
174.2
246.7
53.3
92.3
213.4
Avg. tokens in sentence
16.6
15.9
16.5
12.4
15.9
14.8
12.6
Table 2: Statistics of proposed code-switching ASR emror correction datasets.
the advanced ASR system and automatic annotator.
SEAME-C ASCEND-C
At dialogue level
In this part, we introduce the main components of the data construction. Details of processing and implementation are shown in the Appendix A.
Code-switching dialogues
100%
100%
At sentence level
Code-switching sentences
76.6%
27.8%
Chinese sentences
14.3%
48.6%
ASR Corpus We build the error correction in- stances from two Chinese-English code-switching ASR datasets: SEAME and ASCEND. SEAME is a large Chinese- English code-switching spon- tancous speech corpus, collected from residents of Malaysia and Singapore. The corpus contains 30 hours of word-level transcribed code-switching speech data. ASCEND is high-quality Chinese- English code-switching corpus built sponta- neous multi-turn conversation collected in Hong Kong. It comprises 10.62 hours of clean speech data collected from dialogues.
English sentences
9.1%
23.6%
At token level
Chinese tokens English tokens
33.7%
92.5%
16.3%
7.5%
Table 3: Proportion of languages used in datasets at dialogue, sentence and token levels.
tion important for the analysis. We calculate the proportion of languages used in the datasets at dia- logue, utterance, and token levels. Table presents the details.
From the table, we can find that the sentence- level language distribution of two datasets is quite different. In the SEAME-C dataset, the code- switching sentences take the main proportion, ac- counting for 76.6%. As for the ASCEND-C dataset, the Chinese sentences account for the largest pro- portion, while the code-switching instances only ac- count for 27.8%. In terms of token-level language distribution, both two datasets are dominated by Chinese tokens. It shows that the code-switching phenomenon in Asia area mainly occurs by mixing few English words into Chinese speech. Consid- ering that, we generate pseudo data from Chinese sentences our data augmentation method the following sections.
ASR System With code-switching ASR datasets, we adopt the ASR system to obtain recognition re- sults. For being close to the real recognition setups, we employ advanced ASR model architecture, namely XLSR-53 (Xu et al., 2021). We fine-tune the XLSR-53 model and obtain the recognition re- sults for datasets. The code-switching ASR error correction instances in our datasets are built by the pairs of manually annotated transcriptions and corresponding recognition results.
Annotator We develop automatic annota- for for the Chinese-English code-switching cor- pus, based on prior works in English and Chinese (Bryant et al., 2017: Hinson et al., 2020). takes three steps, tokenization, alignment, and merging, to generate annotations that are suitable for Max- Match (M2) evaluation metrics. Besides, the auto- matic annotator is able to classify errors into four types: redundant (R), missing (M), word selection (S), and word ordering (W). It allows researchers conduct more detailed experiments and analyses with different error types.
3.2.2 Error Distribution
With the proposed automatic annotator, the errors in datasets are classified into four types. To ana- lyze the characteristic ASR errors in proposed datasets, we calculate the error distributions on the test sets of SEAME-C and ASCEND-C datasets. The results are presented in Table 4.
As shown in the table, the error distributions on the test sets of two datasets are similar. The word selection errors account for the largest proportion, with 85.7% in SEAME-C and 81.3% in ASCEND- C. This type of error is caused by mistakes the
3.2 Analysis
3.2.1
Language Distribution
As the code-switching datasets contain multiple languages, the observation of language distribu-

---

Code-switching Sentence
Code-switching Sentence with Errors Sigg-ine && good friend
Code-switching
without Errors
Sentence with Mare Errors
S//ESingaeore make
HE Singapre makes &
Output X2
1 good friends
friend
Iteration
Confidence
Ittrrttioon
Confidence
2nd Iteration Confidence
Iteration
Confidence Blas C
Blas En Meveing
Bias 8
Bias
(Balance Generation Probabilit)
[Balarum Gomeration
Unfold
Transfrmation)
Procabilly!
Sequence Editing Model
Sequence Editing Model
Sequence Friting Modal
Sequence Editing Modal
A A make many good friends in Singapore)
Input Xt-1
Monolingual Sentence
Figure I: An example of the process of generating pseudo data in controllable iterative method. We highlight the edited parts of sentences in each iteration. The controllable iterative method produces pseudo training instances by iteratively generating outputs from input sentences. We apply the sequence editing model to generate synthetic data and build training pairs with iterations.
Error Type
SEAME-C
ASCEND-C
rithm plays a crucial role in the proposed method. adopts the sequence editing model to iteratively produce pscudo training data under the control of confidence bias. With our designed transforma- tions, the sequence editing model is able to directly generate training instances from monolingual texts. Figure 1 shows example of the process of gen- erating pseudo data in our controllable iterative method.
#
Redundant (R)
908
5.6
134
11.2
Missing (M)
1,381
8.6
86
7.2
Word Selection (S)
13,831
85.7
972
81.3
Word Ordering (W)
15
0.1
0.3
Table Error distribution on the test sets SEAME-C and ASCEND-C datasets. The # columns present the amounts of specific error.
recognition of words or phrases with similar pro- nunciation. Besides, there are number of missing and redundant errors, which is challenging cor- rect for requiring to change the length of the input. Additionally, the frequency of word ordering errors is very few. We can find that the code-switching ASR eπror correction is task with imbalanced er- ror distribution, and we need to consider it when designing the correction method.
4.1 Sequence Editing Model
The data augmentation method for the code- switching ASR error correction task aims to gen- erate instances with errors pseudo training data. Error generation can be treated the process of noising the correct source input with edit opera- tions. Based on this idea, we adopt the sequence editing model to obtain the editing sequence re- quired to produce output with errors.
4 Controllable Iterative Data Augmentation Method
4.1.1 Predefined Transformations
The sequence editing method requires predefined editing operations called transformations. In this work, we adopt the custom token-level transforma- tions developed in GECToR (Omelianchuk et al.. 2020), the details are shown in Appendix B. These transformations are originally designed for En- glish texts. We denote these transformations ex- cept STRANSLATIONS as noising transforma- tions. For generating code-switching sentences from monolingual sources, we introduce cross- lingual transformations into original ones. Firstly, we add the translation transformation (STRANS- LATION$) that replaces the current token with the most common translation in a Chinese- Chinese-English
To address the problem of lacking sufficient train- ing data, we propose the controllable iterative data augmentation method. Due to the shortage of high-quality Chinese-Englist code-switching cor- pus, our method is designed to directly generate training instances from monolingual texts. Our method consists of two parts: sequence editing model and pseudo data generation algorithm.
Sequence editing model is the fundamental struc- ture of our method. It adopts the Transformer archi- tecture and the predefined editing operations called transformations to generate raw augmented data from seed corpus. Pseudo data generation algo-

---

dictionary. Secondly, to cover the specific cases that the most common translation can not fit, we insert additional Chinese characters to the vocabu- lary of SAPPENDS and SREPLACES operations. With predefined transformations, we can get the target sentence by applying the sequential outputs of transformations to the tokens in the source sen- tence.
Algorithm 1 Generating pseudo data
Input: Monolingual Sentence zo; Number of it- erations Sequence editing model Φ; Confidence bias e; Similarity Discriminator tे; Hyper parame-
ter X, 60, e1
Output: Synthetic erroncous samples set S Function: Gen(ro, T, X, 60, E1):
1: for t < to T do
2: if then
4.1.2 Model Architecture
3:
< 60
The sequence editing model is an encoder made up of Transformer architecture stacked with two linear layers with softmax layers. Considering the strong performance of pre-trained model, the Trans- former architecture initiated with multilingual pre-trained model m- m-BERT (Devlin et al., 2019). The model is able to generate editing operation sequence from the source input, and then apply the operations on the source to obtain the target output. To make the generation process of the model controllable, we add a confidence bias to the de- coding process. The confidence bias a parame- ter applied to change the generation probability of transformations. For example, adding a positive confidence bias to SKEEPS can increase the proba- bility of generating the SKEEPS. By adjusting the confidence bias, the decoding process of sequence editing model under control.
4:
else
c 6
6:
+ (
if > then
p (011 2t)) if p > À then S.add(z), xt)
10:
return S
4.2 Pseudo Data Generation Algorithm
To generate pseudo training data from the mono- lingual corpus for code-switching ASR error cor- rection, we have to generate both code-switching target sentences and source sentences with ASR errors. For this objective, we propose an iterative generation approach to obtain pseudo training pairs for code-switching ASR error correction with only one model in multiple iterations. The algorithm for generating pseudo training data is summarized in Algorithm
4.1.3 Training Process
To generate pseudo data, we need to train the se- quence editing model to generate noisy sentences from correct ones at first.
Given a monolingual sentence from seed corpus Zg. We apply T times iterations to the sequence editing model & to generate data.
Given a pair of monolingual sentence æ with- out noise and a code-switching sentence with noise y. Then we convert the sentence y to a sequence of transformations z. Let denote the preset con- fidence bias and denote all trainable parameters of the sequence editing model ø. Our objective is to find the optimal parameter set Ô that minimizes the following negative log-lik le-liiklihood function:
In the first iteration, bias 6g that limits the noising transformations and makes the model use cross-lingual transfor- mations and $KEEPS to generate code- switching sentence ø1. This process can be formulated as:
adjust the confidence
I1 + (( ep)
L = ---log log P (z|2, +)
(1)
In the 1-th (I<t S T) iteration, the confidence bias is set to 61 that balances the generation proba- bility of each transformation. It makes the model apply diverse noises to the output of the (t-1)-th iteration I7-1 and obtain the pseudo instances with errors zt. This process can be formulated as:
where (xx|x, e, θ) denotes the conditional proba- bility.
After training on constructed instances, our model can not only learn the editing operations apply errors to sentences but also the transforma- tions to generate Inglish-Chinese code-switching texts. Some details of training are shown in Ap- pendix C.
t- 6(x1-1,61)
(3)
where <

---

With above pseudo instances 21,...,277 we treat the output in the first iteration as the correct target sentence and the output in the later turn as noised source sentence to get the sentence pairs. In order to improve the quality of synthetic sam- ples, we use BERTSccre (Zhang et al., 2019) as the similarity discriminator to filter sentence pairs with low similarity. Given sentence pair (311, 26)) (I <t 2 T), we use similarity discriminator v to get a score p C [0, 1] that presents the semantic similarity between sentences #1 and Ty. We set a threshold À that if the score p of a sentence pair is greater than the threshold, the pair can be selected as an augmented training sample.
ChatGPT as the baseline. In our experiment, we test the performance of the GPT-3.5-turbo model on the code-switching ASR error correction task with zero-shot and few-shot settings. The prompt templates for ChatGPT are shown in Appendix F.
5.3 Data Augmentation Baseline
There are other ways generate pseudo code- switching ASR error correction training instances. In our experiment, we implement two data aug- mentation baselines to compare with our control- lable iterative data augmentation method. Firstly, we apply a code-switching text generation ap- proach to get the Chinese-Fngliss code-switching texts from parallel seed corpus. With the code- switching texts, implement the rule-baseds and back teee--ranslationbaseee data augmentation methods to improve the ASR error correction models.
5 Experimental Setup
In this section, we present our experimental setup including datasets, ASR error correction model, and data augmentation baseline. More details of implementation are presented in Appendix D.
Code-switching Text Generation Following pre- vious work (Hussein et al., 2022), we apply the EC theory (Pratapa et al., 2018) to generate Chinese- English code-switching texts. By taking the pairs of parallel sentences in seed corpus input, the method generates the code-switching texts by placing the token in the parse tree with its word- level alignments. In our experiment, we adopt the fast-align tool (Dyer et al., 2013) to get the word- level alignments and parse the sentences with a neural graph-based parser(Dozat and Manning).
5.1 Dataset for Experiment
Training and Test Data Firstly, we train the model on pseudo training instances generated from data augmentation method. Then, we fine-tune the code-switching ASR error correction model on our proposed Chinese-English code-switching ASR error correction datasets: SEAME-C and ASCEND-C. We take M scorer and mixed er- ror rate (Ade)(Adel al., 2013) as evaluation metrics.
Seed Corpus for Pseudo Training Data Genera- tion For data augmentation methods, we need to generate pseudo training instances from seed cor- pus. Considering proposed datasets were mainly collected from dialogues, we pick the OpenSubti- tles dataset (Lison and Tiedemann, 2016), which is a large collection of subtitles various languages for movies and TV programs, as the seed corpus.
Rule-based Method The rule-based method a simple way to generate errors efficiently. We de- sign rule-based method that generates synthetic training data with five rules: delete, add, replace, shuffle, spelling error. The details of the rules are introduced in Appendix E. With above rules, we can get synthetic training samples with various er- rors. These synthetic training samples can used to improve the performance of the code-switching ASR error correction system.
5.2 ASR Error Correction Model
multilingual grammatical error correction task, the adoption of multilingual version of pre-trained model has achieved great success (Rothe et al., 2021). So we pick two strong multilingual pre- trained models mT5 and mBART as the code- switching ASR error correction models to verify the effectiveness of our data augmentation method. Considering that large language models (LLM) have made significant advancements in various NLP tasks. We adopt the most powerful LLM
fack-translation Method Inspired by the use of tack-translation in downstream tasks (Xie et al., 2018), 8), we train seq2seq noisy model (like the mack-translation model) to generate noisy sentences from correct ones. Then the output of the noising model is paired with the input and the sentence pairs are then used as pseudo data. In our experi- ment, we consider adopting a base mT5 model as the noising model for our baseline.

---

Augmentation Method
Augmented Data Size
SEAME-C
ASCEND-C
Correction Model
Fost MER
RT
Fo.st
MERJ
26.2
18.7
21.14
20.7
15.3
19.3
40.31
Rules
5M
27.4
19.9
25.4
20.74
23.9
17.8
22.4
37.57
mT5
Back-translation
4.1M
27.1
20.5
20.70
23.3
17.4
21.8
38.21
3.3M
28.5
20.7
20.45
25.0
19.6
23.7
36.35
23.8
17.1
22.1
21.63
19.6
14.9
18.4
40.12
Rules
5M
25.4
18.9
23.8
21.18
22.5
17.2
21.2
38.63
mBART
Back-translation
4.1M
25.7
19.1
24.0
21.03
22.8
18.0
21.7
38.44
C1
3.3M
26.9
21.0
25.5
20.73
24.1
18.3
22.7
37.50
7.5
8.1
7.6
40.01
5.7
3.9
5.2
71.28
GPT-3.5-turbo33ssho
13.3
12.6
13.1
35.44
7.2
4.8
6.5
65.62
Table 5: Comparison results of each method. sentences used in each method.
Augmented data sizes show the amounts of additional training
6 Results and Analysis
tion models.
6.1 Comparison Results
Amount -+- FO.5 Soore
We evaluate the performance of our controllable iterative data augmentation method on proposed datasets. Table shows the results.
27.0
26.4
Augmertedd Dota
F0.5 Score
26.0
1.8
21.5
2.8
Firstly, we compare our method with the two data augmentation baselines. For the fairness of comparison, train the correction models with the similar scale of augmentation data. Here we use pseudo data generated from the controllable iterative method with three iterations to train the correction model.
24.5
24.0
2
3
3
Figure 2: Results of the mT5 correction model SEAME-C when trained with outputs collected from different iterations of our controllable iterative method.
As see from the table, proposed controllable iterative data augmentation method achieves the best performance in both 2 scorer and MER metrics on SEAME-C and ASCEND- C datasets. It outperforms the two baselines for both mT5 and mBART correction methods, which shows the effectiveness and robustness of our con- trollable iterative method in Chinese-English code- switching ASR eтror correction task.
6.2 Iteration of Proposed Method
We build the pseudo training pairs with the outputs of the controllable iterative method in multiple iter- ations. In this section, we try to explore the influ- ence of iteration number on the generation of augmented data. Figure 2 shows the results of the mT5 correction model on the SEAME-C dataset when is trained with outputs collected from dif- ferent iterations. The details of results are shown in Appendix G.
Besides, we test CharGPT (GPT-3.5-turbo) on the proposed dataset. From the table, we can find that the performance of ChatGPT is poor in this task. Under the zero-shot setting, it can only correct a few emrors. By learning from examples in the prompt, the performance of ChatGPT under few- shot setting has significantly improved, but still far from the performance of mT5 and mBART models.
As the iteration progresses, controllable itera- tive method continuously generates noisy code- switching data. By collecting the outputs in it- erations, we can obtain more pseudo training pairs at each new iteration. In general, using a larger scale of training data may achieve better perfor- mance. However, the experimental results show the bottleneck of the data augmentation method that further increasing augmented training data the 3-th iteration even has negative impact on per- formance. This bottleneck might caused by the quality reduction the generated data after few iterations. Considering the balance between perfor-
The experiment results show that the code- switching ASR error correction task is challeng- ing present. Due to the scarcity of data and the complexity of code-switching text, LLM method is far from achieving satisfactory results in this task. We present that our data augmentation is an effective method to improve the performance of Chinese- Englishh switehwwt ASR error correc-

---

Dataset
Augmentation
As can be seen from the results, data augmen- tation methods affect the performance of the cor- rection model with respect to different error types. Compared with the controllable iterative method, the rule-based method achieves higher scores on missing errors. The correction model benefits from the operations such as adding and deleting tokens defined in the rules that provide sufficient instances with missing errors. Generally, combining control- lable iterative and rule-based data augmentation methods can cover most error types and generate samples with high diversity. It enables the corree- tion model to achieve better performance.
Method
Rules
27.4 19.9
25.4
20.74
SEAME-C
28.5 20.7 26.5
20.45
Rules+CI
29.2
21.3 27.2
20.15
Rules
23.9 17.8 22.4 37.57
ASCEND-C
CI Rules+CI
25.0
19.6
23.7 36.35
25.6
20.8 24.5 35.46
Table 6: Results of the mT5 correction model on datasets when trained with the combination of pseudo data generated by controllable iterative (CI) and ruled- based methods.
mance and computing costs, we select the outputs generated in the third iteration as the augmented data used for English code-switching ASR error correction task.
Conclusion
In this paper, we explore the code-switching ASR error correction task. To enable the research, we first propose two Chinese-Engllins code-switching ASR correction datasets. To address the problem of lacking sufficient training data, we propose the con- trollable iterative data augmentation method that can generate code-switching ASR error correction instances from the monolingual corpus iteratively. Experiment results show that our method achieves the best performance compared with the rule-based augmentation method, back-translation augmenta- tion method, and ChatGPT method. We further find that combining the data generated by both rule- based method and controllable iterative method can further improve the performance of the error cor- rection model. In future work, we will apply our data augmentation method to other code-switching tasks and also test our method in other language pairs.
6.3 Combination Augmented Data
Rule-based and model-based data augmentation methods are two different approaches pseudo training data generation. Therefore, the distribu- tions of errors generated by these two types of data augmentation methods might be different gen- eral. In this section, we try to explore the effect of combining the data generated by rule-based and our methods.
In our experiments, pick the mT5 model that achieves higher evaluation scores in previous exper- iments as the error correction model. The results of using the combined pseudo data are shown in Table 6. From the results, we can find that training mT5 with the combined pseudo data achieves the best scores on our datasets. The use of the data generated by the rule-based method can further improve the performance of the error correction model.
We further analyze the results different error types. With the automatic annotator, we evaluate the performance in four types: redundant (R), miss- ing (M), word selection (S), and word ordering (W). We conduct the experiment our SEAME- C dataset with mT5 as the eπror correction model.
Limitations
In this paper, only conduct experiments on Chinese-English datasets. There are code- switching phenomena that occur in other languages, for example, the Spanish-English and French- Italian code-switching in America. The empiri- cal conclusions we draw in this study might be changed under different language settings. An- other limitation of our study is the lack of man- ual evaluation. Considering the challenge of the code-switching ASR error correction task, manual evaluation can cover the cases that are evaluated inaccurately by automatic metrics. More interest- ing conclusions might be drawn by conducting the manual evaluation.
The results shown in Table 7.
Error Type
M
W
Augmentation Method
Fo.5
Fo.5
Fo
Fo.
8.1
16.3
21.1
9.8
Rules
10.8
25.6
26.1
11.1
CI
17.1
20.5
28.7
14.7
Rules+CI
18.7
21.1
29.1
7.9
Table Results of mTS correction model on SEAME-C with different augmentation methods on different emror types.

---

Acknowledgements
bidirectional transformers for language understand- ing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technolo- gies, Volume (Long and Short Papers), pages 4171- 4186.
This work was supported by National Key R & D Program of China (2021 (021Y9F090))0)) National Science Foundation of China (No. 160339), State Key Laboratory of Media Convergence Pro- duction Technology and Systems and Key Lab- oratory of Science, Technology and Standard in Press Industry (Key Laboratory of Intelligent Press Media Technology). We appreciate the anonymous reviewers for their helpful comments. Xiaojun Wan is the corresponding author.
Suman Dowlagar and Radhika Mamidi. 2023. A code- mixed task-oriented dialog dataset for medical do- main. Computer Speech & Language, 78: 18:101.49.
Timothy Dozat and Christopher D Manning. Deep bi- affine attention for neural dependency parsing.
Chris Dyer, Victor Chahuneau, and Noah A Smith. 2013. A simple, fast, and effective reparameterization of ibm model 2. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 644-648.
References
Heike Adel, Ngoc Thang Vu, Franziska Kraus, Tim Schlippe, Haizhou Li, and Tanja Schultz. 2013. Re- current neural network language modeling for code switching conversational speech. In 2013 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing, pages 8411-8415. IEEE
Luis Fernando D'Haro and Rafael E Banchs. 2016. Au- tomatic correction of outputs by using machine translation. In Interspeech, volume 2016, pages 1469-3473,
Carlo Aliprandi, Cristina Scudellari, Isabella Gallucei, Nicola Piccinini, Matteo Raffaelli, Arantza del Pozo, Aitor Álvarez, Haritz Arzelus, Renato Cassaca, Tiago Luis, et al. 2014. Automatic live subtitling: state of the art, expectations and current trends. In Pro- ceedings of NAB Broadcast Engineering Conference: Papers Advanced Media Technologies. Las Vegas. volume 13.
Roman Grundkiewicz. Marcin Janezys-Dowmant and Kenneth Heafield. 2019. Neural grammatical error correction systems with unsupervised mr-ttrinnng synthetic data. In Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 252-263.
Jinxi Guo, Tara N Sainath, and Ron J Weiss. 2019. A spelling correction model for end-to-end speech recognition. In ICASSP 2019-2019 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5651-5655 IEEE
Peter Auer. 2013. Sode-switching in conversation: Lan- guage, interaction and identity. Routledge.
Christopher Bryant, Mariano Felice, and Edward Briscoe. 2017. Automatic annotation and evalua- tion of error types for grammatical error correction. Association for Computational Linguistics.
Charles Hinson, Hen-Hsen Huang, and Hsin-Hsi Chen. 2020. Heterogeneous recycle generation for chinese grammatical error correction. In Proceedings of the 28th International Conference on Computational Lin- guistics, pages 2191-2201.
Hannan Cao, Wenmian Yang, and Hwee Tou Ng. 2023. Mitigating exposure bias grammatical error cor- rection with data augmentation and reweighting. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Lin- guistics, pages 2115-2127.
Amir Hussein, Shammur Absar Chowdhury, Ahmed Abdelali, Najim Dehak, and Ahmed Ali. 2022. Code- switching text augmentation for multilingual speech processing. arXiv e-prints, pages arXiv-2201.
Rodolfo Corona, Jesse Thomason, and Raymond Mooney. 2017. Improving black-box speech recog- nition using semantic parsing. In Proceedings of the Eighth International Joint Conference on Natu- ral Language Processing (Volume 2: Short Papers), pages 122-127.
Yunqi Gao Jiaju Mei, Yiming Zhu. 1996. Tongyici cilin. In Shanghai Lexicographical Publishing House.
Rebecca Jonson. 2006. Dialogue context- based re- ranking of ast hypotheses. In 2006 IEEE Spoken Lan- guage Technology Workshop, pages 174-177. IEEE.
Daniel Dahlmeier and Hwee Tou Ng. 2012. Better evaluation for grammatical error correction. In Pro- ceedings of the 2012 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 568-572.
Shun Kiyono, Jun Suzuki, Masato Mita, Tomoya Mizu- moto, and Kentaro Inui. 2019. An empirical study of incorporating pscudo data into grammatical error cor- rection. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNNLL pages 1236-1242.
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep

---

Jared Lichtarge, Chris Alberti, Shankar Kumar, Noam Shazeer, Niki Parmar, and Simon Tong. 2019. Cor- pora generation for grammatical erorr correction. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- fional Linguistic:: Human Language Technologies, Volume (Long and Short Papers). pages 3291-3301.
11th International Joint Conference on Natural Lan- guage Processing (Volume 2: Short Papers), pages 702-707.
Djamé Seddah, Farah Essaidi, Amal Fethi, Matthieu Futeral, Benjamin Muller, Pedro Ortiz Suarez, Benoit Sagot, and Abhishek Srivastava. 2020. Building a user- generated content north-african arabizi treebank: Tackling hell, In Proceedings of the 58th annual meeting of the Association for Computational Lin- guistics, pages 1139-1150.
Pierre Lison and Jörg Tiedemann. 2016. Opensub- titles2016: Extracting large parallel corpora from movie and tv subtitles.
Holy Lovenia, Samuel Cahyawijaya, Genta Indra Winata, Peng Xu, Xu Yan, Zihan Liu, Rita Frieske, Tiezheng Yu. Wenliang Dai, Elham J Barezi, et al. 2021. Ascend: A spontaneous chinese-english dataset for code-switching in multi-turn conversation. arXiv preprint arXiv:21
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 86-96.
Aiman Solyman, Marco Zappatore, Wang Zhenyu, Zeinah Mahmoud, Ali Alfatemi, Ashraf Osman Ibrahim, and Lubna Abdelkareim Gabralla. 2023. Optimizing the impact of data augmentation for low- resource grammatical error correction. Journal of King Saud University- Compuute and Information Sci- ences, (5(6):001572.
Dau-Cheng Lyu, Tien-Ping Tan, Eng Siong Chng. and Haizhou Li. 2010. Seame: a mandarin-enge -nglishh code-switching speech corpus in south-east asia. In Eleventh Annual Conference of the International Speech Communication Association.
Anirudh Mani, Shruti Palaskar, Nimshi Venkat Meripo, Sandeep Konam, and Florian Metze. 2020. Asr ror correction and domain adaptation using machine translation. In ICASSP 2020-2020 IEEE Interna- rional Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6344-6348, IEEE.
Hsi- Yao Su. 2001. Code-switching between man- darin and taiwanese in three telephone conversation: The negotiation of interpersonal relationships among bilingual speakers in taiwan. In Proc. of the Sympo- sium about Language and Society.
Yuqing Tang. Chau Tran. Xian Li, Peng-Jen Chen, Na- man Goyal, Vishrav Chaudhary, Jiatao Gu, and An- gela Fan. 2020. Multilingual translation with exten- sible multilingual pretraining and finetuning, arXiv preprint 20Xx:20008000000
Niko Moritz, Takaaki Hori, and Jonathan Lc. 2020. Streaming automatic speech recognition with the transformer model. In ICASSP 2020-2020 IEFE In- ternational Conference on Acoustics, Speech and Sig- nal Processing (ICASSP), pages 6074-607 IEEE.
Jörg Tiedemann and Santhosh Thottingal. 2020. Opus- mt building open translation services for the world. In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation. European Association for Machine Translation.
Kostiantyn Omelianchuk, Vitaliy Atrasevych, Artem Chernodub, and Oleksandr Skurzhanskyi. 2020. error correction: Tag. not rewrite. In Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 163-170.
Deyuan Wang, Tiantian Zhang, Caixia Yuan, and Xiao- jie Wang. 2023. Joint modeling for correction and dialog state tracking. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEF.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, al. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32.
Haoyu Wang, Shuyan Dong, Yue Liu, James Logan, Ashish Kumar Agrawal, and Yang Liu, 2020. Asr er- ror correction with augmented transformer for entity retrieval. In Interspeech, pages 1550-1554.
Adithya Pratapa, Gayatri Bhat, Monojit Choudhury, Sunayana Sitaram, Sandipan Dandapat, and Kalika Bali. 2018. Language modeling for code-mixing: The role of linguistic theory based synthetic data. In Pruceedings of the 56th Annual Meeting of the As- sociation for Computational Linguistics ( (Volume 1: Long Papers), pages 1543 1553.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2020. Transformers: State-of-the-ar natural language processing. In Proceedings of the 2020 con- ference on empirical methods in natural language processing: system demonstrations, pages 38 45.
Sascha Rothe. Jonathan Mallinson, Eric Malmi, Sebas- tian Krause, and Aliaksei Severyn. 2021. A simple recipe for multilingual grammatical error correction. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the
Ziang Xie, Guillaume Genthial, Stanley Xie, Andrew Y Ng, and Dan Jurafsky. 2018 Noising and denois- ing natural language: Diverse backtranslation for

---

grammar correction. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long Papers), pages 619-628.
is computed, the sequence with the lowest score is returned. At last, use a simple merging strategy that merges consecutive sequences of edits of the same type.
Qiantong Xu, Alexei Baevski, and Michael Auli. 2021. Simple and effective zero-shot cross-linguns phoneme recognition. arXiv preprint (iv: 1680.
B Transformations in GECToR
GECToR proposed the token-level transformations for the sequence editing grammatical eπror correc- tion model. Transformations increase the coverage of grammatical error corrections for limited output vocabulary size for the most common types of rors. The work divided transformations into two types: basic transformations and g-transformations. Basic transformations perform the most com- mon token-level edit operations, such as: keep the current token unchanged (tag SKEEPS), delete cur- rent token (tag $DELETES), append new token 1 next to the current token (tag SAPPEND t S) or replace the current token with another token l2 (tag SREPLACE 82 $).
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel, 2021. mt:: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 483-498.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Wein- berger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. In International Confer- ence Learning Representations.
A Processing for Data Construction
A.1
ASR System
Here is the process we use XLSR-53 model to generate code-switching ASR error correction in- stances. Firstly, we build the vocabulary from the transcription in the datasets. Based on the vocab- ulary of 26 English letters, we add all Chinese characters appearing in the transcriptions to the vo- cabulary for the generation of Chinese. Then, fine-tune the XLSR-53 model on the training data from SEAME and ASCEND datasets. We pick the parameters with the best performance in the valida- tion set as our code-switching ASR system for next recognition step. With the fine-tuned XLSR-53 model, we can initially obtain the recognition re- sults required for building the code-switching ASR error correction corpus. As the datasets sponta- neous speeches, the corpus is mainly informal and non-speech sounds often oceur, such as laughing and coughing. Given that, we design some sim- ple rules to polish the corpus, such as removing non-speech sounds and repeated words.
G-transformations perform task-specific opera- tions such change the case of the current to- ken (SCASE$ tags), merge the current token and the next token into single one (SMERGES tags) and split the current token into two new tokens (SSPLITS tags). Moreover, tags from SNOUN NUMBER$ and $VERB FORMş transformations encode grammatical properties for tokens.
C Processing for Sequence Editing Model
C.1 Processing for Transformations
It takes a three-step pre-processing algorithm to convert sentence to sequence of transformations. Firstly, it maps each token from source sentence to sub-sequence of tokens from target sentence. For this purpose, it detects the minimal spans of tokens which define differences between source tokens and target tokens. Thus, such a span is a pair of selected source tokens and corresponding target tokens. To get tags on the token level, it searches for the best-fitting sub-sequence of target tokens by minimizing the modified Levenshtein distance for each source token. Secondly, for each mapping in the list, it finds token-level transformations which convert source token to the target sub-sequence. Finally, leaves only one transformation for each source token. Through the above three steps, we map the tag to each single source token.
A.2 Annotator
The annotator takes three steps: ment, and merging. For tokenization, we use character-level tokenization for Chinese and word- level tokenization for English. Alignment works by computing the alignment score for each pair of tokens in the source and target sentences. In particular, the Cilin thesaurus (Jiaju Mei, 1996) leveraged to give a similarity score between Chi- nese characters. Once the alignment score matrix
tokenization, align-

---

C.2 Processing for Translation
D.3 Pseudo Data Generation Algorithm
Our model is aimed to generate code-switching sen- tences with enrors from correct monolingual sen- tences, but the training pairs in datasets only con- sist of code-switching texts. Considering that, we adopt an open translation tool OPUS-MT (Tiede- mann and Thottingal, 2020) to translate the code- switching ASR transcriptions into monolingual sen- tences. Specifically, we translate the spans of En- glish texts to Chinese and get the Chinese sentences. With regarding the translated monolingual sentence source and the code-switching sentence with er-
We add a confidence bias to control the decod- ing process. In our experiments, the confidence bias co is to add -1 -1.0 to the generation probability of all transformations except $KEEPS, STRANS- LATIONS and other cross- cooss-lingual transformations The confidence bias e1 set to keep the original generation probability of all transformations. The threshold set to 0.85.
Method
Prompt Template
"role": "system", "content": "You are a Chinese- Chinese-English code-switching ASR error correction tool that can identify and correct envors in the text. Especially, code-swirching is a linguistic phenomenon where speaker alternates between two or more languages language varieties within single conversa- tion or within single sentence." " "role": "user", "content": "Please identify and correct any errors in the following sen- tence indicated by <input> ERROR </input> lag, you need to comprehend the sentence as whole before gradually identifying and correcting any errors while keeping the orig- inal sentence structure unchanged much possible. Remember format your rected output results with the tug <output> Your Corrected Version </output>. Please start: <input> Input Sentence </input>:"
rors from error correction datasets as target, we train the sequence editing model.
D Implementation Details
In our work, we adopt PyTorch (Paszke et al., 2019) framework and HuggingFace (Wolf et al., 2020) tool to implement all deep neural models. Each model is trained on four NVIDIA A40 GPUs. The details of each model are described as follows.
Zero-shot
D.1 ASR Error Correction Model
Considering that mT5 and mBART have achieved strong performance in many multilingual tasks, adopt these pre-trained models correct ASR er- rors. In the experiments, we use the base version of mT5 and fine-tune by using the Adam optimiza- tion method with learning rate 5c-4, weight decay parameter 0.01, dropout rate 0.1, and batch size 24. Besides, we use the large version of mBART, and fine-tune it by using the Adam optimization method with learning rate 2e-5, dropout rate 0.1, and batch size 16. Beam search with beam size of 5 is used for decoding.
"role": "system", "content": "You are a Chinese-English code-switching ASR error correction tool that can identify and correct errors in the text. Especially, code-switching is a linguistic phenomenon where a speaker alternates between two or more languages or language varieties within a single conversa- tion or even within a single sentence." "role": "user", "content": "Please identify and correct any errors in the following sen- tence indicated hy <input> ERROR </input> tag, you need to comprehend the sentence as whole before gradually identifying and correcting any errors while keeping the orig- inal sentence structure unchanged as much as possible Here are some in-context exam- ples:
Few-shot
(1). <input> SRC-1 </input>: <output>
D.2 Sequence Editing Model
TGT-1 </output>:
(2), <input> SRC-1 </input>:
<output>
TGT-1 </output>;
The sequence editing model is an encoder made up of Transformer architecture stacked with two linear layers with softmax layers. We adopt the base version of m- BERT to initialize the encoder. Our model consists of 12 Transformer layers and two linear layers. We train the model with learning rate le-5, weight decay parameter 0.01, dropout rate 0, and batch size 32. The size of the vocabulary is 12229. It consists of 4971 basic transformations, 29 token-indenn -iken-independeett g- g-transformations and 7229 cross- lingual transformations.
(n), <input> SRC-n </input>: <output>
TGT--n </output>;
Please feel free to refer to these examples.
Remember format your corrected output results with the tag <output> Your Corrected Version </output>, Please <input> In- put Sentence </input>:"
Table 8: Zero-shot and few-shot prompt template for Chinese-Eng sode-switching ASR error correction.

---

E
Rule-based Method
The details of the five rules adopted our proposed rule-based method are introduced in the following. Delete. Randomly delete a token with proba- bility.
Add. Firstly, randomly select word or Chinese character from the vocabulary of the ASR datasets, and then add the selected word or character random position.
Replace. Randomly replace token with a word or Chinese character from the vocabulary of the ASR datasets. In particular, if the selected token is a Chinese character, we select the replacement from the homophones with probability.
Shuffle. Shuffle the tokens by adding normal distribution bias the positions of the tokens.
Spell Error. Randomly apply spell error to an English word. We randomly perturb letters with operations, i.e. substitution, deletion, insertion or transposition letters.
F Prompt Template
Zero-shot and few-shot prompt templates for Chinese-English code-switching ASR error correc- tion are shown Table 8.
Additional Experiment Results
Results of the mT5 correction model SEAME-C, when trained with outputs collected from different iterations of our method, are shown in Table
Iteration
Augmented
P
R1
Fo.s MER
Data Size
Iteration I
188
28.0
20.3
26.1
20.54
Iteration 2
2.6M
28.2
20.6
26.3
20.49
Iteration 3
3.3M
28.5
20.7
26.5
20.45
Iteration 4 3.6M
28.3
20.8
26.4
20.51
Iteration 5
3.8M
28.3
21.1
26.5
20.40
Table 9:
Results of the mT5 correction model on SEAME-C when trained with outputs collected from different iterations of our method.


Reviews:
[{"review_id": "UjHbI6iEJM", "reviewer": "Reviewer_6CD8", "summary": "", "questions": "- Please report the WER scores on the ASR outputs before error correction is applied.\n- The translation transformation replaces the current token with the most common translation in a Chinese-English dictionary (line 300). Is the (subword?) tokenization of the sequence editing model compatible with the dictionary?\n- Will the generated synthetic training data also be published?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your interest in our work. We appreciate your valuable comments and suggestions for improvement. In response to your questions, we would like to address each of the concerns raised.\n\n### Responses for Questions\n\n**Q1**: A discussion of the error statistics across different ASR systems would be missing.\n\n**A1**: XLSR-53 is the multilingual version of wav2vec 2.0 model pre-trained on 53 languages, which shows strong performance and robustness in the code-switching ASR task. It has been widely employed as a strong baseline in previous works [1,2]. \n\nBesides, we evaluate the performance of the XLSR-53 and another ASR system ETEH [3]  before constructing the datasets. The results are shown as follows. \n\n| Dataset | Method | MER |\n| --- | --- | --- |\n| SEAME | XLSR-53 | 24.02 |\n| SEAME | ETEH |31.62 |\n| ASCEND | XLSR-53 | 42.38|\n| ASCEND| ETEH |44.85 |\n\nIn particular, we replace the term WER with MER (Mixture Error Rate) which is a more accurate enunciate. It can be observed that the performance of the XLSR-53 system on both datasets is better than the ETEH system. Considering the above reasons, we adopt XLSR-53 ASR system for constructing the datasets, to ensure their quality and generality. For sure, we acknowledge the utilization of a single system inherently carries limitations. We will make sure to supplement our experiment with other typical ASR systems (e.g. Whisper, ESPNet) in the revised version of the paper.\n\n**Q2**: The results of the ASCEND paper are better than the results of baseline ASR system in our paper.\n\n**A2**: We try to reproduce the method in the ASCEND paper, but we do not achieve the same level of performance. As mentioned in A1, the XLSR-53 method we select is strong and representative. So we adopt it to build the datasets.\n\n**Q3**: The WER scores on the ASR outputs before error correction is applied.\n\n**A3**: As mentioned in A1, the table shows the scores on the ASR outputs before error correction is applied. The scores for SEAME and ASCEND are 24.02 and 42.38 respectively.\n\n**Q4**: Is the tokenization of the sequence editing model compatible with the dictionary?\n\n**A4**: To process the transformation at the token-level, we take the first subword per token\nfrom the encoder’s representation to subsequent linear layers. With the process, the tokenized source sentence can be compatible with the dictionary at the same token level.\n\n**Q5**: Will the generated synthetic training data also be published?\n\n**A5**: Of course, we will release the generated synthetic training data along with the released code and data. We hope that the released resources can assist researchers in further investigating this task.\n\nThank you again for your careful and valuable comments and guidance. \n\n### References\n\n[1] Ascend: A spontaneous chinese-english dataset for code-switching in multi-turn conversation. Lovenia H, Cahyawijaya S, Winata G I, et al. (2021).\n\n[2]TEVR: Improving Speech Recognition by Token Entropy Variance Reduction. Krabbenhöft, H. N., & Barth, E. (2022). \n\n[3]Eteh: Unified attention-based end-to-end asr and kws architecture. Cheng G, Miao H, Yang R, et al. (2022).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed CI data augmentation process is well motivated by the error analysis of the datasets they introduced, and carefully designed targeting the ASR error correction goal.\n- They provide comprehensive details on dataset creation and pseudo data generation.\n- They did not only conduct the evaluation in M^2 / WER, but also analyzed error types and the influence of iteration numbers, and even included the attempts with LLM prompting, which supports deeper understanding of the task.", "reasons_to_reject": "A discussion of the error statistics across different ASR systems would be missing. Their ASR error analysis relies on the outputs from a particular ASR system only. Moreover, their ASR model choice might be questionable especially with the focus on the dialogue in Chinese-English in particular. ASCEND paper [1] reports that the wav2vec model achieved 35.30 WER, while they report 40.31 WER for the baseline error correction model without additional augmented data.\n\n[1] Lovenia et. al., ASCEND: A Spontaneous Chinese-English Dataset for Code-switching in Multi-turn Conversation, LREC 2022.", "questions_to_authors": "- Please report the WER scores on the ASR outputs before error correction is applied.\n- The translation transformation replaces the current token with the most common translation in a Chinese-English dictionary (line 300). Is the (subword?) tokenization of the sequence editing model compatible with the dictionary?\n- Will the generated synthetic training data also be published?", "paper_topic_and_main_contributions": "The paper presents the ASR error correction task for Chinese-English code-switching dialogues. The authors created new datasets with ASR outputs and automatic error type annotations from existing ASR corpora SEAME and ASCEND. They further introduced a controllable iterative (CI) data augmentation method to amplify the training data for error correction, and showed convincing improvements over other data augmentation techniques.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- line 976: [...] are shown in **Table** 8.\n- line 979: [...] are shown in **Table** 9.\n- Some transformation operation names are surrounded by **$** (line 294, line 365), others don't have the ending **$**.\n- To demonstrate how well the generated pseudo data can mimic the true code-switching target and source with ASR errors, it would be nice to have some real examples found in the corpus. For instance, is there any comparable counterpart of the sample generation in Figure 1 (\"我在 Singapore make 很多 good friends\") that represents a typical code-switching in an actual dialogue?"}}, {"review_id": "IENmR63ohP", "reviewer": "Reviewer_BZT9", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our paper and providing us with valuable comments. We appreciate your time and effort in reviewing our work. In response to your comments, we would like to address the concerns raised:\n\n**Q1**: Clarity on adding assumptions across the board.\n\n**A1**: In this paper, we present a data augmentation approach to assist in the code-switching ASR error correction task. Our method is based on the assumption that a greater amount of data can achieve better performance. The efficacy of data augmentation methods has been widely demonstrated in previous works [1, 2, 3]. In this work, we further substantiate it through experiments.\n\n**Q2**: Point of view on generalizability.\n\n**A2**: The lack of data is a common challenge for the research in cross-lingual and low-resource languages. Our proposed CI data augmentation method presents an approach to address the data scarcity issue for Chinese-English code-switching ASR error correction tasks. We anticipate that our method can be extended to relevant code-switching tasks involving other languages in the future. Furthermore, our data augmentation methodology provides insights to researchers studying other tasks in cross-lingual and low-resource languages.\n\n**Q3**: Manual annotation and evaluation by humans can further improve the system.\n\n**A3**: Thanks for your suggestions. We will incorporate manual annotation and human evaluation in our work to further improve the quality of the datasets and the evaluation results. \n\n**Q4**: Any kind of ablation study can help in understanding the modularized aspect of the system and can open up further research areas.\n\n**A4**: In section 6, we present the related contents, hoping to address your concerns. \n\nIn section 6.2, we try to explore the influence of the iteration number on the generation of augmented data. In general, using a larger scale of training data may achieve better performance. However, the experimental results show the bottleneck of the data augmentation method.\n\nData plays a crucial role in our proposed method. In section 6.3, we try to explore the effect\nof combining the data generated by different methods. From the results, we can find that the use of the data generated by the rule-based method can further improve the performance of the error correction model.\n\n**Q5**: Details/explanation around the configuration/hyper-parameters of the rule-based and machine learning-based approaches used can help with more clarity.\n\n**A5**: In Appendix E, we provide a detailed explanation of the rule-based approach. In particular, the error probabilities for the operations \"delete,\" \"add,\" \"replace,\" \"shuffle,\" and \"spell error\" are 0.1, 0.1, 0.5, 0.1, and 0.2, respectively. As for the machine learning-based method, we adopt a base mT5 model and fine-tune it by using the Adam optimization method with learning rate 2e-4, weight decay parameter 0.01, dropout rate 0.1, and batch size 24. Further details of the configuration and hyper-parameters can be referred to the released code.\n\n**Q6**: Paper can benefit further from having detailed analysis of the computational complexity or efficiency of the proposed method.\n\n**A6**: \nIn this work, we proposed a data augmentation method to improve the performance of Chinese-English code-switching ASR error correction model by generating synthetic training data. The method only affects the training data of the error correction model and does not impact the speed of the inference process.\n\nThank you again for your careful and valuable comments and guidance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tWell structured, detailed and crisp paper.\n2.\tProposes a new corpus of dataset for the task.\n3.\tProposes a new approach to improving ASR error correction in code-switching settings. \n4.\tProposes a Controllable Iterative (CI)  data augmentation method that combines rule-based and machine learning-based approaches. \n5.\tProvides detailed descriptions of the datasets and experimental setup.\n", "reasons_to_reject": "1.\tClarity on adding assumptions across the board.\n2.\tPoint of view on Generalizability. \n3.\tManual annotation and evaluation by humans can further improve the system. \n4.\tAny kind of ablation study can help understanding the modularized aspect of the system and can open up further research areas.\n5.\tDetails/explanation around the configuration/hyperparameters of the rule-based and machine learning-based approaches used can help with more clarity.\n6.\tPaper can benefit further from having detailed analysis of the computational complexity or efficiency of the proposed method.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "Authors propose new approach to improving the performance of automatic speech recognition (ASR) error correction systems in code-switching settings (more than one language used in the same conversation). For the task they propose and develop 2 Chinese-English code-switching datasets. They then propose a Controllable Iterative (CI) data augmentation method that combines rule-based and machine learning-based approaches to generate synthetic training data for ASR error correction models. They evaluate the proposed method on the dataset developed for the task. They end with possibility of new research areas in future.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ekphuCkmYa", "reviewer": "Reviewer_XFMs", "summary": "", "questions": "A. Is WER the right metric to evaluate for English-Chinese code-switching? I believe past work (e.g. https://arxiv.org/pdf/1810.13091.pdf ) uses Mixed Error Rate (MER) which computes word-level errors for English and character-level errors for Chinese. Adding MERs would make the results more convincing.\n\nB. How do you finetune the mBERT model on your predefined transformations; if you have predefined deterministic transformations (APPEND, REPLACE, etc.) what is the purpose of using an mBERT model to generate these same transformations?\n\nC. What is the input and output format of your mBERT model? What exactly is in the 12229 sized output vocabulary?\n\nD. What method do you use to do controllable decoding from the mBERT model?\n\nE. What dataset do you use to train the mBERT sequence editing model? Appendix C explains that you use OPUS-MT to translate code-switched data to monolingual data, and use this to train; where does this code-switched data come from?\n\nF. On lines 160-161 you mention that you remove monolingual sentences from the val/test sets. Why not do the same for the train sets as well? Relatedly, are the code-mixing statistics in Table 3 on the training set (since you remove monolingual sentences from Val/test, the proportion of Chinese/English sentences should be 0.00% for those)? Please clarify in the text.\n\nG. “This type of error is caused by mistakes ... with similar pronunciation” on lines 243-245 ; was this verified with an experiment? Word selection errors could be due to replacements with unrelated words as well. It would be good to support the statement with an experiment.\n\nH. Can you motivate the choice of XLSR-53 as the base ASR model as well as why you opted to finetune it on the code-switching datasets in order to generate synthetic ASR hypotheses? Why not use a) models from previous papers trained explicitly for code-switching e.g. the SEAME model from ESPNet or b) try zero-shot non-finetuned models like the base XLSR-53 model or Whisper?\n\nI. What ASR model was used in Table 5. What is the WER of this ASR model without any error-correction applied? (Adding that WER to Table 5 will be helpful).\n\nJ. The inability of an LLM to 3-shot adapt to a novel task does not imply the task is fundamentally challenging (Lines 499-504); is there more evidence for your statement?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your careful and valuable reviews. We appreciate your time and effort in reviewing our work. The improvement suggestions you provide are beneficial, and we will modify our paper based on them. Besides, we provide detailed responses to your questions in the following sections, hoping to address your concerns.\n\n### Responses for Reasons to Reject\n\n**Q1**: Ambiguity of the data used to train baselines.\n\n**A1**: Certainly, data is the key to the methods. In our experiment, the rule-based method, back-translation method, and the proposed CI method adopt the same 5 million sentences sampled from the parallel seed corpus. In particular, the CI method and the back-translation method use the similarity discriminator to filter sentence pairs with low similarity, resulting in synthetic sentences of fewer than 5 million. Since the sources used for data augmentation are identical, the comparison of the methods is entirely fair and reliable.\n\n**Q2**: The limitations of using the single ASR system XLSR-53.\n\n**A2**: XLSR-53 is the multilingual version of wav2vec 2.0 model pre-trained on 53 languages, which shows strong performance and robustness in the code-switching ASR task. It has been widely employed as a strong baseline in previous works [1,2]. \n\nBesides, we evaluate the performance of the XLSR-53 and another ASR system ETEH [3]  before constructing the datasets. The results are shown as follows. \n\n| Dataset | Method | MER |\n| --- | --- | --- |\n| SEAME | XLSR-53 | 24.02 |\n| SEAME | ETEH |31.62 |\n| ASCEND | XLSR-53 | 42.38|\n| ASCEND| ETEH |44.85 |\n\nIn particular, we replace the term WER with MER (Mixture Error Rate) which is a more accurate enunciate. It can be observed that the performance of the XLSR-53 system on both datasets is better than the ETEH system. Considering the above reasons, we adopt XLSR-53 ASR system for constructing the datasets, to ensure their quality and generality. For sure, we acknowledge the utilization of a single system inherently carries limitations. We will make sure to supplement our experiment with other typical ASR systems (e.g. Whisper, ESPNet) in the revised version of the paper.\n\n**Q3**: Many important details of the method used are not explained properly.\n\n**A3**: Due to the limitation of pages, we select to elaborate on the essential aspects of the main text. In the revised version, we will carefully consider your suggestions and move some content from the appendix to the main body. \n\n### Responses for Questions\n\nThe questions you raised are valuable and helpful to the refinement of our paper. Due to the limitation of pages, some details are not explained properly. In this section, we provide detailed responses to your questions, hoping to complete the missing pieces of the description and address your concerns.\n\n**Q1**: Is WER the right metric to evaluate for English-Chinese code-switching?\n\n**A1**: The question you raise is highly helpful. We have referred to preview work [4] that uses WER as the metric, which actually calculates the error rate of Chinese characters and English words as same as the MER. After reading the paper you mentioned in question [5], we searched for related works [6] and found that using the term Mixture Error Rate (MER) is more common. In the revised version, we will replace the imprecise term WER with MER.\n\n**Q2**: The fine-tuning process of the mBERT model. \n\n**A2**: As mentioned in section 4.1.3, we train the sequence editing model by the sequences of transformations that are converted from the source sentences. The fine-tuning process of mBERT model is conducted as a component of the overall training process. It encodes the source sentence and outputs feature representations to the following linear layers.\n\n**Q3**: The purpose of using the mBERT model.\n\n**A3**: We refer to the previous GECToR method, which adopts pre-trained transformers as the encoders and conducts experiments with several pre-trained settings. Their experiments show that utilizing pre-trained models can accelerate the convergence speed of training and improve the performance of method. \n\nConsidering that, we apply the pre-trained models to our method. We conduct the experiments with several multilingual pre-trained models (e.g. mBERT, mBART, mT5) and find that the mBERT model achieves the best results. So we select it for our method to accelerate the convergence process in training and improve the robustness and performance of our method. \n\n**Q4**: What is the input and output format of your mBERT model?\n\n**A4**: As mentioned in A2, the inputs of the mBERT are the source sentences and the outputs are the feature presentations with dim matching the hidden size of mBERT. In particular, the model adopts WordPiece tokenizer to process the inputs.\n\n**Q5**: What exactly is in the 12229 sized output vocabulary?\n\n**A5**: As mentioned in appendix D.2, the vocabulary consists of 4971 basic transformations, \n29 token-independent g-transformations and 7229 cross-lingual transformations. The basic transformations perform the most common token-level edit operations, such as: KEEP, DELETE, APPEND. The g-transformations perform task-specific operations such as: CASE,  MERGE. The cross-lingual transformations perform the translation operation. The Appendix B shows the details. \n\n**Q6**: What method do you use to do controllable decoding from the mBERT model?\n\n**A6**: As mentioned in section 4.1.2, we control the decoding process by adjusting the confidence bias which is a parameter applied to change the generation probability of transformations. In our method, we get the generation probability of each transformation by the linear layers with softmax layers. Then we apply the confidence bias which is the vector composed of float numbers to the generation probability. For example, adding a positive number to KEEP can increase the probability of generating the KEEP. By adjusting the confidence bias, the decoding process of sequence editing model is under control.\n\n**Q7**: The dataset used to train the mBERT sequence editing model.\n\n**A7**: As mentioned in appendix C, we train the mBERT sequence editing model by the data that is translated from code-switching sentences. In particular, we select the code-switching sentences from the training sets of ASCEND-C and SEAME-C for translation. \n\n**Q8**: Why remove monolingual sentences for the val/test sets?\n\n**A8**: The datasets we proposed focus on errors in Chinese-English code-switching sentences. However, the original datasets contain monolingual sentences. So, we remove the monolingual sentences for the val/test sets to ensure that the evaluation is performed in code-switching sentences. Besides, in order to provide more sufficient training data, we make these monolingual data available as well. Users have the option to decide whether to adopt these monolingual sentences to their training process. \n\nIn addition, the proportions of Chinese/English sentences refer to the ratio across the entire dataset, including the training, validation, testing sets. Therefore, the proportion is non-zero and accurate.\n\n**Q9**: The verification of the statements that the word selection errors are caused by mistakes in the recognition of words or phrases with similar pronunciation.\n\n**A9**: Thanks for pointing out the issues. By observing the cases, we find that the majority of errors are caused by mistakes in the recognition of words or phrases with similar pronunciation. \nHowever, the concerns you raised that word selection errors could be due to replacements with unrelated words do exist. In the revised version, we will revise the statement as “This type of error is mainly caused by mistakes in the recognition of words or phrases with similar pronunciation.”\n\n**Q10**: The choice of ASR system.\n\n**A10**: As mentioned in A2 of responses for reasons to reject part, XLSR-53 is a method with strong performance and robustness in the code-switching ASR task, and it has been widely employed as a strong baseline in previous works. So, we adopt it to build the datasets with quality. We conduct tests on XLSR-53 and ETEH models in previous experiments. We will make sure to supplement our experiment with other typical ASR systems in the revised version of the paper. As for the use of zero-shot ASR model, they achieve poor performance (WER > 70%) in our experiments, so we do not consider them.\n\n**Q11**: The details of the results in Table 5.\n\n**A11**: As mentioned in Section 3, the results are on the dataset constructed by XLSR-53 model. So the ASR model is XLSR-53. The WERs of it without any error-correction applied are shown in A2 of responses for reasons to reject part.\n\n**Q12**: More evidence for the statement that the task is fundamentally challenging.\n\n**A12**: GPT 3.5 has achieved great success in many NLP tasks. However, its performance in the Chinese-English code-switching ASR error correction task is far from satisfactory. Besides, the pre-trained models, such as mT5, mBART, that show effectiveness in error correction tasks in the previous works do not achieve a great performance on this task. So, we state that the Chinese-English code-switching ASR error correction task is quite challenging under the existing techniques. \n\nIndeed, the issue you pointed out has some rationality. We will make sure to supplement further experiments with various LLMs to provide substantiating evidence.\n\nThank you again for your careful and valuable comments and guidance. \n\n### References\n\n[1] Ascend: A spontaneous chinese-english dataset for code-switching in multi-turn conversation. Lovenia H, Cahyawijaya S, Winata G I, et al. (2021).\n\n[2] TEVR: Improving Speech Recognition by Token Entropy Variance Reduction. Krabbenhöft, H. N., & Barth, E. (2022). \n\n[3] Eteh: Unified attention-based end-to-end asr and kws architecture. Cheng G, Miao H, Yang R, et al. (2022).\n\n[4] Code-Switching in Automatic Speech Recognition: The Issues and Future Directions. Mustafa M B, Yusoof M A, Khalaf H K, et al. (2022).\n\n[5] Towards end-to-end code-switching speech recognition. Luo N, Jiang D, Zhao S, et al. (2018).\n \n[6] Summary On The ISCSLP 2022 Chinese-English Code-Switching ASR Challenge.Deng S, Li C, Bai J, et al. (2022).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper is a very good investigation into the task of ASR error correction in the code-switched setting which has been previously unexplored and hence is a very novel investigation.\n2. The authors perform insightful data analysis in Section 3.2 and use it to guide their approach to generate synthetic ASR error correction data.\n3. The choice of baselines is good and covers a variety of plausible approaches for data augmentation.\n4. I especially liked Sections 6.2 and 6.3 that provide very interesting insights on how many iterations matter and what types of errors are fixed by what kinds of data augmentation methods.\n5. The paper says that the code (new approach) and data (2 new datasets) will be released, which is quite beneficial to the community.", "reasons_to_reject": "1. The data used to train baselines compared against in Section 5.3 are not properly explained; although the authors mention that they control for dataset size, it is hard to verify whether they are fair comparisons or not.\n2. The training and evaluation datasets in this work (SEAME-C and ASCEND-C) are created by finetuning XLSR-53 on SEAME and ASCEND. All ASR correction models in this work are trained and evaluated on SEAME-C and ASCEND-C, and based on my reading, the XLSR-53 ASR model is also used in the evaluation phase to report M^2 and WER metrics in Table 5. As such, the setup seems to have been fit closely to the XLSR-53 ASR model, SEAME and ASCEND. It is not clear from the paper whether the results are generalizable outside of this setting; e.g., can the ASR correction model trained by the authors correct ASR errors from models other than XLSR-53 (or is SEAME-C and ASCEND-C too sensitive to XLSR-53 errors) , and on evaluation datasets other than SEAME-C and ASCEND-C. This may limit the applicability of the proposed datasets and method. For example, it will be good if the authors can add more rows to Table 5 applying their correction model on other code-switching ASR models apart from XLSR-53.\n3. Many important details of the method used (Data Construction in Sec 3.1 and the Data Augmentation method in Section 4) are not explained properly, which currently affects the quality and reproducibility of the paper. I suggest\n    1. moving some info (e.g. the data used to finetune XLSR-53, a summary of the transformations used, etc.) from the Appendix to the main text.\n    2. adding answers to my questions (in ‘Questions for the Authors’) to complete the missing pieces from the description. Answering these questions is likely to increase my Soundness score.", "questions_to_authors": "A. Is WER the right metric to evaluate for English-Chinese code-switching? I believe past work (e.g. https://arxiv.org/pdf/1810.13091.pdf ) uses Mixed Error Rate (MER) which computes word-level errors for English and character-level errors for Chinese. Adding MERs would make the results more convincing.\n\nB. How do you finetune the mBERT model on your predefined transformations; if you have predefined deterministic transformations (APPEND, REPLACE, etc.) what is the purpose of using an mBERT model to generate these same transformations?\n\nC. What is the input and output format of your mBERT model? What exactly is in the 12229 sized output vocabulary?\n\nD. What method do you use to do controllable decoding from the mBERT model?\n\nE. What dataset do you use to train the mBERT sequence editing model? Appendix C explains that you use OPUS-MT to translate code-switched data to monolingual data, and use this to train; where does this code-switched data come from?\n\nF. On lines 160-161 you mention that you remove monolingual sentences from the val/test sets. Why not do the same for the train sets as well? Relatedly, are the code-mixing statistics in Table 3 on the training set (since you remove monolingual sentences from Val/test, the proportion of Chinese/English sentences should be 0.00% for those)? Please clarify in the text.\n\nG. “This type of error is caused by mistakes ... with similar pronunciation” on lines 243-245 ; was this verified with an experiment? Word selection errors could be due to replacements with unrelated words as well. It would be good to support the statement with an experiment.\n\nH. Can you motivate the choice of XLSR-53 as the base ASR model as well as why you opted to finetune it on the code-switching datasets in order to generate synthetic ASR hypotheses? Why not use a) models from previous papers trained explicitly for code-switching e.g. the SEAME model from ESPNet or b) try zero-shot non-finetuned models like the base XLSR-53 model or Whisper?\n\nI. What ASR model was used in Table 5. What is the WER of this ASR model without any error-correction applied? (Adding that WER to Table 5 will be helpful).\n\nJ. The inability of an LLM to 3-shot adapt to a novel task does not imply the task is fundamentally challenging (Lines 499-504); is there more evidence for your statement?", "paper_topic_and_main_contributions": "The paper is about datasets and approaches for code-switching ASR error correction. It investigates ASR error correction in the context of code-switching for the first time. It creates a new set of training and evaluation datasets (SEAME-C and ASCEND-C) using pairs of ground truth transcripts from SEAME/ASCEND (code-switching ASR dataset) and ASR hypotheses from a finetuned multilingual ASR model (XLSR-53). It also proposes a new data augmentation strategy to cheaply generate synthetic but noisy code-switching ASR error correction examples from large monolingual datasets and shows some improvement over baselines on the newly proposed datasets.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Line 117: swinging -> switching\n- Add citations to lines 034 and 417\n- Add word-level English translations to Figure 1 to help non-Chinese-speaking readers to understand the code-mixing better\n- Specify the version of ChatGPT / gpt-3.5-turbo used (e.g. 0314 or the live version at chat.openai.com) and the hyperparams used (temperature etc.) if any.\n- There is no description about the precision, recall and F1 evaluation metrics (from the MaxMatch scorer) in Table 5; adding a description would be important."}}]

Author rebuttal:

Meta-review: This is an interesting work that addresses for the first time ASR error correction in code-switched data (Chinese-English). The paper proposes data augmentation strategies for training ASR error correction models. The ASR error correction datasets are first created using pairs of outputs from an ASR model (XLSR-53) and ground truth transcripts, and automatically assigning error types. Then the paper proposes an approach to data augmentation via a set of text transformations.

The paper should include more detail on the dataset construction and the data augmentation approach. As the authors use a single ASR model to construct the datasets and to analyze the resulting errors, it is unclear how the choice of the model would affect to constructed dataset and the results on the error correction task. The paper can be strengthened by experiments on a different ASR model.
The related work section should  include research on the more general error correction task and error correction on monolingual ASR (including specific examples would help the reader). How can insights from these tasks benefit the ASR error correction? Can you use existing data augmentation techniques from the general error correction task for ASR error correction?
Decision: Accept-Findings