Optimized Tokenization for Transcribed Error Correction
Authors: Tomer Wullach
Shlomo Chazan
Venue: EMNLP 2023 Main (2023)
Keywords: NLP
Speech Recognition
Error Correction

Abstract: The challenges facing speech recognition systems, such as variations in pronunciations, adverse audio conditions, and the scarcity of labeled data, emphasize the necessity for a post-processing step that corrects recurring errors. Previous research has shown the advantages of employing dedicated error correction models, yet training such models requires large amounts of labeled data which is not easily obtained. To overcome this limitation, synthetic transcribed-like data is often utilized, however, bridging the distribution gap between transcribed errors and synthetic noise is not trivial.
In this paper, we demonstrate that the performance of correction models can be significantly increased by training solely using synthetic data. 
Specifically, we empirically show that: (1) synthetic data generated using the error distribution derived from a set of transcribed data outperforms the common approach of applying random perturbations; (2) applying language-specific adjustments to the vocabulary of a BPE tokenizer strike a balance between adapting to unseen distributions and retaining knowledge of transcribed errors.
We showcase the benefits of these key observations, and evaluate our approach using multiple languages, speech recognition systems and prominent speech recognition datasets.

Optimized Tokenization for Transcribed Error Correction
Tomer Wullach OriginAl
Shlomo E. Chazan OriginAl
tomerw@originai.co
shlomi@originai.. co
Abstract
20
10
The challenges facing speech recognition sys- tems, such as variations in pronunciations, ad- verse audio conditions, and the scarcity of la- beled data, emphasize the necessity for post- processing step that corrects recurring errors. Previous research has shown the advantages of employing dedicated eтrоr correction models, yet training such models requires large amounts of labeled data which is not easily obtained. To overcome this limitation, synthetic transcribed- like data is often utilized, however, bridging distribution gap between transcribed errors and synthetic noise is not trivial. In this paper, we demonstrate that the performance of correction models can be significantly increased by train- ing solely using synthetic data. Specifically, we empirically show that: (1) synthetic data gener- ated using the error distribution derived from set transcribed data outperforms common approach of applying random perturbations; (2) applying language-specifts adjustments to the vocabulary BPE tokenizer strike balance between adapting to unseen distributions and retaining knowledge of transcribed errors. We showcase the benefits of these key observations, and evaluate our approach using multiple lan- guages, speech recognition systems and promi- nent speech recognition datasets.
14
Serman
rench
Max. Token Length
14-
12
10
Dutch Fortuguese
Poiish
taaiaan
Spanish
12.0
BPE Vocabulary S (x10P)
Figure 1: The best performing BPE hyperparameters for seven European correction models. Phonetic (isolating, low morpheme-wool wwor ratio) languages tend to use short tokens with small vocabularies. Morphological-riee (bundles several morphemes in each word, inflectional patterns) requires longer tokens and larger vocabularics memorize inflected words and complex structures.
To this end, researchers suggested text-to-text error correction methods specifically designed to correct transcribed errors (Mani et al., 2020; Leng et al., 2021a; Hrinchuk et al., 2020). These transformer-baset methods typically requires a massive and diverse training data (i.e., transcribed audio segments and their corresponding human- authored transcriptions) which is rarely reach.
1 Introduction
Speech Recognition systems has undergone re- markable progress the last several years, largely due to the growing adoption of the self-supervision
To bridge that gap, synthetic data is frequently incorporated to ensure sufficient amount of train- ing examples. A common approach is to introduce random perturbations or use predefined confusion sets (Ji et al., 2021; Karpukhin et al., 2019; Be- linkov and Bisk. 2017). Nevertheless, this approach might result with perturbations that unrelated to transcribed errors, thereby failing to accurately replicate the distribution of such errors.
paradigm. (Baevski et al., 2020; Hsu et al., 2021; Radford et al., 2022; Balestriero et al., 2023). These systems can attain low error rates given suf- ficient model capacity and quality labeled data, yet they susceptible errors caused by dio and speech factors, such as speakers pronun- ciation, background noise, recording quality and orthographic errors which occurs mostly in non- phonetic languages.
A promising solution would involve generating a set of synthetic examples that follows the error distribution of transcribed data, and utilize it to

---

train correction model. Such model needs to find balance between memorizing the transcribe-like patterns by fitting the synthetic training data, and generalizing to unseen distributions.
recognition models, despite being trained ex- clusively using synthetic data.
3. We provide thorough experiments and abla- tion, demonstrating the trade-off's in adjusting language-specific BPE vocabulary sizes.
Transformer-bane based correction models mostly use subword tokenizers such as BPE (Sennrich et al., 2015), which merges tokens based learned set of rules proportional the tokenizer's vocabulary size. Prior research showed that small vocabularies consisting of short tokens, such as individual char- acters, may assist with spelling errors, natural noise and generalize to unseen distributions (Ma et al., 2020; Kann et al., 2020). However, it may also lead to degraded performance in some settings (Gowda and May, 2020; Clark et al., 2022).
Related Work
Transcribed Errors Correction. Correction methods are commonly applied as a post- processing step, aiming to map transcribed errors to their correct form. Recent works demonstrated the merits of utilizing transformer- based architectures, stretching from auto-regressive (Mani al., 2020; Shen et al., 2022) to non auto-regressive (Leng et al.. 2021b,a approaches.
Alternatively, it has been shown that large vo- cabularies can affect the model's memorization ca- pabilities (Kharitonov et al., 2021), I), which can be significant for recurring transcribed error patterns.
One of the concerns in using these approaches is the alignment problem, which includes determining the correspondence between source tokens and tar- get tokens. Various methods have been explored address this issue. For example, some approaches utilize token duration predictor (Leng et al., 202 lb,a), while others leverage the speech recog- nition system's Connectionist Temporal Classifica- tion (CTC) (Graves et al.. 2006) predictions (Leng et al., 2022). Moreover, some studies focus on de- tecting transcribed errors, allowing the correction model to copy tokens identified as correct (Leng et al., 2022; Gekhman et al., 2022).
To address these shortcomings, we generate ex- amples that mimic the error distribution of tran- scribed texts by incorporating information derived from transcribed errors. We then balance between memorizing erroneous patterns and generalizing to unseen distributions using lunguage-speciff adjust- ments to the vocabulary of BPE tokenizers. These adjustments limit the vocabulary size and the maxi- mum length of vocabulary tokens.
Interestingly, we find that the optimal vocabulary limitations differs between languages (Figure 1). Our experiments reveal a correlation between the vocabulary size and the phonetic complexity of the languages. Phonetically simpler languages tend to perform better with smaller BPE vocabularies. Furthermore, we have noticed that languages with rich morphological structures benefit from longer token length limitation. One possible explanation is that non-phonetic languages with extensive mor- phology structures require memorizing error pat- terns that consist longer sequences. The field of research concerning the maximum length of indi- vidual vocabulary tokens, irrespective of vocabu- lary size, is currently lacking in extensive study. Our contributions can be summarized as follows:
However, these methods require a significant amount of training data, consisting of pairs of tran- scribed segments and their corresponding ground- truth transcriptions. Unfortunately, publicly avail- able resources are often insufficient, and obtaining labeled data can be expensive and time-consuming.
To overcome this challenge, a commonly used approach involves generating synthetic training ex- amples. These examples are intentionally noised to resemble transcribed sequences and are then com- bined with the labeled examples for training the correction model. The perturbations duced through random noise (Shen et al., 2022; Leng et al., 202 bb; et al., 2021), predefined con- fusion sets like homophone dictionaries (Shen al., 2022), or by using predictions from pre-trained model (Leng et al., 2022).
be intro-
1.
We demonstrate the advantages of training er- ror correction models with synthetic data de- rived from transcribed errors and an adjusted BPE vocabulary.
We hypothesize that simply incorporating ran- dom perturbations aiming for similar overall error rate as a labeled example set insufficient. Instead, it is crucial to closely align with the dis- tribution of transcribed errors. Throughout this
2. Our approach generalizes across diverse datasets and outputs from various speech

---

study, we provide empirical evidence to support the validity of this hypothesis.
3.1 Simulating Transcribed Data.
Let G be a model trained to transcribe audio seg- ments of language X, and D be a set of labeled ex- amples, i.e., audio segments and their correspond- ing transcriptions. We transcribe samples of D using G and extract the error distribution. We then generate synthetic examples by applying the error distribution extracted by using G to transcribe D. The transcribed error distribution is extracted by quantifying the operations required to trans- form a sentence d, E D into di, its transcribed form predicted using G. Specifically, we define del(di, di, c) as the number of times a character d$ was deleted while transforming to di, insert(di, di, c, 2) the number of times was inserted between characters z, d, and subs (di, di, C, x) as the number of times a charac- ter c d was replaced with character x E di.
Robustness to Perturbations. Speech recogni- tion systems commonly include a language model in their post-processing stage. The likelihood score produced by the language model allows prioritiz- ing multiple transcribed candidates (Baevski et al., 2020; Hsu et al., 2021; Babu et al., 2021; Chen et al., 2022), and act as a correction mechanism to potentially corrupt transcriptions. However, the transcribed examples may contain character-level errors such as insertions, deletions, and substitu- tions, which can undermine the effectiveness of the language model (Moradi and Samwald, 2021; Sun et al., 2020; Belinkov and Bisk, 2017).
Previous studies have demonstrated the benefits employing small vocabularies, such as character- level representations, when dealing with noisy or misspelled texts (Ma et al., 2020; Belinkov and Bisk, 2017: Boukkouri et al., 2020). Inspired by this line of research, we conduct experiments with low-granularity representations to correct errors. Our research is also related to body of work that focuses creating robust representations through the regularization of the tokenization pro- cess (Provilkov et al., 2019 Kudo, 2018). Our work stands out in that our objective is to discover an optimal segmentation that is specifically tailored for transcribed errors in a particular language.
We then compute the error probability for each character c C D by summing over each error type, normalized by the frequency of c' in D ((c). The probability for deleting, inserting and substituting each character is detailed in equations la-lc.
Σ del di. c) +
dED
(c)
(1a)
lc
insert di, c,
Pinsert (c, v.,.
deD
3 Methodology
(1b)
In this work, we create transcribed errors correction models that acts as a post- proscssocsssing step, correct- ing the outputs of a speech recognition model.
Σ
subs de di, c, ()+))+
+
We derive the error distribution from transcribed labeled data and leverage it to create synthetic ex- amples. To capture an informative error distribu- tion rather than random one, we transcribe audio segments using fine-tuned XLSSR models and com- pare the transcribed outputs with ground-truft tran- scriptions. Based on the extracted distribution, we introduce perturbations to a large set of text se- quence, resulting in synthetic examples that resem- ble transcribed data. These synthetic examples are subsequently utilized to train our error correction models, well as BPE tokenizers.
Poutis (c, x))
DED
(Ic)
|c|
We observed that performance can be improved by including smoothing factor y |cl. where is hyperparameter and |c| the frequency of the character in the dataset D. We posit that it can be attributed to the underrepreseentation of specific errors when comparing different datasets.
Next, we utilize a raw corpora D, to generate synthetic examples. To apply the error distribution to a raw sentence d, D,, we split the sentence into words based on spaces, and then iterate over each character probability or substitute with character x with P Puuds(e2)) probability. Then, we take each two con- secutive characters x, y E dy and insert character c between them with probability Pinsert (c, x. y).
We aim to control the model's ability to gener- alize to unseen distributions and memorize tran- scribed error patterns by adjusting the BPE vocab- ulary. Coneretely, we experiment with the vocabu- lary size and the maximum length of each token.
d, and delete with Par((c)

---

Polish
Portuguese
18.0
VER
VIER
17.0
12.5
0.5
10
15
0.5
15
25
0.5
5
10
15
Vocabulary Size (x101)
Vocabulary Size (X10P)
Vacabulary Size (X30')
Dutch
Spanish
German
14.0
9.00
10.5
WER
WER
13.0
0.30
12.5
0.25
12.0
8.00
9.5
0.3
10
15
25
0.5
1
10
23
0.5
2
10
15
25
vocabulary Size (x10')
Vocabulary Size (K10%)
Vocabulary Size (x30')
French
16
-X- maxlen4
WER
maxlen16
14
0.5
10 15
25
Vocabulary Size (x107)
Figure 2: Error correction performance (WER) on MLS (Pratap et al., 2020) validation sets using varying levels of vocabulary size and maximum token length limitations. The results were obtained by employing XLS-R models fine-tuned for each specific language. The "char" horizontal lines indicates the performance using a vocabulary consisting individual characters found in the training data.
To prevent excessively noisy synthetic exam- ples, we restrict the number of changes made to each word in d, to maximum of one. We utilize these synthetic examples to train eггог correc- tion model that converts the noisy and corrupted sentences into their correct counterparts.
tions, require additional support for memorizing non-phonetic phrases. To address this challenge, we conducted experiments by providing segmented tokens with additional context in the form of maxi- mum token length.
Therefore, our goal to strike a balance between a smaller vocabulary that promotes generalization and a larger vocabulary that facilitates memoriza- tion. For each language included in our experi- ments, explore different vocabulary hyperpu- rameters, such vocabulary size and maximal token length, determine the best performance for correction model.
3.2
BPE Vocabulary Adjustments.
BPE vocabulary size has significant impact on the processing of input sequences, it directly af- fects the number of segmented tokens (Gowda and May, 2020). Using smaller vocabulary leads a coarser segmentation, which can be beneficial for generalizing to unseen distributions and handling transcribed errors effectively.
Experimental Setup
On the other hand, larger vocabulary enhances the model's ability to memorize patterns and less frequent words. Prior studies have shown that vo- cabulary size correlated with the memorization of the training data (Kharitonov et al., 2021).
Transcribed Error Correction. We conduct ex- periments with seven European languages and uti- lize correction models trained synthetically gen- erated examples. Our experiments includes correct- ing texts transcribed using several speech recog- nition models. We employ BPE tokenizers with vocabulary sizes ranging from 500 to 25K tokens, and token length limitation varying from to
We make significant observation that lan- guages with rich morphology, where words are often composed of multiple morphemes and inflec-

---

16 characters. For a coarse-grained segmentation comparison, we utilize character-based tokenizer consisting of the characters found the training sets. Additionally, we compare our approach to randomly generated synthetic noise at the charac- ter level. We follow the procedure described in Leng et al. (2021b) to create randomly noised sen- tences along with their corresponding ground truth counterparts.
architecture, comprised of encoder and decoder layers. We use attention heads in each layer, and the dimension of the feed-forward layers and the embedding layers are set to 2048 and 512, respec- tively. The input sequences, i.e., the transcribed texts produced by the speech recognition models, are tokenized using BPE tokenizer.
4.2 Data.
Our experiments covers seven languages: Polish (pl), Portuguese (pt), French (fr), German (de), Dutch (nl), Italian (it) and Spanish (es). This blend allows to explore languages with both phonetic and non-phonetic charasteristics, as well as differ- ent degrees of morphological complexity.
Generalizing to Unseen Distributions. We con- duct two experimental settings evaluate the gen- cralization abilitics of our approach. Firstly, perform cross cross-dataset evaluation to test the per- formance on datasets not included in the training process. Secondly, we evaluate our approach on samples transcribed using a different speech recog- nition model, Whisper (Radford et al.. 2022), to assess its ability to capture unique patterns specific speech recognition. This evaluation is particu- larly important since approach is trained using errors obtained by XLSSR models, which have dif- ferent training objectives than Whisper and may yield different error distributions.
Multilingual Librispeech Multilingual Lib- rispeech (MLS) (Pratap et al., 2020) consists of a total of 50K hours of read audiobooks in eight european languages. We follow Pratap et al. (2020) and segment the MLS dataset into training, devel- opment and test partitions.
We further divide the training partition into parts: 90% and 10%. The 90% portion is used for fine-tuning XLSS- models, while the remaining 10% is utilized to extract the transcribed error dis- tribution. Except for the Polish language, where we fine-tune the XLS-R models using approximately 98 hours, we utilize the remaining training data to extract the error distribution.
4.1 Models.
Speech Recognition Models. We utilize pre- trained Whisper (Radford al., 2022) and XLS- R (Babu al., 2021) models from the HugginglFace platform', with varying number of model parame- ters, ranging from 39M to IB.
XLS-R is a speech processing model composed of feature extracting convolution layers followed by multiple transformer encoder layers. It was pre- trained using contrastive objective to map masked audio spans latent representations. The model was then fine-tuned using Connectionist Temporal Classification (CTC) (Graves et al., 2006) loss to project contextualized speech representations to the vocabulary space. We utilize a 4-gram language model when reporting the XLS-R models results. Whisper, a transformer encoder-decoder model, has gained renown for its exceptional speech recog- nition performance. It was trained using weak su- pervision on diverse tasks and languages. key advantage lies in its ability to achieve outstanding results without fine-tuning for specific languages or domains, relying solely on its pre-training.
Common Voice (Ardila et al., 2019), crowed-sourced collection of multilingual human voices, collected through volunteer contri- butions. We employ Common Voice evaluate the ability of our approach to adapt to diverse distribu- tions that were not encountered during training.
Common Voice
mC4. We generate synthetic transcribed texts by utilizing the mC4 (Xue et al., 2020) data set, a variant of the Common Crawl dataset that consists of 101 languages. For each language, we randomly select set of 20M sentences that ranges from 3 to 15 words long. We use lowercase synthetic texts, remove punctuation marks and eliminate digits.
Synthetic Data Generation. We utilize fine- tuned XLS-R models to transcribe a dedicated set of examples for extracting error distributions. We then calculate the probabilities of character inser- tions, deletions, and substitutions, as described in Scction 3. Subsequently, we generate pairs of syn- thetic sentences that resemble transcriptions, ac-
Error Correction Model. We employ a standard transformer encoder-decoder (Vaswani et al., 2017)
'htnss//huggingffcc.co//

---

Model
de
nl
es
it
pt
pl
fr
Baseline
9.0
13.5
8.1
13.1
17.0
13.9
12.4
XLS-R (0.3B)
Correction
9.2
11.8
7.5
12.4
16.1
12.8
12.4
Baseline
7.4
11.6
7.1
12.0
15.8
10.5
10.2
(IB-R(IB)
Correction
7.4
10.5
5.6
10.7
13.2
9.3
9.8
Baseline
24.9
39.4
19.2
41.7
31.3
34.2
36.8
Whisper tiny
Correction
24.2
36.6
16.8
37.5
28.7
33.7
36.4
Baseline
17.7
28.4
12.8
31.1
21.9
22.8
26.6
Whisper base
Correction
16.9
26.1
10.3
27.6
18.8
22.4
25.9
Baseline
10.5
17.2
7.8
21.4
13.0
11.2
16.2
Whisper small
Correction
9.8
13.7
6.6
18.4
12.2
11.0
13.8
Baseline
7.4
11.7
5.3
16.0
9.0
6.5
8.9
Whisper medium
Correction
7.1
10.5
5.3
15.3
8.8
6.5
8.9
Baseline
6.6
10.2
5.4
14.3
9.2
6.6
8.9
Whisper large
Correction
6.7
10.0
5.4
14.0
8.4
6.1
8.5
Table Speech recognition performance (WER) on MLS test sets. The results the XLS-R models were obtained using 4-gram language model.
companied by their unmodified counterparts serv- ing as the ground truth. These parallel sentences are then used to train our correction models.
Results
5.1 Transcrihed Errors Correction.
Furthermore, we explore the impact of the amount of labeled data on the quality of synthetic data. We train correction models using synthetic examples that were generated based on the errors extracted from varying amounts of labeled data.
The results of our approach the MLS and Com- mon Voice datasets reported in Table and Ta- ble 2, respectively. We observe that our approach is particularly effective for the smaller models (e.g.- Whisper tiny, base and small). The results demon- strate consistent improvement across multiple lan- guages and showcase the strong generalization per- formance of our approach on unseen distributions, including different datasets such as Common Voice and speech recognition models like Whisper.
4.3 Training Details.
We follow the training settings of Babu et al. (2021) and fine-tune the XLSSR models 20K updates. We employ the Adam optimizer with 10% warm-up updates, followed by constant learning rate for the next 40% of updates, which gradually decays to zero in the remaining steps. As for the Whisper model, we do not apply any additional training and report the results solely based on the predictions provided by the pre-trained models.
We also observe significant perplexity reductions when comparing our correction models to a 4-gram language models utilized for post-pri post-processing. This provides further motivation regarding the advan- tages of correction models. That is, they make an auto-regressiv prediction for each source token, while a language model maximizes the sentence- level likelihood score which could deviate from the source sequence and cause hallucinations.
The correction models are trained using 4 NVDIA V100 GPUs for 300K updates. We use a batch size of 256 samples per device, and ac- cumulate the gradients for every updates. The model's parameters are optimized using the Adam optimizer with learning rate of 3e-5 and use the first 1000 updates as warm-up. up. We set dropout probability to 0. We use SentencePiece (Kudo and Richardson, 2018) to train our BPE tokenizers. The tokenizers were trained using the generated synthetic data, while controling the vocabulary size and the maximum token length parameters.
Table 3 shows a comparison of different meth- ods for handling perturbations. Our suggested ap- proach outperforms correction models with similar architectures but different training methodologies. Specifically, trained the models on random per- turbations, well models trained using the pop- ular method for learning robust representations, BPE dropout (Provilkov et al., 2019).

---

Model
de
nl
es
it
pt
pl
fr
Baseline
8.8
12.3
8.6
13.7
11.0
14.8
24.5
XLS-R (0.3B)
Correction
8.3
11.5
8.2
12.9
10.9
14.9
23.2
Baseline
8.5
11.5
8.1
13.2
9.4
12.7
18.2
(IB-R(IB)
Correction
8.5
11.2
7.7
12.7
8.8
12.6
17.1
Baseline
34.5
43.6
30.3
44.5
35.2
45.3
49.7
Whisper tiny
Correction
31.3
37.1
27.2
40.3
31.3
43.1
45.4
Baseline
24.5
29.5
19.6
30.5
23.7
32.8
37.3
Whisper base
Correction
22.8
25.1
17.0
27.4
22.1
30.3
32.3
Baseline
13.0
14.2
10.3
16.0
12.5
16.9
22.7
Whisper small
Correction
12.1
13.3
8.1
13.0
12.0
15.8
21.5
Baseline
8.5
8.0
6.9
9.4
8.1
10.1
16.0
Whisper medium
Correction
8.3
7.4
6.7
8.8
8.1
10.0
15.7
Baseline
7.7
7.1
6.4
8.1
7.1
9.0
14.7
Whisper large
Correction
7.9
7.0
6.0
7.8
6.9
8.6
14.7
Table Speech recognition performance (WER) on Common Voice Speech test sets. models were obtained using 4-gram language model.
The results the XLS-R
ModhodYMooel
de
nl
05
it
pt
pl
fr
Whisper small
10.5
17.2
7.8
21.4
13.0
11.2
16.2
Random noise (Leng al., 2021b)
10.9
18.1
7.2
18.3
13.0
11.5
16.2
BPE dropout (Provilkov et al., 2019)
11.1
15.5
7.9
22.2
12.5
11.8
16.8
Extracted error dist. + Adjusted BPE (this work)
9.8
15.7
6.6
18.4
12.2
11.0
15.8
Table The performance (WER) of different methods for handling transcribed errors. The results were obtained by employing transformer-based correction models, as well as Whisper small on the Common Voice dataset.
languages, as depicted Figure 2. For instance, an Italian correction model favors shorter tokens and requires a vocabulary of 500 tokens, while a Polish correction model which benefits from longer tokens and larger vocabulary. In General, the re- sults indicates that the vocabulary limitations are shared between languages with similar attributes. Figure shows that phonetic languages tend benefit from small vocabularies and short tokens, while morphological rich languages typically re- quires larger vocabularies and longer tokens. A possible explanation is that the words of phonetic (isolating) languages are widely composed of one to very few morphemes.
23.0
22.5
20.0
17.5
15.0
12.5
10.0
HoUrS transrobbed audio
Figure 3: The performance (WER) of correction models using information extracted from varying amounts of transcribed examples. Languages which are considered phonetic (es, it, pt) achieve good performance with little as 25 hours of labeled data. However, languages that are less phonetic (pl, de, nl) require significantly more data to perform well.
5.3 Does Additional Labeled Data Affects The Distribution of Errors?
Figure 3 shows the effect of information extracted from varying amounts of labeled data correction models. We transcribed the audio samples using IB parameters XI.S-R models, fine-tuned using ML. for each of the languages.
5.2 The Effect of Vocabulary Limitations.
A key observation is that the best performing vo- cabulary size and maximal token differs between
The results indicates that correction models phonetic languages (it, es, pt) are able to perform

---

well with as few as 25 hours of transcribed audio, while languages considered as less phonetic (de, pl, fr, nl) requires larger amounts in order to show increased performance.
African, and Middle Eastern regions. Additionally, our experiments involved using a fixed set of BPE vocabulary size levels for each language we investigated. Nevertheless, further improvements could be achieved by conducting experiments with additional levels of vocabulary sizes. To generate synthetic examples, we utilized the mC4 dataset. However, it would be beneficial to explore matching the raw dataset to the specific domain of the speech data in future studies.
We also observe that the amount of pre-training data is not necessarily correlated with the effec- tiveness of the extracted error distribution. That is, expected that languages with higher pres- ence during pre-training would require less data in order to yield quality transcribed error distri- bution, yet the phonetic attributes which has the most influence (e.g.. pre-training consists of 25K hours of fr data and 17K hours of pt, yet pt correction model performs well with only 25 hours for extracting the error distribution, while fr requires much more).
References
Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M Tyers, and Gregor Weber. 2019. Common voice: massively- multilingual speech corpus. arXiv:1912.06670.
arXiv preprint
6 Conclusions.
In this paper, propose transcribed errors cor- rection models trained on synthetic data generated using the errors derived from transeribed examples. We introduce a noise protocol that leverages the error distribution extracted from transcribed sam- ples, and demonstrate the importance of adjusting the vocabulary size and the maximum token length of the models' BPE tokenizer. Moreover, we show that such vocabulary adjustments varies between languages, they correlated with their levels of phonetic and morphological properties.
Arun Babu, Changhan Wang, Andros Tjandra, Kushal Lakhotia, Qiantong Xu, Naman Goyal, Kritika Singh, Patrick von Platen, Yatharth Saraf, Juan Pino, al, 2021. XIs-r: Self-supervised cross-lingual speech representation learning scale. arXiv preprint arXiv:2111.09200
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed. and Michael Auli. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems, 12449-124600
Randall Balestriero, Mark Ibrahim, Vlad Sobal, Ari Morcos, Shashank Shekhar, Tom Goldstein, Florian Bordes, Adrien Bardes, Gregoire Mialon, Yuandong Tian, al. 2023. A cookbook of self sulf-super learning. arXiv preprint arXiv:2304.11 12210.
We highlight the concerns and emphasize the limitations associated with training error correction models using synthetic data generated by apply- ing random noise. We showcase the generalization abilities and the performance gains of our proposed approach using several datasets and speech recog- nition models. The claims made in the paper are supported by a comprehensive experimental study. We hope that this research will inspire further explo- ration of language-speciff adjustments various natural language processing tasks and domains.
Yonatan Belinkov and Yonatan Bisk. 2017. Synthetic and natural noise both break neural machine transla- tion. International Conference on Learning Repre- sentations (ICLR).
Hicham El Boukkouri, Olivier Ferret, Thomas Lavergne, Hiroshi Noji. Pierre Zweigenbaum, and Junichi Tsu- jii.. 2020. Characterbert: Reconciling elmo and bert for word-level open- open-vocabulary representations from characters. Proceedings of the 28th International Conference on Computational Linguistics
Limitations
Sanyuan Chen, Chengyi Wang. Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al. 2022.
Our proposed method utilizes error distributions derived from transcribed data and employs a language-specific tokenizer to segment the data. Our experiments primarily focused on European languages, specifically those belonging to the Ro- mance. Slavic, and Germanic language families. However, we have not thoroughly validated our approuch on languages from other language fam- ilies, including those commonly spoken in Asian,
Wavim: Large-scale self- self-supervised pre-training for full stack speech processing. JEEE Journal of Se- lected Topics in Signal Processing, 16(6); 1505-1518.
Jonathan Clark, Dan Garrette, Iulia Ture, and John Wieting. 2022. Canine: Pre-training an efficient tokenization-free encoder for language representa- tion. Transactions of the Association for Computa- tional Linguistics, 10:73-91.

---

Zorik Gekhman, Dina Zverinski, Jonathan Mallinson, and Genady Beryozkin. 2022. Red-ace: Robust er- nor detection for using confidence embeddings. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing.
with soft detection for automatic speech recognition. arXiv preprint arXiv:2212.0103.
Yichong Leng, Xu Tan, Rui Wang, Linchen Zhu, Jin Xu, Wenjie Liu, Linquan Liu, Tao Qin, Xiang- Yang Li, Edward Lin, et al. 2021a. Fastcorrect 2: Fast error correction on multiple candidates for automatic speech recognition. arXiv preprint arXiv:2109. 14420.
Thamme Gowda and Jonathan May. 2020. Finding the optimal vocabulary size for neural machine transla- tion. Findings of the Association for Computational Linguistics: EMNLP 2020.
Yichong Leng, Xu Tan, Linchen Zhu, Jin Xu, Renqian Luo, Linquan Liu, Tao Qin, Xiangyang Li, Edward Lin, and Tie- Yan Liu. 2021b. Fastcorrect: Fast error correction with edit alignment for automatic speech recognition. Advances in Neural Information Pro- cessing Systems, 34:21708-21719.
Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. 2006. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning, pages 369-376.
Wentao Ma, Yiming Cui, Chenglei Si, Ting Liu, Shijin Wang, and Guoping Hu. 2020. Charbert: character- aware pre-trained language model. Proceedings of the 28th International Conference on Computational Linguistics.
Oleksii Hrinchuk, Mariya Popova, and Boris Ginsburg. 2020. Correction of automatic speech recognition with transformer sqquence-t--sequnnc model. In Ieassp 2020-2024 ieee international conference on acoustics, speech and signal processing (icassp), pages 7074-7078. IEEE
Anirudh Mani, Shruti Palaskar, Nimshi Venkat Meripo. Sandeep Konam, and Florian Metze. 2020. Asr er- ror correction and domain adaptation using machine translation. In ICASSP 2020-2020 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6344-6348. IEEE.
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia. Ruslan Salakhutdinov, and Abdel- rahman Mohamed. 2021. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions Audio, Speech, and Language Processing, 29:3451-3460
Milad Moradi and Matthias Samwald. 2021. Evaluating the robustness of neural language models input perturbations. arXiv preprint arXiv:2108. 122.27777..277....
Tuo Ji, Hang Yan, and Xipeng Qiu. 2021. Spellbert: A lightweight pretrained model for chinese spelling check. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 35511
Vineel Pratap, Qiantong Xu, Anuroop Sriram, Gabriel Synnaeve, and Ronan Collobert. 2020. Mls: A large- scale multilingual dataset for speech research, Inter- speech 2020.
Katharina Kann, Samuel R Bowman, and Kyunghyun Cho. 2020. Learning to learn morphological inflec- tion for resource-pos languages. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 34, pages 8058-8065.
Ivan Provilkov, Dmitrii Emelianenko, and Elena Voita. 2019. Bpe-dropout Simple and effective subword regularization. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics.
Vladimir Karpukhin, Omer Levy, Jacob Eisenstein, and Marjan Ghazvininejad. 2019. Training on synthetic noise improves robustness to natural noise in machine translation. arXiv preprint 100Xxv::90.000 5059
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brock- man, Christine McLeavey, and llya Sutskever. 2022. Robust speech recognition via large-scale weak su- pervision. arXiv preprint ar arXiv:2212.04335
Eugene Kharitonov, Marco Baroni, and Dieuwke Hup- kes. 2021. How bpe affects memorization in trans- formers. arXiv preprint arx0v:2110.02782.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Neural machine translation of rare words with subword units. arXiv preprint 1508. 07909.
Taku Kudo. 2018. Subword regularization: Improv- ing neural network translation models with multiple subword candidates. 56th Annual Meeting of the As- sociation for Computational Linguistics, ACL 2018.
Kai Shen, Yichong Leng, Xu Tan, Siliang Tang, Yuan Zhang. Wenjie Liu, and Edward Lin. 2022. Mask the correct tokens: An embarrassingly simple approach for error correction. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing.
Taku Kudo and John Richardson. 2018. Sentencepiece: simple and language independent subword tok- enizer and detokenizer for neural text processing. EMNLP 2018.
Lichao Sun, Kazuma Hashimoto, Wenpeng Yin, Akari Asai, Jia Li, Philip Yu, and Caiming Xiong. 2020. Ady-bert: Bert is not robust misspellings! gen- erating nature adversarial samples on bert. arXiv preprint arXhv:2003.04995 04985.
Yichong Leng, Xu Tan, Wenjie Liu, Kaitao Song, Rui Wang, Xiang-Yan Li, Tao Qin, Edward Lin, and Tie-Yan Liu. 2022. Softcorrect: Error correction

---

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami A1-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2020, mt5: massively multilingual pre-trained text-to-text transformer. In NAACL, 2021.


Reviews:
[{"review_id": "EVa1BK37JF", "reviewer": "Reviewer_AQUj", "summary": "", "questions": "1. Regarding to the transferability of error distribution obtained from XLS-R model to Whisper, could you provide more detail analyses to support? Such as in terms of model architecture, training objective, training data, or some statistics on the errors made by these models, maybe they are highly overlapped, etc. And could this transferability also potentially applied to other ASR models?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments and for pointing out these important issues. \n\nIndeed, we observed some degree of overlap between the error distributions of different ASR models. \n\n \n\nUpon closer examination we observed that many errors are characterized by mispronunciation by the speaker (such as merging words or omitting characters) and adverse audio conditions. \n\nThese gaps are increasingly bridged as the ASR models’ capacity and training set sizes increase. That is, different models tend to fail in similar spots, yet the stronger models overcome some of the errors. \n\nThese observations can explain the error distribution overlap and the ability to transfer the learned mapping of the errors to other models. \n\nWe also noticed that the error distributions are more informative when extracted using medium to smaller models (in terms of model size and training data size), and that the smaller models’ distributions can assist larger models (e.g., one of the Whisper variants), but not necessarily the other way around.  \n\nThese observations are indeed very interesting, yet we left it outside the paper’s scope due to the page limitation. We will use the extra page which could be added upon acceptance to address these issues and provide an ablation demonstrating these findings. We will also add an appendix with a comparison of the error distributions extracted using different models and settings and fix the typos and grammar issues.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- A very well written paper with effective method and comprehensive ablation studies across popular languages, ASR models.\n- The use of open-source data is also a plus point.", "reasons_to_reject": "- The fact that the error distribution learnt from one ASR model could improve the performance on another ASR model is very interesting, however, it is not thoroughly analyzed. The authors hypothesized that the approach could \"capture unique patterns specific to speech recognition\". This is a strong claim.", "questions_to_authors": "1. Regarding to the transferability of error distribution obtained from XLS-R model to Whisper, could you provide more detail analyses to support? Such as in terms of model architecture, training objective, training data, or some statistics on the errors made by these models, maybe they are highly overlapped, etc. And could this transferability also potentially applied to other ASR models?", "paper_topic_and_main_contributions": "- A technique to simulate error in automatic speech recognition system, which aims to mimic the error distribution of an ASR model and is tailored to specific languages. \n- The synthetic data generated from the proposed technique could be used to train error correction models with notable performance improvements. Surprisingly, the error distribution learnt from one ASR model could improve on another ASR model transcription.\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Line 231: frequency of c' -> frequency of c\n- Line 490: with only ~25K hours -> with only ~25 hours"}, "extra_text": {}}, {"review_id": "gV6D7aZch7", "reviewer": "Reviewer_UNKv", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback. \n\nWe chose to focus on European languages for two primary reasons. Firstly, we aimed to use solely publicly available and well-known datasets to enable an easy reproduction of our results. Secondly, we have observed that our approach correlates more closely with phonetic and morphological attributes rather than the structural aspects of languages, and from this perspective the experimented languages are diverse. \n\nWe believe that our findings and approaches can be applied to other languages and are not bound to specific language structures. \n\nAlso, our method can be easily reproduced, as we rely entirely on publicly available tools (e.g., SentencePiece package) and fully describe the implementation details.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The field of ASR error correction is strongly limited by the absence of high quality training data, which would reproduce the fine error patterns, but simultaneously would allow large models to generalise well for the unseen cases. The novel technique for distribution-matched and BPE-adjusted data synthesis is shown to be effective for the task of transcribed error correction and is of high interest for the community. The authors’ argumentation, analysis and experimental set-up appear reliable and useful for the challenge they are addressing.", "reasons_to_reject": "The effectiveness of approach is demonstrated solely on a subset (mostly European) languages. This limits the generalisibility of the results.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper is proposing a novel approach to generating synthetic data, which is required for training a transcription-error correcting model. In particular, the authors a) demonstrate effectiveness of modelling the synthetic data closer to the observed error distribution as compared to standard randomised approaches; and b) show that adjusting BPE vocabulary is central to the language-specific data synthesis for transcribed error correction. The paper’s main contributions are computationally-aided linguistic analysis and new synthetic data resources.   ", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "kD8CucURZj", "reviewer": "Reviewer_z7J7", "summary": "", "questions": "1. Why is insertion contextual, while deletions/substitutions are not?\n1. Any reason to only apply the smoothing factor to del and sub?\n", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback and comments. \n\nRegarding the first question, although it does make sense that all error types should be contextualized, we empirically found that only an inserted character (space included) depends on the characters around it, while substituted and deleted characters are not necessarily correlated with any of the other characters.  \n\nRegarding the second question, in most cases the distributions of substitution and deletion errors are skewed which can lead to unrealistic biases when generating synthetic data (with some errors being critically underrepresented). However, we did not observe similar issues with the distributions of insertion errors. \n\nWe will fix the typos you mentioned, thank you.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The results are convincing as the approach leads to WER reduction by applying error-correction on the output of a state-of-the-art ASR system (Whisper). \n* The work is evaluated on multiple languages, and illustrates the language-dependent nature of the optimal tokenizer vocabulary size and max token length. \n", "reasons_to_reject": "Nothing I can think of. Good paper overall.", "questions_to_authors": "1. Why is insertion contextual, while deletions/substitutions are not?\n1. Any reason to only apply the smoothing factor to del and sub?\n", "paper_topic_and_main_contributions": "This paper describes an approach to train an error correction model, that is then used to correct the transcripts generated by an ASR system. The key contribution of the paper is that the error correction model is trained on synthetically generated text data, using a process that mimics ASR errors. A synthetic text dataset is generated by applying an error-distribution model estimated from ASR transcripts generated by the XLS-R model, on the mC4 dataset. The synthetic dataset is used to train an error-correction model, and the study highlights that the performance of the error correction model can be optimized by controlling the vocabulary size of the tokenizer, and the max token length. Word Error Rates are reported on the multi-lingual LibriSpeech and CommonVoice datasets. The proposed error correction approach outperforms an LM-based post-processing for error correction.\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "References are OK\n", "Typos_Grammar_Style_and_Presentation_Improvements": "Line 490, did you mean 25 hours? (instead of 25k hours)\n"}, "extra_text": {}}]

Author rebuttal:

Meta-review: **Summary:**
The paper introduces a novel approach for generating synthetic data to train error correction models for automatic speech recognition (ASR) systems. The proposed technique mimics the error distribution of ASR models, offering noticeable performance improvements in training error correction models. This approach demonstrates the ability to transfer error distributions across different ASR models. The paper presents convincing results across multiple languages, highlighting the importance of vocabulary size and token length adjustments for optimal language-specific data synthesis. Notably, the proposed approach outperforms a language model-based post-processing method for error correction.
 
**Pros:**
 
- The paper provides a clear description of the proposed technique for generating synthetic data, aligning it with ASR error distributions (Reviewer 1, Reviewer 2).



- Comprehensive ablation studies across popular languages, ASR models and publicly available well-known datasets demonstrate the effectiveness of the proposed approach (Reviewer 1, Reviewer 2).



- The approach's ability to transfer error distributions across different ASR models is intriguing and potentially of high interest to the community (Reviewer 1).



- The study covers multiple languages, showcasing the language-dependent nature of optimal tokenizer vocabulary size and token length (Reviewer 3).



- The paper offers a clear and coherent discussion of the findings and contributions (Reviewer 3).
 
 
**Cons:**
 
- The potential transferability of error distributions across different ASR models is intriguing but is not thoroughly analyzed, which could be seen as a limitation (Reviewer 1).



- The paper's effectiveness is demonstrated primarily on a subset of languages, raising concerns about the generalizability of results (Reviewer 2).



- Some details for reproducibility might be lacking, as indicated by one reviewer, potentially affecting the broader applicability of the approach (Reviewer 2).



- The reasons behind the contextual nature of insertion errors versus deletions and substitutions should be further elaborated (Reviewer 3).
 
Reviewer 1: AQUj,
Reviewer 2: UNKv,
Reviewer 3: z7J7
Decision: Accept-Main