Continuous pseudo-labeling from the start
Authors: Dan Berrebbi
Ronan Collobert
Samy Bengio
Navdeep Jaitly
Tatiana Likhomanenko
Venue: ICLR 2023 poster (2023)
Keywords: selt-training
pseudo-labeling
speech recognition
data selection and filtering

Abstract: Self-training (ST), or pseudo-labeling has sparked significant interest in the automatic speech recognition (ASR) community recently because of its success in harnessing unlabeled data. Unlike prior semi-supervised learning approaches that relied on iteratively regenerating pseudo-labels (PLs) from a trained model and using them to train a new model, recent state-of-the-art methods perform `continuous training' where PLs are generated using a very recent version of the model being trained. Nevertheless, these approaches still rely on bootstrapping the ST using an initial supervised learning phase where the model is trained on labeled data alone. We believe this has the potential for over-fitting to the labeled dataset in low resource settings and that ST from the start of training should reduce over-fitting. In this paper we show how we can do this by dynamically controlling the evolution of PLs during the training process in ASR. To the best of our knowledge, this is the first study that shows the feasibility of generating PLs from the very start of the training. We are able to achieve this using two techniques that avoid instabilities which lead to degenerate models that do not generalize. Firstly, we control the evolution of PLs through a curriculum that uses the online changes in PLs to control the membership of the cache of PLs and improve generalization. Secondly, we find that by sampling transcriptions from the predictive distribution, rather than only using the best transcription, we can stabilize training further. With these techniques, our ST models match prior works without an external language model.

Published as a conference paper at ICLR 2023
CONTINUOUS PSEUDO-LABELING FROM THE START
Dan Berrebbi*
Ronan Collobert, Samy Bengio,
Navdeep Jaitly, Tatiana Likhomanenko Apple {collobert, bengio, njaitly, anpplaaappareeepoccccom
Carnegie Mellon University dbcrrcbbe andrew. cmu odu
ABSTRACT
Self-training (ST), or pseudo-labeling has sparked significant interest in the au- tomatic speech recognition (ASR) community recently because of its success in harnessing unlabeled data. Unlike prior semi-supervised learning approaches that relied iteratively regenerating pseudo-labels (PLs) from trained model and ing them to train new model, recent state-of-the-ae methods perform 'continuous training' where PLs are generated using very recent version of the model being trained. Nevertheless, these approaches still rely on bootstrapping the ST using an initial supervised learning phase where the model is trained on labeled data alone. We believe this has the potential for over-fitting to the labeled dataset in low resource settings and that ST from the start of training should reduce over-fitting In this paper we show how we can do this by dynamically controlling the evolution of PLs during the training process in ASR. To the best of our knowledge, this is the first study that shows the feasibility of generating PLs from the very start of the training. We are able to achieve this using two techniques that avoid instabilities which lead to degenerate models that do not generalize. Firstly, we control the evolution of PLs through a curriculum that uses the online changes in PLs to control the membership of the cache of PI.s and improve generalization. Secondly, that by sampling transcriptions from the predictive distribution, rather than only using the best transcription, we can stabilize training further. With these techniques, our ST models match prior works without an external language model.
find
INTRODUCTION
The past few years have witnessed growth in methods that leverage large amount of unlabeled
data in domains such as speech, vision and language to produce state-of-the-art results, e-g. Baevski
et al.] (2020) 2022); Chen et al. (2020a); Caron et al. (202T): He et al. (2022); Cai et al. (2022);
Brown et al. p0200; Ramesh et al. (021)). Amongst the techniques that have made this possible are self-supervised learning (SSL) and self-training (ST) (Scudder 1965 Lee| 2013). While SSL is typically used in unsupervised settings, ST is applied in supervised settings where labeled data can be extended with unlabeled data that is labeled using a prior model, process known as pseudo-labeling (PL). These techniques can reduce the burden of expensive labeling processes while successfully train data hungry models such as transformers using large quantities of unlabeled data.
Current state-of-the -tht SSL. methods in speech (Baevski et al. 2020 Hsu et al. 2021 Baevski et al. 2022] Chung et al.) (202I) are typically trained in two phases. First, the models are pre-trained on thousands of hours of unlabeled speech, and then they are further adapted by fine-tuning on the actual task of automatic speech recognition (ASR) using smaller supervised set. However, because the pre-training (PT) phase is task agnostic, self- self-supervision can under- under-perform on a specific downstream task (Talnikar et al., 202T Dery et al.) (022). Further, SSL pre-training leads to a more complicated pipeline involving multiple phases. By contrast, ST algorithms also use unlabeled data but do not require phases of training with different objectives that makes the training pipeline simpler.
In this paper, we focus on recent ST algorithms that perform 'continuous training' of single model. In contrast to earlier ST training methods that iterate between generating PLs over the entire unlabeled dataset and training model (teacher-student) (Synnaeve et al. 2020 Kahn et al.) 2020a
Zhang
Work done during internship at Apple.

---

Published as a conference paper at ICLR 2023
Table 1:
Continuous ST (using slimIPL) with different pre-training steps (M) using a 10h dataset reveals that more pre-training can lead to worse results (we show word error rate, WER, on dev-clean).
M
10k
20k
40k
WER
14.3
17.1
22.9
et al. 2020 Park et al. 2020), here pseudo-labels (PLs) are generated online with a very recent
version of the model (Xu al. 2020 Likhomanenko et al. 2021 Manohar et al.) [021]] Higuchi
et al. 1 2022a b) and training is faster and more resource-efficien. One of the main challenges
for continuous ST training stability (Likhomanenko al. 2021a Higuchi et al. 202 2022b
Cai et al.,) 2022). While these prior works use various techniques for stabilization, one common ingredient that models are initially trained on labeled data for M steps. slimlPL (Likhomanenko
et al. a02 showed robustness to M in some setrings, but well-established recipe does not seem to exist for the case of small labeled datasets (aka. the low resource setting). Indeed, we find that more pre-training steps, compared to what was shown previously in Likhomanenko et al. (2021a), can lead to worse results (see Table D.. We hypothesize that this is due to over-fitting to the labeled set early in training in low resource settings and in this paper try to improve results by doing ST without any pre-trac vre-training (i.e. M = 0). However, in our experiments, off-the-shelf slimIPL diverges early in training in low resource settings, summarize here:
developed methods to address this problem which we
We show that sampling transcriptions from the output distribution instead of using the best tran- scription makes ST robust and stable, especially when no pre-training is performed. We propose a new curriculum for controlling the PL. distribution during training. The currieulum uses the Levenshtein distance between PLs at different time steps to control how PLs are updated, and how unsupervised examples are chosen for training.
For the first time, with these strategies show that continuous PL can be done from the very start of the training matching prior works without an external language model.
2 EXPERIMENTAL SETUP AND RELATED METHODS
Data All our experiments are performed using the LibriSpeech dataset (Panayotov et al. 2015). We the Irain-clean-360 and train-other-500 regular subsets as unlabeled data, and consider either subset of 10h randomly drawn from Iruin-clean- -00,, or the full 100h set (Irain-clean- 100) as labeled data. Comparisons with existing works are also provided using the 10h subset from I. Libri-Light (Kahn) iet al. 2020b In addition, we evaluate the final configuration of our methods on the Common Voice Ardila et al. (2020) for French language where we sample 10h and 100h from the train set to use as labeled data and the rest as unlabeled data (see Appendix A.3)
Acoustic model Following Likhomanenko et al. (021a), models are trained with English letters token se the Connectionist Temporal Classification Graves et al. (2006) (CTC) loss, identical SpecAugment (Park et al. 2019) parameters, and Adagrad optimizer (Duchi et al. 2011). The acoustic model is the same transformer architecture that introduced in SlimIPL, except that encode positions with either absolute sinusoidal positional embedding (Vaswani et al., (017) or the recently proposed CAPE (Likhomanenko et al.) (021b) instead of relative positional embedding (Shaw et al.) 2018). This allows us to speed up training (by 2-3x) and decrease the memory footprint significantly. All models trained GPUs for a maximum of 500k updates. We use either static batch of examples or dynamic batch that packs ~ 290s of audio per GPU.
Continuous pseudo-labeling (PL) in ASR Let L (I,, yid and = {xy be the labeled and unlabeled datasets, respectively, We consider semi-supervised PL approach where acoustic model
'Libri-Light 10h subset contains only 24 speakers drawn from the whole LibriSpeech (from both clean and noisy subsets). To keep experiments consistent, and also assess domain transfer the unlaheled noisy subsets, we reconstructed the 10h set from the train-clean- 100, sampling randomly from the speakers and retaining the original 250 speakers from this subset.
226 letters augmented with the apostrophe and word boundary token.
2

---

Published as a conference paper at ICLR 2023
Algorithm 1: slimIPI. and our proposed changes (red deletion and green - addition)
Inputs: labeled T, = (at, yi and unlabeled U {m;} data, ã augmentation(a)) initialization 0",
cache [}. learning rate 1/k. losses Lt. and Lv, parameters M, Ni. Nr. Pout and C
PL function ['L( [e; 0, T) PL(x; 0) defined via Eq. (2)
PL function (=; 0, defined sampling with lemperature (see Section 4.2)
Result: Acoustic model A(x; 0)
// Initial pre-training (PT) phase train only on labeled samples
Train A (x. 2/) L for M steps:
gh+1 0 V Lr (A(@;; 0*), y), =- 1,M
Decrease model's A(æ; 0) dropout
// Train on labeled data while filling the cache
for M M + C do
For random x E U generate y
),) and C + (U((2.)))
gh+l l 7/. V C2 (AE(; )² y). (x. y) E
== max 1 - k/k K)
Cont nucus pseudo -1 label ing training with the cache
11
repeat
12
if rand(00 1) < Nr,((N, + No) then
Draw (z.y) E and gh+ 0² 7/h VL
14
else
Draw b == (æ,, v) E C and 0 0³ 7he (A 0"); (²))))
16
ÿ PL(AA (x; ,²) // Compute current model state FL
Pout TER(y, 2) k K else Pout // Compute dynamic Pout
18
rand(0, 1) < Pout then
For random x' E generate j' PL(AAnferenee (x'; 0", and C + C C-XU(((> ())
else
C C (U(( // Same sample and PLs back into the cache
21
C ((((),))) // same sample but new PL. back into the cache
k k
24
== 10
25
until convergence maximum iterations are reached
A(x; 0) with model parameters 0 continuously trained on a combination of and pseudo-labelled set derived from U. The model is trained by minimizing loss
( ()()) + (e),
(1)
where À R+ is tunable hyper-parameter controlling the importance of unlabeled data. The loss for labeled data is defined as (f) = - Ex (x)L log Po (ylar, where po(yyaa ( is the conditional distri- bution defined by A(æ; 0). The loss for unlabeled data is defined as (i)(0) -Er~U log pe ( (z).
where ÿ the PL transcription for data point generated using the model being trained. Specifically,
ÿ argmax log (3)(z))
Continuous PL keeps updating the pseudo-labels via Eq. (2), as the model trains. This procedure is prone to divergence, as without any constraint PLs can self reiffrreef rapidly to trivial distribution.
Methods stabilize training Several approaches have been proposed to stabilize continuous PL. A pre-training phase (PT) on the supervised data only (optimizing the loss (r. for M updates) always a key component. For e.g. in Chen al. (2020b) PT is performed until full convergence. Another technique is the use of an exponential moving average (EMA) of the acoustic model to
generate the pseudo-labels Eq. ( (Likhomanenko al. 2021a: Manohar al. 2021 Higuchi
et al.
2021 2022h
Zhang et al. 2022
To avoid the significant memory footprint EMA Likhomanenko et al. 202 20 introduced slimIPL, which uses dynamic cache instead of the EMA to stabilize the training. The cache maintains a set of unlabeled samples UC (with fixed size U = C") and their associated PLs, generated by previous model states. After the pre-training phase, slimIPI. minimizes the loss in Eq. (1). using the
slimIPL
3

---

Published as a conference paper at ICLR 2023
truin step k
Liaai step k'
Ours, train step K'
data U
cucke &
0t = train 3eppt..9.0")
(x,95 - getl = taann stepli. 9,
(x,
(x.9 PL[a((x;))))
y²= PL(
-PPe Pone
PPP ((o(t,54))
Figure 1: Comparison between slimlPL. (left) and how we control the cache by using PL. evolution (right). The constant Pout from slimlPL. now is dynamic and computed based on the PL evolution.
unlabeled subset UC, which is itself updated as training goes: at each iteration, slimlPL removes sample from the cache with probability Poat, replacing with new one U along with its generated PL. More details about slimlPL. can be found in Algorithm and in Figure
PLs selection Pseudo-labels selection can help to achieve better convergence by filtering out noisy PLs that prevent model from faster training. There are also lot of efforts on the curriculum ascudo-labeled data selection: e.g. confidence filtering (Zhang et al. 2021 or assigning weights pseudo-labeled data based the model uncertainty estimation (Huang al. 2022). One of the recent works (Zhang et al.] 2022) in ASR proposes to use PL. curriculum filtering based on the Levenshtein distance between PLs generated for original and weakly augmented inputs. Later will see that our idea is based solely on the PL evolution rather than on input augmentation.
Relation to consistency regularization Popular consistency regularization methods (Sajjadi et al. 2016, Laine & Aila 2016 Sohn et al. 2020, Berthelot et al. 2019) leverage the idea that model should output the same distribution for an unlabeled example even after it has been augmented. In this paper take inspiration from these works but focus
orthogonal view: consider distances between model outputs at different time steps. Also, contrary to consistency regularization, do not use this distance as objective function train model but data selection criterion.
Hyper-paramee selection All hyper- hapeh--paraaaeeee parameters and model selections are performed using dev- clean and dev-other sets. We report final token (TER) or word (WER) error rates on test-clean and lest-other sets. In all experiments, everything else is kept as in the slimIPL paper. By default we use C = 1000, 1, M In most experiments try different random seeds and report metric mean and standard deviation.
only tune (C, Pout, M. X) from the training procedure while
MOTIVATION
Existing continuous PI. approaches rely on a two-step process: first pre-training (PT) on labeled data only, then continue the model training with both labeled and unlabeled data. While PT is known to be critical for the stability of continuous PL, we are interested in this work to find ways to remove the PT phase to simplify the whole procedure, and possibly improve the overall performance, both in terms of convergence speed and final WER.
PT improves the final WER Initial experiments with slimIPL, Table 2 show that with even its simple cache strategy used to stabilize training. PT helps improving the final WER. It is not surprising, as without PT, Pl.s are of poor quality (>> 90% WER) at the beginning of training as the model mostly produces random outputs. Careful tuning of the number of PT steps however important, especially in low-resource supervised settings, as shown in Table [
Caching as a replacement for PT Vanilla continuous PL very similar to slimIPL with Pout = 1 (see Section 2). With the caching strategy, slimlPL picks unlabeled samples (and their associated PLs) from a cache when needed, and immediately replaces these examples with new unlabeled samples (and their new PLs). This allows always use PLs generated from previous version of the trained model, while efficiently computing these PLs. While being simple, we observe Table 2 that this approach is enough stabilize continuous Pl., assuming large enough cache.
4

---

Published as a conference paper at ICLR 2023
Table 2: Continuous PL w/ and w/o pre-training (PT) phase for slimlPL. 'DV' states for divergence.
dev-clean WER
dev-other WER
Data
Pout
w/o PT
w/ PT
w/o PT
w/ PT
10h
23.31.7 DV
13.8
32.11.3
17.5
10h
0.1
11.4
DV
14.0
100h
4.5g.
3.1
10.6g.3
8.1
100h
0.1
DV
3.6
DV
7.5
When to update the PLs from the cached samples is critical slimlPI. (Algorithm I each sample (ar, ÿ)) in the cache C at step k has Pl. i= (xx 0*)) that was generated with the model @ at step < k' when it was added to the cache. After using the sample (æ. ÿ) for training, slimIPL adds it back into the cache with probability found however that updating PLs with the current model state WER performance. See Table which compares the original slimIPL. strategy ('old"), with the one where the PLs updated when sample has been selected in the cache ("new"). For that reason, in the following experiments, we will be using y (L 0² as a PL strategy, when keeping sample back into the cache.
Pout, leaving its corresponding PLs unchanged. We
(L(A(x;; 0x improves final
Controlling eache contents dynamically can improve WER When the cache is updated less often (Pout 1), we see in Table 2 that one may improve the WER, but then PT is essential to avoid any divergence. In Likhomanenko et al. (2021a), the authors slimIPL have reported robustness (in terms of test WER) with respect to Pout However, in Appendix reveal different learning dynamics for different values of Pout our ablations with specific schedules the probability Pout suggest that models without a PT phase would benefit more from low Pout at the beginning of training, which would make training easier initially by letting the model focus the same examples. In addition, later in training, the training procedure might benefit from high Pout= as seeing a wider range of examples may lead to more stability. While we observe significant changes in dynamics with 10h of supervision, with larger labeled set (100h) the different strategies do not make such huge difference.
experiments reported in Table 3 and Figure
The above observations suggest that by dynamically controlling how the cache evolves we can improve results in limited data settings. One possible way of doing this is by using strategy that depends on the rate of evolution of PLs in the cache. In the next section we present such method.
Table 3: Strategies PLs and cache renewing (w/o PT phase). When Pout 1 and sample goes back into the cache, we compare models using the same PI. as it was ý = PL(A(a; o+)) (old) or the newly re-generated PL PL(A(<: @k)) (new). For cache renewing, compare static Pout and simple scheduling with Pout being different before and after 130k steps.
10h, WER
100h, WER
Pout
PLs
dev-clean
dev-other
dev-clean
dev-other
1
23.3
32.11.3
4.5p.1
10.6p.a
0.1
old
DV
DV
DV
DV
0.1
new
15.30.6
25.40.4
4.5p.1
10.4u.1
+ 0.1
old
23.01.1
32.1o.4
4.5p.1
11.0p.0
1 - 0.1
new 24.81.4
36.1o.s
4.4
10.2
0.1 +
old
DV
DV
DV
DV
0.1 +
new
13.70.s
20.70.8
4.8p.1
11.30.1
4 METHODS OF STABLE TRAINING
4.1 CONTROLLING CACHE BY USING PL EVOLUTION
Let's consider an example x E U to be put into the cache at training step k, see Figure [ Its PL is defined as PT[[A(;; 0k) PL(xx; k). At step 3 > k, this example (æ. ÿ) is selected
5

---

Published as a conference paper at ICLR 2023
from the cache and the model is updated to 6 using the gradient of the loss. Unlike slimIPL, the probability of removing the example from the cache is not constant anymore. Instead, Pout is dynamically computed at step k' for sample that is selected from the cache follows:
Pout ( k) = f[p(PI(a; k)) PL(;;) K))
where p is the Levenshtein edit-distance, and f the function that encapsulates how evolution in PLs should determine the rate at which examples are removed from the cache. Using different choices of we can consider different ways of actively controlling the cache (and hence the model training) using the evolution of the PLs. We consider simple functions f + and f H 1 z. The first function encourages the cache to maintain examples whose Pl.s are stable, which might lead to slower learning. The second function maintains examples whose PLs are changing fast which might lead to faster learning but less stable behavior.
Note that while we explained the method using a single example æ from the unlabelled set, in practice we operate the algorithm on a batch level, and the statistics are computed over a full batch of examples, which are all put back in the cache or removed together.
4.2 ALIGNMENT SAMPLING
As discussed in Section 3 training instability shows up as the acoustic model distribution A(æ; 0%) collapses to a degenerate distribution, e.g. empty transcriptions. While cache and/or an exponential moving average model can stabilize training, they do not resolve the issue entirely, especially in the low data regime, with no pre-training, and the model often collapses degenerate solution. Even our proposed method above (see Section 4.1 is susceptible to this collapse on the 10h dataset.
In order to overcome the collapse issue and still make use of unlabeled data as early as possible. we propose to sample targets from the token distribution for every frame (Likhomanenko et al. 2022). We believe that sampling PLs around the most probable hard labels is an effective stabilization technique which works by adding appropriate noise to the targets: it is a way enforce lower bound on the entropy of the label distribution which mitigates the collapse issuc]] As the model is learnt with CTC, every per frame predicted distribution p/ (//wz) w E for token set w and time frame t is considered to be independent. Thus, for every audio frame, we sample a token label W/ ~ p [w] (w)x). A temperature is introduced to smooth the distribution obtained from the model. After the frame level labels are sampled, they are transformed into the transcription by deduplicating consecutive repetitions of the same output token, and removing the left over auxiliary blank tokens")
Sampling Temperature Schedule As + Do the distribution over tokens p [w) x. r) approaches the uniform one, the PL sequence of tokens becomes purely random. On the other hand, as T - the distribution approaches the argmax function which is equivalent to the hard labels in slimIPI. We find that r > performs poorly. With r a model avoids divergence at the beginning of training but end up with worse final performance than hard PLs (r 0): this happens mostly because of larger noise presence due to sampling (quality of PLs is observed being worse). Lower temperatures, e.g. r = 0.1, give indistinguishable results from hard PLs (r = 0). These observations suggest that decreasing temperature as training proceeds can stabilize training at the beginning and benefit from less noisy PLs in the end. We found that simple linear schedule for 7 from I to 0.1 works well.
The summary of our proposed methods on top of slimIPL is given in Algorithm
5 RESULTS
5.1
DYNAMIC SELECTION FOR PSEUDO-LABELED SAMPLES
In Table 4 we show results from using only the method introduced in Section We experiment with token error rate (TER) distance computed between PLs on an entire batch and the two functions as discussed above. For both settings of 100h and 10h of supervised data the proposed dynamic
A With regularization (cache, and/or alignment sampling), the Pl. procedure often collapses generating just blanks very quickly (Likhomanenko et al.] 2021a) it is biased, has 100% WER, but has no variance. A Alignment sampling avoids this by generating noisy targets that have variance.
F.g. alignment 'cc###aattt'" will be transformed into 'cat', where # is a CTC blank token.

---

Published as a conference paper at ICLR 2023
Table WER on dev-clean and dev-other for different cache selection methods (p). We use either Pout = p or a strategy where Pout for the first 130K steps, switching to Pout = 1 afterwards, as
shown in Section 3 Alignment sampling from Section 4.2 is not used.
10h
100h
Pout =
Pout p
Pout =
p
Pout p -
clean
other
clean
other
clean
other
clean
other
0.1
15.30.6
25.40.4
13.7p.8
20.7a.8
4.5p.1
10.6p.3
4.80.1
11.3g.1
TERPPLA PL())
14.70.5
24.60.3
13.2
19.11.8
4.6p.1
10.50.2
4.40.1
10.10.2
- (E))
16.0p.4
26.50.a
17.812
30.42.3
4.4g.1
11.10.5
4.50.0
10.5g.5
10-
of tamu TE muhh
0.6
144
0.4
0.2
04
50000
200000
0.2
(a) Pour Pour=WEiT WERIEEERRRL per batch along the training.
(b) Correlation between WER[PL goldel and H) k')]
Figure 2: Analysis of our curriculum PLs selection criteria. WER is given in scale of (0, 1).
selection decreases WER over the baseline with constant Pout- This behavior also holds when switch from the dynamic strategy of Eq. 33 to constant Pout = 1 after 130K steps of training. For 10h of labeled data setting the improvement over the baseline is larger and reaches around 1% absolute. The function ff x + x performs worse than f + x and hence we use this setting for subsequent experiments.
Our analysis of dynamic probabilities Pout from Table shows: (1) TER(PL((;; k)), PL(x: k')] is close 100% the beginning of training (the model changes very fast), and quickly decreases (less than 10% after 30k steps); (ii) over training different batches get different values of Post. see Figure 2a] (iii) proposed distance correlates with the oracle WER computed between PLs and ground truth labels for U, see Figure 2b The latter demonstrates that our choice of dynamic selection encapsulates knowledge about actual PLs quality.
5.2 ALIGNMENT SAMPLING
In Table compare results for models trained with hard PLs (T 0), models trained with alignment sampling and constant > 0, and models trained with linear schedule of - from 1 to 0.1 (1 + 0.1). as described in Section 4.2 For this section we do not use dynamic control of the cache as introduced in Section 4.1 Here we highlight some observations. Firstly, alignment sampling with high т reduces the number of diverged models (either T 1 or constant temperature over the training does not provide best results: = 0.1 is similar to the baseline while = is even worse; the differennc more pronounced for the 10h of supervision with Pout 0.1 1. Besides, WER we also report TER to highlight that sampling with 1 leads to notable CER degradation. However, scheduled - = 1 + 0.1 provides both stable training (no divergence is observed in experiments) and similar significantly better TER/WER (1.3%-2.7%% over the baseline. The best results are obtained with Pout = 0.1 + showing compatibility of sampling and dynamic probability.
>> 0.1). Secondly,
7

---

Published as a conference paper at ICLR 2023
Table 5: TER and WER on dev-other for sampling PLs with different temperature т, including linear schedule of 7 in case of constant Pout (left parts) or alternated one (right parts), see Section 3 'DV' denotes the number of diverged models over 3 runs with random seeds. PL evolution via dynamic cache probability from Section 4.1 is not used.
10h
100h
Pout 0.1
Pout
0.1
Pout 0.1
Pout
TER
WER
#DV
TER
WER
#DV
TER
WER
#DV
TER
WER
#DV
0 (angmax)
10.1o.2
25.40.4
0
7.80.7
21.4a.s
3.9a.1
10.4o.1
3.7a.1
10.2o.1
0.1
10.91.a
26.12.0
0
8.40.1
21.21.9
0
3.90.1
10.30.1
3.60.1
10.30.1
2
1 11.41a
26. 26.5..8
0
12.1o.a
31.21.9
0
4.2b.2
10.40.3
0
3.7a.
10.40.2
1+00.
0.1
9.71.
22.71.4
7.50.6
20.1
0
3.8p.
10.20.1
3.7a.
10.1p.1
Table 6: Combination of methods (Sections 4.1 and 4.2 for hard labels (left part) and for sampling (right part) with linear schedule on the temperature. 'DV' states for models divergence, 'old' denotes usage of (L k), while 'new" denotes the use of PL(x; k"). We compare different Pout (all with using 'new"): scheduled Pout 0.1 - 1 (switching at 130K steps), p = TER R and scheduled p = TER > 1 (switching at 130K steps). The WER on dev-other is reported. All results are reported across 3 runs with different seeds.
Argmax
Sampling
Data
old
0.1
P
old
new
0.1
p 1
10h
DV
25.4 DV
21.4g.3 DV
21.60.3 DV
19.1
DV
22.7
20.1 DV
21.2
20.71.9
10h
5
DV
DV
DV
DV
14.70.4
13.30.2
100h
DV
10.60.3 DV
11.3..1 DV
10.50.2 DV
10.10.2
13.5g.3
10.2 DV
10.1o.1
10.50.2
10.20.2
100h
DV
DV
DV
DV
10.70.s
10.0o.3
5.3
COMBINING METHODS FOR BEST RESULTS
In this section we highlight the results that can be achieved by combining together all the methods reported above in Sections 4. and 4.2 In Table we give a detailed comparison for both 10h and 100h of supervision. As we have now stable training pipeline from the start (no PT), we also play with ratio À (see Eq. (I)) searching it in range 5]. This raises training instability risk while larger proportion of unlabeled data may improve the model according to Likhomanenko et al. (2021a).
For 10 hours of supervised data the models benefit lot from the higher À and become competitive with models trained with PT phase as well as with prior works (Baevski et al. 2020; Likhomanenko al. (021a)). ). Note that combining sampling with dynamic Pout based on PLs evolution is necessary to have stable training for À >
To have proper comparison with aforementioned prior works increase the batch size and use dynamic batching for the best configuration. First, we confirm that both sampling and dynamically controlling the cache give stable training (see e.g. Appendix Table 13). Second, in Table 7 for 10h/100h setup (A = 5/A = 3) our models achieve similar or better results with no PT compared to PT-based models (which are reproductions of slimIPL using the same settings that we use for our method) while matching the prior works.
To ensure our methods general enough we probe the final configuration (found for LibriSpeech) on Common Voice, French language data. We use exactly the same models with sinusoidal positional embedding and the same hyper-parame syper-parameters. The only thing we tune slimIPL parameter M. Results in Table 8 show that our methods work out of the box: without PT we are able to match slimIPL baseline for 100h of supervision, while we improve results upon sL.mIPI. for low supervision setting of 10h with an average relative WER reduction of 18%.
$ As we use different 10h split in this work we also report results for 10h set with 24 speakers from Libri-Light used in prior works. We found that training with no PT is more prone to unstable training for this set. while our method is able to stabilize it and get comparable performance with its haseline counterpart which lags behind the prior works.
8

---

Published as a conference paper at ICLR 2023
Table Comparison of our best models with prior works for 10h and 100h of supervision. Results are reported across 3 random seeds. For wav2vec 2.0 and slimlPL. we report the prior work results and our reproduction following official open-sourced recipes. 'Posemb' denotes type of used positional embedding. The 10h set from Libri-Light is marked with **
dev WER
test WER
Model
Data Posemb
clean
other
clean
other
w2v 2.0, Large Baevski et al. 2020
conv
8.1
12.0
8.0
12.1
w2v 2.0, Large, reproduction
conv
8.
12.90.2
8.
13.3
slimIPL Likhomanenko et al. 2021a
relpos
11.4
14.0
11.4
14.7
slimIPI.
10h
CAPE
14.4p.3
18.8
15.
19.30.3
Ours
CAPE
15.8
20.41.6
15.91.5
20.41.3
slimIPL.
sinpos
32.70.6
36.80.3
33.70.7
37.60.4
Ours
sinpos
20.72.0
24.42.
21.42.1
24.9%.9
w2v 2.0, Large
conv
7.40.3
12.70.3
7.7p.3
13.00.4
slimIPL
CAPE
10.0p.4
15. lo.s
9.90.4
15.7u.5
10h
Ours
CAPE
8.2p.2
13.11.4
8.5
13.62.1
slimIPI.
sinpos
22.51.3
28.11.3
22.91.2
29.41.4
Ours
sinpos
8.60.2
13.30.2
8.7u.z
13.40.2
w2v 2.0, Large (Baevski et al.
2020
conv
4.6
9.3
4.7
9.0
slimIPI. (Likhomanenko et al. 2021a
relpos
3.7
7.3
3.8
slimlPL
100h
CAPE
3.7g.1
8.0p.1
3.90.1
8.2p.1
Ours
CAPE
4.1g.1
8.40.1
4.0b.1
8.6p.2
slimIPI.
sinpos
3.7d.1
7.8p.1
3.8a.1
8.0p.1
Ours
sinpos
4.0g.1
8.1
4.1g.1
8.4p.2
Lower bound, fully supervised
960h
CAPE
2.60.1
6.90.1
2.70.1
6.9p.1
Table 8: Comparison fully supervised, slimIPL and our methods on Common Voice French. Results are reported across 6 random seeds. Sinusoidal positional embedding is used for all models.
WER
Model
Data
valid
test
Fully supervised
59.9u.s
62.6u.6
slimIPL.
10h
29.92.0
31.12.1
Ours
24.61.8
26.01.o
Fully supervised
17.3
19.3o.1
slimlPL
100h
12.80.2
14.
Ours
13.0u.2
14.30.2
Fully supervised
540h
10.9..4
12.30.3
CONCLUSION
In this paper we show that we can perform continuous pscudo-labeling from the very start of training and get improved results in low supervision settings. We were able achieve these results by using alignment sampling and a dynamic cache selection strategy that is based on the evolution of the pseudo-labels during training. Being able to perform pseudo-labeling from the very start further simplifies training, avoiding complicated multi-step pipelines and allows us to focus on a simpler one. Our work also provides avenues for explorations into curriculum strategies for pseudo-labeling and hope to build upon the ideas and results presented in this paper. In the future wish to explore the effectiveness these methods to other settings for ASR such as sequence-to-sqqueneee www.--euucccerrranuuueeeeo model -ut-of-domain unsupervised data, and neural models not based on transformers.
'The proposed dynamic control of the cache does not rely on anything specific to CTC. Alignment sampling should be transferable to Transducer directly, while for sooguencc-tt-seqqceeee we would sample transcription directly from the model.

---

Published as a conference paper at ICLR 2023
7 REPRODUCIBILITY STATEMENT
We report detailed settings of our experiments which are based on the previously open-sourced recipes for Likhomanenko et al. (2221a) through the paper and also in Appendix A.2 and B We aim to open source the code of our method and experiments soon.
ETHICS
For this paper we used publicly available datasets. Our goal is build models that work for low supervision settings and hope this is positive contribution towards under-representted data sources for ASR. While one can imagine ASR being used for negative purposes, it is our hope that the advantages generated by improving ASR for low-resource settings outweigh its possible negative uses.
ACKNOWLEDGMENTS
We would like to thank Richard He Bai, Jagrit Digani, David Grangier, Loren Lugosch, Yizhe Zhang, and machine learning research teammates for helpful discussions and support throughout the work.
REFERENCES
Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty Reuben Morais, Lindsay Saunders, Francis Tyers, and Gregor Weber. Common voice: A massively- multilingual speech corpus. In Proceedings of the 12th Language Resources and Evaluation Conference, pp. 4218-4221 2020.
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for supe-supee learning of speech representations, Advances in Neural Information Processing Systems, 33, 2020.
Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. Data2vec: A general framework for self-supervised learning in speech, vision and language. arXiv preprint arXiv:2202.03555 2022.
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach to remi-supervised learning. Advances in neural information processing systems, 32, 2019.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot leamerr. Advances in neural information processing systems, 33:1877-1901, 2020. 2020.
Zhaowei Cai, Avinash Ravichandran, Paolo Favaro, Manchen Wang. Davide Modolo, Rahul Bhotika, Zhuowen Tu, and Stefano Soatto. Semi-supervised vision transformers at scale. arXiv preprint arXiv:2208. 05688, 2022.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings the IEFF/CVF International Conference on Computer Vision, pp. 9650-9660, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pp. 1597-1607. PMLR, 2020a.
Yang Chen, Weiran Wang, and Chao Wang. Semi-supervised asr by end-to-end self-training, Proc. Interspeech 2020, pp. 2787-2791, 2020b.
Yu-An Chung, Yu Zhang, Wei Han, Chung- -Ceng Chiu, James Qin, Ruoming Pang, and Yonghui Wu. W2v-bert: Combining contrastive learning and masked language modeling for self-supervise speech pre-training In 202/ IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 244-250 IEFE, 2021.
10

---

Published as a conference paper at ICLR 2023
Lucio M. Dery, Paul Michel, Ameet Talwalkar, and Graham Neubig. Should we be pre- pre-training? an argument for end-task aware training as an alternative. In International Conference on Learning Representations, 2022. URL
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12(Jul):2211-2153 2011.
Angela Fan, Edouard Grave, and Armand Joulin. Reducing transformer depth on demand with structured dropout. In International Conference on Learning Representations, 2020.
Alex Graves, Santiago Ferúandez, Faustino Gomez, and Jürgen Schmidhuber. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning, pp. 369-376, 2006.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 16000-16009, 2022.
Yosuke Higuchi, Niko Moritz, Jonathan Le Roux, and Takaaki Hori. Momentum pseudo-labeling for semi-supervised speech recognition. Proc. Interspeech, 2021.
Yosuke Higuchi, Niko Moritz, Jonathan Le Roux, and Takaaki Hori. Advancing momentum pseudo- labeling with conformer and initialization strategy. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7672-7676. IEEE, 2022a.
Yosuke Higuchi, Niko Moritz, Jonathan Le Roux, and Takaaki Hori. Momentum pseudo-labeling: Semi-supervised asr with continuously improving pscudo-labels. IEEE Journal of Selected Topics in Signal Processing, pp. 1-14, 2022b. doi: 10. 10.11009SSSP.222.31955531
Wei-Ning Hsu, Yao-Hung Hubert Tsai, Benjamin Bolte, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: How much can bad teacher benefit asr pre-training? In ICASSP 2021- 202/ IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. IEEE, 2021.
Kexin Huang, Vishnu Sresht, Brajesh Rai, and Mykola Bordyuh. Adaptive pseudo-labeling for quan- tum calculations, 2022. URL.ittps.//openeevvee.. wtwp:rrwwwwwweeeeeoononn.o.o
Jacob Kahn, Ann Lee, and Awni Hannun. Self-training for end-to-end speech recognition. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7084-7088. IEEE, 2020a.
Jacob Kahn, Morgane Rivière, Weiyi Zheng, Evgeny Kharitonov, Qiantong Xu, BeerneEEmmnuel Mazaré, Julien Karadayi, Vitaliy Liptchinsky, Ronan Collobert, Christian Fuegen, et al. Libri-light: A benchmark for asr with limited or no supervision. ICASSP 2020-2020 IEEE International Conference Acousties, Speech and Signal Processing (ICASSP), pp. 7669-7673. IEEE, 2020b.
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. arXiv preprint arXiv:1610.02222. 2016.
Dong-Hyun Lee. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop challenges in representation learning, ICML, volume 3, 2013.
Tatiana Likhomanenko, Qiantong Xu, Jacob Kahn, Gabriel Synnaeve, and Ronan Collobert. slimipl: Language-model-ffee mrdel-free iterative pseudo-Labeling. Proc. Interspeech, 2021a
Tatiana Likhomanenko, Qiantong Xu, Gabriel Synnaeve, Ronan Collobert, and Alex Rogozhnikov. Cape: Encoding relative positions with continuous augmented positional embeddings. Advances in Neural Information Processing Systems, 34, 2021b.
Tatiana Likhomanenko, Ronan Collobert, Navdeep Jaitly, and Samy Bengio. Continuous soft pseudo- labeling in asr. In I Can't Believe 's Not Better Workshop: Understanding Deep Learning Through Empirical Falsification, NewrIPS, 2022.
11

---

Published as a conference paper at ICLR 2023
Vimal Manohar, Tatiana Likhomanenko, Qiantong Xu, Wei-Ning Hsu, Ronan Collobert, Yatharth Saraf, Geoffrey Zweig, and Abdelrahman Mohamed, Kaizen: Continuously improving teacher using exponential moving average for semi-supervised speech recognition. arXiv preprint arXiv:2/00600759, 2021.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5206-5210. IEEE, 2015.
Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoc Le. Specaugment: A simple data augmentation method for automatic speech recognition. Proc. Interspeech 2019, pp. 2613-2617, 2019.
Daniel S Park, Yu Zhang, Ye Jia, Wei Han, Chung-Chent Chiu, Bo Li, Yonghui Wu, and Quoe V Le. Improved noisy student training for automatic speech recognition. Proc. Interspeech 2020, pp. 2817-2821, 2020.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot rext-to-image generation. In International Conference on Machine PMLR, 2021.
Learning, pp.
Mchdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transfor- mations and perturbations for deep semi-supervised learning. Advances in neural information processing systems, 29, 2016.
Scudder. Probability of error of some adaptive pattern-recognition machines. IEEE Transactions on Information Theory, 113)3363-371, 1965.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pp. 464-468, 2018.
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Do- gus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying supeiiuperviee learning with consistency and confidence. Advances in Neural Information Processing Systems, 33, 2020.
Gabriel Synnaeve, Qiantong Xu, Jacob Kahn, Tatiana Likhomanenko, Edouard Grave, Vineel Pratap, Anuroop Sriram, Vitaliy Liptchinsky, and Ronan Collobert. End-to-end asr: from supervised to semi-supervised learning with modern architectures. In Workshop on Self-superviste in Audio and Speech, ICML., 2020.
Chaitanya Talnikar, Tatiana Likhomanenko, Ronan Collobert, and Gabriel Synnaeve. Joint masked cpe and ctc training for asr. In /CASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3045-3049. IEEE, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Qiantong Xu, Tatiana Likhomanenko, Jacob Kahn, Awni Hannun, Gabriel Synnaeve, and Ronan Collobert. Iterative pseudo-labeling for speech recognition. Proc. Interspeech 2020, pp. 1006-1010, 2020.
Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. Advances in Neural Information Processing Systems, 14:18480--1819 2021.
Bowen Zhang, Songjun Cao, Xiaoming Zhang, Yike Zhang, Long Ma, and Takahiro Shinozaki. Censer: Curriculum semi-supervised learning for speech recognition based on self-supervised pre-training. arXiv preprint wr.iv:2206.08///// 2022.
Yu Zhang, James Qin, Daniel S Park, Wei Han, Chung-Cheng Chiu, Ruoming Pang. Quoe V Le, and Yonghui Wu. Pushing the limits of sumi-supereised learning for automatic speech recognition. arXiv preprint arXiv:2010. 10504, 2020.
12

---

Published as a conference paper at ICLR 2023
A DETAILS ON EXPERIMENTAL SETUP
A.1 SPEAKERS IN LIBRISPEECH
There no intersection between speakers different LibriSpeech train sets as well as in validation / test sets - all speakers are unique and are present in only one of the LibriSpeech sets. To prepare the 10h set we randomly sampled audio per speaker to gather a total 10h of audio.
A.2 ACOUSTIC MODEL TRAINING
We keep the original 16kHz sampling rate and compute log-mel filterbanks with 80 coefficients for a 25ms sliding window, strided by 10ms which are normalized to zero mean and unit variance per input sequence before feeding into model.
Throughout the paper we consider transformer based models with a convolutional frontend to perform the proper striding. The encoder is composed of 1-D convolution with kernel size and stride followed by 36 4-head Transformer blocks VVaswani et al.) 2017). The self-attention dimension is 768 and the feed-forward network (FFN) dimension is 3072 (with 4 heads) in each transformer block. The output of the encoder is followed by linear layer to the output classes. We use dropout after the convolution, dropout on the self-attention and on the FFN for all transformer layers, and layer drop (Fan et al., 2020). dropping entire layers at the FFN level.
We get rid of relative positional embedding (Shaw al., 2018) and use either sinusoidal one (Vaswani iet al. 2017) or recently proposed CAPE embedding [Likhomanenko et al. (0211) (only global shift of 30s used): this speeds up training by 2-3x and decreases memory usage.
For SpecAugment Park et al. (2019) we follow parameters from Likhomanenko et al. (021a): two frequency masks with frequency mask parameter F 30, ten time masks with maximum time-mask ratio p = 0.1 and time mask parameter T = 50; time warping is not used.
All models are trained with CTC loss and Adagrad optimizer with linear warmup period of 64k steps, constant learning rate of 0.03 and step-wise (by 2) learning rate decay at the end of training. All models are trained on tf32 tensor cores of 8 Ampere A100 40GB GPUs for a maximum of 500k updates.
For slimIPL parameters we use always cache size of Ik. Throughout the paper we vary the proportion X (by default we use À = 1 if not stated otherwise) well as Pout- From experiments observe that it is important to activate SpecAugment later in training (e.g. after 5k training steps) otherwise slimIPL baseline is even more prone divergence.
A.3 COMMON VOICE EXPERIMENTS
We use Common Voice data release from 21 July 2021 with French language. In total, there are 543 hours in train, 25.1h in validation and 25.8 in test sets. We randomly sample speakers from the train and take all audio belonging to the same speaker to form 100h train subset. We end up with 982 speakers and 102h. We further sample speakers from this 100h subset to form a 10h subset: contains 171 speakers with 11.5h. These 10h and 100h subsets are used as labeled data while the remaining 443h are used as unlabeled data. We normalize transcriptions by lower casing, removing any punctuation tokens except apostrophe, changing all diacritical marks to their corresponding English characters and removing any other non-English characters. Later, we use the same token set as for LibriSpeech.
We use the same acoustic model as for LibriSpeech experiments with sinusoidal positional embedding all audios in Common Voice very short (5.2% 1.5s). For fully supervised models we use dropout 0.5, 0.3 and 0.1 for 10h, 100h and 540h sets correspondingly. For slimIPI. we change dropout and layer drop from 0.5 0.1 for 10h and from 0.3 to 0.1 for 100h, while for our methods we use dropout and layer drop of 0.1 from the beginning of training. For slimIPL we tune only parameter M for the 10h setting. The rest of parameters are the same as in original slimIPL work (Likhomanenko et al. 2021a a C is 1000 (100), cache probability Pout is 0.1, data proportion À is 10 (3), M is 40k
https: http://wwwwoooooooooo////////a/a htt.:///.....aaaoooaaaaaaaaaaaa.
com wwwm://mmmmmomoo///////oo/omoooo wwt.:/icceeveaaeeeoeeeo
13

---

Published as a conference paper at ICLR 2023
(20k) for 10h (100h) setting. All models are trained with dynamic batch, same for LibriSpeech. For our methods we use exactly the same parameters as for LibriSpeech experiments with dynamic batch.
A.4 FULLY SUPERVISED MODELS
Table 9: Fully supervised models for 10h and 100h of LibriSpeech. Results are reported across random seeds. Sinusoidal, CAPE and relative positional embeddings are denoted as 'sinpos', 'CAPE' and 'relpos' correspondingly. The 10h set from Libri-Light is marked with
Model
Sup.
WER
dev-clean
dev-other
test-clean
test-other
relpos Likhomanenko 2021a
$2.3
32.6
52.4
CAPE
10h*
37.1o.1
58.40.1
37.7p.3
58.40.a
sinpos
76.00.s
87.1o.5
77.
87.2g.6
relpos
27.70.4
48. 18.4
28.2g.3
48.8p.a
CAPE
10h
28.2
58.5
28.96.1
48.9n.2
sinpos
63.4
78.50.9
64.5p.9
78.91.1
relpos Likhomannnko et al.
2021a
6.2
16.8
6.2
16.8
CAPE
100h
1.9
17.9
6.26.1
18.1p.1
sinpos
6.5o.5
19.1
7.1o.3
19.3n.2
A.5 SUMMARY OF HYYPER-PARAMETE
Plyper-paramedd values for both experiments on LibriSpeech and Common Voice are summarized in Tables 11
12 and 10
Table 10: Detailed typer-parameters for the final experiments on Common Voice from Table g
Parameter
slimIPL (10b)
Our (10h)
slimIPL (100h)
(100h)
M
40k
Ok
20k
Ok
C Post
1000
1000
100
1000
0.1
TER (1 after 130K)
0.1
TER (1 after 130K)
10
dropout/layer drop
0,5-0011
0.1
0.3-0.1 sinpos
0.1
embedding
sinpos
sinpos
0
TV m0x(0.1, 0.1 R/130, 000)
0
max
0.3 a 1/130, 000)
lotal hatch
dynamic 290sx8
dynamic 290xx8
dynamic 290xx8
dynamic 290xx8
Table 11: Detailed hyper-parameter for the final experiments on LibriSpeech from Table 7
Parameter
slimiPL (10b)
Our (1db)
stimIPL (1006)
Our (100b)
C
1000
1000
100
1000
5
3
dropout/layerdrop
0.5++0.1
0.1
0,3-+0.1
0.1
0
max
0.1 k/130, 000)
0
max 1. 1 0.1 k/1300,000 000)
embedling
sinpos
sinpus
sinpos 20k
sinpos Ok
M
30kk
Ok
Post
0.1
TER (1 after 130k)
0.1 dynamic 290sx8
TER (1 after 130k)
total batch
dynamic 290
8x8
dynamic 290
embedding
CAPE
CAPE Ok:
CAPE 20k 0%
CAPE OK Sk
Af
S0k
CAPE is used after
Ok
25k
Pont
0.1
TER (1 after 40k)
0.1 dynamic 290sx8
TER (1 after 130k)
total batch dynamic 290sx
dynamic 290s x
dynamic 290xx
B WAV2VEC AND SLIMIPL REPRODUCTION
To reproduce baselines in Table for slimIPL we follow Likhomanenko et al. (202 and its published recipe. The only change do is positional embedding discussed above and batch size.
14

---

Published as a conference paper at ICLR 2023
Table 12: Detailed syper--arameters for the final experiments LibriSpeech from Table 7 for 10h* setting.
(106")
Our (10h")
1000
1000
(... 0.1 A/130, 000)
SpecAugment
T
25, 20 time masks
50, 10 time masks
embesdding
sinpas
sinpos
dropous/layer drop
0.5 :0.1
0.5 (0.1 after 35k)
70k)
Pous
0.1 dynamic 290sx8
TER after 70k) 8x8
total batch
embodding
CAPE 40k
CAPE Ok
M
CAPE used after
Ok
70k
(5 after 130k) 0.5 (0.1 after 70k)
dropouslaayedropp
0.5 -0.1
Pous
total hatch
dynamic 290xx8
8x8
The rest of the training remains the same. To reproduce wav2vec 2.0 (Baevski al. 2020) we take open-sourced Large model pre-trained on the full LibriSpeect and then perform fine-tuning on our 10h set and the 10h set from Libri-Lighh Light. For fine-tuning 10
sour-sourced configurations for We fine-tune models on 24 GPUs as specified in Baevski et al. (2020) for 3 different seeds.
ABLATIONS: SAMPLING FOR LARGER BATCHES C
Table 13: Comparison (in WER) between different temperatures 7 for sampling when large batch and longer training (600k) are used.
dev-clean
dev-other
test-clean
test-other
0 (argmax)
19.1
26.7
19.3
27.8
+ 0.1
13.9
17.5
13.8
18.0
10
200
Poe-1
Aoe D.l
C8
ob 0000000rr000 WER
C8
2
60
64-
20
100000
250020
20000
40350
60200
50200
fmraiion
(b) Comparison between models trained with dif- ferent pout! constant (blue) or 0.1 (orange), or scheduled 0.1 (green).
(a) Evolution of Pout for the different curriculum selection strategies.
Figure 3: Analysis of the probability Pout-
"Released at fihhtpss: //11 http://pw.....tttpote...ee/////r
They are availble at http://///ghthhcoo///////ee/ee/r o2tppp/is/o//ee/eeoooofffffaaann
15


Reviews:
[{"review_id": "p94hWQwcNH", "reviewer": "Reviewer_ZfEP", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes novel changes to an existing self training algorithm for ASR such that performance is improved without a pretraining step. Specifically, the proposed model generates pseudo labels (PL) for unlabelled data from the very start of supervised training and uses it to augment the dataset. This is done by dynamically evolving a cache of data by using a simple yet effective filtering criteria for unlabelled examples. Furthermore, to avoid instability in the proposed PL generation from the start an alignment sampling is used which samples tokens from the predicted distribution instead of an argmax. Results show improvement over previous work.\n", "summary_of_the_review": "This paper contributes to the field of self training for ASR models. The overall presentation of ideas is good and the proposed techniques are simple yet effective. The authors conduct extensive ablation studies to back their claims. However, the paper could benefit by explaining some intuition behind the alignment sampling criteria. The fact that the authors do not comment on non-CTC based models and that w2v2.0 still works better on 2 out of 4 test sets, does weaken the paper.", "strength_and_weaknesses": "Strengths:\n\n1) The overall presentation and flow of ideas is good. The paper does a good job in setting up the motivation for the work by presenting some early results.\n2) Extensive ablation studies over each component of the proposed methodology helps the paper.\n3) Results are compared with comprehensive baselines.\n4) Details about the model architecture and other results included for reproducibility.\n\nWeaknesses:\n\n1) The major part of this paper builds over the previously proposed slimIPL. Yet, not enough explanation is given about slimIPL in the main text. This might be inconvenient for a reader not conversant in the algorithm. Having a full algorithm in the main text will help the paper.\n2) The algorithm for slimIPL (appendix A) confused me. In the 3rd step, when the cache is being initialized, why is there an update step using (x,y) in L? Hasn’t that already been done prior to generating the cache? Or is this a typo?\n3) It is not very clear why alignment sampling should work. The authors say that it provides effective stabilization by adding noise to the targets. But the targets are noisy anyway during the start of training as the TER is close to 100% initially. So why would sampling produce targets that are any different? There is not enough explanation for this.\n4) All results are shown on CTC based models. Do these techniques translate to encoder-decoder based ASR models like LAS and RNN-T which are very popular? The authors do not comment on this.\n5) In the final results, compared to w2v 2.0 results, the proposed techniques work better only on dev-other and test-other but not the clean subsets using 10h of speech. Why should the proposed method be chosen over w2v2.0 ?\n6) Minor issues: The caption for figure 1 can be more informative. What does “CAPE” and “sinpos” mean in table 7?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Lowos2PT0Z", "reviewer": "Reviewer_fsR2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a method to self-train ASR systems directly, without requiring an initial pre-training stage, which is especially useful in low-resource setups. Similarly to an existing method, slimIPL, the proposed approach generates pseudo-labels as training progresses and maintains a cache of pseudo-labels, regularly updated. The main differences are that the proposed method does not require a first pre-training step, that the pseudo-labels are updated when they are used before putting them back in the cache, and that the probability of removing a pseudo-label from the cache is computed from the difference between the older and updated labels. Additionally, the pseudo-labels are sampled from the predicted token distributions, whereas slimIPL generates 1-best labels. \n\nThe method is evaluated on LibriSpeech with two simulated setups corresponding to 10 and 100h hours of labeled data. The impact of the different hyper-parameters and proposed techniques and tricks are evaluated and the results show a more stable training and improved word error rates.", "summary_of_the_review": "The paper is quite good, nice to read and well explained. The motivation for this idea is interesting and solves a real issue. I think this is worth sharing with the community, but I would like to see more results for different scenarios than just LibriSpeech.", "strength_and_weaknesses": "Strength: The idea is interesting and the results are good. Self-training from the start is indeed useful. The proposed improvements of slimIPL are relevant and well motivated and analysed.\n\nWeaknesses: The proposed method is only evaluated on setups simulated from LibriSpeech. There is no comparison with training with all the labeled data of LibriSpeech. Moreover, the 10h scenario where the proposed approach beats other methods does not look like the one usually explored for this problem: this could be motivated more in the text. The approach described in this paper is not as good as prior work on usual 10h used.  It would also have been interesting to see some results for actual low-resource scenarios.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "EhKs4fbkNld", "reviewer": "Reviewer_6PJR", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "**Summary**\nThis paper considers continuous self-training for ASR. This paper build on the observation that a previous method slimIPL performance degrades as the number of pretraining step M increase. The authors hypothesize that pretraining would cause overfitting when the supervised data is limited. To address this mentioned concerns, the authors propose self-training from the beginning of process. The authors introduces a series of tricks to improve the robustness and convergence of slimIPL.\n- regenerating PL when returning a sample to cache\n- dynamically compute the returning probability with Levenshtein edit-distance\n- sampling PL (with temperature scheduling) instead of using 1-best\n\nThe optimal recipe achieve similar result with the original paper on 100h dataset but much better on 10h dataset.", "summary_of_the_review": "accept", "strength_and_weaknesses": "**Strength**\n- The experiment design is very well motivated and executed.\n- The resulting method is both simple in recipe (1 stage training), also achieve significant WER reduction on 10h dataset.\n- The paper writing is extremely clear.\n\n**Wakenesses**\n- Limited novelty.\n- Absence of hyperparameter table in the appendix.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "10XJwn5snY", "reviewer": "Reviewer_bgqJ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper extends previous work on pseudo labeling for ASR (slimIPL to be specific) and achieve similar performance while much stable training process without even using the initial supervised training.\nGenerally, pseudo-labeling has been proved to be very valuable to industry applications of ASR because usually industries have labeled data for bootsrapping, and also have huge amount of unlabelled data for generation of pseudo labels. So the work in this study could be of great value to real applications.", "summary_of_the_review": "Overall, I recognize the value of value of this paper as further improvement in the research about PL for ASR, though marginal. I hope more (realistic) evaluations could be done to bring more benefits to the community.", "strength_and_weaknesses": "Strength:\n1. Analyzed problems of previous methods (training unstable, easy to diverge) and proposed corresponding solutions (PL evolution and alignment sampling). The solutions are proved to be effective through experiments on Librispeech dataset.\n\nWeakness\n1. The writing style is kind of confusing: could be better at explaining the main process of the proposed method and the difference with and improvements over slimIPL.\n2. The results in table 5 and 6 use different training hyperparameters compared to table 7. Not sure why this inconsistency exist but it could be clearer comparison under the same exp settings.\n3. This line of research (PL for ASR) all uses Librispeech for evaluation. I think this limits the potential value for real scenario. The most suitable scenario of this technique is where we have some labeled data to train an initial usable model, then we utilize unlabeled data collected before hand or even periodically collect unlabeled data from the product, to which the ASR system is applied. In this case, the problem could be much more complex, e.g. the distribution of the initial labeled data could be different from the unlabeled data collected later, or even the unlabeled data have different distribution at different time steps. The conclusion drew from this paper (or other PL related studies) could not be suitable for this scenario. I believe if the authors add exps on more real data, the conclusion could be more convincing and thus more contribution to the community (like studies in https://www.isca-speech.org/archive/interspeech_2022/baby22_interspeech.html, https://arxiv.org/abs/2207.09078)\n4. The alignment sampling methods assume a CTC decoder, which may not be applicable to RNN-T and LAS ASR models.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster