Self-Supervised Speech Recognition via Local Prior Matching
Authors: Wei-Ning Hsu
Ann Lee
Gabriel Synnaeve
Awni Hannun
Venue:  (2020)
Keywords: speech recognition
self-supervised learning
language model
semi-supervised learning
pseudo labeling

Abstract: We propose local prior matching (LPM), a self-supervised objective for speech recognition. The LPM objective leverages a strong language model to provide learning signal given unlabeled speech. Since LPM uses a language model, it can take advantage of vast quantities of both unpaired text and speech. The loss is theoretically well-motivated and simple to implement. More importantly, LPM is effective. Starting from a model trained on 100 hours of labeled speech, with an additional 360 hours of unlabeled data LPM reduces the WER by 26% and 31% relative on a clean and noisy test set, respectively. This bridges the gap by 54% and 73% WER on the two test sets relative to a fully supervised model on the same 360 hours with labels. By augmenting LPM with an additional 500 hours of noisy data, we further improve the WER on the noisy test set by 15% relative. Furthermore, we perform extensive ablative studies to show the importance of various configurations of our self-supervised approach.

Under review as a conference paper at ICLR 2020
SELF-SUPERVISEE SPEECH RECOGNITION VIA LOCAL PRIOR MATCHING
Anonymous authors Paper under double-blind review
ABSTRACT
We propose local prior matching (LPM), a self-supervised objective for speech recognition. The LPM objective leverages strong language model provide learning signal given unlabeled speech. Since LPM uses a language model, it can take advantage of vast quantities of both unpaired text and speech. The loss is theoretically well-motivated and simple to implement. More importantly, LPM is effective. Starting from a model trained on 100 hours of labeled speech, with an additional 360 hours of unlabeled data LPM reduces the WER by 26% and 31% relative on clean and noisy test set, respectively. This bridges the gap by 54% and 73% WER on the two test sets relative to fully supervised model on the same 360 hours with labels. By augmenting LPM with an additional 500 hours of noisy data, further improve the WER the noisy test set by 15% relative. Furthermore, we perform extensive ablative studies to show the importance of various configurations of our self- supervsee approach.
INTRODUCTION
Fully supervised learning remains the mainstream paradigm for state-of-the-art automatic speech recognition (ASR). These systems require huge annotated data sets (Li et al. 2017 Chiu et al.) 2018 Hannun al. 2014 Amodei et al. (016)) which are time-consuming and expensive to collect. This hinders the development of accurate ASR systems for low resource languages (Precoda 2013). In fact, out of over 6,000 spoken languages, fewer than 150 are supported by commercial ASR service providers. In sharp contrast to how we teach machines to recognize speech, humans do not learn by listening to thousand hours of speech and simultaneously reading the corresponding transcriptions. Instead, as noted in (Chomsky, 1986; Kuhl 2004 Glass] [012] Dupoux 2018), humans possess an inherent ability to learn from vast quantities of unlabeled speech. Consider the case of conversing with someone with strong accent. Even when the speaker pronounces several words in an unusual way, one can often correctly understand the sentence. We argue that the source of indirect supervision in processing unlabeled speech comes from prior knowledge about the world and the context of the speech.
Inspired by this, we devise self-supervised learning framework termed local prior matching (LPM). We apply this framework to speech recognition allowing an ASR model to lear from unla- beled speech by leveraging a strong language model, which serves as the prior for self-supervision. Given an unlabeled utterance, the ASR model proposes multiple hypotheses and the language model provides a learning signal by evaluating the plausibility of each one.
We evaluate the LPM method on the LibriSpeech corpus (Panayotov et al. 2015), using 100 hours of labeled speech seed the proposal model. Using 360 hours of additional labeled data reduces the word етrоr rate (WER) by 3.8% and 13.1% absolute on an casier and more-chin more-challenging test set, respectively. Using the same 360 hours but without labels, LPM reduces the WER by 2.1% and 9.5% absolute on the same two test sets, effectively bridging the gap to fully-supervised learning by 54% and 73%. In addition, by augmenting LPM with another 500 hours, for total of 860 hours of unlabeled speech, LPM reduces WER on the noisier test set by 14.2% in total. Hence, LPM is able to surpass the performance of using 360 hours labeled data by taking advantage of about twice the amount of unlabeled data. We also conduct extensive ablation studies and analyses in order to demonstrate the significance of each proposed component. For reproducibility, software will be open source and made publicly available in the camera-ready version.

---

Under review as a conference paper at ICLR 2020
2 METHOD
2.1 PRELIMINARIES
Let x denote an utterance of speech and y be a transcription. We assume speech generated following two-step process:
y
X Pxly
(1)
where the text y is first generated from the language model (LM) py and the speech x then generated from text-to-speech (TTS) model Pxly conditioned y. The posterior Py/x is then the ASR model of interest. For convenience, we refer to Pxy (x. = Py(y) Pxly (x y) as the joint distribution, and (x( = Ey Pxy (x,(x,)) as the marginal distribution. In typical su- pervised learning setting, one has access to a labeled dataset Dj, which contains paired samples ((=()) ) drawn from the joint distribution. An ASR model gy|x can be trained by minimiz- ing the marginal-weighter KL-divergence: [ (Pylx ll gylx)], which can be estimated with E(aa)eD log (y)x(y æ²) + const from samples (i.e. the commonly used cross-entropy loss). In the self-supervised setting we only have access to unlabeled speech dataset Dú yet we still wish to optimize the same marginal weighted KL-divergence. Since we do not have samples from Py|x== directly, we can, under mild assumptions, estimate it.
Assumption Different y can lead to the same pronunciation. One reason is the tokenization of words may not be unique. For example y can be the sequence of word "c at." or "ca t_", both of which result in the same spoken word. Homophones (e.g. "right" and "write") also have the same pronunciation. To formalize this property, let f Y - Z be function which maps a text sequence to its pronunciation z E Z (a broader definition of "pronunciation" here, where = f(y) is a distribution over phoneme sequences for given text y), and let Pxly (x y) Px|x fv)). We let the set Y, {y f(y)= = =} C Y contain all text samples with the same pronunciation and hence the same conditional distribution, Px|z=x
Assumption We assume that two pairs (y, y') that have different pronunciations (i.e., f(y) + (3y)) either sound similar or are very different. For instance, "lose" and "loose" result in almost the same pronunciation whereas an English speaker would rarely mistake "winter coming" for "you know nothing". In the former case, the conditional distributions Pxly y) and Pxly (x y²) may overlap substantially whereas in the latter case they barely intersect. More formally, for each z, we assume the pronunciation space Z can be partitioned by acoustic similarity as Z+ UZ For all z zž. the conditional distributions are similar to that of z, such that DKL. (Pxlz=z || Pxl==>-) S e for small e. On the other hand, we assume the set of dissimilar pronunciations 2g contributes very little the marginal probability of non-outlier speech samples. In other words, for any and where Pxja (x 2 0, we assume Σn EZ Ex-ey, EY, (x Py( < (x)
Canditinnl Distriuttion eix (2)
Joint Distritution pic y)
Posterior Distribution (0
Proposod Approximation Bly (x)
0.5
p0x (2)
20r
16
2.c
põe
60x yol
pln: (x]
ply.
5.000
ply: 1.0
0.10
6.000
plys 1x0
15
15
15
Figure 1: Illstrationn of the assumed generative process and the proposed posterior approximation. The leftmost figure shows Pxiz of three pronunciations: Za, 26 and Ze, where Za: 26 E Z and
z. E Z The following two figures depict the joint and the posterior distributions of four texts, respectively, where it is assumed f(y) (y2) Za, f(ys) = zb. f(44) z., and Py (y()) > Py (Wa) > Py > Py (ya). The rightmost figure presents local prior, our proposed approximation to the py x computed from the priors of texts whose Pxiy are above the threshold d at a given æ.
2

---

Under review as a conference paper at ICLR 2020
2.2 LOCAL PRIOR MATCHING
Equipped with the above assumptions, we next present tractable approximation of the posterior Py x using unlabeled speech data. Figure]]shows toy example of the proposed approach. Let be a non-outlier speech sample that is generated conditioned on a text sample y" whose pronunciation
fly") z=
The posterior can be derived with Bayes' rule as follows:
(y Pxly (xr y)
Py( Pxly (xr y)
Py|x(1 x))
(2)
E BEY Py (g) Pxly (xr y)
Σ
Σ
Py Py( Pxly y)
REZ
DEY²
Py (v) Pxty (x: y)
Lu sExt Y= (3)
(3)
Σ Σ Py (i) Pxly ÿ)
$ DEV2
By (y( Px|z(2 f(v))
1u #EZ Ya (z).
(4)
Σ Σ Py (i) (x|x(( 2)
2E DEY2
The set Y can be partitioned according to pronunciation to arrive at equation 2 which is then ap- proximated using by remark 2 to get equation 3 where 1 is the indicator function. In equation Pxly(e (x: ÿ) is simply replaced with Pxie(#) x (Ø)) as stated in remark Despite the simplifica- tion, estimating equation 4 requires a language model, TTS model, and the enumeration of text sequences with similar pronunciations to that of z" To avoid the latter two requirements, we make the following approximations:
1. Replace Px|a(< | 2) with Px|a(< =*) for all Z by exploiting the assump- tion that these distributions are very similar to each other (i.c., from remark 2 we have (Px/x=== Px(x=s) e).
2. Replace UsEzt. Yz with hypotheses proposed from a beam search using a bootstrapped ASR proposal model. We refer to the proposal model as Tylx and the of hypotheses for generated from the beam search Brry ar k), where & is the beam size.
The approximated posterior Py x)) can then be written as:
Py Px Pxiz (x 2²)
(y x)
E EgeB(r, x R.) Py (8) Px|a(< 2*) 1 B(ry)x
(5)
Py
1 B(rr
(6)
Σ Egen(y Biy Py (i)
The approximated posterior only requires computing language model probabilities of the beam search hypotheses, which is tractable. We refer to equation 6 as the local prior, since it is a re- normalized distribution in the neighborhood of y² We also propose local prior matching (LPM) as an unsupervised objective for training an ASR model 9y|x:
Cipm (y x, Tylxs k) = DKL
= -
Σ
(y
log gy bx x),
BEB(TY k.) Biy Py
which minimizes the KL--divergence between the local prior and the model distribution. For clarity, we term Qyix the online model, and let dyix(y x; 0g) and Tylx(y x; 0z)) denote the models that are parameterized by and 0, and 0,, respectively. Since the LPM objective simply re-weights hypotheses by the local prior, we do not need a continuous relaxation of y to make Py differentiable with respect to 0g as in Yang et al. (2018); Liu et al. (2019). The gradient of the proposed LPM objective is given by: 2 (ylx: Py T xxxx,)
Σ
Py (
i log gylx xx)
a0,
E EgeB(ry b)) Py B(
00,
We use paired audio and text to estimate Tylx and unpaired speech compute the LPM objective. The LPM objective can also take advantage of large unpaired text corpora in order to obtain better language models which directly improve the quality of the approximate posterior.
3

---

Under review as a conference paper at ICLR 2020
2.3 PROPOSAL MODEL
The quality of the posterior approximation Pylx depends on the proposal model Tylx- Instead of using a fixed proposal model throughout the entire training process, we update Ty/x with gylx if the latter is better.
On-policy beam search The first approach always uses the online model gylx as the proposal model. This means B, 6g and is effectively form of on-policy beam search, since the model used to generate hypotheses is also the model we update.
Off-policy beam search While the on-policy method benefits from the immediate improvement of the online model, it also suffers immediately if the gradient update from a mini-batch deteriorates performance. This can result in instability during optimization. We consider second option which does not tie 0, and 0g but instead updates 0, with 8, every T steps only when the performance of the online model gy is better than that of the proposal model ry|x by some metric. We refer to the second option as off-policy beam search. To avoid overfitting to the training set, we use the character error rate (CER) on the validation set as the metric for the proposal model update. We set T 1000 for all experiments with the off- off-policy beam search.
2.4 FILTERING HYPOTHESES USING ESTIMATED LENGTIIS
As noted in Chorowski & Jaitly (017), sequence-tor -oseeneee----euuenee ASR models sometimes predict end- of-sentence (EOS) tokens too early or generate looping n-grams, resulting in hypotheses that are significantly shorter or longer than the set of acoustically matched texts for a given utterance. Of the two failure modes, the former more harmful to training with LPM. The reason is that the LPM objective assumes all hypotheses obtained from beam search are acoustically reasonable, and weights each of them by linguistic plausibility given by an LM. While LMs are effective in discrim- inating plausibility between sentences of similar lengths, we find empirically they tend to assign higher probabilities to shorter sentences than to longer sentences, even when the longer ones are more plausible and grammatically correct than the shorter ones. As result, truncated hypotheses are assigned higher weights than those that are acoustically matched but longer, which in turn en- courages earlier prediction of EOS tokens and forms a catastrophic feedback loop particularly with the on- on-policy beam search.
To address this issue, we propose a simple filtering heuristic based on the text length. Before training the model, a text length L. estimated for each unlabeled speech sample æ. During training, only hypotheses with length close to L are retained for the LPM objective computation. Let len(y) denote the length of y. We keep a hypothesis only if [ris L] < len(y) S [rus L], where rus and Tub are the text length lower and upper bound ratios, respectively. Several methods can be used to estimate the text length on an unlabeled utterance, including using the average speaking rate (Peng et al., 2019) or generating a phoneme/syllable segmentation (Adell & Bonafonte] [2004] Scharenborg et al. 2010 Wang et al.) 2017). In this work, we estimate the length by using using that of the best hypothesis generated from the initial proposal model, generated with either ASR-only greedy decoding or ASR+LM beam search decoding.
EXPERIMENTAL SETUP
Dataset We evaluate our approach on LibriSpeech (Panayotov et al. 2013), a crowd-sourced audio book corpus derived from the Vox Project) The training set contains 960 hours of speech, officially split into three sets: train-clenn 100, train-clean-360, and ttain-other-500, where the first two sets are easier and the third set is noisier and more accented, Similarly, the development and test sets are also split according to difficulty, resulting in four partitions: {dev, test) x (clean, other), each of which contains roughly five hours of speech. In this work use train-clean- 100 as the paired speech data, and the other two training splits as the unpaired data.
We train the language model on the unpaired text data provided with LibriSpeech, which includes approximately 14,500 books collected from Project Gutenberg Some of the books in the text corpus overlap with those in the LibriSpeech training set. To avoid training the LM on the ground truth text of the unlabeled speech, we exclude the 997 overlapping books from the text data. We follow the same recipe as Kahn et al. (2019) pre-process the remaining text.
4

---

Under review as a conference paper at ICLR 2020
Neural Network Architecture The proposal model Tylx and the online model gyix are sequence- to-sequence neural networks (Bahdanau et al. 2016 Choroowki & Jaitly/ 2017) with the same time- depth separable (TDS) architecture proposed in Hannun et al. (2019). The encoder is fully convolu- tional, composed of TDS blocks which reduce the number of parameters while keeping the receptive field large. The decoder is single layer recurrent neural network (RNN) with gated recurrent units (GRUs), equipped with a single-headed inner-product key-value attention (Vaswani et al. 2017) for querying information from the encoder outputs. We follow the recipe of Kahn et al. (2019) which uses fewer TDS blocks in the encoder compared to [Hannun et al. (2019) in order to generalize better when trained on the smaller LibriSpeech train-clean- 100. The output target of the decoder at each step is posterior distribution over 5,000 word pieces, generated with the SentencePiece toolkit (Kudo & Richardson, 2018) using transcripts from train-clean-100
To enable efficient evaluation of the language model probabilities, which is required at each training step, we use the gated convolutional language model architecture (ConvL.) proposed in Dauphin et al. (2017). which achieves competitive performances compared to recurrent models while sig- nificantly reducing the latency. We use the same 5,000 word-piece vocabulary for the LM which is trained with the same model configuration and recipe as Zeghidour et al. (2018) The trained ConvLM achieves a token perplexity of 34.24 on the development set.
Optimization We use both paired and unpaired data to optimize qy x- To simplify the optimiza- tion procedure, the model is provided with either a paired or an unpaired batch at each step, al- ternated with fixed ratio my : my. When given a paired batch of n samples, {(x(() )== the model minimizes the standard cross-entropy loss, A Σ. - log (y)x(1())
x0)).. When
provided with an unpaired batch (x=0) the model minimizes weighted LPM loss, a E; fipm (gylx; (+),, Py:''ylx "ylx: k). The weight and the mixing ratio mı/m. are used to balance the supervised and self-training objectives. For regularization we use 20% dropout (Srivastava et al., 2014) 10% label smoothing, 1% decoder input sampling, and 1% word piece sampling (Kudo, 2018) following Kahn et al. 2019). We use SGD without momentum to train the online model with an initial learning rate of 5e-2. To achieve good CER on the development sets, the model is trained for at least 1.6M steps (paired and unpaired ones combined) with batch size of 16 (8 GPUs x 2 per GPU). The learning rate is annealed by factor of two every 0.64M steps. All experiments in this paper are implemented in the way2letter++ framework (Pratap et al. 2018).
Initialization To initialize the proposal model and the online model, we consider three checkpoints from a baseline model trained on train-clean- 100 for varying number of steps using only the supervised objective. The three checkpoints, whose parameters are denoted as OA, Op, and Oc, are trained for about 300k / 40k / 16k steps, achieving average development set CERs of 13% / 20% / 38%, respectively. To obtain a better approximation Pyjx for the posterior, it is desirable to use the best-performed checkpoint et al. (2019) that training from scratch achieves consistently better performances than starting from well-trained model, we hypothesize that initializing the online model with earlier checkpoint may also have a similar effect. Hence, we initialize 0,- 0A and 0g- = Oc if not otherwise specified.
initialize the proposal model. By contrast, as observed in Kahn
4 RESULTS
The best supervised model trained only on train-clean- 100 (0 achieves a WER of 14.00% .02% on dev-clean/dev-other, respectively. Unless otherwise stated, we use train-clean-23 360 as the unpaired speech dataset, (ru, rub) (0.95, 1.05) for length filtering with reference lengths obtained from ASR-only greedy decoding.
4.1 BEAM SIZE, MIXING RATIO, AND LPM WEIGHTS
Table D shows how the WER varies with the beam size and the mixing ratio. for all mixing ratios, the model improves the most from beam size of k to k = 2, showing the benefit of considering multiple hypotheses. The improvement is greater when a higher mixing ratio of unpaired-to sapaired-to-pairee to-paired speech is used. In addition, probability assigned to each hypothesis is normalized within the beam. If there is only one hypoth- esis, it will be assigned approximate posterior probability of one. The amount of improvement
note that the LM is effectively unused when k = 1 because the LM
5

---

Under review as a conference paper at ICLR 2020
Table 1: Vary mixing ratio and beam size, An LPM weight 0.2 is used.
dev-clean dev-other WER (%) k:
D
= 1
==
k ==
= 16
1:1
10.75 / 31.62
10.60 30.96
10.25 30.67
10.14/ 30.17
10.09 / 29.99
1:1
10.43 / 29.76
9.56/ 28.83
9.37 / 18.10
9.06 27.35
8,88 27.25
360hr
1:4
11.09/
11.09/29.89 29.89
9.34 2.44/27. 27.45
9.00 26,47
/.15/26.52
9.36 / 27.00
1:9
12.11 30.89
10.11 1.11/270 /2..71
9.76 / 27.08
10.17/27.41
10.30 27.62
860hr
1:4
10.59/ 26.05
9.37 23.85
8.68 / 22.53
8.37
8.37 / 21.33
diminishes with larger beam sizes, and the performance even starts to degrade beyond A: = 4 when using higher mixing ratio. This may result from the inclusion of worse hypotheses which have better scored under the LM. We use the best setting for the following experiments with mixing ratio ri Tu 4, a beam size 4, and an LPM weight a 0.2. We present detailed results varying the LPM weight in the Appendix.
4.2 PROPOSAL MODEL UPDATE AND MODEL INITIALIZATION
In addition to the two proposal model update strategies proposed in Section 2.3
termed on-policy and off-policy (better), we experiment with two additional strategies. The first, off-policy (never), uses fixed proposal model throughout training without ever updating. The second, off-policy (al- ways), updates the proposal model with the online model every steps (i.e., set 0, + 8,) regardless of the performance.
The full results are shown in Table Four key takeaways are as follows. (1) For all combinations of (rylx= gy x) initialization, off-policy (never) is the worst. This demonstrates the importance of updating the proposal model to generate better hypotheses during training. (2) Off-policy (always) consistently outperforms on-policy. We observe that training is significantly stabilized by reducing the proposal model update frequency from every step to every 1,000 steps. The effect is particu- larly prominent when initializing Tyix and qyix from an earlier checkpoint (9.62% vs 20.02% on dev-clean, and 27.51% % vs 45.62% on dev-other). (3) Off Off-policy (better) achieves the best WER in all settings outperforms off off-policy (always) by a larger margin when initializing from an earlier checkpoint. (4) Unlike the other strategies, off-policy (better) demonstrates consistent improvement when using a less-trained initial online model. In the following experiments, we initialize models with 0, = A and 0, 0c unless otherwise specified.
Table 2: Vary proposal model update strategies and online model initialization.
dev-clean dev-other WERs (%) Init Qy
Init ryix
Tylx update
Init dylx = A
Init qyix =
C
On-Policy
9.50 28.29
N/A
N/A
Off-Policy (never)
11.19 / 31.74
11.14/ 31.69
11.24/ 31.53
A
Off-Policy (always)
9.40 27.79
9,27 27.33
9.52 / 27.34
Off Off-Policy (better)
9,20 27.42
9.14 26.80
9.00 26.47
On-Policy
N/A
10.17 / 28.35
N/A
Off-Policy (never)
13.61 / 35.39
9.26/27.01 13.95 / /35.43
13.56 35.62
Off-Policy (always)
9
9,56/27
9,79 27.44
Off-Policy (better)
9.30/
9.15 26.63
On-Policy
N/A
N/A
20.20/ / 45.62
Off-Policy (never)
20.59 44.09
22,95 46.43
23.42 46.89
C
Off-Policy (always)
9.52/27.89
9.46 / 27.35
9,62/2755 27.51
Off-Policy (better)
9.44/27.34
9.31 / 27.26
9,43 / 27.19

---

Under review as a conference paper at ICLR 2020
4.3 LENGTH FILTERING
Table shows the impact of length filtering with different proposal model update strategies, where both the proposal model and the online model are initialized with OΛ. As discussed in Section 2.4 on-policy suffers more than off-policy without length filtering. If the proposal model is never up- dated, then length filtering does not affect the final WER. We hypothesize that length filtering keeps the proposal model stable during training. Table 4 compares three different reference lengths: the oracle length (Oracle), the predicted length from ASR LM beam search decoding (ASR L.M Dec), and the predicted length from ASR-only greedy decoding (ASR-only Dec). We observe that the WER does not differ much when using different reference lengths for filtering.
Table 3: Ablation study for length filtering.
Table 4: Length filtering using different ref- erence lengths.
dev-clean / deverther WER (%)
fyix update
No filtering
With filtering
Reference Length L
dev-(clean/ / other)
On-Policy
26.65 59.07
9.50 28.29
Oracle
8.85 / 26.39
Off-Policy (never)
11.18 31.83
11.19 /1.774
Off-Policy (always)
13.99 35.52
27.79
ASR LM Dec
8.99 / 26.36 9.00 / 26.47
Off-Policy (better)
11.42 31.56
9.20 27.42
ASR-only Dec
4.4 CHOICE OF LANGUAGE MODELS
We study how the LM affects the performance by experimenting with three alternative hypothesis weighting methods. The first method uses the LM score normalized by token length, Py (3) ( 1/tem(y) which avoids favoring overly short hypotheses. The second method computes the LM probability of each hypothesis and shuffles them, which preserves the entropy of the original approximate posterior but randomizes the weights. The third assigns an equal weight for each hypothesis. 5 presents the results with three beam sizes: 2, 4, 8. Using the LM weights directly performs the best for all beam sizes. Length normalization (LenNorm) is better than shuffled and uniform, which have similar performance. This suggests that both ranking hypotheses correctly and scaling the weights proportional to the LM probability are important.
We also observe how the quality of the LM affects the results. We test LMs of various quality by using the same ConvLM but trained for a different number of steps. We quantify LM quality by the token perplexity (PPL. on the development set. Table shows a clear positive correlation between the LM quality and the final WER. This is expected given that the better LM results in a more accurate posterior approximation Py|x
Table 5: Comparison of LPM and alternative hypothesis weighting methods.
Table 6: Using Py of different de- velopment token perplexity.
dev-clean dev-other WER (%)
Py PPL
dev-fclean other)
pylx
h: === 2
k=
k =
34.24
9.00 / 26.47
Proposed
9.34 27.45
9.00 / 26.47
9.15 26.52
64.22
10.08 / 26.92
LenNorm
10.52 29.10
10.32/ 28.69
10.20 / 28.39
97.87
10.90 / 27.9)
Shuffled
10.76 / 29.32
10.94 29.17
10.56 / 29.20
142.12
11.53 / 28.74
Uniform
10.96 29.46
10.74/29.00
10.84 / 29.21
180.71
13.18 / 30.74
4.5
FINAL RESULTS AND COMPARISON WITH LITERATURE
The best performing model is trained for 3.2M steps, with a learning rate annealed by a factor two every 1.28M steps when using 360 hours of unpaired speech, and every 0.64M steps when using 860 hours of unpaired speech. Reference lengths for filtering are obtained from ASR+LM beam search
decoding. We compare LPM to fully supervised models and a pseudo labeling (PL) method in T Compared to LPM, PL demands extensive search as well heuristic data filtering methods. to obtain high quality pseudo transcripts. We
tuning for the ASR+LM beam
7

---

Under review as a conference paper at ICLR 2020
chose PL as baseline because follow the same experimental setup, and, more importantly, it achieves the state-of-the-an results on LibriSpeech when using train-clean- 100 as paired data and train-other-360 as unpaired speech. A more complete table with results from the literature is in the Appendix.
The upper half of Table 7 shows greedy decoding results without an LM. For the fully supervised model, when removing train-clean-360 the WER increases by 6.86% on test-clean and 13.36% on test-other. Using train-clean-360 speech without transcripts, LPM reduces the absolute WER by 5.64% and 12.21% % on the two test sets, which recovers 82% and 91 %, respectively, of the WER drop from removing the labels. Adding noisier train-other-500 further reduces the WER, and LPM achieves a better WER on the noisy sets (dev-other and test- other) compared to the supervised model trained on 460 hours of clean paired data. addition, LPM outperforms PL. in all settings. This trend is consistent even when decoding with strong Mo
the unpaired set (total 860hr D;)
Table Results of baselines and the proposed methods.
dev WER (%)
test WER (%)
D
D
LM
clean
other
clean
other
Supervised
100hr
N/A
None
14.00
37.02
14.85
39.95
Supervised
460hr
N/A
None
7.20
25.32
7.99
26.59
PL (ASR greedy) Kahn et al. 2019
100hr
360hr
None
12.27
33.42
12.57
35.36
PL (ASR+LM stable) (Kann et al.
2019
100hr
360hr
None
9.30
28.79
9.84
30.15
PL (ASR+LM stable) (Kahn et al.
2019
100hr
860hr
None
9.03
26.03
9.44
27.25
Local Prior Matching
100hr
360hr
None
8.85
26.33
9.21
27.74
Local Prior Matching
100hr
860hr
None
8.08
21.52
8.37
22.89
Supervised
100hr
N/A
ConvLM
7.78
28.15
30.44
Supervised
460hr
N/A
ConyLM
3,98
17.00
4.23
17.36
PL (ASR greody) Kahn et al.
2019
100hr
360hr
ConvLM
6.19
23.53
6.81
24.99
PL (ASR+LM stable) (Kahn et al.
2019
100hr
360hr
ConvLM
5.73
22.54
6.35
24.13
PL (ASR+LM stable) (Kahn
2019
100hr
860hr
ConvLM
6.31
21.87
6.84
23.29
Local Prior Matching
100hr
360hr
ConvLM
5.69
20.22
20.93
Local Prior Matching
100hr
860hr
ConyLM
5.39
14.89
5.78
16.27
ANALYSIS
5.1 HYPOTHESIS QUALITY OF UNLABELED TRAINING SPEECH
updating the proposal model is crucial to improve the hypotheses used during training. To quantify the improvement, Table T shows WERs on the unlabeled data of an LPM model at the beginning (step=0) and at the end 3.2M= of training. Note that this is a proxy of quality for LPM, since multiple hypotheses are used when setting k 2 2. We compare to the WER of the pseudo transeripts used in PI.. Although generating hypotheses without LM leads to worse WER than that of PI., toward the end of training, the proposal model of LPM produces better predictions on both train-clean-360 and train-other-500 than the fixed ones used in PL (ASR+LM stable). Furthermore, the WER on train-other-500 much higher for PL 121.51% 1%) than for LPM at the end of training (13.00%) which explains why LPM achieves much better performances than PL when using the full 860hr of unpaired data.
As discussed in Section 4.2
5.2 LINGUISTIC PLAUSIBILITY
We expect models trained with LPM to generate more semantically and grammatically correct text since the ASR model receives direct supervision from the LM. Sholessshows the proposed hypothe- ses for two utterances using a supervised baseline model and a model trained with LPM. The base- line model proposes erroncous hypotheses which are easy to discard even without the audio the ground truth transcript. On the other hand, LPM generates hypotheses that are both grammatically and semantically plausible, with acceptable substitution errors in some cases (e.g., might/would).
8

---

Under review as a conference paper at ICLR 2020
Table 8: Comparison of the label quality the unlabeled speech data between the two methods when trained on 100hr D/ and 860hr D
train WER (%)
Labelling Method
Step
clean-360
other-500
Pseudo Label
ASR+LM stable
All
8.25
21.51
Local Prior Match
Proposal greedy
14.81
29.03
Local Prior Match
Proposal greedy
3.2M
7.37
13.00
Table 9: Comparison of beam search hypotheses from the supervised model (100hr D,) and the model trained with LPM (100hr Dr 360hr :.). Hypotheses are ranked by ASR score [y|x-
Model
Rank
log
Beam Search Hypotheses (F
Ref.
34.14
she walk very fisst _after she Jeft the house $
47 58
she looked very shought after she left the house
Jonked yvery fat after ,hhe Jeft Jhe Jouse
Sup.
3
-51.47
she whhut very thought after she left the house $ Leery shought jafter she Jeft the J
-83.85
-32.24
she walk very fast as she left the house $
1.PM
-34.14
333.10
she walk very
fist aafter she Jeft the house $
left the
-36.59
she Jooked vveyy Jast after she Jeft the Jouse $
Ref.
-68.01
.oh jif _had Limagined him still iin such distress .uure J. _might Jave done something Jo Jeepp Jiim
-110.11
.wyy Jave something hhepp i before had imagined him steal ing such distress sure ly want have done something help you
Leffore Jad imagined
such distress
Sap.
-107.81
-107.55
i before had imagined him still ing such distress Jure ly J want have done jomething _eup-yyou
-107.10
_.eeoore
imagined
atcal ing such
sure ly want
something _to _hepp -you
-72.55
,oh Jad jiagined Jimm still iin suuch distress _uure would Jave Jone something _o _help you
LPM
-71.35
had imagined him still in such distrss .uure might have done something help Jad jimagined Jiim jin ssuch distress surr would Jave doon ssomething _1 _help ob if had imagined him still in such distress sure i won't have done something to help you
-69.29
-85.62
We also notice in Table|9that the LM probabilities correlate well with linguistic plausibility for texts of similar lengths (c.g., hypotheses of the same utterance), but not for texts with large diffcrences propose to quantify linguistic plausibility of ASR
in length. Motivated by this observation, model by measuring the LM token perplexity exp(E, log Py (y) Ey len(y)) of the hypotheses on development set obtained using ASR-only greedy decoding. Results are shown Table [T The ground truth text has the lowest perplexity models are worse on dev-other than on dev-clean, LPM exhibits the smallest perplexity difference between the two sets, demonstrating how it successfully distills knowledge from the L.M.
both sets expected. We also observe that while all
Table 10: LM token perplexity of ground truth texts and hypotheses obtained with greedy decoding. lower perplexity indicates more linguistically plausible decoding results.
LM perplexity dev-clean
D:
D
dev-other
Ground Truth
N/A
N/A
39.94
43.26
Supervised
100hr
N/A
96.13
313.38
Supervised
460hr
N/A
58.76
164.77
Psuedo Label (ASR greedy)
100hr
360hr
87.36
273.14
Pscudo Label (ASR+LM stable)
100hr
360hr
64.07
170.72
Local Prior Matching
100hr
360hr
61.73
159.72
Local Prior Matching
100hr
860hr
59.84
125.42

---

Under review as a conference paper at ICLR 2020
6 RELATED WORK
Our work builds on a large body of work in semi-supervised learning for ASR. Research in this direction can be classified based on the required modules and objectives used with unpaired data.
In Drexler & Glass (2018) and Karita et al. (2018). bi-encoder network architectures are used, which map text and speech representations in a shared space with their corresponding encoder, and then apply a shared decoder to map from the shared space to the text space. Another line of work adds a TTS model (Tjandra et al., 2017; 2019) Baskar et al.] 2019) or a -enttiinnn (TTE) model (Hayashi et al. [0188] Hori et al.) 2019) in the loop of ASR training. which can be utilized for back-translation style data augmentation (Sennrich al. 2016) or cycle-consistency training (Zhu et al. 2017). [iu et al. (2019) treats ASR as generative model that conditions on speech instead of random noise vectors, and adopts the generative adversarial network (GAN) (Goodfellow et al.) 2014) framework in order to improve the fidelity of ASR-generater texts (Liu et 2019). All the aforementioned methods involve additional modules that must be jointly optimized with the ASR model and require finding careful balance between multiple training objectives. Moreover, some methods require gradient to flow through samples drawn from the ASR outputs, and hence rely on techniques like Gumbel-softmax (Jang et al. 2017), straight-through estimator (Bengio et al., 2013). or REINFORCE (Williams 1992). The LPM objective can evaluate output quality yet is fully differentiable.
Li et al. 2019 Parthasarathi & Strom 2019) because the teacher model can provide a training target for the unlabeled speech. Rather than learning from a teacher ASR model, LPM learns from a teacher LM.
Knowledge distillation is also easy to used for semi-supervised learning (Cui et al.
2017
Pseudo labelling (Vesely et al. 2017, Manohar al. 2018 Kahn al. 2019) is most similar to our approach. Pseudo labelling uses a seed ASR model to transcribe unlabeled speech. The ASR model then trained on the predicted labels along with the original labeled data used to train the seed model. In addition to being well motivated theoretically, we highlight the key methodological advantages of LPM below. (1) While previous work uses fixed pseudo transcripts throughout train- ing and focuses on data filtering, we demonstrate the benefit of generating hypotheses on-the-fly as the proposal model improves. This is efficient since we only use an end-to-end ASR model for beam search decoding. (2) We motivate and show empirically the benefit from considering multiple hypotheses rather than just the top hypothesis (Vesely et al., 2017 Kahn et al,, 2019). (3) Manohar al. (2018) consider multiple hypotheses in a numerator lattice and maximize the aggregated prob- ability of the lattice regardless as to how the probability is allocated amongst the word sequences. In contrast, we verify in the LM ablation study that considering the allocation of the posterior prob- ability amongst the hypotheses is important.
Aside from semi-supervised learning, this work is also related to unsupervised domain adaptation for ASR, where unlabeled speech of the target domain is provided. Unlike the proposed method, previ- ous studies mainly focus on learning domain invariant features (Sun et al. | 2017 Hsu Glass 2018
Meng et al. 2017 2018 2019) or data augmentation through learned speech transformation Hsu
et al. 2017 2018).
CONCLUSION AND FUTURE WORK
We introduce local prior matching, a supe-sed learning objective for speech recognition, and demonstrate note-able reductions in WER when with the addition of unpaired audio and text. We also perform an extensive empirical study to demonstrate the importance of various configurations of LPM. While LPM is motivated by how humans learn to recognize speech, the proposed method can be applied to other sequence transduction tasks including machine translation (Sennrich et al. 2016) and text summarization (Nallapati et al. 2016), provided good prior for the domain.
We consider two promising directions for future in self-supervised speech recognition with LPM and in general: (1) Empower the model with more capabilities, such learning select whether or not to use a sample for sul-suueevision. (2) Using more context signals for prior estimation which can consider previous sentences as well inputs from other modalities.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Jordi Adell and Antonio Bonafonte. Towards phone segmentation for concatenative speech synthe- sis. In Fifth ISCA Workshop on Speech Synthesis, 2004.
Dario Amodei, Sundaram Ananthanarayanan, Rishita Anubhai, Jingliang Bai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Qiang Cheng, Guoliang Chen, et al. Deep Speech 2: End- to-end speech recognition in English and Mandarin. In Proc. International Conference Ma-
chine Learning (ICML), 2016.
Dzmitry Bahdanau, Jan Chorowski, Dmitriy Serdyuk, Philemon Brakel, and Yoshua Bengio. End- to-end attention- based large vocabulary speech recognition. In Proc. International Conference Acoustics, Speech and Signal Processing (ICASSP), 2016.
Murali Karthick Baskar, Shinji Watanabe, Ramon Astudillo, Takaaki Hori, Lukáš Burget, and Jan Čemocký. cenissuper sequence-to-seve soseeuuee--eeeeeeeeeeeeeeee ASR using unpaired speech and text. In Proc. Annual Conference of International Speech Communication Association (INTERSPEECH)) 2019.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: 300. 3432, 2013.
Chung-Cheng Chiu, Tara N Sainath, Yonghui Wu, Rohit Prabhavalkar, Patrick Nguyen, Zhifeng Chen, Anjuli Kannan, Ron Weiss, Kanishka Rao, Ekaterina Gonina, et al. Statc-of-the-ar speech recognition with sequence-to-sequeet Acoustics, Speech and Signal Processing (ICASSP), 2018.
models. In Proc. International Conference on
Noam Chomsky. Knowledge of language: Its nature, origin, and use. Greenwood Publishing Group, 1986.
Jan Chorowski and Navdeep Jaitly. Towards better decoding and language model integration in quence to sequence models. In Proc. Annual Conference of International Speech Communication Association (INTERSPEECI)), 2017.
Jia Cui, Brian Kingsbury. Bhuvana Ramabhadran, George Saon, Tom Sercu, Kartik Audhkhasi, Abhinay Sethy, Markus Nussbaum- and Andrew Rosenberg. Knowledge distillation across ensembles of multilingual models for low-resource languages. In Proc. International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2017.
Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks, In Proc. International Conference on Machine Learning (ICML), 2017.
Jennifer Drexler and James Glass. Combining end-to-end and adversarial training for low-resource speech recognition. In Proc. IEEE Spoken Language Technology Workshop (SLT), 2018.
Emmanuel Dupoux, Cognitive science in the era of artificial intelligence: A roadmap for reverse- engineering the infant language-learner. Cognition, 173:43-59, 2018.
James Glass. Towards unsupervised speech processing. In International Conference on Information Science, Signal Processing and their Applications (ISSPA), 2012.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Proc. Neural Information Processing Systems (NeurIPS), 2014.
Awni Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Greg Diamos, Erich Elsen, Ryan Prenger, Sanjeev Satheesh, Shubho Sengupta, Adam Coates, et al. Deep Speech: Scaling up end-to-end speech recognition. arXiv preprint arXiv:1412.5555 2014.
Awni Hannun, Ann Lee, Qiantong Xu, and Ronan Collobert. Sequence-to-sequeeee speech recogni- tion with time-depth separable convolutions. In Proc. Annual Conference of International Speech Communication Association (INTERSPEECH) 2019.
Tomoki Hayashi, Shinji Watanabe, Yu Zhang, Tomoki Toda, Takaaki Hori, Ramon Astudillo, and Kazuya Takeda. Back-translation-sty style data augmentation for end-to-end ASR. In Proc. IEEE Spoken Language Technology Workshop (SI.T), 2018

---

Under review as a conference paper at ICLR 2020
Takaaki Hori, Ramon Astudillo, Tomoki Hayashi, Yu Zhang, Shinji Watanabe, and Jonathan Le Roux. sycls-consistency training for end-to-end speech recognition. In Proc. International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2019.
Wei-Ning Hsu and James Glass. Extracting domain invariant features by unsupervised learning for robust automatic speech recognition. In Proc. International Conference Acoustics, Speech and Signal Processing (ICASSP), 2018.
Wei-Ning Hsu, Yu Zhang, and James Glass. Unsupervised domain adaptation for robust speech recognition via variational autoencoder-based data augmentation. In Proc. IEEE Workshop on Automfatic Speech Recognition and Understanding (ASRU), 2017.
Wei-Ning Hsu, Hao Tang, and James Glass. Unsupervised adaptation with interpretable disentangled representations for distant conversational speech recognition. In Proc. Annual Conference of International Speech Communication Association (INTERSPEECH), 2018.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameteriization with gumbel-softmax. In Proc. International Conference on Learning Representations (ICLR), 2017.
Jacob Kahn, Ann Lee, and Awni Hannun. Self-training for end-to-end speech recognition. arXiv preprint arXiv: 160600797 2019.
Shigeki Karita, Shinji Watanabe, Tomoharu Iwata, Atsunori Ogawa, and Marc Delcroix. Semi- supervised end-to-end speech recognition. Proc. Annual Conference of International Speech Communication Association (INTERSPEECI), 2018.
Taku Kudo. Subword regularization: Improving neural network translation models with multiple subword candidates. In Proc. Annual Meeting of the Association for Computational Linguistics (ACL), 2018
Taku Kudo and John Richardson. Sentencepiece: simple and language independent subword tokenizer and detokenizer for neural text processing. In Proc. Empirical Methods in Natural Language Processing (EMNLP), 2018.
Patricia K. Kuhl. Early language acquisition: cracking the speech code. Nature Reviews Neuro- science, 5(11):831-88) 2004. doi: 10.1038/mm11333
Bo Li, Tara N Sainath, Arun Narayanan, Joe Caroselli, Michiel Bacchiani, Ananya Misra, Izhak Shafran, Hasim Sak, Golan Pundak, Kean K Chin, et al. Acoustic modeling for Google home. In Proc. Annual Conference of International Speech Communication Association (INTERSPEECH), 2017.
Bo Li, Tara N Sainath, Ruoming Pang. and Zelin Wu. Semi-supervised training for end-to-end models via weak distillation. In Proc. International Conference Acoustics, Speech and Signal Processing (ICASSP), 2019.
Libri Project. https: wirriirii...... org
Alexander H Liu, Hung-yi Lee, and Lin-shan Lee. Adversarial training of end-to-end speech recog- nition using a criticizing language model. In Proc. International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2019.
Vimal Manohar, Hossein Hadian, Daniel Povey, and Sanjeev Khudanpur. Semi-supervised train- ing of acoustic models using lattice-free MMI. In Proc. International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2018.
Zhong Meng, Zhuo Chen, Vadim Mazalov, Jinyu Li, and Yifan Gong. Unsupervised adaptation with domain separation networks for robust speech recognition. In Proc. IEEE Workshop on Automfatic Speech Recognition and Understanding (ASRU), 2017.
Zhong Meng, Jinyu Li, Yifan Gong, and for unsupervised domain adaptation. In Proc. International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2018.
Juang. Adversarial teacher-student learning
12

---

Under review as a conference paper at ICLR 2020
Zhong Meng, Jinyu Li, and Yifan Gong. Attentive adversarial learning for innariinnnnvriantt training. In Proc. International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2019.
Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing Xiang, et al. Abstractive text summarization using sequence-to-sequeene RNNs and beyond. In Proc. ACL Conference on Natural Language Learning (CoNLL), 2016.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an ASR corpus based on public domain audio books. In Proc. International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2015.
Sree Hari Krishnan Parthasarathi and Nikko Strom. Lessons from building acoustic models with a million hours of speech. In Proc. International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2019.
Kainan Peng, Wei Ping. Zhao Song, and Kexin Zhao. Parallel neural text-to-speech. arXiv preprini 1:X.::9905.08455 2019.
Vineel Pratap, Awni Hannun, Qiantong Xu, Jeff Cai, Jacob Kahn, Gabriel Synnaeve, Vitaliy Liptchinsky, and Ronan Collobert. wav2letter+:: The fastest open-source speech recognition system. arXiv preprint arXiv: 18X2.0///22.0725,, 2018.
Kristen Precoda. Non-mainstream languages and speech recognition: Some challenges. CALICO journal, 21(2):229-243, 2013.
Project Gutenberg. https: www. www.gguueengggoogg org
Odette Scharenborg, Vincent Wan, and Mirjam Ernestus. Unsupervised speech segmentation: An analysis of the hypothesized phone boundaries. The Journal of the Acoustical Society of America, 127(2):1084-1095, 2010.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Improving neural machine translation models with monolingual data. Proc. Annual Meeting the Association for Computational Linguistics (ACL), 2016.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-1955 2014.
Sining Sun, Binbin Zhang, Lei Xie, and Yanning Zhang. An unsupervised deep domain adaptation approach for robust speech recognition. Neurocomputing, 257:79-87, 2017.
Andros Tjandra, Sakriani Sakti, and Satoshi Nakamura. Listening while speaking: Speech chain by deep learning. In Proc. IEEE Workshop on Automfatic Speech Recognition and Understanding (ASRU), 2017.
Andros Tjandra, Sakriani Sakti, and Satoshi Nakamura. End-to-end feedback loss in speech chain framework via straight-through estimator. In Proc. International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proc. Neural Information Processing Systems (NeurIPS), 2017.
Karel Vesely, Lukás Burget, and Jan Cemncký.. Semi-supervised DNN training with word selec- tion for ASR. In Proc. Annual Conference of International Speech Communication Association (INTERSPEECH), 2017.
Yu-Hsuan Wang, Cheng- Tao Chung, and Hung-yi Lee. Gate activation signal analysis for gated re- current neural networks and its correlation with phoneme boundaries. In Proc. Annual Conference of International Speech Communication Association (INTERSPEECH), 2017.
Ronald J Williams. Simple statistical gradient-followine algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):22--256 1992.
13

---

Under review as a conference paper at ICLR 2020
Zichao Yang, Zhiting Hu, Chris Dyer, Erie P Xing, and Taylor Beng-Kirppatrick. Unsupervised text style transfer using language models as discriminators. In Proc. Neural Information Processing Systems (NeurIPS), 2018.
Neil Zeghidour, Qiantong Xu, Vitaliy Liptchinsky, Nicolas Usunier, Gabriel Synnaeve, and Ronan Collobert. Fully convolutional speech recognition. arXiv preprinl 1rXiv:18/2.0668 2018.
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image trans- lation using cycle-con adversarial networks. In Proc. IEEE International Conference on Computer Vision (ICCV), 2017.
14

---

Under review as a conference paper at ICLR 2020
A MORE COMPREHENSIVE COMPARISON WITH THE LITERATURE
Table 11: A more comprehensive comparison with semi-supervised ASR studies using LibriSpeech, including the performances of the bascline/topline supervised model used in each study, since they differ significantly across different papers. D) and Du denote the amount of paired and unpaired data used in each experiment, and (S)(T)((S+T) denote the type of the unpaired data, corresponding speech/tex/booth, respectively. Experiments with the asterisk sign (*) contain results that not reported in the original paper, but are obtained from the authors of the paper.
WER (%)
test WER (%)
Dr
D
LM
clean
other
clean
other
Supervised Supervised
100hr
N/A
None
24.9
25.2
460hr
N/A
None
11.4
11.8
Hayashi et al. 2018
BT
100hr
360hr (T)
None
23.5
23.6
Supervised
100hr
N/A
RNN-LM
23.0
22.9
BT
100hr
360hr (T)
RNN-LM
21.6
22.0
Supervised
100hr
N/A
None
21.6
21.7
Crit-LM
100hr
360hr (T)
None
19.1
19.2
Crit-LM
100hr
860hr (T)
None
18.5
18.7
Liu 2019
Supervised
100hr N/A
RNN-LM
20.0
20.3
Crit-LM
100hr
360hr (T)
RNN-LM
17.1
17.3
Crit-LM
100hr
860hr (T)
RNN-LM
15.3
15.8
Supervised
100hr
N/A
None
24.9
25.2
Supervised
460hr N/A
None
11.4
11.8
Hori et al. 2019
Cycle-TTE
100hr
360hr (S)
None
21.5
21.5
Supervised
100hr
N/A
RNN-LM
22.6
22.9
Cycle-TTE
100hr
360hr (S)
RNN-LM
19.6
19.5
Supervised
100hr
None
21.0
Cycle-TTS
100hr
360hr (S)
None
17.9
Cycle-TTS
100hr
360hr (S+T)
None
17.5
Baskar al. 2019
Cycle-TTS
100hr
360hr (T)
RNN-LM
17.0
Cycle-TTS
100hr
360hr (S)
RNN-LM
16.8
Cycle-TTS
100hr
360hr (S+T)
RNN-LM
16.6
Supervised
100hr
N/A
None
14,00
37.02
14.85
39.95
Supervised
460hr
None
7.20
25.32
7.99
26.59
(ASR)*
100hr
360hr (S)
None
12.27
33.42
12.57
35.36
(ASR+LM)
100hr
360hr (S)
None
9.30
28.79
9.84
30.15
(ASR+LM)*
100hr
(S)
None
9.03
26.03
9.44
27.25
PL. (Ensemble)
100hr
360hr (S)
None
8.60
27.78
9.21
29.29
Kahn al. 2019
Supervised
100h N/A
Convi LM
7.78
28.15
8.06
30.44
Supervised
460hr N/A
ConvLM
3.98
17.00
4.23
17.36
PL (ASR)*
100h
360hr (S)
ConvLM
6.19
23.53
6.81
24.99
PL (ASR+LM)
100h
360hr (S)
ConvLM
5.73
22.54
6.35
24.13
PL (ASR+LM))
100hr
860hr (S)
ConvLM
6.31
21.87
6.84
23.29
PL (Ensemble)
100hr
360hr (S)
ConvLM
5.37
22.13
5.93
24.07
Supervised
100hr
None
37.02
39.95
Supervised
460hr
None
7.20
25.32
7.99
26.59
LPM
100hr
360hr
None
8.85
26.33
9.21
27.74
LPM
100hr
860hr
None
8.08
21.52
8.37
22.89
This work
Supervised
100hr
N/A
ConvLM
7.78
28.15
8.06
30.44
Supervised
460hr
N/A
ConvLM
3,98
17.00
4.23
17.36
LPM
100hr
360hr
ConvLM
5.69
20.22
5,99
20.93
LPM
100hr
860hr
ConvLM
5.39
14.89
5.78
16.27
15

---

Under review as a conference paper at ICLR 2020
B ADDITIONAL RESULTS
Table 12: Results of varying LPM weight
dev WER (%)
Cx
clean
other
2e-2
10.86
31.59
5e-2
10.08
28.92
9.24
27.62
2e-1
9.00
26.47
5e-1
9.41
26.56
C CHARACTER ERROR RATES
Table 13: Results of baselines and the proposed methods.
dev CER (%)
test CER (%)
Dj
D
LM
clean
other
clean
other
Supervised
100hr
N/A
None
6.20
20.27
6.80
22. 24.14
Supervised
460hr
N/A
None
2.86
13.06
3.37
13.73
Local Prior Matching
100hr
360hr
None
3.79
14.00
3.87
14.81
Local Prior Matching
100hr
860hr
None
3.52
11.14
3.60
12.08
Supervised
100hr
N/A
ConvLM
3.83
17.03
3.86
18.52
Supervised
460hr
N/A
ConvLM
1.65
9.51
1.79
9.47
Local Prior Matching
100hr
360hr
ConvLM
2.65
11.56
2.81
11.96
Local Prior Matching
100hr
860hr
ConvLM
2.51
8.70
2.70
9.70
16


Reviews:
[{"review_id": "SJeEslb2tS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overview:\nThis paper is dedicated to proposing a self-supervised objective, local prior matching (LMP), for speech recognition. This approach can take advantage of vase quantities of unlabeled speech data. What' more, the objective is simple to implement and theoretically well-motivated. In the paper, based on a supervised pretrained model, it then finetunes 360 hours with unlabeled data and LPM reduces the WER consistently. They also conduct extensive ablation experiments to show the effect of their self-supervised approach.\n\nStrength Bullets:\n1. I think this self-supervised learning objective (LMP) is very novel. The motivation that the source of indirect supervision on processing unlabeled speech comes from prior knowledge about the world and the context of the speech makes sense to me. The author combines the Bayesian method to build the model which is aligned with the motivation. They also provide clear and well-organized derivations. \n2. The paper performs extensive ablation studies over all components, including beam size, mixing ratio, LPM weights, model update strategies, model initialization, length filtering and choice of language models. It provides convincing evidence of the effect of each component. \n3. It provides interesting experiments results to study the relationship between the amount of unlabelled data for self-supervision and final performance. And LPM can surpass the performance of using 360 hours of labeled data by taking advantage of about twice the amount of unlabeled data.\n\n\nWeakness Bullets:\n1. The paper only evaluates their method on LibriSpeech dataset. Although this dataset is popular, one or two more datasets will be more convincing. \n2. For bayesian based methods, it is well known that it performs badly in high dimensional space. The reason is that we can not sample enough data points to obtain a good posterior estimation. Could you provide more analysis about the quality of posterior with different amount of sampled data?\n3. For the experiment between the amount of unlabelled data for self-supervision and final performance, it would be better the author can provide a curve with more results.\n\nRecommendation:\nI think it is a good paper. The proposed approach is useful. This is a weak accept. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "H1g7aNtaYB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposed a distillation approach which use ASRs to generate hypotheses for unsupervised data, run a LM to get probability for the hypothesis, and perform distillation with the resulting probability. The ASRs being used for generating hypotheses can be either a model trained with the supervised data or the student model, and can switch between the two during training. In the experiments, ASR models are pre-trained with the subset of Librispeech data and use the rest of Librispeech data as unsupervised data, and the LM is trained with Librispeech LM data. The experiments shown the proposed approach improve baseline model trained with the Librispeech subset significantly.\n\nThe use of LM to provide soft target is a good idea as LMs can utilize unsupervised text data as opposed to the requirement of training a strong teacher model with paired data, and can be easily integrated with existing distillation approaches for ASRs. The switching to the student model for generating hypotheses when it outperforms the pre-trained ASR also makes a good sense. The overall novelty however is a bit limited compared to the existing work, as the major contribution is to propose to use LMs as teacher rather than ASRs, with the rest of the design to be similar to existing works.\n\nThe paper relates their method to self-supervised learning, yet I find it having stronger correlation with existing distillation approaches, and can be better understood through the distillation perspective.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1lHoTTIqS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper propose local prior matching to leverage a language model to use unlabeled speech data to improve an ASR system. This is a worthy goal. The details of the proposal were a bit hard for me to understand. The proposed method reminded me of \"posterior regularization\" (K. Ganchev et al. 2010), but I could not understand Section 2.2 well enough to draw a direct link. I encourage the authors to condense 2.2 and make it clearer what, exactly, Local Prior Matching is.\n\nThe paper presents extensive, interesting results. I do want to point that they seem to be considerably off of the LibriSpeech state of the art, e.g. see K. Irie et al. Interspeech 2019.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposed local prior matching that utilizes a language model to rescore the hypotheses generate by a teacher model on unlabeled data, which are then used to training the student model for improvement. The experimental results on Librispeech is thorough. But two concerns on this paper are: 1) limited novelty: LM trained on large tex data is already used in weak distillation and the only difference is the use of multiply hypotheses. As pointed out by the reviewers, the method is better understood through distillation even though the authors try to derive it from Bayesian perspective. 2) Librispeech is a medium sized dataset, justifications on much larger dataset for ASR would make it more convincing. 
Decision: Reject