vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations
Authors: Alexei Baevski
Steffen Schneider
Michael Auli
Venue:  (2020)
Keywords: speech recognition
speech representation learning

Abstract: We propose vq-wav2vec to learn discrete representations of audio segments through a wav2vec-style self-supervised context prediction task. The algorithm uses either a gumbel softmax or online k-means clustering to quantize the dense representations. Discretization enables the direct application of algorithms from the NLP community which require discrete inputs. Experiments show that BERT pre-training achieves a new state of the art on TIMIT phoneme classification and WSJ speech recognition.

Published as a conference paper at ICLR 2020
VQ-WAV2VEC: SELF-SUPERVISED LEARNING OF DISCRETE SPEECH REPRESENTATIONS
Alexei Baevski*
Steffen Schneider" Vt
Michael Auli
A Facebook AI Research, Menlo Park, CA, USA University of Tübingen, Germany
ABSTRACT
We propose vq-wav2vec to learn discrete representations of audio segments through a wav2vec-style self-supervised context prediction task. The algorithm uses either Gumbel-Softmax online k-means clustering to quantize the dense representations. Discretization enables the direct application of algorithms from the NLP community which require discrete inputs. Experiments show that BERT pre-training achieves new state of the art on TIMIT phoneme classification and WSJ speech recognition."
INTRODUCTION
Learning discrete representations of speech has gathered much recent interest (Versteegh et al., 2016; Dunbar et al., 2019). A popular approach to discover discrete units is via autoencoding (Tjandra et al., 2019; Eloff et al., 2019; Chorowski al., 2019) sometimes coupled with autoregressive model (Chung et al., 2019). Another line research is to learn continuous speech representations in self-supervised way via predicting context information (Chung & Glass, 2018; den Oord et al., 2018; Schneider et al., 2019).
In this paper, we combine these lines of research by learning discrete representations of speech via a context prediction task instead of reconstructing the input. This enables us to directly apply well performing NLP algorithms speech data (Figure la).
L1
L2
La
vq-wav2vec
BERT
AM
c
Z
X
(a) vavwev
(b) Discretized speech training pipeline
Figure 1: (a) The wq-wav2vec encoder maps audio (2') to dense representation (2) which quantized (q) to Z and aggregated into context representations (C); training requires future time step prediction. (b) Acoustic models are trained by quantizing the raw audio with vq-wav2vec, then applying BERT the discretized sequence and feeding the resulting representations into the acoustic model to output transcriptions.
Our new discretization algorithm, vq- wav2ve,,, learns discrete representations of fixed length seg- ments of audio signal by utilizing the wav2vec loss and architecture (Schneider et al, 2019; $2). To
"quual contribution.
Work done during a Facebook Al residency.
The code will be made available http: /ii tttp::///gttuuooooooooo..b. comppooccmm/tttto/offfaaaaaaiaia rseq.

---

Published as a conference paper at ICLR 2020
choose the discrete variables, consider Humbel-Soffmaal approach (Jang et al., 2016) well as online k-means clustering. similar to VQ- (Oord et al., 2017: Eloff et al., 2019; $3).
We then train Deep Bidirectional Transformer (BERT: Devlin al., 2018; Liu al., 2019) on the discretized unlabeled speech data and input these representations to a standard acoustic model (Figure lb; $4). Our experiments show that BERT representations perform better than log-mel filterbank inputs as well as dense wav2vec representations on both TIMIT and WSJ benchmarks. Discretization of audio enables the direct application of whole host of algorithms from the NLP literature to speech data. For example, we show that a standard sequence to sequence model from the NLP literature can be used perform speech recognition over discrete audio tokens ($5, $6).
BACKGROUND
2.1 WAV2VRC
wav2vec (Schneider et al., 2019) learns representations of audio data by solving self-supervised context-prediction task with the same loss function as word2vec (Mikolov et al., 2013; van den Oord et al., 2018). The model is based on two convolutional neural networks where the the encoder produces a representation z; for each time step at rate of 100 Hz and the aggregator combines multiple encoder time steps into new representation C for each time step Given an aggregated representation C₁2 the model is trained to distinguish sample Zi+z that is k steps in the future from distractor samples drawn from distribution Pu. by minimizing the contrastive loss for steps k= 1_,.... K:
Wwwe2vee
log (logo(2 (2(1)h((c))) + AE [logo(- ol (11-hhh())))) he(c;)))) 8r-p
(2)
where T the sequence length, (xx = 1/(1 + exp(-r)), and where hhe,4 is the probability of 2i4k being the true sample. We consider step-specific affine transformation (c,);) = WAC, + by that is applied to c; (van den Oord et al., 2018). We optimize the loss L = -K k= Lk, summing (1) over different step sizes. After training, the representations produced by the context network c, are input to the acoustic model instead of log-mel filterbank features.
2.2 BERT
BERT (Devlin et al., 2018) pre-training approach for NLP tasks, which uses transformer encoder model to build representation of text. Transformers uses self-attention to encode the input sequence as well as an optional source sequence (Vaswani et al., 2017). The original BERT model combined two tasks for training: first, masked language modeling randomly removes some of the input tokens and the model has to predict those missing tokens. Second, next sentence prediction splices two different text passages together into a single example and the model needs to predict whether the passages are from the same document.
3
Our approach, wq-eav2vec, learns vector quantized (VQ) representations of audio data using future time-step prediction task. We follow the same architectual choices as wav2vec ($2.1) with two convolutional networks f X' + Z and g 2 + C for feature extraction and aggregation, as well as a new quantization module Z + 2 to build discrete representations (Figure la).
We first map 30ms segments of raw speech to dense feature representation at stride of 10ms using the encoder network f. Next, the quantizer (q) turns these dense representations into discrete indices which are mapped aggregator and optimize the same context prediction task as wav2vec outlined in $2.1.
reconstruction å of the original representation We feed å into the
The quantization module replaces the original representation by e; from fixed size codebook e E V xd which contains V representations of size d. We consider the Gumbel-Sofe -ootbml-l-faaxxx which is a differentiable approximation of the argmax for computing one-hot representations ($3.1; Figure 2a)
2

---

Published as a conference paper at ICLR 2020
1
2
2
B
22
2
62
one-hon hot
argmax
one-hot
argmin
1n
probabilities
Gumbel
0.6
||
logits
distances
Z
(a) Gumbel-Softmax
(b) K-means clustering.
Figure 2: (a) The Gumbel-Softtax quantization computes logits representing the codebook vectors (e). In the forward pass the argmax codeword (e2) is chosen and for backward (not shown) the exact probabilities are used. (b) K-means vector quantization computes the distance to all codeword vector and chooses the closest (argmin).
as well as online k-means clustering, similar the vector quantized variational autoencoder (VQ- VAE; Oord et al., 2017; $3.2; Figure 2b). Finally, we perform multiple vector quantizations over different parts of mitigate mode collapse ($3.3).
3.1 GUMBEL-SOFTMAN SOFTMAX
The Gumbel-Softmal (Gumbel, 1954; Jang et al., 2016; Maddison et al., 2014) enables selecting discrete codebook variables in fully differentiable way and we the straight-through estimator of Jang et al. (2016). Given the dense representation z, we apply a linear layer, followed by a ReLU and another linear which outputs R logits for the Gumbel-Softmax. At inference, we simply pick the largest index in (. At training, the output probabilities for choosing the j-th variable are
expll + v,)TT
ΣV exp(le Uk)/T
where = -log(- Mog(-logge log(u)) and are uniform samples from 4( 1). During the forward pass, i argmax, ,P and in the backward pass, the true gradient of the Guumbe-SSotmax outputs is used.
3.2 K-MEANS
The vector quantization approach of van den Oord et al. (2017) is an alternative to making the index selection procedure fully differentiable. Different to their setup, optimize future time step prediction loss instead of the reconstruction loss of an autoencoder.
We choose the codebook variable representation by finding the closest variable to the input features z in terms of the Euclidean distance, yielding argmin, | e, | During the forward pass, we select e; by choosing the corresponding variable from the codebook. We obtain gradients for the encoder network by back back-propagating wavev-ve /d2 (van den Oord et al., 2017). The final loss has two additional terms:
wave + (Isg(()) - 211² + 7/|2 sg(2)
C
where sg(x) = x, 1 term is the future prediction task and gradients do not change the codebook because of the straight- through gradient estimation of mapping to 2. The second term (z)g()) 2| moves the codebook
d sg()) = 0 is the stop gradient operator and is hyperparameter. The first
vectors closer to the encoder output, and the third term ||z - sg(2) makes sure that the encoder outputs close centroid (codeword).
3

---

Published as a conference paper at ICLR 2020
3.3 VECTOR QUANTIZATION WITH MULTIPLE VARIABLE GROUPS
So far, we considered replacing the encoder feature vector z by a single entry e; in the codebook. This is prone to mode collapse where only some of the codewords actually used. Previously, this problem has been mitigated by workarounds such as re-initializing codewords or applying additional regularizers the loss function (Caron et al., 2019). the following, we describe another strategy where we independently quantize partitions of z. similar to product quantization (Jegou et al., 2011). This results in larger dictionaries and increased downstream performance (Appendix A).
The dense feature vector z E I2 is first organized into multiple groups G into the matrix form z' E RCx(d/G), We then represent each row by an integer index, and hence can represent the full feature vector by the indices E [v]6, where V again denotes the possible number of variables for this particular group and each element i, corresponds to a fixed codebook vector. For each of the G groups, we apply either one of the two VQ approaches ($3.1 and $3.2).
The codebook itself can be initialized in two possible ways: Codebook variables can be shared across groups, i.e., a particular index in group j would reference the same vector as the same index in group j'. This yields codebook e E (x(dG)), In contrast, sharing the codebook variables yields a codebook of size (V/x((d/C) In practise, observe that sharing the codebook variables generally yields competitive results to a non-shared representation.
BERT PRE-TRAINING ON QUANTIZED SPEECH
Once we trained vq- oq-wav2vec model we can discretize audio data and make applicable algo- rithms that require discrete inputs. One possibility is to use the discretized training data and apply BERT pre-training where the task is to predict masked input tokens based on an encoding of the surrounding context (Devlin et al., 2018). Once the BERT model is trained, we can use it to build representations and feed them into an acoustic model to improve speech recognition. We follow recent advances in BERT training which only the masked input token prediction (Liu et al., 2019).
Since each of the discretized tokens represents around 10 ms of audio it is likely too easy to predict a single masked input token. We therefore change BERT training by masking spans of consecutive discretized speech tokens, similar to Joshi et al. (2019). To mask the input sequence, sample = 0.05 of all tokens to be a starting index, without replacement, and mask M 10 consecutive tokens from every sampled index; spans may overlap. This makes the masked token prediction harder and we show later that it improves accuracy over masking individual tokens ($6.5).
randomly
5
EXPERIMENTAL SETUP
5.1 DATASETS
We generally pre-train wq-wav2vec and BERT the full 960h of Librispeech (Panayotov et al., 2015) and after vq-wav2vec training it is discretized to 345M tokens. Where indicated we perform ablations clean 100h subset which is discretized to 39.9M tokens. We evaluate models on two benchmarks: TIMIT (Garofolo et al., 1993b) is a 5h dataset with phoneme labels and Wall Street Journal (WSJ; Garofolo al. 1993a) is a 81h dataset for speech recognition. For TIMIT, we apply the standard evaluation protocol and consider 39 different phonemes. For WSJ, we train acoustic models directly on 31 graphemes, including the English alphabet, the apostrophe, the silence token and tokens for repeating characters.
5.2 VQ-WAV2VEC
We adapt the fairseq implementation of wav2vec (Schneider et al., 2019; Ott et al., 2019) and use vq- way2vec/wav2vee models with 34 x 10 parameters. The encoder has 8 layers with 512 channels each, kernel sizes 1,11 and strides (5.4.2,2,2,1,11.J) yielding a total stride of 160. Each layer contains convolution, followed by dropout, group normalization with single group (Wu & He. 2018) and a ReLU non-linearity. The aggregator is composed of 12 layers, with 512 channels, stride 1, and kernel sizes starting at 2 and increasing by 1 for every subsequent layer. The block
4

---

Published as a conference paper at ICLR 2020
structure is the same for the encoder network, except introduce skip connections between each subsequent block.
We train with the wav2vec context prediction loss (Equation 1) for 400k updates, predicting K 8 steps into the future and sample 10 negatives from the same audio example. Training is warmed up for 500 steps where the learning rate is increased from x 10 to 5 x 10 and then annealed to x 10 using a cosine schedule (Loshchilov Hutter, 2016). The batch size 10, and we crop a random section of 150k frames for each example (approximately 9.3 seconds for 16kHz sampling rate). All models are trained on GPUs.
For ablations and experiments the 100h Librispeech subset, (10,8,4.4.4) and strides (5,4.2.2.2) in the encoder and seven convolutional layers with stride one and kernel size three the aggregator. This model is trained for 40k updates.
smaller model with kernels
Gumbel-Softmax Models. We use G 2 groups and V 322 latents per group and the linear layer projects the features produced by the encoder into produces one-hot vector for each group G. The temperature T is linearly annealed from 2 to 0.5 over the first 70% of updates and then kept constant at 0.5. This enables the model to learn which latents work best for each input before committing to a single latent. After training this model on 960h of Librispeech and quantizing the training dataset, we are left with 13.5k unique codewords combinations (out of /c 102k possible codewords).
640 logits. The Gumbel-Softmax
k-means Models. We use G = 2 groups and V = 320 variables per group. vq-wav2vec on full Librispeech yields 23k unique codewords. Following van den Oord et (2017), we found 0.25 to be robust choice for balancing the VQ auxiliary loss.
5.3 BERT
BERT base models have 12 layers, model dimension 768, inner dimension (FFN) 3072 and 12 attention heads (Devlin al., 2018). The learning rate is warmed up over the first 10,000 updates to a peak value of 1 x 10-5, and then linearly decayed over a total of 250k updates. We train on 128 GPUs with batch size of 3072 tokens per GPU giving total batch size of 393k tokens (Ott al., 2018). Each token represents 10ms of audio data.
BERT small. For ablations we use a smaller setup with model dimension 512, FFN size 2048, 8 attention heads and dropout 0.05. Models are trained for 250k updates with a batch size of 2 examples per GPU.
5.4 ACOUSTIC MODEL
We use wav2letter accoustic model (Collobert et al., 2016; 2019) and train for 1,000 epochs 8 GPUs for both TIMIT and WSJ using the auto segmentation criterion. For decoding the emissions lexicon well separate language model trained on
from the acoustic model WSJ we the WSJ language modeling data only. We consider a 4-gram KenLM language model (Heafield al., 2013) and character based convolutional language model (Likhomanenko al., 2019) and tune the models with the same protocol as Schneider et al. (2019).
6 RESULTS
6.1 WSJ SPEECH RECOGNITION
We first evaluate on the WSJ speech recognition benchmark. We train vq wa-wevee model on the unlabeled version of Librispeech, then discretize the same data with the resulting model to estimate a BERT model. Finally, we train a wav2letter acoustic model on WSJ by inputting either the BERT wq-wav2vec representations instead of log-mel filterbanks.?
We compare to various results from the literature, including wav2vee (Schneider et al., 2019) and consider three setups: performance without any language model (No LM), with an n-gram LM
For vq-wav2vec input the dense representations cow.sssoodcinnngneegooooo the learned discrete units.
5

---

Published as a conference paper at ICLR 2020
nov93dev
nov92
LER
WER
LER
WER
Deep Speech (12K h labeled speech; Amodei al., 2016)
4.42
3.1
Trainable frontend (Zeghidour et al., 2018)
6.8
3.5
Lattice-free MMI (Hadian al., 2018)
5.66
2.81
Supervised transfer-learing (Ghahremani et al., 2017)
4.99t
2.53
No LM
Baseline (log-mel)
6.28
19.46
4.14
13.93
wav2vee (Schneider et al., 2019)
5,07
16.24
3.26
11.20
wq-wav2vec Gumbel
7.04
20.44
4.51
14.67
+ BERT base
4.13
13.40
2.62
9.39
4-GRAM LM (Heafield et al., 2013)
Baseline (log-mel)
3.32
8.57
2.19
5.64
wav2vec (Schneider et al., 2019)
2.73
6.96
1.57
4.32
vq-wav2vec Gumbel
3.93
9.55
2.40
6.10
BERT base
2.41
6.28
1.26
3.62
CHAR CONvLM (Likhomanenko et al., 2019)
Baseline (log-mel)
2.77
6.67
1.53
3.46
wav2vec (Schneider et al., 2019)
2.11
5.10
0.99
2.43
vq-wav2vec Gumbel + BERT base
1.79
4.46
0.93
2.34
Table 1: WSJ accuracy of wa-wav2vec on the development (nov93dev) and test (nov92) in terms
of letter еπrо rate (LER) and word error rate (WER) without language modeling (No LM), 4-gram LM and character convolutional LM. vq-wav2vec with BERT pre-training improves over the best wav2vec model (Schneider et al., 2019).
nov93dev
nov92
LER
WER
LER
WER
No LM
wav2vec (Schneider et al., 2019)
5.07
16.24
3.26
11.20
vq-wav2vec Gumbel
7.04
20.44
4.51
14.67
BERT small
4.52
14.14
2.81
9.69
vq-wav2vec k-means (39M codewords)
5,41
17.11
3.63
12.17
vq-wav2vec k-means
7.33
21.64
4.72
15.17
+ BERT small
4.31
13.87
2.70
9.62
4-GRAM LM (Heafield et al., 2013)
wav2vec (Schneider et al., 2019)
2.73
6.96
1.57
4.32
wq-wav2vee Gumbel
3.93
9.55
2.40
6.10
+ BERT small
2.67
6.67
1.46
4.09
vq-wav2vee k-means (39M codewords)
3.05
7.74
1.71
4.82
wq-wav2vec k-means
4.37
10.26
2.28
5.71
BERT small
2.60
6.62
1.45
4.08
Table 2: Comparison of Gumbel-Softman and k-means vector quantization WSJ (cf. Table 1).
(4-gram L.M and with a character convolutional LM (Char ConvLM). We report the accuracy of wav2letter with log-mel filterbanks input (Baseline) and wav2vec. For vq-wav2vec first ex- periment with the Gumbel-Softmax, with and without a BERT base model ($5.3).
Table shows that vq-wav2vec together with BERT training can achieve new state of the art of 2.34 WER on nov92. Gains are largest when no language model is used which the fastest setting. vg- wa-ee with Cumbel-Softtan uses only 13.5k distinct codewords represent the audio signal and this limited set codewords is not sufficient to outperform the baseline. However, does enable training BERT models which require relatively small vocabulary.

---

Published as a conference paper at ICLR 2020
dev PER
test PER
CNN TD-filterbanks (Zeghidour et al., 2018)
15.6
18.0
Li-GRU fMLLR (Ravanelli et al., 2018)
14.9
wav2vec (Schneider et al., 2019)
12.9
14.7
Baseline (log-mel)
16.9
17.6
vq vq-wav2vec, Gumbel
15.34
17.78
+ BERT small
9.64
11.64
vq-wav2vec, k-means
15.65
18.73
+ BERT small
9.80
11.40
Table 3: TIMIT phoneme recognition in terms of phoneme error rate (PER). All our models use the CNN-8L-PReLLU-do0.7 architecture (Zeghidour et al., 2018).
dev clean
dev other
test clean
test other
Mohamed et al. (2019)
4.8
12.7
4.7
12.9
Irie et al. (2019)
4.4
13.2
4.7
13.4
Park et al. (2019)
2.8
6.8
2.5
5.8
vq-wav2vec Gumbel Transformer Big
5.6
15.5
6.2
18.2
Table 4:
Librispeech results for a standard sequence to sequence model trained on discretized audio without BERT pre-training and results from the literature. All results are without language model.
Next, compare Gumbel-Softtax to k-means for vector quantization. For this experiment the faster to train BERT small configuration ($5.3). We also train a vq-wav2vec k-means model with very large number of codewords (39.90 (39.9M) test whether more expressive model can close the gap to wav2vec. Table shows that Gumbel-Softmax and k-means clustering perform relatively comparably: the no language model setup without BERT, Gumbel-Softmax more accurate than k- k-means but these differences disappear with BERT. For 4-gram LM setup, k-means is better but those differences disappear again after BERT training. Finally, the large codeword model can substantially reduce the gap to the original wav2vec model.
6.2 TIMIT PHONEME RECOGNITION
Next, we experiment on the much smaller TIMIT phoneme recognition task where we also pre- weveev on the full Librispeech corpus. Table 3 shows that vq-wav2vec and BERT achieve 11.64 new state of the of 11.64 PER which corresponds to a 21 % reduction in error over the previous best result of wav2vec.
6.3 SEQUENCE TO SEQUENCE MODELING
So far we used wq-wev2vec to train BERT on discretized speech. However, once the audio dis- cretized we can also train a standard sequence to sequence model to perform speech recognition. In preliminary experiments, we trained an off-the-shelf Big Transformer (Vaswani et al., 2017: Ott et al., 2019) on the vq-wav2vec Gumbel-Softmax discretized Librispeech corpus and evaluated on the Librispeech dev/test sets; we use a 4k BPE output vocabulary (Sennrich et al., 2016). Table shows that results are promising, 2019) which depends on data augmentation that we do not use.
though they are not good the state of the art (Park al.,
6.4 ACCURACY VS. BITRATE
Next, investigate how well wq-wav2vec compress the audio data. Specifically, train models with different numbers of groups G and variables V to vary the size of the possible codebook size FG and measure accuracy TIMIT phoneme recognition without BERT training.

---

Published as a conference paper at ICLR 2020
35.0
vq-wav2vec
TIMIT dev PER
30.0
MP3
Ogg Vorbis
25.0
Codec2
Opus
20.0
15.0
10
10a
10a
Representation Bitrate (bit/s)
Figure 3: Comparison of PER the TIMIT dev set for various audio codecs and wq-wav2vec k-means trained on Librispeech 100h.
We measure compression with the bitrate G Gogz V at sampling rate 100Hz and report the trade-off between bitrate and accuracy on our phoneme recognition task. We experiment with wq-wav2vec k-means and train models with 1 1,2,4,8,16 16 and 32 groups, using 40,80, 160,80,10,,....... 160,..., 1280 vari- ables, spanning a bitrate range from 0.53 kbit/s (G = 1, V = 40) to 33.03 kbit/s (G 32, V = 1280). We place the quantization module after the aggregator module and train all models in the small vq-wav2vec setup ($5.2) on the 100h clean Librispeech subset.
As baselines, we consider various lossy compression algorithms applied to the TIMIT audio data and train wav2letter models on the resulting audio: Codec2 as a low bitrate codec, Opus (Terriberry & Vos, 2012) as a medium bitrate codec and MP3 and Ogg Vorbis (Montgomery, 2004) high bitrate codecs. We use the whole spectrum of both variable and constant bitrate settings of the codecs; we encode and decode with flipeg (ffmpeg developers, 2016). Figure 3 shows the trade-off between the bitrate and TIMIT accuracy. Acoustic models on vq-wav2vec achieve the best results across most bitrate settings.
6.5 ABLATIONS
Table 5a shows that masking entire spans of tokens performs significantly better than individual tokens (M 1). Furthermore, BERT training on discretized audio data is fairly robust to masking large parts of the input (Table 5b).
M
dev
test
P
dev
test
14.94
17.38
0.015
12.65
15.28
5
13.62
15.78
0.020
12.51
14.43
12.65
15.28
0.025
12.16
13.96
20
13.04
15.56
0.030
11.68
14.48
30
13.18
15.64
0.050
11.45
13.62
(a) Mask length.
(b) Mask probabilitics.
Table TIMIT PER for (a) different mask sizes M with pM 0.15 BERT training and (b) mask probabilities p for a fixed mask length M = 10.
CONCLUSION
vq-wav2vee is self-supervised algorithm that quantizes unlabeled audio data which makes it amenable to algorithms requiring discrete data. This approach improves the state of the art on the WSJ and TIMIT benchmarks by leveraging BERT pre-training. In future work, we plan to apply
8

---

Published as a conference paper at ICLR 2020
other algorithms requiring discrete inputs audio data and to explore self-supervised pre-training algorithms which mask part of the continuous audio imput. Another future work avenue is to fine- tune the pre-trained model to output transcriptions instead of feeding the pre-trained features to custom ASR model.
REFERENCES
Dario Amodei, Sundaram Ananthanarayanan, Rishita Anubhai, Jingliang Bai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Qiang Cheng, Guoliang Chen, al. Deep speech 2: End- to-end speech recognition in english and mandarin. In Proc. of ICML, 2016.
Mathilde Caron, Piotr Bojanowski, Julien Mairal, and Armand Joulin. Unsupervised pre- pre-training of image features on non-curated data. In Proceedings of the International Conference on Computer Vision (ICCV), 2019.
Jan Chorowski, Ron Weiss, Samy Bengio, and Aäron van den Oord. Unsupervised speech repre- sentation learning using wavenet autocncoders. arXiv, abs/1901 000000,081,,,,.. 2019.
Yu-An Chung and James Glass. Speech2vec: A sequence-to to--eneee------eeeeeee- framework for learning word embeddings from speech. arXiv, hass/1803.08966, 2018.
Yu-An Chung, Wei-Ning Hsu, Hao Tang, and James Glass, An unsupervised autoregressive model for speech representation learning. arXiv, abs/1 abs/1904.03240, 2019.
Ronan Collobert, Christian Puhrsch, and Gabriel Synnaeve. Wav2letter: an end-to-end convnet- based speech recognition system. arXiv, abs/1609.03193, 2016.
Ronan Collobert, Awni Hannun, and Gabriel Synnaeve. A fully differentiable beam search decoder. arXiv,, abs/ 1ess/992.0602,, 2019.
FFmpeg Developers. fimpeg tool software, 2016. http:///://ffmegg
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv, abs/1810.04805, 2018.
Ewan Dunbar, Robin Algayres, Julien Karadayi, Mathieu Bernard, Juan Benjumea, Xuan-Nga Cao, Lucie Miskic, Charlotte Dugrain, Lucas Ondel, Alan W Black, et al. The zero resource speech challenge 2019; Tts without arXiv, 1904. 1469, 2019.
Ryan Eloff, André Nortje, Benjamin van Niekerk, Avashna Govender, Leanne Nortje, Arnu Preto- rius, Elan Van Biljon, Ewald van der Westhuizen, Lisa van Staden, and Herman Kamper. Unsuper- vised acoustic unit discovery for speech synthesis using discrete latent-variable neural networks. arXiv, abs/1904.00556, 2019.
John S. Garofolo, David Graff, Doug Paul, and David S. Pallett. CSR-I (WSJO) Complete LDC93S6A. Web Download. Linguistic Data Consortium, 1993a.
John S. Garofolo, Lori F. Lamel, William M. Fisher, Jonathon G. Fiscus, David S. Pallett, and Nancy L. Dahlgren. The DARPA TIMIT Acoustic-Phonetic Continuous Speech Corpus CDROM. Linguistic Data Consortium, 1993b.
Pegah Ghahremani, Vimal Manohar, Hossein Hadian, Daniel Povey, and Sanjeev Khudanpur. In- vestigation of transfer learning for asr using If-mmi trained neural networks. In Proc. of ASRU, 2017.
Emil Julius Gumbel. Statistical theory of extreme values and some practical applications: series of lectures, volume 33. US Government Printing Office, 1954.
Hossein Hadian, Hossein Sametil, Daniel Povey, and Sanjeev Khudanpur. End-to-end speech recog- nition using lattice-free mmi. In Proc. of Interspeech, 2018.
Kenneth Heafield, Ivan Pouzyrevsky, Jonathan H. Clark, and Philipp Koehn. Scalable modified Kneser-Ney language model estimation. In Proc. of ACI., 2013.

---

Published as a conference paper at ICLR 2020
Kazuki Irie, Rohit Prabhavalkar, Anjuli Kannan, Antoine Bruguier, David Rybach, and Patrick Nguyen. On the choice of modeling unit for perueence-t--eqqeeee speech recognition. Inter- speech 2019, Sep 2019. doi: 10.21437 10.21457/meesppechh201---4 10. 21 437/ Interspeech. 201 9-2277.
URL http://dx..c http::///xdooi.oogg//..
Eric Jang. Shixiang Gu, and Ben Poole. Categorical repaaraaaaaaaaaaeeiiiiiitooo with gumbel-softmax. arXiv, abs/1611.01144 2016.
Herve Jegou, Matthijs Douze, and Cordelia Schmid. Product quantization for nearest neighbor search. IEEE Trans. Pattern Anal. Mach. Intell., 33(1):117-128, January 2011.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoye, and Omer Levy. Span- bert: Improving pre-training by representing and predicting spans. arXiv, abs/1907.10529, 2019.
Tatiana Likhomanenko, Gabriel Synnaeve, and Ronan Collobert. Who needs words? lexicon-free speech recognition. In Proc. of Interspeech, 2019.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprini arXiv: 1907.11699 2019.
Ilya Loshchilov and Frank Hutter. abs/ 000803 2016
SGDR: stochastic gradient descent with restarts.
arXiv,
Chris Maddison, Daniel Tarlow, and Tom Minka. A sampling. In Advances Neural Information Processing Systems, pp. 3086-3094, 2014.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representa- tions of words and phrases and their compositionality. In Proc. of NIPS, 2013.
Abdelrahman Mohamed, Dmytro Okhonko, and Luke Zettlemoyer. Transformers with convolutional context for ASR. CoRR, abs/ 904. 11660, 2019.
C Montgomery. Vorbis specification, 2004.
Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. Scaling neural machine translation.
In Proc. of WMT, 2018.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: fast, extensible toolkit for sequence modeling. In Proc. of NAACL System Demonstrations, 2019.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus based on public domain audio books. In Proc. of ICASSP, pp. 5206-5210. IEEE, 2015.
Daniel S. Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D. Cubuk, and Quoc V. Le. Specaugment: A simple data augmentation method for automatic speech recognition, 2019.
Mirco Ravanelli, Philemon Brakel, Maurizio Omologo, and Yoshua Bengio. Light gated recurrent units for speech recognition. IEEE Transactions on Emerging Topics in Computational Intelli- gence, 2(2):92-102, 2018
Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. wav2vee: Unsupervised pre-training for speech recognition. CoRR, abs/1904.05862, 2019. URL http://arxiv org/abs aataabss190 1904 4. 05862.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proc. of ACL, 2016.
Tim Terriberry and Koen Vos. Definition of the opus audio codec, 2012.
Andros Tjandra, Berrak Sisman, Mingyang Zhang, Sakriani Sakti, Haizhou Li, and Satoshi Naka- mura. Vqvae unsupervised unit discovery and multi-scale code2spec inverter for zerospeech chal- lenge 2019. arXiv, 1905. 1449, 2019.
10

---

Published as a conference paper at ICLR 2020
Aaron den Oord, Oriol Vinyals, al. Neural discrete representation learning. In Advances in Neural Information Processing Systems, pp. 6306-6315, 2017.
Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv, abs/ 2018
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proc. of NIPS, 2017.
Maarten Versteegh, Xavier Anguera, Aren Jansen, and Emmanuel Dupoux. The zero resource speech challenge 2015: Proposed approaches and results. Procedia Computer Science, 81:67- 72, 2016.
Yuxin Wu and Kaiming He. Group normalization. arXiv, abs/1803.000
01118003.888 00440, 2018.
Neil Zeghidour, Nicolas Usunier, lasonas Kokkinos, Thomas Schaiz, Gabriel Synnaeve, and Em- manuel Dupoux. Learning filterbanks from raw speech for phone recognition. In Proc. of (ICASSP), 2018.

---

Published as a conference paper at ICLR 2020
APPENDIX A NUMBER OF VARIABLES VS. GROUPS
We investigate the relationship between number of variables and groups G. Table 6 shows that multiple groups beneficial compared single group with large number variables. Table shows that with a single group and many variables, only a small number of codewords survive
group
2 groups
4 groups
8 groups
16 groups
32 groups
40
33.44
0.24
23.52 + 0.53
18.76 + 0.20
17.43 + 0.14
15.97 = 0.21
15.44 0.32
80
29.14 + 0.70
25.36 + 4.62
17.32 + 0.28
16.36 0.27
17.55 0.27
15.49 + 0.14
160
24.27 0.35
17.35 0.03
16.36 0.13
15.64 = 0.03
15.11 0.10
320
27.22 0.25
20.86 0.09
16.49 + 0.07
15.88
5.74400.11
15.18 0.02
640
26.53 2.02
0.12
16.60 + 0.22
15.62
1.14
15.54 0.31
1280
32.63 + 5.73
18.04 + 0.26
16.37 0.07
15.85 0.05
15.13 0.29
15.18 0.05
Table 6: PER TIMIT dev set for vq-wav2vee models trained on Libri100. Results are based three random seeds.
V
group
groups
4 groups
8 groups
16 groups
32 groups
40
100 % (40)
95.3 % (1.6k)
27.4 % (2.56M)
74.8 % 9M)
99.6 % (39 9M)
99.9 % (39 9M))
80
92.5 % (80)
78.5 % (6.4k)
11.8% (39.9M)
91.5 % (39.9M)
99.3 % (39.9M)
100 % (39.9M)
160
95 % (160)
57.2% % (25.6k)
35.2% (39.9M)
97.6% % (39.9M)
99.8 % (3).9M)
100 (39.9M)
320
33.8 % (320)
24.6 (102.4k)
57.3 (39.9M)
98.7 % (39.9M)
99.9% (39.9M)
100 % (39.9M)
640
24.6% % (640)
10 % (409.6k)
60.2 (39.9M)
99.3 % (39.9M)
99.9 % (39.9M)
100 % (39.9M)
1280
7.2 % (1.28k)
4.9% (1.63M)
67.9 % (39.9M)
99.5 % (39.9M)
99.9 % (39.9M)
100 % (39.9M)
Table 7: Fraction of used codewords vs. number of theoretically possible codewords V in brackets; 39. 39.9M is the number of tokens in Librispeech 100h.
12


Reviews:
[{"review_id": "rJxLRp2s9H", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a method for unsupervised representation learning of speech. The idea is to first learn discrete representation (vector quantization is done by Gumbel softmax or k-means) from audio samples with contrastive prediction coding type objective, and then perform BERT-style pre-training (borrowed from NLP). The BERT features are used as inputs to ASR systems, rather than the usual log-mel features. The idea, which combines those of previous work (wav2vec and BERT) synergetically, is intuitive and clearly presented, significant improvements over log-mel and wav2vec were achieved on ASR benchmarks WSJ and TIMIT. Based on these merits, I suggest this paper to be accepted.\n\nOn the other hand, I would suggest directions for investigation and improvements as follows.\n\n1. While I understand that vector quantization makes the use of NLP-style BERT-training possible (as the inputs to NLP models are discrete tokens),  there are potential disadvantages as well. One observation from the submission is that the token set may need to very large (from tens of thousands to millions) for the system to work well, making the BERT training computationally expensive (I noticed that the BERT model is trained on 128 GPUs). Also, without BERT pre-training, using directly the discrete tokens seems to consistently give worse performance for ASR. I think some more motivations or explorations (what kind of information did BERT learn) are needed to understand why that is the case.\n\n2. Besides the computational expensive-ness of the three-step approach (vector quantization, BERT, acoustic model training), the combined model complexity is large because these steps do not share neural network architecture. A more economical approach is to use BERT-trained model as initialization for acoustic model training, which is the classical way how RBMs pre-training were used in ASR.\n\n3. One concern I have with discrete representation is how robust they are wrt different dataset. The ASR datasets used in this work are relatively clean (but there does exists domain difference between them). It remains to see how the method performs with more acoustically-challenging speech data, and how universally useful the learned features are (as is the case for BERT in NLP).\n\n4. Another curious question is whether the features would still provide as much improvement when a stronger ASR system than AutoSeg (e.g., Lattice-free MMI) is used.\n\nOverall, while I think the computational cost of the proposed method is high, rendering it less practical at this point, I believe the approach has potential and the result obtained so far is already significant.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "r1e9Ipmo_r", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nOverview:\n\nThis paper considers unsupervised (or self-supervised) discrete representation learning of speech using a combination of a recent vector quantized neural network discritization method and future time step prediction. Discrete representations are fine-tuned by using these as input to a BERT model; the resulting representations are then used instead of conventional speech features as the input to speech recognition models. New state-of-the-art results are achieved on two datasets.\n\nStrengths:\n\nThe core strength of this paper is in the results that are achieved on standard speech recognition benchmarks. The results indicate that, while discritization in itself does not give improvements, coupling this with the BERT-objective results in speech features which are better in downstream speech recognition than standard features. I think the main technical novelty is in combining discritization with future time step prediction (but see the weakness below).\n\nWeaknesses:\n\nThe main weakness of the paper is that it does not situate itself within existing literature in this area. Over the last few years, researchers in the speech community have invested significant effort in learning better speech representations, and this is not discussed. See e.g. [1]. Even more importantly, very recently there has been a number of papers investigating discrete representations of speech; see the review [2]. Some of these papers specifically use VQ-VAEs [3]. [4] actually compares VQ-VAE and the Gumbel-Softmax approach. These studies should be mentioned. This paper is different in that it incorporates future time step prediction. But context prediction has also been considered before, also for speech [5, 6, 7]. This paper can be situated as a new contribution combining these two strands of research. In the longer run it would be extremely beneficial to the community if this approach is applied to the standard benchmarks as set out in [2].\n\nAs a minor weakness, some parts of the paper is not described in enough detail and the motivation is weak or not exactly clear (see detailed comments below).\n\nOverall assessment:\n\nI think the results as well as the new combination of existing approaches in the paper warrants publication. But it should be amended significantly to situate itself within the existing literature. I therefore award a \"weak accept\".\n\nDetailed questions and suggestions:\n\n- Section 1: As motivation for this work, it is stated that \"we aim to make well performing NLP algorithms more widely applicable\". As noted above, some NLP-like ideas (such as prediction of future speech segments, stemming from text-based language modelling) have already been considered within the speech community. Rather than motivating the work in this way, it might be helpful to focus the contribution as a combination of future time step prediction and discretization (both of which have been considered in previous work, but not in combination).\n- Section 4: Would it be possible to train the vq-wav2vec model jointly with BERT, i.e. as one model? I suspect it would be difficult since, for the masking objective, the discrete units are already required, but maybe there is a scheme where this could work.\n- Section 2.2: Similarly to the above question, would there be a way to incorporate the BERT principles directly into an end-to-end model, e.g. by randomly masking some of the continuous input speech?\n- Section 3.3: What exactly does \"mode collapse\" refer to in this context? Would this be using only one codebook entry, for instance?\n- Section 6: It seems that in all cases to obtain improvements from discritization, BERT is required on top of the vq-wav2vec discrete symbols. Is it possible that the output acoustic model is simply better-matched to continuous rather than discrete input (direct vq-wav2vec gives discrete while BERT gives continuous)? Would it make sense to train the wav2vec acoustic model on top of the vqvae codebook entries (e) instead of directly on the symbols?\n\nTypos, grammar and style:\n\n- \"gumbel\" -> \"Gumbel\" (throughout; or just be consistent in capitalization)\n- \"which can be mitigated my workarounds\" -> \"which can be mitigated *by* workarounds\"\n- \"work around\" -> \"workaround\"\n\nMissing references:\n\n1. Versteegh, M., Anguera, X., Jansen, A. & Dupoux, E. (2016). The Zero Resource Speech Challenge 2015: Proposed Approaches and Results. In SLTU-2016 Procedia Computer Science, 81, (pp 67-72).\n2. https://arxiv.org/abs/1904.11469\n3. https://arxiv.org/abs/1905.11449\n4. https://arxiv.org/abs/1904.07556\n5. https://arxiv.org/abs/1904.03240\n6. https://arxiv.org/abs/1807.03748 (this paper is cited)\n7. https://arxiv.org/abs/1803.08976\n\nEdit: Based on the feedback from the authors, I changed my rating from a 'weak accept' to an 'accept'.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HyxsH-ORFH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a way to pre-train quantized representations for speech. The approach proposed is a two-stage process: 1. train a quantized version of wav2vec [my understanding is that wav2vec is the same thing as CPC for Audio except for using a binary cross-entropy loss instead of InfoNCE softmax-cross entropy loss]. the authors propose to use gumbel softmax / VQ codebook for the vector quantization.\n2. once you have a discrete representation, you could train BERT (as if it were a seq of language tokens). this makes a lot of sense especially given that CPC / wav2vec recovers phonemes and quantizing the phonemes will recover a language-like version of the raw audio. And running BERT across those tokens will allow you to capture the dependencies at the phoneme level. \n\nAfter pre-training, the authors use the learned representations for speech recognition. They compare this to using log-mel filterbanks. \n\nThe results (WER / LER) is lower for the proposed pipeline compared to using dense wav2vec representation for n-gram and character LM.  It also makes sense that BERT helps for the k-means (vq) setting since the number of codes is large. \n\nThe authors also cleverly adopt/adapt span-BERT which is more suited to this setting.\n\nI think this paper presents a useful contribution as far as improving speech / phoneme recognition using self-supervised learning goes, and also has useful engineering aspects in terms of combining CPC and BERT. I would like to see this paper accepted.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "B1xXOITI9H", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Though rather dense in its exposition, this paper is an interesting contribution to the area of self-supervised learning  based on discrete representations. What would make it stronger imo is to address the issue of how much is gained from a discrete vs. continuous representation. The authors take it as a given that discrete is good because it allows us to leverage work in NLP. That makes sense -- but at what cost?\n\n\"Table 4 shows that our first results are promising, even though they are not as good as the state of the art.\" The state of the art on LibriSpeech is not Mohamed at al. 2019. See e.g. Irie et al. Interspeech 2019 for better result\n\nThe Conclusion is very sparse. \"In future work, we are planning to apply other algorithms requiring discrete inputs to audio data\": can  you elaborate?", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new self-supervised pre-trained speech model that improves speech recognition performance. 
 The idea combines an earlier pre-training approach (wav2vec) with discretization followed by BERT-style masked reconstruction.  The result is a fairly complex approach, with not too much novelty but with a good amount of engineering and analysis, and ultimately very good performance.  The reviewers agree that the work deserves publication at ICLR, and the authors have addressed some of the reviewer concerns in their revision.  The complexity of the approach may mean that it is not immediately widely adopted by others, but it is a good proof of concept and may well inspire other related work.  I believe the ICLR community will find this work interesting.
Decision: Accept (Poster)