Title: Latent Sequence Decompositions
Authors: William Chan
Yu Zhang
Quoc Le
Navdeep Jaitly
Venue: ICLR 2017
Keywords: Speech
Applications
Natural language processing
Deep learning

Abstract: Sequence-to-sequence models rely on a fixed decomposition of the target sequences into a sequence of tokens that may be words, word-pieces or characters. The choice of these tokens and the decomposition of the target sequences into a sequence of tokens is often static, and independent of the input, output data domains. This can potentially lead to a sub-optimal choice of token dictionaries, as the decomposition is not informed by the particular problem being solved. In this paper we present Latent Sequence Decompositions (LSD), a framework in which the decomposition of sequences into constituent tokens is learnt during the training of the model. The decomposition depends both on the input sequence and on the output sequence. In LSD, during training, the model samples decompositions incrementally, from left to right by locally sampling between valid extensions. We experiment with the Wall Street Journal speech recognition task. Our LSD model achieves 12.9% WER compared to a character baseline of 14.8% WER. When combined with a convolutional network on the encoder, we achieve a WER of 9.6%.



Review discussion:
Is it correct to understand the collapse function as a word level mapping from a word to a valid subword sequence (or the other way around). If this is correct, it works like a dictionary, and do you use it to constrain your search of \hat{z} as eq(18)?
Otherwise, if you do best path search or beam search of \hat{z} without this constraint, you may end up with a sequence of \hat{z} that does not correspond to any valid word sequence.
It is still straightforward to incorporate a language model into the decoder, do you have any result of that other than Table 2 only?
The first term in the right side of eq (4) should be \frac{1}{p(y|x; \theta} instead of \frac{1}{p(y|x, z;\theta}?
Hello AnonReviewer2,
Thank you for your question and comment.
Thanks for pointing out the typo -- we have fixed the text.
Yes, you are correct! Our search process is unconstraint (i.e., no dictionary or LM), and you are correct, mis-spellings of invalid words are possible in the search process. This is indeed the power of the model that it is able to produced structured outputs with correct spellings without having any such explicit dictionary. This is due to the seq2seq framework using the chain rule factorization. A similar search process is used for character models (Chan et al., 2016; Bahdanau et al., 2016), and when scaled to larger datasets, there can be virtually no spelling mistakes at all (Chan et al., 2016).
We did not seriously explore searching with a dictionary/LM because we were interested in, 1] exploring the learning of decompositions and 2] dropping the n-gram LM from the search process (since it is big and expensive to carry it around). Adding a dictionary/LM does improve the numbers slightly, we can update the paper with numbers if you would like.
Chan et al., "Listen, Attend and Spell: A Neural Network for Large Vocabulary Conversational Speech Recognition" in ICASSP 2016.
Bahdanau et al., "End-to-End Attention-based Large Vocabulary Speech Recognition," in ICASSP 2016.
Could the authors give more motivation for input (acoustic) dependent segmentation of words?
Why have the authors not compared their method with any publicly available segmenter to get the sub-word units (assuming the segmentation should not depend on acoustic), e.g. morfessor?
During estimation of the gradient and sampling z with (approximated) p(z|y,x;\theta), how do you obtain the posteriors of p(z_t|y,x,z<t;\theta) exactly, when not the uniform distribution is used?
How many sample of z is used to estimate the gradient? If only one, why? Have you tried more?
There is some mismatch between training (sum over many z) and decoding (most probable z). Could you comment on that?
Hello AnonReviewer4,
Thank you for your excellent questions and comments. We will answer your questions below.
Q: Could the authors give more motivation for input (acoustic) dependent segmentation of words?
We believe speaking styles (i.e., speaking rates/accents/noise conditions) can have an effect on the segmentation of the words.
For example, we found the top segmentation discovered by beam search varies from utterance to utterance for many words.
Q: Why have the authors not compared their method with any publicly available segmenter to get the sub-word units (assuming the segmentation should not depend on acoustic), e.g. morfessor?
First, we wanted to build an end-to-end model without dependencies an any external models. This simplifies the training process and removes domain-specific knowledge.
Second, while we did not compare to a segmentation generated from a LM/morfessor, we did compare to a simple heuristic, namely the greedy heuristic wherein we greedily select the longest word piece possible.
Finally, we do believe if we used a LM/expert model to select our word piece vocabulary (as opposed to the current count-based heuristic) under the LSD-framework, we may get some small gains (however, once again, this would make the model rely on prior information).
Q: During estimation of the gradient and sampling z with (approximated) p(z|y,x;\theta), how do you obtain the posteriors of p(z_t|y,x,z<t;\theta) exactly, when not the uniform distribution is used?
p(z_t|y,x,z<t;\theta) can be sampled directly from the model. We sample p(z|y,x;\theta) with ancestral sampling -- note that this sampling is biased, but we address this issue in the paper in Section 2.
Q: How many sample of z is used to estimate the gradient? If only one, why? Have you tried more?
We used 1 sample; we used 1 sample for computational efficiency reasons. We tried using more samples w/ REINFORCE+WER optimization, however we were not able to get any improvement.
Q: There is some mismatch between training (sum over many z) and decoding (most probable z). Could you comment on that?
You are correct, there is a mismatch between the train and test. Exact inference is obviously intractable (even for determining most probable z due to beam search). We could do approximate inference w/ something like importance sampling.
However, we found good results without needing to complicate the search process. We simply did a standard left-to-right beam search on z, and we found good results with that. We however do agree that future work can enhance the search process, which should result in even better WERs.
This submission proposes to learn the word decomposition, or word to sub-word sequence mapping jointly with the attention based sequence-to-sequence model. A particular feature of this approach is that the decomposition is not static, instead, it also conditions on the acoustic input, and the mapping is probabilistic, i.e., one word may map to multiple sub-word sequences. The authors argue that the dynamic decomposition approach can more naturally reflect the acoustic pattern. Interestingly, the motivation behind this approach is analogous to learning the pronunciation mixture model for HMM based speech recognition, where the probabilistic mapping from a word to its pronunciations also conditions on the acoustic input, e.g.,
I. McGraw, I. Badr, and J. Glass, "Learning lexicons form speech using a pronunciation mixture model," in IEEE Transactions on Audio, Speech, and Language Processing, 2013
L. Lu, A. Ghoshal, S. Renals, "Acoustic data-driven pronunciation lexicon for large vocabulary speech recognition", in Proc. ASRU
R. Singh, B. Raj, and R. Stern, "Automatic generation of subword units for speech recognition systems,"  in IEEE Transactions on Speech and Audio Processing, 2002
It would be interesting to put this work in the context by linking it to some previous works in the HMM framework.
Overall, the paper is well written, and it is theoretically convincing. The experimental study could be more solid, e.g., it is reasonable to have a word-level baseline, as the proposed approach lies in between the character-level and word-level systems. the vocabulary size of the WSJ si284 dataset is 20K at maximum, which is not very large for the softmax layer, and it is a closed vocabulary task. I guess the word-level system may be also competitive to the numbers reported in this paper.
Furthermore, can you explain what is the computational bottleneck of the proposed approach? You downsampled the data by the factor of 4 using an RNN, and it still took around 5 days to converge. To me, it is a bit expensive, especially given that you only take one sample when computing the gradient. Table 2 is a little bit misleading, as CTC with language model and seq2seq with a language model model from Bahdanau et al. is much closer to the best number reported in this Table 2, while you may only get a very small improvement using a language model.
Finally, "O(5) days to converge" sounds a bit odd to me.
Hello AnonReviewer2,
Thank you for your review and excellent questions. We will address them below:
Q1: The experimental study could be more solid, e.g., it is reasonable to have a word-level baseline, as the proposed approach lies in between the character-level and word-level systems. the vocabulary size of the WSJ si284 dataset is 20K at maximum, which is not very large for the softmax layer, and it is a closed vocabulary task. I guess the word-level system may be also competitive to the numbers reported in this paper.
We did not compare to a word baseline because we believe a word segmentation will actually do much much more worse than a character baseline. End-to-end seq2seq models w/ word-level ASR are extremely hard to learn due to overfitting problems. Characters (and word pieces) contain much more phonetic information compared to full words (which makes it easier to learn + generalization). CTC experiments show that unless you have large datasets (125k hrs), subword units outperform word units (Sak et al., 2015 and Soltau et al., 2016); and seq2seq models overfit much more easily due to the lack of conditional independence assumptions (especially since there are only ~37k utterances in WSJ train).
Q2: Furthermore, can you explain what is the computational bottleneck of the proposed approach? You downsampled the data by the factor of 4 using an RNN, and it still took around 5 days to converge. To me, it is a bit expensive, especially given that you only take one sample when computing the gradient.
We downsample the encoder RNN not for computational considerations, but rather for model WER considerations. We found when we used hierarchical subsampling, our model performs better (Chan et al., 2016).
The computational bottleneck is due to the attention mechanism. There are several ways to speedup our model:
1] Use a sliding window to transform the model from O(nm) to O(m) where n == size of acoustic signal, m == number of output tokens, see Chorowski et al., 2015.
2] Make the model online, see Neural Transducers by Jaitly et al., 2016.
We also admit our LSTM implementation is suboptimal, NVIDIA recently released the cuDNN LSTM API which our model will substantially benefit in wallclock training times.
Q3: Table 2 is a little bit misleading, as CTC with language model and seq2seq with a language model model from Bahdanau et al. is much closer to the best number reported in this Table 2, while you may only get a very small improvement using a language model.
We do not compare to other models with LM -- we are interested in end-to-end models.
For example, a n-gram LM has a very large memory footprint which makes it impractical to deploy on any small mobile devices. For experiments with seq2seq+LM, please see the recently published Chorowski et al., 2016.
Q4: Finally, "O(5) days to converge" sounds a bit odd to me.
Thank you for pointing this out to us, we will fix this.
Citations
Chan et al., "Listen, Attend and Spell: A Neural Network for Large Vocabulary Conversational Speech Recognition," in ICASSP 2016.
Chorowski et al., “Attention-based models for speech recognition” in NIPS 2015.
Chorowski et al., “Towards better decoding and language model integration in sequence to sequence models” in arXiv 2016.
Jaitly et al., “A Neural Transducer” in NIPS 2016.
Sak et al., “Fast and Accurate Recurrent Neural Network Acoustic Models for Speech Recognition” in INTERSPEECH 2015.
Soltau et al., “Neural Speech Recognizer: Acoustic-to-Word LSTM Model for Large Vocabulary Speech Recognition” in arXiv 2016.
Interesting paper which proposes jointly learning automatic segmentation of words to sub words and their acoustic models.
Although the training handles the word segmentation as hidden variable which depends also on the acoustic representations, during the decoding only maximum approximation is used.
The authors present nice improvements over character based results, however they did not compare results with word segmentation which does not assume the dependency on acoustic.
Obviously, only text based segmentation would result in two (but simpler) independent tasks. In order to extract such segmentation several publicly open tools are available and should be cited. Some of those tools can also exploit the unigram probabilities of the words to perform their segmentations.
It looks that the improvements come from the longer acoustical units - longer acoustical constraints which could lead to less confused search -, pointing towards full word models. In another way, less tokens are more probable due to less multiplication of probabilities. As a thought experiment for an extreme case: if all the possible segmentations would be possible (mixture of all word fragments, characters, and full-words), would the proposed model use word fragments at all? (WSJ is a closed vocabulary task). It would be good to show that the sub word model could outperform even a full-word model (no segmentation).
Your model estimates p(z_t|x,z<t;\theta), but during training both p(z_t|x,y,z<t;\theta) and p(z_t|x,z<t;\theta) are needed. Their connections and calculations, in general Section 2 should be more detailed. The experimental setup should be presented in a replicable way.
The authors use one sample of Z during the gradient calculation. Would not it be more consistent then to use the maximum instead of sampling? Or is even getting the maximum is computationally too expensive? Considering the left-to-right approximation, this might be possible and more consistent with your decoding, please address this issue in details.
Hello AnonReviewer4,
Thank you for your review and excellent questions. We will address them below:
Q1: The authors present nice improvements over character based results, however they did not compare results with word segmentation which does not assume the dependency on acoustic.
We did not compare to a word baseline because we believe a word segmentation will actually do much much more worse than a character baseline. End-to-end seq2seq models w/ word-level ASR are extremely hard to learn due to overfitting problems. Characters (and word pieces) contain much more phonetic information compared to full words (which makes it easier to learn + generalization). CTC experiments show that unless you have large datasets (125k hrs), subword units outperform word units (Sak et al., 2015 and Soltau et al., 2016); and seq2seq models overfit much more easily due to the lack of conditional independence assumptions (especially since there are only ~37k utterances in WSJ train).
Q2: It looks that the improvements come from the longer acoustical units - longer acoustical constraints which could lead to less confused search -, pointing towards full word models.
That is actually not true! We did an analysis in Section 5 (and Figure 1). We compare our LSD model to a maximum extension (MaxExt) model. The MaxExt model is greedy in that it always selects the longest word pieces. In Section 5 and Figure 1, we showed that the MaxExt model by-far chooses to use longer word pieces compared to the LSD model.
However, the LSD model performs significantly better than the MaxExt model.
In fact, the MaxExt model does not even beat the character baseline (see Table 1)!
We agree that the LSD benefits from using the word pieces (as opposed to the smallest unit of characters), however we disagree that full word models would do better.
Q3: As a thought experiment for an extreme case: if all the possible segmentations would be possible (mixture of all word fragments, characters, and full-words), would the proposed model use word fragments at all? (WSJ is a closed vocabulary task). It would be good to show that the sub word model could outperform even a full-word model (no segmentation).
Actually, under the LSD framework, our word piece vocab includes the characters and word pieces. The word pieces are up to size “n”. Since n <= 5, this doesn’t allow for “all possible segmentations”, it does allow common word segmentation assuming the word length is <= 5. The LSD model still chooses to decompose via word pieces rather than whole-words for many decompositions.
For example, “from” is in-vocabulary, and our LSD model chooses to decompose “|from|” as “|fro|m|” (please see Appendix A for full details).
Q4: Your model estimates p(z_t|x,z<t;\theta), but during training both p(z_t|x,y,z<t;\theta) and p(z_t|x,z<t;\theta) are needed. Their connections and calculations, in general Section 2 should be more detailed. The experimental setup should be presented in a replicable way.
Thank you for pointing this out — we will update the text to make the mathematics more clear. As mentioned in the earlier comments, we need p(z_t|x,z<t;\theta), which we can simply sample from the model. We also need p(z|x,y;\theta) which we approximate via ancestral sampling in a left-to-right greedy fashion.
Q5: The authors use one sample of Z during the gradient calculation. Would not it be more consistent then to use the maximum instead of sampling? Or is even getting the maximum is computationally too expensive? Considering the left-to-right approximation, this might be possible and more consistent with your decoding, please address this issue in details.
If we used the maximum (instead of sampling), we may never learn/discover the true posterior (especially if the true posterior is multimodal). Here are a few examples:
1] If we used maximum (instead of sampling), we can only learn a unimodal distribution. The true posterior can be multimodal.
For example, “mister” vs “mr” or “triple a” vs “aaa” (Chan et al., 2016).
2] If we used maximum (instead of sampling), the distribution learnt will collapse to the first mode discovered. There is no guarantee the first mode discovered would even be the best mode.
It is true, our decoding procedure is slightly different than our training procedure. This is due to the fact that during decoding, it is intractable to marginalize over all possible segmentations.
In theory, we could approximate the marginalization via importance sampling, however this is itself still an expensive procedure (especially if you want good estimates). We found empirically that using our left-to-right beam search gave very good results already. We suspect that if we built an importance sampling decoder (which more closely aligns w/ the training procedure), we would achieve even better results simply via the search process. We agree future work can improve in this.
Sak et al., “Fast and Accurate Recurrent Neural Network Acoustic Models for Speech Recognition” in INTERSPEECH 2015.
Soltau et al., “Neural Speech Recognizer: Acoustic-to-Word LSTM Model for Large Vocabulary Speech Recognition” in arXiv 2016.
This paper proposes to learn decomposition of sequences (such as words) for speech recognition. It addresses an important issue and I forsee it being useful for other applications such as machine translation.
While the approach is novel and well-motivated, I would very much like to see a comparison against byte pair encoding (BPE). BPE is a very natural (and important) baseline (i.e. dynamic vs fixed decomposition). The BPE performance should be obtained for various BPE vocab sizes.
Minor points
- Did the learned decompositions correspond to phonetically meaningful units? From the example in the appendix it's hard to tell if the model is learning phonemes or just most frequent character n-grams.
- Any thoughts on applications outside of speech recognition? If this is shown to be effective in other domains it would be a really strong contribution (but this is probably outside the scope for now).
Hello AnonReviewer6,
Thank you for your review and excellent questions. We will address them below:
Q1: While the approach is novel and well-motivated, I would very much like to see a comparison against byte pair encoding (BPE). BPE is a very natural (and important) baseline (i.e. dynamic vs fixed decomposition). The BPE performance should be obtained for various BPE vocab sizes.
We did not compare to BPE, but we did compare to a Maximum Extension (MaxExt) baseline. The MaxExt (described in Section 5) is a left-to-right greedy longest sub-word unit fixed decomposition. Note, the MaxExt baseline is exactly equivalent to BPE (w/ 1 pass) when using 2-grams for MaxExt. For n>2, MaxExt is similar but not exactly equivalent to BPE;
however we believe their performance should be similar (since both algorithms are greedy in nature). We varied the vocab size for MaxExt and we did not find a significant difference in WER.
Q2: Did the learned decompositions correspond to phonetically meaningful units? From the example in the appendix it's hard to tell if the model is learning phonemes or just most frequent character n-grams.
We have not done a formal analysis, but we noticed empirically that units such as |ing| are consistently learnt to be used. Even from the appendix, we find |pro|fi|t| and |sp|ok|e|s|wo|ma|n| are reasonable/meaningful phonetically units.
The model does not focus on just the most frequent character n-grams. Evidence is given in Section 5 Figure 1.
When given a vocab of longer units, the model does shift to prefer them (as opposed to character bigrams).
However, compared to MaxExt, the model does not prefer the shortest decomposition (by using the longest sub-word units). The MaxExt decomposition performs worse than the LSD decompositions.
Q3: Any thoughts on applications outside of speech recognition? If this is shown to be effective in other domains it would be a really strong contribution (but this is probably outside the scope for now).
We believe LSD can be used where there are multiple ways to decompose an output sequence, and where the output decomposition should be a function of the input sequence as well. Possible other domain applications include language, robotics, planning and many games.

Meta-review:
role: PC
data: comment: This work proposes a method for segmenting target generation sequence that is learned as part of the model. Generally all reviewers found this paper novel and interesting.
 
 Pros:
 - Quality: The paper is both containing "with solid theoretically justified" and "present(s) nice improvements over character based result". One reviewer asked for the "the experimental study could be more solid."
 - Impact: methods like "BPE" are now somewhat standard hacks in seq2seq modeling. This type of model could be potentially impactful at disrupting. 
 - Clarity: Reviewers found the work to be a "clearly written paper" 
 
 Cons:
 - Some of the reviewers were not as enthuthiastic about this work compared to other papers. There were several comments asking for further experimental results. However I found that the author's responses clearly explained these away and provided clear justificition for why they were not necessary, already included, or explained by previous results. I feel that this takes care of the major issues, and warrants a small raise in score.

Decision: Accept (Poster)