DEJA-VU: DOUBLE FEATURE PRESENTATION AND ITERATED LOSS
IN DEEP TRANSFORMER NETWORKS
Andros Tjandra1∗ , Chunxi Liu2 , Frank Zhang2 , Xiaohui Zhang2 , Yongqiang Wang2 ,Gabriel Synnaeve2 , Satoshi Nakamura1 , Geoffrey Zweig2
1

Nara Institute of Science and Technology, Japan
2
Facebook AI, USA{andros.tjandra.ai6,s-nakamura}@is.naist.jp,{chunxiliu,frankz,xiaohuizhang,yqw,gab,gzweig}@fb.com

ABSTRACT
Deep acoustic models typically receive features in the first layer ofthe network, and process increasingly abstract representations in thesubsequent layers. Here, we propose to feed the input features atmultiple depths in the acoustic model. As our motivation is to allow
acoustic models to re-examine their input features in light of partial
hypotheses we introduce intermediate model heads and loss function. We study this architecture in the context of deep Transformer
networks, and we use an attention mechanism over both the previous
layer activations and the input features. To train this model’s intermediate output hypothesis, we apply the objective function at eachlayer right before feature re-use. We find that the use of such iterated
loss significantly improves performance by itself, as well as enabling
input feature re-use. We present results on both Librispeech, and alarge scale video dataset, with relative improvements of 10 - 20% for
Librispeech and 3.2 - 13% for videos.
Index Terms— transformer, deep learning, CTC, hybrid ASR
1. INTRODUCTION
In this paper, we propose the processing of features not only in the
input layer of a deep network, but in the intermediate layers as well.
We are motivated by a desire to enable a neural network acoustic
model to adaptively process the features depending on partial hypotheses and noise conditions. Many previous methods for adaptation have operated by linearly transforming either input features
or intermediate layers in a two pass process where the transform islearned to maximize the likelihood of some adaptation data [1, 2, 3].
Other methods have involved characterizing the input via factor analysis or i-vectors [4, 5]. Here, we suggest an alternative approach inwhich adaptation can be achieved by re-presenting the feature stream
at an intermediate layer of the network that is constructed to be correlated with the ultimate graphemic or phonetic output of the system.
We present this work in the context of Transformer networks[6]. Transformers have become a popular deep learning architecture
for modeling sequential datasets, showing improvements in manytasks such as machine translation [6] and language modeling [7]. In
the speech recognition field, Transformers have been proposed toreplace recurrent neural network (RNN) architectures such as longshort-term memory (LSTMs) and gated recurrent units (GRUs) [8].
A recent survey of Transformers in many speech related applications* This work was done while the first author was a research intern at
Facebook.

may be found in [9]. Compared to RNNs, Transformers have several
advantages, specifically an ability to aggregate information across all
the time-steps by using a self-attention mechanism. Unlike RNNs,
the hidden representations do not need to be computed sequentiallyacross time, thus enabling significant efficiency improvements via
parallelization.
In the context of Transformer module, secondary feature analysis is enabled through an additional mid-network transformer module that has access both to previous-layer activations and the raw
features. To implement this model, we apply the objective function
several times at the intermediate layers, to encourage the development of phonetically relevant hypotheses. Interestingly, we find thatthe iterated use of an auxiliary loss in the intermediate layers significantly improves performance by itself, as well as enabling thesecondary feature analysis.
This paper makes two main contributions:
1. We present improvements in the basic training process ofdeep transformer networks, specifically the iterated use ofconnectionist temporal classification (CTC) or cross-entropy(CE) in intermediate layers, and
2. We show that an intermediate-layer attention model with access to both previous-layer activations and raw feature inputs
can significantly improve performance.
We evaluate our proposed model on Librispeech and a largescale video dataset. From our experimental results, we observe 1020% relative improvement on Librispeech and 3.2-11% on the video
dataset.
2. TRANSFORMER MODULES
A transformer network [6] is a powerful approach to learning andmodeling sequential data. A transformer network is itself constructed with a series of transformer modules that each perform
some processing. Each module has a self-attention mechanismand several feed-forward layers, enabling easy parallelization overtime-steps compared to recurrent models such as RNNs or LSTMs[10]. We use the architecture defined in [6], and provide only a briefsummary below.
Assume we have an input sequence that is of length S: X =[x1 , ..., xS ]. Each xi is itself a vector of activations. A transformer
layer encodes X into a corresponding output representation Z =[z1 , ..., zS ] as described below.
Transformers are built around the notion of a self-attention
mechanism that is used to extract the relevant information forloss at the intermediate layer as well. The overall architecture is illustrated in Figure 2. Here, we have used a 24 layer network, withfeature re-presentation after the 12th layer.

Fig. 1. A Transformer Module.

Fig. 2. A 24 layer transformer with one auxiliary loss and feature
re-presentation in the 12-th layer. Z0 represents the input features.
Orange boxes represent an additional MLP network and softmax.
Green boxes represent linear projections and layer-norm.

each time-step s from all time-steps [1..S] in the preceding layer.
Self attention is defined in terms of a Query, Key, Value triplet{Q, K, V } ∈ RS×dk . In self-attention, the queries, keys and values are the columns of the input itself, [x1 , ..., xS ]. The output
activations are computed as:
Q KT
Attn(Q, K, V ) = softmax √
V.
(1)
dk
Transformer modules deploy a multi-headed version of selfattention. As described in [6], this is done by linearly projecting
the queries, keys and values P times with different, learned linear
projections. Self-attention is then applied to each of these projectedversions of Queries, Keys and Values. These are concatenated andonce again projected, resulting in the final values. We refer tothe input projection matrices as WpQ , WpK , WpV , and to the output
projection as WO . Multihead attention is implemented as
MultiAttn(Q, K, V ) = concat(V¯1 , .., V¯P ) WO
where ∀p ∈ {1..P } , V¯p = Attn(QWpQ , KWpK , V WpV ).
Here, WpQ , WpK , WpV
P dm ×dk(2)(3)

dk ×dm∈ R, dm = dk /P , and WO ∈.
After self-attention, a transformer module applies a series of linear layer, RELU, layer-norm and dropout operations, as well as theapplication of residual connections. The full sequence of processing
is illustrated in Figure 1.
R

In the following subsections, we provide detail on the feature
re-presentation mechanism, and iterated loss calculation.
3.1. Feature Re-Presentation
We process the features in the intermediate layer by concatenatinga projection of the original features with a projection of previous
hidden layer activations, and then applying self-attention.
First, we project both the input and intermediate layer features(Z0 ∈ RS×d0 , Zk ∈ RS×dk ), apply layer normalization and concatenate with position encoding:0

Z0 = cat([LayerNorm(Z0 W1 ), E], dim = 1)0

Zk = cat([LayerNorm(Zk W2 ), E], dim = 1)
where d0 is the input feature dimension, dk is the Transformer output dimension, dim = 1 denotes concatenation on the feature axis,
W1 ∈ Rd0 ×dc , W2 ∈ Rdk ×dc and E ∈ RS×de is a sinusoidal position encoding [6].
After we project both information to the same dimension, we
merge them by using time-axis concatenation:
0

3. ITERATED FEATURE PRESENTATIONIn this section, we present our proposal for allowing the network to(re)-consider the input features in the light of intermediate processing. We do this by again deploying a self-attention mechanism to
combine the information present in the original features with the information available in the activations of an intermediate layer. As
described earlier, we calculate the output posteriors and auxiliary0

O = cat([Z0 , Zk ], dim = 0) ∈ R2S×(dc +de )Then, we extract relevant features with extra Transformer layer andfollowed by linear projection and ReLU:(
00Transformer(Q = Z0 , K = O, V = O), split A
Zk+1 =
0
Transformer(Q = Zk , K = O, V = O), split B
0

Zk+1 = LayerNorm(ReLU(Zk+1 W3 ))Fig. 3. Merging input features and intermediate layer activations
with time axis concatenation for the Key and Value. Transformer
layer finds relevant features based on the Query. Split A uses projected input features as the Query and Split B used projected intermediate layer activations as the Query.

0

where W3 ∈ Rdk+1 ×dk+1 is a linear projection. All biases in theformula above are omitted for simplicity.
Note that in doing time-axis concatenation, our Key and Value
sequences are twice as long as the original input. In the standard
self-attention where the Query is the same as the Key and Value, theoutput preserves the sequence length. Therefore, in order to maintain the necessary sequence length S, we select either the first half
(split A) or the second half (split B) to represent the combined information. The difference between these two is that the use of split
A uses the projected input features as the Query set, while split B
uses the projected higher level activations as the Query. In initial
experiments, we found that the use of high-level features (split B) as
queries is preferable. We illustrate this operation on Figure 3.
Another way of combining information from the features withan intermediate layer is to concatenate the two along with the feature
rather than the time axis. However, in initial experiments, we foundthat time axis concatenation produces better results, and focus onthat in the experimental results.
3.2. Iterated Loss
We have found it beneficial to apply the loss function at several intermediate layers of the network. Suppose there are M total layers,
and define a subset of these layers at which to apply the loss function: K = {k1 , k2 , ..., kL } ⊆ {1, .., M − 1}. The total objectivefunction is then defined as
L = Loss(PM , Y ) + λLX

Loss(Pkl , Y )

(4)

l=1

Pkl = Softmax(MLPl (Zkl ))(5)where Zkl is the kl -th Transformer layer activations, Y is theground-truth transcription for CTC and context dependent states forhybrid ASR, and Loss(P, Y ) can be defined as CTC objective [11]or CE for hybrid ASR. The coefficient λ scales the auxiliary loss andwe set λ = 0.3 based on our preliminary experiments. We illustrate
the auxiliary prediction and loss in Figure 2.
4. EXPERIMENTAL RESULTS
4.1. Dataset
We evaluate our proposed module on both the Librispeech [12]dataset and a large-scale English video dataset. In the Librispeechtraining set, there are three splits, containing 100 and 360 hours setsof clean speech and 500 hours of other speech. We combined everything, resulting in 960 hours of training data. For the development
set, there are also two splits: dev-clean and dev-other. For
the test set, there is an analogous split.
The video dataset is a collection of public and anonymized English videos. It consists of a 1000 hour training set, a 9 hour dev
set, and a 46.1 hour test set. The test set comprises an 8.5 hourcurated set of carefully selected very clean videos, a 19 hourclean set and a 18.6 hour noisy set [13]. For the hybrid ASR
experiments on video dataset, alignments were generated with a production system trained with 14k hours.
All speech features are extracted by using log Mel-filterbankswith 80 dimensions, a 25 ms window size and a 10 ms time step
between two windows. Then we apply mean and variance normalization.
4.2. Target Units
For CTC training, we use word-pieces as our target. During training, the reference is tokenized to 5000 sub-word units using sentencepiece1 with a uni-gram language model [14]. Neural networks arethus used to produce a posterior distribution for 5001 symbols (5000
sub-word units plus blank symbol) every frame. For decoding, each
sub-word is modeled by a HMM with two states where the last states
share the same blank symbol probability; the best sub-word segmentation of each word is used to form a lexicon; these HMMs, lexicon
are then combined with the standard n-gram via FST [15] to form astatic decoding graph. Kaldi decoder[16] is used to produce the best
hypothesis.
We further present results with hybrid ASR systems. In this, we
use the same HMM topology, GMM bootstrapping and decision treebuilding procedure as [13]. Specifically, we use context-dependent(CD) graphemes as modeling units. On top of alignments from a
GMM model, we build a decision tree to cluster CD graphemes. This
results in 7248 context dependent units for Librispeech, and 6560
units for the video dataset. Training then proceeds with the CE loss
function. We also apply SpecAugment [17] online during training,
using the LD policy without time warping. For decoding, a standard
Kaldi’s WFST decoder [16] is used.
4.3. Deep Transformer Acoustic Model
All neural networks are implemented with the in-house extensionof the fairseq [18] toolkit. Our speech features are produced by processing the log Mel-spectrogram with two VGG [19] layers that havethe following configurations: (1) two 2-D convolutions with 32 output filters, kernel=3, stride=1, ReLU activation, and max-poolingkernel=2, (2) two 2-D convolutions with 64 output filters, kernel=3,
stride=1 and max-pooling kernel=2 for CTC or max-pooling kernel=1 for hybrid. After the VGG layers, the total number of frames
are subsampled by (i) 4x for CTC, or (ii) 2x for hybrid, thus enabling
us to reduce the run-time and memory usage significantly. After
VGG processing, we use 24 Transformer layers with dk = 512 head
dimensions (8 heads, each head has 64 dimensions), 2048 feedforward hidden dimensions (total parameters 80 millions), and dropout
0.15. For the proposed models, we utilized an auxiliary MLP with
two linear layers with 256 hidden units, LeakyReLU activation andsoftmax (see Sec. 3). We set our position encoding dimensions
de = 256 and pre-concatenation projection dc = 768 for the feature
re-presentation layer. The loss function is either CTC loss or hybrid
CE loss.
1 https://github.com/google/sentencepieceModel(CTC): Baseline+ Iter. Loss

+ Feat. Cat.

Config
VGG+24 Trf.
12-24
8-16-24
6-12-18-24
12-24
8-16-24
6-12-18-24

dev
clean othertest
clean other

Model

4.7
4.1
4.2
4.1
3.9
3.7
3.6

5.0
4.5
4.6
4.4
4.2
4.1
4.0

12.7
11.8
11.9
11.7
10.9
10.3
10.4

13.1
12.2
12.3
12.0
11.1
10.7
10.8

Table 1. Librispeech CTC experimental results without any data
augmentation technique and decoded with FST based on 4-gram
LM. (Notes: Trf is Transfromers, “+ Iter. Loss 12-24” means adding
iterative losses in the 12-th and 24-th layer, “+ Feat. Cat. 12-24”
means adding feature concatenation in the 12-th layer.)
Model

LM

test-clean

test-other(CTC) Zeghidour et al.[20](S2S): Mohamed et al. [8](S2S): Hannun et al. [21](S2S): Park et al. [17]+ SpecAugment(Hybrid): Lüscher et al. [22]+ Trf rescoring.

GCNN
4-gr
RNN

3.3
4.7
4.2
3.2
2.5
3.8
2.3

10.5
12.9
11.8
9.8
5.8
8.8
5.0
9.4
8.4
7.6

4-gr
Trf

(CTC): Baseline (24 Trf)
+ Iter. Loss (8-16-24)+ Feat. Cat. (8-16-24)

4-gr

4.0
3.5
3.3

(CTC): Baseline (36 Trf)
+ Iter. Loss (12-24-36)+ Feat. Cat. (12-24-36)

4-gr

4.0
3.4
3.2

9.4
8.1
7.2

(Hybrid): Baseline (24 Trf)
+ Iter. Loss (8-16-24)+ Feat. Cat. (8-16-24)

4-gr

3.2
3.1
2.9

7.7
7.3
6.7

Table 2. Librispeech experimental results. The baseline consists of
VGG + 24 layers of Transformers trained with SpecAugment [17].
Trf is transformer. 4-gr LM is the official 4-gram word LM. S2S
denotes sequence-to-sequence architecture.

4.4. Results
Table 1 presents CTC based results for the Librispeech dataset,without data augmentation. Our baseline is a 24 layer Transformer network trained with CTC. For the proposed method, we
varied the number and placement of iterated loss and the feature
re-presentation. The next three results show the effect of using CTC
multiple times. We see 12 and 8% relative improvements for testclean and test-other. Adding feature re-presentation gives a further
boost, with net 20 and 18% relative improvements over the baseline.
Table 2 shows results for Librispeech with SpecAugment. We
test both CTC and CE/hybrid systems. There are consistent gains
first from iterated loss, and then from multiple feature presentation.
We also run additional CTC experiments with 36 layers Transformer(total parameters 120 millions). The baseline with 36 layers has thesame performance with 24 layers, but by adding the proposed methods, the 36 layer performance improved to give the best results. Thisshows that our proposed methods can improve even very deep models.
As shown in Table 3, the proposed methods also provide large

curated

Video
clean

noisy

(CTC): Baseline (24 Trf)
+ Iter. Loss (8-16-24)+ Feat. Cat. (8-16-24)

14.0
13.2
12.4

17.4
16.7
16.2

23.6
22.9
22.3

(CTC): Baseline (36 Trf)
+ Iter. Loss (12-24-36)+ Feat. Cat. (12-24-36)

14.2
12.9
12.3

17.5
16.6
16.1

23.8
22.8
22.3

(Hybrid): Baseline (24 Trf)
+ Iter. Loss (8-16-24)+ Feat. Cat. (8-16-24)

12.8
12.1
11.5

16.1
15.7
15.4

22.1
21.8
21.4

Table 3. Video English dataset experimental results.

performance improvements on the curated video set, up to 13% with
CTC, and up to 9% with the hybrid model. We also observe moderate gains of between 3.2 and 8% relative on the clean and noisy
video sets.
5. RELATED WORKIn recent years, Transformer models have become an active research
topic in speech processing. The key features of Transformer networks is self-attention, which produces comparable or better performance to LSTMs when used for encoder-decoder based ASR [23],as well as when trained with CTC [9]. Speech-Transformers [24]also produce comparable performance to the LSTM-based attention
model, but with higher training speed in a single GPU. Abdelrahman
et al.[8] integrates a convolution layer to capture audio context andreduces WER in Librispeech.
The use of an objective function in intermediate layers has beenfound useful in several previous works such as image classification [25] and language modeling [26]. In [27], the authors did pretraining with an RNN-T based model by using a hierarchical CTC
criterion with different target units. In this paper, we don’t need additional types of target unit, instead we just use same tokenizationand targets for both intermediate and final losses.
The application of the objective function to intermediate layers
is also similar in spirit to the use of KL-divergence in [28], which
estimates output posteriors at an intermediate layer and regularizes
them towards the distributions at the final layer. In contrast to thisapproach, the direct application of the objective function does not
require the network to have a good output distribution before the
new gradient contribution is meaningful.
6. CONCLUSION
In this paper, we have proposed a method for re-processing the input features in light of the information available at an intermediate
network layer. We do this in the context of deep transformer networks, via a self-attention mechanism on both features and hidden
states representation. To encourage meaningful partial results, we
calculate the objective function at intermediate layers of the network
as well as the output layer. This improves performance in and of
itself, and when combined with feature re-presentation we observe
consistent relative improvements of 10 - 20% for Librispeech and3.2 - 13% for videos.

7. REFERENCES[1] Yao K. Su H. Li G. Yu, D. and F. Seide, “KL-divergence regularized deep neural network adaptation for improved large vocabulary speech recognition,” Proceedings of the 2013 IEEE
International Conference on Acoustics, Speech, and Signal
Processing (ICASSP), 2013.
[2] Marc Delcroix, Keisuke Kinoshita, Takaaki Hori, and Tomohiro Nakatani, “Context adaptive deep neural networks for fast
acoustic model adaptation,” in 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).
IEEE, 2015, pp. 4535–4539.
[3] Kaisheng Yao, Dong Yu, Frank Seide, Hang Su, Li Deng, andYifan Gong, “Adaptation of context-dependent deep neuralnetworks for automatic speech recognition,” in 2012 IEEE
Spoken Language Technology Workshop (SLT). IEEE, 2012,
pp. 366–369.
[4] Jinyu Li, Jui-Ting Huang, and Yifan Gong, “Factorized
adaptation for deep neural network,” in 2014 IEEE International Conference on Acoustics, Speech and Signal Processing(ICASSP). IEEE, 2014, pp. 5537–5541.
[5] George Saon, Hagen Soltau, David Nahamoo, and Michael
Picheny, “Speaker adaptation of neural network acoustic models using i-vectors,” in 2013 IEEE Workshop on Automatic
Speech Recognition and Understanding. IEEE, 2013, pp. 55–
59.
[6] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia
Polosukhin, “Attention is all you need,” in Advances in neural
information processing systems, 2017, pp. 5998–6008.
[7] Zihang Dai, Zhilin Yang, Yiming Yang, William W Cohen, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov,“Transformer-XL: Attentive language models beyond a fixedlength context,” arXiv preprint arXiv:1901.02860, 2019.
[8] Abdelrahman Mohamed, Dmytro Okhonko, and Luke Zettlemoyer, “Transformers with convolutional context for ASR,”
arXiv preprint arXiv:1904.11660, 2019.
[9] S. Karita, N. Chen, T. Hayashi, et al., “A Comparative Study on
Transformer vs RNN in Speech Applications,” arXiv preprint
arXiv:1909.06317, 2019.
[10] Sepp Hochreiter and Jürgen Schmidhuber, “Long short-term
memory,” Neural computation, vol. 9, no. 8, pp. 1735–1780,
1997.
[11] Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen
Schmidhuber, “Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks,” in Proceedings of the 23rd international conference
on Machine learning. ACM, 2006, pp. 369–376.
[12] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev
Khudanpur, “Librispeech: an ASR corpus based on publicdomain audio books,” in 2015 IEEE International Conferenceon Acoustics, Speech and Signal Processing (ICASSP). IEEE,
2015, pp. 5206–5210.
[13] Duc Le, Xiaohui Zhang, Weiyi Zheng, Christian Fügen,
Geoffrey Zweig, and Michael L Seltzer, “From senones
to chenones: Tied context-dependent graphemes for hybrid
speech recognition,” Proceedings of ASRU, 2019.

[14] Taku Kudo, “Subword regularization: Improving neural network translation models with multiple subword candidates,”arXiv preprint arXiv:1804.10959, 2018.
[15] M. Mohri, F. Pereira, and M. Riley, “Weighted finite-statetransducers in speech recognition,” Computer Speech & Language, vol. 16, no. 1, pp. 69–88, 2002.
[16] Daniel Povey, Arnab Ghoshal, Gilles Boulianne, Lukáš Burget, Ondřej Glembek, Nagendra Goel, Mirko Hannemann, Petr
Motlı́ček, Yanmin Qian, Petr Schwarz, et al., “The Kaldi
speech recognition toolkit,” Proc. ASRU, 2011.
[17] Daniel S. Park, William Chan, Yu Zhang, Chung-Cheng Chiu,
Barret Zoph, Ekin D. Cubuk, and Quoc V. Le, “Specaugment:
A simple data augmentation method for automatic speech
recognition,” Interspeech 2019, Sep 2019.
[18] O. Myle, E. Sergey, B. Alexei, F. Angela, et al., “fairseq: A
Fast, Extensible Toolkit for Sequence Modeling,” in Proceedings of NAACL-HLT 2019: Demonstrations, 2019.
[19] Karen Simonyan and Andrew Zisserman, “Very deep convolutional networks for large-scale image recognition,” in 3rd
International Conference on Learning Representations, ICLR
2015, San Diego, CA, USA, May 7-9, 2015, Conference Track
Proceedings, 2015.
[20] Neil Zeghidour, Qiantong Xu, Vitaliy Liptchinsky, Nicolas
Usunier, Gabriel Synnaeve, and Ronan Collobert, “Fully convolutional speech recognition,” CoRR, vol. abs/1812.06864,
2018.
[21] Awni Hannun, Ann Lee, Qiantong Xu, and Ronan Collobert,“Sequence-to-sequence speech recognition with time-depth
separable convolutions,” Interspeech 2019, Sep 2019.
[22] Christoph Lüscher, Eugen Beck, Kazuki Irie, Markus Kitza,
Wilfried Michel, Albert Zeyer, Ralf Schlüter, and Hermann
Ney, “RWTH ASR Systems for LibriSpeech: Hybrid vs Attention,” in Proc. Interspeech 2019, 2019, pp. 231–235.
[23] Matthias Sperber, Jan Niehues, Graham Neubig, SebastianStüker, and Alex Waibel, “Self-attentional acoustic models,”arXiv preprint arXiv:1803.09519, 2018.
[24] Linhao Dong, Shuang Xu, and Bo Xu, “Speech-transformer: ano-recurrence sequence-to-sequence model for speech recognition,” in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2018, pp.
5884–5888.
[25] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet,
Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich, “Going deeper with convolutions,” in Proceedings of the IEEE conference on computer
vision and pattern recognition, 2015, pp. 1–9.
[26] Rami Al-Rfou, Dokook Choe, Noah Constant, Mandy Guo,and Llion Jones, “Character-level language modeling withdeeper self-attention,” in Proceedings of the AAAI Conferenceon Artificial Intelligence, 2019, vol. 33, pp. 3159–3166.
[27] Kanishka Rao, Hasim Sak, and Rohit Prabhavalkar, “Exploring architectures, data and units for streaming end-to-end
speech recognition with rnn-transducer,” 2017 IEEE Automatic
Speech Recognition and Understanding Workshop (ASRU),
2017.
[28] Liang Lu, Eric Sun, and Yifan Gong, “Self-teaching networks,”
arXiv preprint arXiv:1909.04157, 2019.