DEJA-VU: DOUBLE FEATURE
PRESENTATION AND ITERATED LOSS
IN DEEP TRANSFORMER NETWORKS
Andros Tjandra1*, Chunxi Liu2, Frank Zhang2, Xiaohui Zhang2, Yongqiang Wang2,Gabriel Synnaeve2, Satoshi Nakamura1, Goeffrey Zweig2
1) NAIST, Japan

2) Facebook AI, USA* This work was done while Andros was a research intern at Facebook

Motivation• Make feature processing adaptive to what is being said.

• Different feature processing, depending on what words need to be differentiated inlight of a specific utterance.

• To achieve this, we allow a Transformer Network to (re)-attend to the audio
features, using intermediate layer activations as the Query.
• Imposing the objective function on the intermediate layer ensures that it
has meaningful information – and trains much faster.
• Net – using these two methods lowers error rate 10-20% for Librispeech
and Video ASR datasets.

Review: Self-attention in Transformers
Dot Product Attention

Multihead Self Attention

Image ref: Attention is all you need (Vaswani et al., NIPS 2017)

Transformer module

Review: VGG + Transformer Acoustic Modelℒ𝑃, 𝑌ℒ𝑃, 𝑌Softmax

Loss function is either CTCor CE (for hybrid DNN-HMM)

Transformer…

Stack of Transformer layers

Transformer
VGG

Blocks of 3x3 convolution + stride (for sub-sampling Mel-spectral)Problems?
• Stacking more and more layers has empirically give better result.
• Computer vision: AlexNet (<10 layers) -> VGGNet (20 layers) -> ResNet (>100layers).

• However, training such deep models are difficult.
• With improvements in this paper, we can reliably train up to 36 layer
networks.

Idea #1: Iterated Loss
• In the deep neural network, the loss arealways the furthest node from theinput.
• Early nodes (layers) might received lessfeedback (due to vanishing gradients).
• We add auxiliary loss in theintermediate node.

𝑍ℒ 𝑃 ,𝑌

Transformer…
Transformer
𝑍Transformer

𝜆 ∗ ℒ 𝑃 ,𝑌
1…

Auxiliary layer
to project 𝑍 toprediction 𝑃

Transformer(removed aftertraining finished)𝑍Effect of Iterated Loss
• Comparison:• Baseline 1 CTC (24)
• 2 CTC (12–24)• 3 CTC (8-16-24)• 4 CTC (6-12-18-24)• CoeffEffect of••= 0.3 vs 1.0= 0.3
consistentbetter
compared to1.0 on 2 CTCand 3 CTC

Idea #2: Feature Re-presentation2…

• After the iterated loss, we want todynamically integrate the input
features.
• Why?
• The layer after iterated loss might havepartial hypothesis.
• We could find correlated features basedon the partial hypothesis.

• There are several ways we have
explored (next slide -> )Transformer
𝑍𝑍
Linear Proj+ LayerNorm

Transformer
𝑍
Transformer…
Transformer
𝑍

Transformer
combines input
feature andhidden state(Cont.) Feature Concatenation𝑍• (Top) Feature axis. concatenation

Transformer

Linear proj. + LNQKV𝑍𝑍• (Btm) Time axis. Concatenation✓ best performance• Split A : input as Query• Split B : hidden state as Query
Time Cat + Post Projection

Split A

Split B𝑍

𝑍

Transformer

Transformer

QKVQKV

𝑍

𝑍

Z

𝑍

𝑍

Z

Final architecture𝑍ℒ 𝑃 ,𝑌

Transformer
2…
Transformer

Transformer
combines input
feature andhidden state𝑍

𝑍
Linear Proj+ LayerNorm

Transformer
𝑍
Transformer

𝜆 ∗ ℒ 𝑃 ,𝑌
1…

Auxiliary layer
to project 𝑍 toprediction 𝑃Transformer(removed after
training finished)𝑍Result: Librispeech (CTC w/o data augmentation)
ModelConfig

dev

test

clean other clean other
CTC Baseline

VGG+24 Trf.

4.7

12.7

5.0

13.1

+ Iter. Loss

12-24

4.1

11.8

4.5

12.2

8-16-24

4.2

11.9

4.6

12.3

6-12-18-24

4.1

11.7

4.4

12.0

12-24

3.9

10.9

4.2

11.1

8-16-24

3.7

10.3

4.1

10.7

6-12-18-24

3.6

10.4

4.0

10.8

+ Feat. Cat.

12% test-clean & 8% test-otherrelative improvement
20% test-clean & 18% test-otherrelative improvement

Librispeech with data augmentation

ModelConfig

LM

CTC (Baseline) VGG+24 Trf.
+ Iter. Loss

8-16-24+ Feat. Cat

8-16-244-gram

CTC (Baseline) VGG+36 Trf.
+ Iter. Loss

12-24-36+ Feat. Cat

12-24-36

4-gramtest-clean test-other
4.0

9.4

3.5

8.4

3.3

7.6

4.0

9.4

3.4

8.1

3.2

7.2

Without iter-loss & feat-cat,increasing Transformer layers
doesn’timproveperformance
Withiter-loss& feat-cat,
we still get improvementwith deeper Transformer

Librispeech with hybrid DNN-HMM

ModelConfig

LMHybrid (Baseline) VGG+24 Trf.
+ Iter. Loss

8-16-24+ Feat. Cat

8-16-244-gramtest-clean test-other
3.2

7.7

3.1

7.3

2.9

6.7

9% test-clean &12% test-otherimprovement

Video dataset
ModelConfig

video
curated

clean

other

CTC (Baseline)

VGG+24 Trf.

14.0

17.4

23.6+ Iter. Loss

8-16-24

13.2

16.7

22.9+ Feat. Cat

8-16-24

12.4

16.2

22.3

CTC (Baseline)

VGG+36 Trf.

14.2

17.5

23.8+ Iter. Loss

12-24-36

12.9

16.6

22.8+ Feat. Cat

12-24-36

12.3

16.1

22.3

Hybrid (Baseline)

VGG+24 Trf

12.8

16.1

22.1

+ Iter. Loss

8-16-24

12.1

15.721.8+ Feat. Cat

8-16-24

11.6

15.4

21.4

13% curated
8% clean
6% other
improvement
9% curated
4% clean
3% other
improvement

Conclusion• We have proposed a method for re-processing the input features inlight of the information available at an intermediate network layer.
• To integrate the features from different layers, we proposed selfattention across layers by concatenating two sequences in time-axis.
• Adding iterated loss in the middle of deep transformers helps theperformance (tested on hybrid ASR as well).
• Librispeech: 10-20% relative improvements• Video: 3.2-13% relative improvements

End of presentation

 Thank you for your attention 