ASR Posterior-based Loss for Multi-task
End-to-end Speech Translation
Yuka Ko1, Katsuhito Sudoh1,2, Sakriani Sakti1,2, Satoshi Nakamura1,2
1Nara Institute of Science and Technology (NAIST), Japan

2RIKEN Center for Advanced Intelligence Project (AIP), Japan

Yuka Ko @ AHC Lab, NAIST | INTERSPEECH 2021 | August 30th – September 3rd, 2021

Motivation

2src speech⚫End-to-End Speech Translation (ST)
➢Data size is small : small pairs of src speech - tgt text
→ Multi-task is main approach➢Cross entropy (CE) loss only using correct answer→ Multi-task doesn’t consider ASR hypotheses⚫Proposed➢Multi-task ST training with ASR posterior distribution

tgt text⚫Purpose➢Robust multi-task End-to-End STfor ASR hypotheses

Yuka Ko @ AHC Lab, NAIST | INTERSPEECH 2021 | August 30th – September 3rd, 2021

End-to-End ST

Related Works

Related1 : Robust cascade ST for ASR hypotheses● [Osamura+, 2018] (cascade ST)➢ASR output : 1-best → ASR posterior distribution➢Proposed : Robust NMT for ASR hypotheses

1ASR Encoder

ASR Encoder

ASR Decoder

ASR Decoder

000000.4

0.2

0.1

0.0

0.050.0

ASR 1-best (One hot)

ASR posterior distribution

NMT Encoder

NMT Encoder

NMT Decoder

NMT Decoder
backprop

Previous[Osamura+, 2018]

Yuka Ko @ AHC Lab, NAIST | INTERSPEECH 2021 | August 30th – September 3rd, 2021

4

Related 2 : Single-task and Multi-task ST➢ Single-task End-to-End ST➢ Multi-task End-to-End ST [Weiss+ 2017]

ST-task

ASR-task

ST-taskTargetText

SourceText

TargetText

TargetDecoder

SourceDecoder

TargetDecoder[ℎ1 , ℎ2 , … , ℎ 𝑇 ][ℎ1 , ℎ2 , … , ℎ 𝑇 ]Encoder Time (s)0

Encoder Time (s)1.09

0-0.8104-0.8104

000.7942

0.7942[Weiss+ 2017] Sequence-toSequence Models Can DirectlyTranslate Foreign Speech

English__Great_Britain__Female1

51.09

English__Great_Britain__Female1

Yuka Ko @ AHC Lab, NAIST | INTERSPEECH 2021 | August 30th – September 3rd, 2021

Related3 : Robust Multi-task ST for semantic similarity

6⚫ Using cosine similarity in ASR-task loss calculation [Chuang+, 2020]
ASR-task
0100Cross
Entropy0.020.38

𝑒1 𝑒2 𝑒3 … 𝑒Ƹ 𝑉

0.01

0.40

SourceDecoder

Cosine similarity

TargetText[…, 𝑒𝑚 , …]TargetDecoder

SourceDecoder

Encoder Time (s)[…, 𝑦ෞ𝑚 , …][…, 𝑃𝐶𝑆 (𝑦ෞ𝑚 ), …]
Cross
Entropy

TargetDecoder[ℎ1 , ℎ2 , … , ℎ 𝑇 ][ℎ1 , ℎ2 , … , ℎ 𝑇 ]
0𝐸෠

01.09

-0.8104-0.8104

000.7942

0.7942

Encoder Time (s)English__Great_Britain__Female1
Yuka KoEnglish__Great_Britain__Female1@ AHC Lab, NAIST | INTERSPEECH 2021 | August 30th – September
3rd, 2021

1.09

Proposed Method

Previous and Proposed➢ Previous : 𝐿hard➢ Reference : One-hot△ Same score in mistaken word
△ Not consider acoustic similarity
01

𝐿ASR= 𝐿 hard
0.020.38➢ Proposed : 𝐿soft➢ Reference : ASR posterior distribution〇 Consider acoustic similar word &unsimilar word
0.01

000.010.40

SourceDecoder0.5

0.02

Pre-train
ASR

0.40

0.38

0.01

TargetText
TargetDecoder[ℎ1 , ℎ2 , … , ℎ 𝑇 ]Encoder Time (s)Encoder Time (s)01.09-0.8104-0.8104

000.7942

0.01

SourceDecoder[ℎ1 , ℎ2 , … , ℎ 𝑇 ]

00.4

𝐿ASR
= 𝐿softTarget
TextTarget
Decoder

80.7942

Yuka Ko @ AHC Lab, NAIST | INTERSPEECH 2021 | August 30th – September 3rd, 2021

1.09

Previous and Proposed➢ Previous : 𝐿hard➢ Reference : One-hot△ Same score in mistaken word
△ Not consider acoustic similarity
01

𝐿ASR= 𝐿 hard
0.02

0.38

〇 Consider acoustic similar word &unsimilar word
0.01

0.40

0.01

I catch a ball[ℎ1 , ℎ2case, … , ℎ 𝑇1]][PredictI cat a ball
Encoder Time (s)[Predictcase 2]
0-0.8104
I lost a ball
00.50.40.01

𝐿ASR= 𝐿soft

TargetText

SourceTargetDecoder
Decoder[One-hot reference]

0.7942➢ Proposed : 𝐿soft➢ Reference : ASR posterior distribution

0090.02

Pre-train
ASR

0.40

0.38

0.01

SourceDecoder

TargetText
TargetDecoder[ℎ1 , ℎ2 , … , ℎ 𝑇 ]Encoder Time (s)01.09-0.810400.7942

Yuka Ko @ AHC Lab, NAIST | INTERSPEECH 2021 | August 30th – September 3rd, 2021

1.09

Experiments

Experiment : Dataset
datasrc-tgt

speech featureFisher Spanish

Es-En

fbank + pitch(80+3=83dim)

BPE model

dict size

SentencePiece

1000
Es-En Joint

dataset
BPE modelTrainfisher_train
SentencePiece
1000
Dev
Test

fisher_dev

data size
dict size
140K

1000
Sp-En Joint(Before 3 timesaugmentation)

3.9K

fisher_dev2

3.9K

fisher_test

3.6K

Yuka Ko @ AHC Lab, NAIST | INTERSPEECH 2021 | August 30th – September 3rd, 2021

11

Experiment : Model Parameter⚫ Implement : ESPnet [Watanabe+, 2018], Transformer⚫ Pre-trained ASR model (for soft labels of ASR posterior distributions)➢ 30 epochs➢ Dev best model (in accuracy)⚫ ST model➢ 30 epochs➢ Checkpoint averaging : 5 (in BLEU)➢ 𝐿ST : CE + Label smoothing➢ ASR-task loss
➢ Baseline : 𝐿ASR : {CE, CE + Label smoothing}➢ Proposed : ASR Posterior-based Loss
● Dev best model in BLEU
𝜆ASR = 0.3, 0.4, 0.5 , 𝜆soft = 0.1, 0.3, 0.5, 0.7, 0.9, 1.0
𝐿ASR = 𝜆soft 𝐿soft + 1 − 𝜆soft 𝐿hard➢ 𝐿 = 𝜆ASR 𝐿ASR + 1 − 𝜆ASR 𝐿ST
Yuka Ko @ AHC Lab, NAIST | INTERSPEECH 2021 | August 30th – September 3rd, 2021

12

Result : Pre-trained ASR WER⚫ WER in dev best accuracy in epoch 30 (beam size 10)⚫ Our model without language model (LM) and CTC➢ When decoding soft labels, we set beam size 1
Model
OurASR

LM

CTC

dev
30.2

dev2
29.1

test
27.2

ESPnet✓

✓

24.2

23.6

22.8

Yuka Ko @ AHC Lab, NAIST | INTERSPEECH 2021 | August 30th – September 3rd, 2021

13

BLEU of test data (dev best on each parameters) 14

ModelTask

ASR-task loss

Single-task STMulti-task STBLEU
𝜆ASR

𝜆soft

dev

dev2

test---

41.10

41.61

40.66

CE (Baseline)

0.4

-

44.50

46.20

44.88

CE + Label smoothing (Baseline)

0.5

-

45.29

46.34

45.16

ASR Posterior-based loss (Proposed)

0.4

0.5

45.54

46.46

45.64

Yuka Ko @ AHC Lab, NAIST | INTERSPEECH 2021 | August 30th – September 3rd, 2021

BLEU of test data (dev best on difference 𝝀𝐀𝐒𝐑)
ModelTask

ASR-task loss

Single-task
ST-BLEU

𝜆ASR

𝜆soft

dev

dev2

test--

41.10

41.61

40.66

-

44.08

45.07

44.69-

44.23

45.25

44.21

0.9

44.99

45.73

44.66

-

44.50

46.20

44.88

-

44.61

45.30

45.01

0.5

45.54

46.46

45.64

-43.64

45.28

43.83

-

45.29

46.34

45.16

0.5

45.46

46.37

46.04

CE
CE + Label smoothing

ASR Posterior-based loss (Proposed)
Multi-task
ST

0.3

CE

CE + Label smoothing
ASR Posterior-based loss (Proposed)

0.4

CE
CE + Label smoothing

ASR Posterior-based loss (Proposed)

0.5

Yuka Ko @ AHC Lab, NAIST | INTERSPEECH 2021 | August 30th – September 3rd, 2021

15

𝐁𝐋𝐄𝐔 𝐢𝐧 𝐅𝐢𝐬𝐡𝐞𝐫 𝐭𝐞𝐬𝐭 𝐨𝐧 𝐝𝐢𝐟𝐟𝐞𝐫𝐞𝐧𝐭 𝝀𝐀𝐒𝐑

Yuka Ko @ AHC Lab, NAIST | INTERSPEECH 2021 | August 30th – September 3rd, 2021

16

Dev loss in each 𝝀soft (𝝀ASR = 𝟎. 𝟒)⚫ Total loss is higher than CE➢ However (b) 0.3, (c) 0.5 improved⚫ Maybe proposed soft loss work as regularization➢ It is good in 0.5, in this method

𝜆soft = 0.0

𝜆soft = 0.3

Yuka Ko @ AHC Lab, NAIST | INTERSPEECH 2021 | August 30th – September 3rd, 2021

17

Examples in Fisher test 𝝀ASR = 𝟎. 𝟒 𝝀soft = 𝟎. 𝟓
ExampleGround Truth (Es)para relajacio´nGround Truth (En)for relaxation

CE + Label smoothing (En)for relationships (↔relaciones (es) ?)ASR Posterior-based loss (En) for relaxing
Ground Truth (Es)quie´n no su sobrina

Ground Truth (En)

who no your niece

CE + Label smoothing (En)

who his nephews (↔sobrino (es) ?)ASR Posterior-based loss (En) who are your nieces
Yuka Ko @ AHC Lab, NAIST | INTERSPEECH 2021 | August 30th – September 3rd, 2021

18

Conclusion⚫ Purpose : Robust Multi-task End-to-End ST for ASR hypotheses⚫ Proposed : ASR posterior-based loss
➢ Using ASR posterior distribution in End-to-End ST training⚫ Results : BLEU improvement in proposed➢ Baseline : CE, CE + Label smoothing➢ Mixing hard loss & soft loss was better⚫ Future work
➢ More analysis on ASR-task output➢ Using pre-trained ASR models of different performances➢ Using pronunciation information (phone [Salesky+, 2020]) in loss calculation➢ Fine-tuning (first train with 𝐿hard , after training with 𝐿soft )⚫ Our code is available at:➢ https://github.com/ahclab/st-asrpbl

Yuka Ko @ AHC Lab, NAIST | INTERSPEECH 2021 | August 30th – September 3rd, 2021

19

Reference⚫[Osamura+, 2018] K. Osamura, T. Kano, S. Sakti, K. Sudoh, and S. Nakamura, “Using Spoken Word

20

Posterior Features in Neural Machine Translation,” Proceedings of the 15th International Workshop onSpoken Language Translation, 181-188, Oct. 2018, vol. 21, p. 22, 2018.
⚫[Weiss+, 2017] Ron J Weiss, Jan Chorowski, Navdeep Jaitly, Yonghui Wu, and Zhifeng Chen. 2017.
Sequence-tosequence models can directly translate foreign speech. Proc. Interspeech 2017, pages 2625–2629.
⚫[Chuang+, 2020] S. Chuang, T. Sung, A. H. Liu, and H. Lee, “Worse WER, but Better BLEU? Leveraging
Word Embedding as Intermediate in Multitask End-to-End Speech Translation,” in Proceedings of the58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020,D. Jurafsky, J. Chai, N. Schluter, and J. R. Tetreault, Eds. Association for Computational Linguistics, 2020,
pp. 5998–6003.
⚫[Watanabe+, 2018] S. Watanabe, T. Hori, S. Karita, T. Hayashi, J. Nishitoba, Y. Unno, N.-E. Y. Soplin, J.
Heymann, M. Wiesner, N. Chen et al., “ESPnet: End-to-End Speech Processing Toolkit,” Proc. Interspeech
2018, pp. 2207–2211, 2018.
⚫[Salesky+, 2020] E. Salesky and A. W. Black, “Phone Features Improve Speech Translation,” in Proceedingsof the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10,
2020, D. Jurafsky, J. Chai, N. Schluter, and J. R. Tetreault, Eds. Association for Computational Linguistics,
2020, pp. 2388– 2397.

Yuka Ko @ AHC Lab, NAIST | INTERSPEECH 2021 | August 30th – September 3rd, 2021