Improved Consistency Training for Semi-Supervised
Sequence-to-Sequence ASR via Speech Chain Reconstructionand Self-Transcribing

Heli Qi1, Sashi Novitasari1, Sakriani Sakti2, Satoshi Nakamura1
1. Nara Institute of Science and Technology, Japan
2. Japan Advanced Institute of Science and Technology, Japan

2022/11/1

2022©Heli Qi AHC-Lab, IS, NAIST

1

Table of Contents
Research Background• Semi-supervised ASR

Related Work
Proposed Method•••

Traditional Paradigm
Existing Problems to be solved
Our solutions

Experiment• Experiment Setups• Experiment Results

Conclusion & Future Work
2022/11/1

2022©Heli Qi AHC-Lab, IS, NAIST

2/15

Research Background—Semi-supervised ASRSupervised ASR training with speech-text paired data
Training
Speech-text Paired Data

ASRExpensive!!!!

Semi-supervised ASR training with both speech-text paired data and speech-only data

Speech-text Paired Data

Training

ASR

Speech-only Data

2022/11/1

2022©Heli Qi AHC-Lab, IS, NAIST

3/15

Related Work—Semi-supervised ASR Strategy
Self-training [Jacob et al. ICASSP2020]
Iterative Self-training [Qiantong et al. Interspeech2020]
Noisy Student Training [Daniel S. et al. Interspeech2020]
Speech-only Data

Consistency Regularization [Felix et al. Interspeech2020]……
Jacob et al. ICASSP2020, “Self-training for end-to-end speech recognition”
Qiantong et al. Interspeech2020, “Iterative Pseudo-Labeling for Speech Recognition”
Daniel S. et al. Interspeech2020, “Improved Noisy Student Training for Automatic Speech Recognition”
Felix et al. Interspeech2020, “Semi-Supervised Learning with Data Augmentation for End-to-End ASR”

2022/11/1

2011©Name AHC-Lab, IS, NAIST

4/15

Related Work—Consistency Regularization
FixMatch algorithm [K. Sohn et al. NIPS2020]• Designed for semi-supervised image classification (IC)• Make pseudo labels by the weakly-augmented image• Train the model by the strongly-augmented image and the acquired pseudo label.
K. Sohn et al. NIPS2020, “Fixmatch: Simplifying semi-supervised learning with consistency and confidence”
2022/11/1

2022©Heli Qi AHC-Lab, IS, NAIST

5/15

Handle the difference between IC and S2S ASR
During training, we could view teacherforcing ASR as the next token prediction task:• Multiple predictions for a single queryspeech• Also need prefix tokens as a part of input

ASR
Encoder

Hidden Representation

Pseudo label
at time step nToken prediction
at time step n

ASR
Decoder

ASR
Decoder…

ASR
Decoder

Prefix Tokens<SOS>

𝑇𝑜𝑘𝑒𝑛…

𝑇𝑜𝑘𝑒𝑛

Weakly-augmented Speech2022/11/1

2022©Heli Qi AHC-Lab, IS, NAIST

6/15

Semi-supervised ASR Training Paradigm
1. Base ASR Training on labeled data
Training

Base ASR

Speech-text Paired Data

2. Pseudo Transcript Generation

Base ASR
Speech-only Data

Pseudo Transcript

3. Semi-supervised Training on the enlarged dataset
Semi-supervised Training
Real Labeled Data

2022/11/1

Target ASR

Pseudo Labeled Data2022©Heli Qi AHC-Lab, IS, NAIST

7/15

Fixmatch-based Semi-supervised ASR Training• Pseudo transcript generated by the base ASR model acts as prefix tokens at each time step
• SpecAugment [Daniel S. et al. Interspeech2019] is adopted as both weak augmentation and strong augmentation
WeakAugmentation𝜶(·)

TargetASR
Pseudo Labels

Weakly Augmented Speech

Pseudo Transcript

𝐿𝑜𝑠𝑠＋𝐿𝑜𝑠𝑠

Prefix Tokens 𝒚

Unlabeled Speech𝒙𝒖StrongAugmentation
𝑨(·)

TargetASR
Strongly Augmented Speech

Predictions

Daniel S. et al. Interspeech2019, “SpecAugment: A Simple Data Augmentation Method for Automatic
Speech Recognition”
2022/11/1
2022©Heli Qi AHC-Lab, IS, NAIST
8/15

Existing Problems to Be Solved• Ideal weak augmentation for high quality pseudo labels
 Modify the acoustic style while preserving thelinguistic contents

ASR Training by Teacher-Forcing:

Pseudo Label

Token Prediction

ASR• Hard to decide the masking width Too narrow  Meaningless augmentation
 Too wide  Hurt the linguistic contents

Time-frequency
Masking

Strongly-augmented
SpeechNon-augmented
Prefix Tokens

As a part of input, the augmentation onprefix tokens also need to be considered.
2022/11/1

2022©Heli Qi AHC-Lab, IS, NAIST

9/15

Speech Chain Reconstruction
Text-to-Speech Synthesis (TTS)

TTS
Query Text

Synthetic Speech

Speaker Embedding

Speech Chain Reconstruction [A. Tjandra et al. ASRU2017]

Complete Linguistic Content

TTS

ASR
Unlabeled Speech

Reconstructed Speech

Pseudo Transcript

Speaker Embedding
A. Tjandra et al. ASRU2017, “Listening while speaking: Speech chain by deep learning”
2022/11/1

2022©Heli Qi AHC-Lab, IS, NAIST

10/15

Different Prefix Token Production Strategies
Static prefix tokens

produced by original speech𝒖Dynamic prefix tokens

produced by original speech

Static prefix tokens
𝒖speechproduced by weakly-augmented

Dynamic prefix tokens
𝒖speechproduced by weakly-augmented

Weakly Augmented Speech𝜶(𝒙𝒖 )

BaseASR
Static Prefix Tokens 𝒚

Augmented
input

ororTarget
ASR

Non-augmented
input

Pseudo Labels

TargetASR
Original Unlabeled Speech𝒙𝒖

2022/11/1

Dynamic Prefix Tokens 𝒚
Self-transcribing at each epoch2022©Heli Qi AHC-Lab, IS, NAIST

11/15𝒖Experiment Setups
Datasets:ASR model:

1. LJSpeech (Single Speaker):••Labeled data: 6300 utterances
Unlabeled data: 6300 utterances

2. LibriSpeech-100h (Multiple speakers):••

Labeled data: 8750 utterances (75 speakers)
Unlabeled data: 19968 utterances (176 speakers)

Mode Input & Output:

1. Acoustic Features:• 16,000 sampling rate
• 50ms frame length & 12.5ms frame shift• 80d log Mel-spectrogram
2. Tokenization:• Character-based models• 26 English letters (a~z) + 3 special tokens(apostrophes, space, and “sos/eos”)

1. Encoder:• Single-Speaker Setting: 3 Bi-LSTM layers (2 * 256 dim)
• Multi-Speaker Setting: 5 Bi-LSTM layers (2 * 256 dim)
2. Decoder:• 1 LSTM layer (512 dim)
• Additive Attention

TTS model:• Same structure as Tacotron2 [J. Shen et al. ICASSP2018]• X-vector [D. Snyder et al. ICASSP2018] for speakerembedding

J. Shen et al. ICASSP2018, “Natural tts synthesis by conditioning wavenet on mel spectrogram predictions”
D. Snyder et al. ICASSP2018, “X-vectors: Robust dnn embeddings for speaker recognition”
2022/11/12022©Heli Qi AHC-Lab, IS, NAIST

12/15

Experiment Results
Contrast Experiments:
1. Weak Augmentation 𝜶(·):

 SpecAugment with narrow masking width（Weak SpecAugment）
 Speech Chain Reconstruction

CER Results(Red boxes represent the best performance in each scenario)

2. Strong Augmentation 𝑨(·):

SpecAugment with wide masking width

3. Different Pseudo Labeling Threshold 𝝉:

The larger 𝜏 is, the less pseudo labels used for training

Experiment Conclusions:• Speech Chain Reconstruction outperforms
Weak SpecAugment in all scenarios.
• Static prefix tokens generate by 𝜶(𝒙𝒖 )benefit semi-supervised ASR training a lot• No large improvement of dynamic prefix
tokens has been observed

2022/11/1

2022©Heli Qi AHC-Lab, IS, NAIST

13/15

Conclusion & Future Work
Conclusion:
1. Speech Chain Reconstruction protects the linguistic information of the speech
after augmentation.
2. As a part of ASR input, prefix tokens also need augmentation for the applicationof consistency regularization.
3. Updating prefix tokens during training need more smart designs to better
evaluate its effectiveness, such as updating interval.
Future work:
1. Move from RNN-based ASR to Transformer-based ASR models.
2. Explore other semi-supervised ASR training strategies.
3. Conduct experiments on more challenging datasets, e.g. large-scale speechdatasets, noisy speech dataset, and so on.
2022/11/1

2022©Heli Qi AHC-Lab, IS, NAIST

14/15

Thank you very much for listening!
Really appreciate your patience so far!

2022/11/1

2022©Heli Qi AHC-Lab, IS, NAIST

15/15