Aiming for Generalization, Efficiency andInterpretability in Machine Learning for Speech
and Audio
Cem Subakan

April 21, 2023

1 / 70Plan

Introduction

Generalization

Efficiency

Interpretability

SepFormer

CSSL

PIQ

REAL-M

Cross-Modal
Representation
Learning

Healthcare
Applications

DSLD

BabyCry

2 / 70Table of Contents
Introduction
Bio
Studying Generalization on Source Separation
Problem Definition
SepFormer
REAL-M: Towards Speech Separation on Real Mixtures
Efficiency
Continual Representation Learning
Cross-Modal Representation Learning
Interpretability
PIQ:Posthoc Interpretation via Quantization
Healthcare Applications with Audio
ML for Speech and Language Disorders
ML for Infant Cry Analysis

3 / 70Table of Contents
Introduction
Bio
Studying Generalization on Source Separation
Problem Definition
SepFormer
REAL-M: Towards Speech Separation on Real Mixtures
Efficiency
Continual Representation Learning
Cross-Modal Representation Learning
Interpretability
PIQ:Posthoc Interpretation via Quantization
Healthcare Applications with Audio
ML for Speech and Language Disorders
ML for Infant Cry Analysis

4 / 70Short Biography

BSc+MSc, EE - Signal Processing, Bogazici U.

5 / 70Short Biography

BSc+MSc, EE - Signal Processing, Bogazici U.
PhD, CS - Machine Learning, UofI UrbanaChampaign

5 / 70Short Biography

BSc+MSc, EE - Signal Processing, Bogazici U.
PhD, CS - Machine Learning, UofI UrbanaChampaign
Postdoc, Mila (Quebec AI Institute)5 / 70Short Biography

BSc+MSc, EE - Signal Processing, Bogazici U.
PhD, CS - Machine Learning, UofI UrbanaChampaign
Postdoc, Mila (Quebec AI Institute)
Applied Research Team-Mila

5 / 70Short Biography

BSc+MSc, EE - Signal Processing, Bogazici U.
PhD, CS - Machine Learning, UofI UrbanaChampaign
Postdoc, Mila (Quebec AI Institute)
Applied Research Team-Mila
Postdoc, USherbrooke+Mila
Asst. Prof. ULaval+Adjunct Prof. at Concordia U.

5 / 70My Research

 I work on developing machine learning methods for speech andaudio.

6 / 70My Research
 I work on developing machine learning methods for speech andaudio.
 My current research goals revolve around,
I Generalization under real-life settings
I Efficiency (e.g. Continual Learning)
I Interpretability, Explainability

Efficiency

Generalization

ML forSpeech&Audio

Interpretability

6 / 70Speech and Audio Modeling Speech Enhancement
Noisy

Enhanced
model

t

t

 Speech Recognition

model
I WORK ON SPEECHt

7 / 70Speech and Audio Modeling Speech Separation
Estimated Source 1
MixtureSeparator
Estimated Source 2

 Text-to-Speech

model
I WORK ON SPEECHt
8 / 70Speech and Audio Modeling Speaker Diarization

model

Who spoke when?

t Speaker Verification

model

Same Speaker?

 Other problems: Spoof Detection, Music Source Separation, Music
Transcription, Sound Event Detection/Classification...
9 / 70Speech and Audio Modeling Field with huge economic value & job opportunities,
I Speech Recognition (e.g. Siri)
I Speech Enhancement (e.g. Google meet, Zoom)
I Text-to-Speech
I Speaker Verification, Spoof Detection(Banks)
I Speaker Diarization for Meeting Analysis (Nuance, Microsoft)
I Source Separation (e.g. Beatles Rock Band, Meeting Analysis)

10 / 70

Table of Contents
Introduction
Bio
Studying Generalization on Source Separation
Problem Definition
SepFormer
REAL-M: Towards Speech Separation on Real Mixtures
Efficiency
Continual Representation Learning
Cross-Modal Representation Learning
Interpretability
PIQ:Posthoc Interpretation via Quantization
Healthcare Applications with Audio
ML for Speech and Language Disorders
ML for Infant Cry Analysis

11 / 70

Table of Contents
Introduction
Bio
Studying Generalization on Source Separation
Problem Definition
SepFormer
REAL-M: Towards Speech Separation on Real Mixtures
Efficiency
Continual Representation Learning
Cross-Modal Representation Learning
Interpretability
PIQ:Posthoc Interpretation via Quantization
Healthcare Applications with Audio
ML for Speech and Language Disorders
ML for Infant Cry Analysis

12 / 70

Source Separation

Estimated Source 1
MixtureSeparator
Estimated Source 2

13 / 70

Source Separation

s1

s2

...

sK

x
 The observation x is dependent on latent factors s1 , s2 , . . . , sK .
I Technical definition:
s1 ∼ p(s1 ) . . . sK ∼ p(sK )x ∼ p(x|s1 , . . . , sK )I Goal: Obtain p(s1 |x), p(s2 |x), . . . , p(sK |x)

14 / 70

Single-Microphone Source Separation Problem
Source 1

Estimated Source 1
MixtureSource 2

Separator

Estimated Source 2

 Goal: To recover the original sources from the observed mixture Applications: Music production, hearing devices, meeting analysis,
editing software, and more...
 Some of my contributions
I Hierarchical tensor factorizations
I Globally optimal unsupervised source separation with FHMM.
I Neural network analogs to matrix factorization (best paper award)
I GANs in source separation
I SepFormer, a self-attention based source separation architecture andobtain state-of-the-art results on multiple datasets.
I REAL-M dataset and evaluation framework
15 / 70

Table of Contents
Introduction
Bio
Studying Generalization on Source Separation
Problem Definition
SepFormer
REAL-M: Towards Speech Separation on Real Mixtures
Efficiency
Continual Representation Learning
Cross-Modal Representation Learning
Interpretability
PIQ:Posthoc Interpretation via Quantization
Healthcare Applications with Audio
ML for Speech and Language Disorders
ML for Infant Cry Analysis

16 / 70

WSJ0-2Mix Leaderboard

 Taken from https://paperswithcode.com/sota/
speech-separation-on-wsj0-2mix on October 2022. SepFormer
stayed state of the art on WSJ0-2Mix from October 2020-September
2022.
 ICASSP2021 + IEEE TASL. Currently has 125 >200 citations
according to google scholar. ∼1000 Monthly downloads.
17 / 70

SepFormer Architecture

18 / 70

SepFormer Architecture

the time axis
chunk h̄
h̄

h̄0

h̄1

h̄2

h̄h3

h̄4

h̄5

h̄6

concatenate chunks

the time axis
0

h0
0

h1
0

the chunkaxis

h2IntraTransformer

InterTransformer

0

h3
0

h4
0

h5
0

h6
18 / 70

Best Results on Mixtures of 2 speakers(WSJ0-2Mix)
Model
TasnetSignPredictionNet
ConvTasnet
Two-Step CTN
DeepCASA
FurcaNeXt
DualPathRNN
sudo rm -rf
VSUNOS
DPTNetWavesplit
SepFormer

SI-SNRi
10.8
15.3
15.3
16.1
17.7
n.a.
18.8
18.9
20.1
20.2
22.2
22.3

SDRi
11.1
15.6
15.6
n.a.
18.0
18.4
19.0
n.a.
20.4
20.6
22.3
22.4


SNR ∝ 10 log# Param
n.a
55.2M
5.1M
8.6M
12.8M
51.4M
2.6M
2.6M
7.5M
2.6M
29M
26M

Ener. Signal
Ener. Noise

Stride
20
8
10
10
1
n.a.
1
10
2
1
1
8



19 / 70

Best Results on Mixtures of 3 Speakers(WSJ0-3Mix)

Model
ConvTasnet
DualPathRNNVSUNOS
Wavesplit
Sepformer

SI-SNRi
12.7
14.7
16.9
17.8
19.5

SDRi
13.1
n.a
n.a
18.1
19.7

# Param
5.1M
2.6M
7.5M
29M
26M

Example Results on Test Set:
Click for Mixture
Click for Estimated Source1
Click for Estimated Source2
Click for Estimated Source3

20 / 70

Speed/Memory Comparison with Other
Methods
Speed and Memory Comparison on Forward Pass:
Average Forward-Pass Time
288.0

40.0
21.0

GBytesmiliseconds

Memory Usage

SepFormer
DP-RNN
DPTNetWavesplit

501.0

166.0
95.0
55.0
32.0

11.0
6.0

SepFormer
DP-RNN
DPTNetWavesplit

3.0
1.0

2.0

3.0

4.0

Input sequence length in seconds

5.0

2.0

1.0

2.0

3.0

4.0

Input sequence length in seconds

5.0

21 / 70

Speed/Memory Comparison with Other
Methods
Speed and Memory Comparison on Forward Pass:
Average Forward-Pass Time
21.0

GBytes

288.0

40.0

166.0
95.0
55.0
32.0

11.0
6.0

SepFormer
DP-RNN
DPTNetWavesplit

3.0
1.0

2.0

3.0

4.0

Input sequence length in seconds

5.0

2.0

1.0

2.0

3.0

4.0

Input sequence length in seconds

5.0

Training Curve Comparison:
Training Speed on WSJ0-2Mix
20
18

SI-SNRi (dB)miliseconds

Memory Usage

SepFormer
DP-RNN
DPTNetWavesplit

501.0

16
14

SepFormer
DP-RNN
DPTNet

12
10
8

0

10

20

30

Training time (hours)

40
21 / 70

Environmental Corruption
We try our model with environmental noise / reverberation.
Best results on the WHAM dataset (noise).

Model
ConvTasnet
Learnable fbank
Wavesplit
Sepformer

SI-SNRi
12.7
12.9
16.0
16.4

SDRi
16.5
16.7

Best results on the WHAMR (noise + reverb) dataset.

Model
ConvTasnet
BiLSTM Tasnet
Wavesplit
Sepformer

SI-SNRi
8.3
9.2
13.2
14.0

SDRi
12.2
13.0
22 / 70

Cross-Dataset Experiment
We test our model trained on WSJ0-2Mix on LibriMix.
Model
ConvTasnet
Sepformer trained on WSJ0-2Mix
Wavesplit
Sepformer
Sepformer + FT

SI-SNRi
14.7
17.0
20.5
20.2
20.6

SDRi
17.5
20.7
20.5
20.8

We test our model trained on WSJ0-3Mix on LibriMix.
Model
ConvTasnet
Sepformer trained on WSJ0-3Mix
Wavesplit
Sepformer
Sepformer + FT

SI-SNRi
10.4
15.0
17.5
18.2
18.7

SDRi
15.6
18.0
18.6
19.0

Note: We release our pretrained models, training scripts on SpeechBrain!
23 / 70

Synthetic vs Real Life Mixture

Synthetic: WSJ0-2Mix test set
Click for Mixture
Click for Estimated Source1
Click for Estimated Source2
Real-life: One mic, two people speaking, reverberant environment
Click for Mixture
Click for Estimated Source1
Click for Estimated Source2
Click for Mixture
Click Estimated Source 1
Click Estimated Source 2

24 / 70

Table of Contents
Introduction
Bio
Studying Generalization on Source Separation
Problem Definition
SepFormer
REAL-M: Towards Speech Separation on Real Mixtures
Efficiency
Continual Representation Learning
Cross-Modal Representation Learning
Interpretability
PIQ:Posthoc Interpretation via Quantization
Healthcare Applications with Audio
ML for Speech and Language Disorders
ML for Infant Cry Analysis

25 / 70

Closing the reality gap

 We need evaluation sets that represents the challanges of real-life sothat researchers can more meaningfully benchmark theirperformance.
 We can then design data augmentations, and models to improve
performance on real-life data.

26 / 70

Closing the reality gap

 We need evaluation sets that represents the challanges of real-life sothat researchers can more meaningfully benchmark theirperformance.
 We can then design data augmentations, and models to improve
performance on real-life data.
 An important hurdle: Ground truth data.

26 / 70

Lack of ground truth in real-life separation
 Synthetic source separation datasets
Source 1

Estimated Source 1
MixtureSource 2

Separator

Estimated Source 2

Ground truth

27 / 70

Lack of ground truth in real-life separation
 Synthetic source separation datasets
ks1 − sb1 kSource 1

Estimated Source 1

MixtureSource 2

Ground truth

Separator

Estimated Source 2

ks2 − sb2 k

27 / 70

Lack of ground truth in real-life separation
 Synthetic source separation datasets
ks1 − sb1 kSource 1

Estimated Source 1

MixtureSeparator

Source 2

Estimated Source 2

ks2 − sb2 k

Ground truth Real-life source separation
Estimated Source 1
MixtureSeparator
Estimated Source 2

27 / 70

Lack of ground truth in real-life separation
 Synthetic source separation datasets
ks1 − sb1 kSource 1

Estimated Source 1

MixtureSeparator

Source 2

Estimated Source 2

ks2 − sb2 k

Ground truth Real-life source separation
Estimated Source 1
Mixture?
Separator
Estimated Source 2?

27 / 70

Tackling the lack of ground truth The lack of ground truth prevents evaluating estimation quality on
real-life data.

28 / 70

Tackling the lack of ground truth The lack of ground truth prevents evaluating estimation quality on
real-life data.
 We can however estimate the performance!
 We can train a model to estimate the performance.

28 / 70

Tackling the lack of ground truth The lack of ground truth prevents evaluating estimation quality on
real-life data.
 We can however estimate the performance!
 We can train a model to estimate the performance.
1.00

1.00

0.75

0.75

0.50

0.50

0.25
0

20

40

0.00

0.25
60

80

100

120

0

20

40

0.00

0.25

0.25

0.50

0.50

0.75

0.75

1.00

BAD

60

80

100

120

1.00

GOOD

28 / 70

Tackling the lack of ground truth
 Synthetic source separation datasets
ks1 − sb1 kSource 1

Estimated Source 1

MixtureSeparator

Source 2

Estimated Source 2

ks2 − sb2 k

Ground truth Real-life source separation
Estimated Source 1
Mixturef (mix, sb1 )Separator
Estimated Source 2
f (mix, sb2 )29 / 70

REAL-M: Towards Speech Separation on Real
Mixtures Goal: Systematic Evaluation of Speech Separation Models on
Real-Life Speech Mixtures.
 Contributions:
I We propose a dataset for real-life speech separation. The dataset
is crowdsourced, hence scalable and diverse in acoustic conditions,recording hardware, speakers.
I We show that blind SI-SNR estimation is a feasible way to evaluate
real-life speech separation.
I Therefore, this opens up a scalable methodology for large-scale
real-life source separation evaluation.
I The 5th most viewed poster in ICASSP 2022! (out of 1900 posters)

30 / 70

Table of Contents
Introduction
Bio
Studying Generalization on Source Separation
Problem Definition
SepFormer
REAL-M: Towards Speech Separation on Real Mixtures
Efficiency
Continual Representation Learning
Cross-Modal Representation Learning
Interpretability
PIQ:Posthoc Interpretation via Quantization
Healthcare Applications with Audio
ML for Speech and Language Disorders
ML for Infant Cry Analysis

31 / 70

Table of Contents
Introduction
Bio
Studying Generalization on Source Separation
Problem Definition
SepFormer
REAL-M: Towards Speech Separation on Real Mixtures
Efficiency
Continual Representation Learning
Cross-Modal Representation Learning
Interpretability
PIQ:Posthoc Interpretation via Quantization
Healthcare Applications with Audio
ML for Speech and Language Disorders
ML for Infant Cry Analysis

32 / 70

Continual Learning

Batch Learning
D → fθ (.)
Continual Learning
D1 → fθ (.)
D2 → fθ (.)..
.
DT → fθ (.)

33 / 70

Continual Learning

Batch Learning
D → fθ (.)
Continual Learning
D1 → fθ (.)
D2 → fθ (.)..
.
DT → fθ (.)

33 / 70

Continual Supervised Learning vs Continual
Self-Supervised Learning Continual Supervised Learning (CSUP)
...
D1
D2
DT
gθ

gθ

...

gθ

34 / 70

Continual Supervised Learning vs Continual
Self-Supervised Learning Continual Supervised Learning (CSUP)
...
D1
D2
DT
gθ

gθ

...

gθ

 Continual Representation Learning
...
D1
D2
DT
fθ

fθ

...

fθ

hθ

hθ

...

hθ

34 / 70

Continual Supervised Learning vs Continual
Self-Supervised Learning Continual Supervised Learning (CSUP)
...
D1
D2
DT
gθ

gθ

...

gθ

 Continual Representation Learning
...
D1
D2
DT
fθ

fθ

...

fθ

hθ

hθ

...

hθ

 Potential Advantages of CRL: Flexible, Empirically less prone toforgetting, Computational Savings

34 / 70

Continual Supervised Learning vs Continual
Self-Supervised Learning Continual Supervised Learning (CSUP)
...
D1
D2
DT
gθ

gθ

...

gθ

 Continual Representation Learning
...
D1
D2
DT
fθ

fθ

...

fθ

hθ

hθ

...

hθ

 Potential Advantages of CRL: Flexible, Empirically less prone toforgetting, Computational Savings
 Handles the realistic case where only a subset of labels is available.
34 / 70

Continual Self-Supervised Learning Published in SPL 2023, Will be presented in ICASSP 2023.
 Training the encoder

35 / 70

Continual Self-Supervised Learning Published in SPL 2023, Will be presented in ICASSP 2023.
 Training the encoder Training the output head

35 / 70

Continual Self-Supervised Learning Published in SPL 2023, Will be presented in ICASSP 2023.
 Training the encoder Training the output head Linear Evaluation Protocol (LEP): A linear layer is trained on topof the pretrained encoder using data from current and previous tasks
D1 , D2 , . . . , Dt .
35 / 70

Continual Self-Supervised Learning Published in SPL 2023, Will be presented in ICASSP 2023.
 Training the encoder Training the output head Linear Evaluation Protocol (LEP): A linear layer is trained on topof the pretrained encoder using data from current and previous tasks
D1 , D2 , . . . , Dt .
 Subset Linear Evaluation Protocol (SLEP): Only a percentage ofdata is labeled. This simulates the real-life use cases.
35 / 70

Continual Self-Supervised Learning Published in SPL 2023, Will be presented in ICASSP 2023.
 Training the encoder Training the output head Linear Evaluation Protocol (LEP): A linear layer is trained on topof the pretrained encoder using data from current and previous tasks
D1 , D2 , . . . , Dt .
 Subset Linear Evaluation Protocol (SLEP): Only a percentage ofdata is labeled. This simulates the real-life use cases.

35 / 70

Datasets In Domain Evaluation
I UrbanSound8k, 10 possible urban sound classes (car horn, street
music, air conditioner, ...), Each class has 1000, 4second longrecordings. In total 8.75 hours. (smallish)
I DCASE, TAU19, 10 possible urban sound classes, 40 hours of audio.

36 / 70

Datasets In Domain Evaluation
I UrbanSound8k, 10 possible urban sound classes (car horn, street
music, air conditioner, ...), Each class has 1000, 4second longrecordings. In total 8.75 hours. (smallish)
I DCASE, TAU19, 10 possible urban sound classes, 40 hours of audio.

 Out-of-domain EvaluationI VGG Sound, 560 hours of audio/visual data scraped from youtube.
300 sound classes such as instruments, horns, city sounds. Labels are
not reliable, so we use it for unsupervised learning.

36 / 70

Empirical Findings on In Domain Data CSSL is more robust to forgetting than Supervised Representation
Learning (CSUP).

MethodCSUP
CSSL-SimCLR
CSSL-Barlow Twins
CSSL-MoCo

USound8K
A (↑) F (↓)
65.6
19.6
70.3
15.3
68.5
14.1
68.4
15.6

DCASE
A (↑) F (↓)
48.2
27.6
59.7
17.8
55.9
19.0
49.5
20.2

37 / 70

Empirical Findings on In Domain Data CSSL is more robust to forgetting than Supervised Representation
Learning (CSUP).
 CSSL (even without an explicit mechanism against forgetting) isrobust against forgetting (comparable perf. with distillation).

MethodCSUP
SimCLR
Barlow Twins
MoCo
CSUP + LMSE
CSUP + Lsim
CSUP + LKLDSimCLR + LMSE
SimCLR + Lsim

USound8K
DCASE
A (↑) F (↓) A (↑) F (↓)
No distillation
65.6
19.6
48.2
27.6
70.3
15.3
59.7
17.8
68.5
14.1
55.9
19.0
68.4
15.6
49.5
20.2
With distillation
58.6
27.0
49.1
26.1
70.6
13.8
56.2
19.7
69.8
15.9
55.7
19.4
70.9
14.6
56.2
19.6
70.6
14.0
60.0
17.6
37 / 70

Empirical Findings on In Domain Data CSSL is more robust to forgetting than Supervised Representation
Learning (CSUP).
 CSSL (even without an explicit mechanism against forgetting) isrobust against forgetting (comparable perf. with distillation).
 Practical use case: CSSL is robust against forgetting in the scarcelabel case as well.
MethodUrbanSound8K
LEP
SLEPA (↑) F (↓) A (↑) F (↓)

CSUPSimCLR
Barlow Twins
MoCo

65.6
70.3
68.5
68.4

19.6
15.3
14.1
15.6

48.4
50.3
49.7
50.3

CSUP + LMSE
CSUP + Lsim
CSUP + LKLDSimCLR + LMSE
SimCLR + Lsim

58.6
70.6
69.8
70.9
70.6

27.0
13.8
15.9
14.6
14.0

43.8
54.9
55.4
50.6
51.1

DCASE TAU19LEPSLEPA (↑) F (↓) A (↑) F (↓)

No distillation
33.1
48.2
26.6
59.7
20.5
55.9
25.8
49.5
With distillation
39.2
49.1
27.1
56.2
27.4
55.7
25.1
56.2
25.0
60.0

27.6
17.8
19.0
20.2

32.5
42.1
41.0
34.8

38.4
27.9
23.4
25.8

26.1
19.7
19.4
19.6
17.6

35.1
42.1
42.6
42.0
42.8

35.8
32.4
30.3
26.5
25.9

37 / 70

Out-of-domain evaluation
 We train the encoder on a stream of unlabeled data. We test on a
fixed, out-of-domain test set.

Accuracy (%)

66
64
62
60
58

CSUP

1

2

3

CSSL

4 5 6 7 8
Training Task ID

91038 / 70

Out-of-domain evaluation
 We train the encoder on a stream of unlabeled data. We test on a
fixed, out-of-domain test set.

Accuracy (%)

66
64
62
60
58

CSUP

1

2

3

CSSL

4 5 6 7 8
Training Task ID

910

 We observe that CSSL results in better OOD generalization thancontinual supervised representation learning!

38 / 70

Out-of-domain evaluation
 We train the encoder on a stream of unlabeled data. We test on a
fixed, out-of-domain test set.

Accuracy (%)

66
64
62
60
58

CSUP

1

2

3

CSSL

4 5 6 7 8
Training Task ID

910

 We observe that CSSL results in better OOD generalization thancontinual supervised representation learning!
 Current Objectives:
I Long term goal is to have domain generalization under the continual
learning setting.
I Accelerating learning. (Similar to how humans learn)
38 / 70

Table of Contents
Introduction
Bio
Studying Generalization on Source Separation
Problem Definition
SepFormer
REAL-M: Towards Speech Separation on Real Mixtures
Efficiency
Continual Representation Learning
Cross-Modal Representation Learning
Interpretability
PIQ:Posthoc Interpretation via Quantization
Healthcare Applications with Audio
ML for Speech and Language Disorders
ML for Infant Cry Analysis

39 / 70

Cross-Modal Representation Learning

INPUT TEXTText Encoder>>>
t1 a1 t1 a2 t1 a3 t1
t2 a1> t2 a2> t2 a3> t2
t3 a> t a> t a> t
1 3
2 3
3 3

a1 a2 a3
Audio Encoder

 CLAPI We maximize ai> tj for i = j, and minimize for i 6= j.
I This enables text-based audio retrieval, zero-shot classification.

40 / 70

Cross-Modal Representation Learning Zero-shot evaluation

This is a CAT
This is a DOG
This is a GUN

Text Encoder>
t1 a1 t1
t2 a1> t2
t3 a > t
1 3

a1
Audio Encoder

41 / 70

Cross-Modal Representation Learning Audio Retreival
Cat Noises

Text Encoder

t1 a1> t1 a2> t1 a3> t1
a1 a2 a3

Audio Encoder

42 / 70

Cross-Modal Representation Learning

INPUT TEXTText Encoder>>>
t1 a1 t1 a2 t1 a3 t1
t2 a1> t2 a2> t2 a3> t2
t3 a> t a> t a> t
1 3
2 3
3 3

a1 a2 a3
Audio Encoder

 CLAPI Training this model requires large number of paired data.

43 / 70

Cross-Modal Representation Learning

INPUT TEXTText Encoder>>>
t1 a1 t1 a2 t1 a3 t1
t2 a1> t2 a2> t2 a3> t2
t3 a> t a> t a> t
1 3
2 3
3 3

a1 a2 a3
Audio Encoder

 CLAPI Training this model requires large number of paired data.
I Ongoing work: We are working on a method where we improve themodel performance using unpaired text and audio.

43 / 70

Table of Contents
Introduction
Bio
Studying Generalization on Source Separation
Problem Definition
SepFormer
REAL-M: Towards Speech Separation on Real Mixtures
Efficiency
Continual Representation Learning
Cross-Modal Representation Learning
Interpretability
PIQ:Posthoc Interpretation via Quantization
Healthcare Applications with Audio
ML for Speech and Language Disorders
ML for Infant Cry Analysis

44 / 70

Table of Contents
Introduction
Bio
Studying Generalization on Source Separation
Problem Definition
SepFormer
REAL-M: Towards Speech Separation on Real Mixtures
Efficiency
Continual Representation Learning
Cross-Modal Representation Learning
Interpretability
PIQ:Posthoc Interpretation via Quantization
Healthcare Applications with Audio
ML for Speech and Language Disorders
ML for Infant Cry Analysis

45 / 70

Neural Network Explanation

 Why does this particular input lead to that particular output?

46 / 70

Neural Network Explanation

 Why does this particular input lead to that particular output?

46 / 70

Neural Network Explanation

 Why does this particular input lead to that particular output?

Recording, Classified as DOG

46 / 70

Neural Network Explanation

 Why does this particular input lead to that particular output?

Recording, Classified as DOG
Interpretation

46 / 70

Posthoc Interpretation via Quantization

 We have developed a method that learns “high-level” concepts foreach class in form of latent VQ dictionary, and then reconstructs the
input using this VQ dictionary conditioned on the class information.

47 / 70

Posthoc Interpretation via Quantization

Above shows the inference time. In training, we only use images withsingle classes. NOT mixtures.
48 / 70

Qualitative Results on Images

Left-to-Right: Input, PIQ (ours), VIBI, L2I, LIME, FLINT
49 / 70

Mean-Opinion-Scores on Images

50 / 70

Quantitative Results on Images

 Input Fidelity
FID-I =

Ni
1 Xh
arg max fc (xn ) = arg max fc (xint,n ) ,
c
c
N n=1

 Faithfulness
Faithfulness = fcb(x) − fcb(x − xint ),51 / 70

Mean-Opinion Scores on Audio
5

The Opinion Scores for Interpretations on Audio Mixtures, Set 1

4
3
2
1

5

PIQ L2I-1 L2I-2 PIQ L2I-1 L2I-2 PIQ L2I-1 L2I-2 PIQ L2I-1 L2I-2
The Opinion Scores for Interpretations on Audio Mixtures, Set 2

4
3
2
1

PIQ

L2I-2

PIQ

L2I-2

PIQ

L2I-2

PIQ

L2I-2

Click for More Example Results
52 / 70

Conclusions on Interpretability

 I am trying to build a research axis on Interpretability /Explanations. One incoming PhD student.
 Several interesting applications on audio domain.
 Working on generalizing our approach to more complex audio /images.

53 / 70

Table of Contents
Introduction
Bio
Studying Generalization on Source Separation
Problem Definition
SepFormer
REAL-M: Towards Speech Separation on Real Mixtures
Efficiency
Continual Representation Learning
Cross-Modal Representation Learning
Interpretability
PIQ:Posthoc Interpretation via Quantization
Healthcare Applications with Audio
ML for Speech and Language Disorders
ML for Infant Cry Analysis

54 / 70

Table of Contents
Introduction
Bio
Studying Generalization on Source Separation
Problem Definition
SepFormer
REAL-M: Towards Speech Separation on Real Mixtures
Efficiency
Continual Representation Learning
Cross-Modal Representation Learning
Interpretability
PIQ:Posthoc Interpretation via Quantization
Healthcare Applications with Audio
ML for Speech and Language Disorders
ML for Infant Cry Analysis

55 / 70

ML for Speech and Language Disorders
 We have just submitted a grant application from CIHR-SickKids
foundation with Prof. Selcuk Guven from UdeM Audiologie
department.
 The goal is to use ML to Diagnose and Understand Speech andLanguage Disorders.
I Example 1
I Example 2
Model1:CHUCK SEEMS THIRSTY AFTER THE RACE TRIFLING
THIRSTY OVER THE LADS
Model2: CHUCK SEEMS THIRSTY AFTER THE RACE CHRISHENG
THIRSTY OVER THE WAYS
Model3: CHUCK SEENS BURSTY AFTER THE RACE CHICKINGTHIRSDAY OVER THE WAGE

56 / 70

ML for Speech and Language Disorders
 We have just submitted a grant application from CIHR-SickKids
foundation with Prof. Selcuk Guven from UdeM Audiologie
department.
 The goal is to use ML to Diagnose and Understand Speech andLanguage Disorders.
I Example 1
I Example 2
Model1:CHUCK SEEMS THIRSTY AFTER THE RACE TRIFLING
THIRSTY OVER THE LADS
Model2: CHUCK SEEMS THIRSTY AFTER THE RACE CHRISHENG
THIRSTY OVER THE WAYS
Model3: CHUCK SEENS BURSTY AFTER THE RACE CHICKINGTHIRSDAY OVER THE WAGE Subgoals include,
I Data collection under clinical setting
I ML for diagnosis models + Active learning for noisy label re-labeling
I Robust phoneme based ASR to interpret the diagnosis.
I Applying and developing neural network interpretation methods.

 This project is included in an accepted Compute Canada RRG
application.
56 / 70

Table of Contents
Introduction
Bio
Studying Generalization on Source Separation
Problem Definition
SepFormer
REAL-M: Towards Speech Separation on Real Mixtures
Efficiency
Continual Representation Learning
Cross-Modal Representation Learning
Interpretability
PIQ:Posthoc Interpretation via Quantization
Healthcare Applications with Audio
ML for Speech and Language Disorders
ML for Infant Cry Analysis

57 / 70

ML for Infant Cry Analysis
 Collaboration with UbenwaAI, a Mila based startup.
 The goal is to develop machine learning methods for Infant Cry
Analysis.
 Recent accepted paper:
Self-Supervised Learning for Cry Analysis,
ICASSP 2023 Workshop on Self-Supervision in Audio, Speech and
Beyond (gets into conference proceedings).

58 / 70

Baby Identification Challenge: CryCeleb

The CryCeleb2023 Challenge!

59 / 70

Ubenwa Collaboration

We will have one PhD position in collaboration with Ubenwa on
Interpretability, Continual Learning, Self-Supervised Learning.
Contact me if you are interested!

60 / 70

SpeechBrain

On August 28th, We will have the first annual SpeechBrain summit (withInterspeech endorsement). There will be talks from industry, academia,
and a panel discussion with creators of torchaudio, Kaldi, Librosa,
ESPNet, NeMO on open source software for speech.

61 / 70

Conclusions

 I tried to summarize my recent work and goals concerning
generalization, efficiency and interpretability, centered around speech
and audio applications.

62 / 70

Conclusions

 I tried to summarize my recent work and goals concerning
generalization, efficiency and interpretability, centered around speech
and audio applications.
 Today I talked about:
Obtaining SOTA results (SepFormer)

62 / 70

Conclusions

 I tried to summarize my recent work and goals concerning
generalization, efficiency and interpretability, centered around speech
and audio applications.
 Today I talked about:
Obtaining SOTA results (SepFormer)
Systematic Evaluation of Real-data Perf. (REAL-M)

62 / 70

Conclusions

 I tried to summarize my recent work and goals concerning
generalization, efficiency and interpretability, centered around speech
and audio applications.
 Today I talked about:
Obtaining SOTA results (SepFormer)
Systematic Evaluation of Real-data Perf. (REAL-M)
Efficient representation learning (CSSL+CrossModal)

62 / 70

Conclusions
 I tried to summarize my recent work and goals concerning
generalization, efficiency and interpretability, centered around speech
and audio applications.
 Today I talked about:
Obtaining SOTA results (SepFormer)
Systematic Evaluation of Real-data Perf. (REAL-M)
Efficient representation learning (CSSL+CrossModal)Interpretability (PIQ)

62 / 70

Conclusions
 I tried to summarize my recent work and goals concerning
generalization, efficiency and interpretability, centered around speech
and audio applications.
 Today I talked about:
Obtaining SOTA results (SepFormer)
Systematic Evaluation of Real-data Perf. (REAL-M)
Efficient representation learning (CSSL+CrossModal)Interpretability (PIQ) Machine Learning for Signal Processing Class in fall!
 Would love to chat if anything picks your attention!

62 / 70

Thanks!
Introduction

Generalization

Efficiency

Interpretability

SepFormer

CSSL

PIQ

REAL-M

Cross-Modal
Representation
Learning

Healthcare
Applications

DSLD

BabyCry

63 / 70

Appendix 1 - The SepFormer Masking
Architecture

63 / 70

Appendix 1 - The SepFormer Masking
Architecture

We train this architecture with permutation invariant SI-SNR.


sb> s
kstarget k2
starget :=s, enoise := sb − starget , SI-SNR := 10 log10
ksk2
kenoise k2!
kX
kstargetk2
PIT-SISNR =
min
10
log
0
10
k
k 0 ∈Pkb
s k − starget k2k63 / 70

Appendix2 - Tackling the lack of ground truth The same concept in speech separation:
Click for Mixture 1
Estimated Source 1
Estimated Source 2

Click for Mixture 2
Estimated Source 1
Estimated Source 2

63 / 70

Appendix2 - Tackling the lack of ground truth The same concept in speech separation:
Click for Mixture 1
Estimated Source 1
Estimated Source 2

Click for Mixture 2
Estimated Source 1
Estimated Source 2

 As humans we know if the estimation is good or not.
 Standard practice in evaluation:
Performance ∝ −d(estimate, groundtruth) However, it is not easy to have ground truth always. We can instead use amodel to predict the performance as
Performance estimate = f (estimate, input)
where, f is a neural network, input is the mixture in speech separationcase.

63 / 70

Accuracy (%)

Appendix3 - CSSL More Tasks

100
80
60
40
CSUP T = 5
CSUP T = 10

Accuracy (%)

100

CSSL T = 5
CSSL T = 10

80
60
40
1

2

3

4

5
6
Task ID

7

8

9

10

 SSL effectively combats
forgetting, even without
explicitly combatting
forgetting!
 Offline accuracies: 80.9%,
68.2% with CSUP, 74.3%,
62.5% with CSSL.

63 / 70

Appendix4 - CSSL, More SSL Methods

MethodUrbanSound8K
LEP
SLEPA (↑) F (↓) A (↑) F (↓)

CSUPSimCLR
Barlow Twins
MoCo

65.6
70.3
68.5
68.4

19.6
15.3
14.1
15.6

CSUP + LMSE
CSUP + Lsim
CSUP + LKLDSimCLR + LMSE
SimCLR + Lsim
Barlow Twins + LMSE
Barlow Twins + Lsim
MoCo + LMSE
MoCo + Lsim

58.6
70.6
69.8
70.9
70.6
69.5
70.0
67.7
68.5

27.0
13.8
15.9
14.6
14.0
14.0
13.4
14.8
15.0

DCASE TAU19LEPSLEPA (↑) F (↓) A (↑) F (↓)

No distillation
33.1
48.2
26.6
59.7
20.5
55.9
25.8
49.5
With distillation
43.8
39.2
49.1
54.9
27.1
56.2
55.4
27.4
55.7
50.6
25.1
56.2
51.1
25.0
60.0
47.3
26.1
56.2
49.5
23.9
55.1
51.4
25.4
49.4
50.9
26.8
50.7
48.4
50.3
49.7
50.3

27.6
17.8
19.0
20.2

32.5
42.1
41.0
34.8

38.4
27.9
23.4
25.8

26.1
19.7
19.4
19.6
17.6
19.8
19.8
21.4
18.4

35.1
42.1
42.6
42.0
42.8
41.1
41.2
34.1
35.7

35.8
32.4
30.3
26.5
25.9
25.6
23.1
27.9
24.0

63 / 70

Appendix 5 - Evaluating the SI-SNR
Estimator (Mismatch) We first evaluate the SI-SNR Estimator on synthetic data.

 Evaluating dprnn separator (left), convtasnet separator (right). The
estimator is trained with SepFormer.
 Both scatter plots correspond to Pearson correlation coefficient of
0.8.
63 / 70

Data collection
 We crowdsource a dataset to construct an evaluation of audio mixtures inreal-life. Click to hear examples.

64 / 70

Data collection
 We crowdsource a dataset to construct an evaluation of audio mixtures inreal-life. Click to hear examples.

64 / 70

Data collection
 We crowdsource a dataset to construct an evaluation of audio mixtures inreal-life. Click to hear examples.

64 / 70

Data collection
 We crowdsource a dataset to construct an evaluation of audio mixtures inreal-life. Click to hear examples.

 Contributors are asked simultaneously read the shown sentences.
 This gives a way to collect real-life speech mixtures in a scalable way. We
interface our platform with Mechanical Turk.
 We collected 3 hours of speech, from 50 unique speakers, with various
native (e.g. US, UK) and non-native (e.g. French, Italian, Persian, Indian,African) accents, in various conditions, with various recording equipment.
64 / 70

SI-SNR Estimator Training Pipeline
 We construct a performance estimator f (.) such that,f (x, sb) ≈ SI-SNR(s, sb),
where f (.) is a neural network, x is the mixture, sb is the source
estimate, s is the true source.

65 / 70

SI-SNR Estimator Training Pipeline
 We construct a performance estimator f (.) such that,f (x, sb) ≈ SI-SNR(s, sb),
where f (.) is a neural network, x is the mixture, sb is the source
estimate, s is the true source.
 We train the estimator on synthetic mixtures on which ground truth
information is available. (Also a lot of data!)x ŝ1 Pre.Tr. Separator P ss12ŝ2

SI-SNR

SNR1

SI-SNR

SNR2

65 / 70

SI-SNR Estimator Training Pipeline
 We construct a performance estimator f (.) such that,f (x, sb) ≈ SI-SNR(s, sb),
where f (.) is a neural network, x is the mixture, sb is the source
estimate, s is the true source.
 We train the estimator on synthetic mixtures on which ground truth
information is available. (Also a lot of data!)
SNR1
 ŝ1 SI-SNRx
Pre.Tr. Separator P ss12ŝ2
SNR2
SI-SNRx, ŝkSI-SNR Estimatordk
SNR

d 1 k + kSNR2 − SNR
d 2 k.
L = kSNR1 − SNR
 SI-SNR Estimator is a 5-layer convolutional NNet in the time
domain.

65 / 70

SI-SNR Estimator Training Pipeline
 We construct a performance estimator f (.) such that,f (x, sb) ≈ SI-SNR(s, sb),
where f (.) is a neural network, x is the mixture, sb is the source
estimate, s is the true source.
 We train the estimator on synthetic mixtures on which ground truth
information is available. (Also a lot of data!)
SNR1
 ŝ1 SI-SNRx
Pre.Tr. Separator P ss12ŝ2
SNR2
SI-SNRx, ŝkSI-SNR Estimatordk
SNR

d 1 k + kSNR2 − SNR
d 2 k.
L = kSNR1 − SNR
 SI-SNR Estimator is a 5-layer convolutional NNet in the time
domain.
 Important Question: Is this estimator going to work well(generalize to) real-mixtures?
65 / 70

Evaluating the SI-SNR Estimator We first evaluate the SI-SNR Estimator on synthetic data.

 Evaluating on (left) LibriMix, (right) WHAMR!
 Both scatter plots correspond to Pearson correlation coefficient of
0.8.

66 / 70

Evaluating the SI-SNR Estimator We first evaluate the SI-SNR Estimator on synthetic data.

 Evaluating on (left) LibriMix, (right) WHAMR!
 Both scatter plots correspond to Pearson correlation coefficient of
0.8.
 Important Question: Is this estimator going to work well(generalize to) real-mixtures?
66 / 70

Evaluating the SI-SNR Estimator on REAL-M
 We validate the SI-SNR estimator with a user study on real-life data.
 We presented 50 random mixtures and the separation results to 5 users.
 We asked the users to rate the presented separation result between 1-5.

67 / 70

Results of User Study Results of the user study suggest that on average the opinion scores
correlate-well with SNR estimates.

8
6
4
2
0

All

bad
poor
fair
good
exclt

8
6
4
2
0

bad
poor
fair
good
exclt

8
6
4
2
0

bad
poor
fair
good
exclt

8
6
4
2
0

bad
poor
fair
good
exclt

8
6
4
2
0

bad
poor
fair
good
exclt

User 1 10 User 2 10 User 3 10 User 4 10 User 5 10

bad
poor
fair
good
exclt

10
8
6
4
2
0

 Y-axes show the estimated-SNR, X-axes show the user rating.

68 / 70

Further evaluation of SI-SNR Estimator The performance rankings of models on synthetic data holds true for
REAL-M as well.
 We also observe that with training epochs performance on REAL-M
dataset improves.

Estimated
SI-SNR on REALM dataset SI-SNR on WHAMR! dataset
3.0
8
6

2.0

SI-SNR

Estimated SI-SNR

2.5

1.5
1.0

4
2

sepformer
dprnn
convtasnet

0

0.5
0

20

40

60

Epoch

80

100

0

20

40

60

Epoch

80

100

69 / 70

Conclusions
 With the REAL-M framework, we are moving towards working withreal-mixtures. It provides,
I Scalable data collections
I Variability
I Blind Performance estimation

70 / 70

Conclusions
 With the REAL-M framework, we are moving towards working withreal-mixtures. It provides,
I Scalable data collections
I Variability
I Blind Performance estimation

 The performance on real-life data is far worse than synthetic
benchmarks.
\\
Separator SNR
Synth SNRReal
SepFormer
8.40
2.88
DPRNN
7.04
2.43
CTN
5.49
1.59

70 / 70

Conclusions
 With the REAL-M framework, we are moving towards working withreal-mixtures. It provides,
I Scalable data collections
I Variability
I Blind Performance estimation

 The performance on real-life data is far worse than synthetic
benchmarks.
\\
Separator SNR
Synth SNRReal
SepFormer
8.40
2.88
DPRNN
7.04
2.43
CTN
5.49
1.59
 Next steps:
I Casual talking, Meeting settings
I Scaling up

70 / 70

Conclusions
 With the REAL-M framework, we are moving towards working withreal-mixtures. It provides,
I Scalable data collections
I Variability
I Blind Performance estimation

 The performance on real-life data is far worse than synthetic
benchmarks.
\\
Separator SNR
Synth SNRReal
SepFormer
8.40
2.88
DPRNN
7.04
2.43
CTN
5.49
1.59
 Next steps:
I Casual talking, Meeting settings
I Scaling up
I Improving the generalization (Data augmentations, Using theperformance estimators, Using pretrained models, ...)

70 / 70