Filter-Recovery Network for Multi-Speaker Audio-Visual Speech Separation
Authors: Haoyue Cheng
Zhaoyang Liu
Wayne Wu
Limin Wang
Venue: ICLR 2023 poster (2023)

Abstract: In this paper, we systematically study the audio-visual speech separation task in a multi-speaker scenario. Given the facial information of each speaker, the goal of this task is to separate the corresponding speech from the mixed speech. The existing works are designed for speech separation in a controlled setting with a fixed number of speakers (mostly 2 or 3 speakers), which seems to be impractical for real applications. As a result, we try to utilize a single model to separate the voices with a variable number of speakers. Based on the observation, there are two prominent issues for multi-speaker separation: 1) There are some noisy voice pieces belonging to other speakers in the separation results; 2) Part of the target speech is missing after separation. Accordingly, we propose \textbf{BFRNet}, including a {\bf B}asic audio-visual speech separator and a Filter-Recovery Network (\textbf{FRNet}). FRNet can refine the coarse audio separated by basic audio-visual speech separator. To have fair comparisons, we build a comprehensive benchmark for multi-speaker audio-visual speech separation to verify the performance of various methods. Experimental results show that our method is able to achieve the state-of-the-art performance. Furthermore, we also find that FRNet can boost the performance of other off-the-shelf speech separators, which exhibits its ability of generalization.

Published as a conference paper at ICLR 2023
FILTER-RECOVERY NETWORK FOR MULTI-SPEAKER SPEECH SEPARATION
Haoyue Cheng 1.2
Zhaoyang Liu
Wayne Wu² Limin Wang 1,2,
'ttate Key Laboratory for Novel Software Technology, Nanjing University, China
Shanghai AI Laboratory
[chenghaoyue 98, zyliumy, e com
1mwangenju.c edu.
ABSTRACT
In this paper, we systematically study the audio-visual speech separation task multi-speaker scenario. Given the facial information of each speaker, the goal this task to separate the corresponding speech from the mixed speech. The existing works are designed for speech separation in controlled setting with fixed number of speakers (mostly or speakers), which seems to be impractical for real applications. As result, we try utilize single model separate the voices with a variable number of speakers. Based on the observation, there are two prominent issues for multi-speaker separation: There are some noisy voice pieces belonging to other speakers in the separation results; 2) Part of the target speech missing after separation. Accordingly, we propose BFRNet, including a Basic audio-visual speech separator and Filter-Recoven Network (FRNet). FR- Net can refine the coarse audio separated by basic audio-visual speech separator. To have fair comparisons, we build a comprehensive benchmark for multi-speaker audio-visual speech separation to verify the performance of various methods. Ex- perimental results show that our method is able to achieve the state-of-the-arg per- formance. Furthermore, we also find that FRNet can boost the performance other off-the-shelf speech separators, which exhibits its ability of generalization.
INTRODUCTION
Audio-visual speech separation has been extensively used in various applications. such as speech recognition (Radford et al.; Chan et al. 2015), assistive hearing device (Kumar et al. 2022). and online video meetings (Taam et al. 2022). As human voices are naturally mixed together in public places, it would be challenging to directly extract the information of interest from such raw audio- visual signals containing multiple speakers. As a result, separating audio signals for each speaker could serve an effective pre-processing step for further analysis on the audio-visual signal.
Convolutional neural networks (Gogate et al. 2018 Makishima et al. 2021 Gao Grauman 2021 and Transformers (Ramesh al.) 2021 Montesinos et al. 2022 Rahimi et al. 2022) has made prominent progress in the field of audio visual speech separation. However, previous works (ee et al. 2021 Gao & Grauman, 2021 Montesinos et al./ (202))
mostly focus two-speaker speech Afouras et al. 2018b 2019) manage 2018 requires customized models for each
separation. Although other researches (Ephrat et al.)
2018
separate voices for more speakers, (Ephrat et al. kind of mixture instead of separating all kinds of mixtures with a single model, and (Afouras et al.] 2018b) mainly contributes to enhancing the voice of the target individual while ignoring others. Fur- thermore, (Alouras et al., 2019) uses pre-enrolled speaker embeddings to extract the corresponding speech, but still leaves a performance gap compared with unenrolled speakers.
Therefore, how to efficiently and effectively separate speech under a multi- environment still requires further study. Through our explorations, prior works Gao & Grauman (2021): Montesinos et al. (2022); (CCuanggChgggg et al. (2020); Makishima et al. (2021): Afouras et al. (2019) show their superi- ority two-speaker speech separation but yield disappointing results with more speakers (e.g.. 3, 4, or even speakers). It exhibits simple yet important fact that the complexity of speech separation
"Corresooniigg author.

---

Published as a conference paper at ICLR 2023
strongly correlates with the number of speakers in the mixed audio. As shown in Fig. | the problems for multi-speaker speech separation can be empirically summarized as two folds:
1) The noisy parts (red box in Fig- ure D contains the components from other speakers within the separated audio 2) The missing parts (gray box in Figure [] represent some tar- get speech pieces dropped by mod- els. These two problems usually oc-
Tirgpe
Speedh
Mistiam
cur in complex conditions, reflecting that models fail to accurately separate audio signals under such challenging scenarios.
Sequeration Notvoaks
Viuo Separation Netwoakk
Separaation Naturrtt
Viued
As result, there is still long way to go toward solving audio-visual speech separation for multi-speaker conditions.
Patt
Harts
Figure 1: Illustration of two critical issues in multi-speaker audio-visual speech separation. The missing parts in separa- tion results compared ground truth speech are indicated by gray boxes, while the noisy parts are marked by red boxes.
To this end, we come up with an effective method BFRNet, to con- quer the aforementioned challenges in multi-speaker audio-visual speech separation. As illustrated in Figure 2 the BFRNet consists of basic audio-visual speech separator and Filter-Recovrry Network (FRNet). The FRNet aims at solving the two issues in the results of any identically structured basic separator. It comprises two serial modules, i.e., Filter and Recovery. Firstly, the Filter module utilizes the visual features of the target speaker to query corresponding audio signals from the coarse prediction and suppress components from other speakers. Then, the Recovery module uses the clean audio yielded by the Filter module to query the missing components that belong to the target speaker from others' pre- dictions. Essentially, FRNet aims to calibrate the coarsely separated audio predicted by off-the-shelff models (Gao & Grauman 2021 Montesinos et al. 2022 Chuang al. 2020).
Furthermore, we have noticed that there still one obstacle in evaluating audio-visual speech sep- aration. As we found that most works (Gao & Grauman) 2021 Montesinos et al.) 2022) evaluate performance on unfixed numbers of samples generated by randomly mixing the audio signals dur- ing each inference. it sometimes leads to hard reproductions and unfair comparisons. Consequently, to unify the evaluation protocols, we create a comprehensive benchmark to verify the models' per- formance fairly. Specifically, for each type of mixture, randomly sample test videos without replacement to make up the speech mixtures. The constructed fixed test sets serve for all experi- ments ensure fairness and reproducibility of the results.
To sum up, our contributions are as follows:
First, we design a Filter-Recoved Network (FRNet) with multi-speaker training paradigm to improve the quality of multi-speaker speech separation.
Second, to test the different methods on fair basis, a well-established benchmark for multi-speaker audio-visual speech separation created. We not only unify the evaluation protocol but also re-implement several state-of-the-an methods this benchmark.
Finally, in the experiments, we demonstrate that our proposed FRNet can be equipped with other models to further improve the quality of audio separation and achieve the state-of- the-art performance.
2 RELATED WORKS
Audio-Only Speech Separation. Using only audio modality for speech separation faces the prob-
lem of speaker agnosticism. Some works (Liu et al. 2019 Wang et al. 2018) utilize the speaker's voice embedding the hint to isolate the target speech. Current methods mostly treat the audio-only speech separation as label permutation problem. (Chen et al., (207)) cluster the similar speech to
perform speech separation. (Luo et al. 2018) requires prior information of speaker number. (Luo
2

---

Published as a conference paper at ICLR 2023
& Mesgarani] 2019) adopts a deep learning network comprising a series of convolutional layers and trains the network with permutation invarrinn loss Yu et al. 2017).
Visual Sound Separation. There various kinds of sound separation explored in the literature.
One type focuses on music separation (Zhao et al., 2018 Gao & Grauman 2019; Xu et al.) 2019
Gan et al. 2020), where the diverse shapes of musical instruments and the distinguished patterns of music sounds are the key clues. Other works concentrate on the in-the-wild sounds (Gao et al. 2018
Tzinis al., 2020 Chen et al. (2020), such animal sounds, vehicle sounds, etc. (Gao
2018
incorporate the image recognition results as the advisor and learn prototypical spectral patterns for each sounding object. (Tzinis et al., 2020) extend to unsupervised, open- open-domain audio-visual sound separation and develop a large new dataset. (Chen et al. 2020) propose an algorithm to generate temporally synchronized sound given mismatched visual information.
Visual Speech Separation. Speech is extensively studied as the most closely associated audio with humans, and has a broad range of applications. Due to the natural correlation between face and speech, many works (Gabbay et al.) [2018] [u et al., [2018; Ephrat et al.] 2018 Afouras et al.) 2019 Chung et al.) 2020 Hegge et al.] 2021] Rahimi et al. |l.]|202. (2022) employ face-related information
to separate speech from mixture in the literature. (Chung et al. 2020) use only the still images containing facial appearance to isolate speech, with the assistance of the consistency of face identity
and speech identity. Numerous methods (Afouras al. 20186 [ao & Grauman [202] Ephrat et al.
2018) explore the simultaneous lip motions and voice fluctuations clues. (Lu et al., (018) integrate
optical flow and lip movements to predict the spectrogram masks. (Hegde et al.) 2021) propose
synthesizing a virtual visual stream to deal with the situation where the visual stream is unreliable
or completely absent. Another family of works (Owens & Efros 2018
Afouras al. 2020
Truong
et al.
T02T combines multiple tasks for joint learning.
The most relevant works to ours are (Afouras et al., 2018b), (Sh al. 2020 and (Yao al. 2022)
Affours et al. 2018b) also pays attention to the uncontrolled environment with several speakers, but only focuses on enhancing the target speech and suppressing the noisy voices. Our method further separates every speech component for different mixtures in single model. (Shi et al., 2020) and procedure by adopting an additional refining separation phase. However, the refining phase applies the same model as the coarse phase. Thus if the coarse phase cannot achieve a clean separation, the refining results will likewise be sub-optimal.
(Yao et al.] 2022) perform a coarse-to-
3 METHOD
To best of our knowledge, this is the first work that systematically studies the audio-visun speech separation under a multi-speaker setting. In this section, we first elaborate on our proposed multi- speaker training strategy in Sec. [.1] Next, introduce the adopted basic audio-visual speech separator in Sec. Sec. which takes the visual information and the speech mixture as input and outputs separated speech; Then the Filter-Recovrry Network is detailedly described in Sec. 3.3 Finally, we formulate the objective function for training model in Sec. 3.4
3.1 OVERVIEW
Given a video containing S simultaneous speakers, our goal is to isolate the individual speech for each speaker. Formally, we denote time-domain speech mixture as z E Iz, 2, E RT= where Tr represents the time length, and x, is the separate speech of the i-th speaker. Since acquiring the exact individual ground truth data from the mixtures in real scenes is yet impossible, we follow previous works (Afouras et al. 2018b Gao & Grauman 202T) to synthesize mixtures by adding individual speech together.
Current works are designed for definite speakers in the mixture, mostly 2 or 3 speakers. However, these models present poor effects for practical applications, where there are usually a variable num- ber of speakers. Alternatively. they take an inefficient strategy of training separate models for each kind of mixture different in speaker numbers. To meet the demand for practical applications, create different mixtures with various numbers of speakers during training, i.c., S ranges from 2 to 5. The network is required to isolate all speech components for all kinds of mixtures.
3

---

Published as a conference paper at ICLR 2023
AudicAudio Vinsd Speoch
Filus BarnAay Nitwork
Ilkanned sovetom
Specct Nichan
Lipete'' geskens
Fller
Removal Fetieval
Nokey para
Miotag parts
Filter
Filte
Bacesary
Srid
Rrowwey
Dosoreer Neil
Figure 2: Overview of the proposed framework. It consists basic audio-visual speech separator and a Eiteer-eeooery Network (FRNet). Given mixture spectrogram containing S pieces of speech, the basic separator takes each speaker's visual clue separately and outputs the corresponding visual feature and speech mask. For i-th speaker, the basic separator outputs V; and M,. Then the visual features and speech masks of all S speakers a mixture are fed into the Recovery Network to obtain a more precise speech mask M, for cach speaker. The Filter Net utilizes visual embedding V. to reduce the noisy components in Mr to obtain Mi. Then the Recovery Net takes the clean mask M, as the query to extract the corresponding missing parts from the speech masks of other speakers. To better illustrate the effect of the FRNet, we visualize masks into the form of waveforms present speech components by ->. The red/gray boxes and the corresponding -.. represent the noisy/missing speech pieces that removed retrieved.
Fig. 2 illustrates the architecture of the proposed model, consisting of basic audio-visuee speech separator and additional Filter-Recoeery Network (FRNet). During training, we feed mixtures into the audio-visual speech separator and obtain the separated coarse speech. The model takes the frequenyy spectrogram X E 22 as input, which converted from time-domain audio æ via Short- Fourier Transform (STFT). F and Tx indicate the spectrogram's 's maximum frequency and time span. Each time-frequency bin contains the real and imaginary parts of the complex spectrogram. For i-th speaker, the basic separator outputs target mask M, E 2x2xxxxx of input spectrogram X and visual feature V,
Afterward, for each mixture, the separated S masks and visual features are sent to the FRNet to filter the noisy components and recover the missing ones. The FRNet outputs an improved mask for each speaker in the mixture, which is finally used to restore the exact time-domain speech.
3.2 BASIC AUDIO- VISUAL SPEECH SEPARATOR
In our framework, the basic audio-visual speech separator can be replaced with any network that outputs the masks of the mixture spectrogram. Following (Gao & Grauman] 2021). we adopt both lip and face clues guide target speech separation. Specifically, lip net taken dig out the con- sistency of lip motion and continuous pronunciation, and a face net is used to explore the relationship between speech and face attributes.
Lip Net. Following the previous structure (a et al. 2021: Gao & Grauman 2021 }, we feed Tü consecutive frames of lip regions into a 3D convolutional layer followed by a ShuffleNet (Ma et al. 2018) network to extract mouth features. A temporal convolutional network is further utilized to output the lip motion features Lip; E Cix2 for i-th speaker.
Face Net. The Face Net aims at leveraging the correspondence between face attributes and speech. 18 network takes a single face image as input, and outputs face embedding for i-th speaker. We repeat the face embedding along the time dimension to obtain Face; E C
Encoder-Decoder Separator. As for the speech analysis and separation end, we adopt U-Net network consisting of an encoder, a fusion module, and decoder. The encoder is composed of multiple convolutions and pooling layers. takes the mixture spectrogram X as input and outputs an audio feature Au of dimension C. x T.. Following (Xiong et al. 2022), take AV-Fusion module, an attention like operation, to obtain vision-relate audio representation for a given speaker. We first concatenate the lip feature Lip; and face feature Face; together along the time dimension
4

---

Published as a conference paper at ICLR 2023
to get visual feature V, RCxXT., then feed V2 and Au AV-Fusion module to obtain an enhanced feature. further delivered into the decoder to predict a mask M, of input spectrogram for i-th speaker, which denotes the projection values of the prediction onto the mixture. Note that the values might be positive or negative, as the spectrograms have both positive and negative values.
3.3 FILTER AND RECOVERY NETWORK
Due to the high complexity of speech mixture with multiple speakers (especially above 3 speakers), the separated target speech remains two apparent issues: 1) There still exists voices from others; 2) The isolated speech is partially missing compared the ground truth speech. To solve the two problems, we introduce the Filter-Recovery Network (FRNet), which takes the visual features and the separated coarse speech masks as input, and outputs more precise target speech masks. For convenience, we concatenate the real and imaginary parts of M, obtained by the basic separator to form dimension 2F x Tx before feeding it into FRNet, which still denote M,. Besides, to align the dimension, we transform Vi to V of shape 2F x Tx with convolutional layer.
As Fig. 2 shows, the FRNet consists of Filter Net and a Recovery Net. The former utilizes visual features to remove the noisy voices from the separated speech, and the latter learns the correlation between all separated speech to extract the missing voices from others.
Since the attention mechanism (Vaswani et al. 2017) enhances some parts of the input data while declining other parts, we adopt attention as the component module, For the sake fluent description, we define some operations here. Given three tensors q E PRx,,, k, v E DDxNNo, we compute the weighted sum of v:
1q' k, v'
U,q. Ugk, U.V].
U Ugk,Y E DAXDD
W softmax (q')</VVV)
W WE WERNN,
(2)
Attn(q, k, v) = V
Attn E RDNNN
(3)
Filter Net. Since the visual information is highly correlated with speech, we utilize each speaker's visual knowledge to reduce irrelevant voices the separation results during the Filter phase. The Filter Net consists of L basic layers, each consisting of an Attn and an MLP module, where the MLP block contains two fully-convolutionall layers and a ReLU activation function.
For I-th layer, the model takes V/ and M as input, and outputs M!+1 where M. = Mi, and LN denotes Layer Normalization (Ba et al. (016);
2 J+1 LN (Attn(V', "Vttn(V M, M,) + M,),
1+1 E RFXXX
(4)
M LN (MLP(s'+)) + 2
Mf M 22XXXX
(5)
For i-th speaker, the Filter Net output mask M, = M where the noise speech pieces are removed.
pieces of speech from other speakers in the target separation result of basic separator. So we design the Recovery Net to pull out the missing voice pieces M, for i-th speaker from the separation results of other speakers, which is much easier than recovering the missing parts from the original mixture. We define a rearrange operation by stacking S masks of other speakers:
Recovery Net. According to analysis, there are
temp- = [M;..... M,-1; M++1; Ms}. M,
reshape
temp,
M, E (5
(6)
reshape operation rearranges the input tensor to the target dimension.
where the <
The Recovery Net aims to learn the association between M, and M,, It consists of L basic layers, similar the decoder of Transformer (Vaswani et al. 2017). Specifically, the output of layer can be conveyed by the following equations, where qf M,;
df LN (Attn (g. df. gf) qf),
d reshape %
of 12 o Txx2Fx1
(7)
z 2'[]] = LN (Attn (2(10, M. M.) + d()),
z 2 E 22
(8)
s [[t] = LN (MLP(e'(t)++ +
2 [t] E 2F
(9)
sl [ 210) [x]]]
1+1 reshape 2 2l RTx22xx H+1 E 2F2xX
(10)
5

---

Published as a conference paper at ICLR 2023
We hold the view that the speech separation issues analyzed above do not always manifest to the same degree at different time slots. As result, in Equ. 8 the attention operations are performed separately for each time slice of tensor recover missing parts. Finally, the missing components M; can be obtained by applying a fully-convolutional layer on q²: i.e., Mf = FC(
After obtaining the noise- noise-filtered result M, and the recovered missing part M,, we add them up to obtain the clean and complete separation target mask:
M, M. Mr.
(11)
3.4 LOss FUNCTION
We jointly optimize the outputs of the basic separator and the FRNet. Following previous work (Pan et al. 2022). We take the scale-invariant signal-to-noise ratio (SI-SNR) (Le Roux et al. 2019) as the loss function. The predicted masks M; and M, are separately multiplied by the mixture spectrogram X to get the separated spectrograms, which are finally transformed by inverse STFT to restore the time-domain speech y, and Dt Given any prediction and ground truth s, the SI-SNR loss can be computed with the following formula:
(8.a)
(sI-SNn($, -10logio
|
(12)
|
(8.8)
We jointly train the basic separator and the FRNet by overall loss for k-th speaker:
L = DCCSS-SSN((,,,. x.) + (1 A)Cst-sNn((s...))
(13)
where À is the factor to control the ratio of the two loss parts. For batch of mixtures containing total number of speakers, the training loss is the average of all N individual losses.
EXPERIMENTS
4.1 DATASETS
VoxCeleb2 (Chung al, 2018). This dataset is organized in the identity labels, with 5994 speakers in the training set and another 118 in the test set. It contains more than 1 million samples. each consisting of an utterance and synchronized face tracks. Following (Gao & Grauman| 202I), we hold out two videos for each speaker in the original training set and utilize the rest videos training set. For the remaining videos in the original training set, we randomly sample 7200 videos build the seen test set. The speakers also appear in the training set, but the specific utterances do not. Our unseen test set consists of 7200 videos randomly chosen from the original test set. All of the videos in the VoxCeleb2 dataset form our validation
Lip Reading Sentences 2&3 (Afouras et al. ca) LRS2 and LRS3 datasets contain 144k and 15lk video clips from TV programs. To evaluate the generalization of models, we train them on the VoxCeleb2 dataset and test them on cross-domain LRS2 and LRS3 datasets without fine-tuning. For each dataset, the test set consists of 1200 randomly selected videos from the original test set.
Table 1: Numbers of videos and mixtures for four test sets.
VonCeleh2 unseen set
7200
3500
2400
1800
1440
VonCele22 sens set
7300
3600
2400
1800
1440
LRS2
1200
600
400
300
240
LRS3
1200
600
400
300
240
Test Benchmark. During the test. thoroughly evaluate the models' performance mixtures with various numbers speakers, i.c., from mixtures with speakers to 5 speakers. Meanwhile, to eliminate the uncertainty of results caused by testing randomly created mixtures each time, construct fixed test sets mixtures and perform all experiments on these predetermined mixtures.
For each dataset, randomly mix the test videos to build 2-mix, 3-mix, 4-mix, and 5-mix test sets without any subjective considerations. Each video appears only once in each mixture set. To be more explicit, we list the number of videos and each kind of mixture for all test sets in Tab. 1

---

Published as a conference paper at ICLR 2023
4.2 IMPLEMENTATION DETAILS
Data Process. Following the previous setting (Gao & Grauman 2021). we randomly cut 2.55- second long clips from videos sampled at 25fps and the corresponding audios sampled at 16kHz as the training pairs. For all utterances that make up mixture, we first normalize the energy of each one to the same, which corresponds to the same loudness for each utterance. Then we add up all normalized speech to obtain the mixture. STFT is conducted on the mixture waveform using a Hann window length of 400, a hop size of 160, and an FFT window size of 512 to output the complex spectrogram of dimension x 257 X 256, which taken as the input to the U-Net encoder. A randomly selected frame from the video is rescaled to 224 x 224 and sent to the face analysis net- work. The input to the lip reading network is 64 consecutive frames of cropped gray mouth regions of dimension 88 88. We adopt the official implementation of 2D face landmark detection (Bulat & Tzimiropoulos 2017) to detect the mouth landmarks and crop the mouth regions.
Training Setting. To achieve great separation results for mixtures containing different numbers of speakers, we separate all types of mixtures into individual speech simultaneously. In each batch, to 2:1:1:1, and the total number of speakers is 256. As for selecting utterances to create mixtures, some methods (Gao & Grauman 2021 Rahimi et al. 2022) perform put-back random sampling. Such an approach results in some utterances being sampled multiple times while others are ignored, in which case the model might overfit some samples and underfit others. Instead, the beginning of each training epoch, we shuffle all samples randomly and pick them one by one to ensure no duplication or omission.
the ratio of 2-mix, 3-mix, 4-mix, and 5-mix numbers
Optimization. Empirically, we adopt the Adam optimizer train the network with a weight decay of le-4 and learning rate of le-4. We drop the learning rate by a factor of 0.1 after epochs 12 and 15, and train models for 19 epochs, when the loss almost reaches a plain. Tab. g displays experimental results of setting different lambda values, and we finally set it to 0.5.
Evaluation. Following previous methods (Afouras et al. 2018b Ephrrat et al. 2018 Gao & Grau- man 2021: Rahimi et al.] 2022). we adopt the standard blind source separation metric Signal-to- Distortion-Rate (SDR) (Vincent et al. (006). which measures the ratio between the energy of the target signal and that of the errors. To further assess the speech quality and intelligibility, also employ the Perceptual Evaluation of Speech Quality (PESQ) (Rix et al. 200T) metric.
4.3 COMPARISON WITH STATE-OF-THE-AA METHODS
Tab. 2 and compare our method other open-source ofte--of-thearr models on the test sets. All experiments follow the same test and evaluation protocols. We choose two types of methods for comparison: audio-only methods and audio-visual methods.
Audio-Only Methods:
U-Ner-AO. By removing the face net and the lip net from the adopted basic audio-visual speech separator and making corresponding adaptations for the U-Net, we obtain audio-only speech separator and train it with permutation-invaritt
loss.
VoiceFilter (Wang et al. 2018). This method uses the speech mixture and reference utterance to extract the target speech from the mixture. Note that the reference audio is piece of randomly sampled utterance of the target speaker different from the target speech.
Conv-TasNet (Luo Mesgarani 2019 This method is widely adopted in audio-only speech sep- aration practices. It adopts fully- convoluuoolll audio separation network, which takes the time- domain speech mixture as input and outputs all speech components simultaneously.
Audio-Visual Methods:
LAVSE (Chuang et al., 2020). To reduce processing costs, this work employs a lightweight but efficient framework, where lip encoder extracts the lip motion feature the synchronization signal for target speech extraction.
VisualVoice (Gao & Grauman 2021). It contains the same visual networks and U-Net our basic audio-visual separator, except the U-Net simply concatenates the audio and visual features, while we perform AV-Fusion obtain enhanced audio features and achieve better results VoxCeleb2. Be-

---

Published as a conference paper at ICLR 2023
Table 2: Results on VoxCeleb2 dataset. The metries are the average of all speakers for each test set. All methods are trained in the proposed spetk-sp setting. BFRNet achioves the best performance among all methods for all types of mixtures. (e.g. we achieve 11.06 dB SDR on VoxCeleb2 unseen 2-mix set.) Besides, we list the overall (O.A.) performance of all methods for convenient comparison, which is the average performance of each speaker in all kinds of mixtures. It is worth mentioning that each audio-visual method combined with the FRNet results a large improvement.
Method
Spk.
Unseen
O.A.
STIE (IR)
Seen
PRSQ
Austustily
U-Net-O
0,36
-2.93
1,01
9.46
4.57
3.73
1.75
2.05
9:35
4.76
3.38
1.79
2.07
Audio Visull
LAVSE
5.38
1.55
0.6
1.05
2.17
1.45
1.29
1.64
5.54
1.45
0.72
2.31
0.59
2.20
1.45
1.24
1.44
LAVSE + FRNNs
8445
4.3
2.35
0.45
3.05
2.57
2.00
1.70
1.58
2.00
8.30
4.66
2.16
0.34
3.04
2.59
2.07
1.37
1.56
2.00
ViuuiNinge
R97
4.59
2,05
0.25
3.96
2.64
2.12
1882
1.62
2.05
8.61
3.96
1.39
-0.4
3.39
2.61
2.05
1.7g
1.54
1.98
Vioue -I HRNee
10.78
7.08
4.63
2.76
6.30
286
2.43
2.15
1.91
2.33
10
6.95
4.57
2,40
6.17
287
2.41
2,04
1.86
2:31
178
773
6.14
7.15
177
1:51
2003
991
1:70
2.17
4011
4:31
770
1.13
1.301
1:1
200
NONT FRNe:
10.72
6.74
3.82
1.44
5.65
2.76
2.27
1.92
1.65
2.15
11:05
6.97
3.98
1.55
5.89
2.79
2.30
1.95
1.68
2.18
1010
391
3,36
105
5.05
2.72
2.23
190
1.65
2.12
1000
3,44
255
0.28
4.57
2.72
2.17
1.3
1.36
2.07
BIFENN (ouEs)
11.06
7.88
333
3.36
6.73
139
11.27
7.48
2356
0.63
2.16
191
2.36
Table 3: Results on LRS 2&3 datasets. To validate the generalization of separation models, they are trained on VoxCeleb2 and validated on LRS2/LRS3 without fine-tuning. BFRNet outperforms all base methods that do not integrate FRNet. Although some methods combined with FRNet achieve the hest performance in certain metries, it still proves the effectiveness of FRNet. Similarly, we give the overall (O.A.) performance on all kinds of mixtures for each test set for direct comparison.
Spk.
U:No.40
0.25
-2.65
-4.5
-5.57
-3.01
1,14
0.97
0.90
0.19
-2.70
-434
-5.45
-3.08
1,40
1,03
0.85
0.76
Vileeilerr
4.10
3:00
4.55
278
1,95
1.92
9.th
1.11
-110
1,36
1.35
T:1
1.30
1.29
111.5
4.55
1.35
4.48
7.54
1:55
1.75
1.15
1.95
Andin Voual
LAYSE
5.74
1.64
-0.44
2.15
1.18
2.04
1.3%
1.30
1.17
1.51
6.10
1.50
0.39
1.995
1.39
2,05
1.50
1.25
1.30
1.45
LAVSE ERNer
8,85
4.58
1.92
2.05
3.82
2.45
1,50
1.55
1.37
1.83
9.46
4.78
2.11
0.04
4.10
2.53
1.91
1.58
1.35
1.84
10.31
5:55
2.62
0.56
4.76
2.61
202
1.65
1:47
1.95
11:21
5.00
3.20
1.04
5.38
2.72
2.09
1.75
1:47
2.00
Viuce FRN
11.70
7.83
5.00
2.90
6.86
2.79
2.35
1.99
1.73
2.22
12.66
8.47
5.73
3.54
7.60
2.93
2.45
2.09
1.79
2.32
VoNY
1003
4.00
1:15
1123
3.50
2.55
1.3
1.35
1.29
1.84
1004
5.57
1.41
-0.5
4.14
1.64
1.77
1.31
1.20
1.84
WHiT FRNer
11.99
6.90
-0.0
2.00
1.67
7.10
3.35
0.70
3.75
2.17
1.30
TETRNG
935
-110
1:91
1.30
1.25
5.00
7.98
700
1.0
1:30
1.92
BERNe (OuES)
11.8 7.3
0.36
135
1.3
5.37
3.38
7.52
242
2.0
sides, Visual contains an additional vocal analysis network designed for only 2-mix separation, so we remove the vocal analysis network to adapt to our multi-speaker setting.
2022) The newly pro- posed method uses a mark based graph convo- lutional network (Yan et al. 2018) to capture the fa- cial motion cues. It adopts an AV spectro-temporal transformer for target speech separation.
VoVIT (Montesinos et al.,
3mix
Smi
SON 1881
DeBaSe. In order to offset the impact of increased model capacity brought by the additional FRNet, we compare the results of a deeper basic audio-visual speech separator (DeBaSe) to that of BFRNet, which has more layers in the encoder and has almost the same parameter counts as BFRNet.
Besides, we combine all audio-visual methods apart from DeBaSe with the proposed FRNett to verify its generality. We call these methods *#+FR'.
Visualization of SDR for each method VoxCeleb2 unseen test set. The in- crements on cach bar denote the improvement af- ter combining FRNet with base methods. 'BaSe' is our basic audio-visual separator.
Figure 3:
As seen in Tab. 2 Tab. 3 and Fig. 3 BFRNet achieves the best results in all test sets compared other baseline methods with at least dB SDR ad-
vantage. Besides, the FRNet combined with any audio-visue visual method can significantly improve per- formance. For instance, VisualVoice (Gao & Grauman (02TT) combined with FRNet receives a 1.67 dB gain of SDR on the VoxCeleb2 unseen 2-mix test set. It worth noting that there is still large gap between DeBaSe and BFRNet, although their parameter amounts are nearly equal. It means that just expanding the model capacity does not necessarily lead to significant performance gain. In contrast, BFRNet focuses on the main issues of the separation task, thus significantly enhancing the separation results. The difficulty of separation grows with the increase of speaker number in the mixture, resulting in lower metrics. For the VoxCeleb2 dataset, although the speakers in the seen test set appear in the training set, the results of the seen set are not certainly higher than that of the unseen set. We argue that the speakers' voiceprint features is not very critical for speech separation.
8

---

Published as a conference paper at ICLR 2023
4.4 ABLATION STUDIES
We conduct ablation studies and report average results on VoxCeleb2 unseen and seen test sets.
The effect visual knowledge. We remove the
Table 4: Experiments on effect visual clues.
Face Net and Lip Net separately while remaining other parts. Tab. 4 shows that modified models con- sistently yield lower results than BFRNet. However, as lip motion is more related to speech, it plays a principal role in speech separation, while the static face image only serves as extra information.
Medsles
SDK (dil)
PHSO
1.28
7.8
306
1.94
Table 5: Ablation study on FRNet modules.
Medules
SDR (dB)
PESO
The effect of FRNet modules. To validate the cessity of each module, we conduct experiments in
10,05
5.79
3.12
1.15
2774
2.34
1.55
1.70
10.3
6.65
4.51
2.48
288
3.58
2.10
1.66
10.3
6.30
3.75
1.81
288
2.37
2,65
1.82
11.17
7.48
5.00
3.06
288
2,48
2.18
1.94
Tab. 5 that remove individual modules. When the
Filter net is removed, the Recovery net adopts Mi to
Table 6: Ablation study module design.
replace M. When the Recovery net is removed, the output of Filter net M, is considered the final output.
Exps
PESO
Fllter
Module design FRNet. Tab Texplorss deformed structures of FRNet. For Filter Net, we filter voice si-
11.17
2.08
Table 7: Ablation study on the layers L.
noise from M, with itself as a clue instead multancous visual signal V. We name this ablation 'filter-sa'. As to Recovery Net, we replace the clean mask M, with Mj as the clue to extract the missing voices from M. and we name 'recovery-noise"
SDR (dB)
PESQ
Layers
1,86
11.17
7.48
5.01
3.06
2.88
2.48
2.18
1.91
11.20
7.52
5.17
3,31
2,88
2.50
2,18
1.95
Layers of the FRNet. Both the Filter and Recovery Nets are composed of T. basic layers. We here study the impact of T.. As seen in Tab. 7 there is great improvement for the network with T. compared to T= 1. When the number of layers T. increases to 3, the performance improvement is insignificant. To balance performance and efficiency, we adopt the 2-layer FRNet.
4.5 QUALITATIVE RESULTS
We visualize the intensity of ground truth (GT) spec- trograms and predictions by models in Fig 4 The first row presents the ground truth, the separation sult of the basic separator, and the result of BFRNet, respectively. The second row shows the separation results of other approaches. The results demonstrate that the separation results of baseline methods are subject to the two issues we claim, including the re- sults the proposed method without the FRNet. We use black boxes to highlight the missing parts com- pared GT, and red boxes to indicate the noisy parts from other speakers. As seen in Fig. 4 the two prob- lems are greatly suppressed after utilizing FRNet.
BFRNN
Figure 4: Spectrogram visualization of ground truth and predictions.
CONCLUSION
In this paper, we have focused on the multi-speaker audio-visual speech separation task. We are the first propose separating mixtures with variable number of speakers simultaneously during train- ing, and we also provide standard test benchmark for fair comparison. There are two significant problems for speech separation, especially in the multi-speaker setting: part of the voice missing in the separated speech; the separated speech may still be mixed with others' voices. To deal with this, we propose a Filter and Recovery network to solve these two problems. The filter module filters out other people's 's voices, and the recovery module compensates for their missing voices. We con- duct various experiments to demonstrate the effectiveness of this module, and its addition to other audio-visual speech separation methods has led considerable improvements.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENTS
This work is supported by the National Key R&D Program of China (No. 20222D0160900,, the National Natural Science Foundation of China (No. 62076119, No. 61921006), the Fundamental Research Funds for the Central Universities (No. 020214380091), and the Collaborative Innovation Center of Novel Software Technology and Industrialization.
REFERENCES
Triantafyllos Afouras, Joon Son Chung, Andrew Senior, Oriol Vinyals, and Andrew Zisserman. Deep audio- visual speech recognition. IEEE transactions on pattern analysis and machine intel- ligence, 2018a.
Triantafyllos Afouras, Joon Son Chung, and Andrew Zisserman. The conversation: Deep audio- visual speech enhancement. arXiv preprint arXiv:1804.04122 2018b.
Triantafyllos Afouras, Joon Son Chung, and Andrew Zisserman. Lrs3-ted: large-scale dataset for visual speech recognition. arXiv preprint
2018c 8c.
Triantafyllos Afouras, Joon Son Chung, and Andrew Zisserman. My lips are concealed: Audio- visual speech enhancement through obstructions. Proc. Interspeech 2019, pp. 1295-4295 2019.
Triantafyllos Afouras, Andrew Owens, Joon Son Chung, and Andrew Zisserman. Self-supervised learning of audio-visual objects from video. In European Conference on Computer Vision, pp. 208-224. Springer, 2020.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint 1607.06600 2016.
Adrian Bulat and Georgios Tzimiropoulos. How far are we from solving the 2d & 3d face align- ment problem? (and a dataset of 230,000 3d facial landmarks). In International Conference Computer Vision, 2017.
William Chan, Navdcep Jaitly, Quoc V Le, and Oriol Vinyals. Listen, attend and spell. arXiv preprint arXiv: 02..880.2220 0/2 2015.
P Chen, Y Zhang, M Tan, H Xiao, D Huang, and C Gan. Generating visually aligned sound from videos. IEEE Transactions on Image Processing: a Publication of the IEEE Signal Processing Society, 2020.
Zhuo Chen, Yi Luo, and Nima Mesgarani. Deep attractor network for single-microphone speaker separation. 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 246-250. IEEE, 2017.
Shang-Yi Yi Chuang, Yu Tsao, Chen-Chou Lo, and Hsin-Min Wang. Lite audio-visun visual speech enhance- ment. arXiv preprint arXiv:2005. 1769, 2020.
Joon Son Chung. Arsha Nagrani, and Andrew Zisserman. Voxceleb2: Deep speaker recognition. arXiv preprint arXiv: arXiv:/806.000 05622, 2018
Soo-Whan Chung, Soyeon Choe, Joon Son Chung, and Hong-Goo Kang. Facefilter: Audio-visual speech separation using still images. Proc. Interspeech 2020, pp. 3481-3485, 2020.
Ariel Ephrat, Inbar Mosseri, Oran Lang, Tali Dekel, Kevin Wilson, Avinatan Hassidim, William T Freeman, and Michael Rubinstein. Looking to listen at the cocktail party: a speaker-independent audio-visual model for speech separation. ACM Transactions on Graphics (TOG), 37(4):1-11, 2018.
Aviv Gabbay, Asaph Shamir, and Shmuel Peleg. Visual speech enhancement. Proc. Interspeech 2018, pp. 1170-1172 1174, 2018.
Chuang Gan, Deng Huang. Hang Zhao, Joshua B Tenenbaum, and Antonio Torralba. Music gesture for visual sound separation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10478-10487, 2020.
10

---

Published as a conference paper at ICLR 2023
Ruohan Gao and Kristen Grauman. Co-separating sounds of visual objects. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.
2019.
Ruohan Gao and Kristen Grauman. Visualvoice: Audio-visual speech separation with cross- modal consistency. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15490-15500. IEEE, 2021.
Ruohan Gao, Rogerio Feris, and Kristen Grauman. Learning to separate object sounds by watching unlabeled video. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 35-53, 2018.
Mandar Gogate, Ahsan Adeel, Ricard Marxer, Jon Barker, and Amir Hussain. Dnn driven speaker in- dependent audio-visual mask estimation for speech separation. arXiv preprint arXiv: 00060, 2018.
Sindhu B Hegde, KR Prajwal, Rudrabha Mukhopadhyay, Vinay P Namboodiri, and CV Jawahar. Visual speech enhancement without a real visual stream. Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pp. 1926-1935, 2021.
L Ashok Kumar, D Karthika Renuka, S Lovelyn Rose, Made Wartana, et al. Deep learning based assistive technology on audio visual speech recognition for hearing impaired. International Jour- nal of Cognitive Computing in Engineering, 3:24-30, 30, 2022.
Jonathan Le Roux, Scott Wisdom, Hakan Erdogan, and John R Hershey. Sdr-half-baked well done? In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 626-630. IEEE, 2019.
Jiyoung Lee, Soo-Whan Chung, Sunok Kim, Hong-Goo Kang, and Kwanghoon Sohn. Looking into your speech: Learning cross-modal affinity for audio-visual speech separation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1336-1345. 2021.
Shuo Liu, Gil Keren, and Björn Schuller. Single-channel speech separation with auxiliary speaker embeddings. arXiv preprint 1906.
2019.
Rui Lu, Zhiyao Duan, and Changshui Zhang. Listen and look: Audio-visual matching assisted speech source separation. IEEE Signal Processing Letters, 25(9):1315-131, 2018.
Yi Luo and Nima Mesgarani. Conv-tasnet: Surpassing ideal frequency magnitude masking for speech separation. IEEE/ACM transactions audio, speech, and language processing, 27(8): 1256-1266, 2019.
Yi Luo, Zhuo Chen, and Nima Mesgarani. Speaker-independent speech separation with deep at- tractor network. IEEE/ACM Transactions Audio, Speech, and Language Processing, 26(4): 787-796, 2018.
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient cnn architecture design. In Proceedings of the European conference on computer vision (ECCV), pp. 116-131, 2018.
Pingchuan Ma, Brais Martinez, Stavros Petridis, and Maja Pantic. Towards practical lipreading with distilled and efficient models. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1608-7613 IEEE, 2021.
Naoki Makishima, Mana Ihori, Akihiko Takashima, Tomohiro Tanaka, Shota Orihashi, and Ryo Masumura. Audio-visual speech separation using cross-modal correspondence loss. In ICASSP 2021-2021 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 6673-6677. IEEE, 2021.
Juan Montesinos, Venkatesh S Kadandale, and Gloria Haro. Vovit: Low latency graph-based audio-visual voice separation transformer. arXiv preprint 220.::20030049999 2022.
Andrew Owens and Alexei A Efros. Audio-visual scene analysis with self-supervised multisensory features. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 631-648, 2018.
11

---

Published as a conference paper at ICLR 2023
Zexu Pan, Ruijie Tao, Chenglin Xu, and Haizhou Li. Selective listening by synchronizing speech with lips. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:1650-1664, 2022.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and lyy Sutskever. Robust speech recognition via large-scale weak supervision.
Akam Rahimi, Triantafyllos Afouras, and Andrew Zisserman. Reading to listen at the cocktail party: Multi-modal speech separation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10493-10502, 2022.
Karthik Ramesh, Chao Xing, Wupeng Wang, Dong Wang, and Xiao Chen. Vset: A multimodal transformer for visual speech enhancement. In ICASSP 2021-2021 IEEE International Confer- ence on Acoustics, Speech and Signal Processing (ICASSP), pp. 6658-6662. IEEE, 2021.
Antony Rix, John G Beerends, Michael Hollier, and Andries P Hekstra. Perceptual evaluation of speech quality (pesq)-a new method for speech quality assessment of telephone networks and codecs. In 2001 IEEE international conference on acoustics, speech, and signal processing. Proceedings (Car. No. 01CH37221), volume 2. pp. 749-752. IEEE, 2001.
Ziqiang Shi, Rujie Liu, and Jiqing Han. Lafurca: Iterative refined speech separation based on context-aware dual-path parallel bi-lstm. arXiv preprint arXiv:2001.0899) 2020.
Bastiaan Tamm, Helena Balabin, Rik Vandenberghe, et al. Pre-trained speech representations as feature extractors for speech quality assessment in online conferencing applications. Proceedings Interspeech 2022, pp. 4083-4087, 2022.
Thanh-Dat Truong, Chi Nhan Duong, Hoang Anh Pham, Bhiksha Raj, Ngan Le, Khoa Luu, et al. The right talk: An audio-visual transformer approach. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pp. 105-1114, 2021.
Efthymios Tzinis, Scott Wisdom, Aren Jansen, Shawn Hershey, Tal Remez, Daniel PW Ellis, and John R Hershey. Into the wild with audioscope: Unsupervised audio-visual separation of on- screen sounds. arXiv preprint arXiv:2011.01143, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- fion processing systems, 30, 2017.
Emmanuel Vincent, Rémi Gribonval, and Cédric Févotte. Performance measurement in blind audio source separation. IEEE transactions on audio, speech, and language processing, 14(4):1462- 1469, 2006.
Quan Wang, Hannah Muckenhirn, Kevin Wilson, Prashant Sridhar, Zelin Wu, John Hershey, Rif A Saurous, Ron Weiss, Ye Jia, and Ignacio Lopez Moreno. Voicefilter: Targeted voice separation by speaker-conditioned spectrogram masking. arXiv preprint arXiv:1810.04822 2018.
Junwen Xiong, Peng Zhang, Lei Xie, Wei Huang, Yufei Zha, and Yanning Zhang. Audio-visus speech separation based on joint feature representation with cross-modal attention. arXiv preprint arXiv:2203.02655. 2022.
Xudong Xu, Bo Dai, and Dahua Lin. Recursive visual sound separation using minus-plut net. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 882-891, 2019.
Sijie Yan, Yuanjun Xiong, and Dahua Lin. Spatial temporal graph convolutional networks for sreleton-based action recognition. In Thirty-second AAAI conference on artificial intelligence, 2018.
Zengwei Yao, Wenjie Pei, Fanglin Chen, Guangming Lu, and David Zhang. Stepwise-refining speech separation network via fine-grained encoding in high-order latent domain. IEEE/ACM Transactions Audio, Speech, and Language Processing, 30:378-393, 2022.
12

---

Published as a conference paper at ICLR 2023
Dong Yu, Morten Kolbæk, Zheng-Hur Tan, and Jesper Jensen. Permutation invariant training of deep models for speaker-independemt multi-talker speech separation. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 241-245. IEEE, 2017.
Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl Vondrick, Josh McDermott, and Antonio Torralba. The sound of pixels. In Proceedings of the European conference on computer vision (ECCV), pp. 570-586, 2018.
13

---

Published as a conference paper at ICLR 2023
A APPENDIX
We provide an additional ablation study and more visualization examples in the appendix.
Ablation study À in loss function. We conduct experiments to explore the impact of different train- ing in Eq. 13 on the results. A smaller À means higher training loss weight of FRNet. On the con- trary, larger À implies lower weight FRNet in the whole model. Tab. displays the average results of Vox unseen and seen test sets. As seen in the table, attaching a higher weight the train-
Table 8: Ablation Study A.
SDR (dB)
PESQ
0.2
11.13
7.47
5,01
3,08
2,86
2.47
2.19
1.95
0.5
11.17
7.48
5.01
3.06
2,83
2.48
2.18
1.94
0.8
7.20
ing of FRNet (λ is 0.2) yields similar results to giving an equal weight (A is 0.5). Nevertheless, lower loss weight of FRNet (A is 0.8) results non-negligible performance drop, which proves the necessity of FRNet.
Spectrogram visualization BFRNet. We visualize the intensity of spectrograms of ground truth (GT) and predictions by each network of BFRNet in Fig. 5 The red boxes indicate the noisy part generated by the basic separator and then suppressed by Filter Net. The black boxes denote the missing part yielded by the basic separator and further recovered by Recovery Net. The results demonstrate the effects Filter Net and Recovery Net.
Basic Separator
GT
Basic Seperator
Filter Net
Recover Net.
Filter Net
Recovery Net
(a) Example
(b) Example
GT
Basic Separator
Basic Soparator
Filter Net
Recovery Not
Filter Net
Recovery Net
(c) Example
(d) Example
Figure 5: Visualization of the intensity spectrogram: ground truth (GT), outputs of the basic separator, outputs of the Filter Net, and outputs of the Recovery Net.
14


Reviews:
[{"review_id": "go9iHFRbnm7", "reviewer": "Reviewer_X56m", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies masking-based audio-visual source separation, which predicts a complex spectral mask for the audio mixture for each speaker conditioning on the mixture speech and the video of the target speaker. The authors proposed BFRNet, which is composed of a audio-visual source separation model that predicts a mask for each speaker, and a filter-and-recovery network (FRNet) that refines predicted masks to a) remove non-target residual speech and b) to recover target speech removed from the initially predicted mask.", "summary_of_the_review": "Experimental results are strong compared to the prior work. The main novelty is a module for mask refinement, which can improve several existing audio-visual speech separation models as demonstrated by the authors. More controlled experiments are required to justify the gain does not simply result from the increase in the number of parameters.", "strength_and_weaknesses": "Strengths\n\n- BFRNet yields better performance on VoxCeleb2, LRS2, LRS3 compared to prior works\n- The mask refinement module, FRNet, is complementary to other masked-based audio-visual speech separation models according to the experiments in Table 2 and 3.\n- Ablation studies confirm that both the filter and the recovery module contribute to the improvement. It also shows video facilitates the filter network, and using the cleaned mask to the recovery module improves.\n\nWeaknesses\n- While the empirical results are strong, this paper can be improved by providing more explanations of why the FRNet further improves the base audio-visual speech separation module. \n  - Why does the filter network remove the non-target speakers’ voice better than the base separation module? The filter network only takes predicted mask and video features as the input. It does not even take the mixture audio as the input. It is surprising that it can tell where the noise is by just looking at a mask.\n  - Similarly, why does the recovery network learn what to keep better than the base audio-visual source separation network?\n  - It is possible that adding FRNet improves because now the entire model is bigger than the one without. To justify the FRNet carries benefits other than increasing the model capacity, the authors are recommended to compare with a stronger base model (e.g., Basic Audio-Visual Speech Separator) that have a deeper encoder/decoder where the parameter counts match roughly that of the BFRNet.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "bQcHA5Ntfgp", "reviewer": "Reviewer_VNdU", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work presents an audio-visual speaker separation method suitable for multi-speaker scenarios. An audio-visual speaker separator is first introduced which is followed by a filter-recovery network which aims to further improve the separation quality. The latter can also be combined with other separator networks and it is shown that it can improve their performance as well. The proposed approach is tested on standard benchmarks and it  outperforms other state-of-the-art methods.  ", "summary_of_the_review": "Overall, this is an interesting contribution but there are several weakness as explained above.", "strength_and_weaknesses": "Strengths\n\nSeveral results and an ablation study is presented which convincingly show that the proposed approach outperforms other existing approaches.\n\nThe proposed FRNet can improve the performance of other speech separation approaches and this is a nice contribution.\n\nWeaknesses\n\nFirst of all, writing can be improved. Although the text is understandable there are several errors/typos. More details in the next section.\n\nThe impact of facenet/lipnet is missing in the ablation study. It would be good to show how important each of them is. Since most of the information is contained in the lip movements the contribution of facenet might be small.\n\nThe authors emphasise that one of the main contributions is that the proposed method can separate mixtures with a variable number of speakers simultaneously during training. Isn’t this something the existing approaches can already do? And also the number of speakers needs to be known in advance. It’s not clear why this is an important contribution.\n\nIt is not explained why the authors chose a two-step approach instead of an end-to-end approach, i.e., why not integrating the filter and recovery networks in the speech separation model and optimise them jointly?\n\nThe proposed audio-visual speech separation network (without the FRNet) is very similar to other existing audio-visual approaches. Why is this model better? This is also not clear.\n\nSome details are not explained, e.g. the lambda coefficient in eq. 13 is set to 0.5, how is this value chosen? Why is training performed for 19 epochs? Why 2.55 seconds are used for training? It seems these numbers have been chosen via an optimisation stage but it is not explained how this was performed.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "44sTgjJOTzY", "reviewer": "Reviewer_t4UV", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper points out that there are two prominent issues in complex multi-speaker separation results: 1) There exist some noisy voice pieces belonging to other speakers; 2) Part of the target speech is missing. A Filter-Recovery Network (FRNet) is hence proposed to solve these problems. The authors also emphasize that their single model can separate voices for a variable number of speakers, which is simply achieved by proper training data setting (i.e., including mixtures with a different number of speakers). Overall, the model design is quite interesting with a good performance improvement.", "summary_of_the_review": "The prosed method is motivated by two observed problems in the multi-speaker separation. FRNet can also be used as a post-processing module to improve the performance of different audio-visual separation frameworks is a plus. Overall, the paper is well written with good experimental results.", "strength_and_weaknesses": "This paper first points out that there are two issues in complex multi-speaker separation and solve these two issues with the proposed Filter-Recovery Network (FRNet). The model structure of FRNet is reasonable, but there are still some unclear parts that need to be addressed:\n\n1. The loss function used to train the proposed model is only based on e.q. 13 without any constraints, how do you guarantee the Filter Net and Recovery Net work properly as their original goals (removing noise voices and extracting missing voices, respectively)?\n2. Some audio samples should be released in the supplementary material, especially the separated result at each stage: i.e., a) after Basic Audio-Visual Speech Separator, b) after Filter Net and c) after Recovery Network. This can also somewhat solve my first concern.\n3. The inputs of the Filter Net are Vi and Mi, why the mixture X doesn’t need to be fed into the model?\n4. For the experimental parts, how do you calculate the scores (SDR and PESQ) if there are multiple output-separated speeches?\n\nThe proposed method to separate voices for a variable number of speakers with a single model is not very novel, to my understanding, it is mainly based on a proper setting of the training set.\n\ncomparison\nTypo:\n1.\tPage 2, Specifically, Specifically, -> Specifically.\n2.\tPage 6, k-th speaker -> i-th speaker.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster