RAVSS: Robust Audio-Visual Speech Separation in Multi-Speaker Scenarios with Missing Visual Cues

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 10:23:55

Abstract: While existing Audio-Visual Speech Separation (AVSS) methods primarily concentrate on the audio-visual fusion strategy for two-speaker separation, they demonstrate a severe performance drop in the multi-speaker separation scenarios. Typically, AVSS methods employ guiding videos to sequentially isolate individual speakers from the given audio mixture, resulting in notable missing and noisy parts across various segments of the separated speech. In this study, we propose a simultaneous multi-speaker separation framework that can facilitate the concurrent separation of multiple speakers within a singular process. We introduce speaker-wise interactions to establish distinctions and correlations among speakers. Experimental results on the VoxCeleb2 and LRS3 datasets demonstrate that our method achieves state-of-the-art performance in separating mixtures with 2, 3, 4, and 5 speakers, respectively. Additionally, our model can utilize speakers with complete audio-visual information to mitigate other visual-deficient speakers, thereby enhancing its resilience to missing visual cues. We also conduct experiments where visual information for specific speakers is entirely absent or visual frames are partially missing. The results demonstrate that our model consistently outperforms others, exhibiting the smallest performance drop across all settings involving 2, 3, 4, and 5 speakers.

Main paper: 
 introduction: 
 Humans can selectively concentrate on desired sounds in complex acoustic environments with mixed speech signals from multiple speakers and diverse background noises, which is termed the "Cocktail Party Problem" [4, 8, 15] . Audio-Visual Speech Separation (AVSS) is a task that aims to separate and isolate individual speeches from the overlapped speech mixtures along with corresponding visual cues like speaker's face [14] or lip-movements [31, 32] . AVSS serves as a front-end process for tasks like speech recognition and speaker diarization, finding applications in hearing aids devices [41] and teleconferencing systems [6, 35] .Recent advancements in AVSS methods [9, 14, 21, 22, 33] have primarily focused on enhancing audio-visual fusion strategies for the two-speaker separation scenario. They typically utilize visual cues as guidance to sequentially separate each corresponding speaker's speech from the same mixture. While this approach yields satisfactory results in separating two speakers without significant performance degradation, it faces challenges when dealing with three or more speakers. In such scenarios, the conventional one-byone speaker extraction method often leads to numerous instances of missing and noisy segments [7] within each separated speech stream, as depicted in Figure 1(a) . As a result, this separation approach exhibits a significant performance drop, as illustrated in Table 1 , indicating its limited capability to differentiate and capture interactions among multiple speakers comprehensively. Several methods [1, 3, 13] have also been developed to address multispeaker scenarios. L2L [13] employs customized models for each mixture type in each one-by-one speaker extraction process. And Afouras et al. [1, 3] enhance the one-by-one speaker extraction process with specifically designed modules or pre-enrolled speaker embeddings. Furthermore, BFRNet [7] proposes a filter-recovery network as a further post-processing refinement step. However, these methods also fail to address the separation of multiple speakers in a unified way. As illustrated in Figure 1 (b), we propose a robust multi-speaker audio-visual separation framework that can separate multi-speakers collaboratively in a single separation process. To achieve this, we suggest two speaker-specific interaction forms: speaker-wise audio-only and speaker-wise audio-visual interaction. The speaker-wise audio-only interaction assesses the relationships of distinctions and connections within each speaker. It effectively mitigates the relatively notable missing segments and eliminates irrelevant parts for each speaker, thereby achieving a more accurate separation of multiple speakers. The speaker-wise audio-visual interaction can further enhance the inter-speaker disparity by providing more evident visual distinctions between speakers. It offers a more detailed refinement to rectify or remove segments that are initially similar between speakers. Incorporating these two speakerspecific interactions enhances the model's robustness, particularly when the number of speakers in the audio mixture increases.In real-world scenarios, various factors such as a speaker moving off-screen or the camera being turned off can result in missing video information [3, 5, 11, 16, 20, 42] . We additionally evaluate the robustness of our model in handling missing video data, where visual information for specific speakers is absent or where visual frames are partially missing. Firstly, we expand the scope to situations where only 𝑃 visual cues for 𝑁 -speaker audio mixture are available, with 𝑁 ≥ 𝑃. The extracted speech features are then split into two components: 𝑃 visually-guided speech features and 𝑁 -𝑃 non-visually-guided speech features. Existing AVSS methods [9, 14, 21, 22, 33] typically aim to separate 𝑁 individual speakers from the 𝑁 -mixed audio mixture using the corresponding 𝑁 visual cues. However, in this scenario, these existing AVSS methods encounter the challenge of accurately separating the target speaker without the guidance provided by the corresponding video. In contrast, our separation framework facilitates a comprehensive interaction between visually-guided and non-visually-guided speech features, where the latter serves as the many "missing" parts for frames and speakers as illustrated in Figure 1(b) . Specifically, our model effectively enhances the discriminability of separated non-visually-guided speech features by leveraging the complete audio-visual information of other speakers during the separation process. We also consider the scenario where some video frames are partially missing. We randomly select frames for each corresponding video in 𝑁 speakers with some rates and set those frames to zero, leading to some frame-wise visual information loss. Our model can efficiently compensate for the loss by utilizing both the speech content-related and the speaker-related contextual information. Overall, our model enables a more robust multi-speaker audio-visual separation process, addressing the limitations of the conventional methods.To sum up, our contributions are as follows:• We propose a robust multi-speaker audio-visual separation framework that demonstrates superior performance, particularly as the number of speakers in the mixture increases. • We introduce more challenging scenarios where visual information for specific speakers is absent or visual frames for every speaker are partially missing. Our model achieves the smallest performance gap, further demonstrating its robustness. 
related work 2.1 audio-only speech separation: 
 The methods for audio-only speech separation [12, 19, 23-26, 39, 43, 46] can be generally divided into T-domain methods [23] [24] [25] 39] and TF-domain methods [43, 46] . Conv-TasNet [24] is an efficient and low-latency end-to-end fully-convolutional time-domain method. DPRNN [23] incorporates dual-path recurrent neural networks to capture long-range information that may be lacking in 1D convolutional networks. Sepformer [22] achieves improved separation performance by leveraging the transformer architecture. However, traditional time-domain methods often face challenges in reverberant conditions due to the absence of explicit frequency-domain modeling. To address this, recent studies have focused on TF-domain approaches [33, 43, 46] . TFPSNet [46] leverages three-path scanning, including T-path, F-path, and TF-path, to extract more comprehensive information. TF-GridNet [43] introduces sub-band and full-band attention LSTM blocks to enhance performance in TFdomain separation tasks.Visual information is naturally aligned with audio in videos, which has become the dominant mode of communication on contemporary social platforms. Many studies [14, 22, 30, 31, 33] have consistently demonstrated that utilizing face-related features, such as still face images and lip movements, can significantly enhance the separation of speech from a mixed audio signal. Convolutional neural networks [7, 14, 27] are widely utilized for effectively handling time-frequency speech features. More recently, the transformer architecture [22, 30, 34] proves to be effective in dealing with time-domain speech features. These approaches all employ the cross-attention mechanism to facilitate the interaction between visual information with time-domain or time-frequency domain speech features. These methods utilize visual features to extract corresponding speech features from the mixture in either a parallel manner [32, 44] or a recursive fashion [45] . Additionally, certain approaches [7, 47] explore two-stage refinement techniques to refine features that may exhibit inconsistencies during the separation process. Our work aims to propose a unified audio-visual speech separation framework that can be more accurate and efficient. 
method 3.1 preliminaries: 
 Expressly, we represent the time-domain speech mixture as 𝑥 = 𝑁 𝑖 𝑥 𝑖 , 𝑥 𝑖 ∈ 𝑅 𝑇 𝑥 , where 𝑁 denotes the total number of speakers present in the mixture, and 𝑇 𝑥 represents the time length. Our basic task is to separate and extract the target audio 𝑥 𝑖 from the mixed audio 𝑥 in the time domain, utilizing the provided visual features 𝑉 𝑖 as a reference for the 𝑖-th speaker. Considering the situation of visual information deficiency, the number of available visual cues 𝑃 may be less than the total number of speakers 𝑁 (i.e., 𝑃 ≤ 𝑁 ). Following most previous settings [21, 28, 31] , our primary focus is on the AVSS task of single-channel audio-visual speech separation, addressing the challenge of separating speech signals from overlapped audio mixtures that contain speeches from different speakers.The overall pipeline of our model is shown in Figure 2 . For clarity, we hypothesize that a 3-speaker mixture with two visual cues was sent into the model. As for the visual input, we send the two visual cues into the visual encoder and obtain visual features with a shape of 𝑅 𝑃 ×𝑇 𝑣 ×𝐷 , where 𝑇 𝑣 represents the temporal length of visual features, and 𝐷 represents the feature dimension. As for the speech part, passing through the audio encoder followed by a chunking operation, we obtain speech features represented as 𝐴 𝑚 ∈ 𝑅 𝑁 ×𝑆 ×𝐿×𝐷 . Here, 𝑁 , 𝑆, 𝐿, and 𝐷 denote the number of speakers, the number of chunks, the length of each chunk, and the channel-wise dimension, respectively. In Figure 2 , we draw 𝑁 = 3, 𝑆 = 3, 𝐿 = 4 and omit 𝐷 for better illustration.We follow prior research on AVSS methods [22, 31] that utilize a lip embedding extractor. This extractor consists of consecutive frame inputs with length 𝑇 𝑣 from the lip regions, processed through a 3D convolutional layer followed by an 18-layer ResNet integrated with multi-layer temporal convolutional networks. These modules collectively generate lip motion features 𝑉 𝑖 ∈ R 𝑇 𝑣 ×𝐷 for each speaker 𝑖, capturing lip movement characteristics in the visual domain. The 𝑃 speakers' available visual cues form 𝑉 ∈ R 𝑃 ×𝑇 𝑣 ×𝐷 .As for the audio part, the audio encoder extracts the features 𝐴 𝑜 𝑚 from the speech mixture sequence 𝑥 ∈ 𝑅 𝑇 𝑥 using the 1D convolution operation with a kernel size 𝐾 and stride 𝐾/2, which is:EQUATIONwhere 𝑇 𝑎 = ⌈2×𝑇 𝑥 /𝐾⌉ with proper zero padding, and 𝐷 is the audio embedding dimension. Due to the typically substantial length of audio features, denoted as 𝑇 𝑎 , computational challenges arise, hindering the learning of short-term, fine-grained features. To address this, a chunk operation is employed to split the audio feature 𝐴 𝑜 𝑚 into chunks of length 𝐿 with a hop size of 𝐿/2. These chunks are then concatenated to form a reshaped 3D audio chunked feature 𝐴 𝑚 ∈ R 𝑁 ×𝑆 ×𝐿×𝐷 , where 𝑆 is the number of generated chunks, and 𝑁 is the number of speakers in the mixture.The separator takes the extracted visual features 𝑉 ∈ 𝑅 𝑃 ×𝑇 𝑣 ×𝐷 and audio features 𝐴 𝑚 ∈ 𝑅 𝑁 ×𝑆 ×𝐿×𝐷 as inputs, and output feature masks 𝑀 𝑖 , 𝑖 ∈ {1, . . . , 𝑁 } for the 𝑁 speakers collaboratively with shape 𝑆 × 𝐿 × 𝐷. The overall flow for the separator is as follows.The separator in our study is comprised of 𝐵 blocks, and each block consists of three modules: a local-content attention module, a global content attention module, and a speaker-wise interaction module.Fine-grained audio feature extraction. Speech exhibits a continuous flow lacking explicit segment boundaries, thereby encapsulating intricate and hierarchical content [29] . In contrast, visual cues offer structured data, underscoring the challenge posed by the absence of inherent structure in speech signals. Consequently, the construction of short-time content becomes imperative. We incorporate intra-chunk interactions within each chunk of length 𝐿 to generate fine-grained information from the short-time features. Feed audio mixture 𝐴 𝑚 ∈ 𝑅 𝑁 ×𝑆 ×𝐿×𝐷 as input, the process is as follows:EQUATIONin which LCA represents the local content attention module. By stacking 𝑅 transformer layers, we enable the interaction among audio tokens of length 𝐿 within each chunk, capturing both local and contextual dependencies within the speech features.Visually guided audio feature enhancement. After applying the LCA module, we get 𝑁 speaker features with shape 𝑆 × 𝐿 × 𝐷. However, the differentiation among the features for the 𝑁 speakers is often limited. Despite the facilitation of intra-chunk interactions by the LCA module, the 𝑆 chunks for each speaker lack comprehensive global information and tend to be similar to each other. In real-world video recordings, there is a strong correlation between a speaker's audio and their lip movements over time. Therefore, the speech and visual features are generally aligned despite different sampling rates. Based on these observations, we introduce the global audio-visual interaction (GAV) module to enhance the disparity for 𝑆 chunks in each speaker with aligned video features.Given the temporal length 𝑇 𝑣 of the visual features and the number of chunks 𝑆 in the audio features, we ensure that 𝑇 𝑣 equals 𝑆 through appropriate padding. Considering that visual cues may be absent for specific speakers, cross-modal interaction is exclusively applied to speakers with available visual cues. The process is illustrated in Part (a) in Figure 2 . For the separated speech of shape 𝑆 × 𝐿 × 𝐷 accompanied by lip movements of shape 𝑇 𝑣 × 𝐷, we utilize every 1 × 𝐷 visual frame to interact with each 𝐿 × 𝐷 speech chunk. This can enhance the disparity between 𝑆 chunks and improve the global feature distribution for 𝐿 tokens within each chunk. Specifically, given 𝑃 available visual cues for the 𝑁 speakers, we split the audio features into two parts of length: 𝑃 and 𝑁 -𝑃. Moreover, we only use the first part to interact with the corresponding 𝑃 visual features 𝑉 . This interaction can be mathematically represented as follows:EQUATIONwhere Concat denotes the concatenate operation. This alignment allows audio features in each chunk to be temporally synchronized with their corresponding visual features, thereby enhancing the diversity of content represented by each audio chunk.Inter-chunk global speech interaction. To facilitate inter-chunk global speech interaction between 𝑆 chunks, we employ the global content attention module (GCA). The mathematical representation Mask M M V 1 V 1 V 2 V 2 V frames 1 V frames 1 V frames 2 V frames 2 Ām 1 Ām 1 Ām 2 Ām 2 Ām 3 Ām 3Ãchunksm 1 Ãchunks m 1 Ãchunks m 2 Ãchunks m 2 Ãchunks m 3 Ãchunks m 3 [ Â m 1,2 , Ãm 3 ] [ Â m 1,2 , Ãm 3 ] Â chunks m 1 Â chunks m 1 Â chunks m 2 Â chunks m 2 Ām Ām • A m 1 • A m 1 • A m 2 • A m 2 • A m 3 • A m 3 Ǎm 1 Ǎm 1 Ǎm 2 Ǎm 2 [ Ǎm 1,2 , • A m 3 ] [ Ǎm 1,2 , • A m 3 ]Spk. Additionally, in scenarios where the available visual information is insufficient for the number of speeches to be separated, we employ a split-and-concat operation to achieve cross-modal interaction.A m A m V V Permute&Reshape N × T a × D N × T a × D N × S × L × D N × S × L × D (N × S) × L × D (N × S) × L × D (N × L) × S × D (N × L) × S × D (S × L) × N × D (S × L) × N × D N × S × L × D N × S × L × D P × T v × D P × T v ×of this process is as follows:EQUATIONin which Ā𝑚 represents the result of the inter-chunk global speech interaction. By applying the stacked 𝑅 transformer layers, we enable the inter-chunk speech features to interact globally and capture long-range dependencies within the audio features. This process enhances the understanding and integration of audio information across different chunks.Speaker-wise feature interaction. The previous modules, namely LCA, GAV, and GCA, have been developed to enhance speech content through a blend of fine-grained and coarse-grained interactions, complemented by visual cues. However, these modules have not adequately addressed the precise characterization of speaker attributes for each speaker. As depicted in Part (b) of Figure 2 , when handling separated audio features Ā𝑚 , there are instances of missing segments or contamination with unrelated segments within each speaker's representation Ā𝑚 1 , Ā𝑚 2 , and Ā𝑚 3 , respectively. Specific audio segments intended for one speaker may erroneously be assigned to others, causing interruptions in the intended speaker's speech. Similarly, segments from other speakers may be incorrectly attributed to a given speaker, leading to contamination. These challenges primarily arise from the limited interaction among speakers. Since the mixture comprises contributions from each speaker, each separated speech component maintains a distinct ratio relative to the whole and to each other. To address these challenges and enhance the model's robustness in multi-speaker scenarios, we propose a speaker-wise audio interaction (SAI) module followed by a speaker-wise audio-visual interaction (SAVI) module. Starting with the separated speech features Ā𝑚 obtained from Equation (4), the SAI module is employed to facilitate interaction among speakerspecific features more precisely and rigorously. The formulation is as follows:EQUATIONThe SAI module serves to establish comprehensive global relationships among speakers. It facilitates the reallocation of each component to its corresponding speaker, ensuring accurate assignment of the extracted audio segments. This becomes particularly crucial in scenarios where the distinct separation of speakers within Ā𝑚 lacks distinguishable visual cues. Given that visual information typically improves the accuracy of the speaker-wise attributes, the absence of visual cues can lead to a significant drop in performance. To mitigate this challenge, the SAI module leverages speakers equipped with complete audio-visual information to aid in identifying those solely reliant on audio data. Specifically, we obtain the processed 𝐴 𝑚 after applying the SAI module. Part (c) of Figure 2 demonstrates that the SAI module can filter out unrelated segments for each separated speech from other speakers. However, when speakers articulate identical sentences with similar pacing and volume, the discrete speech attributes may demonstrate similarity at the semantic level. In such cases, the SAI module can cause confusion between the two similar separated speeches. To address this issue, it is essential to introduce the speaker-wise audio-visual interaction (SAVI) module to enhance the speakerrelated information in each separated speech. The SAVI module plays a crucial role in reducing inherent noises and enhancing the distinctive characteristics specific to each speaker. We firstly divide the separated speaker features 𝐴 𝑚 into two groups: the visuallyguided 𝑃 speakers, denoted as 𝐴 𝑚 1 , and the non-visually-guided 𝑁 -𝑃 speakers, denoted as 𝐴 𝑚 2 . The SAVI module is then employed to further enhance the speaker-specific information for the 𝑃 visually-guided speakers 𝐴 𝑚 1 . The process for the SAVI can be described as follows:EQUATIONwhere SAVI represents the global cross-attention process between visual and audio features, which can be described as follows:EQUATIONwhere 𝑊 𝑄 , 𝑊 𝐾 , and 𝑊 𝑉 represent the weights for query, key, and value embeddings, respectively. Then the processed 𝑃 visuallyguided speaker features in SAVI denoted as Ǎ𝑚 1 , are concatenated with 𝑁 -𝑃 non-visually-guided speaker features 𝐴 𝑚 2 along the speaker dimension, which can be described as:EQUATIONwhere we get the combined 𝑁 speaker features Ǎ𝑚 . Overall, the cohesive operation of the SAI and SAVI modules effectively establishes speaker-specific attributes through inter-speaker interactions and audio-visual cross-modal interactions. These modules ascertain the relative ratio for each speaker to the mixture and to each other, building distinct correlations. They significantly enhance the model's robustness, especially in cases where visual information for certain speakers is completely absent or where visual frames are partially missing.To generate the mask 𝑀 = OverlapAdd( Ǎ𝑚 ) ∈ R 𝑁 ×𝑇 𝑎 ×𝐷 , we apply an overlap-add operation on Ǎ𝑚 , which is the inverse process of the chunk operation described in Section 3.2. Next, we perform elementwise multiplication between the output of the audio encoder 𝐴 𝑚 and the mask 𝑀, which is given by:EQUATIONSubsequently, we transform 𝐴 𝑜 back into audio waveforms for each speaker using the transposed convolution version of the encoder. This version employs a kernel size of 𝐾 samples and a stride size of 𝐾/2 samples, and the process can be expressed as follows:x = TransposedConv1D(𝐴 𝑜 ). [18, 48] combined with the scale-invariant signal-to-distortion ratio (SI-SDR) [37] as the loss function. On the other hand, for the video-guided speech signals ( x1 ) with labels 𝑥 1 , we solely employ the SI-SDR loss as the loss function. The total loss function can be described as follows:EQUATIONin which the L 𝑆𝐼 -𝑆𝐷𝑅 can be described as:L 𝑆𝐼 -𝑆𝐷𝑅 ( â, 𝑎) = -10𝑙𝑜𝑔 10 ( ∥ ⟨ â,𝑎⟩𝑎 ∥𝑎 ∥ 2 ∥ 2 ∥ â - ⟨ â,𝑎⟩𝑎 ∥𝑎 ∥ 2 ∥ 2).(EQUATIONThis approach ensures that the model is trained effectively using both visually-guided and non-visually-guided audio features, considering each case's specific requirements. 
experiment 4.1 datasets: 
 VoxCeleb2 [10] comprises over 1 million samples organized by identity labels. It includes 5994 speakers in the training set and an additional 118 speakers in the test set. Each sample contains an utterance with synchronized face tracks. We followed the data generation instructions outlined in [32] to generate the dataset. We randomly selected speech samples from different speakers based on their unique IDs and combined them by adding them together using varying scales. This process resulted in 20,000, 5,000, and 3,000 samples for the training, validation, and test sets. LRS3-TED [2] consists of 433 hours of audio and 151k corresponding video clips extracted from TED and TEDx talks. To assess model generalization, we train them on the VoxCeleb2 dataset and evaluate them on cross-domain LRS3 datasets without fine-tuning. Similar to the VoxCeleb2 mixing process, we generate 660 samples each for 2, 3, 4, and 5 mixture samples as the testing sets.Experiment configurations. The visual frames are sampled at a rate of 25 frames per second (FPS), and the audio data is sampled at a rate of 16kHz. The training length for each mixture is set to 6s. The encoder/decoder kernel size 𝐾 in Section 3.2 and Section 3.4 is set to 16 with a stride of 8. And the chunk size 𝐿 and dimension 𝐷 is set to 160 and 256 respectively. In Section 3.3, the local content attention (LCA) module and global content attention (GCA) module in the separator are stacked with 𝑅 layers each, which is set to 2. Additionally, all other modules in Section 3.3 are single-layer operations, forming one block for the separator. The one separator block is repeated 𝐵 times, which is set to 5. Optimization. Following [22] , the model is trained using an Adam optimizer [17] with an initial learning rate of 1.5×10 -4 . The learning rate will be halved if there is no loss decrease on the validation set for three epochs. The training process stops if there is no loss decrease for five epochs.Evaluation. SI-SDR measures the ratio between the energy of the target signal and that of the errors. We also employ the Perceptual Evaluation of Speech Quality (PESQ) [36] metric to further assess speech quality and intelligibility.Training and testing settings. During the training phase, we generate a dataset with dynamically varying mixture numbers from VoxCeleb2 dataset [10] . This dataset encompasses combinations of 2, 3, 4, and 5 speakers concurrently. Consistent with prior methodologies [7] , a distribution ratio of 2:1:1:1 is employed for the respective combinations, thereby fostering equilibrium in performance across diverse compositional scenarios. Additionally, a parameter setting is introduced wherein a 10% probability is designated for the absence of 1-2 visual cues. Subsequent to the training phase, distinct sample sets are generated for each mixture configuration, namely 2, 3, 4, and 5 speakers, during the testing phase. To facilitate a comprehensive evaluation, 3000 samples are specifically generated from the VoxCeleb2 dataset [10] , augmented by an additional 660 samples obtained from the LRS3 dataset [2] for each mixture configuration. In order to thoroughly assess the performance of the proposed model and validate its robustness across various scenarios, emphasis is primarily placed on two distinct testing situations.• Situation 1: Our primary aim is to assess the robustness of our model to variations in the number of speakers, ranging from 2 to 5. Like most previous settings, we send 𝑁 -mix mixtures with corresponding 𝑃 visual information, in which 𝑃 = 𝑁 . • Situation 2: Our primary objective is to assess the robustness of our model in the face of missing visual cues, which encompasses both complete absence of visual cues and partial loss of visual frames. Firstly, to simulate the visual cues absence scenarios, we provide 𝑁 -mix mixtures with corresponding 𝑃 visual information, in which 𝑃 < 𝑁 . Secondly, we introduce a random masking process to simulate the scenario of missing visual frames. For each video in the set of 𝑁 videos, we randomly select a portion of frames based on the specified missing rates and mask them by setting their values to zero. In Situation 1, we mainly compare the performance of our model with other open-source audio-visual speech separation methods. However, most of these methods present their two-speaker separation results under the two-speaker mixture training setting. To ensure a fair comparison, we have re-trained these methods using the same training strategy employed in our model, which involves simultaneously training multiple speakers. Note that the difference between a two-speaker training setting and a simultaneously multispeaker training setting is only the linear increase in the number of speakers in the mixture and the corresponding increase in the number of speakers to be separated during training. To provide further evidence for the superiority of our simultaneous separation framework, we conduct additional comparisons between our methods and selected approaches in the missing visual cues scenarios. The experimental results for the two situations are presented below.Experiment results for Situation 1. The experimental results, presented in Table 1 , demonstrate the robust performance of our model across all mixture scenarios. As the number of speakers increases, the task of separation becomes more challenging. On the VoxCeleb2 dataset [10] , most methods experience a significant decrease in their SI-SDR performance [37] when the number of speakers to be separated increases from 3 to 4 or from 4 to 5. Our model consistently outperforms other models as the number of mixtures increases, highlighting the effectiveness of our simultaneous separation framework with speaker-wise interactions. The audio-only and audio-visual speaker-wise interaction modules in our model significantly enhance speaker identity in each separated speech and facilitate improved audio-visual fusion. Furthermore, our model's performance is evaluated on the LRS3 dataset [2] without any fine-tuning, demonstrating its generalization and transferability to unseen data. In addition, a consistent positive correlation between the PESQ metric measuring speech quality and the SI-SDR metric can be observed.Experiment results for Situation 2. We test the robustness of our model under conditions where visual information for specific speakers is entirely absent or where visual frames are partially missing, shown in Figure 3 and Figure 4 respectively.In Figure 3 , we compare our results with other methods in scenarios where one corresponding lip movement video is absent. Specifically, we choose AV-Sepformer [22] and BFRNet [7] for comparison. AV-Sepformer is selected because it is also based on the transformer architecture [40] , which enables us to demonstrate that the performance improvement is not solely a result of using different backbone architectures. On the other hand, BFRNet is chosen for its specifically designed modules to address the challenges posed by multi-speaker chaos. The results presented in Figure 3 include both the visual-complete scenario in Situation 1 and the part visual-absence scenario for the three methods. Specifically, we demonstrate the visual-absence scenario for situations where only one visual cue is missing in the figure . For example, we demonstrate the performance of 5 separated speech from 5-mix speech mixture with only 4 visual cues. As shown in Figure 3 , AV-Sepformer [22] experiences a significant performance drop in the visual-absence scenario, while BFRNet [7] shows a comparatively smoother decline.In contrast, our model consistently outperforms both methods in visual-absence scenarios across all mixtures (2, 3, 4, and 5 speakers). It handles missing visual information effectively, resulting in the smallest performance gap between complete and absent visual scenarios. However, we have yet to present results for scenarios with multiple missing lip movements, which often yield negative values and limit their reference value and interpretability. Bridging the gap between complete and lacking visual information remains an important avenue for future research.In Figure 4 , we present the performance trend of our model as the missing frames rate increases. We compare our model's performance with AV-Sepformer [22] to assess the performance under specific mixture settings. Firstly, we analyze the performance of our model across different mixture settings. It is evident that the performance decreases as the percentage of missing frames increases. Furthermore, as the number of mixtures increases from Figure 4 : The visual robustness performance with different missing video frame rates. It is important to note that we apply the missing frame rate to all corresponding speakers' video frames. The x-labels of our model represent different missing frame percentages in each corresponding video, and the y-label represents the corresponding SI-SDR metric.2 to 5, the performance drop becomes more pronounced. Though this can be partly attributed to the overall increase in the number of missing frames, it poses challenges for both speaker extraction and inter-speaker interaction processes. Secondly, we compare our model with AV-Sepformer [22] for each specific mixture setting. Notably, there is a noticeable performance decline as the percentage of missing frames increases. However, our model exhibits a relatively smoother performance drop. This comparison highlights the robustness of our model in scenarios with visual cues part missing. Through this comparison, we observe that our model demonstrates resilience in handling scenarios with missing visual cues, showcasing its ability to maintain performance in challenging conditions. 
ablation study: 
 The effectiveness of each module. Table 2 demonstrates the effectiveness of each module used in our study. In Row a, we present the basic model [23] consisting solely of LCA and GCA modules, which capture both intra-chunk and inter-chunk correlations of T-domain speech features. By integrating the GAV model within the dual-path framework in Row b, we significantly enhance the speech content disparity for each separated speaker. Compared to the model structure in [22] , we make modifications by using fewer layers in each intra-chunk and inter-chunk module, reducing redundancy and increasing the number of cross-modal interactions. As shown in Row b, this modification significantly improves separation performance in mixtures of 3,4,5 speakers by up to 2 dB in SI-SDR [37] . The effectiveness of the single-modal speaker-wise interaction (SI) module is demonstrated in Row c. The SAI module can build the distinctions and correlations among speakers and between each speaker and the whole mixture. It can effectively Downstream audio-visual speech recognition performance. AVSS serves as the front end for various downstream tasks such as speech recognition and speaker diarization. To evaluate the compatibility of separated speech samples with downstream speech recognition tasks, we employ a publicly pretrained model from AV-HuBERT [38] and measure the word error rate (WER) metric on the separated speech. Specifically, we save the separated speech outputs generated by our separation model when provided with 2-mix, 3-mix, 4-mix, and 5-mix inputs, respectively. Due to the absence of text labels in the VoxCeleb2 dataset, we exclusively evaluate performance on the LRS3 dataset. We feed both the separated clean speech and its corresponding video into the audio-visual speech recognition model. As presented in Table 3 , the initial clean samples used for mixing exhibit a WER of 13.41. We compare our approach with two classical methods, AV-ConvTasNet [44] and AV-Sepformer [22] . These methods shed light on the developmental progress of speech separation. In Table 3 , we notice a direct relationship between the clarity of separated samples (indicated by higher SI-SDR metrics, as shown in Creftab:situation1) and the accuracy of downstream speech recognition tasks, reflected in lower WER. Comparing Row c and Row b, we notice a significant performance improvement resulting from the transition from ConvTasNet to Sepformer. Moreover, our separated speeches achieve the best WER, surpassing others by a large margin. Besides, when comparing Rows b, c, and d with Row a, we identify a remaining gap that the speech separation module needs to bridge.Qualitative results. As illustrated in Figure 5 , we showcase the separation results for a 5-mixture audio. We visualize the differences between our separated samples, AV-Sepformer, and the original clean samples before the mixing operation. Each spectrogram represents an individual sample, with time on the horizontal axis and frequency on the vertical axis. A spectrogram illustrates the energy distribution across frequencies over time. The red box highlights temporal discrepancies, while the black box indicates frequency distinctions. In Figure 5 , we notice disparities in the temporal predictions of the AV-Sepformer model, highlighted in red boxes in Columns (a), (c), and (e). In contrast, our model makes accurate predictions for these segments, closely matching the clean samples depicted in the first row. Moreover, our model adeptly predicts missing or noisy segments across both high and low frequencies, as illustrated by the black boxes in Columns (a), (c), and (e). Additionally, in Column (b), we observe chaotic energy distribution in specific segments of the AV-Sepformer predictions, indicating a more pronounced decline in performance. 
conclusion: 
 In this paper, we propose a robust simultaneous multi-speaker audio-visual separation framework that is robust to both multispeaker scenarios and missing visual cues. Our framework utilizes cross-modal audio-visual interaction modules to enhance the distinctions between speakers in terms of both speech content and speaker attributes. Additionally, an inter-speaker interaction module is employed to establish relationships among speakers and between individual speakers and the entire audio mixture. It effectively establishes the connection between visually-guided separated speakers and non-visually-guided separated speakers. The framework achieves state-of-the-art results in settings involving 2, 3, 4, and 5 speaker mixtures and significantly reduces the performance gap in scenarios where visual cues are absent.  
 
 Appendix:  


Decision: The paper proposes a new simultaneous multi-speaker separation framework that considers speaker-wise interactions. The paper received 2 WAs, 1 BA, and 1 BR. The BR reviewer is mainly concerned that the paper's presentation is not clear. Otherwise, the reviewers liked that the problem and method have good motivation, the method is robust, and the evaluation is thoughtful. The AC recommends acceptance and thinks the presentation is addressable and urges the authors to improve paper writing, especially regarding the modules LCA and GCA.

Reviews: review: paper_summary: paper_summary: The paper focuses on target speaker extraction using visual cues, even in scenarios where some visual cues are missing or incomplete. The authors propose a novel RAVSS network capable of handling scenarios with 2-5 mixed speakers, achieving state-of-the-art results. Additionally, they demonstrate the robustness of their network in the absence of visual cues.
strength_weakness: strength_weakness: 1.This paper is novel enough to address missing visual cues problems in target speaker extraction tasks using visual cues. In the experimental section, it compares various existing methods in Table 1 with exhaustive experiments in scenarios with two to five speakers, and demonstrates that its method works best in normal audio-visual scenario. 
2.The most important comparison is in Figure 3 where visual cues are missing, and the results in the figure demonstrate the strong robustness of this paper to the case of missing visual cues, which is a very significant advantage over previous work.
3.The interpretability of the results is high. Figure 5 clearly demonstrates the advantages of the proposed method compared to others. Across low, medium, and high frequencies, RAVSS more effectively removes extraneous noise.
limitations: limitations: 1.The paper is not clear enough in its description of specific modules, such as LCA, GAV and GCA etc., and does not give detailed structures or descriptions, which only seem to be presumed to be some kind of attentions or transformers in the paper's description. While the supplementary material may be given in the code, it still needs to be described clearly in the paper.
2.The RAVSS network seems to require prior knowledge of which speaker's visual cues will be missing at the input, rather than learning this automatically. For instance, in the 3mix scenario, if the presence or absence of visual cues is unknown during the inference stage, will RVASS assume P=N by default? In this case, does the specially designed approach of RAVSS for handling situations where P<N still have practical significance?
paper_summary: paper_summary: This paper introduces RAVSS, a robust framework for audio-visual speech separation that excels in multi-speaker scenarios, even with missing visual cues. Unlike existing methods that sequentially isolate speakers and struggle with performance drops as the number of speakers increases, RAVSS performs simultaneous separation and utilizes speaker-wise interactions to distinguish and correlate among speakers effectively. Experimental results on the VoxCeleb2 and LRS3 datasets demonstrate that RAVSS achieves state-of-the-art performance in separating mixtures of up to five speakers, maintaining robustness in scenarios with missing or partial visual information.
strength_weakness: strength_weakness: The paper includes extensive evaluation results. it also includes qualitative samples to show how the separation works well.
limitations: limitations: The writing of this paper needs to improve. It is not clear to show the novelty and strengths of this paper. For example, the contribution "We propose a robust multi-speaker audio-visual separation framework that demonstrates superior performance, particularly as the number of speakers in the mixture increases" does not include concrete instances. Figure 2 is complex but it is not clear the key contribution and module in the model. There are several fusion modules but how they are unique and effective is not well presented. 
Figure 5 c and d do not have annotations. Furthermore, it makes more sense to show how a segment speech of one person is incorrectly mapped to another speech. 
The paper does not show the novelty to solve the problem. It introduces several modality fusion modules but does not show how they are different from previous work. There are no figures on the details inside several proposed modules.
paper_summary: paper_summary: This paper addresses the problem of separating speech from multiple speakers using both audio and visual modalities. While existing approaches using audio-visual modalities handle this problem for up to two speakers, the authors propose an end-to-end framework that solves the problem for up to five speakers. Their framework also demonstrates robustness when visual information is incomplete or missing.
strength_weakness: strength_weakness: 1. **Clarity and Detail**: The paper is well-written, providing clear and detailed explanations of the proposed methods and the problem being addressed.
2. **Problem Statement and Motivation**: The authors clearly define the problem and provide strong motivation for their work, thoroughly reviewing relevant literature to position their contributions within the broader research context.
3. **Innovative Contributions**: The paper introduces novel attention mechanisms to capture local, global, and audio-visual interaction features, significantly advancing the state-of-the-art in multi-speaker speech separation.
4. **Evaluation**: The proposed method is evaluated on challenging datasets ([VoxCeleb2](https://www.robots.ox.ac.uk/~vgg/data/voxceleb/vox2.html) and [LRS3](http://www.robots.ox.ac.uk/~vgg/data/lip_reading/lrs3.html)), demonstrating its effectiveness in separating mixtures with up to five speakers and showing robustness to missing visual cues.
limitations: limitations: 1. **Inconsistency in Notation**: The variable \( N \) denotes the number of speakers and \( P \) denotes the number of available visual cues. However, the notation does not consistently guarantee clarity throughout the paper.
2. **Reliance on arXiv References**: Many references are from arXiv, which are not peer-reviewed. This raises concerns about the reliability and validation of the referenced works.
3. **Small Margin of Improvement**: In Table 1, the performance differences between the proposed approach and existing methods (e.g., AV-Sepformer) are relatively small for some metrics (e.g., SI-SDR and PESQ) on the VoxCeleb and LRS3 datasets. Without running the approaches multiple times and performing statistical analysis, it is challenging to conclusively determine superiority.
4. **Lack of Detailed Explanation for LCA and GCA**: The paper lacks detailed descriptions of the Local-Content Attention (LCA) and Global Content Attention (GCA) modules. The authors simply state they use "stacking RR transformer layers" without further elaboration. Additionally, the ablation study does not evaluate the individual contributions of LCA and GCA, leaving their impact unclear.
5. **High Number of Parameters**: The proposed approach has 24.3 million parameters, which is high for real-time and practical applications. Given that speech separation is often a front-end task for other speech processing applications, the model's complexity might limit its usability in practical scenarios.
paper_summary: paper_summary: This paper addresses the challenge of robustness in scenarios involving multiple speakers and missing visual cues for audio-visual separation. To tackle this, the authors introduced audio-visual interaction modules that leverage the correlation between speech and lip movements. Specifically, they utilize content and speaker-wise interaction modules to address cases where visual cues of speakers are missing using complete audio-visual information.
strength_weakness: strength_weakness: This paper is well-motivated and exhibits clear and focused writing. 
It's good to see that the authors effectively introduce each module, providing insights into their design rationale and specific functions. The model design is aligned with the stated motivation, ensuring coherence throughout.
Furthermore, the experimental design revolves around assessing the system's robustness to variations in the number of speakers and missing visual cues, showcasing a thoughtful approach to evaluation.
limitations: limitations: 1. As the authors claimed in the paper, the feature after SAI and SAVI module will contain the distinctive characteristics specific to each speaker. A direct approach to validate this claim would be through speaker verification tests. I recommend that the authors conduct experiments to demonstrate the capability of these features in speaker verification tasks.

2. Nowadays audio-only separation can also perform well. Can you compare with audio-only separation systems when all visual cues in your model are missing?

3. Considering recent advancements, it would be beneficial to include comparisons with newer models such as the Audio-visual TF-GridNet [1].

[1] Pan, Z., Wichern, G., Masuyama, Y., Germain, F. G., Khurana, S., Hori, C., & Le Roux, J. (2023, December). Scenario-Aware Audio-Visual TF-GridNet for Target Speech Extraction. In 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) (pp. 1-8). IEEE.