RTFS-Net: Recurrent Time-Frequency Modelling for Efficient Audio-Visual Speech Separation
Authors: Samuel Pegg
Kai Li
Xiaolin Hu
Venue: ICLR 2024 poster (2024)
Keywords: Audio-Visual
Multi-Modal
Time-Frequency-Domain
Speech-Separation
Model-Compression
Primary area: general machine learning (i.e., none of the above)

Abstract: Audio-visual speech separation methods aim to integrate different modalities to generate high-quality separated speech, thereby enhancing the performance of downstream tasks such as speech recognition. Most existing state-of-the-art (SOTA) models operate in the time domain. However, their overly simplistic approach to modeling acoustic features often necessitates larger and more computationally intensive models in order to achieve SOTA performance. In this paper, we present a novel time-frequency domain audio-visual speech separation method: Recurrent Time-Frequency Separation Network (RTFS-Net), which applies its algorithms on the complex time-frequency bins yielded by the Short-Time Fourier Transform. We model and capture the time and frequency dimensions of the audio independently using a multi-layered RNN along each dimension. Furthermore, we introduce a unique attention-based fusion technique for the efficient integration of audio and visual information, and a new mask separation approach that takes advantage of the intrinsic spectral nature of the acoustic features for a clearer separation. RTFS-Net outperforms the prior SOTA method in both inference speed and separation quality while reducing the number of parameters by 90% and MACs by 83%. This is the first time-frequency domain audio-visual speech separation method to outperform all contemporary time-domain counterparts.

Published as a conference paper at ICLR 2024
RTFS-ÑET: RECURRENT TIME-FREQUENCY MOD- ELLING FOR EFFICIENT AUDIO- VISUAL SPEECH SEP- ARATION
Samuel Pegg', Kai Kai Li Xiaolin Hu 1.2,51
axh. 2 iinguua.ttniin...a org.en, LK2 1ee_mmmalls eemallsttiinhheeeee.....ee. tsinghua eduu cn xlhuttsinghua, hew.:gghhaaaeuua..ooooo
I. Department of Computer Science and Technology, Institute for AI, BNRist, Tsinghua University, Beijing 100084, China
2. Tsinghua I Laboratory of Brain and Intelligence (THBI),
IDG/McGovemn Institute for Brain Research, Tsinghua University, Beijing 100084, China
3. Chinese Institute for Brain Research (CIBR), Beijing 100010, China
ABSTRACT
Audio-visual speech separation methods aim to integrate different modalities to generate high-quality separated speech, thereby enhancing the performance of downstream tasks such speech recognition. Most existing state-of-the-art (SOTA) models operate in the time domain. However, their overly simplistic ap- proach to modeling acoustic features often necessitates larger and more compu- tationally intensive models in order to achieve SOTA performance. In this paper, we present a novel time-frequency domain audio-visual speech separation method: Recurrent Time-Frequence Separation Network (RTFS-Net), which applies its al- gorithms on the complex time-frequency bins yielded by the Short-Time Fourier Transform. We model and capture the time and frequency dimensions of the audio independently using multi-layered RNN along each dimension. Furthermore, we introduce unique attention- based fusion technique for the efficient integration of audio and visual information, and new mask separation approach that takes ad- vantage of the intrinsic spectral nature of the acoustic features for clearer separa- tion. RTFS-NG outperforms the prior SOTA method both inference speed and separation quality while reducing the number of parameters by 90% and MACs by 83%. %. This is the first time-frequency domain audio-visual speech separation method to outperform all contemporary time-domain counterparts.
INTRODUCTION
The 'cocktail party problem' (Bronkhorst 2000; Haykin & Chen 2005 Cherry 2005) describes
our ability focus on a single speaker's voice amidst numerous overlapping voices in noisy environment. While humans effortlessly tackle this problem, replicating this ability in machines remains a longstanding challenge in the field signal-processigg Audio-only Speech Separation (AOSS) methods (Luo & Mesgarani 2019 Luo et al. 2020) Subakan et al., (02T). solely utilizing the mixed-speaker audio signal, face limitations in scenarios with strong background noise, rever- beration, heavy voice overlap. To overcome these issues, researchers turned
multi-modh approach: Audio-visual Speech Separation (AVSS) (Gao & Grauman 2021: Lee et al., 2021 Li let al., 2022]. AVSS methods integrate additional visual into the paradigm and can be gen- erally split into two main classifications: Time-domain (T-domain) and Time-Frequency domain (TF-domain) methods, each with their own benefits and challenges.
T-domain methods (Wu et al. 2019 Li et al. 2022 Martel et al., 2023 Lin et al. 2023) work on the
long, uncompressed and nigh-diimnsiionll audio features returned by the ID convolutional encoder design proposed by Luo & Mesgarani (2018). This approach facilitates fine- -re-grained, high qual- ity audio separation, but the high parameter count and computational complexity leads extended training periods, intensive GPU usage and slow inference speeds. On the other hand, TF-domain
Made significant contrihutions. Corresponding author.

---

Published as a conference paper at ICLR 2024
methods (Afouras et al. 2018a Alfouras al. 2018 Gao & Grauman, 2021 apply their algorithms on the complex 2D representation yielded by the Short-Time Fourier Transform (STFT). Typically, the STFT uses large windows and hop lengths to compress the data, resulting in more computation- ally efficient separation methods. However, from a historical perspective, all TF-domain methods have been substantially outperformed by T-domain methods. Based our research, this gap in performance stems from three critical factors.
Firstly, while some attempts have been made (Afouras al., 2020 Lee al. 2021 to model am- plitude and phase separately, no TF-domain AVSS methods explore the independent and tailored modelling of the two acoustic dimensions (time and frequency) in order exploit this domain' advantage over T-domain methods. Indeed, recent research by [uo & Yu (2023) and Wang et al. (203)) the AOSS domain have capitalized on this advantage and outperformed their T-domain counterparts by large margins. However, their usage of large LSTM (Hochreiter & Schmidhuber, 1997] and transformer (Vaswani al., 2017) architectures leads to a heavy computational burden, making them an unattractive solution in the AVSS space. Secondly, while existing AVSS studies et al. 2022 Lin et al. 2023) have explored various audio-visual fusion strategies, they neglect using visual features from multiple receptive fields to increase model performance. This type of visual information serves as important ä priöri knowledge crucial for accurately extracting the tar- get speaker's voice. Thirdly, TF-domain AVSS studies (Afouras et al. 2020 Lee et al. [202]] Gao & Grauman] (021) often overlook the underlying complex nature of the features, and hence lose critical amplitude and phase information when extracting the target speaker's voice from the audio mixture. This degrades the reconstruction performance of the inverse STFT (iSTFT) and leads to poorer model performance.
In this work, we propose a novel TF-domain AVSS method: Recursive Time- Fre-ueryyy Separation Network (RTFS-Nec Figure I that provides a computationally efficient solution the cocktail party problem using an STFT based audio encoder, high-fidelity pretrained video encoder and series of recursive RTFS Blocks to perform target speaker extraction. Our contributions are threefold:
1. Our RTFS Blocks tackle the first issue by projecting the features to compressed subspace. There, we explicitly model both acoustic dimensions individually, then in tandem before sub- sequently applying attentional mechanism (TF-AR) to restore the dimensions with minimal information loss, allowing us to reap the benefits of independent time-frequency processing with- out bearing substantial computational cost.
2. Our Cross-dimensional Attention Fusion (CAF) Block provides low parameter, computation- ally efficient solution the second issue by aggregating the multi-modal information through a multi-head attention strategy in order to optimally fuse the visual cues of the target speaker into the audio features, facilitating high quality separation.
3. Our Spectral Source Separation (St) Block addresses the third issue by explicitly reconstructing the plural features of the target speaker, achieving a higher quality separation without increasing computational cost.
We conducted comprehensive experimental evaluations on three widely used datasets: LRS2 (Afouras et al. 8a). ) LRS3 (Afouras et al. 20188) and VoxCeleb2 (Chung et al.) 2018), to demon- strate the value of each contribution. To the best of our knowledge, RTFS-Net is the first To-domiin AVSS method to outperform all contemporary T-domain methods, achieving this while also exhibit- ing clear advantage by reducing computational complexity by 83% and reducing the parameter count by 90%. We additionally provide a Web page where sample results can be listened to, and our code will be open-sourced after publication for reproducibility purposes.
2 RELATED WORK
Audio-only speech separation. Modern deep learning advances introduced neural networks for speaker-agnostic speech separation, with Conv-TasNet (Luo & Mesgarani 2019) and DualPathRNN (Luo et al. 2020) making significant T-domain breakthroughs. However, T-domain methods (Luo & Mesgarani 2019: Luo et al. 2020) often exhibit a marked performance degradation in reverber- ant conditions, attributed to their neglect of explicit requenccydomall modeling. Consequently, recent research focuses on high high-performance speech separation models in the TF-domain. For in-
stance, TFPSNet YYng et al. (022) incorporates the transformer from DPTNet (Chen 2020) to
2

---

Published as a conference paper at ICLR 2024
assimilate spectral-temporal information. TF-GridNet (Wang al. 2023) extends this by introduc- ing a cross frame self-attention module in order to achieve SOTA performance on the WSJO-2mix [Hershey al. 2016). However, the inclusion of many LSTM and Transformer layers results in extremely high computational complexity, leading to increased training times and GPU memory requirements. Lightweight models like A-FRCNN (Hu al.) 202T and TDANet (Li et al., 2023) strike a balance between performance and efficiency by using an encoder-decoder paradigm with recurrent connections and top-down attention. However, their separation prowess noisy scenarios remains suboptimal compared to multi-modal approaches.
Audio-visual speech separation. Conventional AVSS statistical methods relied on knowledge- driven modeling (Loizou 2013; Wangg & Brown, 2006). Early deep-learning explorations into AVSS mainly occurred in the TF-domain, focusing on tasks like amplitude and phase reconstruction for target speakers (Afouras et al., [01 Alfouras et al. 2018). With the advent of AV-ConvTaste (Wu et al. 2019), became a prevailing belief that T-domain AVSS methods consistently outper- formed TF-domain methods. Notably, CTCNet (Li al. 2022), inspired by thalamic brain struc- tures, introduced a unique multiple-fusion approach. Leveraging multiscale contexts from visual and auditory cues, this module greatly enhanced the spatial scale of fused features, thus improving the model's separation capacity. However, as mentioned previously, T-domain methods come with higher computational load. TF-domain AVSS techniques often larger windows and hop sizes, which curtails computational complexity. Nevertheless, their full potential is yet to be realized. The recent Visualvoice model (Gao & Grauman 202I) combined multi-task learning framework for both AVSS and cross-modal speaker embedding, incorporating facial expressions, lip movements, and audio cues. Despite its touted efficiency, Visualvoice lacks robust modeling, and it lags behind many modern T-domain methods.
3 METHODS
Expanding on prior SOTA methods (Wu et al. 2019 Li al. (022), we present our AVSS pipeline, illustrated in Figure 1 The mono-aural mixed-speech audio signal, E RIXL, in conjunction with the video frames capturing the target speaker's lip movements, y XIXL.xHXW are used as inputs to RTFS-Net in order to derive the target speaker's estimated audio signal, å E R I In this context, La and Ly signify the durations of the audio and video inputs, while H and correspond the dimensions of the single-channel (grey-scale) video frames.
Separation
Video
VP Block
I R-sstacked RTFS Blocks
Decoder
CAF Block
Audio Encoder
Block
8 Block
Estimated
do
The overall pipeline of RTFS-Net. The red and blue solid lines signify the flow directions of auditory and visual features respectively. The snowflake indicates the weights are frozen and the component is not involved in training.
Figure I:
Firstly, the audio and video encoders extract auditory ag and visual ty features. These serve as the inputs for our separation network, which fuses these features and extracts the salient multimodal features, az- Next, our Spectral Source Separation (59) method is applied to separate the target speaker's audio z from the encoded audio signal ag using an- Finally, the target speaker's estimated audio feature map z is decoded into the estimated audio stream and compared with the ground- truth signal for training.
3.1 ENCODERS
Our encoders distill relevant features from both audio and visual inputs. For the video encoder F, (?), we employ the CTp (Li et 2022) pretrained network extract the visual features
3

---

Published as a conference paper at ICLR 2024
My of the target speaker,
vo= = Fo= vo E TCCT.
(1)
For the audio encoder, we firstly define a as the complex-valus hybrid TF-domain bins obtained using the STFT on æ. For a mixed audio of пapk speakers, we define as the speech of speaker i and as the presence of some background noise, music, or other extraneous audio sources. Then,
Papk
ft - f)t, Σ f),
a(t,f)
(2)
where & E 10, Tx) is the time dimension and [0, F| is the frequency dimension. We concatenate the real (Re) and imaginary (Im) parts of a along a new 'channels' axis, then apply a 2D convolution Ea (-) with 3 x 3 kernel and C. output channels across the time and frequency dimensions to obtain the auditory embedding ag of æ. Using the symbol| for concatenation, we write.
a Ea (Re(a)))lm(()),
ao &.xxF.FF.
(3)
3.2 SEPARATION NETWORK
The core of RTFS-Net a separation network that uses recursive units to facilitate information in- teraction in the two acoustic dimensions, and efficiently aggregates multimodal features using an attention-based fusion mechanism. The initial step is to preprocess the auditory Oo and visual vo features separately in preparation for fusion. For the Visual Preprocessing (VP) Block we adopt modified version of the TDANet Block (Li et al., 2023), see Appendix [A] For the Audio Prepro- cessing (AP) Block we use single RTFS Block, whose structure will be defined in Section 332.22 The outputs of the two preprocessing blocks are fed to our proposed CAF Block to fuse the multi- media features into a single enriched feature map (see Figure [ and 2 This audio-visual fusion is subsequently processed with an additional R stacked RTFS Blocks. Following CTCNet (Li et al.) 2022), these R sequential blocks share parameters (including the AP Block), which has been shown to reduce model size and increase performance, since it turns the series of blocks in to a recurrent neural architecture (see Appendix B
3.2. CROSS-DIMENSIONS ATTENTION FUSION BLOCK
The CAF Block (Figure 2 is a depth-wise and group-convolution based architecture designed to consume as little resources as possible while fusing the 2D visual data into the 3D audio data. It in- volves two separate fusion operations that we call the attention fusion (f1) and the gated fusion (f2)- The attention fusion considers multiple visual sub-representation spaces to aggregate information from wide receptive field and apply attention the audio features. The gated fusion up- up-samples the visual information's 's time dimension, then expands the visual features into the T'Fdoomiin using F gates produced from the preprocessed audio features. We use Vaswani et al. (2017)'s 'keys' and "values" nomenclature.
c g00d
Qoen Ji
c2
Maan
Intepolation
CornD F4
Ao & Reripor
F%
Sew c
Noarso negghtx Inseppitiionn
Oownu
Yhy S00
Figure 2: Structure of our CAF Block. As in Figure the red and blue solid lines signify the flow directions of auditory and visual features respectively.
Firstly, let both P, and Pa denote a depth-wise convolution with a x kernel and following global layer normalization (gLN) (Luo Mesgarani 2019). We generate the audio 'value" embeddings and the aforementioned 'gate" from the preprocessed audio signal a1.
a (ai)),
agate ReLU (P2(a1)),
aval agate RR C
4

---

Published as a conference paper at ICLR 2024
Attention Fusion. We apply ID group convolution F1 with Ca groups and C. X output channels to V1. followed by gLN layer. By chunking across the channels, we can decompose the visual features into h distinct sub-feature representations, attention 'heads', vh. Next, we take the 'mean' of the h heads to aggregate the information from the different sub-feature representations into Um- then subsequently apply the Softmax operation in order create multi-head attention style set of features Vattn with values between O and 1. To align the video frame length Zv with the audio's 's time dimension Ta use nearest neighbor interpolation, ø. This is written,
2A Fi (v1),
dA R(CxXN)XT
(5)
Um mean mean (A
Um E RCXX,
(6)
attn ((Softmax(Um)),
Vattn E RCXT.
(7)
Our attention mechanism is applied each of the F 'value' slices of aval with length Ta
fill] = Vattn O avalli), Vi {1...., where [i(i] C.TT Vi = fi R.C.T.FF...
(8)
Gated Fusion. We use ID convolutional layer F2 with kernel size 1, C. output channels and Ca groups (since C. C.), followed by gLN layer to align C, with Ca. Next, we again use interpolation é to align Tv with T, and generate the visual 'key" embeddings.
Ukey = (F2(v1)), Ukey E C.xT
(9)
Next, we utilize of the Ti-dimensional slices of agate as unique gates to comprehensively expand the visual information into the To-danain,
f2[1] agate(? [i] ④ Ukey,
Vi (I,..,,,
(10)
where fall] C.CT Vi
f2 R.R.RTTTP
CAF Block. Finally, we sum the two fused features together. We can denote our CAF Block, 4, as:
42 = ((1, = f1 f2. a2 E R.x.TX
(11)
3.2.2 RTFS BLOCKS
Compared to previous To-domnin AVSS methods (Afouras et al. 2020 Gao & Grauman, 2021
Alfouras et al., 2018 Lee et al.) (021), our RTFS Blocks use a dual-path architecture to explicitly model audio in both acoustic dimensions to improve the separation quality, as shown in Figure 3 We denote the auditory features input into the RTFS Block A. Given our recurrent structure, A represents either dg (input to AP Block) or the output from the previous RIFS Block with a skip connection: a, + ao for j E {1,..., R}. Note that in Figure the residual connection is not shown for simplicity. Our RTFS Block processes auditory features In the four steps discussed below.
bequeryy
ICoev1D
TOOWID
trepuerry
T2
SEU MoNTa
SROU Layoer
IF dondaan voberconn
Unioa
Ae
Compressioo
Phose
Dustspoth Archilecuur
Peconstuctio
Phose
Figure 3: RTFS Block design. After compressing the data to more efficient size, we process first the frequency dimension, then the time dimension, then both dimensions in tandem using TF- domain self-attention to capture deer-dependescces. We then carefully restore the data its original dimensions using our Cemporal-Frequeng Attention Reconstruction units.
Compression of time and frequency resolution. We use 2D convolution with to convert A to a smaller channel dimension D < C.. This means we can effectively employ a larger Ca value for detailed fusion (CAF) and separation (S²), while maintaining lightweight and efficient block design. Similar to Li et al. (2023). in the compression phase we employ q stacked 2D depth- wise convolutional layers with kernels and stride 2, see Appendix C for the effects
kernel

---

Published as a conference paper at ICLR 2024
of different q values. The resultant scati-scale set with varying temporal and frequency resolutions can be denoted {Aili E { }}}, where A, E IR Dx x We use adaptive average pooling compress each member of the obtain the compressed global features Ac. This is written:
the dimensions of the smallest member, A then sum to
1
AG = Σ p(A.), AG E RR D i=0
(12)
Dual-path architecture. The Dual- -ath RNN architecture has been extensively deployed in AOSS tasks (Luo et al., 2020) Chen al., 2020, Wang et al. 2023). However, their usage of large LSTMs leads to high parameter counts and an elevated computational complexity. In natural language pro- cessing, Simple Recurrent Units (SRU (Le et al. 2018)) were introduced to replace LSTTMs by processing most of the operations in parallel, speeding up model training and inference time. In- spired by this, we adopt SRUs for the AVSS task, see Appendix D for a detailed analysis between different recurrent architectures.
we first process the frequency dimension, then the time dimension. As with all dual-path methods, the SRUs are applied across each slice in the time dimension to process the frequency dimension. Similar to /Wang et al. 2023). we unfold" the features by zero pero-padding the frequency dimension of Ag, then unfolding with kernel size 8 and stride 1,
As seen in Figure 3
1a/20-1]] R = [Untol((Ac(t, t E (0/
E 8D T XF
(13)
where F" is the resulting padded and unfolded frequency dimension. Note that while this process results in high input dimensions for the SRUs, which significantly increases computational complex- ity, it is essential for achieving outstanding separation performance. This is why it is important that we first compress the time-frequency resolution. After unfolding, layer normalization is applied in the channel dimension, and then a bidirectional, 4 layer SRU with hidden size ha is applied,
R SRU (R 4), L LE (O....,T.T T./29 1]]
E R22 3 XF
(14)
A transposed convolution T with kernel 8 and stride is used to restore the unfolded dimensions, - T (R,) + AG, R e TR. Px 20_1 29-1
(15)
We next process the time dimension using the same method, and then finally apply Wang et al. 2023 's TF-domain self-attention network, denoted Attn. These two steps are expressed below as: T
T ) R AG= = Attnt R.) + R., R,, AG E Dx =ff..
(16)
Reconstruction of time and frequency resolution. Reconstruction of high-quality temporal and frequency features presents a formidable challenge. Delving into the underlying causes, the recon- struction process often relies on interpolation or transposed convolutions for up-sampling, resulting in the emergence of checkerboard artifacts in the reconstructed outputs. To solve this problem, we propose the prioritizes the reconstruction of key features by exploiting an attention mechanism, thus reducing information loss. For two tensors m and n, we define the TF-AR unit as:
Attention Reconstruction (TF-AR) unit, denoted (-.-). This unit
(17) where (-).). (-)() and W.(.) denote 2D depth-wise convolutions with 4 x 4 kemels followed by gLN layer. We use the notation for the sigmoid function, @ for element-wiss multiplication and ó for nearest neighbour interpolation (up-sampling). To conduct the reconstruction, we firstly use TF-AR units to fuse AG with every element of A.,
1(m, n) = ó (a (W1 (n))) @ W2 (m) + + (W' (n))
A (A,, Ac), A E Rx Vi E 90,
(18)
Next, the multi-scale features are continuously up-sampled and aggregated using q - additional TF-AR units to obtain the finest- finest-grained auditory features, A" E E kDxx,xx xF A residual connection to {A E {0, q 2}} is crucial, and creates U-Net (Ronneberger et al. 2015) style structure:
A- = I(A A/_ + Aq-1-i, Vi (0.... (o......q 2).
(19)
Finally, A" is converted back to C, channels using 2D convolution with 1 x 1 kernel and residual connection to the input of the RTFS Block is added. The output features are used as the audio input for the CAF Block after the AP Block, and as the input to the next RTFS Block during the repeated R stacked RTFS Blocks stage of the separation network.
https::/pytorrch.orgg///// htt.://////.....eeeeeee///e////a
http:///ss....ooooos/s.ss/////tt unw.onn.l..n.o..mm..m hmml

---

Published as a conference paper at ICLR 2024
3.3 SPECTRAL SOURCE SEPARATION
The majority of existing T-domain AVSS methods (Wu et al. 2019 Li et al. 2022 Martel et al., 2023) generate a mask from the refined features aR, then use element-wiss -iie multiplication @ between the encoded audio mixture 4g and the mask in order to obtain the target speaker's separated speech 2. This is written.
(20)
Some TF- domainnin AVSS methods (Afouras et al., 201 8a Gao & Grauman 2021 Lee et al., 2021) directly apply this approach to the TF-domain setting without modification, while other methods Alfouras et al. 2018 Owens & Efros 2018) choose not to use masks at all, and directly passes the output of the separation network aR to the decoder. However, we found both of these TF- domain methods for target speaker extraction to be suboptimal. We need to pay attention to the underlying complex nature the audio features produced by the STFT in order to obtain a clearer distinction. This leads us to introduce our S Block, which utilizes high-dimensional application of the multiplication of complex numbers, Equation 21 to better preserve important acoustic properties during the speaker extraction process, see Section 5.2 and Appendix E
(a + bi)(c + di) = ac bd + i(ad + bc).
(21)
Firstly, a mask is generated from ag using a 2D convolution M with 1 kernel,
m ReLU (M (PReL.(aa)))
E E C.xTTXXF.
(22)
Without loss of generality, we choose the top half of the channels as the real part, and the bottom half of the channels as the imaginary part. We hence define,
m m [0 C2/2 1],
E' = 00: Ca/2-1]
(23)
m' C2.2 Ca),
E /Ca/2 Ca].
(24)
Next, with || denoting concatenation along the channel axis, we apply Equation 21 and calculate:
z' = m'
m
E
(25)
z' m ② m x R to obtain 2, the target speaker's 's separated encoded audio features.
+ E
(26)
(z
(27)
3.4 DECODER
The decoder D(-) takes the separated target speaker's audio features and reconstructs the estimated waveform ø = (z), where 8 E R glxx,, Specifically, z is passed through a transposed 2D convolu- tion with 3 x 3 kernel and 2 output channels. Mirroring the encoder, we take the first channel as the real part and the second channel as the imaginary part and form a complex tensor. This tensor is passed to the iSTFT recover the estimated target speaker audio.
4 EXPERIMENTAL SETUP
Datasets. We utilized the same AVSS datasets as previous works (Gao & Grauman 2021 Li et al, 2022 in the field in order to create fair comparison of performance: LRS2-2Min (Afouras al, 2018a), LRS3-2Mix (Afouras et al.) 2018b) and VoxCeleb2-2Mix (Chung et al. 2018). 8). The models were trained and tested on two second 251ps video clips with an audio sampling rate of 16 kHz. This equates to 32,000 audio frames and 50 video frames, see Appendix F for more details.
Evaluation. Following recent literature (L et al.) 2022), SI-SNRi and SDRi were used evaluate the quality of the separated speeches. see Appendix G We also provided PESQ (Rix et al., (00T)) for reference. For these metrics, higher value indicates better performance. The parameter counts displayed in the results tables are the number of trainable parameters, excluding the pretrained video model. Likewise, the number of Multiply-ACeumulle
(MAC) operations indicates the MACs used while processing two seconds of audio at 16 kHz, excluding the pretrained video network. In our main results table, we also include inference time: the time taken to process 2 seconds of audio on a NVIDIA 2080 GPU. For parameters, MACs and inference speed, lower value is preferable and is the main focus this work. Model hyperparameter configurations are available in Appendix All training settings are available in Appendix
7

---

Published as a conference paper at ICLR 2024
5 RESULTS
5.1
COMPARISONS WITH STATE-OF-THE-ARY METHODS
The goal of this research is to obtain SOTA performance while significantly reducing computational complexity and model size with a lightweight, streamlined approach. In the comprehensive com- parison presented in Table I we evaluated RTFS-Net against a range of SOTA AVSS methods. We explored three variants of RTFS-Net, corresponding to R {4, 6, 12) RTES Blocks, including the AP Block. On the LRS2-2M 2Mix dataset, RTFS-Net-4 demonstrated an SI-SNRi of 14.1 dB, only marginally lower than the 14.3 dB achieved by the previous SOTA, CTCNet. However, this was achieved with a tenfold reduction in model parameters and an eightfold decrease in computational cost. Meanwhile, RTFS-Net-6 surpassed CTCNet on the LRS2-2Mix dataset and delivered com- parable results on VoxCeleb2-2Mixx striking good balance between performance and efficiency. RTFS-Net-12 outperformed all other techniques across all datasets, showcasing its superiority in complex environments and the robustness of our TF-domain approach. Moreover, despite utiliz- ing 12 layers, CTFS-Net-1R 12 still manages reduce the computational costs of CTCNet by threefold while using only a tenth of the parameters. To our knowledge, RTFS-Net is the first AVSS method to employ fewer than million parameters, and the first TF-domain model to outperform T-domain counterparts.
LRS2-2Mix
LRS3-2M1
VoxCeleb2 2Mix
Params MACs Time
Model
SI-SNRi SDRi PESQ SI-SNRi SDRi PESQ SI-SNRi SDRi PESQ (M) (G)
(ms)
CaffNet-C* 2021
12.5
1.15
12.3
Thanh-Dat 2021
11.6
AV-Conv ConyTasner2200
12.5
11.2
11.7
16.5
60.3
Vosuel 20ice/022 2021
11.5
11.8
2.78
9.9
10.3
9.3
10.2
77.8
130.2
AVLIT 2023
12.8
13.1
2.56
13.5
13.6
2.78
9.4
9.9
2.23
5,8
36.4
53.4
(202
14.3
14.6
3.08
17.4
17.5
3.24
11.9
13.1
3.00
7.0
167.2
122.7
RTFS-Nets
14.1
14.3
3.02
15.5
15.6
3.08
11.5
12.4
2.94
0.7
21.9
57.8
RTFS-Nct-6
14.6
14.8
3.03
16.9
17.1
3.12
11.8
12.8
2.97
0.7
30.5
64.7
RIFS-Net-12
14.9
15.1
3.07
17.6
3.25
12.4
13.6
3.00
0.7
56.4
109.9
Table 1: Comparison of RTFS-Net with existing AVSS methods on the LRS2-2Mix, LRS3-2MiX and VoxCeleb2-2Mix datasets. These metries averaged across all speakers for each test set, larger SI-SNRi and SDRi values indicate better performance. '-' indicates the results are not reported in the original paper. *** indicates that the audio is reconstructed using the ground-truth phase.
Important Note: we investigated higher R values in Appendix J and found that RTFS-Net-20 outperform CTCNet by substantial 1.1 dB SI-SNRi and 1 dB SDRi. In Appendix K randomly selected two long, uninterrupted audio samples from the VoxCeleh2 test set and mixed them together generate sample outputs for RTFS-Net, AV-ConvTasNee. VisualVoice. AVLIT and the previous SOTA method, CTCNet. In Appendix D we studied additional downstream tasks to test the generalizability of RTFS-Net.
5.2
ABLATION STUDY
LRS2-2Mix SI-SNRi SDRi
RTFS-NG RTFS-Net/ Params (K) MACs (G)
Fusion Params (K)
Fusion MACs (M)
AV Fusion Strategies
CTCNet Fusion (adapted)
11.3
11.7
528
14.3
197
6365
CAF Block (ours)
11.7
12.1
339
8.0
7
83
Table 2: Comparison of audio-visual fusion strategies on the LRS2-2Mix dataset. RTFS-Net Param- elers and MACs indicate the entire network structure. The Fusion Purameters and MACs represent those used only during the fusion process.
cross-dimensional Attention Fusion (CAF). To demonstrate the value of our CAF Block, we used a reduced version of RTFS-Net- that limited the power of the separation network and em- phasised the fusion network. Appendix H.I As baseline, we substituted the CAF block with a To-domain adaptation of CTCNet's fusion strategy. The previous SOTA method, CTCNet, uses concatenation approach: it uses interpolation upscale the video dimensions (C, x T.) to the
8

---

Published as a conference paper at ICLR 2024
dimensions of the audio (C, X T.). concatenates along the channels and then uses convolution to restore the audio channel dimension, Ca. Similar to Equation g we adapted this to a 3D TF- domain audio setting. In Table we observed that our CAF Block outperformed the bascline by large mar- gin while reducing the number of parameters by 96.4% % and cutting the numher of MACs by 98.7%, suggesting multimodal fusion be effectively performed with well-designed, small network.
Cemporal-Frequenee Attention Reconstruction (TF-AR). To test the efficacy of our RTFS Block's reconstruction method, i.e. the TT-AR units, compared RTFS-Net-4 with a baseline that used interpolation and addition to perform the work of the TF-AR units, similar to the up- sampling process in U-Net (Ronneberger al., (013). We observe in Table that our TF-AR units boosted performance by a substantial 1dB in both performance metrics with an almost negligible affect on computational complexity due to the depth-wise convolutional nature of our TF-AR units.
LRS2-2Mix SI-SNRi SDRi
RTFS-Net RTFS-Net TE-AR
TE-AR
TF-AR
Params (K) MACs (G) Params (K) MACS (M)
Without
13.0
13.3
729
21.4
With (ours)
14.1
14.3
739
21.9
10
494
Table 3: Validation for the effectiveness of the TF-AR units the LRS2-2Mix dataset. The TF-AR Parameters and MACs represent those used only in the TF-AR block.
Spectral Source Separation (S*). In these experiments we further reduced the model configuration seen in Appendix H.l setting C. = 128. tested our 5 Block against four alternative methods. Regression gives a directly to the decoder decode into the target speaker's audio. Mask is the approach used by many SOTA AOSS methods such as Luo et al. (2020) and discussed Section in Luo et al.
3.3 Mask Gate and Mask DW-Gate both apply convolutional gate, (2019), after the mask. The mask is fed into two convolutions with respective Tanh and Sigmoid activations. The outputs are multiplied together to form the final mask. DW here indicates the usage of west convolutions.
Target Speaker
LRS2-2Mix SI-SNRi SDRi
RTFS-Net
RTFS-Net
Extraction Extraction
Extraction Method
Params (K)
MACs (G)
Params (K)
MACs (M)
Regression
10.0
9.9
208
3,0
0
Mask
10.8
11.2
224
3.6
16
534
Mask DW-Gate
10.8
11.3
225
3.6
542
Mask + Gate
11.1
11.6
257
1595
5d (ours)
11.3
11.7
224
3.6
16
534
Table 4: Comparison source separation methods on the LRS2-2N5 LRS2-2Mix dataset. The Extraction Parameters and MACs represent those used only in the Target Speaker Extraction Method.
Table shows the Regression approach was the least effective, with significantly lower metrics than the other methods. However, it also utilized the fewest MACs and parameters. For a very small increase in parameters, the Mask and Mask+DW-Gard approaches yielded far better results. A non depth-wise gate increased the performance further, but the number of MACs and parameters is significantly higher. However, all methods were eclipsed by the performance of our S Block. This mathematics- based approach does not significantly increase the parameter or MAC count, thereby providing performance enhancement all To-domiin AVSS methods no additional cost likely benefiting AOSS methods as well. A visualization is available in Appendix E
6 CONCLUSION
In this study we introduced RTFS-Net, a novel approach to AVSS that explicitly models time and frequency dimensions at a compressed subspace to improve performance and increase efficiency. Empirical evaluations across multiple datasets demonstrate the superiority of approach. No- tably, our method achieves remarkable performance improvements while maintaining a significantly reduced computational complexity and parameter count. This indicates that enhancing AVSS per- formance does not necessarily require larger models, but rather innovative and efficient architectures that better capture the intricate interplay between audio and visual modalities.

---

Published as a conference paper at ICLR 2024
REPRODUCIBILITY STATEMENT
The code for RTFS-Net was written in Python 3.10 using standard Python deep learning libraries, specifically PyTorch and Py PyTorch Lightning. In order to accommodate full reproducibility, will open-source the code for RTFS-Net under the MIT licence on GitHub once this paper has been ac- cepted into the conference. The code shall include in Table including the conda environment we used to run the code, the full config files for RTFS- Net-4, RTFS-Net-6 and RTFS-Net- 12, the weights for the pretrained video model and the RTFS-NO Net model code itself, including TDANet and all layers, blocks and networks mentioned in this paper. Datasets must be obtained separately from the references provided, see Appendix [ as they are the property of the respective publishers, but we provide the data preprocessing scripts alongside this code-base. The GPU optimized PyTorch implementation of the SRU is already open-source and available on PyP[] supplied by the original author. Experimentation and training accomplished using a single server with NVIDIA 3080 GPUs, see Appendix [ for additional details. For those who wish to recreate the code themselves, all hyperparameters are listed in Appendix [H] Evalua- tion metrics and loss functions are described and mathematically defined in Appendix L.1 and G respectively, but we will additionally provide these in the code-base.
files used reproduce the experiments seen
ACKNOWLEDGEMENTS
This work was supported in part by the National Key Research and Development Program of China (No. 20212I0200301) and the National Natural Science Foundation of China (Nos. 62061 136001, 61836014).
REFERENCES
Triantafyllos Afouras, Joon Son Chung, Andrew Senior, Oriol Vinyals, and Andrew Zisserman. Deep audio-visual speech recognition. IEEE Transactions on Pattern Analysis and Machine In- telligence, 44(12):87178877, 2018a.
Triantafyllos Afouras, Joon Son Chung, and Andrew Zisserman. Lrs3-ted: large-scale dataset for visual speech recognition, 2018b.
Triantafyllos Afouras, Andrew Owens, Joon Son Chung, and Andrew Zisserman. Self-supervised learning of audio-visual objects from video. Proceedings of the European Conference on Computer Vision (ECCV), pp. 908-224. Springer, 2020.
T Alfouras, JS Chung, and A Zisserman. The conversation: deep audio-visual speech enhancement. In Interspeech, volume 2018, 2018.
Adelbert Bronkhorst. The cocktail party phenomenon: A review of research on speech intelligibility in multiple-talker conditions. Acta Acustica united with Acustica, 86:117-128, 01 2000.
Jingjing Chen, Qirong Mao, and Dong Liu. Dual-path transformer network: Direct context-aware modeling for end-to-end monaural speech separation. In Proc. Interspeech 2020, pp. 2642-2646,
2020. doi: 10.214377/nnersseechh.o https/ppecc...20-2222 2020-2205.
E. Colin Cherry. Some Experiments on the Recognition Speech, with One and with Two Ears. The Journal of the Acoustical Society of America, 25(5):975-979, 06 2005. ISSN 0001-4966.
J Chung, A Nagrani, and A Zisserman. Voxceleb2: Deep speaker recognition. In Interspeech, 2018.
Martin Cooke, Jon Barker, Stuart Cunningham, and Xu Shao. An audio-visual corpus for speech perception and automatic speech recognition. The Journal of the Acoustical Society of America, 120k5):2421-2424 2006.
Ruohan Gao and Kristen Grauman. Visualvoice: Audio-visual speech separation with cross- modal consistency. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15490-15500. IEEE, 2021.
10

---

Published as a conference paper at ICLR 2024
Simon Haykin and Zhe Chen. The Cocktail Party Problem. Neural Computation, 1779):1875-1922, 09 2005. ISSN 0899-7667.
John R Hershey, Zhuo Chen, Jonathan Le Roux, and Shinji Watanabe. Deep clustering; Discrimina- tive embeddings for segmentation and separation. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 31-35. IEEE, 2016.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8): 1735-1780, 1997.
Xiaolin Hu, Kai Li, Weiyi Zhang, Yi Luo, Jean-Marie Lemercier, and Timo Gerkmann. Speech sep- aration using an asynchronous fully recurrent convolutional neural network. Advances Neural Information Processing Systems (NeurIPS), 34:22509-22522, 2021.
Jonathan Le Roux, Scott Wisdom, Hakan Erdogan, and John R Hershey. Sdr-half-baked well done? In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 626 630. IEEE, 2019.
Jiyoung Lee, Whon Chung, Sunok Kim, Hong-Goo Kang, and Kwanghoon Sohn. Looking into your speech: Learning cross- modal affinity for audio-visual speech separation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1336-1345. 1345. IEEE, 2021.
Tao Lei, Yu Zhang, Sida I Wang, Hui Dai, and Yoav Artzi Simple recurrent units for highly par- allelizable recurrence. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4470-4481, 2018
Kai Li, Fenghua Xie, Hang Chen, Kexin Yuan, and Xiaolin Hu. An audio-visual speech separation model inspired by catiioo-hhllmm-cortiiee circuits. arXiv preprint arXiv:2212.10744, 2022.
Kai Li, Runxuan Yang, and Xiaolin Hu. An efficient encoder-decoder architecture with top-down attention for speech separation. In International Conference Learning Representations (ICLR), 2023.
Jiuxin Lin, Xinyu Cai, Heinrich Dinkel, Jun Chen, Zhiyong Yan, Yongqing Wang, Junbo Zhang, Zhiyong Wu, Yujun Wang. and Helen Meng. Av-sepformer: Cross-attention sepformer for audio- visual target speaker extraction. In ICASSP 2023-2023 IEEE International Conference on Acous- tics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Philipos C Loizou. Speech enhancement: theory and practice. CRC press, 2013.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations (ICLR), 2018.
Yi Luo and Nima Mesgarani. Tasnet: time-domain audio separation network for real-time, single- channel speech separation. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 696-700. IEEF, 2018
Yi Luo and Nima Mesgarani. Conv-tasnet: Surpassing ideal time free-fryyynccy magnitude masking for speech separation. IEFF/ACM Transactions on Audio, Speech, and Language Processing, 27) 1256-1266, 2019.
Yi Luo and Jianwei Yu. Music source separation with band-split mnn IEEF/ACM Trunsactions on Audio, Speech, and Language Processing, 31:1893-1901, 2023. doi: 10. 109/TAS1.P2023. 3271145.
Yi Luo, Cong Han, Nima Mesgarani, Enea Ceolini, and Shih-Chii Liu. Fasnet: Low-latency adap- tive beamforming for multi-min multi-microopone audio processing. In Automatic Speech Recognition and Understanding (ASRU), pp. 260-267. IEEE, 2019.
Yi Luo, Zhuo Chen, and Takuya Yoshioka. Dual-path mn: efficient long sequence modeling for time-domain single-channel speech separation. ICASSP 2020-2020 IEEE International Con- ference Acoustics, Speech and Signal Processing (ICASSP), pp. 46-50. IEFF, 2020.

---

Published as a conference paper at ICLR 2024
Héctor Martel, Julius Richter, Kai Li, Xiaolin Hu, and Timo Gerkmann. Audio-Visual Speech Separation in Noisy Environments with Lightweight Iterative Model. In Proc. Interspeech 2023, pp. 1673-1677, 2023.
Andrew Owens and Alexei A Efros. Audio-visual scene analysis with self-supervised multisensory features. In Proceedings of the European conference on computer vision (ECCV), pp. 631-648, 2018.
Antony W Rix, John G Beerends, Michael P Hollier, and Andries P Hekstra. Perceptual evaluation of speech quality (pesq)-a a new method for speech quality assessment of telephone networks and codecs. In 2001 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Proceedings (Cat. No. OICH37221), volume 2, pp. 749-752. IEEE, 2001.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomed- ical image segmentation. In Medical Image Computing and Computer-Assisted Intervention- MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceed- ings, Part III 18, pp. 234-241. Springer, 2015.
Joon Son Chung, Andrew Senior, Oriol Vinyals, and Andrew Zisserman. Lip reading sentences in the wild. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (CVPR), pp. 6447-6456, 2017.
Cem Subakan, Mirco Ravanelli, Samuele Comell,, Mirko Bronzi, and Jianyuan Zhong. Attention is all you need in speech separation. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 21-25. IEEE, 2021.
Thanh-Dat Truong, Chi Nhan Duong, Hoang Anh Pham, Bhiksha Raj, Ngan Le, Khoa Luu, et al. The right talk: An audio-visual transformer approach. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision (ICCV), pp. 1105-1114, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz. Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Informa- fion Processing Systems (NeurIPS), 30, 2017.
Emmanuel Vincent, Rémi Gribonval, and Cédric Févotte. Performance measurement in blind audio source separation. IEEF/ACM Transactions on Audio, Speech, and Language Processing, 14(4): 1462-1469, 2006.
DeLiang Wang and Guy Brown. Computational auditory scene analysis: Principles, algorithms, and applications. Wiley-IEEE press, 2006.
Zhong-Qiu Wang, Samuele Cornell, Shukjae Choi, Younglo Lee, Byeong-Yeas Yeol Kim, and Shinji Watanabe. Tf-gridnet: Making fime-freyuency domain models great again for monaural speaker separation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Sig- nal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Jian Wu, Yong Xu, Shi-Xiong Zhang, Lian-We Wu Chen, Meng Yu, Lei Xie, and Dong Yu. Time domain audio visual speech separation. In 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 667-673, 2019.
Lei Yang. Wei Liu, and Weiqin Wang. Tipsnet: fime-frey domain path scanning network for speech separation. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6842-6846. IEFE, 2022.
Kaipeng Zhang, Zhanpeng Zhang, Zhifeng Li, and Yu Qiao. Joint face detection and alignment using multitask cascaded convolutional networks. IEEE signal processing letters, 23(10):1499 499-1503, 2016.
12

---

Published as a conference paper at ICLR 2024
A VIDEO PREPROCESSING BLOCK
The VP Block a ID version of the RTFS Block, which is similar the TDANet Block architecture [L ct al. 2023).
Compression of time resolution. Here, we change all 2D convolutions to ID convolutions. As result, we obtain the compressed global video features Va. This is written:
VG Σ PVV.), 1=0
Ve E DX 2011
(28)
Dual-path architecture. Since there only one compressed dimension, we no longer need dual path architecture. Following TDANet, obtain
V G (V G) + VG.
V
(29)
where Attn denotes the transformer used in TDANet, which consists of multi-head self-attention (MHSA) and a convolutional feed-forward network (FFN), with respective dropout and residual connections.
This attentional mechanism was first introduced to AOSS by DPTNet (Chen et al. 2020). but Li
al. (2023) found the subsequent bidirectional LSTM used in this implementation had negligible impact on performance, and could be replaced with lightweight series three 1D convolutions. See the paper and open-source codo more details.
Reconstruction of time and frequency resolution. There is no frequency dimension, just the time dimension. We form ID versions of the the TF-AR units by replacing the 2D convolutions and interpolations with ID equivalents. The following reconstruction process is exactly the same, and so are all the equations in this section. VG hence fused with each member of the multi-scal set, and then fused into single feature map V
B SHARING PARAMETERS
Shared Parameters
LRS2-2Mix SI-SNRi SDRi
Params
MACs
(K)
(G)
Not Shared
13.6
13.9
2,183
21.9
Shared
13.7
14.0
739
21.9
Table 5: Effects of sharing parameters between the RTFS Blocks on the LRS2-2MiX dataset. We experimented with RTFS-Net-4.
In table we consider the effects of sharing parameters between the RTFS Blocks, including the AP block. Sharing parameters means we instantiate single RTFS Block instance, and use this single block as the AP Block and all of the subsequent RTFS Blocks after the fusion. In other words, we take the output of the first RTFS Block, add the residual connection, and then pass that data to the exact same block again as an input. This means that all RTFS Blocks share the same weights and biases. Not sharing the parameters means that we instead instantiate R separate RTFS Blocks, and iterate through them in turn.
Echoing the findings of Li et al. 2022). ) our study observed small but significant increase in model performance, evidenced by the 0.1 dB increase both SI-SNRi and SDRi. Addition-
ally. sharing the parameters cuts the model size down from 2 million parameters down to just 739 thousand. The number of MACs is unchanged, as the number of calculations remains the same.
C RESULTS FOR VARIOUS COMPRESSION SCALES
In Table 6 we compared using different compression levels g using the same RTFS-Net-4 configura- tion Table Increasing the compression factor results in a large drop in performance. However,
httpp::/ /ttt
13

---

Published as a conference paper at ICLR 2024
the reduction in computational complexity is correspondingly large. RTFS-Net with 2 and both outperform the previous TF-domain SOTA method, VisualVoice (Gao Grauman 2021 As seen in Table [ adding more layers (increasing R) will increase performance significantly, using 12 or 16 layer configurations with a larger q value would also be a valid option. However, we chose lower q and R for simplicity. The RTFSSNett with q 1 shows the result of applying TF-GridNeet Waan et al. (023) directly to the AVSS setting. This model is powerful, and so with only 4 layers nearly obtains the performance of our RTFS-Net-12 model in Table However, the memory con- sumption is extremely large, and the model is extremely slow to train. A batch size of 1 is required for most standard GPUs. In addition, the purpose of this research is not to outperform the previous SOTA method, but to provide a lightweight and efficient solution to AVSS problem.
LRS2-2Mix
RTFS-Net RTFS-Net
Total Block Total Block Training Params (k)
SI-SNRi SDRi
Params (K)
MACs (G)
MACs (M) Memory (GB)
1
14.7
14.9
747
60.5
489
55.9
5.74
2
14.1
14.3
739
21.9
481
17.3
3.99
3
12.8
13.1
737
12.4
479
7.8
3,44
11.5
12.0
740
10.2
482
5.5
3.33
5
11.0
11.4
746
9.7
487
5.0
3.29
Table 6: Comparison of different q values (number of up-sampling steps) on the LRS2-2Mix dataset.
The Total Block Parameters and MACs represent the parameters and MACs used in the R stacked RTFS Blocks.
D STRUCTURES FOR INDEPENDENTLY MODELING TIME AND FREQUENCY
In Table 7 we examine the effects of using different RNN structures as described Equation 14 Each model employs the same RTFS-Net configuration as presented in Table [ We observe that both LSTM and SRU yield similar results when modeling the time and frequency dimensions. How- ever, SRU demonstrates advantages over LSTM in terms of parameters, computational complexity, inference time, and training memory usage.
Additionally, we tested the effects of employing transformers. Specifically, we replaced the inter and intra SRUs with DPTNet (Chen et al., (0200) and adopted the transformer architecture from TDANet (Li et al. 2023) to avoid using heavy BiLSTM operations pOSt-MHSA. The fime-frey domain transformer Wang et al. (2023) remained unchanged following the dual-path processing. In the table, this model is denoted as MHSA it possesses fewer parameters and half the number of MACs, but shows an increase in training memory due to the transformer operations. Additionally, both the SI- SNRi and the SDRi metrics exhibited 1dB decrease in performance, significant drop, leading us to opt against this architecture despite its compact size.
We hypothesized that the unfolding before both SRUs might enhance transformer performance as well. However, this approach resulted in an extremely large model in all metrics: the training mem- ory surged 5GB, and the parameter count nearly reached million. This configuration applies only the MHSA component of the transformer, Incorporating the FFN (MHSA- (nHos--nfold+FFN led to an even larger and slower model. Despite these modifications, the performance remained substan- tially lower than that of the SRU configuration.
We can conclude that CTFS-Net's improvement in separation quality comes from the model design, not the choice of RNN. The SRU is implemented specifically to improve inference time and reduce the computational complexity.
E LOSS OF AMPLITUDE AND PHASE INFORMATION IN THE TRADITIONAL MASK APPROACH
In Figure 4 we examined the separation quality of two different target speaker extraction methods: our proposed 3 Block and the traditional masking technique discussed in Section 3.3] For these results, reused the 3 and Mask models trained the LRS2-2Mix dataset and shown in Table
14

---

Published as a conference paper at ICLR 2024
Structure Type
SI-SNRi SDRi
Params (K)
MACs (G)
Time (ms)
Training Memory (GB)
RNN
13.5
13.8
509
15.2
58.73
3.85
GRU
13.8
14.1
724
21.6
64.10
4.04
LSTM
14.0
14.3
832
24.8
61.92
4.08
MHSA
13.1
13.4
404
11.1
4.09
MHSA-Unfold
12.8
13.1
2,966
27.0
5.00
FFN
13.1
13.4
4,028
58.0
5.65
SRU
14.1
14.3
739
21.9
57.84
3,95
Table 7: Comparison of structures for modeling time and frequency on the LRS2-2Mix dataset. Here, Time denotes the time required for inference at 2s audio input. Training Memory denotes the GPU memory used for training with batch size of 1.
We then selected three audio mixtures from the VoxCeleb2-2NM dataset, creating an extremely challenging setting for the models, and applied the two models on each mixture.
In Figure 4a we can see that the traditional masking approach completely failed to capture the gold, vertical column in the middle of the expanded section. On the other hand, our S not only captured this information, but also managed to preserve the striations seen in the original spectrogram.
In Figure 4b we see a similar result, our S approach managed to accurately capture the gold stripes seen just right of the centre of the expanded section, whereas the the traditional masking approach completely excluded this part of the audio signal, resulting in purple/black region in its place.
In Figure we can see that our 5 approach managed to accurately maintain the complex horizontal striations of the original audio, whereas the traditional masking approach produced much more blurred result, with large purple regions showing it completely disregarded large chunks of important acoustic information.
The spectrograms in Figure 4 alongside the clear performance advantage seen in the evaluation metrics of Table 4 show the importance of respecting the underlying complex nature of the audio features obtained from the STFT. Failing to do so results in critical acoustic information being lost during the element-wise multiplication of the traditional masking approach, and hence the result obtained from the iSTFT is visibly degraded.
F DATASET DETAILS
We conducted our AVSS experiments on the LRS2 (Afurass et al. 2018a), LRS3 (ffouras et al., 2018b), and VoxCeleb2 (Chung et al. 2018) datasets. The different speakers are separated into different folders, allowing us to explicitly test speaker-independer performance. Additionally, we used a dataset partitioning consistent with existing AVSS methods to ensure fairness in obtaining results. For lip motion video acquisition, we used the FFmpeg too f to change the sampling rate of all videos to 25 FPS. We then used an existing face extraction network (Zhang et al., (016) to segment the lip region from the image and adjusted it 96 x 96 grayscale frame that contains only the image frames of the speaker's 's lip motion. For audio extraction, we similarly extracted the audio from the video using the FFmpeg tool and uniformly changed the sampling rate to 16 KHz. For comparison with existing AVSS methods, we used a mix of two speakers for training, but this dataset creation method can be used to create datasets for three more speakers
The LRS2-2MiX (Afouras et al.] 8a) dataset is derived from BBC television broadcasts. The videos contain variety of natural speakers and lighting conditions, and the audio contains back- ground noise to bring it closer to real-world environments. Compared to previous lip-reading datasets (e.g. GRID (Cooke et al., 20066) and LRW (Son Chung et al. 2017)), LRS2-2Mix con- lains unrestricted vocabulary. It has full sentences rather than isolated words and shows greater diversity terms of speakers and recording conditions. The dataset contains 11 hours of training, hours of validation, and 1.5 hours of testing data.
https://ffmpeg..n
15

---

Published as a conference paper at ICLR 2024
The LRS3-2Mix (Afouras et al., 2018b) dataset contains over 400 hours of video extracted from 5,594 TED and TEDx langush talks downloaded from YouTube. TED talks are profes- sionally produced and speakers typically microphones, it represents relatively clean envi- ronment. The dataset includes 28 hours of training, 3 hours of validation and 1.5 hours testing data.
The VoxCeleb2-2Mix (Chung et al. 2018) dataset was collected from YouTube videos containing naturally occurring noises such as laughter, cross-talk and different room acoustics. This provides a diverse and challenging set of speech samples for training the separation system. In addition, the faces in the videos differ in terms pose, lighting, and image quality. This suggests that VoxCeleb2- 2Mix provides more challenging real-world environment. The dataset consists of 56 hours of training, 3 hours of validation and 1.5 hours of testing data.
G EVALUATION METRICS FOR AUDIO-VISUAL QUALITY
Following recent literature (Li al. 2022), the scale-invariant signal-to-noise ratio improvement (SI-SNRi) and signal-to-noise ratio improvement (SDRi) were used to evaluate the quality of the separated speeches. These metrics were calculated based on the scale-invariant signal-to-noiss ratio (SI-SNR) (Le Roux et al., 2019) and source-to-distorii ratio (SDR) (Vincent et al., 2006). For these evaluation metrics, a higher value indicates better performance. SI-SNRi and SDRi are defined
SI-SNRi(z, 8, 8) SI-SNR 8) - SI-SNR(s, z),
(30)
SDRi(1, 8, 8) SDR(s, 8) SDRR(s, x).
H MODEL HYPERPARAMETERS
For all model versions, R E {4,6, 12), used the same hyperparamer settings. Encoder. The STFT used a Hanning analysis window with a window size of 256 and a hop length of 128. The encoded feature dimension was C₂- 256.
VP Block. We used a TDANet block (Li et al., 2023) with a hidden dimension of 64, a kernel size of 3, an up-sampling depth of 4, replaced the gLNs with batch normalization and set the number of attention heads in the MHSA to 8. The feed forward network (FFN) had 128 channels. Model code was supplied by the original auther.
AP and RTFS Blocks. We used a hidden dimension of D = 64 as the reduced channel dimension for the RTFS Blocks. We chose q = 2 up-sampling layers in order halve the length of the time and frequency dimensions, see Appendix C for different choices of q. The 4-layer bidirectional SRUs had hidden dimension of ha 32, and an input dimension of 64 used the 4-ha attention configuration for the TF-domain attention (Wang et al. 2023), as defined in their paper. The model code for the TF-domain attention was taken from the ESPnet GitHub project where it was published after the release of Wang et al. (2023)'s 's paper.
256 after unfolding. We
CAF Block. We used h attention heads.
H.1 REDUCED MODEL
The reduced model used in Table 2 was achieved by using modified version of the RTFS-Net-4 configuration defined above. Firstly, we set the hidden dimension of the RTFS Blocks to D = 32, replaced the multi-layer bidirectional SRUs with single layer unidirectional SRUs, and changed the kernel size and stride of the unfolding stage in Equation 13 4 and respectively. This configuration also used in T but with C2 128.
16

---

Published as a conference paper at ICLR 2024
I TRAINING HYPERPARAMETERS
For training we used batch size of 4 and A (Loshchilov & Hutter 2018) optimization with a weight decay of 1 x 10-1 The initial learning rate used was 10 but this value was halved whenever the validation loss did not decrease for 5 epochs in row. We used gradient clipping to limit the maximum L2 norm of the gradient to 5. Training was left running for a maximum of 200 epochs, but early stopping also applied. The loss function used for training was the SI- SNR (Le Roux et al. (2119) between the estimated & and original target speaker audio signals, see Appendix I.1
1.1 TRAINING OBJECTION FUNCTION
The training objective used is the scale-invariant source-to-noise ratio (SI-SNR) between the target speaker's s original audio signal and the estimate returned by the model &. The SI-SNR is defined follows,
812 | w s|
||
S
SI-SNR(s, (s, 8) = 10 log
w
(31)
s
J INCREASING THE NUMBER OF LAYERS
Obtaining SOTA performance was not the focus of this paper. Rather, we aimed to produce a model with high efficiency while utilizing small number of parameters. However, ablation study explore higher values of R in Table 8 including R 16 and R = 20.
Model
SI-SNRi SDRi Params (K) MACs (G)
CTCNet 2022
14.3
14.6
7.043
176.2
RTFS-Net-12
14.9
15.1
739
56.4
RTFS-Net-16
15.2
15.5
739
73.7
RTFS-Net-20
15.4
15.6
739
90.9
Table Exploring higher values (number of RTFS Blocks, including the AP Block) on the LRS2- 2Mix dataset.
From Table we observed that increasing It results in significant increase in performance, with an associated increase in computational complexity. Note that since the RTFS Blocks share parameters, increasing R does not increase the model size. both SDRi and SI-SNRi, while still utilizing only 91G MACs just over half the MACs utilized by CTCNet. However, 20 RTFS Blocks requires much larger GPU memory for training. As result, NVIDIA 4090s were necessary in order to obtain these results.
outperforms CTCNet by over 1dB in
K VISUALIZATION SAMPLES ON THE WEB PAGE
In this section, aim detail how constructed the AVSS samples seen our Web page To validate the generalizability of our approach, we purposefully chose videos from different scenarios and different genders with the aim of demonstrating the robustness of our model across wide range of environments. We selected six males and two females to form four different mixed audio scenarios, including variety of environments such broadcasting studios, outdoors and variety shows.
We selected videos from the VoxCeleb2 (Chung al. 2018 test set, meaning none of the models saw these videas in their training or validation data. In order to extract the facial features of the target speaker, we used an existing face detection model (Zhang et al. 2016) as described in Appendix F We then spliced the audios together in order to create the audio mixtures for the models, and placed the two original videos side by side for the demo.
httts:://ncnnmonnn 4open sci eeieenn//eeee wwtpeecc///ettteeeeeeeeee//e
Demo. html
17

---

Published as a conference paper at ICLR 2024
For each model, passed in the mixed audio signal and the target speaker's facial features to obtain the target speaker's estimated separated audio stream for each the 8 speakers. On our Web page, under each video is link for each of the models tested. Upon opening the video, it will play the mixed audio in time to the video. Hovering the mouse over the desired speaker will switch the audio to the model's s estimated separated audio stream for that speaker.
L GENERALIZING TO OTHER TASKS
In this section explore how RTFS-Net generalize other tasks, namely: the presence of more speakers, and word error rate. These downstream tasks further test the robustness of our method.
L.1 MORE SPEAKERS
Our model incorporates visual information from a single "target" speaker to isolate and extract the audio of this target speaker, hence mixture. This integration visual cues plays a crucial role in enhancing the accuracy of both our model and other AVSS models. We randomly sampled speakers from the LRS2-2Mrr test set to obtain a dataset with three speakers. Adding more speakers increases the difficulty the task, as from the perspective of extracting the target the speaker, the amount of backgroonnd/ettraeeuus noise has increased.
not limited by the number of speakers present in the
Model
SI-SNRi SDRi
Params (M) MACs (G)
AV-ConTTasNet
10.0
10.3
16.5
AVLIT-8
10.4
10.8
5.8
36.4
CTCNet
12.5
12.8
7.0
167.2
RTFS-Net-12
13.7
13.9
0.7
56.4
Table 9: Effects training on the LRS2-2Mix dataset, and testing on the LRS2-3Mix dataset.
In Table 9 we observed that RTFS-Net-12 scaled much better to this harder task and outperformed CTCNet by significant margin. While the presence of multiple speakers somewhat impacts the models' performance, it is important to note that these models were not trained the dataset with three speakers, only tested on
L.2 WORD ERROR RATE
In Table 10 we tested the speech recognition accuracy on the LRS2-2Mix dataset. We utilized the publicly available Google Speech-to-Texx Text API to obtain the recognition results. Our focus was on measuring the Word Error Rate (WER), where lower rate indicative of superior performance. The experimental results demonstrate that our model can achieve competitive speech recognition accuracy, validating the performance of our model in downstream tasks.
Models
SDRi WER (%)
Mixture
84.91
Ground-truth
17.74
CaffNet-C
12.5
32.96
Visualvoice
11.8
34.45
Wi-ConVTasNel
12.8
31.43
AVLIT-8
13.1
31.85
CTCNet
14.6
24.82
RTFS-Net-4
14.3
29.66
RTFS-Net-8
14.8
27.42
RTFS-Net-12 15.1
24.93
Table 10: Word Error Rate (WER) results on the LRS2-2Mix dataset.
18

---

Published as a conference paper at ICLR 2024
Original audio
7320
3310
2000
Time
S
5000
0000
2 4320
15
25
8000
3300
3000
Sample
Original
audio
1.00
(b) Sample
(c) Sample
Figure 4: Comparison of target speaker extraction methods. In 4a 4b and 4c the top spectrogram
the target speakers original audio, the middle spectrogram obtained using RTFS-Net-4 with $ and the bottom spectrogram is obtained using RTFS-Net- using the traditional masking approach.
19


Reviews:
[{"review_id": "0rNdAm3Siu", "reviewer": "Reviewer_1FXL", "summary": "In this paper, the authors present a novel time-frequency domain audio-visual speech separation method (Recurrent Time-Frequency Separation Network), a unique attention-based fusion technique for the efficient integration of audio and visual information, and a new mask separation approach. Results show that the proposed approach outperforms the previous SOTA method using only 10% of the parameters and 18% of the MACs. The authors claim that this is the first time-frequency domain audio-visual speech separation method to outperform all contemporary time-domain SOTA ones.", "questions": "Are there any overlapping speech in the train/test data?\nDo the authors perform any downstream task like speech recognition on the reconstructed speech?\nOne possible downstream task for speech separation is speech-to-speech dubbing. In this case, both the speech and background/nonspeech sound are needed. Have the authors looked into reconstructing background/nonspeech sound?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "One main strength of this paper is the proposed Recurrent Time-Frequency Separation Network that processes the data in the frequency dimension, the time dimension, and the joint time-frequency dimension.", "weaknesses": "It is mentioned that the RTFS blocks share parameters (including the AP Block), leading to reduced model size and increased performance. Therefore, more description/explanation for this would be helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hNmXJUkMcS", "reviewer": "Reviewer_siqV", "summary": "The paper proposes RTFS-Net, a new time-frequency (TF) domain audio-visual speech separation method. It introduces three main innovations:\n\n* RTFS Blocks independently model time and frequency dimensions of audio\n* Cross-dimensional Attention Fusion Block efficiently fuses audio and visual data\n* Spectral Source Separation Block preserves phase/amplitude information\n\nExperiments show RTFS-Net matches or beats prior time domain methods on LRS2, LRS3, and VoxCeleb2 datasets, while using 10x fewer parameters and 3-6x fewer computations.\n\nRTFS-Net is the first TF model to surpass most contemporary time domain methods for audio-visual speech separation. It demonstrates TF domain methods can achieve good performance at lower computational cost through novel modeling of time-frequency spectrograms.", "questions": "None", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Achieves near state-of-the-art performance for audio-visual speech separation while being very parameter and computationally efficient\n* Outperforms all compared time domain methods, proving time-frequency domain modeling can achieve better performance if done effectively\n* Innovative modeling of time and frequency dimensions independently in RTFS Blocks\n* Attention-based fusion mechanism in CAF Block is very lightweight but fuses audio and visual data very effectively\n* Spectral Source Separation Block properly handles phase/amplitude to avoid losing audio information\n* Model code and weights will be released for full reproducibility", "weaknesses": "* Testing is limited to only 2 speaker mixtures. Performance with more speakers is uncertain.\n* Missing PESQ evaluation in results which most other target speech extraction papers provide\n* Doesn't include latest SOTA model comparison: Dual-Path Cross-Modal Attention for Better Audio-Visual Speech Extraction, ICASSP 2023.\nhttps://arxiv.org/pdf/2207.04213.pdf. This gives superior performance for SI-SNRi and provides PESQ results as well. It does not provide MACs analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YiPLEJMydw", "reviewer": "Reviewer_ZTD4", "summary": "The authors build upon previous research in audio-only and audio-visual speech recognition by focusing on improving efficiency and fidelity of separated speech. They draw a lot of inspiration from the CTCNet paper and extend it to the TF domain to improve the efficiency. The solution has been evaluated on standard benchmark datasets and compared to previous state-of-the-art methods.", "questions": "1. Since the best performance is achieved with R=12, why not explore a higher R?\n2. Have the authors considered conducting studies with human listeners? If the target application is ASR, would it be helpful to measure WER in a recognition task?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Audio samples of separation are available and source to be made available when the paper is published.\n2. The writing is easy to follow.\n3. Clear modeling details are provided.", "weaknesses": "1. The baseline methods listed in table 1 should include their references.\n2. RTFS-Net-12 is only about 10% more efficient that CTCNet. How much difference does that make in practical applications?\n3. Some of the comparison examples are not distinguishable to this reviewer. This makes me wonder how to interpret the relative SNR gains.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EbDNEhvCRP", "reviewer": "Reviewer_A4oN", "summary": "The paper presents a novel Recurrent Time-Frequency Separation Network architecture that performs audio-visual source separation tasks effectively and efficiently. The model is characterized by three parts. First, each modality goes through its own processing module, and then the cross-dimensional attention fusion (CAF) consolidates information from both modalities. The spectral source separation block performs masking-based separation. The separation results show promising improvement given the compact size and computational efficiency the new model architecture introduces.", "questions": "- The authors chose to \"add\" f_1 and f_2 (eq 11) after the CAF processing. I think it's a little abrupt in the sense that there might be other choices that preserve the unique information that each vector learns, such as concatenation. Have the authors considered other ways to combine the two vectors?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- The paper presents solid improvement compared to the existing baseline systems. Considering the amount of model compression the proposed model introduced, the improvement is significant.\n\n- All the procedures and modules are well-defined with enough details.\n\n- The choice of the model architectures makes sense, including the dual-path structure, attention-based consolidation, and complex masks.\n\n- Ablation studies are thorough.", "weaknesses": "While the paper is packed with useful information, there are still some parts that need elaboration.\n\n- As the authors mention, the dual-path RNN idea is not new to this problem. I understand that the authors chose SRU for their complexity-related considerations, but I also wonder if the audio processing module could benefit from its own self-attention mechanism, such as in the SepFormer model. \n\n- The spectral source separation module might be the weakest contribution, because complex masks have been extensively studied in the audio-only source separation literature. \n\n- I wish the paper provides more details on the TDANet block for video processing, which is relegated to the reference in the current version.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)