Patent Description:
One disclosed example provides a computing system, comprising one or more processors, and memory comprising instructions executable by the one or more processors to implement a speech enhancement system. The speech enhancement system comprises a spatio-temporal residual network configured to receive video frames from a video containing a target speaker and extract visual features from the video frames, an autoencoder comprising an encoder, a decoder, and skip connections between the encoder and the decoder. The autoencoder is configured to receive input of an audio spectrogram, the audio spectrogram obtained based upon an audio waveform extracted from the video containing the target speaker, and to extract audio features from the audio spectrogram. The speech enhancement system further comprises a squeeze-excitation fusion block configured to receive input of a visual feature Fv, from a last layer of the spatio-temporal residual network and input of audio features Fa from all layers of the autoencoder, to apply global average pooling to the audio features and to the visual feature to compress the audio features over a channel dimension to obtain audio feature representations <MAT>, and to compress the visual feature over a spatial dimension to obtain visual feature representation <MAT>, to apply a 1D convolution across a frequency dimension of an audio feature representation resulting in <MAT>, and apply a 1D convolution across a channel dimension of the audio feature representation, resulting in Fv", to reshape and concatenate <MAT> and Fv" into a fused feature vector Fav, to apply a linear transformation h and a rectified linear unit (ReLU) activation to the fused feature vector, such that M = ReLU(h(Fav)), to apply a second linear transformation g, such that G = σ(g(M)), where σ is sigmoid activation, to broadcast G to the dimension of Fa and apply gating over a time dimension of the audio features as <MAT> <MAT>, and to provide an output to the decoder. The decoder is configured to output a mask configured based upon the fusion of audio features and visual features by the squeeze-excitation fusion block. The instructions are further executable to apply the mask to the audio spectrogram to generate an enhanced magnitude spectrogram, and to reconstruct an enhanced waveform from the enhanced magnitude spectrogram.

Another disclosed example provides a computer-implemented method of audiovisual fusion. The method comprises extracting video frames from a video containing a target speaker, and for each video frame, detecting a mouth of the target speaker in the video frame and cropping the video frame around the mouth. The method further comprises inputting the video frames, after cropping, into a spatio-temporal residual network configured to extract visual features from each video frame, extracting an audio waveform from the video containing the target speaker, and applying a transform to the audio waveform to obtain an audio spectrogram. The method further comprises inputting the audio spectrogram into an autoencoder comprising an encoder, a decoder, and skip connections between the encoder and decoder, and extracting, via the autoencoder, audio features in the audio spectrogram. The method further comprises inputting a visual feature Fv from a last layer of the spatio-temporal residual network and audio features Fa from all layers of the autoencoder into a squeeze-excitation fusion block, applying global average pooling to the audio features Fa and to the visual feature Fv to compress the audio features over a channel dimension to obtain audio feature representations <MAT>, and to compress the visual feature over a spatial dimension to obtain visual feature representations <MAT>, applying a 1D convolution across a frequency dimension of an audio feature representation resulting in <MAT>, and apply a 1D convolution across a channel dimension of the audio feature representation, resulting in Fv", reshaping and concatenating <MAT> and Fv" into a fused feature vector Fav, applying a linear transformation h and a rectified linear unit (ReLU) activation to the fused feature vector, such that M = ReLU(h(Fav)); applying a second linear transformation g, such that G = σ(g(M)), where σ is sigmoid activation; broadcasting G to the dimension of Fa, and applying gating over a time dimension of the audio features as <MAT>, providing an output from the squeeze-excitation fusion block to the decoder, outputting, via the decoder, a mask, generating an enhanced magnitude spectrogram by applying the mask to the audio spectrogram, and generating an enhanced waveform from the enhanced magnitude spectrogram.

Computerized speech enhancement may have many practical applications, such as in automatic speech recognition systems, telecommunication and teleconferencing systems, and hearing aids, as examples. Work in this domain may be at least in part motivated by the cocktail party effect, which refers to the ability of humans to selectively attend to auditory signals of interest within a noisy environment. Experiments in neuroscience have demonstrated that cross-modal integration of audio-visual signals may improve the perceptual quality of a targeted acoustic signal. Additionally, psycho-linguistic studies have investigated the effects of visual information on auditory encoding. Such studies have shown that visual cues may speed up linguistic recognition under noisy listening conditions, and that spatial decoherence between audio and visual sources may affect percept encoding.

Some approaches to computerized AV speech enhancement models utilize deep neural networks. Other approaches utilizes late fusion, where the audio and visual information is processed separately and then integrated at a singular point via channel-wise concatenation. These approaches have achieved some degree of a computational cocktail party effect. However, late fusion does not allow for feature-level correlation across modalities.

In computer vision, work has been done on video classification using deep neural networks that compared late fusion, slow fusion, and early fusion of time-based, unimodal visual signals. This work demonstrated that slow fusion may be most effective. Further, while channel-wise concatenation of cross-modal signals is currently used for conditional generative models, concatenation of cross-modal information may not effectively model the underlying correlations between features.

Squeeze-excitation (SE) networks may offer an alternative to channel-wise concatenation via a lightweight, adaptive gating mechanism that has been demonstrated to work well on sound event detection. Accordingly, examples are disclosed that relate to approaches to audiovisual (AV) fusion within a deep learning framework that utilize a squeeze-excitation (SE) fusion block that integrates AV information across multiple levels of audio and visual feature representations within both late and slow fusion schemes. In the disclosed examples, visual features are generated according to Stafylakis and Tzimiropoulos (<NPL>). Following <NPL>) and Michelsanti et al. (<NPL>), an autoencoder neural network with skip connections is used as the audio network. In other examples, though not recited in the claims, other suitable networks may be used for the audio and visual data processing.

The disclosed examples differ from the findings of Michelsanti et al. on optimal target domains. In the disclosed examples, the objective is trained using the indirect mapping approach of Afouras et al. (<NPL>) and Ephrat et al. (<NPL>) with log-mel transformed inputs. The disclosed method, referred to herein as AV(SE)<NUM>, when compared against the late fusion methods for AV speech enhancement, demonstrates that the SE fusion block yields improvements in performance and up to a <NUM>% reduction in the number of model parameters.

<FIG> schematically shows an architecture of an example AV(SE)<NUM> system <NUM>. System <NUM> comprises a video stream <NUM>, an audio stream <NUM>, and an SE fusion block <NUM> comprising (in this example) a plurality of SE fusion layers <NUM>. System <NUM> converts an audio waveform <NUM> to a mixed magnitude spectrogram <NUM> via short-time Fourier transform (STFT) <NUM>. The mixed magnitude spectrogram <NUM> is log-mel transformed <NUM>, and input to an autoencoder (indicated as encoder <NUM>, bottleneck <NUM>, and decoder <NUM>). Video <NUM> is ingested by the visual network <NUM> and integrated with audio via SE fusion <NUM>. While decoder <NUM>-<NUM> fusion is depicted in <FIG>, the decoder <NUM> may use other audio-visual feature relationships in other examples. The output comprises a mask <NUM> that is applied element-wise to the mixed magnitude spectrogram <NUM>, resulting in an enhanced magnitude spectrogram <NUM>. The enhanced magnitude spectrogram <NUM> is applied to mixed spectrogram phase information <NUM> via inverse short-time Fourier transform (iSTFT) <NUM> to produce an enhanced waveform <NUM>. In other examples, other suitable transforms than the STFT and iSTFT may be used.

Visual information is sampled as non-overlapping frames from video containing a target speaker at <NUM> frames per second (FPS), or other suitable frame rate. The visual processing includes using S<NUM>FD (described by <NPL>) to detect the speaker's face in each frame. In other examples, other suitable methods may be used to detect a speaker's face. Similar to Zhang et al. , visual processing may include discarding redundant visual information and cropping the mouth region. Any suitable methods may be used to perform these processes. In one example, a Facial Alignment Network (e.g. as described in <NPL>) ingests the face detection and produces facial landmarks usable to crop the region around the speaker's mouth. In this example, the resulting lip frame is resized to <NUM> × <NUM>, transformed to grayscale, and normalized using global mean and variance statistics from a training set to produce video data to provide to the visual network <NUM>.

The visual network <NUM> extracts visual features, Fv, using a spatio-temporal residual network, such as that proposed by Stafylakis and Tzimiropoulos. The visual network <NUM> comprises a 3D spatio-temporal convolution followed by a 2D ResNet-<NUM> (such as that proposed by <NPL>), and a bi-directional long short-term memory (LSTM) network. In some examples, the visual network <NUM> input accepts T×<NUM>×<NUM> lip frames, where T = <NUM> in this example. These parameters may be varied in other examples. Features are extracted from the ResNet. Let <MAT> denote visual feature at layer ResNet layer i. <MAT>, where h, w are spatial dimensions and b, t, c are the batch, temporal, and channel dimensions, respectively. To apply 2D convolutions to 3D features, in this example the temporal dimension, t, is packed into the first dimension alongside batch. Consequently, t is unpacked from the first dimension via reshaping such that <MAT> during feature extraction for AV fusion. The visual network is randomly initialized and trained on the speech-enhancement task. Example methods for integrating Fv with audio features are described in more detail below.

For audio processing, the audio waveform extracted from the video containing a target speaker may be processed following the methods of Afouras et al. and Gabbay et al. Per these methods, the spectrogram is obtained using STFT with a Hanning window function. As mentioned above, video frames may be sampled at <NUM> FPS. In some such examples, a window length of <NUM>, hop size of <NUM>, and sampling rate of <NUM> for STFT may be used, to align one video frame to four steps of audio spectrogram. The resulting spectrogram, Xspec ∈ RT×F, has a frequency resolution of F = <NUM> that represents frequencies from <NUM> to <NUM> in this example. In other examples, other suitable parameters may be used.

As mentioned above, the audio network comprises an autoencoder <NUM> with skip connections. The encoder <NUM> may use four residual blocks (two 2D convolution operations per block) and take the log <MAT> representation, described above, as input. Convolution may be performed over time-frequency dimensions and may be followed by batch normalization and a leaky rectified linear unit (lReLU) activation with a slope, α, of <NUM> in this example. In other examples, other suitable parameters may be used.

The decoder <NUM> may comprise, for example, eight upsampling blocks, each comprising subpixel upsampling (e.g., as described in <NPL>), a 2D convolution along time-frequency dimensions, batch normalization, and lReLU(α = <NUM>), except for the final layer, which is sigmoid activated in this example. In other examples, other suitable decoder architectures may be used. The output of the example autoencoder network is a magnitude mask <NUM>, M, and the enhanced magnitude spectrogram, Xenh, may be obtained via indirect mapping: Xenh = M ⊙ Xmag. Skip connections are employed between the encoder <NUM> and decoder <NUM> layers via channel-wise concatenation. Experiments using various direct mapping and mask approximation methods (described in Michelsanti et al. , supra) indicate that indirect mapping may be more effective in the disclosed examples. Dropout is not used. Table <NUM>, below, details example architecture parameters for the audio network.

Audio features, Fa, for use in the SE fusion block <NUM> are extracted from the encoder <NUM>, bottleneck <NUM>, or decoder <NUM>. <MAT> denotes the audio feature at layer j of the autoencoder. <MAT>, where b, t, f, and c are the batch, temporal, frequency, and channel dimensions, respectively.

The SE fusion block <NUM> is a gating block that models relationships between dimensions-temporal, channel, frequency-of audio and visual features. This may provide a light-weight mechanism for adaptive cross-modal integration of audio and visual features that may enhance the representational power of the network. <FIG> depicts an example architecture <NUM> usable for each SE fusion layer <NUM> in <FIG>. In the example of <FIG>, the SE fusion block architecture <NUM> is a tf-gating (temporal and frequency gating) version of an AV fusion block. Dimensionality is represented in (·). Subscripts denote dimension of operation: GAPc is global average pooling (GAP) over channel-dimension c, GAPhw is GAP over spatial dimensions wh, Conv1Df is a one-dimensional convolution across the frequency dimension f, and Con1Dc is a one dimensional convolution across the channel dimension c.

The SE fusion block architecture <NUM> takes an audio feature <MAT> from some layer j of the autoencoder and a visual feature <MAT> from some layer i of the ResNet as input. The following descriptions drop i, j to simplify notation without loss of generality. The SE fusion block architecture <NUM> applies GAP to both input features to obtain compressed feature representations, <MAT> and <MAT>. Fa is compressed over the channel dimension, c, and Fv over spatial dimensions, wh, resulting in <MAT> and <MAT>. The SE fusion block architecture <NUM> applies 1D convolutions across the f and cv dimensions of <MAT> and <MAT>, respectively, by a dimension reduction factor k. In this example, k = <NUM>. This results in <MAT> and <MAT>. <MAT> and <MAT> are reshaped such that <MAT> and <MAT>, where ma = ta · f' and <MAT>, then concatenated into a fused feature vector, Fav ∈ <MAT>. A linear transformation, h, is applied such that M = ReLU(h(Fav)), with M ∈ <MAT>, where r is a squeeze factor. In this example, the selected squeeze factor is r = <NUM>. The SE fusion block architecture <NUM> applies a second linear transform, g, such that G = <MAT>, where σ is sigmoid activation and z varies with gating dimension. For example, with t-gating, z = ta, and with tf-gating, z = ta · f. Finally, G is broadcast to the dimension of Fa and the SE fusion block architecture <NUM> applies an audio feature gating as <MAT>.

The disclosed network may be trained to optimize (e.g. minimize) the reconstruction loss between the enhanced magnitude, Xeng = M ⊙ Xmix, and the target magnitude, Xspec, via L<NUM> loss. An optimization objective may be given by <IMG> =∥ M ⊙ <MAT>.

Datasets: The disclosed example AV(SE)<NUM> model described above was trained on the VoxCeleb2 dataset (described by <NPL>), which contains over <NUM> million utterances for <NUM>,<NUM> celebrities. The dataset is split by celebrity ID (CID) such that the train, validation, and test sets are disjoint over CID. The following datasets are used as noise: CHiME-<NUM>/<NUM> (described by <NPL>; also described by <NPL>), NonStationaryNoise (described by <NPL>), ESC50 (described by <NPL>), HuCorpus (described by <NPL>), and private noise datasets.

Experimental conditions: The effectiveness of the SE fusion block <NUM> was shown by comparing to an AV baseline model having the architecture described above with respect to the audio network. The baseline uses late fusion with channel-wise concatenation of AV features. Experiments were conducted over three factors of variation to compare the disclosed method of late and slow fusion using SE fusion blocks <NUM>.

First, in a claimed example, SE fusion can occur at the bottleneck <NUM> (BN-only). In non-claimed examples, SE fusion can occur at the encoder <NUM> (E-only), or decoder <NUM> (D-only). BN-only fusion is similar to the architecture of the AV baseline wherein AV features are fused via late integration with channel-wise concatenation, except that late fusion is applied via the SE block <NUM>. Conversely, in unclaimed examples, E-only and D-fuse visual features at each audio feature layer of the encoder <NUM> or decoder <NUM>, respectively.

Second, for E-only and D-only approaches which are not claimed, SE fusion is applied such that there exists a <NUM>-to-<NUM> mapping between each layer i in the visual network and each layer j in the encoder/decoder of the audio network. In a claimed example, the last feature of the visual network, <MAT>, is mapped to all layers j in the audio network. This is referred to as <MAT> - to - all. For BN-only, <MAT> was mapped to the audio network's bottleneck <NUM>.

Third, within the SE fusion block <NUM> gating is applied over different audio feature dimensions: time (t-gating), and though not recited by the claims, channel (c-gating), frequency (f-gating), and combinations thereof.

Training: Training samples were generated as follows. The full set of lip frames and audio spectrograms pairs, (Xvid, Xspec), were order-shuffled and iterated over each epoch. A noise spectrogram, Xn, was randomly sampled from the training noise sets, and an additional non-stationary noise spectrogram, Xn2, was sampled with probability p = <NUM>. The mixture signal, Xmix, is obtained as Xmix = Xspec + αXn + βXn2, where α, β are mixing coefficients computed to achieve a specific signal-to-noise ratio (SNR) value; SNR values were randomly sampled from the range of <NUM> to <NUM> decibels (dB). The magnitude and phase were separated from the spectrogram, and the magnitudes were transformed to a log-mel representation, <MAT>, where F = <NUM>. Input audio representations were not normalized. The visual frames, Xvid, were augmented via random cropping (± <NUM> pixels) and left-right flips. Augmented frames were resized to <NUM> × <NUM> and fed into the visual network. The model was trained to optimize the objective given by L =∥ M ⊙ Xmix - Xspec ∥<NUM> using a batch size of <NUM>. The Adam optimizer was employed with an initial learning rate of <NUM> and validation loss was monitored to avoid overfitting. Models were trained for ~<NUM> epochs.

Results: Objective evaluation results are shown below in Table <NUM>. Enhanced speech was evaluated using the perceptual quality of speech quality (PESQ) (described by ITU-T Recommendation, "<NPL>), which is an objective measure of subjective listening tests. Enhanced speech was also evaluated using the short-time objective intelligibility (STOI) (described by <NPL>), which is correlated with the intelligibility of noisy speech. Version <NUM> of the AV(SE)<NUM> system (indicated as A V(SE)<NUM>-v2 in Table <NUM>) was found to achieve the best PESQ and STOI measures of those evaluated and outperformed the AV baseline by <NUM> PESQ and the AO baseline by <NUM> PESQ. The AV baseline has ~<NUM> million (M) parameters, whereas A V(SE)<NUM>-v2 has ~<NUM>, a <NUM>% reduction in parameters. The other AV(SE)<NUM> versions that outperformed the AV baseline afford a <NUM>-<NUM>% parameter reduction. Overall, fusion at the bottleneck and encoder showed relatively better performance, and <NUM>-to-<NUM> fusion outperformed <MAT> - to - all when fusion occurred at the encoder or decoder. Time-based gating at the bottleneck offered relatively better performance over channel-wise gating. However, no global trend was observed for gating. The disclosed model may generalize to speakers unseen during training since CID is disjoint across train and test sets. Similarly, AV(SE)<NUM> generalizes across noise types.

Various alterations to the disclosed model and training procedure may be used. For example, introducing a progressively increasing number of interference speakers during training, coupled with curriculum learning across SNR values (e.g., as is done in Afouras et al. ), may lead to performance gains. Further, pretraining a visual network on a word-level lip reading task may also lead to improvements in system performance. Lastly, improvements in PESQ and STOI measures (greater than <NUM> in PESQ and greater than <NUM> in STOI) were observed when using the target phase to reconstruct the enhanced signal. During inference, the target phase is unavailable; however, experimental observations indicate potential boosts to performance afforded by modeling phase.

<FIG> shows a flow diagram that illustrates an example computer-implemented method <NUM> for performing speech enhancement. Method <NUM> may be performed on any suitable computing system. Method <NUM> comprises, at <NUM>, extracting video frames from a video containing a target speaker. Method <NUM> further comprises, at <NUM>, for each video frame, detecting a mouth of the target speaker in the video frame and cropping the video frame around the mouth. Any suitable methods may be used to detect the mouth of the target speaker, including but not limited to the S<NUM>fd method described above. After cropping, method <NUM> comprises, at <NUM>, inputting the video frames into a spatio-temporal residual network configured to extract visual features from each video frame. In some examples, extracting the visual features may comprise applying a three-dimensional spatio-temporal convolution followed by a two-dimensional residual network and a bi-directional long short-term memory network, as indicated at <NUM>.

Method <NUM> further comprises, at <NUM>, extracting an audio waveform from the video containing the target speaker, and at <NUM>, applying a transform to the audio waveform to obtain an audio spectrogram. In some examples, applying the transform may comprise applying one or more of a short-time Fourier transform or other time/frequency transform as indicated at <NUM>. Further, applying the transform may comprise applying a log-mel transform, as indicated at <NUM>.

Method <NUM> further comprises, at <NUM>, inputting the audio spectrogram into an autoencoder comprising an encoder, a decoder, and skip connections between the encoder and decoder. The skip connections may comprise channel-wise concatenation between encoder and decoder, as indicated at <NUM>. Continuing, method <NUM> comprises, at <NUM>, extracting via the autoencoder audio feature in the audio spectrogram.

At <NUM>, method <NUM> comprises inputting the visual features from a layer of the spatio-temporal residual network and audio features from a layer of the autoencoder into a squeeze-excitation fusion block. In some examples, squeeze-excitation fusion may be used between multiple layers of the autoencoder and spatio-temporal residual network. The squeeze-excitation fusion block may be configured to perform various processes. For example, the squeeze-excitation fusion block may be configured to apply global average pooling to the audio feature and to the visual feature to obtain audio feature representation and visual feature representation, as indicated at <NUM>. The global average pooling may be performed on any suitable dimensions. In some examples, as indicated at <NUM>, the global average pooling compresses a channel dimension of audio features. Further, in some examples, as indicated at <NUM>, the global average pooling compresses a spatial dimension of visual features. As another example process, the squeeze-excitation fusion block may reduce dimensions of the audio feature representation and/or the visual feature representation by a dimension reduction factor, as shown at <NUM>. As a more specific example, the audio frequency dimension and visual channel dimensions may be reduced by the dimension reduction factor.

The squeeze-excitation fusion block forms fused feature vectors by concatenating the audio feature representations and the visual feature representations, as shown at <NUM>. After forming the fused feature vector, the fused feature vector is upsampled, for example, by applying one or more linear transform to the fused feature vector, as indicated at <NUM>. For example, as described above, a linear transformation, h, may be applied such that M = ReLU(h(Fav)), with <MAT>, where r is a squeeze factor. Further, a second linear transform, g, may be applied such that <MAT>, where σ is sigmoid activation and z varies with gating dimension. For example, with t-gating, z = ta, and with tf-gating, z = ta · f, as indicated at <NUM>.

Continuing, method <NUM> comprises, at <NUM>, providing outputs from the squeeze-excitation fusion block to the decoder, and at <NUM>, outputting a mask via the decoder. Method <NUM> next comprises, at <NUM>, generating an enhanced magnitude spectrogram by applying the mask to the audio spectrogram. The application of the mask to the spectrogram may be element-wise (multiplicative), as shown at <NUM>. In other examples, and equivalently, an additive operation may be used when converting both the magnitude and mask in the log domain. Further, at <NUM>, method <NUM> comprises generating an enhanced waveform from the enhanced magnitude spectrogram. This may comprise, at <NUM>, integrating phase information from the audio spectrogram with the enhanced magnitude spectrogram. In some examples, this may be performed via an inverse short-time Fourier transformation, as indicated at <NUM>.

Accordingly, the disclosed examples provide an approach to audio-visual fusion via squeeze-excitation blocks. The disclosed SE fusion block integrates visual information through slow fusion across multiple feature layers of the audio network by learning a time-based adaptive gating on audio features. Objective measures demonstrate that the disclosed method may outperform the baseline AV fusion model while maintaining an improvement over audio-only models. Moreover, the improvements in objective measures may be accompanied by a reduction in the number of model parameters.

The logic machine <NUM> may include one or more processors configured to execute software instructions.

The term "program" may be used to describe an aspect of computing system <NUM> implemented to perform a particular function. In some cases, a program may be instantiated via logic machine <NUM> executing instructions held by storage machine <NUM>. It will be understood that different programs may be instantiated from the same application, service, code block, object, library, routine, API, function, etc. Likewise, the same program may be instantiated by different applications, services, code blocks, objects, routines, APIs, functions, etc. The term "program" may encompass individual or groups of executable files, data files, libraries, drivers, scripts, database records, etc..

Claim 1:
A speech enhancement system (<NUM>), comprising:
one or more processors; and
memory comprising instructions executable by the one or more processors to implement:
a spatio-temporal residual network configured to receive video frames (<NUM>) from a video (<NUM>) containing a target speaker and extract visual features from the video frames;
an autoencoder comprising an encoder (<NUM>), a decoder (<NUM>), and skip connections between the encoder (<NUM>) and the decoder (<NUM>), the autoencoder configured to receive input of an audio spectrogram (<NUM>), the audio spectrogram obtained based upon an audio waveform (<NUM>) extracted from the video (<NUM>) containing the target speaker, the autoencoder being configured to extract audio features from the audio spectrogram (<NUM>); and
a squeeze-excitation fusion block (<NUM>) configured to:
receive input of a visual feature Fv from a last layer of the spatio-temporal residual network and input of audio features Fa from all layers of the autoencoder,
apply global average pooling to the audio features and to the visual feature to compress the audio features over a channel dimension to obtain audio feature representations Fa', and to compress the visual feature over a spatial dimension to obtain a visual feature representation Fv';
apply a 1D convolution across a frequency dimension of an audio feature representation resulting in <MAT>, and apply a 1D convolution across a channel dimension of the audio feature representation, resulting in Fv";
reshape and concatenate <MAT> and Fv" into a fused feature vector Fav;
apply a linear transformation h and a rectified linear unit (ReLU) activation to the fused feature vector, such that M = ReLU(h(Fav));
apply a second linear transformation g, such that G = σ(g(M)), where σ is sigmoid activation;
broadcast G to the dimension of Fa and apply gating over a time dimension of the audio features as <MAT>; and
provide an output to the decoder (<NUM>) of the autoencoder;
wherein the decoder (<NUM>) is configured to output a mask (<NUM>) based upon the fusion of audio features and visual features by the squeeze-excitation fusion block (<NUM>), and wherein the instructions are executable to apply the mask to the audio spectrogram (<NUM>) to generate an enhanced magnitude spectrogram (<NUM>), and to reconstruct an enhanced waveform (<NUM>) from the enhanced magnitude spectrogram (<NUM>).