Learning Audio-Visual Dynamics Using Scene Graphs for Audio Source Separation
Authors: Moitreya Chatterjee
Narendra Ahuja
Anoop Cherian
Venue: NeurIPS 2022 Accept (2022)
Keywords: Audio Source Separation
Multimodal Learning
Multi-task Learning
Scene Graphs

Abstract: There exists an unequivocal distinction between the sound produced by a static source and that produced by a moving one, especially when the source moves towards or away from the microphone. In this paper, we propose to use this connection between audio and visual dynamics for solving two challenging tasks simultaneously, namely: (i) separating audio sources from a mixture using visual cues, and (ii) predicting the 3D visual motion of a sounding source using its separated audio. Towards this end, we present Audio Separator and Motion Predictor (ASMP) -- a deep learning framework that leverages the 3D structure of the scene and the motion of sound sources for better audio source separation. At the heart of ASMP is a 2.5D scene graph capturing various objects in the video and their pseudo-3D spatial proximities. This graph is constructed by registering together 2.5D monocular depth predictions from the 2D video frames and associating the 2.5D scene regions with the outputs of an object detector applied on those frames. The ASMP task is then mathematically modeled as the joint problem of: (i) recursively segmenting the 2.5D scene graph into several sub-graphs, each associated with a constituent sound in the input audio mixture (which is then separated) and (ii) predicting the 3D motions of the corresponding sound sources from the separated audio. To empirically evaluate ASMP, we present experiments on two challenging audio-visual datasets, viz. Audio Separation in the Wild (ASIW) and Audio Visual Event (AVE). Our results demonstrate that ASMP achieves a clear improvement in source separation quality, outperforming prior works on both datasets, while also estimating the direction of motion of the sound sources better than other methods.

Learning Visual Dynamies Using Scene Graphs for Audio Source Separation
Moitreya Chatterjee metro. smi smiles@gmail. com
Narendra Ahuja' edu 'University of Illinois, Aarana-Champper Urbana, IL Mitsubishi Electric Research Labs, Cambridge, MA
Anoop Cherian 2- cheriame aeriannmerl... com
Abstract
There exists an unequivocal distinction between the sound produced by static source and that produced by moving one, especially when the source moves towards or away from the microphone. In this paper. we propose to use this connection between audio and visual dynamics for solving two challenging tasks simultaneously, namely: (i) separating audio sources from a mixture using visual cues, and (ii) predicting the 3D visual motion of sounding source using its sepa- rated audio. Towards this end, we present Audio Separator and Motion Predictor (ASMP)- deep learning framework that leverages the 3D structure of the scene and the motion of sound sources for better audio source separation. At the heart of ASMP is 2.5D scene graph capturing various objects in the video and their pseudo-3D spatial proximities. This graph is constructed by registering together 2.5D monocular depth predictions from the 2D video frames and associating the 2.5D scene regions with the outputs of an object detector applied on those frames. The ASMP task is then mathematically modeled the joint problem of: (i) recur- sively segmenting the 2.5D scene graph into several sub-graphs, each associated with constituent sound in the input audio mixture (which then separated) and (ii) predicting the 3D motions of the corresponding sound sources from the separated audio. To empirically evaluate ASMP, we present experiments on two challenging audio-visual datasets, viz. Audio Separation in the Wild (ASIW) and Audio Visual Event (AVE). Our results demonstrate that ASMP achieves clear improvement source separation quality, outperforming prior works on both datasets, while also estimating the direction of motion of the sound sources better than other methods.
Introduction
Events around us are often audio-visual in nature and our senses have evolved to leverage this multimodal synergy to better reason about the world. For example, the sight of a kid laughing aloud while sliding down slide allows
associate the laughing sound with the kid and get sense of his/her direction of motion, even when a myriad of other sounds are present in the scene. this paper, propose to leverage this synergy between sight and sound for solving two challenging tasks simultaneously, viz.: (i) separating audio sources from a mixture using visual cues, and (ii) the novel task of predicting the 3D visual motion of the sounding source using its corresponding separated audio.
Typical approaches to visually-guided audio source separation weakly- self- supervised models trained to separate mixture of acoustic sources into its constituents by conditioning on appropriate visual regions 8 3
55 56). Such approaches impose constraints the space of the separated audio (e.g., cyclic consistency, object identifiability, etc.) to derive gradients for training the underlying
"Equal Contribution.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

neural models. A few of these methods additionally seek to ground the separated audio with the appearance of the sounding object 8 T 46]. Chatterjee et al. 8 construct 2D scene graph on the video frames to capture the visual context of the audio source for better separation; their key intuition being that certain sounding objects (e.g., a guitar) cannot produce the sound by itself, but must have a suitable spatial context around them (e.g.. person). However, all the above approaches ignore one key aspect of the physical world- - that it is three dimensional and this 3D structure influences the sound being heard. For example, if person playing a guitar is spatially distant from the microphone, then the audio mixture to be separated is unlikely to have the sound of that guitar. Further, the 3D scene structure also allows for incorporating the motion of sounding objects. Imagine the whistling of a train moving towards you. There is an inevitable relationship between the evolution of the sound of the train being heard and its 3D motion, which can help distinctly separate its sound from other whistling trains or background clamour and vice-versa. Thus, if the audio of this train is well-separated via visual-guidance, then the separated audio must also be able to predict the direction of the train's 's motion. Leveraging these insights, we present Audio Separator and Motion Predictor (ASMP) innovative graph neural network for rideo-guide audio source separation from an acoustic mixture that can also predict the direction of motion of the sound source.
Inspired by Cherian et al. [9], our ASMP framework begins by computing dense 2.5D representation of the frames of a video where 2.5D refers to the 2D visual context of the frames enriched with the pseudo-depth for that frame produced using 2D-to-3D monocular depth prediction method [39]. Next, we succinctly capture the semantic context of this 2.5D visual scene by means of a novel 2.5D scene graph representation. The nodes of this scene graph capture the various objects in the scene (projected onto the pseudo-31) scene via the depth map). while the graph edges characterize the approximate 3D spatial distances between the objects. Note that our graph does not explicitly capture any spatio-temporal dynamics of the scene objects, instead is constructed on singleton frames. To achieve audio source separation, we propose recurrent graph neural network that is trained to segment this 2.5D scene graph into sub-graph embeddings; each of which is trained to be associated with a potentially unique sounding object or interaction and is used to induce separation of that sound source from the audio mixture. During training. we enforce this uniqueness via imposing orthogonality constraints between the generated sub-graph embeddings. To make ASMP associate the evolution of sound with the 3D motions of their sources, we propose to include an auxiliary task that demands the prediction of the 3D direction of motion of sounding object from its separated audio, where the ground truth 3D motion is estimated from the 2.5D) scene graph using optical flow.
A natural question one may have this point is: how can method predict the 3D motion from monaural audio using a 2.5D scene graph constructed on a single video frame? The key insights come from two observations: (i) there are often implicit object motion cues present in singleton video frames, which may be embedded within the scene graph node features (e.g., a graph node corresponding
train may have the inductive priors suggesting the train is moving in the direction it faces), and (ii) when the audio is separated via risual-guidamg the audio separation masks, thereby incorporating motion cues into the separated audio spectrograms, Thus, when trained end-to-end for the joint task of sound separation and motion prediction, the model leverages these implicit cues to minimize the learning objective.
these implicit features modulate
To demonstrate the efficacy of ASMP on the dual tasks of sound source separation and motion direction prediction, we present experiments on two challenging datasets, namely (i) Audio Separation in the Wild (ASIW) [8] and (ii) Audio Visual Event (AVE) [47). Both of these datasets feature videos of sounding objects and sounding interactions in the wild. Our results clearly show that ASMP outperforms competing prior methods on both these datasets as well as on both the tasks, underscoring the importance of incorporating 3D spatial structure and the benefits of learning audio- visual dynamics.
Below, we summarize the key contributions of the paper:
We introduce novel 3D geometry-aware scene graph representation 20 for visually- guided audio source separation, called ASMP.
We introduce a novel task of predicting 3D motion direction of a sound source in the scene from the the temporal evolution of the sound it makes, aided by appropriate visual context, and use it to improve audio separation.
ASMP demonstrates state-of of-te--o-----rr audio source separation and motion prediction perfor- mances on two challenging datasets for this task, viz. ASIW and AVE.
2

---

Conditioning
Sub Graph
Sub Graph
TANH
Audio
Craph
Direction Vecter Predkttoo
Video Reference Frame
Vote
Psoudo Grapts
Graph
Separator
Mixdd Aucio
Mieed Audio Spectragrom
Figure 1: A detailed illustration of our proposed ASMP model.
2
Related Works
Below, we briefly review some of the important prior works closely related to our approach.
Visually-guided Audio Source Separation is the task of separating mixture of audio signals into its constituent sources by discovering the association between the separated sound sources and their visual appearances the physical world. Typical methods for solving this task find wide range of real-world applications, such as separating sounds of musical instruments [133 g
[]] 55] [56].
separating speech signals [ 10 33), and separating on-screen sounds from off-screen ones for generic objects [36 48). A learning pipeline often used to train such methods conditions sound- U-net (41) with an appropriate visual object representation, to induce source separation on a mixed audio input, obtained by mixing the audio streams of two different videos. In recent years, research in this area has moved from using global visual features of motion and appearance [55] towards extracting very fine-grained visual conditioning information to induce more effective separations [13 46 Additional refinement steps have also been explored [8] to ward off potential false separation triggers. However, none of these approaches factor in the 3D geometry of the scene captured by the video - an important gap that we attempt to fill using ASMP.
separation network (such
Localizing Sound in Video Frames has been attempted in several recent methods by learning to ground the sound source the visual space, e-g., identifying the pixels of the sound source 17 While, these approaches strive to learn the association between the visual appearance and acoustic signatures of sound sources, they do not apply such methods to audio source separation a task that is the focus of this work.
21
42 46
Sound Synthesis from Videos constitutes another category of important techniques in the audio- visual realm 37 (57) that has become popular lately. Towards this end, several approaches have recently been proposed for the task of generating both monaural and binaural audio starting from videos [12 34 53 27). However, differently, we seek to solve the task of sound source separation
and motion prediction in the visual space.
Application of Scene Graphs to Videos has resulted massive strides in video understanding tasks. Scene graphs, while traditionally used for capturing the static content of images (20] (28 have lately been used for several video understanding tasks. For instance, Ji et al. [19 applied them for action recognition, Geng al. [14] for visual dialog, and Chatterjee al. g for visually-guided
sound-source separation. However, these scene graphs are usually 2D, while we attempt to explicitly incorporate the 3D scene geometry into the scene graphs. While, Cherian al. 9 proposes (2.5+1)ID scene graphs for video question answering, our task of audio source separation and motion prediction brings several novel components beyond their setup.
Audio Separation Using Direction of Arrival has been an important topic of recent interest the where the direction of arrival of sound microphone array is explicitly used for improved sound source separation. While, our approach is inspired by their key findings, attempt to explore this in the audio-visun domain.
audio research community [52 31 35 44
3

---

Panal Dugth
Reference Frame
Deeuh Image
Optical Flow
Pseudo-3D Scene Graph
A step-wise illustration of the pseudo-3D scene graph construction process on an example frame from an ASIW video. We show the original (reference) frame, its predicted monocular 3D depth map, its optical flow estimate with an adjacent frame, and pscudo-3D scene graph for the frame. The highlighted part the graph (blue edges) is the sub-graph associated with the sound of the horse's s motion, while the black-arrow denotes the motion direction of the horse in pscudo-3D space. Note that the motion direction is not used in our graph embeddings, instead provides ground truth guidance when minimizing our losses.
Figure 2:
3 Proposed Method
3.1 Task Setup and Method Overview
Let V denote an unlabeled video, and m(t) = Em (t) be its accompanying discrete-time audio arising from a linear mixture of N acoustic sources a, (t). The main goal of visually- gasually-guided guided audio source separation is to use cues derived from V to induce a separation of m(t) into its constituent acoustic sources a, (t), for E {1.2.... N.. For effectively capturing the audio-visus vviual semantic alignment, we represent the video V as scene graph 9 = (V,E) with nodes V {n1 { ........ nx) denoting the objects in V and & capturing the set of edges cij characterizing the spatial proximity of a pair of nodes (n,, n2y). The key idea in our proposed Audio Separator and Motion Predictor (ASMP) framework is to map each of the acoustic sources at( to a sub-graph of G. We realize this mapping by auto-regressively partitioning the graph G into mutually-orthogone and using these embeddings to condition an Acoustic Separator sub-network tasked with extracting a sound source (t) from the mixture m(t). A key ingredient in our model is equipping the audio separator to also predict the direction of motion of the sounding object. We incorporate this ability by including a direction prediction training loss. Figure L provides an overview our model and the details follow.
neural embeddings
3.2 Estimating Ground Truth Sound Source Motion
One novel aspect of ASMP is that it can predict the motion direction of source from its sound, which in turn derived from an embedding of sub-graph of G. However, the ground truth for this task is not directly available from the 2D frames of given video. In the following, the pre- pre-processing steps for estimating this ground truth, and present an overview of these steps in Figure 2 We denote motion displacement vector, d, for the source (t),, which will be subsequently used as an auxiliary cue to train the audio source separator.
detail
Computing Pseudo-Depth Maps: Constructing a 3D scene from 2D images is classic problem explored in computer vision for which a variety of solutions exist [5]. However, many of these methods make strong assumptions, such as: (i) the scene being static, (ii) existence of sufficient overlap between the frames, or (iii) knowledge of the intrinsic parameters of the camera, none of which might hold for the type of videos that are typically used in our task. To this end, we propose to build upon the recent progress made in monocular 3D reconstruction realm towards mapping each video frame Fy of V into 2.5D pscudo-depti image [54). In particular, use the MiDAS algorithm 39 because of its easc-of-use and accuracy of depth predictions.
Rectif ing Frames for Camera Motion: Since we do not assume that the frames in V have been captured by static camera, we next align them to a common 3D reference frame F,. To this end, we apply the classic out of the scene or shot-switches in the video, we found that tracking the static objects the video typically does not sustain its full length. Thus, we group the frames into fixed-size windows, each with a maximum of & continuous frames. A video can therefore, be considered as a collection of W windows, with each window producing its own motion displacement vector. This consequently
Kanade tracker (30 across the frames. As there may be objects moving

---

also implies that our frames are aligned (rectified) within their respective windows. Concretely, let x denote a tracked point, in pseudo 3D, within window. Suppose x, R and x, E R be the 3D coordinate corresponding to x in F and that in the reference frame F,, respectively. Then, using the full set of such tracked points in this window, we rectify F to Fr- by estimating 3D transformation matrix, accurate upto a scale, using the iterative closest point (ICP) algorithm [6] Once the target frame Fy has been rectified, we also rectify its corresponding pseudo-depth map using the same transformation matrix.
To construct our scene graph, we first use a Paster-RCNN (FRCNN) 40 model for detecting set of M objects and their 2D spatial bounding boxes in the reference frame F, of a window. The detection model is trained (on the Visual Genome dataset (25)) to recognize and localize 1600 commonplace object classes. However, some of the data that we use in our experimental study consists of sound producing objects that are not the Visual Genome classes list. To accommodate such classes (e.g., musical instruments), we use separate FRCNN model pre-trained on images annotated with these classes in the Open Images dataset (24]. Let C denote the set of all detectable classes by either of these two detectors. For frame Fr. the FRCNN model M
Visual Detection of Sound Sources:
produces a set of M quadruples: {C,,, B FX, { / = (FC one for each detected object, consisting of the label C' C of the detected object, its bounding box B in the frame, feature vector F' characterizing the visual appearance of the object, and a detection confidence S.
Tracking a Sound Source: In order to robustly track the visual trail of a potential sound source between a reference frame F, and target frame Fy. we employ RAFT 45 an off-the- the-thehl tf-te-of-thh-art robust optical flow method. RAFT yields a dense optical flow field for each pixel in the reference frame. We filter this flow field using the bounding boxes B,, and associate it with the i-th detected sounding object. Next, we base the flow field in a 3D coordinate space using the estimated pseudo-depth map associated with the respective frames; thus allowing for the recovery of the motion track of the sounding object in a pseudo-3D space.
Our proposed flow-based tracking method yields numerous displacement vectors (one for each pixel) within a detected object bounding box, this box corresponding above, we envisage representing a scene succinctly using a scene graph with each node capturing a single motion vector. To this end, we summarize the flow field associated with a detection box using its median; thus charaterizing the 3D displacement d; of that object in that window.
Estimating the Ground- -ruthh Displacement Vector:
potential sound source in the reference frame F,. However, as alluded to
3.3 Constructing the Pseudo-3D Visual Scene Graph
With the object detections in place, described above, we now proceed to construct the Pseudo-3D Visual Scene Graph for V. While, for most of the prior approaches [4 19]. all detected objects trivially added to the graph as nodes, the constructed graph permits the learning of correlations between appropriate subgraphs and the separated audio sources. Towards this end, each sound source in dataset is associated with set of objects in the visual domain (i.e., from the set of classes C of FRCNN) which we call Auditory Objects and which could potentially create sound. For instance, a crying sound could arise from classes such as kid, baby, etc. Let us denote an auditory object by p. where p E C. The entire video V could then be thought of a collection of auditory objects, P (Pi....,PN) ). which together generate the mixed audio m(t). For each of these auditory objects pt. we identify the subset Vp, of M- N objects that are non-auditory and overlap with p. with an Intersection Over Union (IoU) greater than a pre-defined threshold 7. These objects are called Context Objects. The vertex set V of the graph G associated with the video V is then constructed as V = U. (P:))
use-case of sound source separation dictates that
Next, we describe the process of constructing the edge set E of the graph G. While prior approaches that use scene-gras geene,graphs, typically either design dense fully fully-connected scene-graphs with equal edge weights [8] or deploy a visual relationship detector [4], nonetheless they do not usually embody any 3D scene geometry. Instead in ASMP, we propose to mitigate this shortcoming by computing the edge weights from the pseudo-3D scene we constructed above. To this end, we first extract 3D point clouds for every object i in frame F, by leveraging the rectified pseudo-depth map. Let Qi (for = 1, 2... M) denote the pseudo 3D point cloud of all pixels associated with node n; within its corresponding FRCNN bounding box. Note that each point x²? E =, is 3D vector with pixel-normalized xy coordinates and their pseudo-depths. To define the spatial proximity between two nodes n, and ny, compute the symmetrized Chamfer distance (Dzz) (5) between every pair
5

---

of points in x and ) These distances are then normalized to be within [o, 1] by dividing by the maximum value over all edges in & and are used to create a weighned-adjacenny matrix defining the graph edges & by using a radial basis function on the Chamfer distance matrix, i.e., the weight of an edge Cūj = exp( - -2a) ), for suitable scale o, set in experiments as the 25th percentile of the entries in the distance matrix.
Visual Encoding of Sounding Interactions: The scene graph g is spatio-semantic summary of the visual information V and is the visual analogue to the mixed audio m(() accompanying V. For the purposes of our task, we are interested in extracting appropriate subgraphs of G that are capable of inducing a separation of m (t) into its constituents a, (t) for i = 21.2..... .N. This is accomplished by employing a recurrent graph attention to segment the graph G into sub-graphs. To segment the graph g, we first employ a multi-headed graph attention network [49] which operates on the features associated with the scene graph nodes and performs multi-head graph message passing. This allows ASMP to incorporate the spatial scene geometry (from edge weights) and scene context from the neighbors of auditory node for characterizing the visual representation of a sound source. For capturing the visual representation of interactions (between objects) that produce a sound, we design an edge convolution network [5]] the features from which are then simultaneously pooled with the graph attention features using global max-pooling and global average pooling [26] and concatenated. This results in an embedding vector for the entire graph. Since we need to extract separate sub- sub-graph embeddings to induce a separation of the sound mixture into N sources, we employ a gated recurrent unit (GRU), to recursively extract visual feature vectors Y {y/ y/2, ................................ yN from Ç. We noticed that extracting a separate background sound from every mixture helps. Thus, our GRU is executed for N + steps, rather than N.
To ensure that the extracted graph embeddings Y from the GRU do not repeat a sub-graph, we enforce mutual orthogonality [8] between these embeddings. That is, each embedding y: emerging out of recurrence of the GRU is expected to produce a unit-normalized embedding ýt that is orthogonal to each of the unit-normalized embeddings generated prior to it, i.e., (ύ1, 92. ..... -. This constraint is incorporated as a regularization in our training setup. Mathematically, a softer-version of this constraint given by the following is implemented:
Lortho
Σ ()
0,)²
(1)
Acoustic Separator Network: Finally, to induce the separation in the acoustic signal, we incorporate an Acoustic Separator Network (ASN), which is a U-Net [4] style encoder-decoder network. Such networks have recently shown promise in sound source separation tasks [18] 29]. particularly those 56). The network has three main parts: (i) an encoder consisting of stack of DD-covvolution layers, each coupled with Batch Normalization and Leaky ReI.U, (ii) a bottleneck layer that concatenates the encoder embedding with the conditioning information, and (iii) a decoder which is made up of series of up-convolution layers, followed by non-linear activations, each coupled with a skip connection from a corresponding layer in the U-Net encoder, matching in spatial resolution of its output. The encoder encodes the input, which is a magnitude spectrogram X E ROxT of a mixed audio ml)) produced via the short-time Fourier transform (STFT), where Ω and T denote the number of frequency bins and the number of video frames, respectively. This encoding is then concatenated with each of the normalized visual encoding vectors 9 and decoded to generate time-freque time-freyuency mask, M, E [0. 1]2xT, which when multiplied with the magnitude spectrogram X of the mixture yields an estimate of the magnitude spectrogram of the separated source S, = M, C X, where @ denotes element product. The separated signal in time domain, à (1), is then recoverable by applying inverse Short time Fourier Transform (iSTFT).
that are employed in conditioned settings [3] 32 (3]56)),
Direction Prediction Network: A single-channel audio typically only indicates whether the sound source is moving towards/farther away from the camera [3]. However the sub-graph embeddings, y that the ASN is conditioned on, injects into the separated spectrograms, motion information implicit in the appearance of the objects in the frame. This enriches these spectrograms, permitting us to obtain a coarse estimate of the direction of source motion in pseudo 3D. We therefore cast the task of predicting the motion vector d; of the auditory object as a classification task. We setup two variants of this task using varied quantizations of the unit displacement vector d,, namely: (i) to predict the octant (or corners) of a unit-cube into which an 8-quantized unit displacement vector d; belongs, where the center of the cube refers to the 3D centroid of the corresponding graph node; and (ii) to predict among the 26 directions pertaining to the 8 corners, 6 faces, and 12 edges of unit-cube, d, is

---

quantized into (using cosine similarity). Additionally, for both settings class is created for denoting little/no motion and another class captures the direction vector obtained from the background source. We thus have 10 and 28-classes respectively, in the two settings. We realize this objective in ASMP by incorporating a classifier network h with parameters A, that takes as input the (spectrogram) output of the ASN and produces the class label corresponding to the quantization of d,.
Note that we compute a separate displacement vector d; per window w, while the separated S, spans W the whole length of the video. We thus compute set of displacement vectors {d!") Jw=1 for each window w. Our network takes as input the separated spectrogram. S,, and the visual encoding of the source i (Ú:) and produces as output an estimate of the direction class, d. {0, 1,2..... 9) or a {0, 1, ...,. 27], depending the direction in which the vector lies. In order to predict the vector for particular window, appropriate time-aligned slices of S; are chosen. The sliced spectrogram is where the prediction for each window treated as a different sample the batch. Details in the appendix.
style network. Thus, we have hA S, - { {ay²) w=1
then embedded through a ResNet [6
3.4 Learning Losses
ASMP trains purely with weal/self-supperisory cues. In particular, we train our model by employing the "rixxaad-sepprate" strategy (8 [1 13 55 56), where the audio tracks from multiple videos (typically two) are combined, with the goal being to separate the audio for each possible source from the mixture (X). Towards this end, we employ the following set of losses:
Consistency Loss ensures that the separated spectrograms of a particular class of sound source (such as sound from a guitar, the cry of a baby, etc.) always retains its (visual) identity irrespective of which video it comes from [33. Typically cross-entropy loss satisfies this requirement. However in ASMP, since the separation is induced by a holistic scene graph recurrently embedded by GRU, it is not known in advance, in what order the sources are separated. Thus, we consider all possible combinations of the ground truth labels and assign the one with the minimum loss. This is given by:
No+1 K
Lcons Σ
min
-Σ
(c) log pr.
(2)
ESN
a=1,2
1=1
where N2, + the number of auditory objects the u-th video, SN indicates the set of all permutations on (1...., N₂ + 1}, P². denotes the predicted probability produced by the classifier for class e given Sy as input, (c)) is an indicator for the ground-trute -tuut class of the e-th object in video u, and K denotes the number of sounding object classes in the dataset.
Cyclic Loss attempts to preserve the consistency of the generated audio masks from the same video. Specifically, this loss ensures that the separated masks corresponding to each the sources in the video, when combined, yields mask which separates the audio spectrogram for that video from the mixture audio (that combines audio from two videos). Mathematically, our cyclic loss is written as:
Laye Eu=1,2 EeS M. MD where M 1x=>x denotes the binary mask for the audio of video within the mixture X.
Direction Prediction Loss is a standard cross-entrops loss, which checks for the accuracy of the predicted direction class 1, 1,... D where D₁ E 9, 27 for every separated sound source, per
window. It is given by:
W min ES
No+1 Dr
Ldirpred
Σ 1,2
Σ
Σ
1
uae (c) log
0.00 ".
where N, + 1 is the number of auditory objects in the 4hhh video. SN,+1 indicates the set of all permutations on [l,...,Ng, + 1}, vhe denotes the predicted probability produced by the classifier for class given S as input, and k (c)(() an indicator of the ground-truth class of the c-th object
in video u for window w.
Final Training Loss of ASMP is given by the following, with weights \1, A2, As, A > 0:
C = X Coons + XeLoye + AgCortho + Addrppred
(4)
7

---

Table 1: SDR, SIR, and SAR results on the ASIW and AVE test sets. [Key: Best, second best results.]
ASIW
AVE
Approach
SDR 1
SIR 1
SAR
SDR f
SIR T
SAR
Sound of Motion (SofM) [55]
6.7
9.1
11.1
1.1
9.2
7.6
Cyclic Co-Learn |46
7.0
13.4
12.4
4.2
9.7
8.4
Co-Separation 13
6.6
12.9
12.6
3.9
9.3
7.8
AVSGS 8
8.8
14.1
13.0
5.8
10.4
8.2
ASMP (only 2.5D graph) ASMP (2.5D graph + motion)
9.0
14.3
13.7
6.5
12.4
8.9
9.6
14.5
14.1
7.2
13.3
9.4
Table 2: Direction Prediction results on the ASIW and AVE on test splits.
ASIW
AVE
Direction Prediction
10 10-class (% (%)
28-class (%)
10-class (%)1
28-class (%)1
Majority Vote
27.3
25.4
29.2
24.3
Sound of Motion (SofM) 55
29.6
27.0
31.2
30.6
Cyclic Co-Learn
34.8
32.3
30.7
29.2
Co-Separation 13
32.2
31.7
30.2
28.0
AVSGS
S
39.2
38.7
38.9
34.7
ASMP (Ours)
42.5
41.3
38.5
36.8
Experiments
In this section, we demonstrate the effectiveness of our ASMP approach on challenging visually- guided audio separation benchmarks.
4.1 Datasets and Experimental Setup
Audio Separation in the Wild (ASIW) Dataset: Several prior works present experiments mainly for
separating sounds of musical instruments [1] 46). Such results may not necessarily carry over easily more generic and natural day-to day-to-day life contexts. To this end, we consider the recently proposed Audio Separation in the Wild (ASIW) dataset [8] for our experiments. This dataset consists of 147 validation, 322 test, and 10,540 training videos crawled from the larger AudioCaps dataset [22].
has 14 auditory classes (i.e. K = 15, 14 auditory classes and 1 background audio class), such as baby cries, bell ring, birds chirp, camera clicks, train sounds, automobile sounds, etc. Each video in the dataset is 10s long. has significant camera motion, and captures diverse audio-visual contexts. To evaluate ASMP for direction prediction, we divide each video into temporal windows Is long and predict a discretized 3D direction vector for randomly chosen window.
Audio Visual Event (AVE) Dataset: Apart from the challenging ASIW dataset, we conduct additional experiments by adapting the popular Audio Visual Event (AVE) Dataset [4] for our task. Since this dataset was originally designed for the task identifying audio-visual events, adapting it to evaluate our approach is straightforward. We treat each audio-visual event class as auditory class and associate with set of Auditory Objects (as discussed before). The dataset contains 2211 training, 257 validation, and 261 test set videos. We use videos corresponding to 18 audio-vice visual event classes (i.c. K = 19 here), including both potentially-move static classes like banjo, clock etc. The videos in this dataset are 10s long as well. Similar to ASIW, we divide each video into windows of long for computing the motion displacement vectors.
object classes such, such as bus, train, etc. and
To the best of our knowledge, the task of jointly separating audio sources (using video) and predicting motion directions is novel task, and thus does have prior baseline to compare to. Therefore, we compare ASMP against the following state-of-the -frthe-art methods for audio-visual source separation, specifically those methods that use the "mix-and-sene aanddseaaaate" learning and also evaluate against them for assessing the efficacy of direction prediction. Specifically, we compare to: (i) co-Separation I3 that uses a single auditory object conditioning source separation network,(ii) Sound of Motion (SolM) (55) that incorporates object/human appearances and their pixel-level motion
Baselines:

---

Table 3: SDR, SIR, and SAR Ablation results on the ASIW test set.
[Best results in bold.]
ASIW
Row
Method
SDR [dB]]
SIR [dB]
SAR [dB]]
ASMP (Full Model)
9.6
14.5
14.1
ASMP- Multiscale Chanfer
9.2
14.1
14.0
ASMP- Only 10-class Direction Prediction (A: Ao As
11.2
11.7
ASIW
Video 1
Video 2
Ours Motion Directions
Reference Frame
Mixed Spectrogram
Ground-Truth Spectrograms
Ours
AVSGS
AVE
Video 1
Video
Reference
Mixed
Ground-Truth
Ours
Frame
Spectrogram
Spactrograms
Ours
AVSGS
Mation
Directions
Figure 3: Audio source separation and motion prediction results on ASIW (top) and AVE (bottom). Bounding boxes on reference frames show regions attended by ASMP. Red boxes indicate regions of high differences between ground truth and predicted spectrograms. The orange unit cube shown in the Motion Directions images, indicates the object whose motion is predicted. The green dot shows the ground truth motion direction, while the yellow arrow shows the predicted direction by ASMP.
trajectories condition an audio separation network, (iii) Cyclic Co-Learning 446] which a recent method that trains a deep neural model for the joint task of audio separation and visual grounding of the audio source, and (iv) Audio Visual Scene Graph Segmenter (AVSGS) (8) that uses a scene graph for conditioning audio separator network using the 2D visual context of the audio source.
Evaluation Metrics: We quantify the audio source separation performances using the standard
evaluation metrics [33 46 56), namely: (i) Signal-to-Di -ognal-o-Distortiin Ratio (SDR) (in dB) [38 50
higher SDR indicating more faithful reproduction the original signal, (ii) Signal-to-hntereeeen Interrefeeeen Ratio (SIR) (in dB) quantifying the extent of reduction in interference in the estimated signal, and (iii) Signal-to-Artifan Ratio (SAR) (in dB) capturing the extent which artifacts are introduced by the separator. Further, as our motion direction prediction sub-task is cast as a classification problem in our setup, we also report classification accuracy (Dir. Acc. (%)) for both and 28 class settings.
Implementation Details: For every video in our setup, we detect upto two auditory objects, one background auditory object, and 20 context nodes. The visual feature of the background object is derived from a FRCNN embedding of a random crop of the reference frame F,, from region which does not overlap with other boxes in the frame. The audio streams are sub-sampled at 1kHz and STFT spectrograms are extracted using Hann window of size 1022, with hop length of 256, following prior works [33 (6). We set Ω 256 and T 256. The embeddings, ú: and the GRU- state are 512-dimensional. The loU threshold, is set to 0.1 for both datasets. frames. The weights on the different losses are as follows: A1 = 0.05, A2 = 1.0. X3 1.0, Aa 0.05. Our model is trained using the ADAM optimizer 22 with weight decay of le-4, 81 0.9, ,3 0.999. The learning rate is to and is decreased by factor of 0.1 every 15K iterations.
Each window in a video has

---

4.2 Experimental Results
In Table I we present results for source separation by our method versus competing approaches on the ASIW and AVE datasets. In T we report the direction prediction accuracy of ASMP on the same set of competing methods for both 10 and 28 class setting. Our results clearly attest to the benefit of using the 3D structure of the scene and the motion direction prediction for audio separation. Specifically, ASMP itself (without motion prediction) yields a boost of upto 0.7 dB SDR over prior methods, just on source separation. Additionally, the performance of ASMP improves by atleast another 0.6 dB (on the SDR scale) across the two datasets, when trained to predict the motion direction (Table TT. Furthermore, the results evince that ASMP outperforms prior approaches in almost all of the evaluation categories, by upto 2.0 dB on the SIR scale on the AVE dataset over approach of AVSGS S.. These results clearly demonstrate the
the next best (and generalizability of our method to varied datasets, scene contexts, and audio diversity. From amongst the competing techniques, we find that AVSGS comes closest to matching our performance across the two datasets, which is perhaps unsurprising given that they use 2D scene graph encode the visual context of the sounding object. However, our results show that the presence of 3D scene geometry is also essential for good audio separation (Table laat two rows).
In Table 2 we also report the accuracy of motion prediction of the sounding object (in %) for both ASMP and other baselines for both 10-class and 28-class settings by passing their separated spectrograms through the direction prediction module. As is clear from the table, we find that the accuracy of direction prediction using ASMP is much better than prior methods (by upto 3%). The gains over a simple majority label prediction is especially pronounced, attesting to the efficacy of our direction prediction pipeline.
We report performances of few key ablated variants of our model in Table 3 on the ASIW dataset, viz. (i) by computing the weights of the graph edges via multiscale RBF kernel, (ii) using only the motion direction prediction as a training signal. For the former, we threshold the Chamfer distances at two scales, viz (a) the median of the weights and (b) 100 percentile of the weights. This creates a forest of graphs, which is used the audio separator network. Our results in BableBJshow that this approach of graph construction levels up to our RBF formulation but falls bit short, perhaps because of the hard thresholding of edges. In another ablation study, we train only the motion direction prediction loss (i.c., A X2 = As 0). As the results in Table show, this variant underperforms our full model, corroborating the utility of the other loss terms.
Ablation Study:
Qualitative Results: In Figure 3 we present sample separation results from the ASIW and AVE test sets, juxtaposing the results obtained by our method against the AVSGS baseline. The superiority of our model's separability is evident even at glance of the separated spectrograms that mimic the ground truth spectrograms more closely compared to AVSGS. The estimates of the motion directions seem to agree with the ground truth as well. Additional details and results are in the appendix.
Conclusions
In this work, we presented ASMP, novel algorithm to leverage visual scene geometry to induce better separation of a mixed audio signal into its semantic constituents. Towards this end, we incorporated pseudo-3D cues into algorithmic audio separation pipeline via scene graph data structure. Our work further derives self-supervision cues from the visual dynamics inherent in the videos and use such cues predict the direction of motion of the sound source from the separated audio; our experiments demonstrate this self-supervision to improve source separation. ASMP demonstrates state-of-the-art performances of predicting the direction of motion from the visually-guides separated audio suggests ASMP's potential to used in a variety of other audio-visual problems, including occlusion reasoning, and video generation from audio 77. We will be making our Py Torch implementation public.
two challenging "iilthe-wild" datasets. The success at the task
Limitations: While ASMP presents promising results on both separation and direction prediction tasks, some caveats remain. For example, our model is not yet capable of effectively inducing source separation for sounds emanating from the background. Further, require trained object detectors for sound producing classes, which for some rare/uncommon classes maybe hard to obtain.
Acknowledgements. MC and NA would like to thank the support of the Office of Naval Research under grant N00014- 20-1-2444, and USDA National Institute of Food and Agriculture under grant 2020-67001-32779910244
10

---

References
[1] Triantafyllos Afouras, Andrew Owens, Joon Son Chung, and Andrew Zisserman. Self-supervised learning of audio-visual objects from video. Pruc. ECCV, 2020.
[2] Sameer Agarwal, Yasutaka Furukawa, Noah Snavely, Ian Simon, Brian Curless, Steven M Scitz, and Richard Szeliski. Building rome a day. Communications of the ACM, 54(10):105-112, 2011.
[] EN da C Andrade. Doppler and the doppler effect. 1959.
[4] Relja Arandjelovic and Andrew Zisserman. Objects that sound. In Proc. ECCV, pages 435 451, 2018.
Harry Barrow, Jay M Tenenbaum, Rohert C Bolles, and Helen C. Wolf. Parametric correspondence and chamfer matching: Two new techniques for image matching. Technical report, SRI INTERNATIONAL MENLO PARK CA ARTIFICIAL INTELLIGENCE CENTER, 1977.
[6] Paul Besl and Neil D McKay. Method for registration of 3-d shapes. In Sensor fusion IV: control paradigms and data structures, volume 1611, pages 586 606. Spie, 1992.
[7] Moitreya Chatterjee and Anoop Cherian. Sound2sight: Generating visual dynamics from sound and context. In Proc. ECCV. Springer, 2020.
[8] Moitreya Chatterjee, Jonathan Le Roux, Narendra Ahuja, and Anoop Cherian. Visual scene graphs for audio source separation. In Proc. ICCV, 2021.
[9] Anoop Cherian, Chiori Hori, Tim K Marks, and Jonathan Le Roux. (2.5+ 1) d spatio-temporal scene graphs for video question answering. In Proceedings the AAAI Conference Artificial Intelligence, volume 36, pages 444-453, 2022.
[10] Ariel Ephrat, Inbar Mosseri, Oran Lang. Tali Dekel, Kevin Wilson, Avinatan Hassidim. William Freeman. and Michael Rubinstein. Looking to listen at the cocktail party: a ppeake--nndependent audio-visual model for speech separation. ACM Trans. Graph. (TOG), 37(4) 2018.
[11] Chuang Gan, Deng Huang, Hang Zhao, Joshua B Tenenbaum, and Antonio Torralba. Music gesture for visual sound separation. In Proc. CVPR, pages 10478-10487, 2020.
[12] Ruohan Gao and Kristen Grauman. 2.5 d visual sound. In Proc. CVPR, pages 324-333, 2019.
[13] Ruohan Gao and Kristen Grauman. Co-separating sounds of visual objects. In Proc. ICCV, pages 3879-3888, 2019.
[14] Shijie Geng. Peng Gao, Chiori Hori, Jomathan Le Roux, and Anoop Cherian. Spatio-temporal scene graphs for video dialog. In Proc. AAAI, 2021.
[15] Richard Hartley and Andrew Zisserman. Multiple view geometry in computer vision. Cambridge university press, 2003.
[16] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proc. CVPR, pages 770-778. 2016.
[17] John Hershey and Javier Movellan. Audio vision: Using audio-visual synchrony locate sounds. In Proc. NIPS, pages 813-819, December 1999.
[18] Andreas Jansson, Eric Humphrey, Nicola Montecchio, Rachel Bittner, Aparna Kumar, and Tillman Weyde. Singing voice separation with deep U-net convolutional networks. In Proc. ISMIR, October 2017.
[19] Jingwei Ji, Ranjay Krishna, Li Fei-Fei, and Juan Carlos Nichles. Action genome: Actions as compositions of spatio-temporal scene graphs. In Proc. CVPR, pages 10236-10247, 2020.
[20] Justin Johnson, Ranjay Krishna, Michael Stark, Li-Jia Li, David Shamma, Michael Bernstein, and Li Fei- Fei. Image retrieval using scene graphs. In Proc. CVPR, pages 3668-3678, 2015.
[21] Finat Kidron, Yoay Y Schechner, and Michael Elad. Pixels that sound. In Proc. CVPR, volume 1, pages 88-95. IEFF, 2005.
[22] Chris Dongjoo Kim, Byeongchang Kim, Hyummin Lee, and Gunhee Kim. Audiocaps: Generating captions for audios in the wild. In Proc. NAACL HILT, pages 119-132, 2019.
[23] Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In Proc. ICLR, 2014.

---

[24] Ivan Krasin, Tom Duerig, Neil Alldrin, Vittorio Ferrari, Sami Abu-El-Haaja, Alina Kuznetsova, Hassan Rom, Jasper Uijlings, Stefan Popov, Andreas Veit, et al. Openimages: A public dataset for large-scale multi-label and multi-class image classification. Dataset available from https://gihub. comfopenimages. 2(3):18, 2017.
[25] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanic Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual Genome: Connecting language and vision using crowdsoureed dense image annotations. Int. J. Comput. Vis., (23-13:3,-73, 2017.
[26] Junhyun Lee. Inyeop Lee, and Jaewoo Kang. Self-attention graph pooling, In Proc. ICML, pages 3734 3743, June 2019.
[27] Sijia Li, Shiguang Liu, and Dinesh Manocha. Binaural audio generation via multi-task learning. ACM Transactions on Graphics (TOG), 103, 2021.
[28] Xiaowei Li, Changchang Wu, Christopher Zach, Svetlana Lazebnik, and Jan-Michael Frahm. Modeling and recognition of landmark image collections using iconic scene graphs. Proc. ECCV, pages 427-440. Springer, 2008.
[29] Yuzhou Liu and DeLiang Wang. Divide and conquer: A deep CASA approach to talker-independent monaural speaker separation. IEEE/ACM Trans. Audio, Speech, Lang. Process., 27(12) 20022 2019.
[30] Bruce D Lucas, Takeo Kanade, et al. An iterative image registration technique with an application to stereo vision. Vancouver, 1981.
[31] Yi Luo. Cong Han, Nima Mesgarani, Enea Ceolini, and Shih-Chii Liu. Fasnet: Low-latency adaptive beamforming for multi-microphone audio processing. In 2019 IEEE automatic speech recognition and understanding workshop (ASRU), pages 267. IEEE, 2019.
[32] Gabriel Meseguer-Brood Brocal and Geoffroy Peeters. Conditioned-U-Net: Introducing a control mechanism in the U-Net for multiple source separations. arXiv preprint arxiv: 1907.0/277 2019.
[33] Daniel Michelsanti, Zheng- Tan, Shi-Xiong Zhang. Yong Xu, Meng Yu, Dong Yu, and Jesper Jensen. An overview of haap-learnin--basee audio-visual speech enhancement and separation. arXiv preprint arXiv:2008.09555 2020.
[34] Pedro Morgado, Nuno Vasconcelos, Timothy Langlois, and Oliver Wang. Self-supervised generation of spatial audio for 360 video. In Proc. NeurlPS, pages 360-370, 2018.
[35] Tsubasa Ochiai, Mare Delcroix, Rintaro Ikeshita, Keisuke Kinoshita, Tomohiro Nakatani, and Shoko Araki, Beam-tasnet: Time-domain audio separation network meets frequency- domaii beamformer. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6384 6388. IEEE, 2020.
[36] Andrew Owens and Alexei A Efros. Audio-visual scene analysis with self-supervised multisensory features. In Proc. ECCV, pages 63 1-648, 2018
[37] Andrew Owens, Phillip Isola, Josh McDermott, Antonio Torralba, Edward H Adelson, and William Freeman. Visually indicated sounds. In Proc. CVPR, pages 2405-2413, 2016.
[38] Colin Raffel, Brian McFee, Eric J Humphrey, Justin Salamon, Oriol Nieto, Dawen Liang, Daniel PW Ellis. and C Colin Raffel. mir_eval: A transparent implementation of common mir metrics. In Proc. ISMIR, 2014.
[39] René Ranfil, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In Proc. ICCV, 2021.
[40] Shaoqing Ren. Kaiming He, Ross Girshick, and Jian Sun. Faster r-cun: towards real-time object detection with region proposal networks. IEEE Trans, Pattern Anal. Mach. Intell., 3937)1137--1449 2016.
[41] Olaf Ronneherger, Philipp Fischer, and Thomas Brox, U-net: Convolutional networks for biomedical image segmentation. In Proc. MICCAI, pages 234-241. Springer, 20155
[42] Arda Senocak, Tae-Iyyun Oh, Junsik Kim, Ming- Ming-HSuan Yang. and In So Kweon. Learning to localize sound source in visual scenes. In Proc. CVPR, pages 4358-4366, 2018.
[43] Olga Slizovskaia, Leo Kim, Gloria Haro, and Emilia Gomez. End-to-end sound source separation conditioned on instrument labels. In Proc. ICASSP, pages 306-310, May 2019.
12

---

[44] Aswin Shanmugam Subramanian, Chao Weng, Shinji Watanabe, Meng Yu, Yong Xu, Shi-Xiong Zhang. and Dong Yu. Directional asr: paradigm for e2e multi-speakeer speech recognition with source localization. Proc. ICASSP, pages 8433-8437. IEEE, 2021.
[45] Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. Proc. ECCV, pages 402-419. Springer, 2020.
[46] Yapeng Tian, Di Hu, and Chenliang Xu. Cyclic co-Icarning of sounding object visual grounding and sound separation. In Proc. CVPR, 2021.
[47] Yapeng Tian, Jing Shi, Bochen Li, Zhiyao Duan, and Chenliang Xu. Audio- event localization in unconstrained videos. Proc. ECCV, pages 247 263, 2018.
[48] Efthymios Tzinis, Scott Wisdom, Aren Jansen, Shawn Hershey, Tal Remez, Daniel PW Ellis, and John Hershey. Into the wild with AudioScope: Unsupervised audio-visual separation of on-screen sounds. In Proc. ICLR, 2021.
[49] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. In Proc. ICLR, April 2018.
[50] Emmanuel Vincent, Rémi Gribonval, and Cédrie Févotte. Performance measurement blind audio source separation. IEEE Trans. Audio, Speech, Lang. Process. 14(4):1462-1469 July 2006.
[51] Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph CNN for learning on point clouds. ACM Trans. Graph. (TOG), 38(5)1 2019.
[52] Wenjie Xu, Maoshen Jia, Shang Gao, and Lu Li. Multiple sound source separation by using doa estimation and ica. In 2021 4th International Conference Information Communication and Signal Processing (ICICSP), pages 239.253. IEEE, 2021.
[53] Xudong Xu, Hang Zhou, Ziwei Liu, Bo Dai, Xiaogang Wang, and Dahua Lin. Visually informed binaural audio generation without binaural audios. In Proc. CVPR, 2021.
[54] Chaoqiang Zhao, Qiyu Sun, Chongzhen Zhang, Yang Tang, and Feng Qian. Monocular depth estimation based on deep learning: An overview. Science China Technological Sciences, 13-117612-1627, 2020.
[55] Hang Zhao, Chuang Gan, Wei-Chiu Ma, and Antonio Torralba. The sound of motions. In Proc. ICCV, pages 1735-1744, 2019.
[56] Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl Vondrick, Josh McDermott, and Antonio Torralba. The sound of pixels. In Proc. ECCV, pages 570-586, 2018.
[57] Yipin Zhou, Zhaowen Wang, Chen Fang, Trung Bui, and Tamara I. Berg. Visual to sound: Generating natural sound for videos in the wild. In Proc. CVPR, pages 3550-3558, 2018.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] In a separate subsection, after conclusions.
(c) Did you discuss any potential negative societal impacts of your work? [N/A] Our algorithm targets an already well established task, which has potentially beneficial applications for speech/musie separation, etc.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments...
13

---

(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes] The datasets we use are all public 8 47 And the code will be made available on the web.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Details are available the Implementation Details and the Datasets and Experimental Setup subsection.
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [No] Primarily because the performance gains obtained were quite significant. All experiments were conducted with the same seed.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] See the supplementary material.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either in the supplemental material or as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "tanejKplrqk", "reviewer": "Reviewer_bcMT", "summary": "This paper proposes to model the dynamics of audio-visual objects in the scene with a pseudo-3D scene graph. The model uses a RNN to roll out multiple features to represent the audio-visual features of different objects. This paper also estimates the ground truth sound source motion from the video and adds a direction prediction loss for predicting the direciton of motion from the separated sound source. Authors show that the proposed model outperfroms SOTA models on two in the wild audio-visual separation datasets.", "questions": "Why the features rolled out RNN from one single global feature is better than more object centric features?\n\nWhy is it posible to predict direction from single-channel audio?\n\nCan we see how the baselines do compared to your model for the same qualitative videos?", "limitations": "Yes, both are discussed.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The idea of constructing a scene graph for audio-visual separation is interesting and does go beyond the traditional object based audio-visual separation pipelines. The paper provides sufficient details for modeling and implementation. The performance improvement over the baseline also seems pretty impressive.\n\nHowever, I have doubts about the two proposed contributions. First of all, constructing scene graphs and learning audio-visual dynamics for separating sounds make sense. However, whether use RNN to generate multiple features from one global feature to represent differnet objects is questionable because the rollout is not conditioned on anything else. How do you make sure the generated features correspond to visual objects? It's not obvious to me why this representation is much better than object-centric features with some local motion features.\n\nSecondly, why is possible to predict direction from single-channel audio? The best scenario is you can predict whether the sound is moving closer to or away from the microphone. Even for that case, you need to assume the volume for the source sound is constant, which is certainly not true in many cases. I don't think it's possible to predict directions just from single-channel audio and if it does, then it's really questionable why it does. \n\nFor the qualitative video, it would also be better to display how your model does compared to the baselines. The separation results do not sound great to me, but it could be because of the lack of comparisons. \n\nAnother issue is also that the paper has a very complicated pipeline, which makes it hard to verify it's the proposed components that led to the performance improvement.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "qjPGyYTE6d8", "reviewer": "Reviewer_Zmj2", "summary": "The paper presents a method for visually-guided audio source separation and motion prediction (exclusively from sound). The method is trained using audio-visual data and operates by constructing a 2.5D scene graph, which is segmented into sub-graphs. Each sub-graph is associated with an audio source. The sub-graph embeddings are subsequently fed to an audio separator network that has a U-Net architecture and produces the separated audio for a particular object. An additional head (direction prediction network) is appended to the audio separator network to predict the motion of the object based on the 2.5D. This allows training using an additional direction prediction loss.", "questions": "Since two independent object detectors are used. How is the case where an object is picked up by both detectors (i.e., overlapping bounding boxes) handled?\n\nHow is the list of possible auditory classes obtained?\n\nConsidering that a mirrored image is equally likely to be associated with the same sound how does the network achieve an accuracy of >50% given that 2 (or more) octants are equiprobable? Also, why does this ambiguity not affect the training of the source separation network? Finally, since the motion prediction network works purely based on sound have the authors tested it on audio that has not been separated but contains only sound from one moving object?\n\nIn general, I believe that the method is clearly explained, novel and has been sufficiently evaluated in the paper. However, I do not see how the motion prediction from only mono audio can disambiguate the direction of movement (i.e. how can it tell left from right). I imagine that this ambiguity should also affect the training of the source separation. I would like to hear the authors' response on this before recommending the paper for acceptance. ", "limitations": "I do not see any serious potential negative societal impact that needs discussing in this work.", "rating": 7, "confidence": 3, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths:\n\n- The paper is well written.\n- The use of 3D scene dynamics for audio separation is a novel approach as is motion prediction from audio.\n- The evaluation on visually-guided source separation is reasonable. There are ample ablation studies performed in both the main paper and supplementary material. For audio source separation, the method is also tested on YouTube videos and examples are presented in the supplementary material.\n- The framework and models used are described in sufficient detail in the supplementary material and a code stub is provided.\n\n## Weaknesses:\n- Motion prediction from audio is interesting but the motivation for it is not clear, especially given that the audio is not stereo. This capability of the model I believe has not been sufficiently explored/explained in the paper and raises some questions (see Questions section).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "a2Z62e74uo9", "reviewer": "Reviewer_j9bT", "summary": "This paper proposes an audio-visual separation and 3d motion direction prediction model based on 3D visual scene graphs. For visual conditioning, an object detector is run on video reference frames, and depth is predicted from the video frames, both of which are combined to create a \"2.5D\" scene graph representation. N_u \"auditory objects\" are identified, and each object defines a subgraph along with their associated \"context\" nodes. Then, a recurrent network (a GRU) processes the resulting sub-graphs and predicts N_u+1 orthogonal embeddings (N_u auditory sources, and 1 background source). These embeddings are used to condition a U-Net operating on the input mixture spectrogram, where the U-Net conditions on each embedding to predict a mask for the corresponding source. From the separated spectrogram, a 3d motion direction vector is also predicted.\n\nThe model uses weak and self-supervised losses trained with the \"mix-and-separate\" paradigm (i.e. mixtures of 2 videos), including a classification consistency loss, direction prediction loss, co-separation loss (i.e. consistency of predicted masks with ideal oracle masks of the original two source video), and a loss promoting orthogonality of the conditioned embeddings from the GRU.\n\nThe proposed approach is tested on the ASIW and AVE datasets, and is shown to improve performance on standard SDR/SIR/SAR separation metrics, and also achieve good performance in terms of predicting the motion direction vector.\n\nOverall I think this is a nice paper with novel contributions, but I think a few important details need to be made more clear.", "questions": "## Questions\n\nQ1. What are the reference signals for SDR/SIR/SAR for ASIW and AVE? It seems that you are using a \"mix-and-separate\" approach that mixes videos together, and the references are the soundtracks of the two source videos? Or is the evaluation performed on single videos, in which case the SDR/SIR/SAR reference is between the sum of the separated sources and the original video's soundtrack? Also, this raises the following questions for me:\n\nQ1a. How many sources per source video?\n\nQ1b. Do these sources always originate from on-screen objects, and if so, how is this determined?\n\nQ1c. What is the level of background noise in videos?\n\nQ1d. Do all videos contain both on-screen and off-screen sound?\n\nQ2. Have you discovered any object classes that the proposed system does not cover (i.e. not contained in the 1600 Visual Genome classes or the Open Images classes)?\n\nQ3. \"For each of these auditory objects p_i, we identify the subset V_{p_i} of M − N objects that are non-auditory and overlap\": how do you determine if auditory objects are sounding or not? Does the approach assume that some classes of object are always making sound when on-screen?\n\nQ4. \"For every video in our setup, we detect upto [sic] two auditory objects, one background auditory object, and 20 context nodes.\": Okay, so it seems N_u is a fixed hyperparameter to choose. But I don't see how the consistency loss in Eq. (2) handles less than N_u+1 sources, because it operates over all permutations. Is N_u not fixed, and chosen by how many \"auditory\" objects are determined to be present? But that doesn't seem to be the case, according to the code stub in the supplementary. This again raises my question Q3, as to how auditory objects are identified.\n\nQ5. \"Direction Prediction Loss is a standard cross-entropy loss, which checks for the accuracy of the predicted octant number {0, 1, . . . , 8}\": does 0 indicate no motion, and 1-8 mean the 8 octants?\n\n\"The visual feature of the background object is derived from a FRCNN embedding of a random crop of the reference frame Fr.\": extracting a random crop seems less optimal than extracting a crop that does not overlap with any of the detected objects. Would this work better, and/or did you try this? What if the crop doesn't have visual content correlated with the background sound?\n\n## Minor comments and typos\n\n1. Nit on style: I generally prefer not to capitalize words in an acronym, e.g. \"Audio Separator and Motion Predictor (ASMP)\" -> \"audio separator and motion predictor (ASMP)\"\n\n2. \"with the kid,\": kind of informal\n\n3. \"Audio Separation Using Direction of Arrival\" section: perhaps mention some works on audio-only beamforming and multi-microphone processing?\n\n4. \"Lucak-Kanade\" -> \"Lucas-Kanade\"\n\n5. \"A video can therefore, be\" -> \"A video can therefore be\"", "limitations": "Limitations in terms of negative societal impact are sufficiently addressed.\n\nSome limitations of the model itself that I think would be good to discuss further are as follows:\n1) How many sources the model can handle? I suppose this is set by N_u? I can imagine a lot of real-world scenes, like a concert or busy street, where there are certainly more than 2 sounding objects.\n2) Can the model handle videos that only contain background sounds, i.e. contain only off-screen sounds?\n3) The model seems to rely on source class to do its separations. Are there source classes that the model does not work on because they are not covered, and can these perhaps be mitigated?", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\nS1. The demo videos are convincing, both in terms of separation and predicting 3d motion direction for sources. I think the 3d motion direction prediction is particularly impressive.\n\nS2. The paper is clear and well-written.\n\nS3. The results on ASIW and AVE clearly show improvement in terms of SDR/SIR/SAR.\n\nS4. Paper performs ablations and a user study.\n\n## Weaknesses\n\nW1. The system is quite complicated, with a lot of components. However, they are generally described well, and the approach is effective according to the objective results and demo videos.\n\nW2. More detail about the ASIW and AVE datasets that are relevant to the proposed separation task are required. See my Q1 below. I would be inclined to increase my score if these were made more clear.\n\nW3. Some details about the approach are still a bit unclear. Please see my questions below. I would be inclined to increase my score if these were made more clear.\n\nW4. It would be good to discuss more limitations of the method. See Limitations section below.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "1UsWJG5Y8b", "reviewer": "Reviewer_XeDg", "summary": "This paper proposes a new framework for audio-visual source separation by simultaneously predicting the 3D visual motion of a sounding source. A new Audio Separator and Motion Predictor (ASMP) framework is proposed to leverage the 3D structure of the scene and the\n9 motion of sound sources for better audio source separation. Experiments on two challenging audio-visual datasets demonstrate the usefulness of the proposed method.", "questions": "- As shown in the 3D scene graph, there are many silent objects like trees, sky, etc. These objects do not make sounds. How does the proposed framework handle silent objects? What is the percentage of silent objects and how will they affect the performance of separation?\n\n- When predicting the direction or 3D moving dynamics of the sounding objects, is the network focusing on the volume of the sound? Is there evidence that the model also makes use of the spatial cues in audio?", "limitations": "I cannot find a section to discuss limitations or societal impact.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The method is well motivated. Jointly predicting the 3D location in space of the sounding source can help to better separate the sounds they make.\n\n- The paper is generally well written with clear and sufficient details.\n\n- Extensive experiments are conducted are two datasets and the proposed method compares favorably to prior methods.\n\nWeakness:\n\n- Title is not the most accurate. Although the key idea is to predict 3D dynamics of sounding objects, but this paper is still more focused on audio-visual source separation. Predicting 3D dynamics is to help the separation learning, so it would be better if the title can indicate this aspect.\n\n- Related work, \"Audio Separation Using Direction of Arrival\" can be merged into the approach since only two references are discussed there. For better presentation, it can be briefly discussed when presenting the direction prediction network.\n\n- The performance of the proposed method is pretty close to AVSGS as shown in Table 1. AVSGS is also based on scene graphs but with 2D visual context. More analysis or discussions to compare with AVSGS would be desirable.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept