Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes to model the dynamics of audio-visual objects in the scene with a pseudo-3D scene graph. The model uses a RNN to roll out multiple features to represent the audio-visual features of different objects. This paper also estimates the ground truth sound source motion from the video and adds a direction prediction loss for predicting the direciton of motion from the separated sound source. Authors show that the proposed model outperfroms SOTA models on two in the wild audio-visual separation datasets.
strengths_and_weaknesses: The idea of constructing a scene graph for audio-visual separation is interesting and does go beyond the traditional object based audio-visual separation pipelines. The paper provides sufficient details for modeling and implementation. The performance improvement over the baseline also seems pretty impressive.

However, I have doubts about the two proposed contributions. First of all, constructing scene graphs and learning audio-visual dynamics for separating sounds make sense. However, whether use RNN to generate multiple features from one global feature to represent differnet objects is questionable because the rollout is not conditioned on anything else. How do you make sure the generated features correspond to visual objects? It's not obvious to me why this representation is much better than object-centric features with some local motion features.

Secondly, why is possible to predict direction from single-channel audio? The best scenario is you can predict whether the sound is moving closer to or away from the microphone. Even for that case, you need to assume the volume for the source sound is constant, which is certainly not true in many cases. I don't think it's possible to predict directions just from single-channel audio and if it does, then it's really questionable why it does. 

For the qualitative video, it would also be better to display how your model does compared to the baselines. The separation results do not sound great to me, but it could be because of the lack of comparisons. 

Another issue is also that the paper has a very complicated pipeline, which makes it hard to verify it's the proposed components that led to the performance improvement.
questions: Why the features rolled out RNN from one single global feature is better than more object centric features?

Why is it posible to predict direction from single-channel audio?

Can we see how the baselines do compared to your model for the same qualitative videos?
limitations: Yes, both are discussed.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes an audio-visual separation and 3d motion direction prediction model based on 3D visual scene graphs. For visual conditioning, an object detector is run on video reference frames, and depth is predicted from the video frames, both of which are combined to create a "2.5D" scene graph representation. N_u "auditory objects" are identified, and each object defines a subgraph along with their associated "context" nodes. Then, a recurrent network (a GRU) processes the resulting sub-graphs and predicts N_u+1 orthogonal embeddings (N_u auditory sources, and 1 background source). These embeddings are used to condition a U-Net operating on the input mixture spectrogram, where the U-Net conditions on each embedding to predict a mask for the corresponding source. From the separated spectrogram, a 3d motion direction vector is also predicted.

The model uses weak and self-supervised losses trained with the "mix-and-separate" paradigm (i.e. mixtures of 2 videos), including a classification consistency loss, direction prediction loss, co-separation loss (i.e. consistency of predicted masks with ideal oracle masks of the original two source video), and a loss promoting orthogonality of the conditioned embeddings from the GRU.

The proposed approach is tested on the ASIW and AVE datasets, and is shown to improve performance on standard SDR/SIR/SAR separation metrics, and also achieve good performance in terms of predicting the motion direction vector.

Overall I think this is a nice paper with novel contributions, but I think a few important details need to be made more clear.
strengths_and_weaknesses: ## Strengths

S1. The demo videos are convincing, both in terms of separation and predicting 3d motion direction for sources. I think the 3d motion direction prediction is particularly impressive.

S2. The paper is clear and well-written.

S3. The results on ASIW and AVE clearly show improvement in terms of SDR/SIR/SAR.

S4. Paper performs ablations and a user study.

## Weaknesses

W1. The system is quite complicated, with a lot of components. However, they are generally described well, and the approach is effective according to the objective results and demo videos.

W2. More detail about the ASIW and AVE datasets that are relevant to the proposed separation task are required. See my Q1 below. I would be inclined to increase my score if these were made more clear.

W3. Some details about the approach are still a bit unclear. Please see my questions below. I would be inclined to increase my score if these were made more clear.

W4. It would be good to discuss more limitations of the method. See Limitations section below.
questions: ## Questions

Q1. What are the reference signals for SDR/SIR/SAR for ASIW and AVE? It seems that you are using a "mix-and-separate" approach that mixes videos together, and the references are the soundtracks of the two source videos? Or is the evaluation performed on single videos, in which case the SDR/SIR/SAR reference is between the sum of the separated sources and the original video's soundtrack? Also, this raises the following questions for me:

Q1a. How many sources per source video?

Q1b. Do these sources always originate from on-screen objects, and if so, how is this determined?

Q1c. What is the level of background noise in videos?

Q1d. Do all videos contain both on-screen and off-screen sound?

Q2. Have you discovered any object classes that the proposed system does not cover (i.e. not contained in the 1600 Visual Genome classes or the Open Images classes)?

Q3. "For each of these auditory objects p_i, we identify the subset V_{p_i} of M − N objects that are non-auditory and overlap": how do you determine if auditory objects are sounding or not? Does the approach assume that some classes of object are always making sound when on-screen?

Q4. "For every video in our setup, we detect upto [sic] two auditory objects, one background auditory object, and 20 context nodes.": Okay, so it seems N_u is a fixed hyperparameter to choose. But I don't see how the consistency loss in Eq. (2) handles less than N_u+1 sources, because it operates over all permutations. Is N_u not fixed, and chosen by how many "auditory" objects are determined to be present? But that doesn't seem to be the case, according to the code stub in the supplementary. This again raises my question Q3, as to how auditory objects are identified.

Q5. "Direction Prediction Loss is a standard cross-entropy loss, which checks for the accuracy of the predicted octant number {0, 1, . . . , 8}": does 0 indicate no motion, and 1-8 mean the 8 octants?

"The visual feature of the background object is derived from a FRCNN embedding of a random crop of the reference frame Fr.": extracting a random crop seems less optimal than extracting a crop that does not overlap with any of the detected objects. Would this work better, and/or did you try this? What if the crop doesn't have visual content correlated with the background sound?

## Minor comments and typos

1. Nit on style: I generally prefer not to capitalize words in an acronym, e.g. "Audio Separator and Motion Predictor (ASMP)" -> "audio separator and motion predictor (ASMP)"

2. "with the kid,": kind of informal

3. "Audio Separation Using Direction of Arrival" section: perhaps mention some works on audio-only beamforming and multi-microphone processing?

4. "Lucak-Kanade" -> "Lucas-Kanade"

5. "A video can therefore, be" -> "A video can therefore be"
limitations: Limitations in terms of negative societal impact are sufficiently addressed.

Some limitations of the model itself that I think would be good to discuss further are as follows:
1) How many sources the model can handle? I suppose this is set by N_u? I can imagine a lot of real-world scenes, like a concert or busy street, where there are certainly more than 2 sounding objects.
2) Can the model handle videos that only contain background sounds, i.e. contain only off-screen sounds?
3) The model seems to rely on source class to do its separations. Are there source classes that the model does not work on because they are not covered, and can these perhaps be mitigated?
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper presents a method for visually-guided audio source separation and motion prediction (exclusively from sound). The method is trained using audio-visual data and operates by constructing a 2.5D scene graph, which is segmented into sub-graphs. Each sub-graph is associated with an audio source. The sub-graph embeddings are subsequently fed to an audio separator network that has a U-Net architecture and produces the separated audio for a particular object. An additional head (direction prediction network) is appended to the audio separator network to predict the motion of the object based on the 2.5D. This allows training using an additional direction prediction loss.
strengths_and_weaknesses: ## Strengths:

- The paper is well written.
- The use of 3D scene dynamics for audio separation is a novel approach as is motion prediction from audio.
- The evaluation on visually-guided source separation is reasonable. There are ample ablation studies performed in both the main paper and supplementary material. For audio source separation, the method is also tested on YouTube videos and examples are presented in the supplementary material.
- The framework and models used are described in sufficient detail in the supplementary material and a code stub is provided.

## Weaknesses:
- Motion prediction from audio is interesting but the motivation for it is not clear, especially given that the audio is not stereo. This capability of the model I believe has not been sufficiently explored/explained in the paper and raises some questions (see Questions section).
questions: Since two independent object detectors are used. How is the case where an object is picked up by both detectors (i.e., overlapping bounding boxes) handled?

How is the list of possible auditory classes obtained?

Considering that a mirrored image is equally likely to be associated with the same sound how does the network achieve an accuracy of >50% given that 2 (or more) octants are equiprobable? Also, why does this ambiguity not affect the training of the source separation network? Finally, since the motion prediction network works purely based on sound have the authors tested it on audio that has not been separated but contains only sound from one moving object?

In general, I believe that the method is clearly explained, novel and has been sufficiently evaluated in the paper. However, I do not see how the motion prediction from only mono audio can disambiguate the direction of movement (i.e. how can it tell left from right). I imagine that this ambiguity should also affect the training of the source separation. I would like to hear the authors' response on this before recommending the paper for acceptance. 
limitations: I do not see any serious potential negative societal impact that needs discussing in this work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a new framework for audio-visual source separation by simultaneously predicting the 3D visual motion of a sounding source. A new Audio Separator and Motion Predictor (ASMP) framework is proposed to leverage the 3D structure of the scene and the
9 motion of sound sources for better audio source separation. Experiments on two challenging audio-visual datasets demonstrate the usefulness of the proposed method.
strengths_and_weaknesses: Strengths:

- The method is well motivated. Jointly predicting the 3D location in space of the sounding source can help to better separate the sounds they make.

- The paper is generally well written with clear and sufficient details.

- Extensive experiments are conducted are two datasets and the proposed method compares favorably to prior methods.

Weakness:

- Title is not the most accurate. Although the key idea is to predict 3D dynamics of sounding objects, but this paper is still more focused on audio-visual source separation. Predicting 3D dynamics is to help the separation learning, so it would be better if the title can indicate this aspect.

- Related work, "Audio Separation Using Direction of Arrival" can be merged into the approach since only two references are discussed there. For better presentation, it can be briefly discussed when presenting the direction prediction network.

- The performance of the proposed method is pretty close to AVSGS as shown in Table 1. AVSGS is also based on scene graphs but with 2D visual context. More analysis or discussions to compare with AVSGS would be desirable.
questions: - As shown in the 3D scene graph, there are many silent objects like trees, sky, etc. These objects do not make sounds. How does the proposed framework handle silent objects? What is the percentage of silent objects and how will they affect the performance of separation?

- When predicting the direction or 3D moving dynamics of the sounding objects, is the network focusing on the volume of the sound? Is there evidence that the model also makes use of the spatial cues in audio?
limitations: I cannot find a section to discuss limitations or societal impact.
review_title: null

Meta-review: metareview: All four reviewers agree that this paper demonstrates strong improvements over prior methods, and there is broad agreement among the reviewers that the model is well motivated and novel, and that the paper is clearly written.

There was a good discussion between the authors and reviewers on a number of perceived weaknesses in the paper, and the authors were able to address these concerns with additional experiments and proposed revisions, prompting two reviewers to raise their scores. In the end, all reviewers recommend acceptance to some degree, and in my judgement, the most negative reviewer (who recommends borderline accept) has missed the point, made both in the paper and during the discussion, that the estimation of source direction from single-channel audio depends not only on audio cues, but also on video cues.
confidence: Certain
Decision: Accept