Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In this paper authors introduce a self-supervised learning method from audio-visual data. Specifically, the method revolves around using moments of interaction to learning meaningful audio-visual relation. Authors split the learning in two losses: an standard audio-visual loss, and a loss around the audible change of state and its relation to the visual change of state. Authors evaluate their proposed method in EpicKitchens and Ego4D. 

strengths_and_weaknesses: **Strengths**:

- S1. Audio-visual self-supervised learning is a powerful tool to learn representations. However, typical approaches do not exploit the semantic importance of the moments where to sample the audio. This work is a step towards using the audio content in a most meaningful way.

- S2. The results show how the method improves over the baseline with standard audio-visual learning. 

- S3. Authors evaluate in two very relevant well-known ego centric benchmarks. I believe that is important as it makes the paper stronger.

- S4. Ego-centric data is going to be becoming more important in the near future. With the progress of recording devices and embodied research, works along the lines of this paper will become more relevant.

- S5. I specially like the idea of detecting state changes through audio. According to the paper the procedure is quite simple and the self-supervised method certainly benefits from seeing samples around that moment.

**Weaknesses**

- W1. Authors compare with a single baselines for audio-visual learning. I think other works such as MMV (Alayrac, NeuriPS 2020),  Evolving Losses (Piergiovanni, CVPR 2020), Brave (Recasens, ICCV 2021), XDC (Alwassel, NeurIPS 2020) are very relevant in the community. I understand the topic is slightly different and authors cannot retrain with all the baselines, but I still believe that having a single baseline such as AVID is insufficient for a publication. 

- W2. The authors are missing citations for some of the papers mentioned in W1 (e.g. MMV, Evolving Losses). I think those works are important in the space of self-supervised audio-visual learning.

- W3. I think authors do not properly evaluate the ability of their model to detect moments of interaction. The description of the method is very complete and in Table 1 they ablate using the method, but I think it would be good to somehow evaluate whether the proposed method works.

- W4. I am missing some examples of moments of interaction I understand the reasoning behind the methodology of using the reverse clip as negative, but I believe the readers would benefit from a few visual examples to understand that.


questions: My concerns are listed in Weaknesses. My main questions to the authors would be:

- Have the authors considered adding additional baselines when comparing to the state-of-the-art? If not, why not?

- Which evidence do the authors have that the detection of change works properly if they don't have any evaluation for that signal.

- In general, could the authors show some more examples of moments of interaction (similar to the ones in supplementary) in the main paper. I think it would be specially interesting to see examples where the detection of those moments works and other where it fails.
limitations: Authors discuss the limitations of the method in 4.2. I would suggest expanding that a bit with general negative societal impact that working with ego video can have. I believe egocentric video can be used to track everyday's life which could have some negative impact.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposes leveraging audible state changes for self-supervised representation learning from long-form egocentric videos. Authors initially detect the timestamps where interesting interactions occur (MoI) and then compute a cross modal contrastive objective where the natural transition (before, interaction, after) is more likely than one which is temporally reverse (after, interaction, before). 
strengths_and_weaknesses: Paper is clear in presentation and has provided an interesting view to self-supervised multi-modal representation learning in egocentric videos with audible state change. The idea is pretty close to "Actions ∼ Transformations" <https://arxiv.org/pdf/1512.00795.pdf> which unfortunately is missing from the references. Below are my detailed comments.

- Line 48-49: Authors argue that AVC objective leads to representations that are not informative of the changes over time. If indeed the sound of “opening fridge” is different from that of “closing fridge”, AVC will encourage video representation of “opening fridge” to be more similar to the audio representation of “opening fridge” than “closing fridge” since audio embedding of the latter will be served as a negative instance within the contrastive framework. Note that, you do not have to sample after the fact e.g once the fridge is closed/opened, instead you can sample during the action e.g while fridge is being opened/closed, something that working with video in a cross-modal contrastive setup easily allows you to do. With that said, I would like to see authors clarifying their point on the lack of AVC’s suitability for the task specially given the ablation studies that show dropping AVC vs AStC is not that much different.

- On finding MoI: As paper mentions, egocentric videos are naturally long-form, as they continuously capture daily activities. Hence, a person moving in an environment i.e change in location, even without any interaction can result in changes in audio spectrogram. For example, a person is cooking some food in the kitchen, then walks to the living room to pick up a book. The fact that there is a fan or stereo playing in the living room, will result in a change in audio (note that the sound of fan or stereo is not necessarily audible in the kitchen), however those are not MoI since there has been no interaction with either stereo or fan yet from Sec 3.2 it seems to me that the proposed MoI approach should pick those. In nutshell, change in audio is not only as a result of human-object interaction, it can be due to change of location or variant environment as well and I cannot see how the proposed MoI detection method can work in a realistic environment.

- I am not convinced that learning from audible state change as described in Sec 3.3 is generic enough. For example, the visual state is different after hearing the sound of MoI for “opening fridge” , while before state shows a closed fridge, the after state depicts an open one. Also, due to “distinct” sound of “opening fridge” the backward transition should be less likely. Now consider the example of “cutting cucumber”, the backward transition almost never happens (humans usually don’t stitch cucumber slices together!), while it is reasonable to go from a closed fridge to an open one and vice versa. There are also cases which despite audible MoI, visuals look almost identical in before and after like “stirring a pot”. I suspect that more clear performance gains seen on Ego4D versus Epic-Kitchens is partly related to the state change properties which are more prominent in the former dataset. I would like to hear authors feedback on the different aforementioned types of interactions and why their proposed model should work in a self-supervised setup where we don’t know which type of these interactions are included in the training data.

- Line 269: I do not think it is fair to compare rows 2 and 7 since row 2 has been only trained on AudioSet which is very different from evaluation egocentric datasets. To see the true additional value of your contributions (use of MoI and AStC loss), AVID should be fine-tuned on the egocentric datasets (equally RepLAI w/o AStC and MoI)
questions: - How do you handle the audio diversity of semantically similar interactions? For example, “cutting cucumber” on a wooden board will sound different from doing it on a plastic board, or sound of “sautéing” mushrooms/onions in a pan is meaningfully influenced by the oven/pan/oil temperature.

- As part of ablation studies, have you tried a linear head for $h_{\Delta V}^{AStC}$ since forward delta seems to be the negative of the backward one.

- AVC loss shown in Figure 3.b encourages the embedding of $V_{t-\delta}$ to be close to $V_{t+\delta}$ by both anchoring on audio embedding at t while the AStC encourages them to be different. These two objective functions, to the best of my understanding are pulling in opposite directions! Would not it make more sense to choose the audio at $t-\delta$ and $t+\delta$, instead of t, when computing two AVC losses?

- In Table 1, Top 5 Acc: AVC seems to be doing most of the job while w/o AStC and MoI, performance on “verb” is almost maintained (~73%) but the pattern is different for “noun”. Any insights?

limitations: Authors have addressed the limitations
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a new way for self-supervised learning with egocentric videos to learn from audible interactions. Specifically, it uses audio signals to identify the state changes and use these transition states to learn the audio-visual correlations. The proposed method performs much better than a recent method AVID [42] and the ablation study shows the effectiveness of each model component. 
strengths_and_weaknesses: Strengths

- The idea of using audio to identify the state changes for learning audio-visual correlation is novel and seems effective. 
- The paper is well written and easy to read
- The performance of this method is good. 

Weaknesses

- The example used for illustrating Eq. 1 are not convincing. The given example is the audio of closing the door should be (1) similar to the visual transition opened door → closed door and (2) dissimilar to the (backwards) transition closed → open. This activity has strong temporal order. But how to deal with activities like stirring and washing? 
- There is lack of the fully-supervised performance for comparison in experiments. 
questions: My main question is the generalization ability of the proposed method, i.e., increasing the probability of associating the audio with the visual state change in the forward direction but decreasing the probability in the backward direction. It can work well for activities like opening/closing the door intuitively. However, how can it work for others like stirring and washing? 
limitations: The limitations and potential negative impact are discussed in the supplementary.
review_title: null

Meta-review: metareview: The paper presents self-supervised representation learning from egocentric video data. The reviewers unanimously support the paper. Although WY3G has not updated the rating, the reviewer commented that s/he is upgrading the rating to Weak Accept, making the paper get three unanimous Weak Accept ratings. All three reviewers find the idea of using audio to identify the state changes for learning audio-visual correlation interesting. The authors are encouraged to include added experimental results they provided during the discussion phase to the final version of the paper.

confidence: Certain
Decision: Accept