Learning State-Aware Visual Representations from Audible Interactions
Authors: Himangi Mittal
Pedro Morgado
Unnat Jain
Abhinav Gupta
Venue: NeurIPS 2022 Accept (2022)
Keywords: Video Representation learning
self-supervised learning
contrastive learning
audio-visual learning
egocentric videos
Ego4D
EPIC-Kitchens

Abstract: We propose a self-supervised algorithm to learn representations from egocentric video data. Recently, significant efforts have been made to capture humans interacting with their own environments as they go about their daily activities. In result, several large egocentric datasets of interaction-rich multi-modal data have emerged. However, learning representations from videos can be challenging. First, given the uncurated nature of long-form continuous videos, learning effective representations require focusing on moments in time when interactions take place. Second, visual representations of daily activities should be sensitive to changes in the state of the environment. However, current successful multi-modal learning frameworks encourage representation invariance over time. To address these challenges, we leverage audio signals to identify moments of likely interactions which are conducive to better learning. We also propose a novel self-supervised objective that learns from audible state changes caused by interactions. We validate these contributions extensively on two large-scale egocentric datasets, EPIC-Kitchens-100 and the recently released Ego4D, and show improvements on several downstream tasks, including action recognition, long-term action anticipation, and object state change classification.

Learning State-Aware Visual Representations from Audible Interactions
Himangi Mittal', Pedro Morgado', Unnat Jain', Abhinav Gupta'
'Carnegie Mellon University
Meta Al Research
Abstract
We propose self-supervised algorithm to learn representations from egocen- tric video data. Recently, significant efforts have been made to capture humans interacting with their own environments as they go about their daily activities. In result, several large egocentric datasets of interaction-rich multi-modal data have emerged. However, learning representations from videos can be challeng- ing. First, given the uncurated nature of long-form continuous videos, learning effective representations require focusing on moments in time when interactions take place. Second, visual representations of daily activities should be sensitive changes the state of the environment. However, current successful multi- modal learning frameworks encourage representation invariance over time. To address these challenges, we leverage audio signals identify moments of likely interactions which are conducive to better learning. We also propose a novel self- supervised objective that learns from audible state changes caused by interactions. We validate these contributions extensively SPICCKKicchee
two large-scale egocentric datasets, and the recently released Ego4D, and show improvements several downstream tasks, including action recognition, long-term action an- ticipation, and object state change classification. Code and pretrained model are available here: https: //github com/ www./ooaan/iiiiieeeeee Hiimangii/mepplii....
1 Introduction
Recent successes in self-supervised learning (SSL) [48, 10, 31, 28] has brought into question the need for human annotations in order to learn strong visual representations. However, current approaches are bottlenecked by the lack of rich data - they learn from static images which lack temporal information and restrict the ability learn object deformations and state changes. It is clear that need videos to learn rich representations in self-supervised manner.
Learning representations from videos is however quite challenging. The first challenge is choosing the right SSL loss. Approaches such as [67, 54] have attempted learn representations that are invariant to object deformations//ienn wwwwooonns.i..sissss..... However, many downstream tasks require representations that are sensitive these deformations. Another alternative has been use the multi-modal data [3, 43. 57] and learn representations via audio. But again most of these approaches seek to align audio and visual features in a common space, leading to invariant representations as well. The second challenge dealing with the fact that current video-based SSL. approaches exploit the curated nature of video datasets, such as Kineties [9]. These approaches are designed to leverage carefully selected clips, displaying single action object interaction. This is in contrast to the predominantly untrimmed real-world data characteristic large egocentric datasets of daily activities. Here, unlike action centric datasets, the most 'interesting' human annotators. Thus, learning from untrimmed video poses a major challenge, as a significant portion of the data does not focus on the concepts we want to learn
'interaction-rich' clips have NOT been carefully selected by
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Before moment with ND interaction
Before
moment of interaction
Random Moment
Moment Interaction (Mal)
After moment with NO interaction
After moment interaction
No State Change
Changes bok choy chopped bok choy
Figure 1: Moments of audible interactions in long, untrimmed video. Random moments in time are likely to contain NO interactions, e.g., timestamp shown by the gray box. Since no interactions occur, no changes in the before and after states are observed. By sampling from moments of interaction (Mol), shown in the red box, we can learn representations that are sensitive to the change in the visual state caused by interactions.
In this work, we ask the question, 'Can we learn meaningful representations from interaction-ric rich, multi-modal streams of egocentric data?" Learning from continuous streams of data requires focusing on the right moments when the actual interactions are likely to occur. Consider, for example, the acts of opening fridge or placing pan the stove. Actions like these create clear and consistent sound signatures due to the physical interaction between objects. These moments can be easily detected from audio alone and can be used target training interesting portions of the untrimmed videos. We show that even a simple spectrogram-based handcrafted detector is sufficient to identify interesting moments in time, and that representation learning benefits substantially from using them to sample training clips.
But what should the loss be? Prior work on audio-visual correspondence (AVC) [15, 4, 43] uses the natural co-occurrence of sounds and the visual manifestations of their sources as the source of supervision. However, since the AVC objective still favors invariance, the learned representations not informative of the changes that happen over time (e.g., representations that can distinguish between closed and opened fridge, or vegetables before and after chopping them). To better capture state changes, we introduce a novel audio-visual self-supervised objective, in which audio representations key moments in time are required to be informative of the change the corresponding visual representations over time. The intuition behind this objective is that transitions between object states often marked by characteristic sounds. Thus, models optimized under this objective would associate the distinct sounds not only with the objects themselves (as accomplished with AVC), but also with the transition between two different states of the object.
To this end, we introduce RepLAI- - Representation Learning from Audible Interactions, a self- supervised algorithm for representation learning from videos of audible interactions. RepLAI uses the audio signals two unique ways: (1) to identify moments in time that are conducive to better self-supervised learning and (2) to learn representations that focus the visual state changes caused by audible interactions. We validate these contributions extensively on two egocentric datasets, EPIC-Kitchens- 100 [14] and the recently released Ego4D [27], where we demonstrate the benefits of RepLAI for several downstream tasks, including action recognition, long term action anticipation, and object state change classification.
2 Related Work
Self-supervised learning. Self-supervised learning methods operate on an unlabeled dataset by explicitly defining pretext tasks such as solving jigsaw puzzle [47], patch location prediction [16]. inpainting [50], and image rotation [25] prediction. Following these, the next wave of self- self-super methods has been based on contrastive learning that learns representations with the help of data aug- mentation and instance discrimination [10, 28, 48, 31, 8). These methods have shown rapid progress in self-supervised learning for images. While these approaches explore the spatial information of images, RepLAI leverages the temporal information of videos.

---

Video representation learning, Relevant to our proposed approach is self-supervised representation learning for videos where the spatiotemporal pretext tasks are designed such as temporal order prediction [40, 70, 35, 69], predicting motion and appearance statistics [65], pace prediction [66], temporal cycle consistency [18, 68], and video colorization [64]. Contrastive learning has also been widely adopted in the domain of video [55, 29, 32, 57, 71, 30, 22] with impressive results on action recognition tasks. These methods however learn representations that are invariant to spatio-temporal augmentations, such as temporal jittering. and thus are incapable of representing object state changes. Closer to the objective of RepLAI, we include relevant literature on audio-visual representation learning from videos, where the audio stream is additionally utilized.
Audio-visual representation learning. Learning without additional supervision has also been ex- plored in the context of the audio modality with the help of audio-visual correspondence (AVC) [4, S]. As stated simply, AVC is the binary classification task of predicting if video clip and short audio clip correspond with each other or not (details in Sec. 3.4). Similar tasks like temporal synchroniza- tion [36, 49] between audio and video, audio classification [6. 3, 111, spatial alignment prediction between audio and 360-degree videos [41], optimal combination of self-supervised tasks [52] have been shown beneficial for learning effective multi-modal video representations. Other works ex- plore contrastive learning for both audio and video modality [43, 51, 42] as cross-modal instance discrimination task.
Fine-grained video understanding, Real-world videos are often untrimmed in nature and have multiple actions in a single video. Along this line, fine-grained analysis has been studied for videos in the form of a query-response temporal attention mechanism [72], bi-directional RNN s[58], and remi-supervised learning problem [17]. While these works only utilize the visual modality, another line of work has also explored multi-moda fine-grained video understanding as transformer- model [34], by exploiting the correspondence between modalities [44], or by exploring how to best combine multiple modalities audio, visual, and language [2]. In our work, we try to conduct fine-grained video understanding in a self-supervised manner.
Egocentric datasets. Egocentric datasets offer new opportunities to learn from first-person point of view, where the world seen through the eyes of an agent. Many egocentric datasets have been developed such as Epic-kitchens [13, 14] which consist of daily activities performed in kitchen environment, Activities of Daily Living [53], UT Ego [37, 60], the Disney Dataset [20], and the recently released large-scale Ego4D dataset [27] which consists of day-to-day life activities in multiple scenarios such as household, outdoor spaces, workplace, etc. Multiple challenges and downstream tasks have explored for egocentric datasets like action recognition [34, 33, 38], action
localization [56], action anticipation [26, 59, 39, 1, 23], human-object interactions [45, 12, 7], parsing social interactions [46], and domain adaptation [44]. In our work, we evaluate the efficiency of the representations learned by our self-supervised approach on the EPIC-Kitchens- 100 and Ego4D
datasets over multiple downstream tasks.
3 RepLAI
In this section, we detail our approach to learn audio-visual representations from and for interaction- rich egocentric data in a self-supervised manner, i.e., without relying on human annotated labels. Sec. 3.1 provides overview of RepLAI and motivates the two key contributions of this work identifying 'moments of interaction' (Mol) and learning from 'audible visual state changes'. Sec. 3.2 details the proposed approach for Mol detection and section Sec. 3.3 explains the proposed self- supervised objective for learning state-aware representations. Sec. 3.4 explains the objective of audio-visual correspondence learning used to train RepLAI. Sec. 3.5 brings both objectives together and includes necessary details for reproducibility.
3.1 Overview
Given a dataset D D= ((U;, M containing N long (untrimmed) audio-visual streams, our goal is to learn visual and audio encoders, denoted fv and fA. that can effectively represent egocentric data. An overview of the proposed approach is depicted search for moments of interaction (Mol) using the audio stream, and extract short audio and visual clips around these Mol. These trimmed clips are then encoded into a vectorized representation using fv and fA. The whole system is trained to optimize two self-supervised losses an audio-visual
Fig. 2. For each sample (v, a) D, we
3

---

Clip Extraction
Audio-Visual Encoders
Self-Supervised Loss
Untrimmed visual stream v
Ut 5
Ve-6N111
Visual state change
Trim
fv
4e
ve+6
Mol Detection
Lave
LASIC
Untrimmed
Trim
fa
Audio-Visud Pairs
Untrimmed audio stream
a2
8
seeks to learn audio and visual encoders (fa and fv: by (1) detecting and focusing training data moments of interaction (Mol) present in untrimmed videos and (2) solving combination of two tasks audio-visual correspondence (AVC) and audio identifiable state changes (AStC).
Figure 2: Overview of RepLAL. RepLAI
correspondence loss Lvve. and novel self-supervised loss that learns from audible state changes fAstC.
Why detect moments of interaction (Mol)? Untrimmed video daily activities often contains long periods without interactions, which aren't useful for training. Instead, we search for moments in time that are more likely to contain interactions which we refer to as moments of interaction (Mol).
Why learn from audible state changes? Visual representations of daily activities should be informative of the state of the environment and/or objects being interacted with. Morcover, changes in the environment are usually caused by physical interactions, which produce distinct sound signatures. We hypothesize that state-aware representations can obtained by learning to associate audio with the change of visual representation during a moment of interaction.
3.2 Audio-driven detection of moments of interaction
Audio signals are particularly informative of moments of interaction. To complete day-to-day activities, we physically interact with objects in our environments. These interactions typically produce distinct audio patterns short bursts energy that span frequencies. This illustrated in Fig. 1, where of actions in the kitchen. The audio data is represented as a log mel spectrogram, where the x-axis represents time and y-axis the audio frequency in log-scale. As can be seen, moments of interaction appear in the spectrogram as vertical edges, which can be easily detected. Once detected, short clips around the moments of interaction collected into dataset DMoI. and used for training.
visualize the untrimmed visual and audio data of a person performing series
The remaining question is how locate the timestamp of such vertical edges? Intuitively, we do this by finding robust local maxima in the total energy (summed over all frequencies) of the spectrogram. Concretely, let (t.,,aa w) be the value the log mel spectrogram an audio clip at time and frequency w. To remove the influence of background noise and overall audio intensity/volle volume, we compute the
z-score normalization of the spectrogram for each frequency independently (1,ww) s(t.w) tu
where is small constant for numerical stability. Here, Pau and a.,, the mean and standard deviation of (t, w) over time, respectively.' Next, we define moments interaction as the set of timestamps which are local maxima of Σ., st,, (or peaks for short). Moreover, avoid weak local maxima that may be caused by the noisy nature of audio signals, we ignore peaks with small prominence (lower than 1)². For further robustness, when multiple close peaks found (less than 50ms apart), only the highest prominence peak is kept.
3.3 Learning from audible state changes
Physical interactions often cause both state changes in the environment and distinct audio signals. To leverage this natural co-occurrence, we propose a self-supervised task that seeks to associate the audio with changes in the visual state during moment of interaction.
SSpecifically, μw Ex [t, o3 [x( p.) and le 5.
'The prominence of a peak is defined as the difference between the peak value and the minimum value in small window around it.

---

Learning from Audible State Changes
Audio-Visual Correspondence
Clip Extraction
Audio-Visual Encoders
ASIC Projection Heads
Loss Function
Audio-Visual
Projection
AVC
P4-6
Ve-á
fr
Am
fv
Lave
Video
Trim
CASIC
Video
Ve+-
fo
fv
h
Lave
24-5
Mol
Mol
(ie
Audio
Trim
fx
Audio
(a) The proposed AStC formulation (Sec. 3.3)
(b) Audio-visual correspondence (Sec. 3.4)
Figure 3: RepLAI architecture for AStC and AVC tasks. In both cases, short clips extracted around moments of interaction (Mol). (a). In AStC, representation a visual state change Av, matched to corresponding audio a,. (b). AVC seeks to associate audio a, with the corresponding visual clips vt.
The proposed task is optimized by minimizing loss with two negative log-likelihood terms to: (I) increase the probability of associating the audio with the visual state change in the forward (i.e. correct) direction, (2) decrease the probability of associating the audio with the visual state change in the backward (i.e. incorrect) direction. Consider, for example, the interaction of "closing a fridge door'. To optimize for this task, the audio of closing the door should be (1) similar the visual transition opened door + closed door and (2) dissimilar to the (backwards transition closed + open, This encourages learning of representations that are informative of object states, making them useful for a variety of egocentric tasks. Specifically, the audible state change (AStC) loss is defined as
LASEC = EV,,, a.reeDaan EDaaa [- log (p frwd (wr. ox)) - log (1 bkwd (Ur, az))]
The probabilities (p/rd,, ppbwdd are computed from cross-modal similarities
p (we, ax) = (sim (Avfrdd, fred aa) /T) p hkwd (vr, (t) at) a (sim m Avhiwd
(2)
hkwd at) /T)
(3)
where r 0.2 is temperature hyper-pase prper-parameter, and a denotes the sigmoid function. For better readability, we absorb the notations for the audio projection MLP head h and the state change projection MLP head 1: within sim(;,)) -), but their usage is clearly illustrated in Fig. 3a.
Audio representations (a,) are obtained by encoding the trimmed audio clips ag via the audio encoder fA (shared across all objectives). As explained above, is further projected via h to space where similarity visual state changes is enforced.
State change representations (Avfrrd, Av. are computed by considering two non-overlapping visual clips for each moment of interaction i, timestamps t 5 and + 5. The two clips, 0 and U are encoded via the visual encoder fv (shared across all tasks) and a projection MLP head h hastc (specific to the AStC task). Specifically, we represent forward and backward state changes as
fred hASeC (v+4+))) hAgUC fv (8e-)),
Av Mewd = 1. o fy (v.-))) - hyg (v(+5)).
(5)
In summary, optimizing the loss of Eq. 1 not only requires the audio representation to be aligned with representation of visual change A Ayrrdd that took place, but also be different from the hypothetical backward state change Avhkwd.
3.4 Learning from audio-visual correspondences [15, 4, 43]
Audio-visual correspondence (AVC) is well-studied self seff-supervised methodology for learning unimodal audio and visual encoders. The key idea is to bring visual and audio clips into common feature space, where the representations
audio-visual pairs are aligned. Note that AVC differs from the proposed AStC task, as AVC seeks to associate the audio o, with the corresponding visual clips v,- as opposed the change in visual state Av, As a result, visual representations learned through AVC are biased towards static concepts, while those leamed through AStC are more sensitive to dynamic
5

---

concepts. Since both types of representations can be useful for egocentric tasks, we further train the visual and audio encoders, fy and fA, for the AVC task.
Specifically, consider a dataset of audio-visual pairs (vi, a.) with representations v; = =f=v((i (m:) and a, (a:)) In particular, we let (vi, 0;) be short clips extracted from sample i around one of the detected moments of interest. Then, following [43, 61], audio-visual correspondence is established
by minimizing a cross-modal InfoNCE loss of the form
@in(ve,ao)/T
cain(v,,a)/
Lave a log Σ, ecin(viay)// log
(6)
Σ, esimly,,
where r = 0.07 is temperature hyper-parameter and sim( denotes the cosine similarity. Both terms in Eq. 6 help bring v; and a, (i.e. the positives) together. The key difference is whether the negative set composed of audio representations a, or visual representations V, where #
For readability of Eq. 6, we once again absorb the notation for the audio and visual projection MLP heads (h-Y and h:*C) within :)m and illustrate their usage in Fig. 3b. Fig. 3b also shows that we apply the AVC loss twice to associate both the visual clips (extracted slightly before and after the moment of interaction t) to the corresponding audio.
3.5 Training
The audio-visual representation models /A and fv are trained to minimize both AVC and AStC losses
= L + (1 - a) CastC
(7)
where a a weighting values of a, found that equal weighting produced best results.
between the two terms. While we experimented with different
Implementation details. We follow prior work on audio visual correspondence [43], and use an R(2+1)D video encoder [62] with depth 18 and a 10-layer 2D CNN as the audio encoder. Two video clips are extracted around moments of interaction
frame rate of 16 FPS each with duration of 0.5s, and separated by gap of 0.2s. Video clips are augmented by random resizing, cropping, and horizontal flipping resulting in clips of 8 frames a resolution of 112 x 112, As for the audio, we extract clips of 2s at 44. 1kHz and downsample them to 16kHz. If the audio is stereo, we average the two waveforms to downgrade to mono, and then convert the mono signal to a log mel spectrogram with SO frequency bands and 128 temporal frames. Models are trained with stochastic gradient descent for 100 epochs with batch size of 128 trained over 4 GTX 1080 Ti GPUs, learning rate of 0.005 and momentum of 0.9. For Ego4D, GPUs with a learning rate of 0.05. The two loss terms in Eq. 7 are equally weighted with a = 0.5.
a batch size of 512 trained over RTX 2080 Ti
Experiments
In this section, we demonstrate the benefits of identifying moments of interaction and learning state-aware representations through an audible state-change objective. We also show that, while large scale audiolvisual correspondence (AVC) is beneficial, it is not sufficient to learn state-aware representations required for egocentric tasks. The setup used for our experiments is described in Sec. 1.1. Results and discussion of main takeaways are presented Sec. 4.2.
4.1 Experimental Setup
Datasets. We evaluate two egocentric datasets: SPIC-Kitchens-100 [14] and Ego4D [27]. EPIC- Kitchens- 100 contains 100 hours of activities in the kitchen. Ego4D contains 3670 hours of egocentric video covering daily activities in the home, workplace, social settings, etc. For experiments on Ego4D, we use all videos from the Forecasting and Hand-Object interaction subsets.
Baselines and ablations. We consider various baselines as well as ablated versions of RepLAI. Random represents an untrained (randomly initialized) model. AVID [43] and XDC [3] are two state-of-the-art models pre-trained on 2M audio-visual pairs from AudioSet [24] that only leverage audio-visual correspondence. For the full method RepLAI, we initialize the model weights from AVID before training moments of interaction to minimize both AVC and state change loss, AStC.

---

Method
LAve
LaseC
Mol Sampling
AVC
Topl Ace 1
Top5 Acc 1
Pretraining [43]
Verb
Noun
Verb
Noun
(1) Random
20.38
4.96
64.75
19.83
(2) XDC [3]
24.46
6.75
68.04
22.71
(3) AVID [43]
26.62
9.00
69.79
25.50
(4) RepLAI w/o AVC
29.92
10.46
70.58
29.00
(5) RepLAI w/o AStC
29.29
9.67
73.33
29.54
(6) RepLAI w/o Mol
28.71
8.33
73.17
27.29
(7) RepLAI (scratch)
25.75
8.12
71.25
27.29
(8) RepLAI
31.71
11.25
73.54
30.54
Table 1: Action recognition on EPIC-Kitchens- 100. Topl and top5 accuracy (%) is reported.
T: Higher better.
We also evaluate our method trained without AVID initialization (RepLAI from scratch), trained with only AVC (RepLAI w/o AStC), only state change losses (RepLAI w/o AVC), and trained on random moments in time (RepLAI w/o Mol). Finally, we compare our approach with the fully supervised methods presented in Ego4D [27].
Downstream tasks. After self-supervised pre-training, the models are evaluated on range of egocentric downstream tasks. This is done, as is standard, by appending a task specific decoder to the backbone model, and training the decoder on a small annotated dataset. The tasks are:
100 and Ego4D. Given short video clip, the task to classify the 'verb' and 'noun' of the action taking place. This done using two separate linear classifiers trained for this task. We report the top-1 and top-5 accuracies, following [14] (Tab. 1) and [27] (Tab. 2). We also evaluate on the unseen participants, head classes, and tail classes of EPIC- MECCCKKichhessO00 in Tab. 3. Through this task, we assess the efficacy of the spatial-temporal representations learned by the model differentiating among different verbs and nouns.
Video action recognition (AR) on
Long- ferm action anticipation (LTA) on Ego4D. Given a video, the task is to predict the camera wearer's s future sequence of actions. For this task, the model first presented with 4 consecutive clips of 2s, which are encoded using our visual backbone fv. Following [27], the representations are concatenated and fed to 20 separate linear classification heads to predict the following 20 actions. Performance is measured using the edit distance metric ED @ (Z=20) [27]. With the help of this task, we can evaluate if the representations learned by the model can be employed for long-horizer planning where the actions can change and may be of arbitrary duration. Results are reported in Tab. 2.
State change classification (SICC) on Ego4D. Given video clip, the task is to classify an object undergoes a state change or not. The video clip is encoded by fv and state change classification head is used which performs global average pooling on the entire feature tensor and is followed by a classification layer. Performance measured through the State Change Classification Accuracy (%), and reported in Tab. 2. This task ideal for assessing the ability of the model in understanding the temporal change happening in the state of an object.
4.2 Discussion of results
As can be seen in Tab. I and Tab. 2, RepLAI outperforms all other methods across all downstream tasks. Overall, this can be attributed to its ability focus on interactions, both by detecting when they occur and by learning representations that are sensitive to interactions. A closer analysis of these results reveals several insights that we discuss next.
RepLA AI enhances large-scale AVC driven approaches. Prior work on self-supervised audio-visual learning has shown strong audio-visua representations for action recognition [43, 42]. One question that we seek to answer is, how useful these are representations to egocentric tasks and what are their limitations? To answer this question, we compare our model trained from scratch, RepLAI (Scratch),
'Edit distance measures the minimum number of operations required to convert the predicted sequence of actions to ground truth. To account for multi- multi-modality of future actions, it also allows the model to make 2 = 20 predictions, and only accounts for the best prediction.
7

---

SICC
AR
LTA
Method
Lave
LASEC
Mol
AVC
Acc f
Top1 Ace
(Z@(Z=20)
Pretraining [43]
Verb
Noun
Verb
Noun
(3D-ReNee5000 [9, 27]
NA
NA
NA
NA
68.70
(S2) SlowFast (21,
NA
NA
NA
NA
0.747
0.808
(S3) MVTT [19, 27]
NA
NA
NA
NA
0.707
0.901
(1) Random
51.80
17.4
7.7
0.831
0,936
(2) XDC [3]
58.90
17.90
8.70
0.823
0,928
(3) AVID [43]
61.11
18.3
10.7
0.811
0,919
(4) RepLAI w/o AVC
64.00
20.3
12.4
0.781
0,854
(5) RepLAI w/o ASEC
63.60
21.1
13.5
0.774
0.853
(6) RepLAI w/o Mol
62.90
19.8
11.2
0.792
0.868
(7) RepLAI (scratch)
66.20
22.2
14.1
0.760
0,840
(8) RepLAA
66.30
22.5
14.7
0.755
0.834
Table 2: Performance on several downstream tasks on Ego4D. SICC: State Change Classification (%). AR:
Action Recognition (%). I.TA: Long-term action anticipation. t: Higher is better. J: Lower is better.
Unseen Participants
Tail Classes
Head Classes
Topl Ace f
Top5 Ace
Top1
Acc f
Top5
Acc f
Top1
Acc
Top5 Acc f
Methods
Verb
Noun
Verb
Noun
Verb
Noun
Verb
Noun
Verb
Noun
Verb
Noun
XDC [3]
24.29
6.96
67.79
23.00
15.89
4.17
44.92
9.77
24.78
6,95
72.28
24.74
AVID [43]
26.17
8.67
68.75
24.12
16.80
4.43
47.14
12.89
27.95
9.82
73.20
28.43
RepLAI w/o AVC
28.67
9.38
72.04
27.88
18.49
5.21
47.79
12.63
30.59
10.21
73.33
30.90
RepLAI w/o Mol
27.71
7.92
72.08
26.88
16.80
49.74
12.76
29.36
10.65
76.33
30.41
RepLAI
31.58
10.17
73.46
29.96
20.05
6.12
52.08
16,54
33.41
11.58
77.77
34.33
Table 3: Video action recognition (AR) accuracy (%) on
100 for unseen participants, head
classes, and tail classes. Top1 and top5 accuracy (%) is reported. 1: Higher is better.
with our model using the weights from AVID [43] as initialization for both the visual and audio encoders. We also compare our method to standalone AVID and XDC i.e. without further self- supervised training. Comparing rows (2), (3) and (8) in Tab. and Tab. 2, it is clear that RepLAI enhances large-scale AVC pre-training by significant margins, leading to absolute improvements of 5% in top-1 verb accuracy on WIIC-Kitchens- 100, 4.2% on Ego4D, 5.2% increase in state-change classification accuracy, and 5.6% reduction on the edit distance for long-term anticipation compared to AVID. Comparing rows (7) and (8), we also see that large-scale AVID pre-training enhances the representations learned by RepLAI on EPIC-Kitchens- 100 significantly but only marginally on Ego-4D. This is likely due to the significantly large diversity of scenes in Ego4D. Thus, while relying on large-scale audio- visual pre-training (as with AVID) can help avoid overfitting smaller egocentric datasets, this is less critical when training on larger and more diverse data.
Detecting moments of interaction (Mol) helps representation learning. We hypothesize that to learn good representations for egocentric data of daily activities, self-supervised learning should focus on moments in time when interactions occur. To assess whether our audio-driven Mol detection algorithm helps representation learning, we compare RepLAI with an ablated version, RepLA/ w/o Mol, where the model is trained on audio-visual clips extracted at random from the untrimmed videos. As can be seen by comparing rows (6) and (8) in Tab. and Tab. 2, sampling clips around Mol leads to significantly better representations for all egocentric downstream tasks that we study, Moreover, even though Repl.Al w/o Mol trains with AStC, it is unable to fully leverage the state change objective function without the information of moments of interactions which leads to a worse performance. This suggests that, an explicit state change objective function and sampling video clips around moments of interactions (which are likely to be aligned with the actual state changes) together provide an information-rich feedback our model in better understanding how the state changes by an interaction and how the actions transition over time. These results also clearly show that the proposed Mol detection procedure is able to find moments in time that are especially useful for learning representations of daily activities. We emphasize the simplicity and effectiveness of our audio-driven detector, which shows how informative audio can when searching for moments of interaction. In the future, we believe that learning-basee approaches could further enhance Mol detection, and further improve the learned audio-visual representations. We also show several qualitative examples of detected Mol in the supplement.
8

---

Moment of Interaction:
Moment of Interaction: Grating carrot
RepLAI (ours) I-SNE
Cutting carrot
Audio Spectrogram
Action Timeline
AVID t-SNE
Before Cut Carrot
After Cut Carrot
Before Grate Carrot
After Grate
-20
Carrot
Figure t-SNE visualization of feature representations learned by RepLAI and AVID video consisting of fine- grained actions over time. For simpler visualization, we consider all the vidcos belonging to a single participant. A larger spread in the I-SNE RepLA/ indicates more distinct state-aware representations.
AVC and AStC are complementary. To assess the impact of both terms in Eq. we evaluate RepLA/ trained without Lave and without LASEC. Comparing rows (4), (5) to row (2) and row (3) in Tab. and Tab. 2 shows that each term enhances the representations obtained through large-scale scale audio-visual pre-training (AVID). Furthermore, comparing the ablated models in rows (4) and (5) to the full model row (8) shows that these two terms are complementary each other. This because the AVC and AStC tasks encourage learning of representations with different characteristics. AVC focuses on learning visual representations that are informative of what kind of sounding objects are present in the video, while AStC forces the model to differentiate between visual representations that occur before and after state change interactions.
RepLAI encourages state-aware representation learning. To study the representations learned by our approach for different states, we generate a t-SNE plot [63] for RepLAI and AVID as shown in Fig. 4. For generating a simpler visualization, a small dataset prepared consisting of all the videos corresponding to a single participant, POI, in OPICCKitchens-100 100 and split into clips of 0.5s. We can observe that there is larger spread the t-SNE plot for RepLA/ than AVID. A larger spread indicates that the representations of the various states are significantly different from each other and form more distant clusters as shown by RepLAI. Whereas, the state representations are similar to each other, they are clustered together and show lesser spread as shown by AVID. Mol are the key moments interactions with an object an environment where the state is changing. AVID has no such information about the key moments and also does not have explicit state change objective function. Therefore, is unable to discriminate between the before and after state of an action and has less effective state-aware information in its representations.
RepLAI representation are more generalizable and robust to long-tail. To assess RepLAI in a scenario with domain shift, we evaluate on unseen participants that were fully excluded from pre-training of RepLAI. Tab. 3 shows that RepLAI significantly outperforms baselines and ablations, indicating that representation learning by our model provides much better generalization. Moreover, the verb and noun classes in EPIC-Kitchens 100 exhibit taiget distribution. When further compared on head and tail classes separately in Tab. 3, we can observe that RepLAI outperforms all other methods highlighting its higher robustness a taiget distribution.
Self-supervised vs supervised representation learning Tab. 2 also compares RepLAI to fully supervised methods introduced in Ego4D [27] (rows $1, S2 and S3). We can observe that RepLAI can also perform competitively to the fully supervised approaches when we have access to larger and more diverse data. With further focus on SSL. for untrimmed datasets, SSL methods will be able to match supervised approaches, and our work takes step towards it.

---

Conclusion
In this work, we propose an audio-driven self-supervised method for learning representations of egocentric video of daily activities. We show that in order to learn strong representations for this domain, two important challenges need be addressed. First, learning should focus on moments of interaction (Mol). Since these moments only occur sporadically in untrimmed video data, we show that Mol detection is an important component of representation learning untrimmed datasets. Second, learning should focus on the consequences of interactions, i.e., changes in the state of an environment caused by agents interacting with the world. particular, by seeking to identify visible state changes from the audio alone, can learn representations that are potentially more aware of the state of the environment and hence, particularly useful for egocentric downstream tasks.
Acknowledgements
We would like to thank DARPA MCS, ONR Young Investigator and DARPA SAIL-ON for the funding.
Broader impact
Deep learning models are capable of learning (and sometimes even amplifying) biases existing in datasets. While several steps have been taken datasets like Ego4D increase geographical diversity, we would like to encourage careful consideration of ethical implications when deploying these model. While public datasets are essential to make progress on how to represent visual egocentric data, premature deployment of our models is likely have negative societal impact, as we did not check for the presence or absence of such biases.
References
[1] Y. Abu Farha, A. Richard, and J. Gall. When will you what?-anticipating temporal occur- rences of activities. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 5343-5352. 2018. 3
[2] J.-B. Alayrac, A. Recasens, R. Schneider, R. Arandjelovic, J. Ramapuram, J. De Fauw, L. Smaira, S. Dieleman, and A. Zisserman. Self-supervised multimodal versatile networks. NeurIPS, 2(6):
7, 2020.
[3] H. Alwassel, D. Mahajan, B. Korbar, L. Torresani, B. Ghanem, and D. Tran. Self-supervised learning by cross-modal audio-video clustering. Advances in Neural Information Processing Systems, 33:9758-999 9970, 2020. 1, 3, 6, 7,8
[4] R. Arandjelovic and A. Zisserman. Look, listen and learn. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 609-617, 2017. 2, 3,
[5] R. Arandjelovic and A. Zisserman. Objects that sound. In Proceedings of the European conference on computer vision (ECCV), pages 435-451, 2018. 3
[6] Y. Asano, M. Patrick, C. Rupprecht, and Vedaldi. Labelling unlabelled videos from scratch with multi-modal self-suuperiision Advances in Neural Information Processing Systems, 33: 4660 4671, 2020.
[7] M. Cai, K. Kitani, and Y. Sato. Understanding hand-object manipulation by modeling the contextual relationship between actions, grasp types and object attributes. arXiv preprint arXiv: 187.008807.08254 2018.
[8] M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In Thirty-fourth Conference on Neural Information Processing Systems (NeurIPS), 2020. 2
[9] J. Carreira and A. Zisserman. Quo vadis, action recognition? a new model and the kinet- ies dataset. Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 6299-6308, 2017. 1,8
10

---

[10] T. Chen, S. Korbllith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. Proceedings of the International Conference on Machine Learning. pages 1597-1607. PMLR, 2020. 1,2
[11] S.-W. Chung. J. S. Chung, and H.-G. Kang. Perfect match: Improved cross-modal embeddings for audio-visual synchronisation. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3965-3969. IEEE, 2019. 3
[12] D. Damen, T. Leelasawassuk, O. Haines, A. Calway, and W. W. Mayol-Cuevas. You-do, i-lear:: Discovering task relevant objects and their modes of interaction from multi-user egocentric video. In BMVC, volume 2, page 3, 2014. 3
[13] D. Damen, H. Doughty, G. M. Farinella, S. Fidler, A. Furnari, E. Kazakos, D. Moltisanti, J. Munro, Perrett, W. Price, and M. Wray. Scaling egocentric vision: The epic-kitchens dataset. In European Conference on Computer Vision (ECCV), 2018.
[14] D. Damen, H. Doughty, G. M. Farinella, Furnari, E. Kazakos, Ma, D. Moltisanti, Munro, T. Perrett, W. Price, et al. Rescaling egocentric vision. arXiv preprint arXiv:2006. 13256, 2020. 7,
[15] V. de Sa. Learning classification with unlabeled data. In Advances in Neural Information Processing Systems, pages 1 112-119. Citeseer, 1994. 5,5
[16] C. Doersch, A. Gupta, and A. A. Efros. Unsupervised visual representation learning by context prediction, Proceedings of the IEEE international conference on computer vision, pages 1422-1430, 2015. 2
[17] H. Doughty and C. G. Snoek. How do you do it? fine-grained action understanding with aseudo-adverbs. arXiv preprint arXiv:2203. 12344, 2022. 3
[18] D. Dwibedi, Y. Aytar, J. Tompson, P. Sermanet, and A. Zisserman. Temporal cycle-consistency learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1801-1810, 2019.
[19] H. Fan, B. Xiong. K. Mangalam, Y. Li, Z. Yan, J. Malik, and C. Feichtenhofer. Multiscale vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6824-6835, 2021. 8
[20] A. Fathi, J. K. Hodgins, and J. M. Rehg. Social interactions: A first-person perspective. In 2012 IFEE Conference on Computer Vision and Pattern Recognition, pages 1226-1233. IEEE, 2012.
3
[21] C. Feichtenhofer, H. Fan, J. Malik, and K. He. Slowfast networks for video recognition. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6202-6211, 2019. 8
[22] C. Feichtenhofer, H. Fan, B. Xiong, R. Girshick, and K. He. A large-scale study on unsuper- vised spatiotemporal representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2021. 3
[23] A. Furnari and G. M. Farinella. -unioinnnnnoling Istms for action anticipation from first-person video. IEEE transactions on pattern analysis and machine intelligence, 43(11):402--4036 2020.
[24] Gemmeke, D. P. Ellis, D. Freedman, A. Jansen, W. Lawrence, R. C. Moore, M. Plakal, and M. Ritter. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 776-780. IEEE, 2017. 6
[25] S. Gidaris, P. Singh, and N. Komodakis. Unsupervised representation learning by predicting image rotations. International Conference on Learning Representations, 2018. 2
[26] R. Girdhar and K. Grauman. Anticipative video transformer. Proceedings of the IEEF/CVF International Conference Computer Vision, pages 13505-13515, 2021. 3
11

---

[27] K. Grauman, A. Westbury, E. Byrne, Z. Chavis. A. Furnari, R. Girdhar, J. Hamburger, H. Jiang. M. Liu, X. Liu, al. Ego4d: Around the world in 3,000 hours of egocentric video. In CVPR. 2022. 2, 3, 8,9
[28] J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Pires, Z. Guo, M. Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning. In Advances in Neural Information Processing Systems, 2020. 1,2
[29] T. Han, W. Xie, and A. Zisserman. Self-supervised co-training for video representation learning. Advances in Neural Information Processing Systems, 33:5679-5699 2020.
[30] T. Han, W. Xie, and A. Zisserman. Memory-augmented dense predictive coding for video representation learning. In European conference on computer vision, pages 312-329. Springer, 2020.
[31] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9729-9738, 2020. 1,2
[32] K. Hu, J. Shao, Y. Liu, B. Raj. M. Savvides, and Z. Shen. Contrast and order representations for video self-supervised learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7939-7949, 2021. 3
[33] E. Kazakos, A. Nagrani, A. Zisserman, and D. Damen. Epic-fusion: Audio-visual temporal binding for egocentric action recognition. In Proceedings of the IEEE/CVE International Conference Compuler Vision, pages 5492-5501, 2019. 3
[34] E. Kazakos, J. Huh, A. Nagrani, A. Zisserman, and D. Damen. With little help from my temporal context: Multimodal egocentric action recognition. arXiv preprint arXiv:2111.0100 2021. 3
[35] D. Kim, D. Cho, and S. Kweon. Sule-supe video representation learning with space-time cubic puzzles. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 8545-8552, 2019.
[36] B. Korbar, D. Tran, and L. Torresani. Cooperative learning of audio and video models from self-supervised synchronization. Advances in Neural Information Processing Systems, 31, 2018. 3
[37] K. Lai, Bo, and D. Fox. Unsupervised feature learning for 3d scene labeling. In 2014 IEEE International Conference on Robotics and Automation (ICRA), pages 3050-3057. IEEE, 2014.
3
[38] Y. Li, T. Nagarajan, B. Xiong, and K. Grauman. Ego-exo: Transferring visual representations from third-person to first-person videos. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 6943-6953, 2021. 3
[39] M. Liu, S. Tang, Y. Li, and J. Rehg. Forecasting human object interaction: Joint prediction of motor attention and egocentric activity. 2019.
[40] I. Misra, C. L. Zitnick, and M. Hebert. Shuffle and lear: unsupervised learning using temporal order verification. In Proceedings of the European Conference on Computer Vision, pages 527-544. Springer, 2016. 3
[41] Morgado, Y. Li, and N. Nvasconcelos. Learning representations from audio-visual spatial alignment. Advances in Neural Information Processing Systems, 33, 2020. 3
[42] P. Morgado, I. Misra, and N. Vasconcelos. Robust audio-visual instance discrimination. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1
2021.
[43] P. Morgado, N. Vasconcelos, and I. Misra. Audio-visual instance discrimination with cross- modal agreement. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 12475-12486, 2021. 1, 6,
12

---

[44] J. Munro and D. Damen. Multi-modal domain adaptation for fine the-graindd action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 122-132, 2020. 3
[45] T. Nagarajan, C. Feichtenhofer, and K. Grauman. Grounded human-object interaction hotspots from video. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8688-8697, 2019.
[46] E. Ng. D. Xiang, H. Joo, and K. Grauman. You2me: Inferring body pose in egocentric video via first and second person interactions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9890-99999999 9900,9900,0000 2020.
[47] M. Noroozi and P. Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In Proceedings of the European Conference on Computer Vision, pages 69-84. Springer, 2016.7
[48] A. v. d. Oord. Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: 1807,0370, 2018. 1,2
[49] A. Owens and A. A. Efros. Audio-visual scene analysis with self-supervised multisensory features. In Proceedings of the European Conference on Computer Vision (ECCV). pages 631-648, 2018.
[50] D. Pathak, P. Krahenbuhl, J. Donahue, T. Darrell, and A. A. Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pages 2536-2544, 2016. 2
[51] M. Patrick, Y. M. Asano, P. Kuznetsova, R. Fong, J. F. Henriques, G. Zweig, and A. Vedaldi. Multi-modal self--uupevision from generalized data transformations. arXiv preprint arXiv:2003.04290 042998 2020. 3
[52] A. Piergiovanni, A. Angelova, and M. S. Ryoo. Evolving losses for unsupervised video representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 133-142, 2020. 3
[53] H. Pirsiavash and D. Ramanan. Detecting activities of daily living first-person camera views. In 2012 IEEE conference on computer vision and pattern recognition, pages 2847-2854, IEEE, 2012.
[54] S. Purushwalkam, Ye, S. Gupta, and A. Gupta. Aligning videos in space and time. arXiv preprint arXiv:2007.045155 2020. 1
[55] R. Qian, T. Meng. B. Gong, M.-H. Yang. H. Wang, S. Belongie, and Y. Cui. Spatiotemporal contrastive video representation learning. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 6964-6974, 2021. 3
[56] M. Ramazanova, V. Escorcia, F. C. Heilbron, C. Zhao, and B. Ghanem. Owl (observe, watch, listen): Localizing actions in egocentric video via audiovisual temporal context. arXiv preprint a2Oiv:2202.0 04947, 2022. 3
[57] A. Recasens, P. Luc, J.-B. Alayrac, L. Wang, F. Strub, C. Tallec, M. Malinowski, V. Päträucean, F. Altché, M. Valko, et al. Broaden your views for self-supervised video learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1255-1265, 2021. 1.3
[58] B. Singh, T. K. Marks, M. Jones, O. Tuzel, and M. Shao. multi-stream bi-directional recurrent neural network for fine-grained action detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1961-1970, 2016.
[59] K. Singh, K. Fatahalian, and A. A. Efros. Krishnacam: Using a longitudinal, single- person, egocentric dataset for scene understanding tasks. In 20/6 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1-9. IEEE, 2016.
[60] Y.-C Su and K. Grauman. Detecting engagement in egocentric video. In European Conference on Computer Vision, pages 454-471. Springer, 2016.
13

---

[61] Y. Tian, D. Krishnan, and P. Isola. Contrastive multiview coding. In Proceedings of the European Conference on Computer Vision, pages 776-794. Springer. 2020.
[62] D. Tran, H. Wang. L. Torresani, J. Ray, Y. LeCun, and M. Paluri. A closer look at spatiotemporal convolutions for action recognition. In Proceedings of the IEEE conference Computer Vision and Pattern Recognition, pages 6450-6459, 2018. 6
[63] L. van der Maaten and G. Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 2008. 9
[64] C. Vondrick, H. Pirsiavash, and Torralba. Anticipating visual representations from unlabeled video. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 98-106, 2016. 3
[65] Wang, J. Jiao, L.. Bao, S. He, Y. Liu, and Liu. Self-supervised spatio-temporal repreessett tion learning for videos by predicting motion and appearance statistics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4006-4015, 2019.
[66] J. Wang. J. Jiao, and Y.-H. Liu. supersipe video representation learning by pace prediction. In Proceedings of the European Conference Computer Vision, pages 504-521 Springer, 2020.
[67] X. Wang and A. Gupta. Unsupervised learning of visual representations using videos. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 2794-2802, 2015.
[68] X. Wang, A. Jabri, and A. A. Efros. Learning correspondence from the cycle- conesseonssstency of time. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 2566-2576, 2019.
[69] D. Wei, J. Lim, A. Zisserman, and W. T. Freeman. Learning and using the arrow of time. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018. 3
[70] D. Xu, J. Xiao, Z. Zhao, J. Shao, D. Xie, and Y. Zhuang. Self-supervised spatiotemporal learning via video clip order prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10334-10343, 2019.
[71] Z. Zeng, D. McDfff, Y. Song, et al. Contrastive learning of global and local video representations. Advances in Neural Information Processing Systems, 34:7025-7040, 2021. 3
[72] C. Zhang, A. Gupta, and A. Zisserman. Temporal query networks for fine-grained video understanding. In Proceedings of the IEEE/VVÉ Conference on Computer Vision and Puttern Recognition, pages 148664496 2021. 3
14

---

Checklist
Please do modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] We discuss the limitations after the conclusions section of the main paper along with the potential avenues for future research inspired by the limitations of our method.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] Refer to the section after the conclusions
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material as URL)? [Yes]] The code and instructions are available publicly.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Refer Section 3.5: Implementation Details
(c) Did you report error bars (e.g.. with respect to the random seed after running exper- iments multiple times)? [No] Since the train and test set are too large, it takes lot of time and resource to run the experiments multiple times. However, errors bars are expected to be small, given their sizes.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Refer to Section 3.5: Implementation Details
4. If you using existing assets (e.g., code, data, models) curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
15


Reviews:
[{"review_id": "znwbZoJKoWr", "reviewer": "Reviewer_coBj", "summary": "In this paper authors introduce a self-supervised learning method from audio-visual data. Specifically, the method revolves around using moments of interaction to learning meaningful audio-visual relation. Authors split the learning in two losses: an standard audio-visual loss, and a loss around the audible change of state and its relation to the visual change of state. Authors evaluate their proposed method in EpicKitchens and Ego4D. \n", "questions": "My concerns are listed in Weaknesses. My main questions to the authors would be:\n\n- Have the authors considered adding additional baselines when comparing to the state-of-the-art? If not, why not?\n\n- Which evidence do the authors have that the detection of change works properly if they don't have any evaluation for that signal.\n\n- In general, could the authors show some more examples of moments of interaction (similar to the ones in supplementary) in the main paper. I think it would be specially interesting to see examples where the detection of those moments works and other where it fails.", "limitations": "Authors discuss the limitations of the method in 4.2. I would suggest expanding that a bit with general negative societal impact that working with ego video can have. I believe egocentric video can be used to track everyday's life which could have some negative impact. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n\n- S1. Audio-visual self-supervised learning is a powerful tool to learn representations. However, typical approaches do not exploit the semantic importance of the moments where to sample the audio. This work is a step towards using the audio content in a most meaningful way.\n\n- S2. The results show how the method improves over the baseline with standard audio-visual learning. \n\n- S3. Authors evaluate in two very relevant well-known ego centric benchmarks. I believe that is important as it makes the paper stronger.\n\n- S4. Ego-centric data is going to be becoming more important in the near future. With the progress of recording devices and embodied research, works along the lines of this paper will become more relevant.\n\n- S5. I specially like the idea of detecting state changes through audio. According to the paper the procedure is quite simple and the self-supervised method certainly benefits from seeing samples around that moment.\n\n**Weaknesses**\n\n- W1. Authors compare with a single baselines for audio-visual learning. I think other works such as MMV (Alayrac, NeuriPS 2020),  Evolving Losses (Piergiovanni, CVPR 2020), Brave (Recasens, ICCV 2021), XDC (Alwassel, NeurIPS 2020) are very relevant in the community. I understand the topic is slightly different and authors cannot retrain with all the baselines, but I still believe that having a single baseline such as AVID is insufficient for a publication. \n\n- W2. The authors are missing citations for some of the papers mentioned in W1 (e.g. MMV, Evolving Losses). I think those works are important in the space of self-supervised audio-visual learning.\n\n- W3. I think authors do not properly evaluate the ability of their model to detect moments of interaction. The description of the method is very complete and in Table 1 they ablate using the method, but I think it would be good to somehow evaluate whether the proposed method works.\n\n- W4. I am missing some examples of moments of interaction I understand the reasoning behind the methodology of using the reverse clip as negative, but I believe the readers would benefit from a few visual examples to understand that.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "faC_Uiks94G", "reviewer": "Reviewer_WY3G", "summary": "This paper proposes leveraging audible state changes for self-supervised representation learning from long-form egocentric videos. Authors initially detect the timestamps where interesting interactions occur (MoI) and then compute a cross modal contrastive objective where the natural transition (before, interaction, after) is more likely than one which is temporally reverse (after, interaction, before). ", "questions": "- How do you handle the audio diversity of semantically similar interactions? For example, “cutting cucumber” on a wooden board will sound different from doing it on a plastic board, or sound of “sautéing” mushrooms/onions in a pan is meaningfully influenced by the oven/pan/oil temperature.\n\n- As part of ablation studies, have you tried a linear head for $h_{\\Delta V}^{AStC}$ since forward delta seems to be the negative of the backward one.\n\n- AVC loss shown in Figure 3.b encourages the embedding of $V_{t-\\delta}$ to be close to $V_{t+\\delta}$ by both anchoring on audio embedding at t while the AStC encourages them to be different. These two objective functions, to the best of my understanding are pulling in opposite directions! Would not it make more sense to choose the audio at $t-\\delta$ and $t+\\delta$, instead of t, when computing two AVC losses?\n\n- In Table 1, Top 5 Acc: AVC seems to be doing most of the job while w/o AStC and MoI, performance on “verb” is almost maintained (~73%) but the pattern is different for “noun”. Any insights?\n", "limitations": "Authors have addressed the limitations", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Paper is clear in presentation and has provided an interesting view to self-supervised multi-modal representation learning in egocentric videos with audible state change. The idea is pretty close to \"Actions ∼ Transformations\" <https://arxiv.org/pdf/1512.00795.pdf> which unfortunately is missing from the references. Below are my detailed comments.\n\n- Line 48-49: Authors argue that AVC objective leads to representations that are not informative of the changes over time. If indeed the sound of “opening fridge” is different from that of “closing fridge”, AVC will encourage video representation of “opening fridge” to be more similar to the audio representation of “opening fridge” than “closing fridge” since audio embedding of the latter will be served as a negative instance within the contrastive framework. Note that, you do not have to sample after the fact e.g once the fridge is closed/opened, instead you can sample during the action e.g while fridge is being opened/closed, something that working with video in a cross-modal contrastive setup easily allows you to do. With that said, I would like to see authors clarifying their point on the lack of AVC’s suitability for the task specially given the ablation studies that show dropping AVC vs AStC is not that much different.\n\n- On finding MoI: As paper mentions, egocentric videos are naturally long-form, as they continuously capture daily activities. Hence, a person moving in an environment i.e change in location, even without any interaction can result in changes in audio spectrogram. For example, a person is cooking some food in the kitchen, then walks to the living room to pick up a book. The fact that there is a fan or stereo playing in the living room, will result in a change in audio (note that the sound of fan or stereo is not necessarily audible in the kitchen), however those are not MoI since there has been no interaction with either stereo or fan yet from Sec 3.2 it seems to me that the proposed MoI approach should pick those. In nutshell, change in audio is not only as a result of human-object interaction, it can be due to change of location or variant environment as well and I cannot see how the proposed MoI detection method can work in a realistic environment.\n\n- I am not convinced that learning from audible state change as described in Sec 3.3 is generic enough. For example, the visual state is different after hearing the sound of MoI for “opening fridge” , while before state shows a closed fridge, the after state depicts an open one. Also, due to “distinct” sound of “opening fridge” the backward transition should be less likely. Now consider the example of “cutting cucumber”, the backward transition almost never happens (humans usually don’t stitch cucumber slices together!), while it is reasonable to go from a closed fridge to an open one and vice versa. There are also cases which despite audible MoI, visuals look almost identical in before and after like “stirring a pot”. I suspect that more clear performance gains seen on Ego4D versus Epic-Kitchens is partly related to the state change properties which are more prominent in the former dataset. I would like to hear authors feedback on the different aforementioned types of interactions and why their proposed model should work in a self-supervised setup where we don’t know which type of these interactions are included in the training data.\n\n- Line 269: I do not think it is fair to compare rows 2 and 7 since row 2 has been only trained on AudioSet which is very different from evaluation egocentric datasets. To see the true additional value of your contributions (use of MoI and AStC loss), AVID should be fine-tuned on the egocentric datasets (equally RepLAI w/o AStC and MoI)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "O6KIDaZsuRb", "reviewer": "Reviewer_wUoy", "summary": "This paper proposes a new way for self-supervised learning with egocentric videos to learn from audible interactions. Specifically, it uses audio signals to identify the state changes and use these transition states to learn the audio-visual correlations. The proposed method performs much better than a recent method AVID [42] and the ablation study shows the effectiveness of each model component. ", "questions": "My main question is the generalization ability of the proposed method, i.e., increasing the probability of associating the audio with the visual state change in the forward direction but decreasing the probability in the backward direction. It can work well for activities like opening/closing the door intuitively. However, how can it work for others like stirring and washing? ", "limitations": "The limitations and potential negative impact are discussed in the supplementary. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n- The idea of using audio to identify the state changes for learning audio-visual correlation is novel and seems effective. \n- The paper is well written and easy to read\n- The performance of this method is good. \n\nWeaknesses\n\n- The example used for illustrating Eq. 1 are not convincing. The given example is the audio of closing the door should be (1) similar to the visual transition opened door → closed door and (2) dissimilar to the (backwards) transition closed → open. This activity has strong temporal order. But how to deal with activities like stirring and washing? \n- There is lack of the fully-supervised performance for comparison in experiments. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept