Egocentric Video-Language Pretraining
Authors: Kevin Qinghong Lin
Jinpeng Wang
Mattia Soldan
Michael Wray
Rui Yan
Zhongcong Xu
Difei Gao
Rong-Cheng Tu
Wenzhe Zhao
Weijie Kong
Chengfei Cai
WANG HongFa
Dima Damen
Bernard Ghanem
Wei Liu
Mike Zheng Shou
Venue: NeurIPS 2022 Accept (2022)
Keywords: Video-Language Pretraining
Egocentric Video Datasets

Abstract: Video-Language Pretraining (VLP), which aims to learn transferable representation to advance a wide range of video-text downstream tasks, has recently received increasing attention. Best performing works rely on large-scale, 3rd-person video-text datasets, such as HowTo100M. In this work, we exploit the recently released Ego4D dataset to pioneer Egocentric VLP along three directions. (i) We create EgoClip, a 1st-person video-text pretraining dataset comprising 3.8M clip-text pairs well-chosen from Ego4D, covering a large variety of human daily activities. (ii) We propose a novel pretraining objective, dubbed EgoNCE, which adapts video-text contrastive learning to the egocentric domain by mining egocentric-aware positive and negative samples. (iii) We introduce EgoMCQ, a development benchmark that is close to EgoClip and hence can support effective validation and fast exploration of our design decisions in EgoClip and EgoNCE. Furthermore, we demonstrate strong performance on five egocentric downstream tasks across three datasets: video-text retrieval on EPIC-KITCHENS-100; action recognition on Charades-Ego; natural language query, moment query, and object state change classification on Ego4D challenge benchmarks. The dataset and code are available at https://github.com/showlab/EgoVLP.

Egocentric Video-Language Pretraining
Kevin Qinghong Lin', Alex Jinpeng Wang', Mattia Soldan', Michael Wray?, Rui Yan', Eric Zhongeong Xu', Difei Gao', Rongeheng Tu', Wenzhe Zhao', Weijie Kong", Chengfei Cai², Hongfa Wang", Dima Damen', Bernard Ghanem,, Wei Liu', and Mike Zheng Shou 1s
Show Lab, National University of Singapore University of Bristol King Abdullah University of Science and Technology *Tencent Data Platform
Abstract
Video-Language Pretraining (VLP), which aims to learn transferable representation advance a wide range of video-text downstream tasks, has recently received increasing attention. Best performing works rely on large-scale, 3rd-person video- text datasets, such as HowTo In this work, we exploit the recently released Ego4D dataset to pioneer Egocentric VLP along three directions. (i) We create EgoClip, Ist-person video-text pretraining dataset comprising 3.8M clip-text pairs well-chosen from Ego4D, covering a large variety of human daily activities. (ii) We propose a novel pretraining objective, dubbed EgoNCE, which adapts vidco-text contrastive learning to the egocentric domain by mining egocentric- aware positive and negative samples. (iii) We introduce EgoMCQ. a development benchmark that is close to EgoClip and hence support effective validation and fast exploration of our design decisions EgoClip and EgoNCE. Furthermore, five egocentric downstream tasks across
demonstrate strong performance three datasets: video-text retrieval on EPIC-KITCHENS- Charades-Fgoo natural language query, moment query, and object state change classification on Ego4D challenge benchmarks. The dataset and code are available at https: / thub. com/ http////oobbb//.......
action recognition
1 Introduction
With the recent interest boom in computer vision and natural language processing, Video-Language Pretraining (VLP) has prevailed, which aims learn strong and transferable video-language rep- resentation for powering broad spectrum of video-text downstream tasks, such video-text retrieval [1. 2, 31. video question answering [4, 5, 6], and video captioning [7, 8, 9]. The success of VLP mainly stems from the availability of scale open-world video-text datasets [10], which sub- sume large number of videos sourced from the Web (e.g., YouTube) and pair videos with associated textual information. For instance, HowTo companied by noisy narrations yielded from Automatic Speech Recognition (ASR). WebVid-2Ne [3] scrapes 2.5M descriptive videos with well-formed long captions.
[10] collects 134K hours of instructional videos ac-
Despite reaching an impressive data scale, videos those existing video-text pretraining datasets are often of 3rd-person views and may have been edited before posting on the Web. Yet, there is noticeable domain gap between the existing video-text pretraining datasets and 1st-person view videos such as those videos captured by wearable cameras smart glasses. Egocentric video has received increasing interests from the academia (e.g., activity recognition [11], activity anticipation [12], and video summarization [13]) and industry (various applications in robotics and augmented reality).
E:: Corresponding Author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Dataset
Ego?
Domain
Dur (hrs)
# Clips
# Texts
Example
MSR-VTT [1]
x
diverse
40
10K
200K
YouCook2 [16]
X
cooking
176
14K
14K
NctivityNet Captions [7]
X
action
849
100K
100K
WebVid-2M [3]
X
diverse
13K
2.5M
2.5M
HowTo100M [10]
X
instructional
134K
136M
136M
3rd- red-person view
Charades-Ego [17]
home
34
30K
30K
UT-Ego [18]
diverse
37
11K
11K
Disneyworld [19]
disneyland
42
15K
15K
EPIC-KITCHENS-10 [20]
kitchen
100
90K
90K
EgoClip
diverse
2.9K
3.8M
3.8M
1st-person view
Table 1: Comparison of our proposed EgoClip pretraining dataset against the mainstream video- language datasets (top) and egocentric datasets (bottom).
However, due to such domain gap, directly transferring the existing VLP models egocentric downstream tasks cannot fully unleash the potential of large-scale pretraining approaches, which we have confirmed in the later experimental section. To bridge this gap, are motivated to develop Egocentric VLP models, which can greatly benefit various egocentric video downstream applications.
However, existing egocentric video datasets are of small scale and domain-specinie making Egocentric VLP prohibitive. As illustrated in Tab. the formerly largest egocentric video dataset EPIC- KITCHENS-100 [14] focuses on kitchens scenarios and its size is far smaller than those of the 3rd-person pretraining sets WebVid-2M [3] and HowTo 10M [10]. Fortunately, with the recent introduction of the massive-scale egocentric video dataset Ego4D [15], it becomes possible to unlock Egocentric VLP. Ego4D consists of 3, 670 hours of videos with manually annotated narrations from 74 worldwide locations, covering a large variety of daily-life scenarios and activities.
In this work, roused by the favorable scale and diversity of Ego4D, we make a significant effort to pave the way for Egocentric VLP with the following steps:
(i) To address the aforementioned issue of lacking a suitable large-scale egocentric video-language pretraining dataset, we create a video-text pretraining dataset EgoClip which contains total of 3.8M clean Ist-person clip-text pairs selected from Ego4D and covers diverse human daily activities. (ii) To make full use of EgoClip for video-text representation learning, we propose novel video-text contrastive objective EgoNCE to address unique challenges in egocentric pretraining datasets, (iii) We create development benchmark i.e., Egocentric Multiple-Choices -happlee-Choiieeeeuetton.nn dubbed EgoMCQ, which contains 39K questions created from Ego4D and focuses on evaluating video-text alignment. In contrast to other downstream benchmarks, EgoMCQ has less discrepancy from EgoClip. powering us to accurately validate and quickly iterate our designs of EgoClip and EgoNCE. (iv) We conduct extensive experiments to demonstrate the superiority of Egocentric VLP by trans- ferring our pretrained representation to five egocentric downstream benchmarks and achieving state-of-the-art performance: 59.4% nDCG on video-text retrieval of EPIC-KITCHENS. 100 [14] 32.1% mAP on action recognition of Charades-Ego [17], and significant boosts over three Ego4D challenges? 2; natural language query, moment query and object state change classification.
2 Related Work
Video-Language Pretraining, The introduction of large-scale video-text datasets [10, 3] has enabled the emergence of VLP approaches to improve the video-text representation for various vision- language tasks [21, 22, 4], such as MIL-NCE which [23] proposes to match clips with multiple captions close in temporal to adapt the video-text misalignment of HowTo100M [10]. Dominant VLP methods can be classified into two groups, namely: joint- and dual- encoders The former combines videos and texts as single input to the encoder that performs the multimodal fusion. For instance, [24, 25] concatenate videos and texts together before feeding them to unified transformer. Conversely, methods like [3, 26] exploit dual encoders to independently project the video and text inputs into common space and minimize the distance between the paired representations. These approaches are preferred in retrieval settings as they allow for efficient indexing of a single
Fgnccettric VI.P won championship on Instanee Retrieval, EPIC-Kitchens Challenges @ CVPR 2022. 'Egocentric VLP won championship OSCC and 2nd place on NLQ, Ego4D Challenges @ CVPR 2022.

---

EgoClip
Feedback of EgoClip design
EgoMCQ
Q: C
gets food the dog from pocket
Pretrain
EgoNCE+
Evaluate
X
Fealback of
Video clip
Narrution
Video Enc
Text Enc
Model design
X
X
(a) Pretraining set
(c) Development set
Clip
Text
(b) VLP model
(d) Egocentric downstream tasks
Video-text retrieval EPIC-KITCHENS- 100
Action recognition Charades-Ego
Natural language query Ego4D Challenge
(a) the pretraining set EgoClip. (b) the VLP model, and (c) the development set EgoMCQ. We use EgoClip to pretrain a VLP model with the EgoNCE loss and then evaluate EgoMCQ. According to the feedback, we iteratively refine designs of (a) and (b). We then transfer the pretrained model to downstream tasks relevant to the egocentric domain.
Figure 1: Our Egocentric VLP includes:
modality [27, 28]. For example, Frozen [3] employs two separate transformers to encode video and text features and aligns them by video-text InfoNCE [29]. In our work, we adopt the Frozen [3] but extend its InfoNCE to EgoNCE via positive and negative sampling for egocentric-friende pretraining.
Egocentric Video Datasets. Egocentric videos, collected by participants using wearable cameras, offer natural perspective of people's daily activities and raise a range of challenging research topics [11, 12, 30]. Several egocentric video datasets have been developed in decades, e.g., [20, 17, 31j. However, since the collection of egocentric videos is expensive, previous egocentric datasets tend to be small-scale and domain-specific. These limitations hinder Ist-person view research and fail to match the progress of 3rd-person counterparts, such VIP [23, 24, 3]. Recently, massive egocentric video dataset Ego4D [15] has been released, which consists of 3, 670 hours of videos collected by 931 people from 74 worldwide locations in 9 different countries, where most videos are accompanied by narrations, audio, 3D meshes, and more. Furthermore, Ego4D introduces a suite of new challenging benchmarks (e.g., Natural language query and moment query) to fully explore the Ist-person visual experience. With this step-changing dataset and benchmarks, Ego4D would lead to a new research surge on egocentric visual perception.
EgoClip: An Egocentric Video-Language Pretraining Dataset
Data curation. For our EgoClip dataset, we source data from Ego4D [15], which contains 9, 645 untrimmed videos of varying lengths from 5 see to 7 hrs. From these videos, most are associated with dense timestanp-level narrations assigned by two different annotators, describing the camera wearer's activities and interactions with objects. For example, the narration "#C puts the scrapper down." corresponds to video content that occurred at 3. 70s, where "#C" refers to the camera- wearer. Notably, narrations in Ego4D are well-aligned with the videos, both temporally and visually. Prior pretraining datasets are characterized by much greater level of temporal misalignment between the video and text (e.g., HowTo100M [10] narrations are scraped from ASR, yielding sentences misaligned or even unrelated to video content). We first filter Ego4D videos with missing narrations (7.4% of the total video duration) and exclude videos that belong to the validation and test sets of the Ego4D benchmark challenge [15] (a further 23.9% of the total video duration). Next, we retain textual annotation from both narrators in EgoClip, allowing us to consider narration diversity when pairing vidco and text for pretraining purposes. Finally, adopt several criteria to filter the video and textual narrations, further reducing noise (detailed steps are provided in Supplementary B. Overall, this procedure yields 2.9K hours of videos with 3.85 million narrations which cover 2927 hours of video from 129 different scenarios. EgoClip has 21.9 clips per minute with an average clip length of 1.0 seconds and standard deviation of 0.9 seconds (the longest clip is up to 60s). Additional analyses are included in the Supplementary B.3.
Creation of clip-text pairs. Clip-text pairs are the common data format for VLP, but are usually not present in untrimmed video datasets with only a weak matching between narrations captions and vidcos. This was first discussed in How To 100M [10], which pairs subtitles to video clips with
3

---

corresponding time intervals to produce noisy pairs. This is not suitable for Ego4D since each narration is annotated with single timestamp rather than an interval. Thus, we design contextual variable-length clip pairing strategy. Formally, narrations per video in Ego4D are organized as a sequence of sentences (To.... ,Tn] with exact timestamps (lo.... ,nn), indicating an event described by T, happened in the moment 44- For narration Ti with timestamp 4, we pair a clip V; with following start and end timepoints:
[1ttart [ = -1:-/22,, 4 4 B./2al;
(I)
which represents window centered around the timestamp i; with temporal duration equal to 3:/a. 3, is an adjustable parameter equal to the average temporal distance between pairs of consecutive narrations, i.e., (15+1 - f2)/n. We compute f3 on per video basis. Conversely, a is a scale factor computed as the average of all 3; across all videos in the EgoClip (a = 4.9 seconds). Intuitively. Eq. 1 is derived from three observations: (i) Centering helps involve prior information about the event i; (ii) 3 measures the clip duration according to its scenario, such as longer clips watching television (352.9 seconds) v.s. shorter clips harvesting crops (0.9 seconds); (iii) controls the context granularity of clips (e.g., large a pays more attention to rapid, atomic actions). We ablate these design choices in experimental section.
4 Video-Language Pretraining Model
To efficiently transfer video-language representation to egocentric downstream tasks (e.g.. video-text retrieval on model architecture. In particular, we emphasize devising general pretraining objective EgoNCE to adapt the existing VLP model to the egocentric domain (e.g., EgoClip).
[20]). We prefer the dual-encoder (discussed in Sec. 2) VLP
4.1 Architecture: Dual-encoder Pipeline
We choose Frozen [3] as our pretraining architecture. Frozen [3] design encompasses an elegant and simple dual encoder strategy (one per modality) which has favorable characteristics (e.g., indexability and efficiency [27, 28]). Note that this allows us to use our pretrained network in single-modality tasks (e.g., video-only tasks). In practice, the video encoder adopts the TimeSformer [32] architecture, while the text encoder builds upon DistilIBETT [33]. However, our approach is not limited to the encoder's 's design (e.g., the video backbone can be replaced by SlowFast [34] or Video Swin [35]). In the rest of the paper we adopt this notation: (V,, T:) represents the video-text input to the model,
while v, and t; are used to identify the video and text embeddings.
4.2 EgoNCE: An Egocentric-friendg Pretraining Objective
common pretraining objective for the dual-encoder VLP is InfoNCE [29], where the matching
visual-text pairs in the batch are treated as positives while all other pairwise combinations in the
batch are regarded as negatives. Formally, within a batch B ... .V,, InfoNCE is computed by the sum of the video-to-tex loss L.z and text-to-video loss Lev. For simplicity, we only formulate L.2,, whereas Lizy is defined in a symmetric way:
1
cxp(v. t,/T) log iEB
Cuzs
(2)
B
EjeB exp( v\ tz/7)²
where the i-th video embedding v, and j-th text embedding tj L2 normalized features, and is temperature factor.
However, this simple objective performs not well on large-scale video-text datasets like HowTo [10] due to the serious misalignment between the two modalities of data. There- fore, [36] proposes MIL-NCE which treats temporal nearest captions as positive samples.
In this work, our Ist-person human daily activity dataset, i.e. EgoClip, presents two unique challenges compared to the existing 3rd-person view video-text datasets: Challenge (i): The same action often occurs different scenarios (e.g., "unlock the phone" could happen when "lying bed" or "walking outdoors"). Challenge (ii): Often, different actions appearing in the same scenario tend to have indistinguishable visual differences (e.g., when "working in front of the laptop", "typing on the keyboard" or "moving the mouse" have similar feature representations).

---

Evaluation on the text-video retrieval task is unreliable due to duplications Retrieval result: Top clips are not GT but shall be considered as correct.
Text query:
&C C. closes the refrigerator.
Topl: ::
Top2:
Top3:
TopN (GT)
CCC clesen the refrigerator
#C clowes the lower
findgr
EgoMCQ
Inter-video
Intra-video
Text query
#C picks the silicone sealant
&C C carries paint bucket down the ladder
Select the correct video clip from candidates
(b)
(c)
(d)
(c)
(R)
(c)
(d)
(c)
yC places the cumping
CCCllk the power dril with both londs.
Cpicss the allicone soalett
BC cass
ac Cholds
Cturzs
MC C
5C dimos
CC.CCarii
store
the green bean into
paintrrsh with both
painthuuhh in paintlessh his left hand
painbruali
pont bucket down the ladder
Answer with
seat down
right
pain: bucket
hands
X
X
X
Design of the Egocentric VLP development set. Top: An illustration of why the task of text-video retrieval is not suitable; Bottom: Two settings of EgoMCQ. Left-bottom: The "inter- video" setting, each question contains 5 clips from different videos. Right-bottom: The "intra-video" setting, each question contains contiguous clips from the same video, making it more challenging.
Figure 2:
To overcome these two unique challenges, we propose a novel EgoNCE training objective which takes into account two simple yet efficient sampling strategies based on the vanilla InfoNCE.
Action-aware Positive Sampling, In this work, we make a reasonable assumption that the critical verbs and objects mentioned in the
elements in linking visual actions textual narrations narrations (e.g., "drinking coffee" and "opening fridge"). Following this assumption, we can devise clever method to address challenge (i). Specifically, for each narration, identify its nouns and verbs and merge synonym words based on the Ego4D taxonomy dictionary [15], a thesaurus recording meaningful nouns/verbs in Ego4D narrations. Then, batch samples that shared at least noun and at least one verb are treated as positive samples. At last, for the sample i, we define its positive samples within batch B as P. {j E B noun(j) n noun(i) + Ø, verb(j) verb(i) * Ø}.
Scene-aware Negative Sampling. To address challenge (ii), we consider different actions in the same scenario as hard negative samples. Specifically, for each video clip i, sample adjacent clip i' E (i), which is close to in time within the same video. We augment the original batch B with such hard negative samples and each sample in B has its negative counterparts i'. Hence the batch is updated B (1.2....NV. 2'....... N')
N(B)
With these two sampling strategies, our new pretraining objective EgoNCE can be formulated as:
ΣKEp, exp(v/tx/r) (exp(v) t//T) + explv/ty
ego
log
Σ
iEB
Here the item in purple corresponds our proposed action-aware positive samples and blue cor- responds to our proposed scene-aware negative samples. EgoNCE provides general extension to adapt the existing VLP models for video-text pretraining datasets the egocentric domain.
EgoMCQ: A Benchmark for Egocentric VLP Development
The need for development benchmark. We find that most egocentric benchmarks are domain- specific and focus single-modality tasks (see Tab. 1). However, our purpose is to exploit Ego4D's diversity to learn rich video-text representations. Hence, to validate our design choices of the pretraining dataset (e.g., EgoClip), and model (e.g.. FgoNCE). it is essential to measure performance on benchmark highly aligned with the pretraining task. Therefore, we propose FgoMCQ, new egocentric benchmark for reliable and fast developments of Egocentric VLP.
5

---

Data source, We start from the Ego4D data excluded from constructing the EgoClip, which mainly covers the validation set of the Ego4D challenge benchmarks. Additionally, to assure that the scene is not visible during pretraining, we manually remove videos that share multiple views with the videos in EgoClip. To ensure diversity, we randomly select one annotator's ' narration for each video. We follow the same clip pairing strategy as Eq. to be consistent with the data format of EgoClip.
Benchmarking task design. To determine the task for development, we first consider video-text retrieval since it highly aligns with the VLP pretraining objective. However, as depicted in the top half of Fig. 2, for an action (e.g.. close the refrigerator), there are substantial duplicates or semantically similar captions in Ego4D. This can cause issues in retrieval evaluation [37] making model training unreliable. A straightforward approach to prevent this is deduplication (dedup), but it challenging to devise a dedup criterion and perform well in the retrieval settings of a "one-to-whor validation set", Therefore, we select the Multiple- Hultiple-hooic Questions (MCQ) task for development since repetitions are highly unlikely given a small number of answers.
Grouping strategies. To set up the MCQ task, a naive construction randomly groups five video clips to form options for a question. But we find randomly grouping is not challenging since options are highly likely to come from different videos and vary widely in content. We redefine this basic setting as "inter-video" and ensure that the five clips originate from different videos, aiming to distinguish instances from different scenarios (the left-bottom of Fig. 2). Furthermore, we propose more challenging setting "intra-video" by grouping five continuous clips together.This This setting is regarded as a specific form of video-text localization focused on fine- anneggraied context clues, such as hand interaction (the right-bottom of Fig. 2). Dedup is performed within five options for each question for reliable assessment (see Supp. C.I) and we adopt accuracy as the EgoMCQ metric.
Statistics. We finalize 39K questions covering 198K narrations with 468 hours of video, where the "inter-video" has 24K questions covering 290.3 hours of videos. And the "intra-video" has 15K questions and covers 178.3 hours of videos. The average duration among the five options is 34.2 seconds (More statisties of EgoMCQ are shown in Supplementary C.3).
Experiments
We assess our Egocentric VLP along two directions: (i) We conduct an extensive analysis to explore key components of Egocentric VLP (e.g., EgoClip. EgoNCE, and EgoMCQ); (ii) we transfer our pretrained model to various downstream tasks to validate the quality of our video-text representation.
6.1 Benchmarks and Settings
We evaluate our VLP model on five egocentric benchmarks, spanning video-text tasks and pure video tasks, across three different datasets. We briefly describe each task below.
Multi-Instance Retrieval of EPIC-KITCHENS-100. This task is modelled as a video-text retrieval which considers the semantic overlap between different videos narrations, where multiple videos may correspond to the same narration. The training set contains 67.2K clips and validation set contains 9. 9.7K clips. The evaluation metrics are mean Average Precision (mAP) and the normalized Discounted Cumulative Gain (nDCG).
Natural Language Query of Ego4D Challenges. The Natural Language Query task is modelled as a natural language grounding problem [38, 39, 40]. Given a language query and a video, the task aims at localizing the temporal interval within the video, in which the answer is deducible. The training set contains 11.3K queries annotated from 1K clips for this task, while the validation contains 3.9K queries collected from 0.3K clips. The evaluation metric is Recall @ K for IoU-0 (R @ (o)o==0) [38] where 0 is threshold. We evaluate for K KE(1,5) 5) and (0,0.3, 0.5).
Action Recognition of Charades-Ego. This dataset has 64K instances, spanning 1st-person and 3rd-person views and covering 157 activity categories for training. We train and evaluate only on the 1st-person videos. The validation set contains 847 videos for classification and each video belongs to multiple classes. The evaluation metric is mAP.
Moment Query of Ego4D Challenges. The Moment Query task is a video-only task modelled as Temporal Action Localization [11]. Given a particular high-level activity category, the task solution consists of retrieving all the possible temporal windows where the activity occurs. The training set

---

Clip's length (8)
EgoMCQ Acc (%)
Zero-shot T Retrieval [20]
Clip creation strategy
Avg + Std
Inter-vidco
Intra-videe vidco
mAP (avg)
nDCG (avg)
(a) L1 a]]
5.0 0.0
87.66
39.72
19.6
12.3
(b)
44-0/ 23-a/2,4++a/2
0.0
89.23
41.68
20.6
13.7
(c)
14-1,4441 [1441]
10.0 38.2
88.13
40.62
20.6
13.7
(d)
[4 1,
1,
4.9 4.7
89.74
44.82
21.1
14.5
(c) t. -./44,t++3:/444
2.4
90.23
49.67
21.9
15.3
( f. - (.88/22
1.9
89.36
51.51
22.1
15.5
Table 2: Results on our development set EgoMCQ and video-text retrieval on EPIC-KITCHENS- 100 when using different strategies in the creation of EgoClip, where 4 a, 3r are defined in Eq. 1. In all experiments, we bold the best results and underlined the second best results.
contains 13.6K instances from 1.5K clips, while the validation set contains 4.3K instances from 0.5K clips. The evaluation metrics are mAP and R@ K-loU=0 for [1, 5) and A (0.3, 0.5, 0.7).
Object State Change Classification (OSCC) of Ego4D Challenges. This OSCC task is modelled as an (N+1)-way classification aiming to identify an object's state change in a given video. The training and val. sets contain 41K and 28K clips, respectively. The evaluation metric is accuracy.
Implementation Details. Our codebase is based on the official Frozen one and retains the same settings unless specified. During pretraining, we sample 4 frames for each clip, and use the Adam optimizer [41] with learning rate of 3 10 To select the best method we pretrain our architecture for 10 epochs and use the best performing model on the EgoMCQ benchmark. Pretraining takes two days on 32 A100 GPUs (1, 536 GPU hrs).
6.2 Ablation Studies
Ablation of the strategy used when creating EgoClip. We validate our proposed strategies, i.e., Eq.1 in Tab. 2, by comparing the following variants: (a) fixed length start at timestamp; (b) fixed length a, center at timestamp; (c) variable clip, start and end by adjacent timestamps; (d) our proposed strategy, scaled by 2: (e) our proposed strategy, scaled by 4; (f) our proposed strategy.
We consider that good pretraining dataset creation strategy should satisfy: (1) the VLP model trained on EgoClip should be able to well distinguish instances in EgoMCQ with the same data format; (2) the VLP model pretrained on EgoClip with the specific clip creation strategy should perform well on public downstream tasks (e.g., video-text retrieval on [20] and zero-shot for efficiency).
We draw several conclusions from Tab. 2: (i) The performance of EgoMCQ is well aligned with the zero-shot result on EPIC-KITCHENS 100, especially minor gain on downstream but noticeable on EgoMCQ, which means EgoMCQ provides valid feedback and is suitable development set. (ii) Under the same clip length a, (b) surpassing (a) proves that centering at timestamp includes prior information is helpful. (iii) Variable-length clips make a big difference, as shown in (c) and (d).
Notably, with our designed Br, (d) outperforms (b) with similar average clip length, which validates our key idea of "contextual varied clip length". (iv) Based on (d), (e), and (f), we found proper scale factor greater than is preferred, which helps focus on large of instantaneous actions densely labeled by Ego4D [15]. These ablation studies demonstrate the effectiveness of our proposed EgoClip creation strat- egy and EgoMCQ for development.
Accuracy (%) Inter-video
Variants
Intra-video
InfoNCE
89.4
51.5
(a) w/ Pos, noun
82.9 (6.5 J)
(9.2 J)
(b) w/ Pos, verb
86.9 (2.5 1)
50.5
(c) w/ Pos, noun & verh
89.7 (0.4 1)
53.6 f)
(d) w/ Neg. random
88.3 4)
49.9 L)
(e) w/ Neg, within video
89.7 (0.3 f)
53.0 f)
(f) w/ Neg. within
89.5 (0.2
54.5 (3.0 f)
(g) Pos & Neg. EgoNCE 90.6 (1.3 1) 57.2 (5.7 1)
Effect of EgoNCE. In this section, we evaluate the effect of the proposed sampling strategies for the EgoNCE objective (Eq. 3) on EgoMCQ and compare against a vanilla InfoNCE loss (Eq. 2). We ablate several configurations for positive and negative sampling strategies. The sampling strategy for positive pairs exploits language cues, while negative pairs rely on temporal, visual cues. Given a text-video pair, regard other text-video pairs positive if the textual narrations: (a) share at
Table 3: EgoNCE sampling strategy ablation. We evaluate accuracy performance on our de- velopment benchmark EgoMCQ.
https://github.com// wwwpo////mmmaaa/nnneaaeee bain/frozen-in-tim
7

---

mAP (%)
nDCG (%)
Methods
Vis Enc Input
Frames
Vis-text
V-T
T-V
Avg.
V-+T
T-+
Avg
Random
5.7
5.6
5.7
10.8
10.9
10.9.
MI-MM
S3D [42]
32
HowTol00M
34.8
23.6
29.2
47-1
42.4
44.7
MME [43]
TBN [14]
25
38.5
50.1
46.9
JPOSE [43]
TBN [14]
25
49.9
38.1
44.0
55.5
51.6
53.5
Frozen
Raw Videos
4
38.8
29.7
34.2
50.5
48.3
49.4
Frozen
Raw Videos
4
00w
39.2
30.1
34.7
50.7
48.7
49.7
Frozen
Raw Videos
4
CC3M-WebVid22M
41.2
31.6
36.4
52.7
30.2
51.4
Frozen
Raw Videos
EgoClip
39.6
55.7
52.9
FroZen+EgoNCE
Raw Videos
4
EgoClip
45.1
35.5
40.2
56.2
53.5
54.88
Frozen
Raw Videos
16
CC3M+WebVid22M
40.9
57..2
54.3
55.8
Frozen+EgoNCE
Raw Videos
16
EgoClip
49.9
40.1
45.0
60.9
57.9
59.4
Raw Videos
HowTol00M
6.5
11.6
12.8
12.2
Frozen
Raw Videos
4
CCd
8.6
7.4
8.0
14.5
14.6
14.5
Frozen
Raw Videos
4
EgoClip
17.9
13.1
15.5
23.0
21.2
22.1
Frozen NCE
Raw Videos
EgoClip
19.4
16.6
24.1
22.0
23.
Table 4: Performance of the SPIC-KITCHENS-IIIG Multi-Instance Retrieval. Note that TBN feature [14] is a combination of three modalities: RGB, Flow and Audio. Conversely, our approach only relies RGB input. The grey highlighted rows correspond zero-shot evaluation.
least one noun, (b) share at least one verb, and (c) share at least a verb-noun pair. Conversely, we define the following heuristics for negative sampling: (d) random text-video pair from EgoClip, (e) a text-video pair from the same video, and (f) a text-video pair within minute from the given video-text pair annotation timestamp. Tab. shows that using solely verbs (a) or nouns (b) for positive selection degrades the accuracy performance with respect to naive InfoNCE. However, we successfully push the performance beyond the baseline results when considering both verbs and nouns jointly (c). Moreover, we notice that merely selecting negatives within the same video leads to better performance. Finally, we pick the optimal settings from positive and negative sides and combine them together for (g) EgoNCE and reach the best results.
particular, we obtain the best performance for temporally "hard negatives" (f).
6.3 Comparisons with State-of-the-arts
Multi-Instanee Retrieval. In Tab. 4, we report both zero-shot and fine-tuning evaluation results. In the zero-shot setting, pretraining with EgoClip (3.8M), despite being smaller in scale, still outperforms CC3M+ Wed (5.5M) and HowToo100M (136M), validating the unique benefit of pretraining on egocentric data. When fine-tuned with frames (rows 5-9). EgoClip pretraining maintains margin over the best baseline CC3M+ WIMM+-We within fine-tuning. Lastly, we increase the sample frames of our finalized model as well as the best competitor CC3M+ accompany the frame increase. deem that notable benefits come from better temporal modeling for frequent action in the Ist-person view. Overall, our pretraining model outperforms the best baseline (JPoSE) by 1.0 mAP and 5.9% nDCG while requiring fewer frames and input modalities.
further verifying the viewpoint domain gap
pretraining to 16 (rows 10-11). As expected, performance gains
Natural Language Query. We report validation results on Tab. 5. We adopt the same baselines as introduced in [15], namely: 2DTAN [44] and VSINet [45], and substitute the SlowFast-BFil features with our video and language representations. We observe large boost in performance offered by our pretrained model on all metrics. Notably, we improve R@ for loU=0.3 from 5.45 to 10.84, despite our video branch not being pre-trained on Kinetics400. Besides, we significantly surpass VLP pretrained on CC3M+ VidVi-2M and H0WTo We believe that this increase is due
Methods
Video-text Pre-extrated Features
IoU=0.3
IoU=0.5
Vis-text Enc
Vis-text
R@1
R@5
R@1
R@5
2D-TAN [44]
BlowFast+BERT
5.04
12.89
2.02
5.88
VSLNet [45]
BlowFast+BERT
5.45
10.74
3.12
6.63
VSLNet [45]
Frozen
HowTo100M
3.95
8.72
2.01
4.62
VSLNet [45]
Frozen
WebM+Webiid
5.06
10.30
2.71
6.69
VSLNet [45]
Frozen
EgoClip
10.53
17.94
5.96
11.85
VSLNet [45]
Frozen+EgoNCE
EgoClip
10.84
18.84
6.81
13.45
Table 5: Recall for several IoUs the NLQ task's val. set.

---

Methods
Vis Enc
Frames
Vis-Text PT
Train / FT Data
mAP (%)
Actor [46]
ResNet-152
Charades- (Ist 3rd)
20.0
SSDA [47]
13D
32
Charades-Egg (Ist + 3rd)
23.1
13D [47]
L3D
32
Charades-Bae Sarades-EEo-aa (1st).
25.8
Ego-Exo [48]
FlowFast (ResNet-101)
Charades-Fage (Ist)
30.1
Frozen
TimeSformer
16
Charades- (1st)
28.8
Frozen
TimeSformer
16
HowTo100M
Charades-Egg Eggo (1st)
28.3
Frozen
TimeSformer
CC3M-WehVid-2M
Charades-Fae -go (1st)
30.9
Frozen
TimeStormer
EgoClip
Charades- (1st)
31.2
FroNcE
TimeSformer
16
EgoClip
Charades-Fgee (Ist)
32.1
Frozen
TimeSSormer
HowTo100M
9.2
Frozen
Tiommrormer
CC3M-WebWid-2M
20,9
Frozen
TimeSformer
16
EgoClip
23.6
FroNcc
TimeSformer
16
EgoClip
25.0
Table Performance the action recognition on the Charades-Ego dataset (a first-person test set). The grey highlighted rows correspond to zero-shot evaluation.
to the egocentric data availability and the video-text interaction learned from large-scale pretraining. Please Supplementary E.5 for the test set results.
Action Recognition. We conduct action recognition on Charades- -ggo where categories are short phrases like "Holding some clothes". Thus this task can be solved as a video-text retrieval by leveraging the text representation. We present the result in Tab. under zero-shot and fine-tuning settings. In zero-shot settings, our model outperforms two supervised baselines, which validates the stronger generalization of jointly learning video-text features. After fine-tuning (rows 5-9), our model surpasses all VLP counterparts and improves over the state-of-the-ar classifier Ego-Exo by 2.0% with fewer sampled frames, which shows the superior advantage joint video-text representations.
Methods
Video Pre-extracted Features
IoU=0.3
IoU=0.3
IoU=0.7
mAP (%) (%))) @ IoU
Vis Enc
Vis-text PT
R@1
R@5
R@1
R@5
R@1
R@5
0.1
0.3
0.5
Avg
VSGN [49]
SlowFast
33.45
58.43
25.16
46.18
15.36
25.81
9.10
5.76
3.41
6.03
VSGN [49]
Frizen
1o000000M
31.40
52.61
22.28
41.29
13.41
23.21
9.83
6.72
3.84
6.72
VSGN
Frozen
CC3M&WebVid-22
32.08
23.46
23.77
6.40
3.86
6.58
VSGN [49]
Frozen
EgoClip
40.06
63.71
29.59
48.32
17.41
26.33
15.90
10.54
6.19
10.69
VSGN [49)
Frozen+EgoNCE
EgoClip
40.43
65.67
30.14
51.98
19.06
29.77
16.63
11.45
6.57
11.39
Table 7: Recall and mAP metries for several IoUs on the Moment Query task's val. set.
Moment Query. This task investigates the quality of video-only features. We extract video features and provide them as input to the VSGN model [49]. We report the validation results in Tab. We find that our features achieves the best performance over SlowFast features with an increase of 4.66% in Avg mAP. Moreover, we maintain better performance with respect to Srd-person large-scale pretraining datasets. This demonstrates that the Ist-person VLP model also learns competitive vidco representations. Please see the Supplementary E.6 for the test set results.
Object State Change Classification. We report the validation results on Tab. Once again, our model achieves the best performance of all baselines, 2.4% than CC3M+ Web Vid-2M counterparts, which indi- cates our visual representations are able to focus on fine-grained clues related to state changes.
Methods
Vis-Text PT
Acc. (%)
Always Positive
48.1
Bi-d LSTM [50]
ImageNet
65.3
I3D (ResNet-50) [51]
68.7
Fruzen
70.3
Frozen
HowTo100M
71.7
Summary of EgoNCE. From the above experimen- tal results, Frozen pretrained on EgoClip with the EgoNCE objective brings a consistent improvement over the InfoNCE on all downstream tasks, which comprehensively demonstrates the effect of EgoNCE, as well the decision from EgoMCQ.
Frozen
CC3M-WebVid-22
71.5
Frozen
EgoClip
73.4
Froze++ggoNCE
EgoClip
73.9
Table 8: Accuracy metric on the Object State Change Classification task's val. set.

---

7 Conclusion, Limitations, and Societal Impacts.
To the best of our knowledge, this work is the pioneering work unlock Egocentric VLP. (i) We devise a principled data curation and create EgoClip, an egocentric large-scale text-video pretraining dataset with 3.8M clip-text pairs well-chosen from Ego4D. (ii) We exploit the particular character- isties of egocentric videos and devise EgoNCE with meaningful sampling strategies for effective egocentric pretraining. (iii) We create EgoMCQ. an egocentric video-language benchmark close to the pretraining set to support efficient exploration and development of EgoClip and EgoNCE. Finally, we further demonstrate the strong representation of our egocentric pretraining on five tasks across three datasets. We believe that our EgoClip, EgoMCQ and EgoNCE would greatly benefit the egocentric video community. laying good foundation for the new research trend of egocentric VLP. Limitations. Our pretraining approach does not take into account the long-term temporal dependen- cies in long Ego4D videos. We leave this for future work.
Societal impact. Egocentric VLP learns real-world perception knowledge that may contribute to practical applications such as augmented reality and robotics. However, Ego4D videos collected by participants may contain users' privacy and unintended biases, so should be used cautiously. We refer the readers to the Ego4D paper about further privacy and societal impacts.
8 Acknowledgements
This project is supported by the National Research Foundation, Singapore under its NRFF Award SHRRFFI3-22210000 and Mike Zheng Shou's Start-Up Grant from NUS. The computational work for this article was partially performed on resources of the National Supercomputing Centre, Singapore. Michael Wray and Dima Damen are supported by EPSRC UMPIRE (EP/T004991/1), Mattia Soldan and Bernard Ghanem are supported by the King Abdullah University of Science and Technology (KAUST) Office of Sponsored Research through the Visual Computing Center (VCC) funding, as well as, the SDAIA-KAUS Center of Excellence Data Science and Artificial Intelligence SDAIIA-KUST AI). Thanks to Tencent Data Platform for the support of computing resources. Our work is built upon the Ego4D dataset, and we greatly appreciate the contributions and efforts of the Ego4D community.
References
[1] Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In CVPR, pages 5288-5296, 2016.
[2] Mandela Patrick, Po-Yao Huang, Yuki Asano, Florian Metze, Alexander Hauptmann, Joao Henriques, and Andrea Vedaldi. Support-set bottlenecks for video-text representation learning. In ICLR, 2020.
[3] Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In ICCV, pages 1728-1738, 2021.
[4] Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. MM, pages 1645-1653, 2017.
[5] Youngjae Yu, Jongseok Kim, and Gunhee Kim. A joint sequence fusion model for video question answering and retrieval. In ECCV, pages 471-487, 2018.
[6] Linchao Zhu and Yi Yang. Actbert: Learning global-local video-text representations. In CVPR, pages 8746-8755, 2020.
[7] Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense- captioning events in videos. In ICCV, pages 706-715, 2017.
[8] Bairui Wang, Lin Ma, Wei Zhang, and Wei Liu. Reconstruction network for video captioning. In CVPR, pages 7622-7631, 2018.
[9] Luowei Zhou, Yingbo Zhou, Jason J Corso, Richard Socher, and Caiming Xiong. End-to-end dense video captioning with masked transformer. In CVPR, pages $739-8748, 2018
[10] Antoine Miech, Dimitri Zhukov, Jean- Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. Howtol00ml Learning text-video embedding by watching hundred million narrated video clips. In ICCV, pages 2630-26400 2019.
10

---

[11] Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: large-scale video benchmark for human activity understanding. In CVPR, pages 961-970, 2015.
[12] Yazan Abu Farha, Alexander Richard, and Juergen Gall. When will you do what?-amticipating temporal occurrences of activities. In CVPR, pages
2018.
[13] Yu-Fei Ma, Lie Lu, Hong-Jiang Zhang, and Mingjing Li. A user attention model for video summarization. In MM, pages 333-542, 2002.
[14] Evangelos Kazakos, Arsha Nagrani, Andrew Zisserman, and Dima Damen. Epic-fusion: Audio- visual temporal binding for egocentric action recognition. In ICCV, pages 5492-5501, 2019.
[15] Kristen Grauman, Andrew Westbury, Eugene Byme, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang. Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In CVPR, pages 18995-19012, 2022.
[16] Luowei Zhou, Chenliang Xu, and Jason J Corso. Towards automatic learning of procedures from web instructional videos. In AAAI, 2018.
[17] Gunnar A Sigurdsson, Abhinav Gupta, Cordelia Schmid, Ali Farhadi, and Karteck Alahari. Charades-ego: A large-scale dataset of paired third and first person videos. arXiv preprint arXiv:1804.09626, 2018.
[18] Yong Jae Lee, Joydeep Ghosh, and Kristen Grauman. Discovering important people and objects for egocentric video summarization. CVPR, pages 1346-1353, IEEE, 2012.
[19] Alircza Fathi, Jessica K. Hodgins, and James M Rehg. Social interactions: A first-person perspective. In CVPR, pages 1226-1233, IEEE, 2012.
[20] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Evangelos Kazakos, Jian Ma, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Rescaling egocentric vision:
Collection, pipeline and challenges for epic-Kitchens-100 IJCV, 130(1):33-55, 2022.
[21]
Lisa Anne Hendricks, Oliver Wang. Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. Localizing moments video with natural language. In ICCV, pages 5803-5812, 2017.
[22] Long Chen, Hanwang Zhang, Jun Xiao, Liqiang Nie, Jian Shao, Wei Liu, and Tat-Seng Chua. Sea-cnm: Spatial and channel-wise attention in convolutional networks for image captioning. In CVPR, pages 5659-5667, 2017.
[23] Antoine Miech, Jean-Baptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman. End-to-end learning of visual representations from uncurated instructional videos. In CVPR, pages 9879-9888888 9889, 2020.
[24] Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara Berg, Mohit Bansal, and Jingjing Liu. Less is more: Clipbert for video-and laln-aand-language learning via sparse sampling. In CVPR, pages 7331-7341, 2021.
[25] Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. Videobert: A joint model for video and language representation learning. In ICCV, October 2019.
[26) Jinpeng Wang, Yixiao Ge, Guanyu Cai, Rui Yan, Xudong Lin, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Object-aware video-language pre-training for retrieval. In CVPR, pages 3313-3322, 2022.
[27] Victor Escorcia, Mattia Soldan, Josef Sivie, Bernard Ghanem, and Bryan Russell. Tem- poral localization of moments in video collections with natural language. arXiv preprins 1907 127663 2019.
[28] Antoine Miech, Jean- Caan-Baptiste Alayrac, Ivan Laptev, Josef Sivic, and Andrew Zisserman. Think- ing fast and slow: Efficient text-to-visual retrieval with transformers. In CVPR, pages 9826-9836, 2021.
[29] Aaron Van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv e-prints, pages arXiv-1807 2018.
[30] Benita Wong, Joya Chen, You Wu, Stan Weixian Lei, Dongxing Mao, Difei Gao, and Mike Zheng Shou. Assistq: Affordance-centric question-driven task completion for egocentric assistant. In ECCV, 2022.
[31] Yin Li, Zhefan Ye, and James M Rehg. Delving into egocentric actions. In CVPR, pages 287-295, 2015.

---

[32] Gedas Bertasius, Heng Wang. and Lorenzo Torresani. Is space-time attention all you need for video understanding? In ICML, volume 2. page 4, 2021.
[33] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv: 108, 2019.
[34] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In ICCV, pages 6202-6211, 1, 2019.
[35] Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin transformer. CVPR, pages 3202-3211, 2022.
[36] Antoine Miech, Jean-Baptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman. End-to-end learning of visual representations from uncurated instructional videos. In CVPR, pages 9879-9889, 2020.
[37] Michael Wray, Hazel Doughty, and Dima Damen. On semantic similarity in video retrieval. In CVPR, 2021.
[38] Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. Localizing moments in video with temporal language. In EMNLP, 2018.
[39] Gao Jiyang, Sun Chen, Yang Zhenheng, Nevatia, Ram. TALL: Temporal Activity Localization via Language Query. In ICCV, 2017.
[40] Mattia Soldan, Mengmeng Xu, Sisi Qu, Jesper Tegner, and Bernard Ghanem. Vlg-net: Video- language graph matching network for video grounding. In ICCV, pages 3224-3234, 2021.
[41] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6988 2014.
[42] Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy, Rethinking spa- tiotemporal feature learning: Speed-accuracy trade-off's in video classification. In ECCV, pages 305-321, 2018.
[43] Michael Wray, Diane Larlus, Gabriela Csurka, and Dima Damen. Fine- tine-grained action retrieval through multiple parts-of-speech embeddings. In ICCV, pages 450-459, 2019.
[44] Songyang Zhang, Houwen Peng, Jianlong Fu, and Jiebo Luo. Learning 2d temporal adjacent networks for moment localization with natural language. AAAI, volume 34, pages 12870- 1 12877, 2020.
[45] Hao Zhang, Aixin Sun, Wei Jing, and Joey Tianyi Zhou. Span-based localizing network for natural language video localization. In ACL, pages 6543-6554, 2020.
[46] Gunnar A Sigurdsson, Abhinav Gupta, Cordelia Schmid, Ali Farhadi, and Karteek Alahari. Actor and observer: Joint modeling of first and third-person videos. In CVPR, pages 7396-7404, 2018.
[47] Jinwoo Choi, Gaurav Sharma, Manmohan Chandraker, and Jia-Bin Huang. Unsupervised and semi-supervised domain adaptation for action recognition from drones. In WACV, pages 1717-1726, 2020.
[48] Yanghao Li, Tushar Nagarajan, Bo Xiong, and Kristen Grauman. Ego-exo: Transferring visual representations from third-person first-person videos. In CVPR, pages 1943-6953, 2021.
[49] Chen Zhao, Ali K Thabet, and Bernard Ghanem. Video self-stitching graph network for temporal action localization. In ICCV, pages 13658-13667, 2021.
[50] Alex Graves, Santiago Fernández, and Jürgen Schmidhuber. Bidirectional Istm networks for improved phoneme classification and recognition. In ICANN. pages 799-804, Springer, 2005.
[51] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. CVPR, pages 0299,6308, 2017.
[52] Jie Lei, Tamara I. Berg, and Mohit Bansal. Detecting moments and highlights in videos via natural language queries. NeurIPS, 34:1 14:118448 18:161166118,, 2021.
[53] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition at scale. In ICLR, 2020.
12


Reviews:
[{"review_id": "flMAcj9Wshh", "reviewer": "Reviewer_cPhr", "summary": "This paper proposes a method for video+language learning, specifically focused on the ego-centric setting. The paper introduces a new way to mine positive/negative samples for InfoNCE style training, based on sampling by overlapping nouns and verbs in the video caption. The paper introduces a new evaluation setting on the ego4d dataset and evaluates on epic kitchens and charades ego.", "questions": "Is there any reason the EGOClip method couldn't be applied to 3rd person videos? None of the approach seems specific to egocentric videos in ways that other videos wouldn't be applicable (e.g., ego-motion).", "limitations": "There is some slight discussion on these, but it is very limited. For example, there are many societal impacts of ego-centric videos, such as surveillance, privacy issues, etc. that could be discussed. This goes beyond the impact of the electricity used to train the models.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The proposed mining method for the positive/negative samples is interesting and effective. The results show good performance across multiple datasets and tasks. The state-of-the-art comparisons also show the difference due to different PT datasets and other settings. Overall, the results are fairly complete and show the benefits of the components.\n\nMinor comments:\n\"Fortunately, with the recent introduction of the massive-scale egocentric video dataset Ego4D [16], it becomes possible to unlock Egocentric VLP.\"\n\nThat is a strong claim, given that in Table 1, Ego4D is still a fraction of the size of other datasets, eg HowTo100M, WebVid, YouTube-Temporal-180M, etc. I wouldn't call Ego4D \"massive-scale\". The results also are significantly better than the previous results, which suggests this isn't really \"unlocking\" anything.\n\nThere are typos and grammatical errors throughout the paper. It would benefit from careful editing.\n\nSome of the notation is unnecessary complicated. For example, the subscript $i$ in Eq. 1 doesn't add anything. I.e., that section could be revised without the subscript without any loss in clarity. This would also simplify Table 2.\n\nNote that the answer to the licenses is incorrect, Ego4D does not have the MIT license, and has it's own license with certain restrictions. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Z6Hy9Dcgh9", "reviewer": "Reviewer_pWxJ", "summary": "This paper proposes a new benchmark for egocentric video understanding based on Ego4D, a pre-training set, and a model for the pre-training and downstream fine-tuning. Given the untrimmed Ego4D videos, the authors do the curations and build the EgpClip. Based on the recent work Frozen, a contrastive video learning model with a new loss RgoNCE is proposed. To test the above design, the authors also propose a valid set named EgoMCQ based on the val data of Ego4D. On several downstream tasks, the pre-training set and model show impressive performance compared to existing first-view pre-training works given the large-scale video-text data from Ego4D.", "questions": "L184-185: to assure that the scene is not visible during pretraining, we manually remove videos that share multiple views with the videos in EgoClip.\nAny details about the manually removing?", "limitations": "N/A.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n+ The curation and cleaning for Ego4D are important and beneficial for the community.\n\n+ The illustrations are detailed and easy to follow. The suppl gives much useful information.\n\n+ The experiments indeed show good signs when we use this new curated dataset for ego-centric video learning.\n\n+ The main setting and curation are sound.\n\nCons:\n- The largest concern: this work seems not to differentiate the contribution of Ego4D and this submission. \n\n1. The tab1 is misleading, as the data scale and contribution belong to Ego4D instead of this work. This work conducts mainly the curation and cleaning. So I suggest comparing both Ego4D and EgoClip here to avoid this misleading.\n\n2. The results look good and impressive, but the readers may wonder how much of them come from the curation? Though without curation, the videos in Ego4D may be not easy to directly use, the performance improvements can be definitely thanks to the data contribution of Ego4D when compared with the first-view data pre-training. Only an \"EgoClip\" is weird and misleading. Thus, I suggest the authors add the tests to use the original videos of Ego4D in soma ways, or if not at least discuss this clearly. If we think about this point, the data contribution of this work and the performance claim may be misaligned.\n\n3. Considering the method contribution is relatively smaller compared to the data,  in my opinion, this work may be more suitable for the benchmark and dataset track if the main contributions are not revised.\n\n- The evaluation: The retrieval tests are essential, but more video recognition tests can make the evaluation more solid, e.g., on Egtea, Epic, etc.\n\n- Typo: L246, i.e.Eq.1", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "PohF5UoBOfO", "reviewer": "Reviewer_93Gf", "summary": "The paper introduces a first person video-text pretraining dataset called EgoClip comprising 3.8M clip-text pairs chosen from Ego4D. Ego4D is a large-scale dataset comprising first person videos from a large variety of daily human activities. The paper further proposes a pretraining objective called EgoNCE to adapt the video-text contrastive learning to the egocentric domain by mining egocentric aware positive and negative samples. Finally, the paper introduces a new development benchmark called EgoMCQ that is close to the EgoClip dataset and can therefore be used to effectively validate and explore algorithms for EgoClip (which they showcase by exploring EgoNCE on  EgoClip). \n\nEgoClip is sourced from Ego4D. Through a series of criteria, a set of pairs of video and textual narrations are chosen with low amount of noise in terms of video-text misalignment and video-text irrelevance. The final dataset has 2.9K hours of videos with 3.85 million narrations from 129 different scenarios. There are 21.9 clips per minutes with average clip length of 1s and standard deviation of 0.9s. This is followed by designing a contextual variable-length clip pairing strategy where centered around the instantaneous timestamp of narration, a temporal duration of $\\beta/\\alpha$ is specified such that $\\beta$ is the average temporal distance between pairs of consecutive narrations and $\\alpha$ is the scale factor computed as the average of all $\\beta$s across all videos in EgoClip.\n\nFor video-language pretraining, the paper proposes EgoNCE which is an extension of InfoNCE contrastive loss between video-text pairs. EgoNCE adapts InfoNCE for egocentric videos by doing action-aware positive sampling where batch samples that share at least one noun and at least one verb are treated as positive samples. It further has scene-aware negative sampling where different actions in the same scenario are treated as hard negative samples. \n\nFinally, the paper develops a new development benchmark to evaluate on EgoClip called EgoMCQ. The motivation is that standard video-text retrieval may be ineffective since there are substantial duplicate/semantically similar captions in Ego4D. Multiple Choice Question (MCQ) task makes repetitions highly unlikely given a small number of answers. EgoMCQ considers an inter-video setting where five clips originate from different videos, aiming to distinguish instances from different scenarios. EgoMCQ also considers an intra-video setting by grouping five continuous clips together focusing on fine-grained context clues such as hand interaction. \n\nTo validate the proposed method, experiments are conducted on five egocentric benchmarks. Experiments show the EgoClip and EgoClip along with EgoNCE perform better than existing methods on all benchmarks. This is because EgoClip helps bring pretraining closer to egocentric downstream tasks. Ablation is conducted on the effect of EgoNCE showing that it is more effective than InfoNCE. Ablation is also conducted to validate the effectiveness of the proposed development benchmark EgoMCQ.\n", "questions": "Questions/Suggestions\n\n1. I would request authors to shed some more light on scenarios where the action-aware positive sampling and scene-aware negative sampling can potential conflict and whether scene-aware negative sampling can introduce false negatives.\n\n2. I would request authors, if possible, to validate the dataset, approach and benchmark on another backbone, such as I3D or Video Swin Transformer, to provide another data point on the ability for the proposed elements to be generalizable across different backbones.", "limitations": "I believe the authors have adequately addressed the limitations and potential negative societal impact of their work.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n1. The proposed components of the work overall have significant novelty and help to improve pretraining for egocentric downstream tasks. EgoClip and accompanying EgoMCQ benchmark will be very helpful to the egocentric community to develop better pretrained models suited for egocentric tasks.\n2. The overall paper quality is also high. The paper is easy to read, follow and understand.\n3. Extensive evaluation has been performed to evaluate all aspects of the proposed method. The experiments help gather insight on each aspect in a comprehensive manner.\n\nWeaknesses\n\n1. Since egocentric videos are untrimmed and consecutive actions are quite fine-grained, I have some doubts about the scene-aware negative sampling which considers different actions in the same scenario as hard negative samples. But it is possible that different actions in the same scenario can share the same noun. In that case, it appears it might conflict with the action-aware positive sampling. Is it possible that scene-aware negative sampling can introduce false negatives?\n\n2. All experiments for EgoClip seem to be conducted using the Frozen method and TimeSformer backbone. It would be more helpful to see if, just for proof of concept on maybe one setting, how the dataset and approach performs with another backbone. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "4KROUiTXx8w", "reviewer": "Reviewer_VVdo", "summary": "This paper studies video-language pre-training for egocentric videos.  The authors claim 3 contribution: (1) cleaning the Ego4D dataset to form a large video-text training corpus. (2) proposed EgoNCE to tackle the two challenges (same action in different scenes, and different actions in the same scene) in egocentric video-text pre-training (3) built on top of the Ego4D dataset, the authors introduced EgoMCQ as a development set.", "questions": "- Presentation suggestions:\n    - Since the paper contains a dataset contribution, it would be great to include a datasheet for it.", "limitations": "The limitations are discussed in Sec 7.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- The authors studies pre-training for ego-centric videos, which is an important topic for the community, and could be impactful.\n- The experimental results are mostly positive, supporting the claims in the paper.\n\n**Weaknesses**\n\n- There is a lack of architectural novelty: the model architecture is essentially the same as Frozen [11].\n- L158-165, the authors proposed action-aware positive sampling to address challenge 1 (L151), which relies on the taxonomy annotation specific to the Ego4D dataset and has limited scope, which may not applicable to more general domain ego videos.\n- The scene-aware negative sampling is essentially negative sampling from the same video. This technique is often used for tasks like video moment retrieval [Hendricks et. al, Lei et al.], thus it is also not new.\n- In the ablations, it is shown that using EgoClip alone is better than using 3rd person view data, either HT100M or CC3M+WebVid2M. It would be interesting to also explore whether the model can benefit from training on a combination of 1st and 3rd view data. E.g., pre-training on EgoClip+CC3M+WebVid2M.\n- The two challenges discussed in L150-155 do not seem to unique to 1st person videos. E.g., the same action “running” could happen in a street, a park, or in a playground.\n- The claimed contribution (i) (L6) seems trivial, it is almost simply done by doing a bit transform of the Ego4D dataset annotations.\n- It is not clear why we need another development set (EgoMCQ), if there are already a bunch of tasks which already has their development set.\n\nHendricks et. al, Localizing Moments in Video with Natural Language, ICCV 2017\n\nLet et. al, QVHIGHLIGHTS: Detecting Moments and Highlights in Videos via Natural Language Queries, NeurIPS 2021", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["Inadequate Data and Algorithm Evaluation"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept