Localizing Active Objects from Egocentric Vision with Symbolic World Knowledge
Authors: Te-Lin Wu
Yu Zhou
Nanyun Peng
Venue: EMNLP 2023 Main (2023)
Keywords: Object state change
Pre-conditions
Post-conditions
Egocentric videos
Active grounding
Multimodal

Abstract: The ability to actively ground task instructions from an egocentric view is crucial for AI agents to accomplish tasks or assist humans virtually.
One important step towards this goal is to localize and track key active objects that undergo major state change as a consequence of human actions/interactions to the environment without being told exactly what/where to ground (e.g., localizing and tracking the `sponge` in video from the instruction "Dip the `sponge` into the bucket.").
While existing works approach this problem from a pure vision perspective,
we investigate to which extent the textual modality (i.e., task instructions) and their interaction with visual modality can be beneficial.
Specifically, we propose to improve phrase grounding models' ability on localizing the active objects by:
(1) learning the role of `objects undergoing change` and extracting them accurately from the instructions,
(2) leveraging pre- and post-conditions of the objects during actions,
and (3) recognizing the objects more robustly with descriptional knowledge.
We leverage large language models (LLMs) to extract the aforementioned action-object knowledge,
and design a per-object aggregation masking technique to effectively perform joint inference on object phrases and symbolic knowledge.
We evaluate our framework on Ego4D and Epic-Kitchens datasets.
Extensive experiments demonstrate the effectiveness of our proposed framework, which leads to>54% improvements in all standard metrics on the TREK-150-OPE-Det localization + tracking task, >7% improvements in all standard metrics on the TREK-150-OPE tracking task, and >3% improvements in average precision (AP) on the Ego4D SCOD task.

Localizing Active Objects from Egocentric Vision with Symbolic World Knowledge
TinLin Wu Yu Zhou' Nanyun Peng
University of California, Los Angeles (telimmu,violetpenggeeee .cla ucla.edu, yu yu.zhou@ucla.edd
Abstract
Pre-Cond: (lond:(Rian
ability to actively ground task instructions from egocentric view is crucial for AI agents to accomplish tasks or assist humans. One important step towards this goal is local- ize and track key active objects that undergo major state change as a consequence of hu- man actions/interactions in the environment (e.g., localizing and tracking the 'sponge" in video from the instruction "Dip the sponge into the bucket.") ") without being told exactly what/where to ground. While existing works approach this problem from a pure vision per- spective, we investigate to which extent the language modality (i.e., task instructions) and their interaction with visual modality can be beneficial. Specifically, we propose improve phrase grounding models' (Li et al., 2022) ability in localizing the active objects by: (1) learning the role of objects undergoing change accurately extracting them from the instruc- tions, (2) leveraging pre- and post-conditions of the objects during actions, and (3) recogniz- ing the objects more robustly with descriptional knowledge. We leverage large language models (LLMs) to extract the aforementioned action- object knowledge, and design per-object ag- gregation masking technique to effectively per- form joint inference on object phrases with symbolic knowledge. We evaluate
Object Undergoing Change
Post Cest-Cond: Opened)
Cut the pawpaw into half with the knife.
fruit with yellow-greenist flesh..
Figure I: Active object grounding is the task of localizing the active objects undergoing state change (OUC). In this ex- ample action instruction "cut the pawpaw into half with the knife", the Al assistant is required to firstly infer the OUC (pawpaw) and the Tool (knife) from the instruction, and then localize them in the egocentric visual scenes throughout the action trajectories. Symbolic knowledge including pre/post conditions and object descriptions can bring additional infor- mation to facilitate the grounding.
as Meta, Google, and Apple have attracted growing research in on-device Al that can provide just-in- time assistance to human wearers. While giving (or receiving) instructions during task execution. the AI assistant should co-observe its wearer's 's first- person (egocentric) viewpoint to comprehend the visual scenes and provide appropriate assistance. To accomplish this, it is crucial for Al to first be able localize and track the objects that are un- dergoing significant state change according to the instruction and/or actions performed. For example in Figure 1, it can be inferred from the instruction that the object undergoing change which should be actively grounded and tracked is the pawpaw. Existing works have focused on the visual modal- ity alone for such state change object localization tasks, including recognizing hand-object interac-
frame- work on Ego4D (Grauman et al., 2022) and spic-Kitchens (Dunnhofer al., 2022) datasets. Extensive experiments demonstrate the effec- tiveness of our proposed framework, which leads 54% improvements all standard metrics on the TREK-1500 100KK-----P----- localiza- tion tracking task, > 7% improvements in all standard metrics the TREK-150-0P track- ing task, and > 3% improvements in average precision (AP) on the Ego4D SCOD task.
1 Introduction
Recent technological advancements in smart glasses (and headsets) from industry leaders such
"The authors contribute equally, alphahetical order.
'Cdee at: https: httpt/gitttttbbb

---

tions (Shan et al., 2020a) and object visual state changes (Alayrac al., 2017). However, remains under-explored whether the visual modality by it- self is sufficient for providing signals to enable ro- bust state change object localizing/tracking without enhanced signals from the textual modality. While utilizing phrase grounding model (Liu et al., 2022, 2023) is presumably a straightforward alternative, it leaves unanswered questions of which mentioned objects/entities in the instruction are supposedly the one(s) that undergo major state changes, e-g-- the pawpaw in Figure 1 instead of the knife is the correct tbbeet- Furthermore, how visual ap- pearances of the objects can help such multimodal grounding is yet to be investigated.
Pre-Frame
FNR-Frame
Post-Frame
Use the sprra mozzlemaa to spray the sar windshiel@eag
Figure 2: Ego4D SCOD active grounding: Example object undergo change (OUC) due to the instructed actions and asso- ciated Tools, spanning: the pre-condition, point-of-no-retuee (PNR) and post-condition frames.
narrated egocentric video datasets, Ego4D (Grau- man et al., 2022) and Epic- Spicc-Kithhens (Damen et al., 2022) and demonstrate strong gains. Our main con- tributions are two folds: (1) We design a sophisti- cated prompting pipeline to extract useful symbolic knowledge for objects undergoing state change dur- ing action from instructions. (2) We propose joint inference framework with a per-object knowl- edge aggregation technique to effectively utilize the extracted knowledge for improving multimodal grounding models.
In light of this, we tackle the active object grounding task by first extracting target object mentionsfrom the instructions using large lan- guage models (ChatGPT (OpenAT, 2023)) with a specifically designed prompting pipeline, and then finetuning an open- -oeauloccaylary detection model, GLIP (Li* et al., 2022), for visual grounding.
We further hypothesize that additional action- and object-level symbolic knowledge could be help- ful. As shown in Figure 1, state conditions prior to (pre-conditions: which indicate pre-action states) and after (post-conditions: which suggest at past state changes) the execution of the action are often considered when locating the objects. especially when the state changes more visually signifi- cant. Furthermore, generic object knowledge in- cluding visual descriptions (e.g., "yellow greeniss flesh), are helpful for uncommon objects. Based on this hypothesis, we prompt the LLM to obtain pre- and post-conditions on the extracted object mentions, along with a brief description focusing on specific object attributes.
Tasks and Terminologies
Active Object Grounding. For both robotics and assistant in virtual or augmented reality, the Al ob- serves (or co-observes with the device wearer) the visual scene in the first-person (egocentric) point of view, while receiving (or giving) the task in- structions of what actions to be performed next. To understand the context of the instructions as well as engage in assisting the task performer's actions, it is crucial to closely follow the key objects/entities that are involved in the actions undergoing major state change. 3 We term these actively involved ob- jects as objects undergoing change (OUC), and what facilitate such state change as Tools.
To improve the grounding models by effectively using all the aforementioned action-object knowl- edge, we design an object- mask to sepa- rately attribute the symbolic knowledge to its cor- responding object mentions for training. During in- ference time, pre-/post-condition dependent scor- ing mechanism is devised to aggregate the object and the corresponding knowledge logit scores to produce joint inference prediction.
Tasks. As there is not yet existing resource that directly studies such active instruction ground- ing problem in real-world task task uesk-perfonming situa- tions, we re-purpose two existing egocentric video datasets that can be scamlessly transformed into such a setting: Ego4D (Grauman et al., 2022) and Epic- BitchKitchens (Damen et al., 2018). Both come with per-time-interval annotated narrations tran- scribing the main actions occurred in the videos.* Ego4D: SCOD, According to Ego4D's ' definition,
We evaluate our proposed framework on two
"This should not contradict with the application where an assistive Al judges if the outcomes of the actions are desirable, as here we are only using general commonsensical conditions generated hy an LLM,, while in reality there can be more subtle and task-dependent conditions that need to he examined.
State change can come from objects' physical properties such as composition, textures, and fanctionalities; as well as attributes such as sizes, shapes, and physical affordances. "The narrations are paraphrased as imperative instructions.

---

object state change can encapsulate both spatial and temporal aspects. There a timestamp that the state change caused by certain actions start to oc- cur, i.e., the point-of-no-returm (PNR). Ego4D's state change object detection (SCOD) subtask then defines, chronologically, three types of frames: the pre-condition (Pre), the PNR, and the post- condition (Post) frames, during a performed action. Pre-frames capture the prior (visual) states where a particular action allowed take place, while post-frames depict the outcomes caused by the ac- tion, and hence also record the associated object state change. Each frame annotated with its cor- responding frame-type is further annotated with bounding boxes of the OUC (and Tools, if applica- ble), that is required to be regressed by the models. Figure shows exemplar SCOD data point.
Profrrre
Parframa Logis
2011
Localiuaioon loss
(Miox Regression)
Image Encoder
GUIP(L) Model
Dot Product
(LLM)) Symbolic Knowtedge
Cut Fish Fillasouu with ant
Figure 3: Overview of proposed framework that comprises base multimodal phrase grounding model (GLIP), frame- type predictor, knowledge extractor leveraging LLMs (GPT), and predictions supervised by both bounding box regression the objects and their ranked scores.
Method
Figure 3 overviews the proposed framework, con- sisted of: (1) A base multimodal grounding architecture, where we adopt a strong open vocabulary object detection module, GLIP (Li* et al., 2022). (2) A frame-type prediction sub- component which adds output projection layers top of GLIP to utilize both image (frame) and text features to predict of which frame-type (Pre/PN- R/Post) is currently observed. (Section 3.2.1) (3) A prompting pipeline that is engineered to ex- tract useful action-object knowledge from an LLM (GPT). (Section 3.2) (4) A per-object knowledge aggregation technique is applied to GLIP's word- region alignment contrastive training. (Section 3.3)
Our re- re-purposed active grounding task thus as follows: Given an instructed action and one of a Pre/PNR/Post-typed frames, localize (ground) both the OUC(s) and Tool(s) in the visuals. While the official SCOD challenge only evaluates the PNR frame predictions, we consider all (Pre, PNR, and Post) frames for both training and inference.
Epic-Kitchens: TREK-150. TREK-150 object tracking challenge (Dunnhofer al., 2022, 2021) enriches subset of 150 videos from the Epic- Kitchens (Damen et al., 2018, 2022) dataset, with densely annotated per-frame bounding boxes for tracking a target object. Since the Epic-Kitchens also comprises egocentric videos capturing human performing (specifically kitchen) tasks, the target objects track are exactly the OUCs per the termi- nology defined above. Hence, given an instructed action, the model is required to ground and track the OUC in the egocentric visual scenes.
3.1 Adapting GLIP
GLIP (Li* et al., 2022; Zhang et al., 2022) achieves open vocabulary object detection by pre- training on a contrastive phrase grounding objec- tive. Specifically, GLIP extends the text(caption)- to-image dot product matching objective from CLIP (Radford et al., 2021) a word-region-leve alignment objective. For some (tokenized) words of the textual description of an image, there are certain image region(s) that could be grounded to, while other regions are viewed as the negative sam- ples for the CLIP-like alignment contrastive learn- ing, During pretraining, GLIP utilizes both phrase grounding datasets (Ordonez et al., 2011; Plummer et al., 2015; Sharma al., 2018) and object detec- tion datasets (Krishna et al., 2017; Krasin et al., 2017; Shao et al., 2019).
It is worth noting that some OUCs may occa- sionally go "in-and-out" of the egocentric point of view (PoV), resulting in partial occlusion and/or full occlusion frames where no ground truth anno- tations for the OUCs are provided. Such frames are excluded from the final evaluation. And in Sec- tion 4.2.3 we will show that our proposed model is very successful in predicting the objects when they come back due to the robustness of our symbolic joint inference grounding mechanism.
Unlike Ego4D SCOD lask, TREK-1 -550 does not contain any defined Pre/PNR/Post frames. Our proposed model is trained to perform joint inference and autonomously decide which of the pre- and post-conditions to weigh more based on the frame image and instructed action. And hence, in the TREK- -500 task, frame-type information is not required.
Contrastive Learning. We illustrate the GLIP
"The descriptions of object detection are a simple concate- nation of all the available object class labels.

---

Prenced CO0k// /
Dat ensu Aoe/
GLIP Architecture
Word Festuras
orC baedoo price of
Irage-lagien Famtun
with
Pask rew
shors metslli
skod est
Nust absevied
Shar alaleo
Pru- Framms
Post-Frame
Apprymmss
Word Ragion
Alignment Scores
Moord-Ragion (from Symbolic Knowfedge) Alignment Scores
For Framt Logtt
Localization Loss
Alignment Loss
Figure 4: Model architecture (knoviedge-enhanced grounding): On the left depicts the word-region alignment (contrastive) learning of the base GLIP architecture, where the model is trained to align the encoded latent word and image features with their dot- product logits heing supervised by the positive and negative word-region pairs. On the right illustrates the enhanced object-knowleett knowledge grounding. During training we apply an object-type dependent mask to propagate the positive alignment supervisions; while during inference time the frame-type predictor (offline trained by the encoded textual and image features) phrases
acts as combinator to fuse dot product-logi scores from both (extracted) object phrases and corresponding knowledge. (Note that for simplicity we do not fully split some phrases into individual words.)
training adapted Notice that for simplicity we do not fully expand the tokenized word blocks, e.g., "fish filler" should span two words where each word (fish" and "fll- ler") and its corresponding region is all regarded as the positive matching samples. The model is trained to align the encoded latent word and image features? with their dot-product logits being su- pervised by the positive and negative word-region pairs. The alignment scores will then be used to score (and rank) the regressed bounding boxes pro- duced by the image features, and each box will fea- ture an object-cluss prediction score. Concretely, for jth regressed box, its grounding score to a phrase W = hwr is a mean pooling of the dot-products between the jth region feature and all the word features that compose such phrase: Stoz Σ I, W. In this work, we mainly focus the OUC and Tool object classes, i.e., each textually-grouuse
task in Figure 4 left half.
post-processing to further refine the extractions. 8 Conditioned on the extracted OUC (and Tool), two additional queries are made to generate: (1) the symbolic pre- and post-conditions of such objects induced by the actions, and (2) brief descriptions characterizing the objects and their attributes. Inter- estingly, we empirically find it beneficial to situate GPT with a role, egg., "From the first-person view."
GPT Intrinsic Evaluation. In Table 2, we au- tomatically evaluate the OUC/Tool extraction of GPT against the labelled ground truth entities in both datasets. We report both exact (string) match and word overlapping ratio (as GPT often extracts complete clauses of entities), to quantify the robust- ness our GPT active entity extractions.
Table 3 reports human evaluation results of GPT symbolic knowledge, including pre-/post- conditions and descriptions. Evaluation is based two binary metrics, namely: (1). Textual Cor- rectness: "Based on text alone, does the knowledge make sense?" and (2). Visual Correctness: "Does the conds. match the image?" Despite impres- sive performance on both intrinsic evaluations, we qualitatively analyze in Table some representa- tive cases where GPT mismatches with annotations or humans, including cases where GPT's answer is actually more reasonable than the annotations.
region will further predict whether is an OUC or Tool class.
3.2 LLM for Action-Object Knowledge
Pipeline. As illustrated in Figure 5, we implement LLM query pipeline to extract active entities and relevant symbolic knowledge from an instructional caption. To account for GPT's verbose tendency, we forcibly instruct GPT to produce the active ob- jects (OUC and/or Tool) following a specifically designed format and then apply heuristic-basse
3.2.1 Incorporating Knowledge
Adding Knowledge. We use the following schema to enrich the instruction with the obtained knowl-
FFor details of GLIP's multimodal fusion technique, we refer the readers to Lit ct al. (2022); Zhang* et al. (2022).
More details are in Append. Sec, A.2.

---

Input
Instructional Caption
[Caption]: Slash the portion spinach with the sickle.
Localizable Active Entities
Entiy
Identify Tool
OUC: [|dentify]: a portion of spinach" + [Ground]: "spinach"
Extraction
Pre-frame
Tool: [Identify]: "a sickle"
+ [Ground]: "sickle"
Precise Semantic Grounding
[OUC]: Spinach
[Tool): Sickle
Symbolic World Knowledge
Targets
Ouc
Tool
OUC: [Pre-cond]: "fresh, green' + [Post-cond]: "diced, shredded"
Phil-Frame
Tool: [Pre-cond]: "sharp, metallic" + [Post-cond]: "green, splattered'
Knowlodge Extraction
State Change Forecast
OUC: [Desc.]: "Spinach leafy green vegetable."
Tool: [Desc].A sickle is curved knife with a short handle, used for cutting grain
Object Description
crops.
Post- Fram-
Figure 5: The GPT knowledge extraction pipeline. Demonstrated through an example from the Ego4D SCOD Dataset.
Examples
Explanations
GPT: Pick up some green papersioccy from the table. [No Tool]
Without visual knowledge input, GPT not robust phrase ambiguity, leading to undesirable definition.
Desc.: "Green papers are consultation documents issued by government""
GPT: Cut the fish filletone, with a knife
LLMs may hallucinate exaggerated state changes. in this case claiming the knife to he "hlunt, distorted" after a single use, which unreasonable.
OUC: [Pre-cond]: "fresh, raw" [Post-cond]: "sliced, cut"
Tool: [Pre-cond]: "Sharp, metallic" [Post-cond]: "Blunt, distorted"
GPT: Hold the iron,orcy on the ironing board with your hand. [No Tool]
"Pants" is not mentioned in the narration, GPT fails to capture OUC due to text narration reporting bias.
Ego4D gt-label: [OUC]: "pants" [Tool]] "iron"
GPT: Spin the moprouc, the mop bucket spinner
GPT prediction more reasonable compared to the Ego4D ground truth label.
Ego4D gt-label: [OUC]: "mop" [Tool]]:
"mop"
Table Qualitative Analysis of GPT Knowledge Extraction: Examples of cases where GPT-extracted symbolic knowledge are wrong or conflict with Fgo4D annotations. Here the GPT-extracted or datasct-annotated knowledge are displayed in GREEN if they match human analysis and RED otherwise. Explanations for each example are provided on the right.
Object Type
Ego4D SCOD
TREK-150
edge to PNR frame yield better results. As Fig- ure 4 illustrates (omitting some prefixes for sim- plicity), we propagate the positive matching labels to object/tool's 's corresponding knowledge. In the same training mini-batch, we encourage the con- trastiveness to focus on more detailed visual appear- ance changes grounded to the symbolic condition statements and/or descriptions, by sampling frames from the same video clips with higher probability.
EM (%)
Overlap.
EM (%)
Overlap.
OUC
77.8
88.6
76.0
94.3
Tool
60.3
88.5
Table 2: Automatic evaluation of GPT entity extraction. Abbreviations: EM: exact string matching: Overlap: The ra- tio of GPT extractions fully covering the ground truth phrases
Knowledge Type
Ego4D SCOD
TREK-150
Textual
Visual
Textual
Visual
Pre-Cond.
86.5
81.6
83.0
79.9
Post-Cond.
75.2
70.3
76.6
73.5
Using both the encoded textual and image features, we learn an additional layer to predict the types of frames conditioned on the associated language instruction. Note that the frame-type definition proposed in Ego4D should be generalizable outside of the specific task, i.e,, these frame types could be defined on any kinds of ac- tion videos. In addition to the annotated frames in SCOD, randomly sub-sample nearby frames within 0.2 seconds (roughly 5-6 frames) ex- pand the training data. The frame-type prediction achieves 64.38% accuracy on our SCOD test-set, which is then directly applied to the TREK-150 task for deciding the amount of pre- and post-condition
Frame-Type Prediction.
Desc. 98.9
91.4
99.2
95.3
Table 3: Human evaluation of GPT symbolic knowledge extraction. Abbreviations: Textual: i.c. "textual correctness" "Based text alone, does the GPT cdess/ make sense?". Visual: i.. "visual correctness": ; "Does GPT conds/Jesc match what shown the image?"
edge: "/instr./ [SEP] object/tool (pre/post)-stacr state is [conds.) [SEP] object/tool description is (desc.)", where [SEP] is the separation special token; [conds.) and /desc.) are the pre-/post-condition and object definition knowledge to be filled-in. Empiri- cally, we find diffusing the post-condition knowl-

---

knowledge to use given the multimodal inputs.
(AP) as the main evaluation metric, and utilize the COCO API (Lin et al., 2014) for metric compu- tation. Specifically, we report AP, AP50, (AP at IOUZ 0.5) and AP75 (AP at IOU2 0.75).
3.3 Object-Centric Joint Inference
Masking. As illustrated in Figure 4, a straight- forward way to assign symbolic knowledge to its corresponding object type respectively is to con- struct per-object-
4.1.2 Baselines
mask. For example, an OUC mask Moue will have Is spanning the posi- tions of the words from condition (e-g- "fresh, raw" of the OUC "fish filler" in Figure 4) and descrip- tive knowledge, and Os everywhere else. We omit the knowledge prefixes in Section 3.2.1 (e.g., the phrase "object state i)") so that the models can concentrate on grounding the meaningful words. Such mask for each object type can be determin- istically constructed to serve additional word- region alignment supervisions, and can general- ize object types outside of OUC and Tool (be- yond the scope of this work) as the GPT extraction can clearly indicate the bojeet-to-knowllede corre- spondences. In other words, we enrich the GLIP's phrase grounding to additionally consider symbolic knowledge during the contrastive training. Note that the mask is frame-type dependent, e.g., MOUC and Moas will focus on their corresponding con- ditional knowledge.
We evaluate three categories of baselines: (1) Pure object detection models, where the language in- structions are not utilized. (2) (Pseudo) referential grounding, where certain linguistic heuristics are used to propose the key OUCs. (3) GPT with sym- bolic knowledge, where GPT is used to extract both the OUCs and Tools, with additional symbolic knowledge available to utilize.
Pure Object Detection (OD). We finetune the ofate-of-the-art model of the SCOD task from Chen et al. (2022) (Vidlntern) on all types of frames (Pre, PNR, and Post) to serve as the pure object de- tection model baseline, which learns to localize the OUC from a strong hand hand-object-ineee injecct-nteracttonn prior in the training distribution. We also train an OD ver- sion of GLIP providing generic instruction, "Find the object of change." to quantify its ability to fit the training distribution of plausible OUCs.
Pseudo Grounding (GT/SRL). We experiment four types of models utilizing the instructions and certain linguistic patterns as heuristics: (1) We ex- tract all the nouns using Spacy NLP tool (Honnibal and Montani, 2017) and randomly assign OUC to one of which (Random Entity). (2) A simple yet strong baseline is to ground the full sentence of the instruction if the only object class to be predicted is the OUC type (Full-Instr.). (3) Following (2), we hypothesize that the first argument type (ARG1) of the semantic-roe-llaeelln and Lin, 2019; Gardner et al., 2017) of most simple instructions is likely regarded the OUC ((SRL- ARG1)). (4) Lastly, quantify possible upper bound of simple grounding methods, we utilize the annotated ground truth object class labels from SCOD task and perform a simple pattern matching to extract the OUCs and Tools. For those ground truth words are not easily matched, we adopt the ARG1 method from (3) (GT-SRL-ARG1)
Aggregation. During the inference time, we com- bine the frame-type prediction scores //" with the per-object mask aggregate the dot-product logit scores for ranking the regressed boxes. Specifically, we have SOUC- = E, ftr M where S is a 3-way logit and fr E (Pre, PNR, Post}.
4 Experiments and Analysis
We adopt the GLIP-L variant (and its pretrained weights) for all of our experiments, where its visual encoder uses the Swin-L transformer (Liu et al., 2021). We train the GLIP-L with our framework primarily on the SCOD dataset, and perform zero- shot transfer to the TREK-150 task.
(SRL) parses (Shi
4.1 Ego4D SCOD
4.1.1
Experimental Setups
Data Splits. We split the official SCOD train set following a 90-10 train-validation ratio and use the official validation set as our primary test set."
GPT-based. For our main methods leveraging LLMs (GPT) and its generated action-object sym- bolic knowledge, we consider four types of com- binations: (1) GPT with its extracted OUCs and Tools. (2) The model from (1) with additional uti- lization of object defimitions (GPT+Desc.). (3) Similar to (2) but condition on generated pre- and
Evaluation Metrics. Following the original SCOD task's 's main settings, we adopt average precision
'The official test-set only concerns the PNR frame, and deliberately excluded narrations to make vision only local- ization task, which is not exactly suitable for our framework.

---

Pre-Frame"
PNR-Framet
Post-Framef
Base
Type
Method
Objects
AP
AP50
AP75
AP
AP50
AP75
AP
AP50
AP75
OUC
32.73
49.17
34.05
37.49
$7.04
38.59
29.68
44.43
30,94
Vidlntern
Tool
16.39
23.43
17.25
16.53
24,51
17.14
14.03
21.70
14,44
OD
OUC
26.91
42.83
27.86
29.74
47.70
30.47
24.13
38.74
24.71
Zero- on GTs
OUC
20.18
32.97
20.63
19.51
32,34
19.39
19.34
31.07
19.88
Random Fatily
OUC
25.90
26,20
26.85
24.45
39.17
25.10
Full-Instrr
OUC
32.45
51.62
33.34
33.78
54.44
34.49
31.30
49.23
32.42
SRI-ARG1
OUC
36.41
54.93
37.65
38.32
58.07
39.41
33.59
49.99
34.90
OUC
37.87
$6.35
39.55
39.64
40.73
34,97
51.34
36.69
GT-SRL-ARGI
Tool
45,53
71.22
46.27
43,70
68.96
44.54
43.76
69.56
44.04
GLIP-L
GPT
OUC
37.46
56.05
38.96
39.07
59.17
40.13
34.77
51.34
36.35
Instr.
Tool
38.41
60.66
39.33
37. 37.64
60.26
39.29
37.67
59.73
38.24
OUC
36.97
56.16
38.35
38.49
59.38
39.41
34.09
51.18
35,56
GPT+Desc.
Tool
42.26
64.37
44.59
41.30
64.46
43.53
40,20
63.92
41.60
GPT+Conds.
OUC
38.65
57.55
40.16
40.19
60.39
41.56
35.40
52.15
37.11
Tool
43.48
65.78
45.58
42.37
64.97
44.77
41.08
63.26
42.07
wlo obj-mask
OUC
37.59
56.28
39.19
39.09
59.31
40.58
33,93
50.80
35,38
GPT+Conds.+Dess
OUC
38.27
57.79
39.65
39.96
60.91
41.35
35.37
52.82
36.95
Tool
44.00
66.49
46.12
42.77
66.06
44.82
42.12
65.44
42.45
Table Model performance on Ego4D SCOD, OD: pure object detection. Instr: grounding with instructions. We highlight best OUC performance in RED for and best Tool performance in GREEN.
post-conditions of the objects (GPT+Conds.). (4) Combining both (2) and (3) (GPT+Conds.+Desc))
ment on more accurate extractions and search of better suited symbolic knowledge.
4.1.3 Results
Post-Frame? OD Metrie
Method
Top-K
AP50
AP75
Table 4 summarizes the overall model performance Ego4D SCOD task. Even using the ground truth phrases, GLIP's zero-shot performance is signif- icantly worse than pure OD baselines, implying that many of the SCOD objects are uncommon to its original training distribution. Generally, the instruction grounded performance (Instr.) are all better than the pure OD baselines, even with using the whole instruction sentence the grounding phrase. The significant performance gaps between our models and the VidIntern baseline verifies that visual-only models can be much benefit from in- corporation of textual information (should they be available) for the active object grounding task.
Track from GT PNR Track Pred. PNR
1
20.36
41.15
17.78
10.21
21.27
8.63
GPT+Conds
1
29.85
43.53
31.45
Table 5: PNR to Post OUC tracking ablation study, Since tracking module only produce a single box for each frame, we report the top-1 performance of our grounding model. (Normally COCO API reports max 100 detection boxes.)
However, one may raise a natural question: if the OUC/Tool can be more robustly localized in the PNR frame, would tracker improve the post- frame performance over our grounding framework? We thus conduct an ablation study using the tracker in Section 4.2 to track from PNR-frames using ei- ther the ground truth box and our model grounded box to the post-frames. Results in Table 5 contra- dicts this hypothesis, where we find that, due to viewpoint variations and appearance differences induced by the state change, our grounding model is significantly more robust than using tracking.
Particularly for OUC, with vanilla GPT extrac- tions can almost match the performance using the ground truth phrases, where the both the condi- tional and definition symbolic knowledge further improve the performance. Notice that condition knowledge by itself is more useful than the defini- tion, and would perform better when combined. We also ablate row excluding the per-object aggrega- tion mechanism that the conditional knowledge is simply utilized contextualized suffix for an instruction, which indeed performs worse, espe- cially for the post-frames. As implied in Table best performance on Tool is achieved using the ground truth phrases, leaving room for improve-
4.1.4 Qualitative Inspections
Figure 6 shows six different examples for in-depth qualitative inspections. It mainly shows that, gen- erally, when the models grounding with the sym- bolic knowledge outperforms the ones without, the provided symbolic knowledge, especially the con- ditional knowledge, plays an important role

---

OPE-Det"
OPET
Std. OD Metrict
FPV OD
BBox Rank.
Method
SS
NPS
SS
NPS
AP
AP50
AP75
HiC HiC
MDNet MDNet
LTMU-H
0.267
0.261
0.505
0.520
TbyD-H
0.047
0.018
0.433
0.455
Swin-L, DINO
Vidlntern
0.341
0.340
0.526
0.541
29.49
41.47
30.85
Full-Inste
0.355
0.361
0.521
0.537
38.51
60.06
40.17
SRL-ARGI
0.373
0.377
0.528
0.544
40.00
60.52
40.96
GT-SRL-ARGI
0.383
0.390
0.531
0.548
42.35
61.41
44.27
GLIP-L
GLIP-L
GPT
0.379
0.389
0.522
0.545
41.85
61.89
43.46
GeTc 0.402
0,409
0.528
0.543
41.40
60.70
43.17
GPT+Conds. 0.412
0,422
0.541
0.557
45,90
67.26
47.94
GPT+Desc.+Conds.
0.413
0.424
0.539
0.557
43.49
64.34
45.43
Table 6: Model performance on TREK-150. OPE denotes One-Pass Evaluation (Dunnhofer et al., 2022) and OPE-Det is variant OPE where each tracker is initialized with its corresponding object detector prediction the first frame. Success Score (SS) and Normalized Precision Score (NPS) standard tracking metrics.
4.2
TREK-150
rute the HiC-detector for all 8 GLIP-based models and the VideoIntern baseline trained on the SCOD task and perform zero-shot knowledge transfer (directly from Ego4D SCOD).
4.2.1
Experimental Setups
Protocols. 15E'-150's official evaluation pro- tocol is One-Pass Evaluation (OPE), where the tracker is initialized with the ground-truth bound- ing box of the target in the first frame; and then every subsequent frame until the end of the video. Tracking predictions and the ground-truth bounding boxes are compared based on IOU and distance box centers. However, as the premise having ground- bounding box initialization can be generally impractical, a variant of OPE-Det is additionally conducted, where the first-frame bounding box is directly predicted by our trained grounding model (grounded to the instructions). Evaluation Metrics. Following Dunnhofer et al. (2022), we use common tracking metrics, i.e., Suc- cess Score (SS) and Normalized Precision Score (NSP), as the primary evaluation metrics. In ad- dition, we also report standard OD metric (APs) simply viewing each frame to be tracked as the localization task, as an alternative reference.
4.2.3 Results
Table 6 summarizes the performance on TREK-150. Our best GLIP model trained using GPT-extracted objects and symbolic knowledge outperforms the best HiC baseline by over 54% relative gains in the SS metric and over 62% relative gains in the NPS scores for the OPE-D task. also outperforms the VideoIntern baseline by 16-18% relative gains in SS/NPS and even the GLIP-(GT-SRL-ARGI) model by 7-9% relative gains on both metrics. This demonstrates the transferability of our OUC grounding model in fpv-tracking. For the OPE task with ground-trute initializations, the gains provided by our GLIP-GPT models over LTMU-H narrow to 7-8% relative gains across both metrics while still maintaining lead over all other methods. This shows that the model still able to better help the tracker re- re-focus on the OUCs although the overall tracking performance more empirically bounded by the tracking module.
4.2.2 Baselines.
We adopt the best performing framework, the LTMU-H. in the original TREK-150 paper as the major baseline. LTMU-H integrates an fpv (first- person-view) object detector (HiC (Shan et al., 2020b)) into a generic object tracker (LTMU (Dai et al., 2020)), which is able to re-focus the tracker on the active objects after the (tracked) focus is lost (i.e., identified by low tracker confidence scores). Following the convention of utilizing object de- tection models improve tracking (Feichtenhofer et al., 2017), we focus on improving object tracking performance by replacing the HiC-detector with our knowlede-enhanced GLIP models. We substi-
Related Works
Egocentric Vision. Egocentric vision has recently attracted research attentions thanks advance- ments smart wearable devices and robotics. Datasets used this work, Ego4D (Grauman et al., 2022) and Epic-Kitchens (Damen et al., 2022, 2018; Dunnhofer et al., 2022) are two represen- tative large-scale collections of egocentric videos
Mainly because: (1) The general bounding box annota- tions in Epic-Kitchens videos are machine annotated, and (2) we helieve model learned from Ego4D's more general visual domains should transfer well to kitchen activities.

---

(A)
(B)
(C)
Post: Cord wwwn::OOrrutiirrrao Threwdee)
Fest Teant--aanttt.. (Sistrhed, Textured)
Puat Cond: Chet, un?
Cotdl (Shurp, Netalie)
Souieed)
Pre-Kondi (saonch, Flat)
Pour water the trautetowo from 2i00ooo
roshiine the
the carabnard pco wits cuttarowo
aody-
inttecnng
used outting or truening.
(D)
(E)
(F)
Fro-
Fost-Cood:: Cost-Condd:llerddeee 5)
Fon-Frohd (Neat, (leat,
Fre- Con--oooddddrraeee
Post Cons CCemn sred, sred, Gray)
Pru-Cond Oheut. Catl Lod)
20 LTe woll
the bathouo from the shell hand.
grid
biedting udsttence used In conntraction
Nertzencii surtocs used atartng or Staplaying.
Qualitative inspections, mainly on the effectiveness of the GPT generated symbolic knowledge. Bounding box color code: Ground truth boxes, Models with uses of symbolic knowledge (MD-1), Le.the GPT+Conds,+Dese, Models without uses of symbolic knowledge (MD-2), Le., the vanilla GPT.
Figure 6:
recording tasks performed by the camera wearers. Other existing works have also investigated ego- centric vision in audio-visual learning (Kazakos et al., 2019), object detection with EgoNet (Berta- sius et al., 2017; Furmari et al., 2017), object seg- mentation with eye-gazes (Kirillov et al., 2023) and videos (Darkhalil et al., 2022).
Multimodal Grounding. In this work, we adopt the GLIP model (Li* et al., 2022; Zhang* et al., 2022) for its compatibility with our settings and the joint inference framework, which indeed demon- strate significant improvements for the active object grounding task. There are many related works for multimodal grounding and/or leveraging language (LLMs) to help with vision tasks, including (but not limited to) Grounding-DDie DINO (Liu et al., 2023), DQ-DETR (Liu et al., 2022), ELEVATER (Li* et al., 2022), phrase segmentation (Zou* al., 2022), visually-enhanced grounding (Yang et al.. 2023), video-to-text grounding (Zhou et al., 2023). LLM-enhanced zero-shot novel object classifica- tion (Naeem et al., 2023), and multimodal object description generations (L.i et al., 2022, 2023).
Action-Object Knowledge. The knowledge of ob- jects are often at the center of understanding human actions. Prior works in both NLP and vision com- munities, have studied problems such as tracking visual object state changes (Alayrac et al., 2017; Isola et al., 2015; Yang et al., 2022), understanding object manipulatiins and affordances (Shan et al.. 2020a; Fang et al., 2018) 8), tracking textual entity state changes (Branavan et al., 2012; Bosselut et al.. 2018; Mishra al., 2018 8; Tandon et al., 2020), and understanding textual pre-tpost-condittion from ac- tion instructions (Wu al., 2023). While hand- object interactions (Shan et al., 2020a: Fu et al., 2022) are perhaps one of the most common ob- ject manipulation schemes, the objects undergoing change may not be directly in contact with the hands (see Figure 2). Here additional textual in- formation can aid disambiguating the active object during localization and tracking. In this spirit, our work marries the merits from both modalities to tackle the active object grounding problem accord- ing to specific task instructions, and utilize action- object knowledge to further improve the models.
6 Conclusions
In this work, we approach the active object ground- ing task leveraging two narrated egocentric video datasets, Ego4D and Spic-KKicheee We propose carefully designed prompting scheme obtain useful action-object knowledge from LLMs (GPT), with specific focuses on object pre-/post during an action and its attributional descriptions. Enriching the GLIP model with the aforemen- tioned knowledge as well as the proposed per- object knowledge aggregation technique, our mod- els outperforms various strong baselines in both active object localization and tracking tasks.

---

7 Limitations
useful knowledge.
We hereby discuss the potential limitations of our work:
8 Ethics and Broader Impacts
We hereby acknowledge that all of the co-authors of this work are aware of the provided ACL Code of Ethics and honor the code of conduct. This work is mainly about understanding and localizing the key objects undergoing major state changes during human performing an instructed or guided action, which is crucial step for application such on- device Al assistant for AR or VR devices/glasser
(1) While we make our best endeavours to engi- neer comprehensive and appropriate prompts for obtaining essential symbolic action-object knowl- edge from large language models (LLMs) such as GPT, there still few cases where the extracted objects are not ideal (see Table 1). Hence, our model performance could potentially be bounded by such limitation inherited from the LLM ability to fully and accurately comprehend the provided in- structions, Future works can explore whether more sophisticated in-context learning (by providing ex- amples that could be tricky to the LLM) would be able alleviate this issue. Alternatively, may utilize L.M-self-conssruutee datasets to fine- tune another strong language models (such as Al- paca (Taori et al., 2023)) for the object extraction task. On the other hand, incorporating high-level descriptions of the visual contexts using off-the- shelf captioning models could also be explored to make the LLM more situated to further improve the efficacy of the extracted knowledge.
Datasets. We mainly use the state-change- object-detection (SCOD) subtask provided by Lgo4D (Grauman et al., 2022) grand challenge for training and (testing on their own tasks), and transfer the learned knowledge to TREK- 150 (Dunnhofer et al., 2022) from Epic-Kitchens dataset (Damen et al., 2018). These datasets is not created to have intended biases towards any population where the videos are collected span- ning across multiple regions around the world. The main knowledge learned from the dataset is mainly physical knowledge, which should be generally applicable to any social groups when conducting daily tasks.
(2) As this work focuses on action frames first-person egocentric videos (from both Ego4D (Grauman et al., 2022) and Epic-Kitchens dataset (Damen et al., 2018; Dunnhofer et al., 2022) datasets), the underlying learned model would obvi- ously perform better on visual observations/scenes from first-person viewpoints. While we hypothe- size that, unless heavy occlusion and drastic do- main shifts (of the performed tasks and/or ob- jects involved) occur, the learned models in this work should be able to transfer to third-person viewpoints, we have not fully tested and verified such hypothesis. However, if applied properly. the overall framework as well as the utilization of LLMs for action-object knowledge should be well- vell-generalizable regardless of the viewpoints.
Techniques. We propose to leverage both the symbolic knowledge from large-language mod- els (LLMs) such as GPT to guide the multimodal grounding model and the visual appearances and relations among objects for localizing the object undergo changes. The technique should be gener- ally transferable to similar tasks even outside of the domain concerned in this work, and unless misused intentionally to harmful data (or trained on), should not contain harmful information for its predictions and usages.
Acknowledgments
Many thanks to I-Hung Hsu for his valuable feed- back during our paper preparation, and Liunian Harold Li for his tips on using his great work. GLIP models. This material is based on research supported by the Machine Common Sense (MCS) program under Cooperative Agreement N66001- 19-2-4032 and the ECOLE program under Cooper- ative Agreement HR001 12390060, both with the US Defense Advanced Research Projects Agency (DARPA). The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing DARPA, or the U.S. Government.
(3) There is more object- and action-relevant knowledge that could be obtained from LLMs, such as spatial relations among the objects, size differ- ence between the objects, and other subtle geomet- rical transitions of the objects. During experiments, we attempted to incorporate spatial and size in- formation our models. However, experimental results on the given datasets did not show signifi- cant improvement. Thus we omitted them from this work. We hope to inspire future relevant research along this line to further exploit other potentially

---

References
object tracking in first person vision. In International Journal of Computer Vision (IJCV).
Jean-Baptiste Alayrac, Ivan Laptev, Josef Sivie, and
Simon Lacoste-Julien 2017. Joint discovery of ob- ject states and manipulation actions, In International Conference on Computer Vision (ICCV).
Kuan Fang, Te-Lin Wu, Damiel Yang. Silvio Savarese, and Joseph J. Lim. 2018. Demo2vec: Reasoning object affordances from online videos. In Proceed- ings of the IEEE Conference on Computer Vision and Pottern Recognition (CVPR).
Gedas Bertasius, Hyun Soo Park, Stella X Yu, and Jianbo Shi. 2017. First person action-object detec- tion with egonet. Robotics: Science and Systems (RSS).
Christoph Feichtenhofer, Axel Pinz, and Andrew Zisser- man. 2017. Detect to track and track to detect. In In- ternational Conference on Computer Vision (ICCV).
Antoine Bosselut, Omer Levy, Ari Holtzman, Corin En- nis, Dieter Fox, and Yejin Choi. 2018. Simulating action dynamics with neural process networks. In In- ternational Conference on Learning Representations (ICLR).
Qichen Fu, Xingyu Liu, and Kris Kitani. 2022. Sequen- tial voting with relational box fields for active object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
S.R.K. Branavan, Nate Kushman, Tao Lei, and Regina Barzilay. 2012. Learning high-level planning from text. In Association for Computational Linguistics (ACL).
Antonino Furnari, Sebastiano Battiato, Kristen Grau- man, and Giovanni Maria Farinella. 2017. Next- active-object prediction from egocentric videos. vol- ume 49, pages 401-411. Elsevier.
Guo Chen. Sen Xing, Zhe Chen, Yi Wang, Kunchang Li, Yizhuo Li, Yi Liu, Jiahao Wang. Yin-Dong Zheng,
Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord. Pradeep Dasigi, Nelson F. Liu, Matthew Peters, Michael Schmitz, and Luke S. Zetrlemoyer. 2017. Allennlp: A deep semantic natural language processing platform.
Bingkun Huang, et al. 2022. Internvideo-eoo4d:: pack of champion solutions to ego4d challenges. arXiv preprint arXiv:2211.0955
Kenan Dai, Yunhua Zhang, Dong Wang, Jianhua Li, Huchuan Lu. and Xiaoyun Yang. 2020. High- performance long-term tracking with meta-updater. 2020 IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6297-6306.
Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, Miguel Martin, Tushar Nagarajan, Iija Ra- dosavovic, Santhosh Kumar Ramakrishnan, Fiona Ryan, Jayant Sharma, Michael Wray, Mengmeng Xu. Eric Zhongcong Xu, Chen Zhao, Siddhant Bansal, Dhruv Batra. Vincent Cartillier, Sean Crane, Tien Do, Morrie Doulaty, Akshay Erapalli, Christoph Feicht- enhofer, Adriano Fragomeni, Qichen Fu, Christian Fuegen, Abrham Gebreselasie, Cristina Gonzalez, James Hillis, Xuhua Huang, Yifci Huang, Wenqi Jia, Weslie Khoo, Jachym Kolar, Satwik Kottur, Anurag Kumar, Federico Landini, Chao Li, Yanghao Li. Zhenqiang Li, Karttikeya Mangalam, Raghava Modhugu, Jonathan Munro, Tullie Murrell, Takumi Nishiyasu, Will Price, Paola Ruiz Puentes, Merey Ra- mazanova, Leda Sari, Kiran Somasundaram, Audrey Southerland, Yusuke Sugano, Ruijie Tao, Minh Vo, Yuchen Wang, Xindi Wu, Takuma Yagi. Yunyi Zhu, Pablo Arbelaez, David Crandall, Dima Damen, Gio- vanni Maria Farinella, Bernard Ghanem, Vansi Kr- ishna Ithapu, C. V. Jawahar, Hanbyul Joo, Kris Ki- tani, Haizhou Li, Richard Newcombe, Aude Oliva, Hyun Soo Park, James M. Rehg, Yoichi Sato, Jianbo Shi, Mike Zheng Shou, Antonio Torralba, Lorenzo Torresani, Mingfei Yan, and Jitendra Malik. 2022. Ego4d: Around the World in 3,000 Hours of Ego- centric Video. In Proceedings of the IEEE Confer- ence on Computer Vision and Pattern Recognition (CVPR).
Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. 2018. Scaling egocen- tric vision: The epic-kitchen, dataset. In European Conference on Computer Vision (ECCV).
Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Jian Ma, Evangelos Kazakos, Da- vide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. 2022. Rescaling egocentric vision: Collection, pipeline and challenges for epic- kitchens- 100. International Journal of Computer Vision (IJCV).
Ahmad Darkhalil, Dandan Shan, Bin Zhu, Jian Ma, Am- lan Kar, Richard Higgins, Sanja Fidler, David Fouhey, and Dima Damen. 2022. Epic-kine kitchens visor bench- mark: Video segmentations and object relations. In Proceedings of the Neural Information Processing Systems (NeurIPS) Track on Datasets and Bench- marks.
Matteo Dunnhofer, Antonino Furnari, Giovanni Maria Farinella, and Christian Micheloni. 2021. Is first person vision challenging for object tracking? In Proceedings of the IEEF/CVF International Confer- ence on Computer Vision (ICCV) Workshops.
Matthew Honnibal and Ines Montani. 2017. spaCy 2: Natural language understanding with Bloom embed- dings, convolutional neural networks and incremental parsing. To appear.
Matteo Dunnhofer, Antonino Furnari, Giovanni Maria Farinella, and Christian Micheloni. 2022. Visual

---

Phillip Isola, Joseph J. Lim, and Edward Adelson. 2015. Discovering states and transformations in im- age collections. In Proceedings of the IEEF Confer- ence on Computer Vision and Pattern Recognition (CVPR).
Shilong Liu, Yaoyuan Liang, Feng Li, Shijia Huang. Hao Zhang. Hang Su, Jun Zhu, and Lei Zhang. 2022. Dq-detr: Dual query detection transformer for phrase extraction and grounding. In Association for the Advancement of Artificial Intelligence (AAAI).
Evangelos Kazakos, Arsha Nagrani, Andrew Zisser- man, and Dima Damen. 2019. Epic-fusion: Audio- visual temporal binding for egocentric action recog- nition. In International Conference on Computer Vision (ICCV).
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang. Hang Su, Jun Zhu, et al. 2023. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint 2xXi:2303.00490
Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR).
Ze Liu, Yutong Lin, Yue Cao. Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021. Swin transformer: Hierarchical vision transformer using shifted windows. In International Conference Compuler Vision (ICCV).
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao. Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg. Wan-Yen Lo, Piotr Dollár, and Ross Girshick. 2023. Segment anything. 2344::230..06643.
Bhavana Dalvi Mishra, Lifu Huang, Niket Tandon, Wen-tau Yih, and Peter Clark. 2018. Tracking state changes in procedural text: a challenge dataset and models for process paragraph comprehension. In North American Chapter of the Association for Com- putational Linguistics (NAACL-HLT).
Ivan Krasin, Tom Duerig, Neil Alldrin, Vittorio Fer- rari, Sami Abujai--Hija, Alina Kuznetsova, Hassan Rom, Jasper Uijlings, Stefan Popov. Andreas Veit, al. 2017. Openimages: A public dataset for large- scale multi-label and multi-class image classification. Dataset available from https://github con/openim- ages.
Muhammad Ferjad Nacem, Muhammad Gul Zain Ali Khan, Yongqin Xian, Muhammad Zeshan Afzal, Di- dier Stricker, Lue Van Gool, and Federico Tombari. 2023. 12mvformer: Large language model generated multi-view document supervision for zero-shot image classification. In Proceedings of the IEEE Confer- ence on Computer Vision and Pattern Recognition (CVPR).
Ranjay Krishma, Yuke Zhu, Oliver Groth, Justin John- son, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. 2017. Visual genome: Connecting language and vi- sion using crowdsourced dense image annotations. In International Journal of Computer Vision (IJCV).
OpenAI. 2023. Chatgpt.
Chunyuan Li Haotian Liu*, Liunian Harold Li, Pengchuan Zhang, Jyoti Aneja, Jianwei Yang. Ping Jin, Yong Jae Lee, Houdong Hu, Zicheng Liu, et al. 2022. Elevater: benchmark and toolkit for evalu- ating manguage-augmente visual models. In Track on Datasets and Benchmarks, Neural Information Processing Systems (NeurIPS).
Vicente Ordonez, Girish Kulkarni, and Tamara Berg. 2011. Im2text: Describing images using million captioned photographs.
Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo. Julia Hockenmaier, and Svetlana Lazebnik. 2015. Flickr30k entities: Collecting region-to-phrass correspondences for richer image- to to-sentence models. In International Conference on Computer Vision (ICCV).
Dongxu Li. Junnan Li, Hung Le, Guangsen Wang, Sil- vio Savarese, and Steven C. H. Hoi. 2022. Lavis: A library for language-vision intelligence.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large lan- guage models. arXiv preprini arXiv:2301. 12597..
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. arXiv preprint 1xXiv:2103.0002:
Liunian Harold Li*, Pengchuan Zhang Haotian Zhang*, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang. Lu Yuan, Lei Zhang, Jenq-Neng Hwang, Kai- Wei Chang, and Jianfeng Gao. 2022. Grounded language-ing -mage pre-training, In Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Dandan Shan, Jiaqi Geng, Michelle Shu, and David Fouhey. 2020a. Understanding human hands in con- tact at internet scale. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recog- nition (CVPR).
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In European Confer- ence on Computer Vision (ECCV).
Dandan Shan, Jiaqi Geng, Michelle Shu, and David F. Fouhey. 2020b. Understanding human hands in con- tact at internet scale. pages 9866-9875

---

Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng. Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. 2019. Objects365: A large-scale, high-quality dataset for object detection. In International Confer- ence on Computer Vision (ICCV).
Yu Zhou, Sha Li, Li Manling, Lin Xudong. Shih-Fu Chang, Mohit Bansal, and Heng Ji. 2023. Non- sequential graph script induction via multimedia grounding. In Proc. the 61th Anmual Meeting of the Association for Computational Linguistics (ACI2023).
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic im- age captioning. In Association for Computational Linguistics (ACL), pages 2556-2565.
Xueyan Zou 7i-Yi Dou Jianwei Yang*, Zhe Gan, Linjie Chunyuan Li, Xiyang Dai, Jianfeng Wang. Lu Yuan, Nanyun Peng. Lijuan Wang, Yong Jae Lee". and Jianfeng Gao 2022. Generalized decoding for pixel, image and language. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Peng Shi and Jimmy Lin. 2019. Simple hert models for relation extraction and semantic role labeling. ArXiv, abs/1904.05255.
Niket Tandon, Keisuke Sakaguchi, Bhavana Dalvi, Dheeraj Rajagopal, Peter Clark, Michal Guerquin, Kyle Richardson, and Eduard Hovy. 2020. A dataset for tracking entities in open domain procedural text. Empirical Methods in Natural Language Process- ing (EMNLP), pages 6408-6415
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori Hashimoto. 2023. Alpaca: strong, replicable instruction-following model. Stan- ford Center for Research Foundation Models. 3/aapaaa html,
https//orfmm stanford. 3(6)77.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistae, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jemite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transform- ers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing: System Demonstrations, pages 38 45, Online. Association for Computational Linguistics.
Te-Lin Wu, Caiqi Zhang, Qingyuan Hu, Alex Spangher, and Nanyun Peng. 2023. Learning action condi- tions from instructional manuals for instruction un- derstanding.
Guang Yang. Manling Li, Jiajie Zhang, Xudong Lin, Shih-Fu Chang, and Heng Ji. 2022. Video event extraction via tracking visual states of arguments. In AAAI Conference Artificial Intelligence.
Ziyan Yang, Kushal Kafle, Franck Dernoncourt, and Vi- cente Ordonez. 2023. Improving visual grounding by encouraging consistent gradient-based explanations. In Proceedings of the IEEE Conference on Computer Vision and Pallern Recognition (CVPR).
Haotian Zhang* Pengchuan Zhang* Xiaowei Hu, Yen- Chun Chen, Liunian Harold Li, Xiyang Dai, Lijuan Wang, Lu Yuan, Jenq-Neng Hwang, and Jianfeng Gao. 2022. Glipv2: Unifying localization and vision- language understanding. In Neural Information Pro- cessing Systems (NeurlPS).

---

A
GPT Prompt Engineering Details
future queries will ignore that entity.
Due to GPT's tendency to paraphrase and pro- vide unwanted chain-of-thought explanations, we further pass GPT outputs for OUC and/or tool through an additional query aimed to produce ex- act grounding if it is not contained in the caption. Emperically this query has very high chance of producing
A.1
GPT Prompts
Below the prompts used in the GPT entity and symbolic knowledge extraction pipeline de- scribed in Figure
GPT Prompt Set
crattonngrouddbble answer if exist. In the unlikely event that no groundable OUC is found after running GPT Prompt 2, we resort to using SRL [ARGI] as the OUC. On the other hand, no groundable Tool is found after running GPT Prompt 2, we set the Tool to None.
a Identify OUC Tool:
GPT Prompt 1 = "From the action [O],
please extract the follow objects
exactly as they are written: (a) The
single object being manipulated (b)
The one single tool used to manipulate (a). Please do not explain anything in you answer Please only output one object for each part. If (a) or (b), please simply output [
At this point, the OUC and Tool (if exist). GPT results parsed and stored if following the designated formats.
GPT Prompts 3 and 4 on
cannot find either
None] for that field and print nothing else."
A.3
GPT Situated Role
# Precise Semantic Grounding (for both OUC and Tool):
For small randomly selected subset of Ego4d prompts (220 samples) in the human evaluation of GPT generated answers for pre/post conditions (Table 2), we evaluated results both with and with- out the situated role specification "From the first- person view." " in Table 7.
CPT Prompt 2 "Please output exact subpart of the sentence [[}] where the object [()] is referred to. Please remove leading verbs in your answer Please output only the exact sentence subpart and nothing else. Please make sure that you answer exactly found in the sentence [O]. If question please simply output [None] and nothing else. Please do not output any explaination
not valid,
Prompt Style
Pre Cond.
Post Cond,
Textual
Visual
Textual
Visual
wof Situated Role w/ Situated Role
70.9
75.0
61.4
64.5
your answer.
86.4
82.3
75.5
71.8
g State Change Forcast
GPT Prompt 3 "From the first person
Table 7: GPT prompt comparison on Ego4D sub- set. Here "Textual" refers to "textual correctness": i.e. "based on text and common sense alone, does the GPT cdess./desc. make sense?": whereas "Visual" refers to "visual correctness": i.c. "does the GPT conds./dese. match what is shown in the image?"
view of the action [(], would the object [O] undergo significant change in its visual appearance? If so, please output [yes] in the first line and If not, please output [no] in the first line. If the answer is yes, then on the second line, please simply print one or two visually recognizable adjectives to describe the appearance of the object before the state change, on the third line, please simply print a few (<7) words that visually describe the object after the state change"
We observed that when provided with additional situated viewpoint specification, GPT results dis- played a significant increase in alignment with hu- man judgment, thus it was incorporated into our final prompt.
B Details of Modeling & Learning
# Object Description
CPT Prompt 4 = II one sentence, please define the object [()] visually"
B.1 Narration Processing
Ego4D. Ego4D's original annotated narrations are of third-person deseriptive tone, e.g., "#C cuts the vegetables with the knife.", where the symbol of #C indicates the camera wearer. Since this pattern universally applicable all the available nar- rations of the vidcos, we perform a deterministic string transformation stripping these special indi-
A.2 GPT Pipeline
The GPT prompts are used in the order above. First, we plug the instructional caption to GPT Prompt and generate a tentative OUC-Tool pair. If either entity is not found, GPT would return None and

---

cators of camera wearers other human charac- ters, followed by simple conversion change the third-person singular verb to first-person verb, The rest of the pronouns are also deterministically con- verted, so that the narration becomes a first-person imperative tone. Although we did not empirically find such transformation make any significant im- pacts to the model performance, we conduct this transformation for GPT to more easily grasp the situated roles (given it) as well as conform the main motivation of this work better.
hyperparameters used in Table 8. The basic hyper- parameters such as learning rale, batch size, and gradient accumulation steps, are kept consistent for models based off the same architecture (for base- lines and our GLIP-L model series). All of our models adopt the same search bounds and ranges of trials as in Table 9.
Epie-Kitchens. For TREK-150 where its videos basically belong to Epic-Kitchens, the original re- leased narrations are already in an imperative in- structional tone. In light of this, we simply adopt them as the task instructions seamlessly without needing to perform any modifications.
B.2 Training & Implementation Details
Training Details. For GLIP series of models, simply truncate the textual imputs (the action in- struction texts) at maximum 256 tokens, ensuring all of the textual inputs are below such maximum length even with additional symbolic knowledge is incorporated. The hyperparameters are manually tuned against an Ego4D SCOD validation set, and the checkpoints used for testing are selected by the best performing ones such set.
All the models in this work trained on 2- 4 Nvidia A100/A6000 GPUs on a Ubuntu 20.04.2 operating system. The hyperparameters for each model are manually tuned against the de- velopment datasets, and the checkpoints used for testing are selected by the best performing ones on such held-out development sets.
The implementations of the transformer-based models extended from the HuggingFace! code base (Wolf et al., 2020), and our entire code-base is implemented in Py- Torch.
Implementation Details.
B.3 Hyperparameters
We train our models until performance convergence is observed on the held-out development set (split from the original train set from Ego4D SCOD sub- task). The training time is roughly 12-14 hours, spanning 10-15 training epochs. We list all the
"hnps:/wwwwnnviiiaco http::/wwwwwiiiia.ooooooooo//aaa a6000/ 19 #hinss///stttboooommmgiggghh//ns //mnps//tyyorchhorrg/

---

Gradient mulation Steps
Models
Batch Size
Initial LR
# TTraining Epochs
Params
Vidcolntem Basclines
10-3
218M
GLIP-L Series
429M
Table 8: Hyperparameters in this work: Initial LR denotes the initial learning rate. All the models are trained with Adam optimizers (Kingma and Ba, 2015). include number learnahle parameters of each model the column params.
Type
Batch Size
Initial LR
Training Epochs
Gradient Accumulation Steps
Bound (lower-upper)
2-8
1 x 10 3 1x 10
5-15
Number of Trials
2-3
2-4
Table Search bounds for the hyperparameters of all the models.


Reviews:
[{"review_id": "bqGau4Mkr5", "reviewer": "Reviewer_MyHn", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are thankful to Reviewer MyHn for the insightful remarks! We are encouraged that you find **our paper well-written and easy to follow**, **our baselines appropriate**, and **our analysis detailed** while **opening up multiple areas for future research**!\n\nPlease find your suggestions and concerns addressed below:\n\n---\n\n**[W1] Visual failure cases**\n\nWe absolutely agree with the reviewer on the importance of analyzing visual failure cases, which is precisely why we provide a **comprehensive set of qualitative examples** in Figure 6 under Appendix B.1 (not in the main paper due to space constraints).\nIn Figure 6 (C) and (F), we observe failure cases where our method correctly localizes part of the OUC but not its whole, and hence fails to align perfectly with the ground truth bounding boxes.\nThis type of failure can be mostly attributed to the object detector having difficulty in identifying the OUCs’ differently colored/structured regions.\n\nOther failure cases presented in Figure 6 mostly occur when GPT extracted symbolic knowledge is suboptimal (as analyzed in Table 1), or when drastic shifts in object views (drastically shifted from the objects’ normal views) or lighting conditions of objects occur.\nGiven more space for our final version, we will include detailed qualitative analysis of each failure case in the main paper.\n\n---\n\n**[W2] OUC exiting egocentric view**\n\nWe are happy that the reviewer brings up the issue of object occlusion, which is something we have also observed and carefully examined during our experiments!\nDue to the space limitations, we list our key findings below for your reference and will add them as further details in our final version:\n\n- **Ego4d SCOD**: Pre/PNR/Post frames have considerably low likelihood of exhibiting object occlusion. However, if the object to be grounded is out of the PoV, the GLIP predicted object logits would almost certainly be **lower than a pre-set objectness threshold** (due to the learning process), and hence lead to a ***“no-prediction”***.\n\n- **TREK-150 (Epic-Kitchens)** videos do exhibit the phenomenon of the OUC going “in-and-out” of the egocentric PoV, resulting in *partial occlusion* and/or *full occlusion* frames where no ground truth annotations for the OUC are provided.\nSuch frames are excluded from the final evaluation.\nOur model is very successful in predicting the objects when they come back due to the robustness of our symbolic joint inference grounding mechanism.\n\n---\n\n\n**[W3] Ablation on GPT-prompts**\n\nThanks to the reviewer for bringing up this question! We would like to first provide some context for better understanding the performance gap between our GPT-only model and the GPT+symbolic knowledge model: In terms of AP50, the performance gain for OUC in TREK-150 is 2.45 across all frames, the corresponding gain for OUC and Tool in Ego4D SCOD across all frames are 1.53~1.74 and 5.71～5.83, respectively. Judging from the fact that the results are obtained from ~38K video frames for Ego4D SCOD and ~90K frames from TREK-150, the performance gain on AP-style metrics should be quite significant for the OUC, and even more so for Tool.\n\nFurthermore, our method is not limited to the existing datasets/works, where we aim at devising a generally applicable framework that future works can utilize, perhaps on more challenging active object grounding datasets, where the objects potentially go through more drastic state changes.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is written well and easy to follow.\n- The results and analysis are very detailed, and the chosen baselines for comparison seem appropriate.\n- The authors provide a good amount of detail on limitations of their approach, opening up multiple areas for future study.", "reasons_to_reject": "Some comments - \n\n- Visual failure cases when the OUC does not to get detected would be nice to see.\n- Cases when the OUC goes out of the egocentric view don't seem to be considered. For instance, if the object being spoken about in the instruction moves out of the video for a while and comes back, it is not clear what the GLIP output would be like.\n- While the ablation on GPT-prompts is interesting, the results for all of them seem close to each other. The inference seems unnecessary.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper approaches the \"active target grounding task\", which involves identifying and grounding the object undergoing change in a sequence of images or video. The authors propose a two stage approach, in using ChatGPT to first identify object mentions from the instructions of a video, and then grounding it using GLIP. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Z2JkPRDkQM", "reviewer": "Reviewer_5jwp", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer 5jwp for the constructive feedback! Please find your suggestions and concerns addressed in detail below:\n\n---\n\n**[W1] Comparisons with existing works [1,2,3]**\n\nWe carefully reviewed the suggested related works and will hereby discuss their relations to our paper.\nWe would like to firstly clarify a potential misunderstanding of the term *”active object”* and our newly-defined task *”active object grounding”*.\nIn our paper (**L124-131 and L160-165**), we clearly define the task **active object grounding** as automatically localizing and tracking **object(s) undergoing state change (OUC)** (objects being passively manipulated by the agent) and **Tool(s)** used to enable **an instructed manipulation**. OUCs and Tools are in turn defined as *“active objects”*, which are not directly named and thus need to be extracted from the task instructions.\n\n- **Paper [1]**, while titled *“active object detection”*, is actually discussing the problem of trying to detect a **named object** via an ***\"active\"* mobile agent traversing a 3d environment** and taking pictures of the object from *different angles*. Here the word *“active”* is referring to the mobile nature of the agent and not the object. Unlike our work, here the agent is a mere observer and not in the view. The objects are not going through any form of state change.\n\n- **Paper [2]** also has *“active object detection”* in its title but is actually a work on trying to improve generic **named object detection** (on the Pascal VOC dataset) by **iteratively cropping the image** in an ***“active”* manner**. **Again here *“active”* does not refer to the object itself** but rather describes the authors’ method to improve a completely different task. Here the images are not egocentric and there is no agent involved. Similar to Paper [1], the objects here are not going through any form of state change.\n\n- **Paper [3]** defines *“active objects”* as objects **directly in contact** with the human hand. Recall from above that our definition of *“active object”* is *the object undergoing state change due to an instructed action*, and hence this object **is often not directly in contact with human hands**. The goal of Paper [3]’s task is to exploit the human hands’ role in providing location hints for object detection and dealing with partial object occlusion caused by hands blocking the camera's direct view of the object, which is completely different from our task’s goal. Nevertheless, we conduct an additional experiment by applying this work to our Ego4D SCOD active object grounding task (on OUC localization only). Within the short author response period, we have **reached out to the original authors** and **made our best efforts to re-implement** Paper [3]’s model. We worked closely with them to **fairly re-train and evaluate their model** on our dataset. In this process, **we provided Paper [3]’s framework with the annotated *ground truth* hand bounding boxes** (which already gives it an additional advantage over our method). We report its corresponding performances below:\n\n- **Overall Performance (%) on Ego4D SCOD**\n| **Pre** | **Pre** | **Pre** | **PNR** | **PNR** | **PNR** | **Post** | **Post** | **Post** |\n| :-----: | :-----: | :-----: | :-----: | :-----: | :-----: | :-----: | :-----: | :-----: | \n| **AP** | **AP50** | **AP75**  | **AP** | **AP50** | **AP75** | **AP** | **AP50** | **AP75** |\n| 0.008 | 0.021 | 0.000 | 0.198 | 0.330 | 0.330 | 0.005 | 0.026 | 0.001 |\n\nTo reflect the performance shown above, we additionally analyze the IOUs between the predicted bounding boxes and the ground truth ones, for their mean/std and histogram statistics, which are reported below:\n\n- **Mean and Std of IOUs**\n| **Pre** | **Pre** | **PNR** | **PNR** | **Post** | **Post** |\n| :-----: | :-----: | :-----: | :-----: | :-----: | :-----: |\n| **Mean** | **Std** | **Mean**  | **Std** | **Mean** | **Std** |\n| 0.0484 | 0.0976 | 0.0488 | 0.0993 | 0.0478 | 0.0967 |\n\n- **IOU Histogram (%)**\n| **0.0-0.1**| **0.1-0.2** | **0.2-0.3** | **0.3-0.4** | **0.4-0.5** | **0.5-0.6** | **0.6-0.7** | **0.7-0.8** | **0.8-0.9** | **0.9-1.0** |\n| :-----: | :-----: | :-----: | :-----: | :-----: | :-----: | :-----: | :-----: | :-----: | :-----: |\n| 83.16 | 8.64 | 4.31 | 2.09 | 1.21 | 0.36 | 0.21 | 0.07 | 0.02 | 0.00 |\n\nAbove results show that the performance of Paper [3]’s framework is **far below our lowest performing visual-only baseline** (*i.e.*, VidIntern) on the *active object grounding* task (as defined in our work) especially for Pre and Post frames. The low performance can mostly be attributed to the model’s **fundamentally different objective** to detect *the hand-held object* instead of the actual *object undergoing state change*. This result further highlights the difficulty and unique nature of our proposed task.\n\n---\n\n**[W2] Comparisons with the VidIntern baseline**\n\nWhile we agree that VidIntern does not utilize any caption information, this is **precisely one of the motivations of our work (see L61-65 in the Introduction)** – **whether incorporating textual information can significantly benefit the *active object grounding* task**. This is a very practical setting for many real-world applications (*e.g.*, scenarios depicted in Introduction L41-56) where the textual task instructions are readily available (or uttered). \n\nIt’s important to note that both datasets we experimented on provide such textual information to be used at inference time, *i.e.*, we did not collect or provide any additional annotations for our model. Thus, we believe that our comparison with VidIntern (as our strongest vision-only baseline) is indeed fair as **both frameworks are restricted to training only on dataset-designated resources**. \n\nOur experiments successfully demonstrated that enhancing vision-only object detection with multimodal joint inference can yield significant performance improvements on the Ego4D SCOD task.\nThis result reinforces the emerging research direction of augmenting vision-only tasks with textual information, where effectively utilizing multiple available modalities to learn a stronger model compared to its unimodal counterparts are also well-studied in [4, 5, 6, 7].\n\nRegarding concerns related to evaluating Pre/Post frames in addition to PNR frames, we would like to note that the Ego4D SCOD Track provides official train/val data for all frames including Pre/PNR/Post.\nThe recorded performance of VidIntern on Pre/Post frames is its best performance after being trained on all frames just like our models, thus making certain that our comparison is fair.\n\nWhile our model outperforms VidIntern on PNR frames by a significant amount (3.87 AP50), the performance gap further increases for Pre and Post frames. This can be attributed to the fact that in PNR frames the OUCs are closer to the human actors (and their hands), making it easier for pure vision models to leverage this spurious correlation and disambiguate the OUC without actually understanding which object is undergoing state change.\n\nLastly, GLIP [8, 9] itself is one of the state-of-the-art phrase/object  grounding (*requires knowing which phrase(s) to ground **beforehand***) and open-vocabulary object detection models in public.\nAll our model variants without using the symbolic knowledge in Table 3, are **exactly baseline comparisons with the generic GLIP**, only that we ablate the **automated** *phrase-to-ground generation* with our proposed GPT extractions against heuristic-based counterparts, such as directly using the full task instructions (Full-Instr.) and semantic-role labeling (SRL).\nWe even additionally compare our full framework with the ground truth OUC phrases obtained from both datasets, where our framework still performs generally better.\n\n[4] Plummer, Bryan A., Matthew Brown, and Svetlana Lazebnik. \"Enhancing video summarization via vision-language embedding.\" *CVPR* 2017\n\n[5] Hessel, Jack, et al. \"A case study on combining asr and visual features for generating instructional video captions.\" *CoNLL* 2019\n\n[6] Chen and Lin et al, “Joint Multimedia Event Extraction from Video and Article” *EMNLP* 2021\n\n[7] Peng, Xiaokang, et al. \"Balanced multimodal learning via on-the-fly gradient modulation.\" *CVPR* 2022\n\n[8] Li, Liunian Harold, et al. \"Grounded language-image pre-training.\" *CVPR* 2022\n\n[9] Zhang, Haotian, et al. \"Glipv2: Unifying localization and vision-language understanding.\" *NeurIPS* 2022\n\n---\n\n**[W3] Details of TREK-150**\n\nThank you for bringing up this point, we would like to take this opportunity to clarify that TREK-150 is an **“object tracking”** dataset adapted to our setting (adaptation details in Sec 4.2 L436-L455), where given an instructed action, the model is required to ground and track the OUC throughout all frames in the egocentric video (L166-177). Therefore, unlike the Ego4D SCOD dataset, it **does not contain any defined Pre/PNR/Post frames**. Since our model is trained to perform joint inference and autonomously decide which of the pre- and post-conditions to weigh more based on the frame image and instructed action, **frame-type information is not required for our inference**. \n\nWe will be sure to include this clarification in the final version of our paper. Also please feel free to let us know if you have any further questions, we will be happy to answer in detail and add them to the final version of our paper! We believe these revisions can be easily performed in the camera-ready version and hope you may kindly re-evaluate this point as a reason to reject our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.propose a  prompting pipeline that integrates the knowledge about the objects undergoing state-change from LLM\n2.evaluate the proposed framework on two egocentric video datasets", "reasons_to_reject": "1. The paper lacks comparison with existing active object detection methods, like[1,2,3] in  missing references or other object grounding methods\n2. The comparisons with VidIntern is not fair. For the objects undergoing state-change detection, VidIntern does not take any caption or instruction as inputs, meanwhile, it detect the active object on the PNR not including Pre and Post frames.\n3.The details of the epic-kitchen(trek-150) is not sufficient, such as the definition of pre and pose frames.", "questions_to_authors": "", "paper_topic_and_main_contributions": "**Topic:** This paper aims to address the active object grounding, that is, to find the objects undergoing state-change in egocentric videos.\n\n**Contribution:**\n1. propose a prompting pipeline that integrates LLM to obtain the knowledge about the objects undergoing state-change\n2. propose a joint inference framework that aggregates the pre- and post-condition information to find the objects undergoing state-change\n3. evaluate the proposed framework on two egocentric video datasets", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "1. Active Object Detection With Multistep Action Prediction Using Deep Q-Network, IEEE Transactions on Industrial Informatics 2019\n2.Active object localization with deep reinforcement learning\n3.Sequential voting with relational box fields for active object detection"}}, {"review_id": "Pj1AOYShAz", "reviewer": "Reviewer_h4o5", "summary": "", "questions": "A- In Section 2, you present the task itself. You mention that you rephrase the narrations as imperative instructions. I don't really see the need for that, but I may be missing something. How would the evaluation of your method, or even the method itself, change if you used the original narrations?\nB- You use ChatGPT, a closed model that we don't even know where was trained, to generate symbolic knowledge. Did you check whether ChatGPT is contaminated by Ego4D and Epic-Kitchens narrations? Have you considered the usage of open source models such as Llama and similars?\nC- In Section 3.2 you mention that \"Interestingly, we empirically find it beneficial to situate GPT with a role, e.g., \"From the first-person view.\"\". Can you provide more details on the empirical process? Was it qualitative or quantitative?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer h4o5 for the helpful comments and detailed questions! We are encouraged that the reviewer finds **our main idea very interesting**, and that the **extensive experimental studies justify its effectiveness**. We are particularly delighted that the reviewer acknowledges our framework being **generalizable** and **can be applied to further multimodal research**.\n\nPlease find your suggestions and concerns addressed below:\n\n---\n\n**[W1] Paper presentation improvements**\n\nSorry for the confusions and we really appreciate all the paper presentation suggestions.\nWe hereby address the concerns and elaborate on how we would revise our paper. We believe these revisions can be easily performed in the final version and hope the reviewer can re-evaluate this point as a reason to reject our work:\n\n- **Figure 1.** Revised caption: *”**Active object grounding** is the task of localizing the active objects undergoing state change (OUC). In this example action instruction \"cut the pawpaw into half with the knife\", the AI assistant is required to firstly infer the OUC (**pawpaw**) and the Tool (**knife**) from the instruction, and then localize them in the egocentric visual scenes throughout the action trajectories. Symbolic knowledge including pre/post conditions and object descriptions can bring additional information to facilitate the grounding.”*\n\n- **Figure 4.** We will condense L216-220 to enrich the description of the contrastive learning (left part), L276-279 to the additional object-knowledge grounding (right part), and L285-288 to the frame type prediction part.\n\n- **Table 1.** Defs. (which stands for Definitions) and Desc. (Descriptions of an object provided by the LLM) are equal. It was an editorial error which we will fix. Thanks for pointing it out! This table is a qualitative inspection of representative cases where the LLM fails to capture the desired knowledge from the action instructions. For clarity, we will change the Table 1 caption to the following: *”Table 1: Qualitative Analysis of GPT Knowledge Extraction: Examples of cases where GPT extracted symbolic knowledge are wrong or conflict with Ego4D annotations. Here the GPT-extracted or dataset-annotated knowledge are displayed in GREEN if they match human analysis and RED otherwise.”*\n\n- **Table 2.** As explained in Section 3.2 (L258-260), *“Textual”* refers to **”textual correctness”**: *i.e.* *\"based on text and common sense alone, does the GPT conds./desc. make sense?\"*; whereas *“Visual”* refers to **”visual correctness”**: *i.e.* *“does the GPT conds./desc. match what is shown in the image?\"* For more clarity, we will move this explanation to the image caption in the final version.\n\n- **Table 3.** In our final version we will split the table into two, where one contains the major results comparing our best model combinations with the baselines and other utilization of the action instructions; while the other mainly left for ablating the types of knowledge to be jointly inferred on. We also consider reporting only primary AP and AP50 results for the main paper and moving AP75 to the appendices if that brings better readability.\n\nOverall, we thank and appreciate all the paper presentation feedback and we will surely incorporate them into our final version!\n\n---\n\n**[W2] Novelty of our work**\n\nThanks for the related work! Ever since LLM showed remarkable performance on many general tasks, the incorporation of LLM into a multimodal system has drawn large attention from the community. While we acknowledge that the *“incorporation”* (itself) of an LLM is not novel, the actual usage of LLMs and types of knowledge carried out for different applications distinguishes each work. \n\nParticularly for the I2MVFormer work, LLMs are used to derive multiple *“views”* (*i.e.*, **static descriptions**) of an object class following a Wiki-style.\nWhile this aligns at a high-level with the generic-description (desc.) sub-component of our framework, our main novelty also lies in the utilization of **action- and state- dependent pre- and post-conditions**. These conditions are especially beneficial for **multi-frame action videos** and other **real-time vision applications** where the **state changes of objects** (visual contexts, appearances, etc.) are **spatiotemporally action-dependent beyond their generic attributes [1,2]**.\n\nOur experimental results (Table 3 and 5) also support this claim as *GPT+Conds.* significantly outperforms *GPT+Desc.* in nearly every evaluation category (and achieves best results in Ego4D SCOD when combined).\n\nFurthermore, the inference we propose is applicable to not just one object at a time but to **multiple objects involved in the human tasks** (*i.e.*, in this work, both Tools and OUCs). And hence, to make the derived symbolic knowledge concise but informative, we made efforts to devise an effective prompting scheme that is robust (tested on different versions of OpenAI GPT-series) to be further processed and parsed to help the grounding modules.\n\nWe will include the suggested work into the multimodality related work section of ours. Thanks for the suggestion!\n\n[1] Souček, Tomáš, et al. \"Look for the change: Learning object states and state-modifying actions from untrimmed web videos.\" *CVPR* 2022\n\n[2] Isola, Phillip, Joseph J. Lim, and Edward H. Adelson. \"Discovering states and transformations in image collections.\" *CVPR* 2015\n\n---\n\n**[W3] Supporting details for experimental results**\n\nWe mainly reported the best results obtained from each model. While during the submission period we trained key models on three or more different random seeds, due to time and computational resource constraints, we did not have the luxury to sweep the entire table for the multi-trials.\nHere we supplement the performance mean and std from the major model comparisons:\n\n| **Model** |           | **Pre** | **Pre** | **Pre** | **PNR** | **PNR** | **PNR** | **Post** | **Post** | **Post** |\n| :-----:| :-----: | :-----: | :-----: | :-----: | :-----: | :-----: | :-----: | :-----: | :-----: | :-----: | \n|           |           | **AP** | **AP50** | **AP75**  | **AP** | **AP50** | **AP75** | **AP** | **AP50** | **AP75** |\n| Full-Instr. | Mean | 33.07 | 52.33 | 34.04 | 34.53 | 55.46 | 35.33 | 30.90 | 48.29 | 32.12 |\n| Full-Instr.  | Std. | 0.44 | 0.50 | 0.52 | 0.55 | 0.73 | 0.60 | 0.29 | 0.66 | 0.29 |\n| GPT | Mean | 37.24 | 55.61 | 38.80 | 39.16 | 59.18 | 40.18 | 34.75 | 51.18 | 36.53 |\n| GPT  | Std. | 0.18 | 0.36 | 0.14 | 0.13 | 0.23 | 0.12 | 0.02 | 0.12 | 0.13 |\n| GT-SRL-ARG1 | Mean | 37.45 | 56.28 | 38.92 | 39.10 | 59.38 | 40.19 | 34.65 | 51.32 | 36.25 |\n| GT-SRL-ARG1  | Std. | 0.35 | 0.17 | 0.54 | 0.43 | 0.16 | 0.42 | 0.33 | 0.03 | 0.40 |\n| GPT+Conds. | Mean | 38.44 | 57.33 | 39.91 | 39.99 | 60.38 | 41.35 | 35.28 | 52.06 | 37.01 |\n| GPT+Conds. | Std. | 0.21 | 0.23 | 0.17 | 0.20 | 0.09 | 0.21 | 0.15 | 0.14 | 0.22 |\n| GPT+Conds.+Desc. | Mean | 38.17 | 57.32 | 39.69 | 39.92 | 60.60 | 41.24 | 35.44 | 52.67 | 37.02 |\n| GPT+Conds.+Desc. | Std. | 0.19 | 0.30 | 0.29 | 0.14 | 0.31 | 0.24 | 0.28 | 0.34 | 0.44 |\n\nThe results show close alignment with performance trends reported in the main paper.\n\nWe asked two trained (on the understanding of the pre- and post-conditions of an action-object interaction) non-co-author internal lab members to perform the human evaluations in Table 2, by sending them spreadsheets containing the original action instructions, corresponding LLM extractions, and the LLM-generated symbolic knowledge, where the human evaluated (*binary good/bad*) scores are averaged across their evaluations.\n\nWe appreciate the suggestions on the additional details of our studies, we will incorporate them into either the main paper or a new section in the appendix.\n\n---\n\n**[Q1] Imperative formats**\n\nWe rephrase task narrations to imperative form simply to follow the motivation of situating the assistant AI into the real-world application scenario. You are correct that our method does not (and should not) show any significant difference in the two formats of narrations.\n\n---\n\n**[Q2] ChatGPT data contamination (symbolic knowledge extraction sub-task)**\n\nIn our framework, we task ChatGPT with extracting the following knowledge from task instructions: the objects undergoing state change (OUCs), the tools that facilitate such actions, the pre- and post-conditions of each object, and the descriptions of each object. It is important to note that the Ego4D dataset does not contain annotations for any symbolic knowledge, and they did not previously exist anywhere until we generated them. Therefore, it is impossible for ChatGPT to have seen exact answers to our symbolic knowledge extraction questions even if it had access to all Ego4D annotations. What's more, the Ego4D annotations themselves are by no means a well-structured resource available online, and therefore extremely unlikely to be acquired and effectively trained on. Considering the above points, we do not believe that ChatGPT can effectively exploit the Ego4D narrations and associated symbolic knowledge in training time. \n\nOur method is not limited to the type of the LLMs, and hence high performing Llama family models can be suitable candidates for replacing ChatGPT in our knowledge extraction component. In any case, our framework is concerned only with the LLM's ability to follow our instructions and perform knowledge extraction.\n\n---\n\n**[Q3] Prompt specification of first-person view**\n\n**Some background:**\n- Here the task is for GPT to determine whether the object being questioned goes through visual state change from the action performer’s egocentric view and if so, provide visually identifiable pre/post conditions, thus we use this prompt phrase to provide GPT with viewpoint information and exploit its (*simulated*) visual commonsense knowledge (full prompt in Appendix A.1 L954-970).\n- This similar phenomenon is also observed in some concurrent works/blog posts [3,4,5] when utilizing LLMs via a role-playing fashion.\n\n**The empirical process was both qualitative and quantitative**:\n- **Qualitatively**: given the instruction \"Lay the trousers on the ironing board\" and asked whether “the ironing board” goes through any visual state change, GPT (specified first-person view) would respond with “Yes” (as the ironing board becomes covered by the trousers in the action performer’s egocentric view) and go on to provide the post condition “blocked, covered”. For the same example without specifying \"From the first-person view.\", GPT would respond with “No” and provide no post conditions.\n- **Quantitatively**: For a small subset of Ego4d prompts (220 samples) in the human evaluation of GPT generated answers for pre/post conditions (Table 2), we evaluated results both w/wo the specification \"From the first-person view.\":\n| **Prompt Style** |  Pre Cond.| |  Post Cond.| |\n| :-----:   | :-----: | :-----: | :-----: | :-----: |\n| | **Textual** | **Visual** | **Textual** | **Visual** |\n| wo/ First-Person | 70.9 | 75.0 | 61.4 | 64.5 |\n| w/ First-Person | 86.4 | 82.3 | 75.5 | 71.8 |\n  - We observed that when provided with additional viewpoint specification, GPT results displayed a significant increase in alignment with human judgment, thus it was incorporated into our final prompt.\n\n[3] Ding, Yan, et al. \"Leveraging Commonsense Knowledge from Large Language Models for Task and Motion Planning.\" *RSS 2023 Workshop* 2023\n\n[4] Shanahan, Murray, Kyle McDonell, and Laria Reynolds. \"Role-Play with Large Language Models.\" *arXiv preprint* 2023\n\n[5] Babar M Bhatti. “The Art and Science of Crafting Effective Prompts for LLMs”, *Medium Blog Post* 2023\n\n---\n\n**[Misc.] Missing references and typos**\n\nWe will cite the suggested references appropriately in our final version. Specifically for the I2MVFormer paper, we will include the discussion mentioned above. Thanks again for the presentation suggestion, we will fix the typos and enrich the related works section accordingly.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A- The idea of using LLMs to extract symbolic knowledge from textual instructions to improve object detection and tracking is very interesting.\nB- The proposed method (extended GLIP) can also be used for further multimodal or vision-based tasks.\nC- The obtained results show that leveraging textual information (in various ways) improves over purely visual methods.", "reasons_to_reject": "A- The presentation of the paper could be improved significantly, specially concerning the figures and their captions (they are not very descriptive) and the tables/captions. For example: the caption of Figure 1 does not describe the figure itself; it is a shorter repetition of the abstract. It would be more helpful to describe what the figure shows. The caption of Figure 4 is also very brief, when the depicted diagram is not easy to understand. Table 1: each row is different but it is not explained in the caption. For instance, what is \"Defs\" in the first row? Table 2 shows actually two different tables, and nothing is reported in the caption about the meaning of \"Textual\" and \"Visual\". Finally, Table 3, which is the main table of the paper, is so dense that it is very difficult to find trends and extract conclusions. I think this table should be split.\nB- Although the idea of using LLMs to extract symbolic knowledge and leveraging that knowledge to improve visual tasks is very interesting, it is not novel, from my point of view. Check the following paper, for example:\nMuhammad Ferjad Naeem, Muhammad Gul Zain Ali Khan, Yongqin Xian, Muhammad Zeshan Afzal, Didier Stricker, Luc Van Gool, Federico Tombari; I2MVFormer: Large Language Model Generated Multi-View Document Supervision for Zero-Shot Image Classification. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 15169-15179\nEven though the application is different and the methods differ, I think the main idea is quite similar. I still believe that the instantiation of the idea in this work is interesting, but I have to admit that the idea itself is not novel. \nC- Some doubts about the obtained results: specially for Table 3, authors do not specify how many executions they performed to obtain those results. At least three executions with the average and standard deviation should be reported to assess whether the obtained differences among models is significant. If those three (or more) executions cannot be performed, it should be justified. Similarly, for Table 2, there are missing details. Those are human evaluation results, but authors do not mention how many evaluators were involved, their background, their relation with the research team, etc. I know those are \"additional results\" and not the main target of the work, but I think we have to be more rigorous when presenting results (an Appendix could be added, for example, with the missing details).", "questions_to_authors": "A- In Section 2, you present the task itself. You mention that you rephrase the narrations as imperative instructions. I don't really see the need for that, but I may be missing something. How would the evaluation of your method, or even the method itself, change if you used the original narrations?\nB- You use ChatGPT, a closed model that we don't even know where was trained, to generate symbolic knowledge. Did you check whether ChatGPT is contaminated by Ego4D and Epic-Kitchens narrations? Have you considered the usage of open source models such as Llama and similars?\nC- In Section 3.2 you mention that \"Interestingly, we empirically find it beneficial to situate GPT with a role, e.g., \"From the first-person view.\"\". Can you provide more details on the empirical process? Was it qualitative or quantitative?", "paper_topic_and_main_contributions": "This paper considers the problem of actively grounding task instructions from an egocentric view. More concretely, authors deal with the problem of detecting and tracking objects undergoing change and tools (other objects which might be used to change the state of the main object). Previous works have mainly focused on the vision modality. This work tries to leverage also the textual modality, extracting symbolic knowledge from LLMs and training an extension of an open-vocabulary object detector (GLIP) to improve its grounding capabilities. The proposal is evaluated on two egocentric video datasets, namely Ego4D and Epic-Kitchens.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Muhammad Ferjad Naeem, Muhammad Gul Zain Ali Khan, Yongqin Xian, Muhammad Zeshan Afzal, Didier Stricker, Luc Van Gool, Federico Tombari; I2MVFormer: Large Language Model Generated Multi-View Document Supervision for Zero-Shot Image Classification. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 15169-15179", "Typos_Grammar_Style_and_Presentation_Improvements": "The paper contains many typos. Here is a list (not exhaustive; please, review the paper for typos):\n\nline 163: challengeonly -> challenge only\nFigure 5: State Change Forcast -> State Change Forecast\nline 312: can be deterministic constructed -> deterministically constructed"}}]

Author rebuttal:

Meta-review: The paper proposes a method to use world knowledge from LLMs to improve active object detection and tracking in ego-centric videos.

Strengths:
- Paper is well written (MyHn)
- Interesting idea (h4o5)
- Results and analysis (MyHn, 5jwp)
- Detailed limitations section (MyHn)
- Generality of method (h4o5)

The concerns raised by the reviewers were mostly comments to provide constructive feedback, or mostly addressed during rebuttal (reviewers h4o5, 5jwp). No serious outstanding soundness concerns are identified and there is overall consensus on the soundness of the paper.
Decision: Accept-Main