Look, Remember and Reason: Grounded Reasoning in Videos with Language Models
Authors: Apratim Bhattacharyya
Sunny Panchal
Reza Pourreza
Mingu Lee
Pulkit Madan
Roland Memisevic
Venue: ICLR 2024 poster (2024)
Keywords: Grounding
Reasoning
Language Models
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: We show the effectiveness of off-the-shelf language models for reasoning on videos, when grounded using surrogate tasks.

Abstract: Multi-modal language models (LM) have recently shown promising performance 
in high-level reasoning tasks on videos. However, existing methods still fall short in tasks like causal or  compositional spatiotemporal reasoning over actions, in which model predictions need to be grounded in fine-grained low-level details, such as object motions and object interactions.
In this work, we propose training an LM end-to-end on low-level surrogate tasks, including object detection, re-identification, and tracking, to endow the  model with the required low-level visual capabilities. We show that a two-stream video encoder with spatiotemporal attention is effective at capturing the required static and motion-based cues in the video. By leveraging the LM's ability to perform the low-level surrogate tasks,  we can cast reasoning in videos as the three-step process of *Look, Remember, Reason*, wherein visual information is extracted using low-level visual skills step-by-step and then integrated to arrive at a final answer. We demonstrate the effectiveness of our framework on diverse visual reasoning tasks from the ACRE, CATER, Something-Else and STAR datasets. Our approach is trainable end-to-end and surpasses state-of-the-art task-specific methods across these tasks by a large margin.

Published as a conference paper at ICLR 2024
LOOK, REMEMBER AND REASON: GROUNDED REA- SONING IN VIDEOS WITH LANGUAGE MODELS
Apratim Bhattacharyya, Sunny Panchal, Mingu Lee, Reza Pourreza, Pulkit Madan, Roland Memisevic Qualcomm AI Research*
ABSTRACT
Multi-modal language models (LM) have recently shown promising performance in high-level reasoning tasks on videos. However, existing methods still fall short in tasks like causal or compositional spatiotemporal reasoning over actions, in which model predictions need to be grounded fine-grained low-level details, such as object motions and object interactions. In this work, we propose train- ing an LM end-to-end on low-level surrogate tasks, including object detection, re-identification, and tracking, to endow the model with the required low-level visual capabilities. We show that a two-stream video encoder with spatiotemporal attention is effective at capturing the required static and motion- based cues in the video. By leveraging the LMMss ability to perform the low-level surrogate tasks, we cast reasoning in videos as the three-step process of Look, Remember, Reason, wherein visual information is extracted using low-level visual skills step-by-step and then integrated arrive at final answer. We demonstrate the effectiveness of our framework on diverse visual reasoning tasks from the ACRE, CATER, Something- EEle and STAR datasets. Our approach is trainable end-to-end and surpasses state-of- tate--of-hhearr the-art art task-specifie methods across tasks by large margin.
INTRODUCTION
Autoregressive language models (LMs) have shown impressive results reasoning tasks such as on grade school math problems (Cobbe et al.) 2021) and even on LSAT (abs) 2023). Most language models designed for these problems, however, are trained on only textual data. Since many real-world scenarios require reasoning over heterogeneous sensory inputs. e.g.. visual cues. multi-modal LMs for images or videos have recently gained traction (Alayrac et al. 2022 Koh al., [023] Zhang et al.,
2023b Maaz et al. 2023 Li et al. 2023b b Maaz et al. 2023 The focus of current multi-modal
LMs for videos have primarily been on high-level question answering and instruction following (Maaz et al. 2023 Li et al. 2023b Alayrac et al. 2022). However, many visual reasoning problems in videos require grounding in lineggriined level information, i.e., recognizing objects, and understanding their spatiotemporal interactions. For example, as shown in Fig. I recognition of the high-level compositional action requires a fine-grained understanding of low-level details of motion and interactions between objects. The ability of multi-modal LMs to perform visual reasoning tasks such as compositional action recognition in videos (Materzynska et al. 2020) that require a combination of low-level skills with high-level reasoning has not yet been explored.
To enable multi-modal LMs to solve such reasoning problems we propose our Look, Remember and Reason (LRR) multi-modal LM. Our LRR model architecture extracts dense spatiotemporal features from each imput frame. This is accomplished using a two-stream attention based architecture that captures low-level spatial and temporal details (Simonyan & Zisserman, 2014) at each input video frame using top-down cross-attention Our multi-modal LRR model is grounded to relevant low-level visual information the scene by stochastically introducing low-level surrogate tasks during training. including object recognition, re-identification. and tracking, at randomly selected time-steps (c.f. Fig. M.. We keep these grounded spatiotemporal features, which include low-level visual details, in the working memory, i.e., "remembered" within the context window of the LM. This allows the
"Qualcomm Al Research initiative of Qualcomm Technologies, Inc.

---

Published as a conference paper at ICLR 2024
Video Frames
Surrogate Task for Grounding Object Tracking
Visual Reasoning
Problam Crample Task Composhoonll Action Recogniton
Language Model
Solution Pushing (something) from nggnt
Figure 1. Our Look, Remember, Reason (I.RR) model 'looks' the video frames extract relevant low-level information, e.g., objett motion and interactions, supervised with surrogate tasks like object tracking only during training. It 'remembers' the information from intermediate steps and 'reasons' using the aggregated information.
model combine the low-level visual features with high-level inferences "reason" and generate the final responses using our LRR framework as shown in Fig. !
Our main contributions are: (i) We highlight the importance of grounding for visual reasoning in multi-modal LMs and propose a novel Look, Remember, and Reason framework to this end to instill the required low-level visual skills in the model using surrogate tasks during training: (ii) We introduce two-stream video encoder that captures both the scene structure and object motions, crucial for learning the low-level skills; (iii) We demonstrate the effectiveness of our approach on ACRE (Zhang al. 2021), CATER (Girdhar & Ramanan 2020), and the real-world Something Elsse (Materzynska et al. 2020) and STAR (Wu et al. al.] (021)) datasets. Our approach outperforms the prior state-of-the-art, that is based on highly task-specific architectures, by large margin margin--highlighteee highlighting how our general- general-purpose purpose LRR model can perform varied and complex spatiotemporal reasoning tasks in videos including causal, compositional and situated reasoning.
RELATED WORK
Multi-modal language models. The use of auto-regressive language models with adapters to process visual inputs is an active area of research. Recently proposed models include Pix2seq (Chen et al.) 2022), which trained extract low-level visual information from images. Unlike our work, extracting low-level visual information is a goal in itself in that work not surrogate task, and the model processes images not videos. Other image- based mudti LMs include ViperGPT (Surís et al., 2023), VisProg (Gupta & Kembhavi 2022). and Chameleon (Lu et al. (203)). Similarly, PaIM-E (Driess et al., 2023) provides images and text as interleaved multi-modal latent vectors, allowing process multiple images within any part of sentence which serves
input to the
LM where the model is trained end-to-end. ELaMA-Adaptes (Zhang et al. (2023a) introduces an adapter layer to enable multi-modal inputs with the LLaMA
et al. (023). FROMAGe (Koh et al.) 2023) freezes the language model and fine-tunes the input and output linear layers to encode multi multi-modal interactions. InstructBLIP (Dai al.) (023) improves high-level instruction following abilities using instruction fine-tuning. Qwen-Vi VL (Bai et al. (023) and Kosmos-2 (Peng
al. 2023) propose grounded multi-model LMs that can localize referring expressions in images. In contrast to our work, all these approaches process images not videos.
Video-based multi-modal LMs include Video-ChatGPT Maaz et al. 2023). VideoChat (Li al. 2023b), and Valley (Luo et al., (223). They focus on question answering and instruction following using video. Flamingo (Alayrac et al., 2022), by using Perceiver as input, also able to ingest video data, and similarly is trained to infer high-level concepts. In our work, in contrast, we study visual reasoning tasks that require low-level understanding the form of e.g., motion, actions, and interactions, in addition to high-level reasoning. We show how grounding high-level concepts in these low-level representations greatly improves video-based reasoning tasks.
Spatiotemporal video grounding. Recent work grounding focuses spatially localizing object in an image or in a video given a referring expression (Huang et al. 2018 8 Shi et al. 2019
Vasudevan al.) 20188 [i et al. (202cc). Deng et al. (2021); Yang et al. (2020); Luo et al. 2020);
Yang et al. 2019); Kamath et al. (2021) propose end-to--nd approaches for the spatial local izatton
images. For video, (Chen al. (0118 2019) Zeng al. (2020); ZZhng et al. (2019) 2020
2

---

Published as a conference paper at ICLR 2024
He al. 2 Jang et al. (2023) focus temporal localization given natural language query. Methods like STVGBert (Su et al. 2021) and STCAT (Jin et al. 2022) perform spatiotemporal
localization of natural language query using transformer-basee model. Yang et al. (0022) focuses on action localization given a natural language query. Xíao et al. (2023) propose video grounding using Gaussian mask optimization. In contrast to these methods, the approach we introduce performs visual grounding in pre-trained LM through end-to-end training using surrogate tasks. We show that this makes it possible instill low-level visual capabilities as required for solving reasoning problems.
Attention-based models and visual reasoning. Attention- dleettoo-bbased models have been studied ex- tensively for visual reasoning Ding et al. (2021); Hu et al. (201)); Hudson & Manning (2018);
Kamath et al. p02Tp: Mahajan & Roth |020)]|Santoro et al. (2017). Recent advances include an object-centric encoder and a transformer reasoning module to solve RPM-like benchmarks (Mondal
ct al. 2023), multi-hop feature modulation (Strub et al. 2018) and cascaded modulation networks Yao al. 2018) that
multi-step comprehension process, neural interpreters (Rahaman et al.) 2021) that factorize inference in a self-attention network and ALANS learner (Zhang et al. (022a)) that combines abstract algebra and representation theory. Calibrating concepts and operations Li et al. (2021) enables neural symbolic models to capture underlying data characteristics and perform hierarchical inference. Again, the key difference in our work is that instill the ability extract object-centric information within the LM itself, using surrogate tasks, instead of resorting to separate object detection modules.
3 LOOK, REMEMBER, REASON
To allow for visual reasoning videos using language models, we propose a novel Look, Remember, Reason framework which grounds the model to the low-level structual and motion information in the visual input required for solving high-level reasoning tasks. Grounding is enabled through low-level visual surrogate tasks, e.g. object recognition, tracking and re-identification. In the following, we first describe our auto-regressive LRR architecture including details of our two-stream video encoder, followed by our training pipeline with details of our surrogate tasks.
3.1 AUTO-REGRESSIVE PIPELINE
Our LRR model parameterized by 6, as shown in Fig. 2 based on a pre-trained LM backbone with two-stream auto-regressive video encoder. Our LRR model receives the visual reasoning problem Q and, an interleaved sequence Z of video frames V (Vi....; Vr.) and tokenized text == (51 ... ST,) ) as input. S consists of low-level visual surrogate tasks and the answer indicated by <taskname> and <answer> respectively. The input video frame sequence V is encoded by our two- stream video encoder. The LM backbone receives input Q, and the interleaved sequence of encoded video frames V whose positions are indicated with <frame> special tokens and S. Conditioned on Q and V, we train our LRR model by maximizing log-likelihood of the text sequence S as,
log (po (S/Q. V))
Σ log (s) St -1, VI, Vt.. Q))
(1)
where, (V1.-. Ve.) is the interleaved video frame sub-sequence up to the text token St,- In Section 3.4, we describe the surrogate tasks included in S. The parameters of the LM backbone are initialized from pre-trained LMs. allowing us to exploit their existing high-level reasoning capabilities. We use LMs from the OPT family (Zhang et al., 2022b). but verified that similar performance can be achieved using other pre-trained models (Gao et al. 2021 [Scao et al.) (022). While LM backbones are trained only on text, visual reasoning relies on information from the visual domain V. Therefore, to ground our LRR model, visual information in V needs to be mapped to the text-based representation space of the LM. The key challenge here is that in contrast to text tokens, videos are highly information dense. To address this, we develop two-stream video encoder to extract information relevant for the visual reasoning problem at hand, such that responses to the visual reasoning problems are grounded in the fine-grained low-level details of the video.
3

---

Published as a conference paper at ICLR 2024
(it-r) Vi6-++11)
Yiu
VTU
Two-stream Encoder
Temporal Astention
Spatial Attention
TEMP-ATTS
TEMP-ArIS
TEXP-ATTN
2/
TEMP-ATTN
L blools
SPATIA--ATT
APATTAL-ATTS
SPATIAL-ATTH
SPATIAL-ATTN
TYMP-AFTY
TEMI-ArT
TRSP--ATI
TRMP-ATTN
At trnining time
LM Backbone
<taskname>
<answer>
SELE-NETN
SELE-AFTS
SELF-AFTN
h
SELE-AFTS
SELP-ATIN
SELF-AFTN
M Mocios
CRoss Artr
CROSS-ATIN
CR0sS-ATTS
SELP-ATEN
CRoss ArtN
SELT-AFTS
SELF-AFTN
SELE-ATTS
SEL--ArP
SELE-ATTS
SSUT-ATTN
SELP-ATTN
Surrogate Task
Answer
91
Figure The architecture of our LRR model, highlighting the use of interleaved top-down cross-attention layers in between self-attention layers higher up in the hierarchy.
3.2
TWO-STREAM VIDEO ENCODER
Our autoregressive two-stream video encoder exploits divided space-time attention (Bertasius et al. 202T) and generates for each input video frame patch based embeddings that capture relevant low- level spatio-temporal information. Spatial attention captures structural information, e.g., object identities. Temporal attention autoregressively captures object motion and interaction information using the previous frames as buffer. As first step, converts each input video frame Vt, into P flattened patches Vt. = (V(I.t.))..... V(P.r..)) of size 16 x 16 (Dosovitskiy et al. 2021) as shown in Fig. Our LRR model applies a linear transformation (LN) on the input patches generate the initial patch embeddings z!. {2( (1.t.)) z L(;,,)) as.
(p.c.) LN( LN(V + cm
The spatiotemporal positional embedding (m.....) added to each patch to aid our video encoder ex- tract spatiotemporal features from each patch in the input video frame. Next, these patch embeddings are processed by the blocks of our video encoder where at each block spatial or temporal attention is applied. At every block & E {I...., L.} of our model, the embeddings are linearly mapped to produce the query, keys, and values required for the space- and time-based attention operations,
K.... = LL(Z) ((...) LN(Zf ((P...) LN (p..)))
In case of temporal attention, for the patch embedding 2/ (pp..) the attention is computed over patch embeddings at the same spatial position in the т previous input video frames, 2 (p.(tt.r)) (t)) +)) to z (p.tt)) For the patch highlighted red in Fig. the patches where temporal attention is applied in the previous F input frames highlighted in cyan. The temporal attention vector (aT) is given by,
Linste) Vdm
(oT)),..)) SOFTMAX
k(p..)))
where dm is the dimensionality of the key, queries and values. In contrast, the spatial attention vector, (ass) (m... is calculated over all patch embeddings of the current input video frame, highlighted in green in Fig 2
Lprtol [K KK........... K (P.a))))
(as) (p.t.) = SOFTMAX
(5)
Vdm
The patch embeddings z f+1 are the weighted (p.ta)
of the values vf (p.t..) computed using the attention weights (aT) or (as)(p.t)) in case of temporal or spatial attention, respectively. Finally, the patch embeddings z!;,- containing localized information of the scene structure and motion, are obtained output from the last block I. at time-step i, of the video encoder.
4

---

Published as a conference paper at ICLR 2024
3.3 LM BACKBONE WITH TOW-DOWN CROSS ATTENTION
Visual information encoded by the two-stream video encoder is mapped top-down to the LM backbone positions indicated by the <frame> token time-steps i, (Fig. 2 using cross attention layers. This top-down mechanism with cross attention layers, trained using surrogate tasks (Section (.4). helps us ground the LM backbone to the relevant low-level information from the input video frames.
We modify the backbone LM architecture by inserting cross attention (CROSS-ATTI) layers (Alayrac) al., (022) between self-attention (SELF-ATTN) layers (Fig. 2 The output patch embeddings from our video encoder / (1...)) (f....)....,((....)) ( } are transformed using multi-layer perceptrons (MLP) at every cross-attention layer which help deal with the domain shift between the visual and textual domains. Next, we exploit the rich hierarchical representations in the hidden states J of the LM at timestep by, where M is the number of self-attention layers, to "look" at the visual input and extract visual information in a top-down fashion for grounding. Specifically, we use the representation hig after the application of the (k + 1)th self-attention layer in the LM. The hidden representation i encodes global semantics about the visual reasoning problem Q and previous video frames upto 4, It serves as the query vector after a linear transformation and the visual features z serves as the keys and values of the cross attention layer respectively,
hr, {he ..... 1.......... h.
h SELF-ATIN(H
(6)
2. = CROSS-ATTN(H 2(.))
(7)
htー1 hR. 1 2t.
(8)
h 1-- = FF. k+1
h, K+1
(9)
where FFN is feedforward layer Vaswanni et al. 2017). As the hidden representation h encodes
global semantics, it allows the LM backbone to extract relevant low-level spatiotemporal information
in 2, (Eq. 77). when trained using our surrogate tasks. This information added to hidden states h k+1
(Eq. g and implicitly "remembered" within the context window of the L.M. This information can be aggregated by the LM to "reason" and arrive at the final answer.
3.4 GROUNDING THROUGH SURROGATE TASKS
To ground our LRR model the relevant low-level information in the visual input, we utilize surrogate tasks during training. This is illustrated in Fig. where the model is prompted using special tokens (<taskname>) to solve surrogate task. We consider tasks like object recognition, tracking and re-identification as low-level. From Fig. [ the ability to recognize the performed action rests upon grounding to the motion and interactions of the hand and the stapler. This makes recognition and tracking constituent low-level capabilities for recognizing the (high-level) compositional action.
Our LRR model highly flexible and can be prompted to solve a wide range of low-level surrogate tasks. Here, we focus primarily the tasks of object recognition, localization, re- ce-identification and tracking. These tasks are fundamental to solving a range of visual reasoning problems, and the requisite ground truth can be readily obtained using off-the-shelf vision models (Ren et al. 2017] 7] Tang et al. 2017; Ye et al. 2022). Moreover, these tasks can be encoded as text to be processed by the LM backbone in the following general format: "<taskname> object id,, object class,, object bounding box
object idn, object class,, object bounding box," ", where there are n objects in the scene. The <taskname> special token prompts the model to solve the surrogate task. We use <detect>, cre-identifyz and <track> for detection, re- re-identification, and tracking surrogate tasks respectively. The object id is an integer that assigned by the model based on the spatiotemporal order of appearance and is crucial for re-identification and tracking as same object instance should be assigned the same id across video frames. The bounding box is described as 4-tuple of the x and y coordinates of the upper left and lower right corners. We provide additional details in the experimental section (Section 4).
Our LRR model is prompted to solve these surrogate tasks at randomly selected time steps during training. Random prompting forgoes the need to include surrogate tasks during inference time, leading to faster inference. Random prompting also benefits training efficiency in the case of long video sequences. To train our LRR model using maximum likelihood as described in Eq. (T). We construct the token sequence S in I that includes surrogate tasks for each training example follows:
5

---

Published as a conference paper at ICLR 2024
Table Evaluation results the ACRE dataset, where, D.R. Direct evidence, L.D. Indirect evidence, S.O.
Screened-off. and B. Backward Blocked subsets ('represents results tested by ourselves.).
Compositional
Systematic:
Model
All
D.R.
L.D.
S.O.
B.B.
All
D.R.
L.D.
S.O.
B.B
NS.OPT Zhang 2021
69.0
92.5
76.0
88.3
13.4
67.4
94.7
88.3
82.7
16.0
IV-CL Sun al. 2025
93.2
92.6
ALOE
Drg al
91.7
97.1
90.8
96.8
78.8
93.9
97.1
71.2
98.9
94.4
OpenFlamingo
Awaialla
2025
38.2
42.6
49.5
9.9
47.6
38.6
36.5
25.8
13.7
67.6
LRR
98.2
LRR (w/o Surrogate tasks)
38.1
38.4
30.2
26.2
50.0
36.5
35.2
28.
55.3
LRR (wil "Two-stream" encoder)
92.2
98.6
77.7
96.2
85.8
92.8
98.5
76.2
96.8
89.4
LRR (from scratch)
87.7
96.9
57.7
91.1
84.8
88.0
96.3
58.5
91.5
86.9
Context Trials
Query
<re-identify> 5. medium blue metal sphere; medium purple rubber eylinder; 3. medium groy rubber sphere. <blicketz off.
<re ide-identyy>> L.medium gray rubber eylinder; medium green rubber cylinder.
<re identily> 3, mediuon gray rubber sphere; mediam yellow metal cube. <blicket> on.
<re identify> 1, medium gray rubber cylinder; mediam blue metal sphere. canswere no.
<blicket> on.
Figure 3. Example solutions to surrogate tasks generated by our LRR model on ACRE. Re-identified objects across context trials are underlined in the same color.
We first randomly select a subset video frames (V Wig....,VVy V. We then update the token sequence S include the surrogate tasks at these randomly selected time-steps, the beginning of which marked by a task specific <taskname> special token, e.g., <detect>, <re-identify> and <tracks, The surrogate task itself is added in described above.
EXPERIMENTS
We evaluate on visual reasoning tasks from: ACRE (Zhang et al.) (02T). Something Materzynska al. (2020), CATER (Girdhar & Ramanan 2020) and STAR (Wu et al. 2021).
Models and training details. We fine- tune-tue the pre-trained LM backbone along with the video encoder and cross- aroos-attention layers in our LRR model the visual reasoning problems considered here are challenging and cannot accurately solved by prompting state of the art LMs such as GPTT-4 (Gendron al., 2023). We focus the OPT family of LMs /Zhng al, (2022h), particularly OPT-125M/350M/1.C We use 4 Nvidia A100 GPUs. Additional details in the appendix.
4.1 ACRE
The ACRE dataset (Zhang et al. 2021) evaluates how well vision systems perform causal induction. focuses on causal discovery through "blicket" detection tests (originally designed for children). A blicket detector activates in the presence of a blicket object. The experiment uses context trials where different objects are placed on the detector, revealing its activation status. Subjects must then determine which objects or combinations would trigger the detector. The ACRE dataset includes context trials and 4 blicket detection tests per sample.
Surrogate tasks. The key low-level visual challenge in the ACRE dataset is to associate query objects to the context trials to detect whether the blicket machine activated. Therefore, we consider the surrogate tasks of object recognition and re-identification. The solution to the surrogate task can be generated by the backbone LM the following format: "cre-identify> object id,, object class;; additional surrogate task to identify the state of the blicket machine: "<blicket> on/off"". These surrogate tasks introduced randomly during training with probability of 30% after each context trial query. Baselines and evaluation. We base our LRR models on the 6PT-125M backbone. For comparison (see Table T). include the state-of-the-an the-rrt multi-modal L.LM OpenFlamingo (3B-mosaicml//ntt
object id,, object class, as shown in Fig. 3 We also found it helpful to introduce

---

Published as a conference paper at ICLR 2024
Table 2. Evaluation the Something
Ellse Else dataset (" "reeresents results tested by ourselves.).
Base
Compositionl
Method
Top-1
Top-5
Top-1
Top-5
STIN OIE + NL Materzynska el
2020
78.1
94.5
56.2
81.3
Video-ChatGPT" Mazz el ol. 2025
52.6
75.8
38.6
LRR (Ours)
80.2
96.1
62.0
86.3
LRR (wlo Surrogate tasks)
71.3
89.6
50.1
70.8
LRR (wlo Two-stream encoder)
73.2
90.4
53.6
76.1
aand
objectaa
obert
Objec: Objer's
Figure 4. Example solutions surrogate task tracking generated by our LRR model on Something-FRi Bounding boxes belonging to the same track are highlighted using the same color.
Be-redaajam-220bbdoee Awaddlly et al. (2023)), it has shown success in reasoning problems Awadalla involving multiple images and videos. We also test the following ablations to highlight key compo- nents: 1. LRR (w/o Surrogate tasks): Tests the importance of the surrogate re-identification task; 2. LRR (w/o Two-stream encoder): Tests the importance of our two-stream encoder's temporal attention mechanism for object re-identification; LRR (trained from scratch): Demonstrates the value of pre-trained LM backbones by using the same OPT-125M architecture without pre-training.
Our LRR model outperforms the state of the art powerful transformer based ALOE (Ding al.) 2021) by large margin of 6.5% and 5.3% on the compositional and systematic splits respectively. LRR model with surrogate tasks over explicit object centric imput representations used by ALOE. Similarly, we observe a significant performance gain of 60% over the powerful transformer based multi-modal L.LM OpenFlamingo (Awadalla et al. 2023) which highlights the importance of our LRR framework to ground our model to the relevant low-level details highlighted by the weak performance of the LRR (w/o Surrogate tasks) ablation. We also that our two-stream encoder improves performance over the single-stream model. The weak performance of the LRR R. (from scratch) ablation shows that is crucial to start from a pre-trained LM backbone exploit its high level reasoning abilities. Finally, although the results in Table use the OPT-125M backbone in our LRR model, we obtain an identical 98. 98.2% and 99.0% accuracy on the compositional and systematic splits respectively with OPT-1.3B backbone, indicating that our look, remember, reason framework is applicable across LM backbone sizes.
This shows the advantage of our end-to
4.2 SOMETHING-ELSE
This complex, real-world dataset (Materzynska et al., 2020) focuses on compositional action recogni- tion. Building upon the Something-SSoethhinn dataset (Goyal et al. 2017). it measures compositional generalization by splitting actions into verb, subject, and object combinations. This split allows for benchmarking performance on novel combinations unseen during training, forcing models develop a nuanced understanding of motion rather than simply associating actions with object types.
Surrogate tasks. In this task, we employ tracking as a surrogate task to support the model's ability to capture motion and object interactions. Note that, as object classes are not important for compositional action recognition, the solution to the surrogate task can be generated the following format: "<track> object id1 object bounding box1; ... object id,, object bounding box," as shown Fig. Surrogate tasks are introduced randomly during training with a probability 30% after each input video frame.
Baselines and evaluation. We base our LRR model on the OPT-125M LM backbone and compare to several baselines and ablations in Table 2 We report results both the base split and the compositional split with novel action-object combinations. We consider the state-of-the the-tt art STIN al. (2020) and Video-Chard ChatGPT (Maaz et al. (023) baselines. We
+ OIE + NL Materrzzyss demonstrate the importance of our surrogate tasks and two-stream video encoder through ablations: LRR (w/o Surrogate Tasks) and (w/o Two-stream encoder).
7

---

Published as a conference paper at ICLR 2024
Table Evaluation on the CATER dataset ( ressulls reported only for static camera).
Static Camera
Moving Camera
Method
Top-I(f)
Top-S(t)
Liggrid:)
Top-1(1)
Top-S(T)
Li(grid;))
R3D
LSTM Girdhar Ramanan
2020
46.2
69.9
1.5
38.6
70.2
ALOB
Ding et
2021
74.0
94.0
0.44
59.7
90.1
0.69
IV-CL
2023
OPNe
S Shamsian
2020
74.8
0.54
Hopper
Zhou of al
202
73.2
93.8
0.85
TPC. VaD
Zhang
2022
79.7
95.5
0.47
LRR (Ours)
84.1
97.2
0.34
80.4
96.7
0.42
LRR (wil Surrogate tasks)
68.5
88.7
0.65
62.7
86.7
0.77
LRR (wlo Two-stream encoder)
81.4
97.2
0.44
75.6
96.6
0.53
13, 18, 18,
21, 3, 3,
21, 33, 21
32, 3.
Figure 5. Example answers to the tracking surrogate task generated by our LRR model on CATER. Our LRR model prompted with the "<track>" special token solve the tracking surrogate task randomly selected time-steps during training. Object tracks are over the x grid
surface and are highlighted in color.
Our LRR model also outperforms the STIN + OIE + NL baseline by 2.1 % and 5.8% Top-1 accuracy on the base and compositional split respectively, which highlights the reasoning ability of grounded LM based architectures. The results also show that the performance of the state of the art multi-modal (finetuned), lags very significantly by 23.4% behind our LRR model. This because of the CLIP based video encoder in Video-ChatGPT is not well suited for capturing motion features and due to a lack of grounding. The importance of motion features underscored by the lacking performance of the plain ViT based (w/o Two-stream encoder) ablation: Top-1 accuracy drop of 8.4% on the compositional split. The importance of grounding is highlighted through our ablation without surrogate tasks: Top-1 accuracy drop of 11.9% the compositional split. We illustrate successful tracking of objects in complex real-world scenarios by our L.R model in Fig, and in Appendix B. Note that, although the results in Table uses the OPT-125M backbone, we obtain an identical 61.3% Top-1 and 85.9% Top-5 accuracy on the compositional split with an OPT-1.3B backbone, indicating that our look, remember, reason framework applicable across LM sizes.
LM:
4.3 CATER
The CATER (Compositional Actions and TEmporal Reasoning) dataset challenges models to rec- ognize complex object movement patterns requiring long-term reasoning. The most difficult task is adversarial target tracking, where a "snitch" object must be located at the end of a video sequence despite potential occlusion containment within other objects. This formulated problem on 6 x grid. The dataset includes static and moving camera splits, with the latter posing additional challenges due the need for long-term spatiotemporal analysis.
classification
Surrogate tasks. We employ (multi-target) tracking as surrogate task for which the solution can be expressed the following format: "<track> object id1, object grid position,; ... object id,, object grid position,," as shown in Fig. 5 The tracking task includes the medium and large cones in the scene, as these objects can occlude the snitch. We use grid positions instead of bounding boxes (unlike Something-Elsse because the final goal is to predict the grid position of the snitch. Surrogate tasks are introduced randomly during training with probability of 30% after each input frame.
Baselines and evaluation. Our model uses the OPT- 125M backbone (results in Table 3). We ablate both the surrogate (multi-target) tracking task (w/o Surrogate tasks) and our Two-stream video encoder (w/o Two-stream Encoder) to demonstrate their importance. Following ALOB Ding et al. (2021). we jointly train on static and moving camera splits.
The state-of-the -ftte-rrr- powerful transformer- maaedd IV-CL (Sun et al.) 2023), OPNet (Shamsian et al.) 2020), Hopper (Zhou al.) 202T). TFC V3D Depthwise (Zhang 2022) and Loci (Traub et al. (023) report results only the static camera split. Loci (Traub al. 2023 reports an impressive 90.7%
8

---

Published as a conference paper at ICLR 2024
Table 4. Evaluation of our LRR model STAR (validation set).
Method
Int.
Seq.
Prc.
Fca.
Overall
Internvideo (Wang et al
2022
62.7
65.6
54.9
51.9
58.7
BLIP-2 (Li et al. 2023a
65,4
69.0
59.7
54.2
62,0
SeViLA Yu et 2023
63.7
70.4
63.1
62.4
64.9
LRR (Ours)
73.7
71.0
71.3
65.1
70.5
LRR (w/o Surrogate tasks)
54.5
48.7
44.3
45.5
48.2
accuracy on the static camera split, but is not applicable to the moving camera split due to its static background and camera model. Our LRR model outperforms TFC V3D Depthwise (Zhang [0022) model on the static camera by 4.4% and ALOE (Ding et al. 202T) on the challenging moving camera split by 20.7% 7%. The large performance gain over the LRR (w/o Surrogate tasks) baseline shows the advantage of surrogate tracking tasks, without which the model is not grounded to the motion of the cones and hence fails in cases where the snitch is contained by the cones. Qualitative examples in Fig. 5 illlusrrates that our model is able to successfully track objects in cases of recursive containment and is robust to moving cameras. Finally, the performance advantage over the LRR (w/o Two-stream encoder) confirms that our LRR model is able to better capture the motion of the objects.
4.4 STAR
STAR (Wu et al. 2021) is a situated spatio-tempone reasoning benchmark, consisting of questions built upon real-world videos associated with human actions and interactions. The STAR benchmark thus requires an understanding of low-level human motion, actions and object interactions. However, the STAR dataset does not contain dense object annotations, e-g., object tracks, unlike the CATER and Something -elee datasets. It only contains sparse spatio-temporal scene graphs which cover selected keyframes. We introduce object recognition surrogate tasks localize objects on these keyframes based on the scene graphs. Furthermore, we jointly train our LRR model to recognize actions from the Kineties (Kay et al. (017) and Moments in Time (Monfort et al. (020) dataset, as well as surrogate tracking tasks from the Something- dataset (see Section 4.2 to improve the understanding of object interactions. We also regularize on textual data.
Our LRR model with an OPT-350M LM backbone achieves 70.5% overall accuracy and significantly outperforms powerful trasnformer-base state of the art models, such as ViLA (Yu et al. 2023). Internvideo (Wang et al. 2022) and BLIP-2 (Li et al. (2033)) shown in Table 4 Following Se ViLA (Yu et al. 2023), we report results on the validation set. Note that methods such as Se ViLA are trained much larger set of image/
text pairs and, use the 4.1B parameter BLIP as video encoder and the 3B parameter Flan- T55 XL as an language model (c.f. Section 4.4 in Yu et al. (2023)). In contrast, our LRR model contains fewer parameters and is trained on a much smaller training set. Curicially, our training set includes carefully selected surrogate tasks to endow the model with the requisite low-level visual capabilities. To illustrate this, we include ablation of our LRR model without any surrogate tasks (w/o Surrogate tasks).
In addition to the results on the STAR validation set presented above we also evaluate our LRR model on the STAR challenge leaderboard. The results can be found: here (ranked 1 as of January 2024.)
CONCLUSION
We show that off-the-shell LMs can solve complex visual reasoning tasks on videos using our LRR framework, In this framework, LMs equipped with a two-stream video encoder are supervised and grounded using surrogate tasks. Grounding ensures that the LM can utilize relevant low-level visual cues from in the input video to make predictions. Grounding predictions to low-level visual cues combined with the high-level reasoning ability of the LM is the key to the success of the model. Our LRR model outperforms the state of the art by 6.5% and 5.3% on the compositional and systematic splits of the ACRE dataset; by 5.8% Top-1 accuracy the compositional split of the Something Else dataset; by 4.4% Top- 1 accuracy on static camera and 20.7% Top-1 accuracy on moving camera splits of the CATER dataset; and by 5.6% overall accuracy on STAR.

---

Published as a conference paper at ICLR 2024
REFERENCES
GPT-4 technical report. CORR, abs/2303.08774, 2023.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring. Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro. Jacob L. Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karén Simonyan. Flamingo: a visual language model for few-shot learning. In NeurlPS, 2022.
Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Yitzhak Gadre, Shiori Sagawa, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt. Openflamingo: An open-source framework for training large autoregressive vision-language models. CoRR, abs/2308.00 0133990 2023.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou,, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. CoRR, abs/2308. 12966, 2023.
Gedas Bertasius, Heng Wang, and Lorenzo Torresani. space-time time attention all you need for video understanding? In Marina Meila and Tong Zhang (eds.), ICML, 2021.
Jingyuan Chen, Xinpeng Chen, Lin Ma, Zequn Jie, and Tat-Seng Chua. Temporally grounding natural sentence in video. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun' un'ichi Tsujii (eds.), ACL, 2018
Jingyuan Chen, Lin Ma, Xinpeng Chen, Zequn Jie, and Jiebo Luo. Localizing natural language in videos. In AAAI, 2019.
Ting Chen, Saurabh Saxena, Lala Li, Tsung-Yi Yi Lin, David J. Fleet, and Geoffrey E. Hinton, A unified sequence interface for vision tasks. In NeurlPS, 2022.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. CoRR, abs/21 14164, 2021.
Ekin D. Cubuk, Barret Zoph. Jonathon Shlens, and Quoc V. Le. Randaugment: Practical automated data augmentation with reduced search space. In CVPR Workshops, 2020.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven C. H. Hoi. Instructblip: Towards general-purpose vision- language models with instruction tuning. In NeurIPS, 2023.
Jiajun Deng, Zhengyuan Yang. Tianlang Chen, Wengang Zhou, and Houqiang Li. Transvg: End-to- end visual grounding with transformers. In ICCV, 2021.
David Ding, Felix Hill, Adam Santoro, Malcolm Reynolds, and Matt M. Botvinick. Attention over learned object embeddings enables complex visual reasoning. In NeurlPS, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
Danny Driess, Fei Xia. Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence. Palm-e: An embodied multimodal language model. CoRR, abs/2303.00378. 2023.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima. Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling. CoRR, abs/2101. .000270 2021.
10

---

Published as a conference paper at ICLR 2024
Gaël Gendron, Qiming Bao, Michael Witbrock, and Gillian Dobbie. Large language models not abstract reasoners. CoRR, abs/2305.19555, 2023.
Rohit Girdhar and Deva Ramanan. CATER: A diagnostic dataset for compositional actions & temporal reasoning. ICLR, 2020.
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fründ, Peter Yianilss, Moritz Mueller-Freitae Florian Hoppe, Christian Thurau, Ingo Bax, and Roland Memisevic. The "something something" video database for learning and evaluating visual common sense. In ICCV, 2017.
Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training. CoRR, abs/2211.11559, 2022.
Dongliang He, Xiang Zhao, Jizhou Huang, Fu Li, Xiao Liu, and Shilei Wen. Read, watch, and move: Reinforcement learning for temporally grounding natural language descriptions in videos. In AAAI, 2019.
Ronghang Hu, Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Kate Saenko. Learning to reason: End-to-end module networks for visual question answering. /CCV, 2017.
De-An Huang, Shyamal Buch, Lucio M. Dery, Animesh Garg, Li Fei-Fei, and Juan Carlos Niebles. Finding "it": Weakly-supervised reference-aware visual grounding instructional videos. In CVPR, 2018.
Drew A. Hudson and Christopher Manning. Compositional attention networks for machine reasoning. In ICLR, 2018.
Jinhyun Jang, Jungin Park, Jin Kim, Hyeongjun Kwon, and Kwanghoon Sohn. Knowing where to focus: Event-aware transformer for video grounding. CoRR, abs/2308.06947, 2023.
Yang Jin, Yongzhi Li, Zehuan Yuan, and Yadong Mu. Embracing consistency: A one-stage approach for spatio-temporal video grounding. In NeurlPS, 2022.
Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. MDETR modulated detection for end-to-end multi-modal understanding. In ICCV, 2021.
Will Kay. João Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisserman. The kinetics human action video dataset. CoRR, abs/1705.0695. 2017.
Jing Yu Koh, Ruslan Salakhutdinov, and Daniel Fried. Grounding language models images for multimodal generation. CoRR, abs/2301.13823 2023.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML., 2023a.
Kunchang Li, Yinan Hc, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. CoRR, abs/2305.06635. 2023b.
Mengze Li, Han Wang, Wenqiao Zhang, Jiaxu Miao, Zhou Zhao, Shengyu Zhang, Wei Ji, and Fei Wu. WINNER: weakly-supervised hierarchical decomposition and alignment for spatio-temporal video grounding. In CVPR, 2023c.
Zhuowan Li. Elias Stengel-Eskin, Yixiao Zhang, Cihang Xie, Quan Tran, Benjamin Van Durme, and Alan L. Yuille. Calibrating concepts and operations: Towards symbolic reasoning on real images. In ICCV, 2021.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
Pan Lu, Baolin Peng, Hao Cheng. Michel Galley, Kai- Chang, Ying Nian Wu, Song- Chun Zhu, and Jianfeng Gao. Chameleon: Plug-and-plat compositional reasoning with large language models. CoRR, abs/2304.09842, 2023.
11

---

Published as a conference paper at ICLR 2024
Gen Luo, Yiyi Zhou, Xiaoshuai Sun, Liujuan Cao, Chenglin Wu, Cheng Deng, and Rongrong Ji. Multi-task collaborative network for joint referring expression comprehension and segmentation. In CVPR, 2020.
Ruipu Luo, Ziwang Zhao, Min Yang, Junwei Dong, Minghui Qiu, Pengcheng Lu, Tao Wang, and Zhongyu Wei. Valley: Video assistant with large language model enhanced ability. CoRR, abs/2306.07207, 2023.
Muhammad Maaz, Hanoona Abdul Rasheed, Salman H. Khan, and Fahad Shahbaz Khan. Video- chatgpt: Towards detailed video understanding via large vision and language models. CoRR, abs/2306.05424, 2023.
Shweta Mahajan and Stefan Roth. Diverse image captioning with context-object split latent spaces. In NeurlPS, 2020.
Joanna Materzynska, Tete Xiao, Roei Herzig, Huijuan Xu, Xiaolong Wang, and Trevor Darrell. Something- Compositional action recognition with spatial-temporal interaction networks. In CVPR, 2020.
Shanka Subhra Mondal, Taylor Webb, and Jonathan D. Cohen. Learning to reason over visual objects. CoRR, tbs/2303.02260. 2023.
Mathew Monfort, Carl Vondrick, Aude Oliva, Alex Andonian, Bolei Zhou, Kandan Ramakrishnan, Sarah Adel Bargal, Tom Yan, Lisa M. Brown, Quanfu Fan, and Dan Gutfreund, Moments in time dataset: One million videos for event understanding. IEEE Trans. Pattern Anal. Mach. Intell., 42 (2):502-508, 2020.
Zhiliang Peng. Wenhui Wang, Li Dong. Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. CoRR, abs/2306. 14824, 2023.
Nasim Rahaman, Muhammad Waleed Gondal, Shruti Joshi, Peter V. Gehler, Yoshua Bengio, Francesco Locatello, and Bernhard Schölkopf. Dynamic inference with neural interpreters. In NeurlPS, 2021.
Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Trans. Pattern Anal. Mach. Intell., 39(6): 1137-1149, 2017.
Adam Santoro, David Raposo, David G. T. Barrett, Mateusz Malinowski, Razvan Pascanu, Peter W. Battaglia, and Tim Lillicrap. A simple neural network module for relational reasoning. In NeurIPS, 2017.
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilie, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang, Benoit Sagot, Niklas Muennighoff, Albert Villanova del Moral, Olatunji Ruwase, Rachel Bawden, Stas Bekman, Angelina McMillan-Major, Iz Beltagy, Huu Nguyen, Lucile Saulnier, Samson Tan, Pedro Ortiz Suarez, Victor Sanh, Hugo Laurençon, Yacine Jernite, Julien Launay, Margaret Mitchell, Colin Raffel, Aaron Gokaslan, Adi Simhi, Aitor Soroa, Alham Fikri Aji, Amit Alfassy, Anna Rogers, Ariel Kreisberg Nitzav, Canwen Xu, Chenghao Mou, Chris Emezue, Christopher Klamm, Colin Leong, Daniel van Strien, David Ifeoluwa Adelani, and et al. BLOOM: A 176b- parameter open-access multilingual language model. CoRR, 0bs/2211.051111 2022.
Aviv Shamsian, Ofri Kleinfeld, Amir Globerson, and Gal Chechik. Learning object permanence from video. In ECCV, 2020.
Jing Shi, Jia Xu, Boqing Gong, and Chenliang Xu. Not all frames are equal: Weakly-supervised video grounding wit contextual similarity and visual clustering losses. In CVPR, 2019.
Karen Simonyan and Andrew Zisserman. Two-stream convolutional networks for action recognition in videos. Zoubin Ghahramani, Max Welling, Corinna Cortes, Neil D. Lawrence, and Kilian Q. Weinberger (eds.), NeurlPS, 2014.
12

---

Published as a conference paper at ICLR 2024
Florian Strub, Mathieu Seurin, Ethan Perez, Harm de Vries, Jérémie Mary, Philippe Preux, Aaron C. Courville, and Olivier Pietquin. Visual reasoning with multi-hop feature modulation. In ECCV, 2018.
Rui Su, Qian Yu, and Dong Xu. Stygbert: A visual-linguistic transformer based framework for spatio-temporal video grounding. In ICCV, 2021.
Chen Sun, Calvin Luo, Xingyi Zhou, Anurag Arnab, and Cordelia Schmid. Does visual pretraining help end-to-end reasoning? CoRR, abs/2307. 008566, 2023.
Dídac Surís, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reasoning. CoRR, abs/2303.08125 2023.
Siyu Tang, Mykhaylo Andriluka, Bjoemn Andres, and Bernt Schiele. Multiple people tracking by lifted multicut and person re-identification. CVPR, 2017.
Hugo Touvron, Thibaut Lavril, Gautier Izacard. Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302.13971, 2023.
Manuel Traub, Sebastian Otte, Tobias Menge, Matthias Karlbauer, Jannik Thümmel, and Martin Butz. Learning what and where unsupervised disentangling location and identity tracking. In ICLR, 2023.
Arun Balajee Vasudevan, Dengxin Dai, and Luc Van Gool. Object referring in videos with language and human gaze. In CVPR, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NeurlPS, 2017.
Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang. Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, Sen Xing, Guo Chen, Junting Pan, Jiashuo Yu, Yali Wang, Limin Wang, and Yu Qiao. Internvideo: General video foundation models via generative and discriminative learning. CoRR, abs/2212.03191, 2022.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In NeurlPS, 2022.
Bo Wu, Shoubin Yu, Zhenfang Chen, Josh Tenenbaum, and Chuang Gan. STAR: A benchmark for situated reasoning in real-world videos. In Joaquin Vanschoren and Sii-Kit Yeung (eds.), NeurIPS, 2021.
Junbin Xiao, Angela Yao, Yicong Li, and Tat-Seng Chua. Can trust your answer? visually grounded video question answering. CORR, abs/2309.01127,, 2023.
Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Tubedetr: Spatio- temporal video grounding with transformers. In CVPR, 2022.
Zhengyuan Yang. Boqing Gong, Liwei Wang, Wenbing Huang, Dong Yu, and Jiebo Luo. A fast and accurate one-stage approach to visual grounding. In ICCV, 2019.
Zhengyuan Yang, Tianlang Chen, Liwei Wang, and Jiebo Luo. Improving one-stage visual grounding by recursive sub-query construction. Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), ECCV, 2020.
Yiqun Yao, Jiaming Xu, Feng Wang, and Bo Xu. Cascaded mutual modulation for visual reasoning. In EMNLP, 2018.
Mang Ye, Jianbing Shen, Gaojie Lin, Tao Xiang, Ling Shao, and Steven C. H. Hoi. Deep learning for person re-identification: survey and outlook. IEEE Trans. Pattern Anal. Mach. Intell., 44(6): 2872-2893, 2022.
13

---

Published as a conference paper at ICLR 2024
Shoubin Yu, Jaemin Cho, Prateek Yadav, and Mohit Bansal. Self-chained image-language model for video localization and question answering. CoRR, abs/2305.06990 006988, 2023.
Runhao Zeng, Haoming Xu, Wenbing Huang, Peihao Chen, Mingkui Tan, and Chuang Gan. Dense regression network for video grounding. In CVPR, 2020.
Chi Zhang, Baoxiong Jia, Mark Edmonds, Song-Chun Zhu, and Yixin Zhu, ACRE: abstract causal reasoning beyond covariation. In CVPR, 2021.
Chi Zhang, Sirui Xie, Baoxiong Jia, Ying Nian Wu, Song-Chun Zhu, and Yixin Zhu. Learning algebraic representation for systematic generalization abstract reasoning. In ECCV, 2022a.
Da Zhang, Xiyang Dai, Xin Wang, Yuan-Fang Wang, and Larry S. Davis. MAN: moment alignment network for natural language moment retrieval via iterative graph adjustment. In CVPR, 2019.
Hao Zhang, Aixin Sun, Wei Jing, and Joey Tianyi Zhou. Span-based localizing network for natural language video localization. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (eds.), ACL, 2020.
Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. adapter: Efficient fine-tuning of language models with zero-init attention. CoRR, abs/2303.16199, 2023a.
Shiwen Zhang. Tfcnet: Temporal fully connected networks for static unbiased temporal reasoning. CoRR, 2022.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona T. Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. OPT: open pre-trained transformer language models. CoRR, abs/2205.0100 .01068, 2022b.
Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Multimodal chain-of-thought reasoning in language models. CoRR, abs/2302.00923, 2023b.
Honglu Zhou, Asim Kadav, Farley Lai, Alexandru Niculescu-Mizz Martin Rengiang Min, Mubbasir Kapadia, and Hans Peter Graf. Hopper: Multi-hop transformer for spatiotemporal reasoning. In ICLR, 2021.
14

---

Published as a conference paper at ICLR 2024
APPENDIX
A OVERVIEW
Here we provide, Results on a "generalist" LRR model. 2. Surrogate tasks and overfitting. 3. Additional training details including hyper-parameters used for evaluation in Section 4 of the main paper across ACRE, Something-Eles and CATER datasets. 4. Additional experiments to compare our LRR approach to Schain-of-thoughee (Wei et al. (2022) based approaches. 5. Additional qualitative examples across ACRE, Something-Eii -llseinnng--eeeeeee Else and CATER datasets.
B GENERALIST MODEL
The results in the main paper show that we can leverage the reasoning capabilities of pre-trained LMs create "specialist" models with state of the art performance variety of visual reasoning tasks using end-to-end learning. Next, as both the input modality and the acquired visual skills are generic and in principle applicable across tasks, we train the model with 125M backbone jointly on the two video-based reasoning tasks discussed above, CATER and Something- -Else This a step towards developing a grounded "generalist" for performing reasoning videos. Despite commonalities in underlying low-level features, the diverse nature of reasoning objectives in the two tasks makes this highly challenging. As shown in Table our LRR (Joint) model outperforms even
the dataset specific fine-tuned state of the art models (SOTA) (Materzynska et al. 2020 Zhang 2022
Ding et al. 2021 on these highly diverse visual reasoning tasks.
Table 5. Evaluation of our LRR model trained jointly on CATER and Something Else.
Something-Eil Eise
CATER: Static Camera
CATER: Moving Camera
Method
Top If
Top.51
Top 11
Top 51
Top 1
Top 51
SOTA (ef. Tables and
56.2
81.3
79.7
95.5
59.7
90.1
LRR (Joint, Ours)
61.1
85.4
80.6
97.2
73.7
95.6
C SURROGATE TASKS AND OVERFITTING
To highlight that surrogate tasks prevent LRR model from overfitting, report training and test accuracy of our LRR model trained with and without surrogate tasks on the compositional split of Something Else and on the moving camera split of CATER in Table
The results show that our surrogate tasks clearly prevent overfitting as the model is grounded to the fine-grained low-level details and is thus able to better "understand" the task at hand.
D ACRE
Additional training details. We trained our LRR model with the OPT-125M and OPT-1.3B backbone until convergence (~ 500k iterations) with a batch size of 4. We use the AdamW opti- mizer (Loshchilov & Hutter 2019) with a learning rate of
5 ,1 0.9, B2 0.95 and A (weight decay) = 0.1 and gradient clipping with norm of 1.0.
As described in Section 3.4 of the main paper, we prompt our LRR model to solve certain surrogate tasks at randomly selected time steps. It is also possible to include surrogate tasks after every time-step. In this case, this would resemble a "Chain of Thought" Wei et al. (2022) like process, where the final answer would depend upon the surrogate tasks solved at inference time. However, this is very inefficient and thus impractical, especially for long video sequences. We compare both of these approaches in Table 7 LRR (Random 30%, Ours) where prompt our LRR model with the OPT-125M backbone with a probability of 30% an input video frame to solve surrogate tasks only during training. LRR (Every frame) where the model is prompted to solve surrogate tasks after every frame during training and inference. Note, that in case of the LRR (Every frame) model it is necessary solve surrogate tasks during inference as the (autoregressive LM) model always sees such tasks during training. We see from the results in Table 7 that the
Comparsion "Chain-of-thoughe
It may he possible include
ACRE treating images videos, leave for work.
15

---

Published as a conference paper at ICLR 2024
Table Results highlighting that LRR model prevents overfitting.
Something Else (Top 1 t)
CATER: Moving Camera (Top t)
Method
Train
Test
Train
Test
LRR (w/o Surrogate tasks)
92.6
50.1 62.0
99.4
62.7
LRR (Ours)
87.3
89.7
80.4
Table 7. Evaluation of our random prompting strategy on ACRE cff. Table 2 in the main paper.
Method
Compositional t Systematic f
Inference Speed (msee)
ALOE (Ding
2021
91.7
93.9
LRR (Random 30% during training. Ours)
98.2
99.2
61 1415
LRR (Every frame)
99.3
99.5
performance of LRR (Random 30% during training, Ours) comparable with LRR (Every frame) while the inference speed is order of magnitudes faster. We report the inference speed in milliseconds on single Nvidia A100 GPU. This is because our random prompting during training distills the relevant low-leve information into the hidden states of the LM backbone at feature level and thus does not require solving surrogate tasks inference time in a COT like fashion.
Additional qualitative examples. We include additional qualitative examples in Fig. hhighlighting surrogate tasks. These examples illustrate that using our surrogate re-identification task, our LRR model can aggregate information from multiple context trials to arrive at the final answer following the paradigm "Look, Remember, Reason".
Context Trials
Query
<re identifys 1mmedium purple rubber cube;
<re cre-identfy> I.medium pupple rubber 2. mediumm cyan rubber cylie-
<re-identify>
1mdeiun
<re-identify> medium Cyan rubber cylinder. off.
purple nubber cube <blicket>> on.
2. medium cyan rubber cylinder.
<hlicket>
canswers
cre-identff>> 3, medium
<re-identify>- 5, medium
<re identify>>
red metal cube; S.medium purple metal cube: 3.medium purpie meial sphere; Tmediun green
<re-identify> mediuun pupple metal cube; medium purple metal sphere: 6, mediam
purple metoll sphere: 4meedium brown metal cube; 5,medium purple
purple metal cube;
3.medium parple metal
metal cube; 6ymedium red
metai cube.
sphere: 6,medlum red
rubber yphere; 4.medium brown metal cube. on.
metal cube: 7. medium
metal cube.
<hlickets off.
green sphere. canswers maybe.
<blickets
<re identify> 3, medium rubber cube;
cre-identify> medium purple rubber cube: 6,nedium brown menal cylinder: 3. mediam eyan rubber cube.
<re cre-idenify>> mediium rubber cube;
<ro-identify> 3, medium
mediam purple rubber cube; medium cyon
medium gray rubber cube:
gray rubber cube. <blicket>
4, medium purple rubber cube; 6, medium brown cylinder. canswers yes.
rubber cube; 6,medium brown metal cylinder.
<blicket> off.
<blicket>
Figure 6. Example solutions surrogate tasks generated by our LRR model ACRE, Re-identified objects
across context trials are underlined in the same color.
16

---

Published as a conference paper at ICLR 2024
E SOMETHING-ELSE
Additional training details. We trained our LRR model with the OPT-125M and OPT-1.3B backbone until convergence (~ 700k iterations) with a batch size of We the AdamW opti- mizer (Loshchilov & Hutter 2019) with a learning rate of x 10 50- 31 = 0.9, B2 = 0.95 and (weiight decay) 0.1 and gradient clipping with norm 1.0. We also employ random data augmentation using RandAugment (Cubuk et al. 2020) with magnitude of 15 and label smoothing with 0.3, where applicable (for fairness).
Additional qualitative examples. The qualitative examples in Fig- show that our LRR model can deal with visually complex real-world scenarios such as severe occlusions, e-g., the silver ball in Fig. (bottom row) is occluded by the hand. Here in Fig. 7 we include additional qualitative examples further highlighting that our LRR model can deal with visually complex real-world scenarios such severe deformations, e-g-, the piece of paper in Fig. second row which is torn into pieces; severe changes in appearance, e.g., the pen in Fig. third row, the teal case the fifth row; motion blur in Fig. last row.
Finally, these examples illustrate that using our surrogate tracking task, our LRR model can aggregate information from multiple context trials arrive at the final answer following the paradigm of "Look, Remember, Reason".
F CATER
Additional training details. We trained our LRR model with the OPT-125M backbone until convergence (~~ 600k iterations) with batch size of 4. We use the AdamW optimizer (Loshchilov Hutter 2019) with a learning rate of 1 x 10-5, 31 = 0.9, 32 0.95 and À (weight decay) = 0.1.
use gradient clipping with norm of 1.0.
Additional qualitative examples. We include additional qualitative examples including both the static and moving camera splits Fig. highlighting the surrogate tracking task. We see that our LRR model can successfully deal with containment in both static camera (rows 1-3; Fig. ( and moving camera (row 4-5; Fig. S) settings, due to our rationale that explicitly tracks cones and the snitch. Note that the example in row 5 in Fig. from the moving camera split is especially challenging due to recursive containment.
17

---

Published as a conference paper at ICLR 2024
FaPE
Answer: Pretending or failing to wipe [something] off of [something]-
Obnet
Answer: Tearing [something] into two picces.
Osject
abiecr
Object
nand
Answer: Pushing [something] so that almost falls off but doesn't.
Answer: Pushing [something] from right to left.
hand
1and
CONES
Answer: Poking [something] so that it falls over.
Ogect
Object'
Oayoct
Obeer OR
Answer: Pushing [something] with [something].
Oeaec
Answer: Putting [something] next to [something).
Figure 7. Example solutions to surrogate tasks generated by our LRR model on Something -lls,,, highlighting object tracks and bounding boxes
18

---

Published as a conference paper at ICLR 2024
29, 17
35, 20, 20
20, 20, 20
12, 20
9, 34, 32
32, 3, 32
19, O, 19
7, 2, 19
28, 12, 15, 1
1, 18, 29,
9,7, 23, 9
34, 6, 23, 34
4, 20, 29,
23, 9, 22, 11
23, 9, 22, 11
15, 4. 11
22, 10, 22
22, 22
17, 17
17, 29, 17
Figure 8. Example solution to the tracking surrogate tasks generated by our LRR model on CATER. We show the predicted grid locations of the cones and the snitch below.
19


Reviews:
[{"review_id": "VUlU5E7go7", "reviewer": "Reviewer_tQdf", "summary": "This paper proposed a Look, Rember, and Reason (LRR) framework to solve video reasoning task. The structure utilized LM to extract visual information by using surrogate grouding tasks and then integrated the grouding information to arrive at a final answer. The authors propose a two-stream vide encoder to capture sccene structure and object motions to learn low-level skills. Experiments on two sythetic dataset and one real-world datset shows the effectiveness of proposed method on complex spatialtemporal and causal reasoning tasks in videos.", "questions": "See section weakness", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The proposed LRR structure make use of language models to solve surrage groudning task to benefit final video reasoning task. This structural design better utilize the low-level visual skills and information from videos.\n\n2. This paper conduct experiments on two synthetic datasets and one real-world dataset. The proposed methods achieve competative performance three datasets and outperforms other exsisting baseline models, showing the effectiveness of proposed method.\n\n3. Table 2 and 3 also show the performance of LRR without surrogate tasks and two-stream encoder. The ablation study shows the significance of the two proposed components for the overall structure.", "weaknesses": "1. Writing, section 3 and Figure 2 is a little unclear and hard to follow.\n2. For different surrogate tasks, where do the ground-truth answers such as localization or box come from?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xk7qQ5sIbi", "reviewer": "Reviewer_5Fo1", "summary": "The paper proposes a Look, Remember, Reason (LRR) framework to enable language models to perform visual reasoning in videos. The proposed LRR framework uses a two-stream video encoder to extract dense spatiotemporal features from video frames capturing structural and motion cues. The language model backbone has cross-attention layers inserted between its self-attention layers to enable top-down attention over the visual features. This allows the model to extract relevant visual information based on the reasoning task. LRR is trained end-to-end using surrogate tasks like object detection, re-identification and tracking. These provide supervision to teach the model the required low-level visual skills.\n\nThe authors also demonstrate that training LRR jointly on multiple datasets leads to a \"generalist\" model that performs competitively compared to task-specific \"specialist\" models. In the experimental results, the authors demonstrate that the LRR models significantly outperform prior state-of-the-art on challenging visual reasoning tasks from the ACRE, Something-Else, and CATER datasets, showing the benefit of the proposed grounded reasoning approach.", "questions": "1) Could you comment on the nature of surrogate tasks? Are there some tasks that are more suited for reasoning vs others. Do low level recognition tasks (choice in the paper) work better.\n2) Is there evidence that LRR is not ovefitting to these simplistic datasets due to surrogate tasks?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Demonstrates strong performance on multiple challenging visual reasoning datasets by grounding the language model in low-level visual details.\n+ Good demonstration of using surrogate tasks and end to end training", "weaknesses": "- The datasets used are rather simple with low visual complexity, such as CATER.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8Efbr9kDE1", "reviewer": "Reviewer_4932", "summary": "Thank you for submitting your manuscript. The proposed three-step process of Look, Remember, Reason for training a Language Model (LM) end-to-end on low-level surrogate tasks, which include object detection, re-identification, and tracking, is indeed novel.", "questions": "see Weaknesses", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The approach presented is intriguing and demonstrates significant performance improvements.", "weaknesses": "1-Throughout the paper, there's a recurring mention of \"low-level surrogate tasks\". Could the authors elucidate the definition of these low-level tasks? Moreover, how do they differ from high-level tasks?\n\n2-The Look, Remember, Reason (LRR) model framework is innovative. However, there seems to be a gap in explicitly correlating this framework with the actual operations carried out in the method. The unique contributions of the \"Remember\" and \"Reason\" steps, in particular, are not clearly highlighted. It would be beneficial for the readers if the authors can provide a clearer mapping of these steps to their corresponding operations.\n\n3-Will the codebase for the presented method be made publicly available?\n\n4-Regarding the results of Video-ChatGPT on the Something-Else dataset: Were these results replicated by the authors? I couldn't find a direct reference to such results in the original Video-ChatGPT paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HXmSYIy3u7", "reviewer": "Reviewer_Fa1W", "summary": "The authors claim that they propose training an LM end-to-end on low-level surrogate tasks, including object detection, re-identification, and tracking, to endow the model with the required low-level visual capabilities.", "questions": "Please refer to Weakness.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors claim that they propose training an LM end-to-end on low-level surrogate tasks, including object detection, re-identification, and tracking, to endow the model with the required low-level visual capabilities.", "weaknesses": "1. In the experiments, the authors primarily focus on conducting investigations using synthetic datasets, particularly the ACRE dataset. However, it raises concerns about the generalizability of the conclusions/findings obtained from synthetic datasets to real-world datasets.\n\n2. The experimental results primarily focus on classical models. However, the generalizability of the conclusions/findings derived from these classical models to more powerful transformer-based models, such as the models mentioned in *Related Work* part, remains a concern.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)