Learning to Reason Iteratively and Parallelly for Complex Visual Reasoning Scenarios
Authors: Shantanu Jaiswal
Debaditya Roy
Basura Fernando
Cheston Tan
Venue: NeurIPS 2024 poster (2024)
Keywords: iterative and parallel computation; complex visual reasoning and question answering; neural network based reasoning architectures
Primary area: deep_learning_architectures
TL;DR: We introduce a fully neural reasoning mechanism comprising iterative & parallel computation to address complex image & video reasoning tasks such as AGQA, STAR, CLEVR-Humans and CLEVRER-Humans.

Abstract: Complex visual reasoning and question answering (VQA) is a challenging task that requires compositional multi-step processing and higher-level reasoning capabilities beyond the immediate recognition and localization of objects and events. Here, we introduce a fully neural Iterative and Parallel Reasoning Mechanism (IPRM) that combines two distinct forms of computation -- iterative and parallel -- to better address complex VQA scenarios.  Specifically, IPRM's "iterative" computation facilitates compositional step-by-step reasoning for scenarios wherein individual operations need to be computed, stored, and recalled dynamically (e.g. when computing the query “determine the color of pen to the left of the child in red t-shirt sitting at the white table”). Meanwhile, its  "parallel'' computation allows for the simultaneous exploration of different reasoning paths and benefits more robust and efficient execution of  operations that are mutually independent (e.g. when counting individual colors for the query: "determine the maximum occurring color amongst all t-shirts'"). We design IPRM as a lightweight and fully-differentiable neural module that can be conveniently applied to both transformer and non-transformer vision-language backbones. It notably outperforms prior task-specific methods and transformer-based attention modules across various image and video VQA benchmarks testing distinct complex reasoning capabilities such as compositional spatiotemporal reasoning (AGQA), situational reasoning (STAR), multi-hop reasoning generalization (CLEVR-Humans) and causal event linking (CLEVRER-Humans). Further, IPRM's internal computations can be visualized across reasoning steps, aiding interpretability and diagnosis of its errors.

Learning to Reason Iteratively and Parallelly for Complex Visual Reasoning Scenarios
Shantanu Jaiswal 1.2
Debaditya Roy²
Basura Fernando Cheston Tan
1 Carnegie Mellon University 2 IHPC, A Singapore 3 Centre for Frontier AI Research, SSSRR Singapore Correspondence sj sjaiswa3dcs cmu.edu
Abstract
Complex visual reasoning and question answering (VQA) is a challenging task that requires compositional multi-step processing and higher-level reasoning capa- bilities beyond the immediate recognition and localization objects and events. Here, introduce fully neural /terative and Parallel Reasoning Mechanism (IPRM) that combines two distinet forms of computation - iterative and parallel- better address complex VQA scenarios. Specifically, IPRM's "iterative" com- putation facilitates compositional step-by-step reasoning for scenarios wherein individual operations need be computed, stored, and recalled dynamically (e.g. when computing the query "determine the color of pen to the left of the child in red t-shirt sitting at the white table"). Meanwhile, its "parallel" computation allows for the simultancous exploration of different reasoning paths and benefits more robust and efficient execution of operations that are mutually independent (e.g. when counting individual colors for the query: "determine the maximum occurring color amongst all t-shirts"). We design IPRM as a lightweight and fully-differentiablee neural module that can be conveniently applied to both trans- former and non- non-trans transformer vision-language backbones. It notably outperforms prior task-specific methods and transformer-based attention modules across various image and video VQA benchmarks testing distinct complex reasoning capabilities such compositional spatiotemporal reasoning (AGQA), situational reasoning (STAR), multi hop reasoning generalization (EEVR-Huans)) and causal event linking (CLEVRER SEEV-RERRRHum)) Further, IPRM's internal computations can be visualized across reasoning steps, aiding interpretability and diagnosis of its er- rors. Source code at: https://gii thub. com/ httm//saanannn/IPR... Itttll_ttraiiie Lnd_PParaleel_Her Reasoninnnnnnninn Machnng_ehhhninnne
Introduction
Visual reasoning and question answering (VQA) at its core requires model identify relevant visual operations, execute those operations, and then combine their results make inference. Complex visual reasoning scenarios (depicted in fig. M are particularly challenging in this regard. They require models reason compositionally over large number of reasoning steps and to engage a variety higher-level reasoning operations such as causal linking, logical reasoning, and spatiotemporal processing that extend beyond powerful computational priors exist iterative and parallel. While each has its own limitations, when combined, they can complement each other and effectively address the challenges of complex VQA tasks. Specifically, iterative computation, wherein individual operations are identified and composed in step-by-step manner, beneficial prior for multi-step reasoning scenarios explored by past VQA works 2
perception capabilities. In this context, two
25 18]
However, pure iterative computation can exhibit limitations in
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

What is the color of small object in front green objectt with the max occurrin shape?
Are both the ball to the right of other balls and the black holmot made plastic?
collision eyan and cylinder responsible for collision biw gray cylinder and eyan cube?
they put camera before the longest occuring action?
What they do complating current action?
Figure 1: Complex VQA scenarios (CLEVR-Humanl 35 GQA AGQA 20 and STAR(79)) wherein combination of iterative sstep-by-step computation (blue phrases) and parallel computation (orange phrases) can be beneficial for reasoning.
29 CLEVRER-Humang (5I).,
scenarios wherein the entailed visual operations are independent of one-another, where distinct stimuli need to be processed and tracked simultaneously.
For example, consider the first scenario shown in fig. 11 When executing the language phrase "maximum occurring shape" " (i.e. "what shape appears the most") ) purely iterative method would: (i) compute the count of each shape (each of which itself could take multiple iterations), (ii) then update and maintain the counts in memory (without forgetting count all previous shapes), and (iii) finally, recall each shape's count to compute the "maximum" " Besides taking more reasoning steps than required, such computation also increases the demand for information retention and recall in memory, which in this scenario could scale by the number of shapes to be counted. In complex video reasoning scenarios, a purely iterative method would similarly struggle in tracking and reasoning multiple co-occurring events. In such scenarios, from both efficiency and efficacy perspectives, is advantageous to process operations or stimuli in parallel, rather than solely iteratively.
More generally, parallel computation facilitates the simultancous maintenance and exploration of distinet reasoning paths, and thereby enables reasoning to be more comprehensive, efficient and robust. For example, to compute "maximum occuring shape", parallel compute can enable distinet shape queries to be simultaneously computed prior to computing the "maximum" " operation. Similarly, it is effecctive for other scenarios illustrated in fig. [ such as in processing independent logical clauses (" ("[are X) and (Y made of plastic/") or when tracking and processing co-occurring events videos.
Such computation can be implicitly realized in conventional transformer-based parallel attention mechanisms [70]. However, transformer- based attention does not explicitly incorporate iterative com- positional computation [4] A1I, which as described is beneficial for multi-step reasoning scenarios wherein operations need be composed sequentially. Accordingly, while parallel computation may effectively compute the result of "maximum occurring shape" in fig. [ it would potentially struggle to integrate the result with further operations such as "green object with.. " "small object in front of green and "color of "hht need to be computed step-by-step to answer the question.
Based on the above insights, design the Iterative and Parallel Reasoning Mechanism (IPRM), a novel neural reasoning architecture that combines step-by-step iterative computation with the ability process multiple independent operations and stimuli simultaneously. Inspired by how humans utilize working memory I7 9 to facilitate complex reasoning, IPRM internally maintains latent memory of parallel "operation states", keyed to which are "results states". Given vision and language inputs. IPRM performs the following iterative computation. First, it forms new set of parallel operations by retrieving relevant language information conditioned on its prior operation states. Then, it "executes" these operations parallelly by retrieving relevant visual information conditioned on its new operations as well as prior result states. Finally, it integrates its new operations (and their results) into memory by dynamically composing these operations with me-annothr well prior operation states, and subsequently, repeats the entire process in its next iterative step.
This strategy effectively enables us to take advantage of both parallel and iterative computations and notably helps improve ofate-of-arts across various complex image and video reasoning tasks using a single reasoning mechanism. Equally importantly, IPRM's internal computations can be visualized
AAssuming "mar" applied after counts of all shapes computed.

---

Lang. Input (Xt)
Vis. Input (Xv)
repeat
T
Mast
May-
KV
Zrpt
Ky Frs Bag Merel)
Map
Operation Composition
Q
Mapa
Mrea
Map
Zaya
Maz
Mrua
Mapo
M
May
Zipii
Drealt
Mapll
Mecot
Memory (M4)
Operation Formation
Operation Execution
Mem. state (Mas))
Figure 2: IPRM's computation flow diagram. First, new set of N- N-parallel latent operations Zop are retrieved from language features Xr. conditioned on prior operation states Map- Then, visual features X queried conditioned on both Zop and prior result states results M to form the new results Zres Finally, both Zrs and Zop are passed to the Operation Composition Unit (see the output of which becomes the new memory state M.
across reasoning steps, which helps better interpret what operations it was doing and accordingly where it was looking visually when processing a complex reasoning scenario.
2 Iterative and Parallel Reasoning Mechanism
Our proposed iterative-and parallel-reasoning mechanism (IPRM) is fully-differentiable neural architecture. Given visual features Xy Ry DD and language or task-description features XL E ND IPRM outputs "reasoning result" ys E RDDm and, optionally, set of "reasoning result tokens" " YR N. x Dm, As previously mentioned, IPRM operates iteratively for T' reasoning steps and internally, maintains explicit memory M (Mop- Mres). The memory is modelled as set of "operation states" Mop XD. keyed to which are "result states" Mres E oo x as shown fig. 2 Here, Nop denotes the number of parallel operations be computed while D. denotes the mechanism's internal feature dimension. On high level, at each reasoning step (denoted by t (1....,T)) IPRM performs the following computations:
1. First, conditioned on the existing operation states M we retrieve relevant information from language or task-description features Xr. to form a new set latent operations Zop,t D We term this computation as "Operation Formation".
Zop.t For_FormXXxx Mop.t)
2. Then, conditioned on the latent operations Zop.t and the existing result state Mres.t, we attend and retrieve relevant information from visual features Xv which represents new set of latent results Zres,t E R Xoy corresponding to Zop.t. We term this computation as "Operation Execution".
Zres,t Op Exp_Exee[vv: Zoppe. Mres.t.)
(2)
3. Finally, facilitate interaction amongst parallel operations, perform inter-operation k.; attention. Here, each operation Zop_k.;; E (1,.... NNoph h is composed with other oper- ations in Zop,t as well as prior operation states Mopit- W:t] within a lookback-window W. The corresponding result Zres is similarly composed with other results Zres.t and prior result states denoted as Mres :):)-- We term this computation "Operation Composition"
M
Zres.t) M(tt-)..e))
As shown in eq. 3. this output is the memory state M441
(Mop- Mres]:
The overall computation flow is illustrated in fig. 2 and we provide specific details and intuitions behind these computations in the following sub-sections.

---

2.1 Operation Formation
The "operation formation" stage conceptually models reasoner that based on its prior set of operations, decides what language features to retrieve in order to form the next set of relevant opera- tions. This can be effectively implemented through conventional attention mechanisms. Specifically, the cumulative set of prior operations (maintained in Mop,t) can be projected to form the 'query" Q1, E RNaap x Dm representing "what features to look for". The language features XL can be projected to form the 'key' KL E RN Pm and 'value' VL E R x Pm, Finally, the new set of latent operations Zop,t can be retrieved by computing attn QL, KL, VL). These steps are formally represented below:
(4) Here, WL E RDDm W WL RDnxDDm, WL.k RDxxDr and WL.v E RDmxD. Note KL and VL are not computation-sted dependent and only computed once. We use simple linear- modulated formulation (with appropriate broadcasting and projection weight Wa E RPxx)) to implement attn(.) (further details in appendix sec.
QL.t = WL.gg(Tanh) L. (Mop.t)) 1), KL = WL.k(XL), VL (L.
Kr Zop.t attn QI,t, KL, VL.
2.2 Operation Execution
In the "operation execution" " stage, the reasoner determines what visual features need to be retrieved depending on both the newly formed operations and existing result states. To model the constituent visual attention mechanism, we draw insights from cxisting recurrent visual reasoning methods that incorporate feature modulation for memory-guided attention. Specifically, we retrieve set of feature modulation weights Sv,t E IR Nop Dm./ through joint projection of the new operations Zop.t and prior results Mres.t as shown eq.
28 69
Sv, Wv.,([Wv (Zo,,t), ree (Mres.)))
(6)
Here, is a feature reduction ratio A 31
Sv, is then applied dimension wise projection
of Xx retrieve an intermediate attention key Kvv, E Ro XNxDDmrr The final attention key Kv. then obtained through joint multilayye-prrojccionn of Ky,, and the previously projected representation of Xv shown in eq.
Kvv: S O (Vv.)(Xvv, Kv, =
K
Finally, the attention query and value are formed through separate projections of Zop.t and Xv respectively. These are then fed together with Kv.t to the attention function to retrieve the new operation results Zres.t as shown in eq. 48. Intuitively, the overall process allows for both prior results and the new set of operations to jointly guide visual attention.
QV.+. Vv Wv.q (Zop,), Wv. (Xv), Zres,t attn(Qv.t, Vv.))
(8)
Here, Wv. E DDm/rxDm, Wv.res E Dm/rxDm W.s E DDm//x2DD/rr Wv.kI E RDn/ /rm//xD, Wv, E Pmmlrx2Dm/r Wv.ka E RDm//xDDirr Wv.q E RDm/rxDm and Wv. E TR. Dm x
2.3 Operation Compusition
Finally, in the "operation composition" stage, the reasoner first integrates the executed operations Zop.t and their results Zrea,t into the existing memory state M, through a simple recurrent update as shown in eqs. O and (TO.. Then, to mitigate redundancy amongst parallel operations and to retrieve relevant knowledge from prior-step operations, it dynamically composes individual operation states M' M with other operation states in M' opp and also prior operation states in Map,t- W.t- Here, W is an attention look-back window.
This composition is achieved through computing inter-operation attention as illustrated in fig. Specifically, Mop.t+1 is projected to obtain set queries Qop.t. while the token-wise concatenation of M' Mr op.t+1 and Mop.t-we a are projected to obtain the operation attention keys Kop,t and values V op.t: A second set of values V Vres.t also formed through projection of respective result states

---

as shown in eq. (4) Further, an identity attention mask IN.p is used to ensure that operations in Qop.t. can only attend to other operations and not themselves. This done to enable higher degree of operation composition. As shown in eq. 15 Qop.t. Kop.t. V Vop.t and IN are passed to the attention operation, which outputs an intermediate representation M. and the softmaxed- attention weights Aop.t. M oop,t+2 t-+1 is then added to a projection of M
Mop.t+1 fop.t+1 to effectively combine
attended operation states with the original operation states, and thereby form the next mem. operation state Mop,tt+.
Finally, the next result states obtained by applying Aop,t V r and then adding projection of Mresstt+1 as shown in eq. (17 Note Aop,t is specifically utilized to ensure that results are composed based on attentions between operation states. Here, all the mentioned weights W E RR... P... and
[:: represents token-wise concatenation. IN.p in eq. (15 an identity matrix which is concatenated with zeros (i.e. unmasked) for window tokens if window len. W 0.
op.t+1 Wopu (Zop.t) + W WopH WopH(Mop.)) (op.tt
(9)
resstt+1 Wrest (Zrox,t) (resu(Zres.t) W WrsH (Mres,t)
(10)
Qop.t W Wop.q
(11)
Kop,t Wo (M t1 Mop,t- w:+)
(12)
Vop.t Wop.v MM Mop.t-W:tt
(13)
Vres, = Wres
M Mres.t-Wit))
(14)
M op.t+1 App.t = attn Qop.t: Kop.t, Vop,t. mask=INop)
(15)
Mop.t+1 M op,t++ + W Wop.u2(M
(16)
Mres = Aop.t(Vrestl (es,,t + ve MM res.t+1)
(17)
Obtaining Reasoning Summary mentioned before, our proposed mechanism outputs a set of "reasoning result tokens" Y and "reasoning result" ys. YR is simply equivalent the last memory result states Mres,T+D To obtain Ys. we perform attention on the last operation states Mop, T+1 by utilizing a summary representation ls Di of X1. as the attention-query.
We set la to be the first token in case of transformer based language backbones and as last hidden state case of SSTM-basee language backbones. As shown in eq. 18). 1g is projected to obtain a single-token attention query Pq while M 1 is projected obtain the attention keys Pk- The attention value is simply the result states Mres.T+1. and the output of the attention function is the "reasoning result". Intuitively, this computation corresponds to the reasoner deciding which final oper- ation states in Mop. T+1 are most relevant to the summary of the input language task-description XL, based on which corresponding result states M T+1 are weighted and retrieved.
Mr
M
KVpc Mayil
M
Q
Map
Aup (seif maskad))
W
MapNN
Mag
cupy Avg
M
Asp
W
M
Pq: Pk = W Wpg.((.)) W Pk.k (Mo..T))) ys attn(Pq, Pk, Mres.T+1
(18)
(19)
M
M (Messtiiin
Figure 3: Operation
Here, Wpq.q E P.. DD and W Wpk.k E Pm..D...
Composition Unit
Reasoning mechanism general applicability. Our proposed iterative and parallel reasoning mechanism is an end-to-end trainable neural module. It can be conveniently applied on top of different vision and language backbones, and trained directly as a new computational block with no specific adjustments. Further, IPRM is weight-tied which means that its number of parameters is constant regardless of number of computation steps and parallel operations. We provide parameter and computational details along with module implementation in appendix sec. C
5

---

Table 1: Comparison of IPRM with videoQA methods STAR (left) and AGQAv2 (right). All
methods operate on 32 frames unless otherwise mentioned in O. *oot directly compared as utilizes additional surrogate tasks benchmarks and num. of frames not reported.
Model
Int.
Seq.
Pred.
Feas.
Avg.
LRK
73.7
71.3
65.1
70.3
LRR (w/o surrogale)
54.5
48.7
44,3
45.5
48.2
Metri
HCRN 190
A10
Tciup
XBST
GF
IPAM
All-in-One MM
47.5
50.8
47.7
44,0
47.5
objenel
supertativc
40.3
33.6
48.3
57.5
50.
19.8
42.1
51.7
55.0
44.6
51.8
48.0
Temp[ATP
50.6
52.8
49.3
40.6
48.3
49.7
49.5
48.3
67.2
53.2
75.6
MIST [8
55.5
54.2
54.2
44.4
$1.1
caist
50.0
50.8
51.8
60.3
99.1
62.4
InteroVideo (8)
62.7
54,9
51.9
ALL rocoz.
dunsion
43.5
5.5
45.4
19.0
49.6
19.0
54.6
19.7
52.8
14.2
50.7
20,0
SeViLA-BLIP2 No
63.7
70.4
63.1
62.4
64.9
open
36.3
50.6
56.1
58.6
Concat-Att--l
71.4
66.6
55.2
bmary
43.0
58.5
54.2
62.3
Cross-Att-4L
67.5
72.1
64.4
58.5
all
42.1
48.6
49.8
54.4
55.1
60.4
IPRM
71.8
77.7
71.0
39.1
69.9
3 Experiments
We evaluate IPRM STAR 79
AGQAv2 20 and CLEVRER-Humans 51] for video reasoning
tasks and FLEVR-Humane K5 GQA 29 and CLEVR-CoGenT 34 for image reasoning tasks.
For all tasks, we set IPRM's s parallel operations (Nop) to 6, reasoning steps (7") to 9, reduction ratio (r) to 2 and window length (W) to 2 (informed by ablative analysis detailed in sec. 3.3 We follow task-specific practices (detailed in appendix C.I for respective vision and language backbones in our primary experiments, besides also demonstrating integration of IPRM with large-scale VL backbones such CLIP. Further, besides task task-specific methods, we also consider transformer-base VL modules as baselines - concat-att (where lang. and vis. tokens are concatenated and cross (where lang. tokens "query" to vis. tokens as "key" and "value"; as in Further implementation and training details are provided in appendix sec C
prominent
in 36
4
3.1
Video Reasoning and Question Answering (STAR, AGQAv2 and CLEVRER-Humand
We first evaluate IPRM on recent video reasoning benchmarks. STAR 70 and AGQAv2 comprise real-world vidcos and test multiple reasoning skills in context of situational reasoning and composi- tional spatiotemporal reasoning respectively. STAR contains 60K questions testing four broad types of video reasoning abilities: feasibility, interaction, prediction and sequence. Meanwhile, AGQAv2 contains 2.27M balanced questions distributed across 16 different question types. As shown in table I IPRM obtains 69.9% average acc. on STAR and 60.4% overall acc. on AGQAv2, outperforming prior -specfie methods by 5% both benchmarks.
Interestingly, on STAR IPRM obtains an 8% and 7% improvement over SeViLA-BLIP2 on predictive and sequencing scenarios respectively. This is possibly due to IPRM's capability to reason over multiple events simultancously across frames, which may enhance its capacity to retrieve and cumulatively reason relevant information needed predict future events and determine appropriate sequences. However, IPRM performs less effectively than ViLA--LIP in feasibility scenarios, possibly because these scenarios require not only visual reasoning but also commonsense knowledge, which can benefit from integration with larger-scale vision-language backbones and auxiliary training tasks such introduced in LRR (5].
Similarly, AGQAv2, IPRM improves performances across various question types, notably achiev- ing a 8% improvement in questions that require determining sequence of events. Further, IPRM also outperforms both 4-layer concat- and cross-attention modules on STAR (scaling further attention- layers not found to benefit performance detailed in appendix table
Next, evaluate IPRM on the CLEVRER- Humans benchmark (51) which comprises synthetic videos of simultaneous object motions and multiple collisions, and tests model's ability to determine causal links between events. We perform zero-shot, finetuned and from-scratch evaluation. As shown in table 2 IPRM outpeforms task task-specific neurosymbolic models NS-DR and VR-DP as well as Mfate-oftth-aa ALOE across the three settings. Specifically, IPRM improves zero-shot per-question acc. by 7%, finetuned per-question acc. by 18.8% and scratch per-question acc. by 6.2%. These results further suggest that IPRM can better track and process co-occuring events, and in this case, more accurately determine causal links.

---

Table 2: Comparison of methods for CLEVRER-
Table 3: Comparison of methods on CLEVR-
Humans [1 (Opt. is per option acc. and Qs. is
Humans (zero-shot and finetuned setting), CLEVR- CoGenT (ValA is in-domain; ValB is out-of- domain setting) and CLOSURE. IPRM performs strongly without requiring extra supervision.
per question acc.). IPRM achieves state-of-art
across settings.
Scratch
NS.DR K4
Opl.
51.0
32.0
Qs.
Opt.
Qs.
Opt.
Qs.
Midel
Extra supv.
CLV-lum
VaA
CEVe
Vala
CLOSURE
VROP
31.6
PGFEE F51
Frograms
54.0
66.5
96.6
73.2
75.5
CNNESTMGI
50.3
30.0
51.7
34.2
51.5
30.8
NS VQA
XJ
Programs
67.8
99.8
63.9
77.2
52.0
FIM
XT
Noue
$6.6
75.9
998.3
78.8
56.9
ALOF
54.0
26.9
51.8
31.7
52.7
32.1
MAC
Nore
57.4
81.5
99.0
78.3
73,8
IPRM
61.7
38,9
74.1
53.0
62.0
38.3
MOFTR
Round Box
50.9
81.7
99.8
76.7
76-
Concat, Att
85
98.5
Validation Accuracy (%)
$2.8
99.0
80.5
97.5
75.
97.0
IPRM (4.4M params)
Concat-Alt (12.6M)
Ctoss (16.8M)
MOEER (17.4M)
(5.BM)
95.5
Concat Att
-X-
0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 1.0
Training Data Ratio
Functionaa program langth
Functional program length
Figure 4: Acc. of IPRM (blue) across program lengths for CLEVR (left) and STAR (right). IPRM has signi- cantly higher aces, longer program lengths.
Figure 5: IPRM performance on at different training data ratios of Cross- and Concat-Att.
3.2 Compositional Image Reasoning (CLEVR-Humans CLEVR-CoGen and GQA)
Here, we evaluate IPRM on challenging compositional image reasoning benchmarks. CLEVR- Humans tests generalization of multi-hop reasoning to free-form human crowdsourced questions which entail reasoning skills/scenarios beyond model's training on the original CLEVR dataset and provides a limited finetuning set (2.5% of CLEVR). Similarly, CLEVR-CoGen' tests generalization on novel attribute compositions not observed in training (e.g."s "gray cubes" and "red cylinders" are in training but eval. is on "gray cylinders" and "red cubes"; see suppl. for exact specification). The CLOSURE t benchmark further tests systematic generalization for different question type compositions.
As shown in table performance over prior state-of-art vision-language model MDETR. Further, IPRM neither requires bounding-box pre-training supervision (as done MDETR) nor functional programs, and can be trained directly with only Dision-language inputs and task supervision. fig. illustrates IPRM's per- formance across different training ratios compared to MDETR and cross- and concat-att transformer modules. Notably, IPRM exceeds MDETR's 's fully- filly-fffnuued performance by 1.1% with only half of training data. Further, IPRM exhibits these improvements while being relatively lightweight (4.4M params) in comparison to MDETR's transformer blocks (17.4M; 44 more parameters) and cross- and concat- VL attention methods (16.8M and 12.6M respectively). These results suggest that IPRM exhibits strong generalization capabilities and more sample-efficien learning of novel reasoning skills and scenarios context of multi-step imageQA.
IPRM achieves 3.9% and 3.8% improvements in zero-shot and fully- fintt-finetuned
For CLEVR-CoGenT, IPRM achieves state-of-art results in out-of-domain generalization on novel attribute compositions and outperforms MDETR (having parallel transformer compute) by 3.6% and MAC (iterative method) by 2%. This suggests the combination of iterative and parallel computation as done IPRM can implicitly enable more disentangled feature processing and thereby improve compositional learning of primitive attributes in context of multi-step imageQA. Similarly, on CLOSURE, IPRM achieves an average zero-shot accuracy of 75.6% which is highest amongst fully neural reasoning methods (that require no extra supervision) and is close to the neurosymbolic method NS-VQA which utilizes ground truth programs and object bounding boxes supervision. Further, detailed breakdown of models CLOSURE question types is provided in appendix table
7

---

Table 4: Performance comparison GQA with imageQA methods and large-scale models that do not utilize ground-trutt truth scene graphs. indicaaes large-scale pretrained VL model. ** **Utilizes ground truth scene graphs, programs and bounding boxes for auxiliary training.
LCGN 25 55,8
8C
LXMERT 66
12-in-1*
49
OSCAR
46
CFR
3
IPRM 60.3
GQA
57.4
60.0
60.0
61.6
72.1
We also evaluate IPRM on the GQA benchmark which tests compositional VQA on real-world images. We perform comparisons with prior VQA methods as well as large-scale VI. models such as LXMERT and 12-in-1 that do not utilize the ground truth scene graphs in GQA. As shown in table IPRM achieves 60.3%. outperforming prior reasoning methods such MCAN and LCGN as well as large-scale VL models such as LXMERT and 12-in-1. However, IPRM's performance behind the larger VL model OSCAR which performs pretraining on 4.3M data samples collated from COCO, VG, SBU and Flickr. Note, IPRM is standalone reasoning module only trained on GQA balanced with no pre-training Further, IPRM when trained with perfect perception (i.e. ground truth object bounding boxes and attributes), achieves 87.2% on GQA validation set suggesting strong reasoning performances can achieved through advancements visual detectors and backbones. Finally, in appendix [E3 we demonstrate that IPRM more effectively enhances the performance of frozen CLIP variants on complex reasoning benchmarks compared to scaling traditional cross- and concaf- incat-attention modules.
Vol. Accuracy (55)
82.3
50
100
80
80
Val. Accurscy
76.4
75
75-
70
70
Number of compxtation
Rumber computation 50035 (T)
Reduction Ratio
Window length
Figure 6: IPRM Model ablations in order: (i) Impact of number of parallel operations (Nop) vs computation steps (T"). (ii) Impact of Operation Composition Block (OPC). (iii): Impact of reduction ratio (r) and (iv) memory window length (W).
3.3 Model ablations and reasoning visualization
We perform ablations to study the contributions of IPRM's salient components. First, we analyze the impact of varying number of parallel operations (Nop) against number of iterative computation steps (T)) We compare models with Nop {1,3, 6, 9} and T E {1,3, 6, 9), resulting in 16 different models. Given the large amount of ablative models, we perform analysis on a seduced-resolutin CLEVR for 15 epochs. As shown in fig. 6 (plot (i)), we find that T and Nop appear to be co-dependent and that neither by itself can lead to high performance. E.g. setting T - generally results in peformances around 75% regardless of Nop. while setting Nop= 1 or 3 results in a sharp performance drop of 3% when changing T to 9 from 6. In contrast, for Nop > 3, we observe that performance increases steadily with higher T, suggesting that a higher number of parallel operations may prevent overfitting in a model with high computation steps. Overall, we find that (Nop 6, T = 9) and (Nop = 9, T 9) are the two best performing models achieving above 82% accuracy (with the former preferred as Nop = 6 performs better for diff. T compared to Nop 9).
setting of CLEvR-Humans with pretraining
Next, we study the impact of the operation composition block (OPC) by evaluating Nop = 6 (and diff. computation steps 2) with and without OPC. As shown in fig. (plot (ii), while T = 1 has relatively low drop of ~2%, the performance drops are more significant for higher T. The (Noop = 6,
T = 9) model which reached 82% acc. with OPC, drops to 74% acc. without OPC.
Finally, we study the impacts of the dimension reduction ratio (r) and memory- lookbbcc window length (W). As shown in fig. 0 (plot (ii)). compared to no dimension reduction (i.e. r = 1) while being computationally faster and requiring less memory. However, setting to 8 significantly deteriorates performance. Similarly, as shown in fig. x (plot (iv)), a setting of window-length W is works sufficiently well, and decreasing it below that significantly impacts performance.
2 leads to negligible drop (0.3%) in performance
8

---

Parelel Operaation
Paralbll Openatoons
What shape is behind
How many objects shape
the large cyllinder with
occuring color? cylinder;
one on the farthast right?? (QT: two; Pred:: two)
Predi cylinder)
Reasoning Computation Staps
Computation Steps
kiles or flags? (GT: yea; Pred: yes)
any
that match each other terms of matorial and color also match each other in size?
(GI Yes; Prod: No)
Reasoning Computation Steps
Figure 7: Condensed reasoning visualization of IPRM. In the top examples, IPRM correctly utilizes both parallel and iterative computation to arrive at the correct answer. The bottom left example shows IPRM's cumulative lang. and visual atlentions when solving real-world GQA example. The bottom right example, shows an error case where IPRM seems to misunderstand question and outputs wrong ans. with less relevant attentions. See appendix for further reasoning visualizations and error cases, and supplemental for further CLIP integrated visualizations on GQA examples.
In fig. 4 we assess the performance of IPRM across different functional program lengths (a proxy for reasoning steps) on the CLEVR and STAR benchmarks. As shown, IPRM maintains a high performance across both short and longer reasoning steps and for both image and video VQA scenarios. also provide a condensed illustration of IPRM's reasoning visualization in fig. 7 Here, for brevity, only show the model's visual attention for subset of parallel operations and computation steps (see suppl. for full visualization). The top left example ("what shape max occuring color") illustrates the model's 's usage of both iterative and parallel computation. At the first reasoning step, the model appears to doing parallel operations to both identify "large cylinder" and compute "max occuring color". In the and seems to check which of the two large cylinders match the color. Next, it highlights the correct cylinder and in the final step, it locates and predicts the correct object behind. The top right example in fig. 7 similarly shows computation for a question involving spatial ("farthest right"), similarity ("same shape") and counting ("how many") operations. The bottom left example illustrates IPRM's simultaneous language and visual attentions updated across reasoning steps when solving a real world (GQA) example. Finally, the bottom right example is an error-case where IPRM seems incorrectly understand execute requisite operations, thereby not producing expected attentions for "items with matching material and color" throughout steps.
step, it appears to have found "max occuring color" (cyan)
Related Work
Visual reasoning methods and vision-language models. Multiple prior works have introduced
effective visual reasoning methods in context of image and video question answering 35
52
35
50
69
56
B2
54
53
[6]. Prominent works include NMN 2 FILM
[60]
NSM
MAC
X
MCAN
37
NS-VQA (85), ALOE [1), VR-DP 1. SHG-VQA
68
MIST
18
and OCRA 76
contrast to these works that show applicability of methods for particular VQA
benchmarks/tasks our work explores a more general direction of integrating parallel and iterative
computation in reasoning framework that we show be effective for multiple complex VQA benchmarks and reasoning scenarios. More recently, vision-language models AKI 73
37
66
45
46. /
and multimodal large-langugge-modees (57 4i7 with transformer-based mechanisms have
shown impressive reasoning capabilities at scale. Notable examples include CLIP
6
GPT 57
Gemini MDETR (36) LXMERT [66), VInVI. [X) BLIP
Flamingo
1.
LlavA
BETT 74 and All-in-One
72
We believe our work is complimentary to these developments,

---

contributes an alternative and possibly more effective reasoning mechanism that can be integrated with such models in future to enhance complex VQA capabilities.
Memory and recurrence-aggmented transformers. Multiple works have identified limitations of purely feedforward computation as realized in transformers and worked encoding recurrence EC and
26
Notably, Recurrent Memory Transformer and MemFormer(ßI. introduce recurrent and dynamic memory to improve language modelling introduces efficient memory to augment knowledge retrieval, MemViT XQ introduces cache memory to retain prior context for long-video tasks, and [ introduces memory for more effective action anticipation. While these methods study recurrent and memory- augmeettd computation on specific natural language processing and computer vision tasks, our work focuses on the integration of pnerative-parallee computation and working memory in single neural reasoning mechanism beneficial for complex VQA scenarios.
computation 80 g
capabilities, More recently, EMAT X2
5 Conclusion
We introduced a novel fully- dilleeiifeeeettiable and end-to-end trainable iterative and parallel reasoning mechanism (IPRM) to address complex VQA scenarios. We comprehensively evaluated IPRM on various complex image and video VQA benchmarks testing distinct reasoning capabilities, and found improves state-of-arts on multiple such henchmarks. We also performed quantiative ablations to study individual impacts of parallel and iterative computation besides qualitative analysis of IPRM's reasoning computation visualization.
Limitations and Future work
Here, we note possible limitations of IPRM. Similar to existing VQA and deep-learning methods, IPRM may reflect biases that present in the training distribution of VQA benchmarks. This may lead it to overfit to certain image inputs or question forms and possibly provide skewed answers in such scenarios. Further, the utilized vision-language backbones in our experiments may also entail visual, language and cultural biases in their original training distribution which may permeate to IPRM upon integration for VQA scenarios. In this regard, we hope the capability visualize intermediate reasoning of IPRM and diagnose its error cases (as shown in section 3.3 can serve useful tool to benefit interpretability in VQA and identify possible reasoning biases that may emerge in the model.
For future work, scaling the IPRM architecture to foundational uideo-language model by integrat- ing it with large-scale transformer-base vision-language models and relevant instruction-tuning approaches presents an exciting research opportunity. Moreover, while designed and evaluated IPRM in the context of complex VQA, we believe it has the potential to operate as general reasoning mechanism applicable to tasks beyond visual reasoning and question answering, such language processing and embodied reasoning.
Acknowledgment This research/project is supported by the National Research Foundation, Singapore, under its NRF Fellowship (Award## NRF-NRFFI 47FFF142002 0-22144-22--0011 This research also supported by funding allocation to B.F. and C.T. by the Agency for Science, Technology and Research (A STAR) under its SERC Central Research Fund (CRF), as well as its Centre for Frontier Al Research (CFAR).
References
[1] J.-B. Alayrac. J. Donahue, P. Luc, A. Miech, Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances in 2022.
Neural Information Processing Systems,
[2] Andreas, M. Rohrbach, T. Darrell, and D. Klein. Learning compose neural networks for 2016.
question answering. arXiv preprint
[3] D. Bahdanau, H. de Vries, T. J. O'Donnell, S. Murty, P. Beaudoin, Y. Bengio, and A. Courville. Closure: Assessing systematic generalization clevr models, arXiv preprint arXiv: M9:::992.000 1912.05783, 2019.
10

---

[4] Z. Bai, R. Wang, and X. Chen. Glance and focus: Memory prompting for multi-event video question answering. Advances in Neural Information Processing Systems, 36, 2024.
[5] A. Bhattacharyya, S. Panchal, R. Pourreza, M. Lee, P. Madan, and R. Memisevic. Look, remember and reason: Grounded reasoning in videos with language models. In The Twelfth International Conference on Learning Representations.
[6] B. Bogin, S. Subramanian, M. Gardner, and J. Berant. Latent compositional representations im- prove systematic generalization in grounded question answering. Transactions of the Association for Computational Linguistics, 9:195-210, 2021.
[7] Buch, C. Eyzaguirre, A. Gaidon, J. Wu, Fei-Fei, and J. C. Niebles. Revisiting the" video" in video-language understanding. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 2917-2927, 2022.
[8] A. Bulatov, Y. Kuratov, and M. Burtsev. Recurrent memory transformer. Advances in Neural Information Processing Systems, 15:11071-1001, 2022.
[9] A. Capon, S. Handley, and I. Dennis. Working memory and reasoning: An individual differences perspective. Thinking Reasoning, 93):203-224, 2003.
[10] Y. Cong, W. Liao, H. Ackermann, B. Rosenhahn, and M. Y. Yang. Spatial-temporal transformer for dynamic scene graph generation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 16382, 2021.
[11] M. Dehghani, S. Gouws, O. Vinyals, J. Uszkoreit, and L. Kaiser. Universal transformers. arXiv preprint 1807.188000035 08889, 2018
[12] D. Ding, F. Hill, A. Santoro, M. Reynolds, and M. Botvinick. Attention over learned object embeddings enables complex visual reasoning. Advances in neural information processing systems, 34:91 14:9122-9124 2021.
[13] M. Ding, Z. Chen, T. Du, P. Luo, J. Tenenbaum, and C. Gan. Dynamic visual reasoning by learning differentiable physics models from video and language. Advances in Neural Information Processing Systems, 34:887-899 187-899, 2021.
[14] N. Dziri, X. Lu, M. Sclar, X. L. Li, L. Jiang. B. Y. Lin, S. Welleck, West, C. Bhagavatula, R. Le Bras, et al. Faith and fate: Limits of transformers on compositionality, Advances in Neural Information Processing Systems, 36, 2024.
[15] C. Fan, X. Zhang, S. Zhang, W. Wang, C. Zhang, and H. Huang. Heterogeneous memory enhanced multimodal attention model for video question answering. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 1999-2007,, 2019.
[16] H.-S. Fang, S. Xie, Y.-W. Tai, and C. Lu. Rmpe: Regional multi-person pose estimation. In Proceedings of the IEEE international conference on computer vision, pages 2334-2343, 2017.
[17] D. Fougnie. The relationship between attention and working memory. New research short-term memory, 1:45, 2008.
[18) D. Gao, I. Zhou, I. Ji, L. Zhu, Y. Yang, and M. 7.. Shou. Mist: Multi-modal iterative spatial- temporal transformer for long-form video question answering. In Proceedings of the IEEK/VF Conference on Computer Vision and Pattern Recognition, pages 14773-14783, 2023.
[19] R. Girshick. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 1440-1448, 2015.
[20] M. Frunde-MeLaughlee R. Krishna, and M. Agrawala. Agqa: A benchmark for compositional spatio-temporal reasoning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 11287-11297. 2021.
[21] M. Grunde-Mcl.aughinn R. Krishna, and M. Agrawala. Agqa 2.0: An updated benchmark for compositional spatio-temporal reasoning, arXiv preprint arXiv:2204.06105. 2022.
11

---

[22] J. Hsu, J. Mao. J. Tenenbaum, and J. Wu. What's left? concept grounding with logic-enhanced foundation models. Advances in Neural Information Processing Systems, 36, 2024.
[23] Hu, L. Shen, and G. Sun. Squeeze-and-excitatin networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7132-7141, 2018.
[24] R. Hu, J. Andreas, M. Rohrbach, T.. Darrell, and K. Saenko. Learning to reason: End-to-end module networks for visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 804-813, 2017.
[25] R. Hu, A. Rohrbach, T. Darrell, and K. Saenko. Language-conditioned graph networks for relational reasoning. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10294-10303, 2019.
[26] F. Huang, K. Lu, C. Yuxi, Z. Qin, Y. Fang, G. Tian, and G. Li. Encoding recurrence into transformers, In The Eleventh International Conference Learning Representations, 2022.
[27] D. Hudson and C. D. Manning. Learning by abstraction: The neural state machine. Advances in Neural Information Processing Systems, 32, 2019.
[28] D. A. Hudson and C. D. Manning. Compositional attention networks for machine reasoning. arXiv preprint
2018
[29] D. A. Hudson and C. D. Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700-6709, 2019.
[30] D. Hutchins, I. Schlag, Y. Wu, E. Dyer, and B. Neyshabur. Block-recurrent transformers. Advances Neural Information Processing Systems,
2022.
[31] S. Jaiswal, B. Fernando, and C. Tan. Tdam: Top-down attention module for contextually guided feature selection in cnns. In European Conference on Computer Vision, pages 259-276. Springer, 2022.
[32] A. Jha, B. Patro. L. Van Gool, and T. Tuytelaars. Barlow constrained optimization for visual question answering. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1084-1093, 2023.
[33] C. Jing, Y. Jia, Y. Wu, X. Liu, and Q Wu. Maintaining reasoning consistency in compositional visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5099-5108, 2022.
[34] J. Johnson, B. Hariharan, L.. Van Der Maaten, L. Fei-Fei, C. Lawrence Zitnick, and R. Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2901- 2910 101 2017.
[35] J. Johnson, B. Hariharan, I.. Van Der Maaten, J. Hoffman, L. Fei-Fei, C. Lawrence Zitnick, and R. Girshick. Inferring and executing programs for visual reasoning. In Proceedings of the IEFE international conference on computer vision, pages 2989-2998, 2017.
[36] A. Kamath, M. Singh, Y. LeCun, G. Synnaeve, I. Misra, and N. Carion. Mdetr-modulated detec- tion for end-to-end multi- modal understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1780-1790, 2021.
[37] W. Kim, B. Son, and I. Kim. Vilt: Vision-and-language transformer without convolution region supervision. In International Conference Machine Learning, pages 5583-5594. PMLR, 2021.
[38] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[39] T. M. Le, Le, S. Venkatesh, and T. Tran. Hierarchical conditional relation networks for video question answering. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 9972-9981, 2020.
12

---

[40] J. Lei, Li, L. Zhou, Z. Gan, T. L. Berg, M. Bansal, and J. Liu. Less is more: Clipbert for video-and-lange and-language learning via sparse sampling. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7331-7341, 2021.
[41] M. Lewis, N. V. Nayak, P. Yu, Q. Yu, J. Merullo, S. H. Bach, and E. Pavlick. Does clip bind concepts? probing compositionality in large image models. arXiv preprint arXiv:2212.10537, 2022.
[42] C. Li, H. Xu, J. Tian, W. Wang, M. Yan, B. Bi, J. Ye, H. Chen, G. Xu, Z. Cao, et al. mplug: Effective and efficient -aiiguge learning by cross-modal skip-connections. arXiv preprini arXiv:2205. /2005, 2022.
[43] Li, D. Li, Savarese, and Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597 2023.
[44] J. Li, D. Li, C. Xiong, and S. Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning, pages 12888-12900. PMLR, 2022.
[45] I.. Li, M. Yatskar, D. Yin, C.-J. Hsieh, and K.-W. Chang. Visualhert: A simple and performant baseline for vision and language. arXiv preprint arKiv: (.0iv:/9008.035555 2019.
[46] X. Li, Yin, C. Li, P. Zhang, X. Hu, L. Zhang, L. Wang. H. Hu, L. Dong, F. Wei, et al. Oscar: Object-semantics aligned pre-training for vision-language tasks. In Computer Vision-ECCe 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXX 16, pages 121-137. Springer. 2020.
[47] H. Liu, C. Li, Q. Wu, and Y. Lee. Visual instruction tuning. arXiv preprint arXiv:300.008885, 2023.
[48] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov. Roberta: robustly optimized bert pretraining approach. arXiv preprint 190iv 1692, 2019.
[49] J. Lu, V. Goswami, M. Rohrbach, D. Parikh, and S. Lee. 12-in-1: Multi-task vision and language representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10437-10446, 2020.
[50] Mao, C. Gan, Kohli, J. B. Tenenbaum, and Wu. The neuro-symbolic concept learner: Inter- preting scenes, words, and sentences from natural supervision. arXiv preprint 1eXiv:1904.125:0 2019.
[51] J. Mao, X. Yang, X. Zhang, N. Goodman, and Wu, Clevrer-humans: Describing physical and causal events the human way. Advances in Neural Information Processing Systems, 35:7755- 7768, 2022.
[52] D. Mascharka, P. Tran, R. Soklaski, and A. Majumdar, Transparency by design: Closing the gap between performance and interpretability in visual reasoning. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4942-4950, 2018.
[53] S. S. Mondal, J. D. Cohen, and T. W. Webb. Slot abstractors: Toward scalable abstract visual reasoning. arXiv preprint arXiv:2403.0s4 0334588 2024.
[54] S. S. Mondal, T. Webb, and J. D. Cohen. Learning to reason over visual objects. arXiv preprint arXiv:2303.02260 2023.
[55] B. X. Nguyen, T. Do. H. Tran, E. Tjiputra, Q. D. Tran, and A. Nguyen. Coarse-to-fin reasoning for visual question answering. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 4558-4566, 2022.
[56] D. K. Nguyen, V. Goswami, and X. Chen. Movie: Revisiting modulated convolutions for visual counting and beyond. In International Conference on Learning Representations, 2020.
[57] R. OpenAl. Gpt-4 technical report. ArXiv, 2303, 2023.
13

---

[58] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. Pytorch: An imperative style, nigh-perforrance deep learning library. Advances in neural information processing systems, 32, 2019.
[59] J. Pennington, R. Socher, and C. D. Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543, 2014.
[60] E. Perez, F. Strub, H. De Vries, V. Dumoulin, and A. Courville. Film: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI conference on arlificial intelligence, volume 32, 2018.
[61] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, J. Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
[62] R. Shrestha, K. Kafle, and C. Kanan. Answer them all! toward universal visual question answering models. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10472-10481, 2019.
[63] A. Suhr, M. Lewis. J. Yeh, and Y. Artzi corpus of natural language for visual reasoning. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 217-223, 2017.
[64] A. Suhr, S. Zhou, A. Zhang, Zhang, H. Bai, and Y. Artzi A corpus for reasoning about natural language grounded in photographs. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6418-6428, 2019.
[65] H. Tan and M. Bansal. Object ordering with bidirectional matchings for visual reasoning. In Proceedings of NAACL-IILT, pages 444-451, 2018
[66] H. Tan and M. Bansal. Lxmert: Learning gross-modality encoder representations from trans- formers. In Proceedings of the 2019 Conference Empirical Methods in Natural Lan- guage Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages
2019.
[67] G. Team, R. Anil, S. Borgeaud, J.-B. Alayrac, Yu, R. Soricut, Schalkwyk, A. M. Dai, A. Hauth, K. Millican, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.118000
2023.
[68] A. Urooj, H. Kuchne, B. Wu, K. Chheu, W. Bousselham, C. Gan, N. Lobo, and M. Shah. Learning situation hyper-graphs for video question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14879-14889, 2023.
[69] M. Vaishnav and T. Serre. Gamr: A guided attention model for (visual) reasoning, In The Eleventh International Conference on Learning Representations, 2022.
[70] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[71] J. Wang, G. Chen, Y. Huang, L. Wang. and T. Lu. annoor--add-nticipattin transformer for online action understanding. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pages 13824-13833 2023.
[72] J. Wang, Y. Ge, R. Yan, Y. Ge, K. Q. Lin, S. Tsutsui, X. All in one: Exploring unified vadeo-language pre-training. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 6598 6608, 2023.
Lin, G. Cai, J. Wu, Y. Shan, al.
[73] Wang, A. Yang, R. Men, J. Lin, S. Bai, Z. Li, J. Ma, C. Zhou, J. Zhou, and H. Yang. Ofa: Unifying architectures, tasks, and modalities through simple sequence-to-sequence learning framework, In International Conference on Machine Learning, pages 23318-23340. PMLR, 2022.
14

---

[74] W. Wang. H. Bao, L. Dong, J. Bjorck, Z. Peng, Q. Liu, K. Aggarwal, O. K. Mohammed, S. Singhal, S. Som, al. Image as a foreign language: Beit pretraining for vision and vision- language tasks. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition, pages 19175-19186, 2023.
[75] Y. Wang, K. Li, Y. Li, Y. He, B. Huang, Z. Zhao, H. Zhang, J. Xu, Y. Liu, Z. Wang, et al. Interviieoo: General video foundation models via generative and discriminative learning. arXiv 2022.
preprint arXiv:2212.03193
[76] T. Webb, S. S. Mondal, and J. D. Cohen. Systematic visual reasoning through object-centric relational abstraction. Advances in Neural Information Processing Systems, 36, 2024.
[77] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, Cistac, T. Rault, R. Louf, M. Funtowicz, et al. Huggingface's 's transformers: ofate-of-the-ar natural language processing. arXiv preprint 1rXiv:1910.03330 2019. 2019.
[78] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, Cistac, T. Rault, R. Louf, M. Funtowiez, al. Transformers: State-of-the-an natural language processing. In Proceed- ings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pages 38-45, 2020.
[79] B. Wu, Yu, Z. Chen, J. B. Tenenbaum, and C. Gan. Star: A benchmark for situated reasoning in real-world videos. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
[80] C.-Y. Wu, Y. Li, K. Mangalam, H. Fan, B. Xiong, J. Malik, and C. Feichtenhofer. Memvit: Memory-augmenter multiscale vision transformer for efficient long-term video recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13587-13597, 2022.
[81] Q. Wu, 7. Lan, K. Qian, J. Gu, A. Geramifard, and 7.. Yu. Memformer: A memory-augmentt transformer for sequence modeling. In Y. He, H. Ji, S. Li, Liu, and C.-H. Chang, editors, Findings of the Association for Computational Linguistics: AACL-IJCNLP 2022, pages 308-318, Online only, Nov. 2022, Association for Computational Linguistics.
[82] Y. Wu, Y. Zhao, B. Hu, P. Minervini, P. Stenctorp, and S. Riedel. An efficient memory- augmented transformer for knowledge-intensive nlp tasks. arXiv preprint 2rXiv:2210.1677 2022.
[83] Z. Yang. X. He, J. Gao, L. Deng, and A. Smola. Stacked attention networks for image question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 21-29, 2016.
[84] K. Yi, C. Gan, Y. Li, P. Kohli, J. Wa, A. Torralba, and J. B. Tenenbaum. Clevrer: Collision events for video representation and reasoning. arXiv preprint arXiv:1910.014: 2019.
[85] K. Yi, J. Wu, C. Gan, A. Torralba, P. Kohli, and Tenenbaum. Neural-symbolic vqa: Disen- tangling reasoning from vision and language understanding. Advances in neural information processing systems, 31, 2018.
[86] S. Yu, J. Cho, P. Yadav, and M. Bansal. Self-chained languaae model for video localiza- tion and question answering. arXiv preprint htti:2335.06888 2023.
[87] Z. Yu, J. Yu. Y. Cui, D. Tao, and Q. Tian. Deep modular co-attention networks for visual question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6281 6290, 2019.
[88] Zhang, X. Li, X. Hu, J. Yang, I.. Zhang, L.. Wang, Y. Choi, and J. Gao. Vinvi: Revisiting visual representations in vision-language models. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 3579-5588, 2021.
15

---

A Appendix supplemental material
B Elaborated Experiments and Results Discussion
B.1 Further comparisons on CLEVR-Humans, CLEVR-CoGenie CLOSURE and NLVR<1
Here, we provide further comparisons with benchmark-specific methods for CLmaR-Humans 35 CLEVR-CoGenT [34], CLOSURE B and NLVRv1 [33 (not reported in main paper due to space limitations). As mentioned main paper, these benchmarks utilize synthetic images and are a test of pure visual reasoning capabilities that are minimally influenced by increased world knowledge or usage of stronger visual backbones.
CLEVR-Humans as already mentioned in main paper evaluates a model's reasoning generaliza- tion capabilities to unseen scenarios or question forms. CLEVR-CoGenTT studies compositional attribute generalization. Specifically, it has two conditions - i) cond.A wherein all cubes have color (red, green, purple, cyan) (spheres
{gray, blue, brown, yellow) and cylinders color), and ii) cond.B wherein color-sets are switched b/w cubes and cylinders. A model is then trained one condition and evaluated on both the original and alternate condition. A higher accuracy on the alternate condition indicates that the model learns more 'composttionally" as it generalizes better to novel shape-color combinations with less feature/attribute combination overfitting.
be any
Table 5: Elaborated results on CLEvR-Humans (left), CLEVR-CoGenT (middle) and NLVRv1 (right), IPRM achieves state-of-art across the three benchmarks and does not require additional supervision such as bounding boxes or functional programs. * requires func. programs supervision pre-defined dataset-specifie neural modules. requires object bounding boxes supervision.
Model
ZS
CLV-Hum
FT
Model
ValA
CoGenTr-A
ValB
ValA
CoGenFF-B
ValB
Model
NLVRI
Test
RGGEE
351
54.0
66.6
NS NS-VQA
351
99.8
63.9
CNN
RNNN
63
56.3
NS-VQA
67.8
MDETR
3G
99.8
76.7
MAC PE
59.4
RAMEN
57.8
StackAtt-MLP
80.3
68.7
75.7
75.8
61.2
GLT
FILM 100
56.6
75.8
75.9
+ BE
96.6
73.7
76.1
92.7
NMN A
62.0
LEFT
78.8
Tbd-Net*
594
98.8
75.4
96.9
96.3
N2NMN 251
66.0
MAC 20
57.4
81.55
MAC MA
99.0
78.3
97.2
96.1
CNN-BiATT
66.1
MDETR
59.9
81.7
78.8
81.1
96.9
IPRM (scratch)
63.8
IPRM
63.8
X3.5
IPRM
99.1
80.3
98.0
98.2
IPRM-CLV-FT 73.0
Finally, NLVRv1 evaluates language- -aroundd- visual reasoning. Each sample of this benchmark comprises of three synthetic images and composite natural language statement about the images which can evaluate to True or False and requires various visual-linguistic reasoning skills.
As shown in table 5 IPRM achieves state-of-art results across the three benchmarks and does not require pre-annotated bounding-boo boxes or functional programs as additional supervision. For CLEVR- R-Hmmans (table / left), it outperforms larger-scale models such as MDETR and RAMEN in zero-shot performance even though the latter is pre-trained on multiple VQA datasets. It also increases ofate-of-art in finetuned setting by 3.8%.
For SLEVR-CogenTT (table centre). IPRM achieves the highest generalization results amongst methods in both the CoGen- Train A and Finetune B. Specifically, it obtains 80.3% acc. cond. B (when trained on cond. A), which is 1.5% higher than the previous state-of-art cond.B method FILM and 3.6% higher than MDETR. When further finetuned on cond.B, IPRM generalizes for both cond.A and cond.B achieving 98.0% and 98.2% unlike FILM which overfits to cond.B and thereby has poor performance cond.A. Further, its performance Cond.A (99.1%% 1%) is highest amongst methods that do not utilize bounding box or localization supervision and marginally lower than MDETR and NS-VQA (which utilize bounding-box supervision).
For NLVRv1 (table rrgght) IPRM model trained from scratch achieves 63.8% acc, and performs competitively with existing task specific state-of of model CNN-BiAtt. When finetuned from its CLEVR checkpoint, we find IPRM achieves 73.0% acc. which is 7% higher than existing visual inputs state-of-ar art for NLVRvl and suggests strong reasoning transfer capabilities of IPRM. It further outperforms the N2NMN method which requires pre-defined neural modules to be identified for the dataset.
16

---

Finally, we have also provided results of models on CLOSURE for different subsets / questtin types in table For the -LEVR-ennvronment, we have visualized some failure cases in figs. 10 and More cases can be run through the visualization framework in provided source code.
Table 6: Zero-shot performances of models CLOSURE subsets (2 trials run for IPRM)
(ZeroShot)
embed, mat_.pa
spa max
oc_mat
or_mat upa
compare
compare, mat_spa
FLMM
41.4+18
62=11
93.462
34,4+11
35 245.555
26.258.5
65.825
MAC
63.7425
76.8+11
99ed,.1
75.2+13
70.4+15
66 66.256.4
NS-VQA
33,3+18
97.3+4.6
98.345
69. 09.54228
50.4+27
96.146.2
95.446.7
GG-Tenso-NMM'
37.6117
79.8+1.4
95.645.6
34.158.1
25.5t11
89.2:2.3
89 89.8
Vecter-
32,5+17
97.342.4
97,2+3.9
64.9+25
47,4+23
95.345.7
44,4664
IPRM
83.136.4
80,347.2
99.250.1
75.7494
67.148.1
62,247.3
61.52588
B.2 Elaborated STAR results and ablations for video reasoning tasks
We provide results on the STAR Test, further baselines and model ablations for video reasoning tasks in table 7
Table Left: Results on STAR ri official hidden test set (evaluation server) with ground-trute vision (GT V) and predicted vision (PR V); Right: Results on STAR val. set with num. of sampled frames =32 unless otherwise stated O.
Model
Int.
Seq.
Pred.
Feas.
Avg.
All-in-Onc
Model
Setup
STAR-Toa
Temp f
50.6
52.8
49.3
40.6
48.3
Seg
Avg.
MIST [8]
55.5
54.2
54.2
44.4
51.1
GTV
34.7
35.9
31:2
31.4
34.7
16
58.7
NS 69
CLIP BERT 201
GTV V
26.3
42.6
38.9
46.3
43.4
30.7
29.8
43.9
48.5
36.5
Concal-AIt-21
SeViLA-BLI BLIP
5
63.7
70.4
63.1
62.4
64.9
Concat-Att-4l
68.1
71.4
66.6
55.2
65.3
33.6
37.2
30.3
34.8
Att-
67.5
72.1
64.4
58.5
65.6
CLIP-BERT
29.8
Concat-At--6.
55.7
66.0
SHIG-VQA
PR V
20.9
18.0
42.0
31.8
35.3
30.2
32.5
29.7
39.5
30.7
Cross Au 6L
59.8
63.0
56.4
49.9
$7.3
GF A
56.1
61.3
52.7
45.7
53.9
(PRM(ml.t))
63.8
IPRMal 19)
70.0
75.7
70.2
57.8
68.4
IPM
PRV
72.0
78.1
71.3
39.4
70.3
IPRM
75.2
68.6
57.4
67.7
IPRM
71.8
77.7
71.0
59.1
69.9
IPRM (GT
78.7
85.5
81.7
71.2
79.3
B.3 CLIP Integration Results
We provide results with frozen CLIP 61 visual backbones including CLIP VIT-L/14, CLIP VIT- B/16 and CLIP VIT-L/I4@ 336px on GQA 29), NLVRv2 6-4 and CLEVR-Human
table g Wt-Proj-Att). As shown in the table, IPRM can enhance performance for the CLIP variants across GQA, v and CLV-Humans in comparison to concat and cross-att blocks. Further, it is more parameter efficient with only 5.5M additional parameters in comparison 4-layer well as 2-layer stacks of Cross-Att (9.2M 2-layer, 17.6M 4-layer) and Concat-Att (7.2M 2-layer, 13.6M 4-layer). With regards computational FLOPs, IPRM consumes 5.9GFLOPs which marginally higher than Cross-Att l-yer config (3. GFLOPs) and lower than Concat- Att 4-yayer config (8 (8.9GFLOPs). Note, that the performance benefits of adding further layers of cross- or concat-un blocks are observed to be minimal after 4 layers, and can also depend on the amount of training data available. F.g. Both cross- and concat-att blocks of layers had better performances on NLVRv2 (which has limited set of training questions relative to GQA and CLEVR) in comparison to 4 layer config.
We empirically found utilizing a Distil-rowe Distil-roberta language backbone to be more effective than the associated CLIP language backbone, and hence used the former for question processing. We compare with alternate prominent lision-language attention mechanisms including Cross-att and Concat-att blocks as well as simple joint projection of vision and language pooled representations (referred as
B.4 Further reasoning computation visualizations
We provide elaborate reasoning computation visualizations of IPRM showing the lang. and vis. attentions across parallel operations and computation steps during operation formation and operation execution stages. Fig. g shows scenario wherein IPRM correctly utilizes parallel and iterative computations to compute intermediate operations of "find object close to front", "retrieve/compare
17

---

Table 8: Left: Comparison of IPRM with prominent vision-language attention mechanisms with CLIP VIT-L/14 backbones on CLEVR- GQA and NLVRv2 benchmarks ('4L' indicates att layers; 'x' indicates model did not converge). Right: Results with other CLIP variants VIT-B and VIT-L@ 336 on GQA and NLVRv2.
Model (CLIP
GQA TextD
NLVR2
ICLIP
GQA
NLVR2
CLV-
VIT-R/16 hhone)
Test
VIT-1/14 bhone)
+Param
+GLLOPS
Test
Test
ZS
FT
Wt-Proj-Fusion
51.4
59.9
Wr.Proj-Fusion
53.5
Cross-Att
54.6
56.6
Cross-Au (2L)
9.2M
1.5
55.1
62.1
Concar-Arl
56.0
57.4
Concat-Att (2I.)
7.2M
4.4
55.3
60.5
IPRM
55.9
60.8
Cross-All (4L)
17.6
57.4
60.3
Model (CTIP
GQA
NEVR2
Concat-Att (4L)
13.6M
8,9
58.7
55.9
61.2
81.1
VIT-L/14@ 336)
TestD
Test
Cross-Att (61.)
26.0M
4.5
56.8
60.8
80.4
Wt-Proj-Fusion
54.0
61.1
Concur-Att
19.7M
57.4
62.0
81.8
Cress-Alt
57.4
58,4
IPRM
5.2M
5,9
59.2
65.1
64.3
84.6
Concar-Art
57.3
59.1
IPRM
59.0
65.3
shape and size", "find applicable objects with both same shape and size". Fig. 2 shows another correct prediction of IPRM, and this time, its intermediate reasoning visualization is useful to determine that the entailed reasoning appears sensible. Fig. [0] shows an incorrect prediction by IPRM and its intermediate reasoning visualizations also suggest that IPRM did not understand the question and thereby did not attend to relevant objects. Finally, Fig. M shows a scenario wherein while IPRM produces the correct answer, it's intermediate reasoning appears imprecise which makes the prediction (and underlying reasoning) less reliable. We provide further visualizations with a CLIP VIT-L/14 backbone on GQA samples in the supplemental jupyter notebook output (html format for easier viewing).
C Model implementation and experiment details
We implement IPRM in Py PyTorch 58 generic vision-language module receiving set of input vision (or represetttttitiinnn tokens and input language (or reseeeepreeetttioonn) tokens. We provide Python-style pseudocode of IPRM in figs L2 13 and L4 For all experiments, we set the internal dimension of IPRM to 512 and use the same configuration of num. parallel operations (Nop)=6, num. computation steps (T)=9, reduction ratio (r)=2 and window size (W)=2. We follow benchmark-speet conventions for vision-language backbones that are detailed below in sec. C.T For CLIP [61, we utilize the official models from Huggingface [77). All experiments are performed on single NVIDIA A40 GPU with 46GB memory and averaged over 3 trials with different random seeds wherever possible (done for primary experiments on STAR, AGQA,
le-4 with Adam optimizer and gradient clipping value of 8. The learning-rate rate is reduced based on validation acc. plateau with reduction factor 0.5, threshold 0.001 and patience 0. Further experiment hyper- parameters and settings are provided below. Source code at: htttps://github. com/ shantanu
CLEVR-Humane Unless otherwise specified, the learning rate is initialized
38
IPRM_ and and_aanadllae_...aaaaaaaaaaaaaae Kww.onaninnnnnnn www.ooonninnMeehiiineiiee Mechanism
C.1 Benchmark-speciffe experiment details
CLEVR-Humane
We use the CLEVR-Humans dataset from 35 which comprises images from
original CLEVR dataset 34 and human crowdsourced questions. We use batch size of 216 for
training. We use the same language encoder (Distil-Robertal 48 from Huggingface 78) as in existing
state-of-art MDETR (36] and frozen ResNet101 backbone layer 3 spatial features (as in 2x 351
We perform all ablation experiments with 14x 1024 visual features. Each ablation model is first pretrained for 10 epochs on the original CLEVR dataset (the initial learning rate for IPRM le-4 and for language encoder is le-5) and then finetuned on SLEVR--Huanne for 40 epochs with early stopping (learning rate of le-4 throughout). As observed in prior work (52]. we similarly found in multiple scenarios with occluded objects that visual attention only partially identified such objects. Hence, we simply resampled (bilinear sampling) visual input to obtain 16x 16x 1024 features and empirically found more complete visual attentions with corresponding 1.1% improvement in accuracy. The final two best performing model configurations (Nop=6, T=9, W=2, R=2 and Nop=6, T=9, W=2, R=1) from ablations were then pre-trained for 35 epochs CLEVR and finetuned
18

---

on CLEVR- R-Hmmaas... While we found that configuration Nop= 7=9, W=2, R=/ obtains highest zero-shot (ZS) acc. of 65.6% and finetuned (FT) acc. of 86.3% we adopt Nop=6, T=9, W=2, R=2 (with 63.3% 7S and 85.4% FT acc.) as our optimal model given its lesser parameters and FI.OPs.
GQA. We use the GQA compositional real-world image question answering dataset from Based on prior VQA methods on GQA
33 we utilize pre-extracted founding-box object proposal features and object label predictions obtained from pretrained object detector The bounding box coordinates is normalized to range of 0 to 1 based on the original distributed representation through a learned nonlinear projection. This representation is concatenated with learned projection of the predicted object labels (initialized with glove [59 300dim embeddings) to form the final visual input. We train IPRM for 25 epochs with a batch-size of 192 and same hyperparameters as before. We evaluate the final model on both the dest-dev split and the official test evaluation server (https //eval. wwtfeatureeeeeeeeeeeaea cenetturrdcchallle Satallengee/222 225/ wewgeeee////aalaaaaiioooo evaluation.
22
25
5
119 88.
input image size, and the coordinates are transformed
STAR- QA. We use the STAR-VideoQ) Vi dataset for situational reasoning on real-world videos from 179] Based on previous videoQA methods 779 40 [ for STAR, we utilize object bounding boxes, labels, human pose and human-object relations across frames (note: we do not use the situation hyper-graphs or functional programs). We first perform experiments with the provided ground truth object bounding boxes, labels and human-object relations as well as provided human pose predictions from Alphapose [T as reported in main paper. Each of these is projected to distributed representation through learned non-linear projections obtain object token-wise representations. A further learnable positional embedding for each frame is added to these representations which are then flattened across frames to form the visual input to IPRM. For the language encoder, we found both a simple Bi-LSTM and Distil-roberta language encoder obtain similar performance, and hence choose the simpler Bi-LSTM as the language model. We evaluated models on both 16 sampled frames and 32 sampled frames, and empirically found using 32 frames has ~ 1.33% higher performance and used it for our primary experiments. A batch size of 64 was used with learning rate le-4 over 20 epochs with early stopping. We evaluated the models on both the validation split and official test evaluation server htttps://vval
htt//eb/ckt cha chweeengeesaeeeeeecccccccaeeeeel challenge the all-predicted (no ground-truth visual input setup, similar to 79. we utilize fasterRCNN [19 object extractor, ST-Trans scene graph extractor [TO] and the same Alphapose predictor to obtain predicted object bounding boxes, labels, human-object relations and human poses. We observe drop of ~99 in predicted setup similar to observations in 77) suggesting further performance can be achieved through better object and relationship detection backbones.
/325// overview For
AGQAV2. We use the AGQAv2 2C 21 benchmark that comprises balanced training and test splits. We followed the same methodology in STAR for language and visual backbones. Since AGQAv2 has a very large training and validation set, we found epochs to sufficient number (with most performance increment observed in the 1st epoch itself).
Humans dataset introduced in [5] for temporal, physical and causal video reasoning which comprises videos from the original CLEVRER dataset
HAEVRER-Humann We use the CLEVRER-Humans
14 Similar to in STAR-Videole
and neurosymbolic models 84 35
we utilize a pretrained We again form box coordinates
faster-RCNN based object localization and attribute prediction network from [84]
object-level representations by concatenating learned projections of
and predicted object attributes (i.e. color, shape and material). A frame-level learnable positional embedding is added and object-tokens across frames are flattened to form the final visual input to IPRM. For the language encoder, we used simple bi-LSTM similar to existing methods. Note we do not use the functional programs or event causal graphs in our model. The batch size was 128 with learning rate of 8 and every 4 frames sampled (resulting on average 32 sampled frames). We evaluated models in the three setups - from scratch, zero-shot (CLEVRER- - and finetuned (CLEVRER "'EER-retrriiedd) Since the CLEVRER-HHman dataset is relatively small (comprising only 1076 questions ~ 8 batches; ~ 0.5% of original CLEVRER). for scratch training we trained for 250 epochs (with early stopping) while for finetuning we finetuned for 150 epochs (with 35 epochs for original CLEVRER training).
We use the CLEVR-COGen dataset from +- and follow the same setup in We use a simpler bi-L.TMM language encoder for experiments since the questions are synthetic program-gentt generated unlike in CLEVR-Human (crowdsourced free-form). We trained our model on condition A for 40 epochs (with carly stopping) and used the best cond. A validation
CLEVR-CoGen.
19

---

performance model to evaluate generalization performance cond.B. For finetuning on cond. B we finetuned the best cond.A model for 20 epochs and used the best cond.B validation performance model to also evaluate on cond.A. All other hyperparameters are the same mentioned for CLEVR- Humans.
NLVR. We use the NLVRv1 and NLVRv2 datasets from 63 64]. NLVRvI comprises 3 synthetic images and a language statement while NLVRv2 comprises real-world images and lang. statement. For both datasets, the obtained visual tokens for each images was flattened to obtain the final visual input and an image-wiss wise positional embedding was added indicate image order. For the language encoder, we used a simple Bi-LSTM.
D Potential Negative Impact
In relation to VQA and deep-learning methods in general, the deployment of IPRM in real-world applications without thorough consideration of dataset training distribution biases, could inadver- tently reinforce existing vision, language and cultural biases present in the data, leading to erroneous outcomes or skewed answers. Further, the deployment of VQA methods such as IPRM in sensitive domains such as healthcare scene/footage analysis could raise ethical concerns, including privacy violations, algorithmic reliability, and the potential for unintended consequences stemming from erroneous biased predictions.
20

---

Are there any objects that have both shape and size in common with the object that is found closest to the front? (GT: Yes; Pred: Yes)
any
hade
add
common
found
th
Parallel Operations
Reasoning Computation Steps
Figure 8: Top: original image and question; middle: language attentions across parallel operations (clubbed together; op_k represents parallel operation k) and computation steps. Bottom: Visual attentions across parallel ops and computation steps. Here, IPRM correctly utilizes parallel and iterative compute to locate the correct candidate object for prediction (to which all operations attend last step).
21

---

What shape is the object closest to the gray object with the maximum occuring shape? Pred: cube GT: cube
What
Shape
the
dijett
gray
shicrt
occ
uting
snape
de 9e
9
Paralle! Operations
Reasoning Computation Steps
Figure 9: In this example, IPRM predicts the correct answer and its visual attention trace provides evidence of correct intermediate reasoning. In penultimate reasoning step, IPRM correctly localizes the gray object with maximum occuring shape (cylinder) and in the final step. the parallel operations attend to both the cyan cube and the brown cylinder closest to previously identified gray cylinder.
22

---

Are the two objects that are of a primary color, but not red, of the same shape? Pred: no GT: yes
obrects
color
bul
dhap
9
Parallel Operations
Reasoning Computation Steps
Figure 10: Example where IPRM outputs incorrect answer and the intermediate reasoning appears faulty possibly due to lack of understanding what "primary color is". The pair of blue (a primary color) cubes in this case should have been identified but are not visually attended in any of the operations across reasoning steps).
23

---

What shape the object left of the blue small object with the maximum occuring shape? Pred: sphere GT: sphere
What
shupe
the
etject
the
cijert
the
mazimum
Shape
Parallel Operations
Reasoning Computation Steps
Figure 11: Example wherein IPRM produces correct answer but its visual attention trace suggests intermediate reasoning may be imprecise. The maximum occuring shape is cube; however both the blue small cylinder and blue small cube appear to be attended the penultimate step as the "blue small object with max occuring shape" making the reasoning and prediction less reliable.

---

def iprm_forwart (vis, tokent #Bx Nux Dm
2
lang. tokens, #Bx lang_ sunaayyumar nun_parallel_ops=6, num_iteratiive_seee==;,
#B
mem_ mem__windos_len-2):
7
mem_op_states
mem_res_ res_stttes 0
lang_atts 0
vis_atta []
12
#0. Initialize memory
b, d - vis_tokens www.:toeessssseeeee. si)e((), vis_tokens (-1)
mem_op_state, mem_res_state mem_oppstates. appeed (mem_ (moponnmmemoo_sttataa op__t_ops
_ (nrrpaalllll b,d)
15
mem_ com__re__ee wwt_rre__stttss.e append (mem_ res_state)
for in range caluu__ttraaiv_seeeeee
#1. Form new set of latent operations from lang. token features new_opa, lang_ att
faarrrco__ormmaiiiaaaann...ooe tokens, semm_op_statee
21
#2. Ezecute new operations on vés. input to form new results new_ops_resultte vis_att mem mmm_res_staaoe
oiaerioon..cccttttoooiiistttttt nev_ops,
#3. Apply operation composition
hem__p__ttate mem_ree res_ res_stte_ = operation conpriiiioomooiiiiion ceppoositiinnnww_oooooo (new_ops, new_ops_results, mem_op_states, mem_res_states)
#4. Mointain memory states within lookback window
28
mem_op stattssteees...eeeessssssss append (men_op_state)
mem mem_res_ seem_res_state. append (mem_ree res_state)
ot_tteess mem_op_ op_states [min (-1, -mem mem_windov__ern :] cem_res_statee mem_res_staaee
(min(-1, mem_windov__en
33
#5. Store lang. and vis. atts for visualization apaetts.ppend vis_atts append (vis_att)
37
#6. 'Pool' final result
38
Final_result = pool_ final, final_result (mem_res, rea_state, mem op_stat,, sang_sumaryy_rep
return final resultt lang_atts, vis_atts
Figure 12: IPRM pseudocode (1/3)
25

---

#Below, "Lin" refers to lsnear layer $and ""MPP" refers to 2-layer multi- wults-layer-pereetrre layer def operation_formation (lang- tokens, (laen, #Bx M
prev_op_state #Bx Noxx Dm (Nop=num parallel ops)
):
#1. Form op "query" based on prior state
op-9 MLP_1(prer_op_stattee #paper eq.
#2. Use lang_ token_ as attn "key" and "value" (paper eq. 5)
lang_k k lang_ tokens
lang_v = tokens
13
#3. Retrieve new latent ops from lang. rep through attention latent_ops, lang_attn mod_attn(op_q. lang_k, lang_v, aang_attn_proj) #paper eq. 6; L194
14
return latent_ops, attg
18
def ierration_xxecttionn triistokens, #8x Nux Dm new_ops, #Bx Dm prev_res_stan #Bx #1. Form feature modulation weights (paper eq. 7)
23
$_v concat [Lin_ op (Lin__p(new__pps) Lin_res azis
#concat across feat.
s_v Lin_s(s_v)
26
#2. Form visual attention "key" (paper egs. and 9)
vis_ vis_red_rep rep_reedppp- mod_vis s_v vis_red_rep Nop sid_ vis_k MLP_v(concat vis,_ vis_r)d_repp)) #concat across feat. azis
sin_viivis_taa tokens)
#3. Form visual attention "query" and "value" (paper 10)
vis_q ain_oo_q(nev... vis_v (vis_ttooveeess
36
#4. Obtain new latent "results" through vis attention (paper eq. latent_resulta, vis_attn pod_ttnnvvis_g) vis_k, vis_v, vis_att.proo)
return latent_results, vis_attn
40
def mod_ aq,_attn(q, k, v, att_ppooj layer, attn_mask):
[K_mutt q*k #element-wse wise product
attn_wt att_proj llyerr (ayelayee_aaaaaaae (qk]m #linear projection (paper L194)
(attn_mask -1e30)) out (attn_wt v) sum)) #sum across feature azis
attn_st coftmax(attn__
return out, attn_vt
Figure 13: IPRM pseudocode (2/3)
26

---

def operation_.ommosittiooo (new_ ops,, #8x Nopx
2
new_res, #Bx Nopx Dm
mem_op_ otastee, #list of elements: Bx Nopx Dm mem_res_states #list of elements: Bxpo Dm
#1. Integrate new-ops results memory (paper eq. 12 inter_op_state Lin_op_u(new__pps Lin_res hin_rros_hme__ressstttteete
(wpppphhee__p_pttaaaeeeeeeeeeaee (-1])
Lin_res__unnw_rres
#2. Concat operation and result states memory lookback window wtsssttt__iiddottee concat (micat[[ntter.an coanterrpp_sttte, mem_op op_ateee) res res_states_windowe concat ((inter_res_statee mem_res_states))
12
#3. Form inter-operation queries and keys (paper eq. 14)
op- queries wttp:oppg@iitttooo______ op_keys ce..pp.(ap_sttte__iiii
16
#4. Form inter-operation values and res values (paper eq. 15-16) va_values Lin_op_v httprooopsstte__iiieeeeee vindowed)) res_values Lin.ree_((re_sstteeeeiiddd
#5. Compute inter-operation attention (paper eq. 17) sttn_mask new_op_state, op_attn_at mod_ (ootattooop_ueeeees op_ op_keys, op_ values, op_attn_proj, attn_mask)
seatrix(op.kessssszeiiii (1)) [:op_queries size(1))
#6. Obtain operation and result states (paper eq. 18-19) new_op_state new_op_state (immoop_u(inttt_opp_tttt new_res_state opaatta_ttrooovressstttt Lin_res_v (inter___ettrerr_stttttl
29
return stt_ee_ new jew_res_state
def wiiiiiinnnisttte_eeeeee state (nw_ppralllelops.
#slice specified num perallel ops from initialized params ~ (0,1))
34
op_ ip_init_atate = op_init_ppraml paaarammlmmmpprlllllle..o.
res res_init_stater ses_init_paran ceatt_param:nnum_ppp mumm_parallell_ps #broadcast batch-wise to get Bx opx Dm
return po_init_state..rped (b, res res_init_stace rb,
38
def pool hol_finnal. reaiall_reatttreeeeteeee pool_q q = (aa_pq@lan__amme summary_rep)
op_state, Lang_sumaryy_rep)
pool_k Lin_pk state)
return mod_attn mpdlatt pool_k, res_state)
Figure 14: IPRM pseudocode (3/3)
27

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Abstract and introduction reflects the motivation of method and experimental Answer: [Yes]
Justification:
results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
28

---

Justification:
Limitations are provided in appendix.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sellispecifiitiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-texi system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: Paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Benchmark and training details are provided along with module pseudocode and example CLIP integration. Source code for experiments will made publicly available along with checkpoints.
Guidelines:
29

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer:[Yes Yes]
Paper provides training and implementations details and provides module pseudocode. Full source code will be made publicly available upon acceptance.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ /mips.cc/ppblic//uuiie http://///.ateee/oooeesmsmiiaaaa www.iiissooooooooooooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
30

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Appendix provides details on hyperparameters and dataset specific settings for all experiments.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification: Appendix mentions that results were averaged over atleast 3 seeds for primary experiments.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Appendix mentions the type of GPU and its memory for all experiments along with batch size of experiments.
31

---

Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental rums well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: Authors reviewed code of ethics during submission.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer:[Yes)
Justification:
Paper mentions potential negative impacts of work in appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
32

---

Answer: answerNA
Justification: Paper does not explicitly pose such risks; however in limitations and potential negative impact this is mentioned.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
Licenses existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Yes, all coding libraries and datasets are properly cited and credited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperrswi theode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [NA]
Justification:No new asset
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
33

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Paper does not involve crowdsourcing or human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification:
Paper does not involve crowdsourcing or human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
34


Reviews:
[{"review_id": "V744Hruy7W", "reviewer": "Reviewer_oSE3", "summary": "This paper introduces the Iterative and Parallel Reasoning Mechanism (IPRM), a neural architecture for complex visual reasoning and question answering tasks. IPRM involves a three-step iterative process: Operation Formation, Operation Execution, and Operation Composition. In Operation Formation, IPRM generates new parallel operations by extracting relevant language information based on previous operation states. During Operation Execution, these new operations are processed simultaneously, retrieving pertinent visual information guided by both newly formed operations and existing result states. Finally, in Operation Composition, IPRM integrates the new operations and their results into memory by dynamically combining them with each other and with previous operation states.  Experiments are performed on video reasoning like STAR, AGQAv2, visual reasoning datasets like GQA, CLEVR.", "questions": "Please see weaknesses.", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "-\tThe motivation behind the architecture is interesting and sound.\n\n-\tThe experiments show improved results comparing with other methods on several benchmarks.\n\n-\tThe visualization is interesting and somehow can explain intuition of the model.", "weaknesses": "•\tSome notations need further clarifications: The paper needs to improve consistency and clarity in its notation. For example, it is unclear if K in Figure 2 and its caption is equivalent to N_op used elsewhere. Some variables in equations are not properly defined or explained, such as U and H in equation (9), and u2 and v2 in equations (15) and (16). \n\n•\tModel scaling:  How does the performance change if the number of parameters is increased/decreased? While the authors claim IPRM can be applied to different vision and language backbones, they don't provide evidence of its performance when scaling up. This omission leaves questions about the model's scalability and efficiency unanswered.\n\n•\tWhat is the intuition behind using fixed numbers of parallel operators and computation steps, given that different questions likely require different numbers of operations and reasoning steps? Is there any potential drawback of using fixed parameters in this context?\n\n•\tCan the author provide some analysis about failure cases of model?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful review and questions. We provide point-by-point clarifications below:\n\n**Some notations need further clarifications:**\nThank you for pointing these out and we apologize for the resulting confusion. \n- Yes, K in fig.2 is equivalent to N_op (which denotes number of parallel operations) used elsewhere. We will use N_op in fig. 2 (and its caption) in our updated paper version to avoid any confusion. \n- U and H in eq.(9) and u2 and v2 in eqs. (15) and (16) are simply used to specify different unique transformation weights and do not denote any variable by themselves. For clarity, in our updated paper, we will use W_{opU} and W_{opH} in eq.(9) (instead of W_{op,U}, W_{op,H}) and W_{opU’} and W_{resV’} in eqs. (15) and (16).\n\n**Model scaling and performance change when number of parameters is increased/decreased. Concerns on model’s scalability and efficiency when scaling up.**\n- Currently, in our ablative analysis in paper section 3.3 fig. 6, we studied four primary factors influencing our method IPRM’s performance and its resultant scale (in terms of parameter size and computational FLOPs). These are: i) number of iterative steps (T), ii) number of parallel operations (N_op), iii) reduction ratio (r), and iv) window length (W). As shown in fig.6, these factors have significant bearing on IPRM’s performance and more capable/higher-scale model variants are obtained as T, N_op and W are increased and r is decreased. E.g. a model with T=1,N_op=1 obtains ~ 75% acc. and when scaled to T=9,N_op=9 obtains ~82% (a 7% performance improvement). We have further reported the computational FLOPs and parameter sizes of IPRM when varying   above factors in table 1 of the attached rebuttal pdf. We will also include it in the appendix of our updated paper version for clarity. \n- Further, in appendix section B.3 table 7 we have also investigated the performance increments when IPRM is applied to different scales of backbone CLIP vision-language (VL) models. As shown, applying IPRM on the larger CLIP VIT-L/14 (369M params) backbone results in 3.4% and 4.3% performance increments on GQA and NLVR2 respectively in comparison to the smaller CLIP VIT-B/16 (149M params). Further, IPRM also outperforms scaling of additional Concat-Att or Cross-Att transformer blocks on CLIP while requiring lesser parameters and comparable GFLOPs. We believe these results help better illustrate performance of IPRM at different backbone model scales and also highlights how integrating IPRM with VL backbones such as CLIP can be more effective than adding further transformer blocks in context of visual reasoning tasks. \n\n**Intuition behind using fixed number of parallel operations and computation steps given varying question demands. Potential drawbacks of fixed number of operations.**\n- Thank you for raising this intriguing point. We agree that different questions can require different number of iterative and parallel operations based on complexity. Halting-based adaptive computation methods such as DACT [1], PonderNet [2] and early-exit strategies [3] have been specifically designed for such a capability. These can certainly be also applied to our architecture owing to its iterative computation nature. However, these approaches typically introduce additional learning losses and task-sensitive hyperparameters (such as computational budget or halting penalty) that can vary across tasks and datasets.\n\n- To enable our proposed reasoning model to be uniformly applicable to different image and video reasoning tasks without modification, we opted for weighted residual connections in updating the memory state (eqs. 15 & 16) over fixed steps. This implicitly enables the model to perform identity operations for steps where no additional computation may be required. This can be seen in reasoning visualizations in rebuttal pdf fig.1 and appendix fig. 8 to 11, where the lang. and vis. attention remains minimally changed b/w consecutive iterative steps. \n\n- A potential drawback of such fixed computation is that the amount of GFLOPs remains constant regardless of the question complexity. However, we note that this is true for other baseline reasoning methods and transformer-based models (such as SeViLA-BLIP2, Concat-Att, MIST, and MDETR) as well, and hence can be an interesting direction to explore in future. \n\n\n**Analysis about failure cases of model**\n- Currently, in appendix fig. 10 & 11 and examples in rebuttal pdf fig.1, we provide qualitative analysis for a few representative failure cases of our model. These largely stem from the model either not having sufficient commonsense or the model not utilizing visual information correctly. E.g. in fig. 10, for the question, “Are the two objects that are of a primary color, but not red, of the same shape?”, the model appears to not understand what a “primary color” is, and hence fails in its reasoning. Similarly, in the rebuttal pdf bottom-left example, the model appears to correctly find the ‘device that is on?’, but outputs an imprecise label (‘computer’) instead of the ground truth label (‘monitor’). \n\n- We have also performed quantitative analysis on the impact of visual encoding quality on our model’s performance in appendix table 5 (left). As shown, when utilizing ground-truth visual inputs, our model can achieve a 9.4% performance boost on STAR. Similarly on GQA, we observed a 25% performance boost compared to when using predicted visual inputs. This suggests that many errors of our model may stem from imprecise backbone visual outputs rather than the reasoning process itself, and can thereby be improved with advancements in visual backbones.   \n\n[1] Eyzaguirre, Cristobal, and Alvaro Soto. \"Differentiable adaptive computation time for visual reasoning.\" CVPR. 2020.\n\n[2] Banino, Andrea, Jan Balaguer, and Charles Blundell. \"Pondernet: Learning to ponder.\" arXiv preprint arXiv:2107.05407 (2021).\n\n[3] Guo, Qiushan, et al. \"Dynamic recursive neural network.\"CVPR. 2019.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ea3xfT2AhD", "reviewer": "Reviewer_nzfV", "summary": "The paper proposes an iterative and parallel reasoning mechanism (IPRM) for VQA tasks, combining step by step iterative computation with parallel processing of independent operations, and also maintaining an internal memory of operation states and results states. Firstly at each reasoning step, previous operation states attend to language features to form latent operations, then latent results are formed by attending to visual features based on the newly formed latent operations and prior result states. Finally new memory state is computed by combining the latent operation and result states alongwith prior memory states with a lookback window. IPRM outperforms previous methods on several complex image and video question answering tasks. IPRM is also lightweight and sample efficient. Ablation experiments demonstrate the importance of different components.", "questions": "1. What is the reason behind applying a MLP with a nonlinear activation instead of just linearly transforming prior operations to get query in Eq 4, as is done in standard attention mechanisms?\n\n2. Can the authors perform some ablations on operation execution component, regarding the importance of using previous result states and newly formed operations in forming attention key?\n\n3. Based on the ablation plots, figure 6 (iii) it seems r=1 performs better than r=2, then why was r=2 chosen for the experiments? \n\n4. For the window length ablation it would be informative to perform the ablation for W>3, to make sure the performance doesn’t increase further with window length. Also it seems W=3 performs slightly better than W=2?\n\n5. How does IPRM do on the CLOSURE [1] dataset (both zeroshot and finetuned)?\n\nSuggestions:\n\n1. It would be informative to also include the MAC model in the comparison of the parameters and sample efficiency in Figure 5.\n\n2. Some of the recent prior work on visual reasoning [2,3,4] haven’t been cited.\n\n3. The authors should include a discussion of limitations/future directions in the main paper.\n\n[1] - Bahdanau, D., de Vries, H., O'Donnell, T.J., Murty, S., Beaudoin, P., Bengio, Y. and Courville, A., 2019. Closure: Assessing systematic generalization of clevr models. arXiv preprint arXiv:1912.05783.\n\n[2] - Mondal, S.S., Webb, T. and Cohen, J.D., 2023. Learning to reason over visual objects. arXiv preprint arXiv:2303.02260.\n\n[3] - Webb, T., Mondal, S.S. and Cohen, J.D., 2024. Systematic visual reasoning through object-centric relational abstraction. Advances in Neural Information Processing Systems, 36.\n\n[4] - Mondal, S.S., Cohen, J.D. and Webb, T.W., 2024. Slot abstractors: Toward scalable abstract visual reasoning. arXiv preprint arXiv:2403.03458.", "limitations": "The authors have discussed the limitations in the appendix.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well written and easy to follow.\n2. A novel way of combining iterative and parallel processing for VQA tasks.\n3. The approach demonstrates decent improvements on three video (STAR, AGQAv2, CLEVERER-Humans) and three image question answering tasks (CLVER-Humans, CLEVR-CoGenT, and GQA), alongwith the capability to generalize zeroshot and in out of domain settings (CLEVR-CoGenT).\n4. The approach also requires fewer training samples and parameters.", "weaknesses": "No major weaknesses, please see questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review, questions and suggestions. We provide point-by-point clarifications below:\n\n**Reason behind applying a MLP with nonlinear activation to get query in eq. 4**:\nIn applying a nonlinear activation for eq.4, we drew inspiration from the cross-attention transformer architecture. In the cross-attention transformer, after each attention block, an MLP is applied which comprises nonlinearities on top of linear projection to model a nonlinear transformation. The output of this MLP then becomes the new ‘query’ for the next transformer block. Hence, for our method, the MLP in eq.4 similarly utilizes a nonlinearity to model a nonlinear transformation and obtain more nuanced ‘queries’ for operation formation at each iterative step. We have also ran an experiment to compare performance without any nonlinearity, and as shown in the table below the next paragraph, IPRM-no-nonlin results in a 1.2% lesser accuracy compared to the original model. \n\n\n**Ablations on operation execution component:**\nWe have done two further ablations to analyze the impact of using previous result states (M_{res,t}) and newly formed operations (Z{op,t}) in forming the attention key for operation execution. As shown below, we find that using only M_{res,t} (w/o new-form-ops) results in 0.4% drop while using only Z{op,t} (w/o prev-res-state) results in a 1.1% drop suggesting the previous result states M_{res,t} are more informative in forming the attention key. We will include these results in an elaborated ablations section in the appendix of our updated paper.\n\n| Model | Acc. (%) | \n| --- | --- |\n| IPRM-original | 82.3 |\n| w/o op-form-nonlin | 81.1 |\n| w/o new-form-ops-key | 81.9 |\n| w/o prev-res-state-key | 81.2 |\n\n\n**Why choose r=2 when r=1 performs better?**\nWhile r=1 does perform better than r=2 by 0.3% accuracy, it requires 0.9M more parameters and 14.5GFLOPs more compute than r=2. Hence, we had chosen r=2 for our optimal model configuration given its parameter and computational efficiency. We will elaborate on this reason in L267 of the updated paper version. We have additionally also reported the breakdown in number of parameters and GFLOPs for different configurations of our model in rebuttal pdf table 1 and will also include it in the appendix of our updated paper.\n\n**Results for window size (W) > 3 and W=3 performs better than W=2?**\nWhile W=3 does perform marginally better (0.1%) than W=2, similar to the above point, we adopted W=2 as our optimal model configuration as it required 0.9 GFLOPs less compute than W=3. We additionally ran a further experiment with W=4, and found it again does only marginally better (0.13%) better than W=2 while requiring 1.7 GFLOPs more.\n\n**Performance on CLOSURE:**\nWe have reported zero-shot results of our model IPRM on CLOSURE below. Note, similar to previous methods (e.g. MAC, FILM, NS-VQA, etc) on this benchmark, we find high standard deviation values ranging from 6% to 8% (currently with 2 random seeds) unlike observed in our experiments with other datasets (where deviation was found to be generally below 0.7%). Further, we have not been able to run finetuned results as of now as the dataset repository does not provide the used finetuning set. IPRM outperforms prior methods on the and_mat_spa, embed_spa_mat and or_mat splits but particularly lags behind on compare_mat and compare_mat_spa splits.\n\n| Model (ZeroShot) | and_mat_spa | embed_mat_spa | embed_spa_mat | or_mat | or_mat_spa | compare_mat | compare_mat_spa |\n| ------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- | \n| IPRM | 83.1±6.4 | 80.3±7.2 | 99.2±0.1 | 75.7±9.4 | 67.1±8.1|  62.2±7.3 | 61.5±5.8 |\n| FiLM | 41.4±18 | 62±11 | 93.4±2 | 34.4±11 | 35.2±5.5 | 66.2±8.5 | 65.8±5 | \n| MAC | 63.7±25 | 76.8±11 | 99±0.15 | 75.2±13 | 70.4±15 | 65.3±8.6 | 66.2±6.4 | \n| NS-VQA* | 33.3±18 | 97.3±4.6 | 98.3±5 | 69.5±28 | 50.4±27 | 96.1±6.2 | 95.4±6.7 |\n| PG-Tensor-NMN* | 37.6±17 | 79.8±1.4 | 95.6±5.6 | 34.1±8.1 | 25.5±11 | 89.2±2.3 | 89.8±2.7 |\n| PG-Vector-NMN* | 32.5±17 | 97.3±2.4 | 97.2±3.9 | 64.9±25 | 47.4±23 | 95.3±5.7 | 94.4±6.4 |\n\n\n*Methods use annotated programs during training for program parser\n\n\n**Include MAC in comparison of parameters and sample efficiency.**\nWe have added MAC model in rebuttal pdf fig. 1 (bottom right) for comparison and will update the fig.5 of main paper.\n\n**The authors should include a discussion of limitations/future directions in the main paper.**\nThank you for mentioning these works on visual reasoning. We will cite and discuss them in the related work section of our updated paper. We will also move the section on limitations/future directions from appendix to the main paper in our updated version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fiNzPxBENh", "reviewer": "Reviewer_3tYc", "summary": "This paper introduces the Iterative and Parallel Reasoning Mechanism (IPRM), a novel neural architecture designed to enhance visual question answering (VQA) by combining iterative and parallel computation. The IPRM aims to address the limitations of existing methods that rely solely on either iterative or parallel processing. The iterative component allows for step-by-step reasoning essential for complex queries, while the parallel component facilitates the simultaneous processing of independent operations, improving efficiency and robustness.\n\nThe authors propose a lightweight, fully differentiable module that can be integrated into both transformer and non-transformer vision-language backbones. The IPRM outperforms current state-of-the-art methods across several benchmarks, including AGQA, STAR, CLEVR-Humans, and CLEVRER-Humans, demonstrating its capability to handle various complex reasoning tasks. Additionally, the mechanism’s internal computations can be visualized, enhancing interpretability and error diagnosis.", "questions": "1. Can the authors provide more insights into the scalability of IPRM? Specifically, how does the computational complexity of IPRM compare to other state-of-the-art methods?\n2. It would be helpful to include an analysis of the sensitivity of IPRM’s performance to its hyperparameters, such as the number of parallel operations and the length of the reasoning steps.", "limitations": "The authors acknowledge the limitations of their work, particularly regarding the potential scalability challenges and the need for further evaluation on diverse datasets.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The combination of iterative and parallel reasoning within a single framework is novel, addressing specific limitations of existing VQA methods.\n2. The IPRM demonstrates superior performance across multiple challenging VQA benchmarks, demonstrated by extensive experimental results.\n3. The paper is generally well-structured.", "weaknesses": "1. While the paper is well-structured, certain sections, particularly those describing the technical details of the IPRM, can be dense and challenging to follow.\n2. The paper could benefit from a more detailed discussion on the scalability of the IPRM, particularly in terms of computational resources and training time.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your informative review and useful suggestions. We provide point-by-point clarifications below:\n\n\n**Technical details can be dense and challenging to follow:**\nTo enable easier readability and understanding of the method, We have currently provided pseudocode mapping to each equation in appendix Pg. 30 -32. We will reference the pseudocode in the main paper\n\n**Scalability of IPRM in terms of computational resources used and training time; Comparison with other state-of-art methods**\nAs currently mentioned in appendix section C L627 on “Model implementation and experiment details”, we perform all our experiments on a single NVIDIA A40 GPU with 46GB memory and 6 CPU threads/number of workers. We will shift these details to the main paper.  \nFor further clarification and transparency, we report total training times and peak GPU memory consumption for primary datasets we have considered below:\n\n\n| Dataset | Training time | Peak GPU memory | \n| ---------- | ------------------ | ------------------------- |\n| STAR |        ~ 6 hours |  44GB |\n| AGQA | ~ 14 hours | 43GB |\n| CLEVRER-Humans | ~13 hours | 32GB |\n| CLEVR-Humans | ~10 hours | 33GB |\n| GQA | ~13 hours | 40GB |\n\nFurther, in appendix table 7 (left), we have currently reported the GFLOPs and parameter comparison of our architecture IPRM against different scales of baseline concat-attention and cross-attention transformer blocks. IPRM has 5.2M params and 5.9GFLOPs. In comparison, a 4 layer configuration of concat-attention has 13.6M params, 8.9GFLOPs and a 4 layer configuration of cross-att has 17.6M params, 3.1 GLOPs. Hence, IPRM exhibits higher parameter efficiency than both concat-attention and cross-attention transformer blocks while consuming a comparable number of GFLOPs. \n\nRegarding computational comparison with existing videoQA and imageQA state-of-the-art methods, it is difficult to perform an apples-to-apples comparison with majority of these methods as they utilize different computational setups and additional pretraining objectives. Hence, in the table below, we have compared computational resources and training time of our model with two selected state-of-arts that have provided information on computational resources and complexity in their respective papers or source code. As shown, our model (IPRM) requires lesser GPUs and training time than the state-of-art videoQA method SEVILA-BLIP2, and has a higher inference speed (i.e. more samples are processed per second). Similarly for imageQA, we compare IPRM with a replication of the 6-layer concat attention transformer decoder used in MDETR, and find IPRM has marginally lesser peak GPU memory consumption and has comparable training time, while having a marginally higher inference speed.\n\n| Model | GPUs | Total Training time | Inference speed (sample/sec.) |\n| ------- | ------- | ------------------------- | ---------------------------------------- |\n| SEVILA-BLIP2 | 4 x 48GB (NVIDIA A6000) | 8 hours | 0.31 |\n| IPRM | 1 x 46 GB (NVIDIA A40) | 6 hours | 1.11 |\n\n\n\n\n| Model | Peak memory consumption | Total Training Time | Inference speed (samples/sec.) |\n| ------- | ------- | ------------------------- | ---------------------------------------- |\n| Concat-Att-6Layers (MDETR-configuration replication) | 37GB | 10 hours | 192 |\n| IPRM |  33GB | 10 hours | 211 |\n\n\n\n **Sensitivity analysis of IPRM to hyperparameters**:\nWe have provided ablative analysis of IPRM in fig. 6 page 8 of the paper with sensitivity analysis on four primary hyperparameters. These include: i) impact of number of iterative steps (T) and parallel operations (N_op), ii) impact of operation composition unit, iii) impact of reduction ratio and iv) impact of window length . In relation to your suggestion, the first subfigure of fig.6 reports model performances for different combinations of number of iterative steps and parallel operations (with values for each drawn from {1,3,6,9}). Notably, we find that a higher number of iterative steps and parallel operations in IPRM lead to higher performances.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fPdvvAO7Lm", "reviewer": "Reviewer_P4wf", "summary": "he paper addresses visual reasoning problems. It introduces a fully neural iterative and parallel reasoning mechanism (IPRM) that combines iterative computation with the ability to perform distinct operations simultaneously. Evaluation is performed on visual reasoning datasets such as CLVER, AGQA, CLEVR-Humans and STAR dataset.", "questions": "* Can the proposed architecture be scaled to provide zero-shot results on multiple datasets similar to current SOTA video-language models?\n* Can the proposed architecture handle multi-turn conversations?\n* The paper should compare/discuss SOTA approaches such as  \"Look, Remember and Reason: Grounded reasoning in videos with language models, ICLR 2024\" on STAR and “Coarse-to-Fine Reasoning for Visual Question Answering, CVPR Workshops 2021” on GQA.", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The proposed method is novel and interesting. \n* The paper is well written and easy to understand. \n* The paper includes in-depth analysis of the proposed method.\n* The method shows promising zero-shot performance on CLEVRER-Humans,", "weaknesses": "* Generalization Results: The method shows zero-shot results only on CLEVRER-Humans. This is in contract to current SOTA video-language models such as LLaVA-Next or LLaMA-Vid which show zero-shot results across multiple datasets such as MSRVTT to TGIF-QA. Can the proposed model architecture be scaled to create more general purpose models similar to current video-language models?\n\n* Multi-turn QA (related to the above point): The standard transformer architecture allows for multi-turn dialogue, where the user can ask the model multiple questions about a video (e.g. LLaVA-Next or GPT-4V). Can the proposed model architecture be extended to allow for multi-turn dialogue?\n\n* Results on STAR: Recent works such as \"Look, Remember and Reason: Grounded reasoning in videos with language models, ICLR 2024\" outperform the proposed model.\n\n* Results on GQA:  “Coarse-to-Fine Reasoning for Visual Question Answering, CVPR Workshops 2021” achieves 72.1% compared to 60.5% reported for IPRM. (Note that the use of scene graphs during training should not be a reason not to compare, instead the paper should discuss approaches to close the performance gap).\n\n* The reasoning steps in Figure 7 are not very interpretable. In Figure 7 (left) the model does not seem to attend to the large cylinders in the first step. Similarly, in the Figure 7 (middle) the bottom reasoning chain seems to empty steps where the model does not look at any object. It would also be interesting to see the reasoning chains in case of real-world data such as STAR.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your informative review and insightful questions. We provide point-by-point clarifications below:\n\n**Can proposed architecture be scaled to a video-language model that can provide zero-shot results on multiple video-datasets?**\nScaling our architecture (IPRM) to a video-language model through integration with large-scale foundation models and instruction-tuning approaches such as done in LLaVA-Next or LLaMA-Vid is indeed an intriguing avenue of research.  Since IPRM receives language and visual token inputs parallelly and is an end-to-end trainable module (similar to conventional transformer blocks), we believe it can be effectively applied on top of transformer-based large-scale foundation models and be directly trained end-to-end with necessary pretraining and instruction-tuning objectives for general video-language models. \nHowever, we note that this scaling will be a significant research undertaking in itself (in terms of computational resources for pretraining/instruction tuning and constituent experiments). Hence, we believe this will be an exciting avenue of research to comprehensively explore in future work. Currently, for this paper, we believe we have already conducted extensive work in i) designing the IPRM architecture, ii) performing thorough architectural analysis, and iii) showing performance improvements across multiple complex videoQA and imageQA benchmarks (such as STAR, AGQAv2 and CLEVR-Humans), which remain challenging even for finetuned state-of-art vision-language models and reasoning methods (such as SeViLA-BLIP2 and MDETR).\n\n**Can the proposed architecture handle multi-turn conversations?**\nAs stated in our previous response, our proposed IPRM architecture receives language and visual token inputs parallelly and is an end-to-end trainable module  similar to conventional transformer blocks. We hence believe IPRM can exhibit multi-turn conversation capabilities observed in transformer blocks when combined with vision-language foundation models and instruction-tuning objectives (such as as in LLaVA-Next and LLaMA-Vid). Additionally, since IPRM explicitly models an internal memory state, it may be particularly advantageous for multi-turn conversational tasks that require the retention and recall of past dialogues or interactions in memory over long periods. We believe this will be another exciting avenue for future work, which we can jointly explore with the aforementioned direction on scaling IPRM  to a large-scale video-language model. We will add these points and necessary discussion to our paper’s section on “Limitations and Future Work” (appendix sec. D), and move the section to the main paper in our updated version.\n\n**Discussion of SOTA approaches such as “Look, Remember and Reason” (LRR) and “Coarse-to-Fine Reasoning” (CFR)**\nThank you for mentioning these works. We will discuss both these works in related work as well as add them to table 1 and table 4 of our paper. We note that our work’s core contribution is a new neural reasoning architecture, while these works primary contribution is introducing necessary surrogate tasks (in LRR) and coarse-to-fine training strategies (in CFR) to address compositional visual reasoning. As such, we believe our work can complement these methods by incorporating their proposed surrogate tasks and training strategies. \n\nFurther, inspired by methods that utilize scene graphs to bridge the performance gap, we performed an experiment using symbolic object labels and attributes from ground truth GQA scene graphs as inputs to IPRM. We interestingly found that in such a setting, our model can achieve 87.2% performance on the GQA validation set resulting in approx. 25% performance improvement over utilizing predicted object visual features. We have similarly found that on STAR, with ground truth scene graphs, our model can achieve 79.7% on the test set (approx. 9% improvement over predicted visual inputs).  These results further indicate that the performance gap on visual reasoning can indeed be closed by obtaining more accurate backbone scene graph models. We will include these results and related discussion in appendix section B.1.\n\n**The reasoning steps in Figure 7 are not very interpretable.**\n- We acknowledge that visualization of reasoning steps may be unclear  at first glance (especially for synthetic datasets such as CLEVR). We have plotted the parallel operations vertically and iterative steps horizontally. In our view, the model does show a correct and sensible reasoning trace across iterative steps. \n\nE.g. In the paper fig. 7 (leftmost scenario), the first parallel operation (top left) does indeed attend to two large cylinders, meanwhile the bottom two parallel operations for that iteration attend generally to different objects in the image (possibly to compute results for \"max occuring color\"). In the subsequent 2nd iterative step, the model more sharply identifies the two large cylinders across parallel operations, while in the third step it again sharply filters the correct 'large cylinder having the max occuring color' (i.e. blue). Finally in the last step, the model attends to the correct object behind the identified cylinder.  \n\nSimilarly, in fig. 7 (middle scenario), the bottommost reasoning chain does attend to certain objects in iterative step 2 and 4. We wish to clarify that the the certain cases where there is no visual attention is a reflection of identity operations (that arise due to the inclusion of weighted residuals in memory state update as specified in eqs.15 & 16).\n\n\n**Reasoning chains in case of real-world data**\nWe agree that it can be more interesting and perhaps also clearer when visualizing reasoning on real-world data. For this, we had provided visualizations on the GQA dataset in the suppl. jupyter notebook (IPRM_CLIP_Visualization.html) and also in fig.1 of uploaded rebuttal pdf. These examples better illustrate our model's entailed reasoning chains.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their time in reviewing the paper and for their positive and thoughtful feedback. We are encouraged that they found our proposed architecture and underlying motivation to combine iterative and parallel computation for complex visual reasoning tasks to be novel (nzfV, 3tYc, P4wf), interesting and sound (oSE3, P4wf). 

Further, we are pleased that they found the paper to be well written and generally well-structured (P4wf, nzfV, 3tYc) and the experiments to be extensive with decent performance improvements (nzfV, 3tYc, oSE3) across several challenging videoQA and imageQA benchmarks along with an in-depth analysis of the method (P4wf) and how it requires fewer training samples and parameters (nzfV).

We have addressed the concerns raised by the reviewers pointwise. Some **primary concerns** are highlighted below with a brief description of our response:

**Reviewer oSE3** -- We have i) addressed concerns regarding _scalability of the model and efficiency of the model when scaling up_ (with model ablations and results using different CLIP backbones), ii) provided intuition behind using fixed number of parallel operations and computation steps given varying question demands, and iii) provided analysis about failure cases of model. We have also clarified some confusion regarding the notation which we will rectify in the updated paper.

**Reviewer nZFw** -- We have added _zero-shot results of IPRM on CLOSURE dataset_ where IPRM demonstrates superior performance on multiple dataset splits. We have also clarified our motivation behind query formation, reported additional ablations regarding 'key formation' in operation execution, and included the MAC model in comparison of parameters and sample efficiency of visual reasoning methods. 

**Reviewer 3tYc** -- We have provided more extensive details on the scalability of IPRM in terms of computational resources and training time. We have also compared it with baseline transformer modules and state-of-art methods on multiple scalability factors such as peak GPU memory usage, training time and inference speed. Further, we have also specified ablations related to sensitivity analysis of model hyperparameters.

**Reviewer P4wf** -- We have provided directions on how IPRM can be scaled to a video-language model that can provide zero-shot results on multiple video-datasets and support multi-turn conversations given its similarity to conventional transformer modules in terms of parallel processing of inputs and end-to-end modular integration with vision-language models. We also discuss how ideas in SOTA methods such as  “Look, Remember and Reason” (LRR) and “Coarse-to-Fine Reasoning” (CFR)  can be used to make IPRM even better and provide an experiment to validate one such idea based on usage of symbolic scene graph inputs. Further, we have demonstrated language and visual attention visualization on a real-world dataset i.e. GQA, and provided clarifications regarding model interpretability visualization for paper fig. 7.

We hope we have addressed the major questions and concerns of reviewers, and look forward to further discussion.

Meta-review: Learning to

Summary of Review Process:
- 4 reviews.
- Leaning positive scores: 5,6,7,7
- Authors submitted rebuttals/responses
- 1 of 4 reviewers did not respond to rebuttal

Meta Review: 
In this work, LLMs and image generation models are leveraged to create synthetic image-text pairs for VLM training. An LLM generates captions and a T2I model uses these captions to generate synthetic images. 

Reviewers appreciated the following points about this paper:
- novelty of the framework, especially combining iterative and parallel reasoning processes
- extensive experiments and clear demonstration of performance improvement
- well written / well structured manuscript

Reviewer oSE3 asked about scalbility and efficiency, intution behind number of parallel operators, failure analysis, and notation confusion. The reviewer did not acknowledge the authors' response. The AC has read the responses and thinks that these address the questions. The AC recommends that it would be important to integrate these responses into the final version to improve readibility and the flow of the paper

Reviewer nZFw asked about the motivation behind using an MLP with nonlinear activattion, additional ablations, additional results on CLOSURE, additional baseline MAC, error analysis, and limitations section.  The reviewer deemed the authors' responses satisfactory and increased the rating to 7 (Accept).

Reviewer 3tYc suggested some improvements to the structure of the paper and a detailed discussion about IPRM, including scalability and sensitivity analysis.

Reviewer P4wf asked about generalization on multiple video dataset, multiturn conversations, comparison to recent work and improvements to the design of Fig 7.

AC: There is clear support from all reviewers to accept this paper. The questions raised by the reviewers were important and the authors were able to address those during the discussion phase. I strongly encourage the authors to incorporate these changes and integrate clarifications and discussions into the final paper.
Decision: Accept (poster)