Learning to Reason Iteratively and Parallelly for Complex Visual Reasoning Scenarios

Authors: shantanu  jaiswal
debaditya  roy
basura  fernando
cheston  tan
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 22:30:23

Abstract: Complex visual reasoning and question answering (VQA) is a challenging task that requires compositional multi-step processing and higher-level reasoning capabilities beyond the immediate recognition and localization of objects and events. Here, we introduce a fully neural Iterative and Parallel Reasoning Mechanism (IPRM) that combines two distinct forms of computation -- iterative and parallel -- to better address complex VQA scenarios.  Specifically, IPRM's "iterative" computation facilitates compositional step-by-step reasoning for scenarios wherein individual operations need to be computed, stored, and recalled dynamically (e.g. when computing the query “determine the color of pen to the left of the child in red t-shirt sitting at the white table”). Meanwhile, its  "parallel'' computation allows for the simultaneous exploration of different reasoning paths and benefits more robust and efficient execution of  operations that are mutually independent (e.g. when counting individual colors for the query: "determine the maximum occurring color amongst all t-shirts'"). We design IPRM as a lightweight and fully-differentiable neural module that can be conveniently applied to both transformer and non-transformer vision-language backbones. It notably outperforms prior task-specific methods and transformer-based attention modules across various image and video VQA benchmarks testing distinct complex reasoning capabilities such as compositional spatiotemporal reasoning (AGQA), situational reasoning (STAR), multi-hop reasoning generalization (CLEVR-Humans) and causal event linking (CLEVRER-Humans). Further, IPRM's internal computations can be visualized across reasoning steps, aiding interpretability and diagnosis of its errors.

Main paper: 
 introduction: 
 Visual reasoning and question answering (VQA) at its core requires a model to identify relevant visual operations, execute those operations, and then combine their results to make an inference. Complex visual reasoning scenarios (depicted in fig. 1 ) are particularly challenging in this regard. They require models to reason compositionally over a large number of reasoning steps and to engage in a variety of higher-level reasoning operations such as causal linking, logical reasoning, and spatiotemporal processing that extend beyond core perception capabilities. In this context, two powerful computational priors exist -iterative and parallel. While each has its own limitations, when combined, they can complement each other and effectively address the challenges of complex VQA tasks. Specifically, iterative computation, wherein individual operations are identified and composed in a step-by-step manner, is a beneficial prior for multi-step reasoning scenarios explored by past VQA works [28, 25, 18] . However, pure iterative computation can exhibit limitations in Figure 1 : Complex VQA scenarios (CLEVR-Humans [35] , GQA [29] , CLEVRER-Humans [51] ), AGQA [20] and STAR [79] ) wherein combination of iterative (step-by-step) computation (blue phrases) and parallel computation (orange phrases) can be beneficial for reasoning. scenarios wherein the entailed visual operations are independent of one-another, or where distinct stimuli need to be processed and tracked simultaneously.For example, consider the first scenario shown in fig. 1 . When executing the language phrase "maximum occurring shape" (i.e. "what shape appears the most"), a purely iterative method would:(i) compute the count of each shape (each of which itself could take multiple iterations), (ii) then update and maintain the counts in memory (without forgetting count of all previous shapes), and (iii) finally, recall each shape's count to compute the "maximum"1 . Besides taking more reasoning steps than required, such computation also increases the demand for information retention and recall in memory, which in this scenario could scale by the number of shapes to be counted. In complex video reasoning scenarios, a purely iterative method would similarly struggle in tracking and reasoning over multiple co-occurring events. In such scenarios, from both efficiency and efficacy perspectives, it is advantageous to process operations or stimuli in parallel, rather than solely iteratively.More generally, parallel computation facilitates the simultaneous maintenance and exploration of distinct reasoning paths, and thereby enables reasoning to be more comprehensive, efficient and robust. For example, to compute "maximum occuring shape", parallel compute can enable distinct shape queries to be simultaneously computed prior to computing the "maximum" operation. Similarly, it is effective for other scenarios illustrated in fig. 1 such as in processing independent logical clauses ("{are X} and {Y made of plastic}") or when tracking and processing co-occurring events in videos.Such computation can be implicitly realized in conventional transformer-based parallel attention mechanisms [70] . However, transformer-based attention does not explicitly incorporate iterative compositional computation [14, 41] , which as described is beneficial for multi-step reasoning scenarios wherein operations need to be composed sequentially. Accordingly, while parallel computation may effectively compute the result of "maximum occurring shape" in fig. 1 , it would potentially struggle to integrate the result with further operations such as "green object with ..", "small object in front of green ..", and "color of .." that need to be computed step-by-step to answer the question.Based on the above insights, we design the Iterative and Parallel Reasoning Mechanism (IPRM), a novel neural reasoning architecture that combines step-by-step iterative computation with the ability to process multiple independent operations and stimuli simultaneously. Inspired by how humans utilize working memory [17, 9] to facilitate complex reasoning, IPRM internally maintains a latent memory of parallel "operation states", keyed to which are "results states". Given vision and language inputs, IPRM performs the following iterative computation. First, it forms a new set of parallel operations by retrieving relevant language information conditioned on its prior operation states. Then, it "executes" these operations parallelly by retrieving relevant visual information conditioned on its new operations as well as prior result states. Finally, it integrates its new operations (and their results) into memory by dynamically composing these operations with one-another as well as prior operation states, and subsequently, repeats the entire process in its next iterative step.This strategy effectively enables us to take advantage of both parallel and iterative computations and notably helps improve state-of-arts across various complex image and video reasoning tasks using a single reasoning mechanism. Equally importantly, IPRM's internal computations can be visualized are retrieved from language features X L conditioned on prior operation states M op . Then, visual features X V are queried conditioned on both Z op and prior result states results M res , to form the new results Z res . Finally, both Z res and Z op are passed to the Operation Composition Unit (see 2.3), the output of which becomes the new memory state M.across reasoning steps, which helps better interpret what operations it was doing and accordingly where it was looking visually when processing a complex reasoning scenario. 
iterative and parallel reasoning mechanism: 
 Our proposed iterative-and parallel-reasoning mechanism (IPRM) is a fully-differentiable neural architecture. Given visual features X V ∈ R N V ×D V and language or task-description features X L ∈ R N L ×D L , IPRM outputs a "reasoning result" y s ∈ R Dm and, optionally, a set of "reasoning result tokens" Y R ∈ R Nm×Dm . As previously mentioned, IPRM operates iteratively for T reasoning steps and internally, maintains an explicit memory M : {M op , M res }. The memory is modelled as a set of "operation states" M op ∈ R Nop×Dm , keyed to which are "result states" M res ∈ R Nop×Dm as shown in fig. 2 . Here, N op denotes the number of parallel operations to be computed while D m denotes the mechanism's internal feature dimension. On a high level, at each reasoning step (denoted by t ∈ {1, • • • , T }), IPRM performs the following computations:1. First, conditioned on the existing operation states M op,t , we retrieve relevant information from language or task-description features X L to form a new set of latent operations Z op,t ∈ R Nop×Dm . We term this computation as "Operation Formation".EQUATION2. Then, conditioned on the latent operations Z op,t and the existing result state M res,t , we attend and retrieve relevant information from visual features X V which represents a new set of latent results Z res,t ∈ R Nop×Dm corresponding to Z op,t . We term this computation as "Operation Execution".EQUATION3. Finally, to facilitate interaction amongst parallel operations, we perform inter-operation attention. Here, each operation Z op_k,t ; k ∈ {1, EQUATIONAs shown in eq. ( 3), this output is the new memory state M t+1 : {M op , M res }.The overall computation flow is illustrated in fig. 2 , and we provide specific details and intuitions behind these computations in the following sub-sections.The "operation formation" stage conceptually models a reasoner that based on its prior set of operations, decides what language features to retrieve in order to form the next set of relevant operations. This can be effectively implemented through conventional attention mechanisms. Specifically, the cumulative set of prior operations (maintained in M op,t ) can be projected to form the 'query' Q L,t ∈ R Nop×Dm representing "what features to look for". The language features X L can be projected to form the 'key' K L ∈ R N L ×Dm and 'value' V L ∈ R N L ×Dm . Finally, the new set of latent operations Z op,t can be retrieved by computing attn(Q L , K L , V L ). These steps are formally represented below:EQUATIONHere,W L,q2 ∈ R Dm×Dm , W L,q1 ∈ R Dm×Dm , W L,k ∈ R Dm×D l and W L,v ∈ R Dm×D l .Note K L and V L are not computation-step dependent and only computed once. We use a simple linearmodulated formulation (with appropriate broadcasting and projection weight W a ∈ R D k ×1 ) to implement attn(.) (further details in appendix sec. 13).In the "operation execution" stage, the reasoner determines what visual features need to be retrieved depending on both the newly formed operations and existing result states. To model the constituent visual attention mechanism, we draw insights from existing recurrent visual reasoning methods [28, 69] that incorporate feature modulation for memory-guided attention. Specifically, we retrieve a set of feature modulation weights S V,t ∈ R Nop×Dm/r through a joint projection of the new operations Z op,t and prior results M res,t as shown in eq. ( 6).EQUATIONHere, r is a feature reduction ratio [23, 31] . S V,t is then applied dimension wise to a projection of X V to retrieve an intermediate attention key K ′ V,t ∈ R Nop×N k ×Dm/r . The final attention key K V,t is then obtained through a joint multi-layer-projection of K ′ V,t and the previously projected representation of X V as shown in eq. (7) .EQUATIONFinally, the attention query and value are formed through separate projections of Z op,t and X V respectively. These are then fed together with K V,t to the attention function to retrieve the new operation results Z res,t as shown in eq. ( 8). Intuitively, the overall process allows for both prior results and the new set of operations to jointly guide visual attention.EQUATIONHere,W V,op ∈ R Dm/r×Dm , W V,res ∈ R Dm/r×Dm , W V,s ∈ R Dm/r×2Dm/r , W V,k1 ∈ R Dm/r×Dv , W V,k2 ∈ R Dm/r×2Dm/r , W V,k3 ∈ R Dm/r×Dm/r , W V,q ∈ R Dm/r×Dm and W V,v ∈ R Dm×Dm .Finally, in the "operation composition" stage, the reasoner first integrates the executed operations Z op,t and their results Z res,t into the existing memory state M t through a simple recurrent update as shown in eqs. ( 9) and (10) . Then, to mitigate redundancy amongst parallel operations and to retrieve relevant knowledge from prior-step operations, it dynamically composes individual operation states M ′ op,t+1 with other operation states in M ′ op,t+1 and also prior operation states in M op,t-W:t . Here, W is an attention look-back window. This composition is achieved through computing inter-operation attention as illustrated in fig. 3 . Specifically, M ′ op,t+1 is projected to obtain a set of queries Q op,t , while the token-wise concatenation of M ′ op,t+1 and M op,t-W:t are projected to obtain the operation attention keys K op,t and values V op,t . A second set of values V res,t are also formed through projection of respective result states as shown in eq. ( 14). Further, an identity attention mask I Nop is used to ensure that operations in Q op,t , can only attend to other operations and not themselves. This is done to enable a higher degree of operation composition. As shown in eq. ( 15), Q op,t , K op,t , V op,t and I Nop are passed to the attention operation, which outputs an intermediate representation M ′′ op,t+1 and the softmaxedattention weights A op,t . M ′′ op,t+1 is then added to a projection of M ′ op,t+1 to effectively combine attended operation states with the original operation states, and thereby form the next mem. operation state M op,t+1 .Finally, the next result states are obtained by applying A op,t on V res,t and then adding a projection of M ′ res,t+1 as shown in eq. ( 17). Note A op,t is specifically utilized to ensure that results are composed based on attentions between operation states. Here, all the mentioned weights W .. ∈ R Dm×Dm and [•; •] represents token-wise concatenation. I Nop in eq. ( 15) is an identity matrix which is concatenated with zeros (i.e. unmasked) for window tokens if window len. W > 0.EQUATIONEQUATIONEQUATIONEQUATIONEQUATIONEQUATIONEQUATIONEQUATIONEQUATIONObtaining Reasoning Summary As mentioned before, our proposed mechanism outputs a set of "reasoning result tokens" Y R and a "reasoning result" y s . Y R is simply equivalent to the last memory result states M res,T+1 . To obtain y s , we perform attention on the last operation states M op,T+1 by utilizing a summary representation l s ∈ R D l of X L as the attention-query. We set l s to be the first token in case of transformer-based language backbones and as last hidden state in case of LSTM-based language backbones. As shown in eq. ( 18), l s is projected to obtain a single-token attention query p q while M op,T+1 is projected to obtain the attention keys P k . The attention value is simply the result states M res,T+1 , and the output of the attention function is the "reasoning result". Intuitively, this computation corresponds to the reasoner deciding which final operation states in M op,T+1 are most relevant to the summary of the input language or task-description X L , based on which corresponding result states M res,T+1 are weighted and retrieved.EQUATIONHere, W pq,q ∈ R Dm×D l and W pk,k ∈ R Dm×Dm .Reasoning mechanism general applicability. Our proposed iterative and parallel reasoning mechanism is an end-to-end trainable neural module.It can be conveniently applied on top of different vision and language backbones, and be trained directly as a new computational block with no specific adjustments. Further, IPRM is weight-tied which means that its number of parameters is constant regardless of number of computation steps and parallel operations. We provide parameter and computational details along with module implementation in appendix sec. C. 3 ExperimentsWe evaluate IPRM on STAR [79] , AGQAv2 [20] and CLEVRER-Humans [51] for video reasoning tasks and CLEVR-Humans [35] , GQA [29] and CLEVR-CoGenT [34] for image reasoning tasks. For all tasks, we set IPRM's parallel operations (N op ) to 6, reasoning steps (T ) to 9, reduction ratio (r) to 2 and window length (W ) to 2 (informed by ablative analysis detailed in sec. 3.3). We follow task-specific practices (detailed in appendix C.1) for respective vision and language backbones in our primary experiments, besides also demonstrating integration of IPRM with large-scale VL backbones such as CLIP. Further, besides task-specific methods, we also consider two prominent transformer-based VL modules as baselines -concat-att (where lang. and vis. tokens are concatenated as in [36, 37] ) and cross-att (where lang. tokens are "query" to vis. tokens as "key" and "value"; as in [43, 1] ). Further implementation and training details are provided in appendix sec.C. 
video reasoning and question answering (star, agqav2 and clevrer-humans): 
 We first evaluate IPRM on recent video reasoning benchmarks. STAR [79] Interestingly, on STAR IPRM obtains an 8% and 7% improvement over SeViLA-BLIP2 on the predictive and sequencing scenarios respectively. This is possibly due to IPRM's capability to reason over multiple events simultaneously across frames, which may enhance its capacity to retrieve and cumulatively reason on relevant information needed to predict future events and determine appropriate sequences. However, IPRM performs less effectively than SeViLA-BLIP2 in feasibility scenarios, possibly because these scenarios require not only visual reasoning but also commonsense knowledge, which can benefit from integration with larger-scale vision-language backbones and auxiliary training tasks such as introduced in LRR [5] .Similarly, On AGQAv2, IPRM improves performances across various question types, notably achieving a 8% improvement in questions that require determining sequence of events. Further, IPRM also outperforms both 4-layer concat-and cross-attention modules on STAR (scaling further attentionlayers was not found to benefit performance as detailed in appendix table 7 ).Next, we evaluate IPRM on the CLEVRER-Humans benchmark [51] , which comprises synthetic videos of simultaneous object motions and multiple collisions, and tests a model's ability to determine causal links between events. We perform zero-shot, finetuned and from-scratch evaluation. As shown in table 2, IPRM outpeforms task-specific neurosymbolic models NS-DR and VR-DP as well as state-of-the-art ALOE across the three settings. Specifically, IPRM improves zero-shot per-question acc. by 7%, finetuned per-question acc. by 18.8% and scratch per-question acc. by 6.2%. These results further suggest that IPRM can better track and process co-occuring events, and in this case, more accurately determine causal links. Here, we evaluate IPRM on challenging compositional image reasoning benchmarks. CLEVR-Humans tests generalization of multi-hop reasoning to free-form human crowdsourced questions which entail reasoning skills/scenarios beyond a model's training on the original CLEVR dataset and provides a limited finetuning set (2.5% of CLEVR). Similarly, CLEVR-CoGenT tests generalization on novel attribute compositions not observed in training (e.g."gray cubes" and "red cylinders" are in training but eval. is on "gray cylinders" and "red cubes"; see suppl. for exact specification).The CLOSURE [3] benchmark further tests systematic generalization for different question type compositions.As shown in table 3 , IPRM achieves 3.9% and 3.8% improvements in zero-shot and fully-finetuned performance over prior state-of-art vision-language model MDETR. Further, IPRM neither requires bounding-box pre-training supervision (as done in MDETR) nor functional programs, and can be trained directly with only vision-language inputs and task supervision. fig. We also evaluate IPRM on the GQA benchmark which tests compositional VQA on real-world images. We perform comparisons with prior VQA methods as well as large-scale VL models such as LXMERT and 12-in-1 that do not utilize the ground truth scene graphs in GQA. As shown in table 4 , IPRM achieves 60.3% , outperforming prior reasoning methods such as MCAN and LCGN as well as large-scale VL models such as LXMERT and 12-in-1. However, IPRM's performance is behind the larger VL model OSCAR which performs pretraining on 4.3M data samples collated from COCO, VG, SBU and Flickr. Note, IPRM is a standalone reasoning module only trained on GQA balanced with no pre-training. Further, IPRM when trained with perfect perception (i.e. ground truth object bounding boxes and attributes), achieves 87.2% on GQA validation set suggesting strong reasoning performances can be achieved through advancements in visual detectors and backbones. Finally, in appendix B.3, we demonstrate that IPRM more effectively enhances the performance of frozen CLIP variants on complex reasoning benchmarks compared to scaling traditional cross-and concat-attention modules. We perform ablations to study the contributions of IPRM's salient components. First, we analyze the impact of varying number of parallel operations (N op ) against number of iterative computation steps (T ). We compare models with N op ∈ {1, 3, 6, 9} and T ∈ {1, 3, 6, 9}, resulting in 16 different models. Given the large amount of ablative models, we perform analysis on a reduced-resolution setting of CLEVR-Humans with pretraining on CLEVR for 15 epochs. As shown in fig. 6 (plot (i)), we find that T and N op appear to be co-dependent and that neither by itself can lead to high performance. E.g. setting T = 1 generally results in peformances around 75% regardless of N op , while setting N op = 1 or 3 results in a sharp performance drop of 3% when changing T to 9 from 6.In contrast, for N op > 3, we observe that performance increases steadily with higher T , suggesting that a higher number of parallel operations may prevent overfitting in a model with high computation steps. Overall, we find that (N op = 6, T = 9) and (N op = 9, T = 9) are the two best performing models achieving above 82% accuracy (with the former preferred as N op = 6 performs better for diff. T compared to N op = 9).Next, we study the impact of the operation composition block (OPC) by evaluating N op = 6 (and diff. computation steps T ) with and without OPC. As shown in fig. 6 (plot (ii)), while T = 1 has a relatively low drop of ∼2%, the performance drops are more significant for higher T . The (N op = 6, T = 9) model which reached ∼82% acc. with OPC, drops to ∼74% acc. without OPC.Finally, we study the impacts of the dimension reduction ratio (r) and memory-lookback window length (W ). As shown in fig. 6 (plot (iii)), r = 2 leads to negligible drop (0.3%) in performance compared to no dimension reduction (i.e. r = 1) while being computationally faster and requiring less memory. However, setting r to 8 significantly deteriorates performance. Similarly, as shown in fig. 6 (plot (iv)), a setting of window-length W is 2 works sufficiently well, and decreasing it below that significantly impacts performance. In fig. 4 , we assess the performance of IPRM across different functional program lengths (a proxy for reasoning steps) on the CLEVR and STAR benchmarks. As shown, IPRM maintains a high performance across both short and longer reasoning steps and for both image and video VQA scenarios. We also provide a condensed illustration of IPRM's reasoning visualization in fig. 7 .Here, for brevity, we only show the model's visual attention for a subset of parallel operations and computation steps (see suppl. for full visualization). The top left example ("what shape .. max occuring color") illustrates the model's usage of both iterative and parallel computation. At the first reasoning step, the model appears to be doing parallel operations to both identify "large cylinder" and compute "max occuring color". In the next step, it appears to have found "max occuring color" (cyan) and seems to check which of the two large cylinders match the color. Next, it highlights the correct cylinder and in the final step, it locates and predicts the correct object behind. The top right example in fig. 7 similarly shows computation for a question involving spatial ("farthest right"), similarity ("same shape") and counting ("how many") operations. The bottom left example illustrates IPRM's simultaneous language and visual attentions updated across reasoning steps when solving a real world (GQA) example. Finally, the bottom right example is an error-case where IPRM seems to incorrectly understand or execute requisite operations, thereby not producing expected attentions for "items with matching material and color" throughout steps. 
related work: 
 Visual reasoning methods and vision-language models. Multiple prior works have introduced effective visual reasoning methods in context of image and video question answering [35, 52, 2, 85, 15, 50, 69, 56, 32, 54, 53, 76] . Prominent works include NMN [2] , FILM [60] , NSM [27] , MAC [28] , MCAN [87] , NS-VQA [85] , ALOE [12] , VR-DP [13] , SHG-VQA [68] , MIST [18] and OCRA [76] . In contrast to these works that show applicability of methods for particular VQA benchmarks/tasks, our work explores a more general direction of integrating parallel and iterative computation in a reasoning framework that we show to be effective for multiple complex VQA benchmarks and reasoning scenarios. More recently, vision-language models [44, 73, 37, 66, 45, 46, 74] and multimodal large-language-models [57, 47] with transformer-based mechanisms have shown impressive reasoning capabilities at scale. Notable examples include CLIP [61] , GPT [57] , Gemini [67] , MDETR [36] , LXMERT [66] , VinVL [88] , BLIP [44, 43] , Flamingo [1] , LlavA [47] ,BEiT [74] and All-in-One [72] . We believe our work is complimentary to these developments, as it contributes an alternative and possibly more effective reasoning mechanism that can be integrated with such models in future to enhance complex VQA capabilities.Memory and recurrence-augmented transformers. Multiple works have identified limitations of purely feedforward computation as realized in transformers and worked on encoding recurrence [30, 26, 11, 71] and memory-augmented computation [80, 8] . Notably, Recurrent Memory Transformer [8] and MemFormer [81] introduce recurrent and dynamic memory to improve language modelling capabilities. More recently, EMAT [82] introduces efficient memory to augment knowledge retrieval, MemViT [80] introduces a cache memory to retain prior context for long-video tasks, and [71] introduces memory for more effective action anticipation. While these methods study recurrent and memory-augmented computation on specific natural language processing and computer vision tasks, our work focuses on the integration of iterative-parallel computation and working memory in a single neural reasoning mechanism beneficial for complex VQA scenarios. 
conclusion: 
 We introduced a novel fully-differentiable and end-to-end trainable iterative and parallel reasoning mechanism (IPRM) to address complex VQA scenarios. We comprehensively evaluated IPRM on various complex image and video VQA benchmarks testing distinct reasoning capabilities, and found it improves state-of-arts on multiple such benchmarks. We also performed quantiative ablations to study individual impacts of parallel and iterative computation besides qualitative analysis of IPRM's reasoning computation visualization.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer:[Yes]Justification: Paper provides training and implementations details and provides module pseudocode. Full source code will be made publicly available upon acceptance.Guidelines:• The answer NA means that paper does not include experiments requiring code. • At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable). • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
limitations and future work: 
 Here, we note possible limitations of IPRM. Similar to existing VQA and deep-learning methods, IPRM may reflect biases that are present in the training distribution of VQA benchmarks. This may lead it to overfit to certain image inputs or question forms and possibly provide skewed answers in such scenarios. Further, the utilized vision-language backbones in our experiments may also entail visual, language and cultural biases in their original training distribution which may permeate to IPRM upon integration for VQA scenarios. In this regard, we hope the capability to visualize intermediate reasoning of IPRM and diagnose its error cases (as shown in section 3.3) can serve a useful tool to benefit interpretability in VQA and identify possible reasoning biases that may emerge in the model.For future work, scaling the IPRM architecture to a foundational video-language model by integrating it with large-scale transformer-based vision-language models and relevant instruction-tuning approaches presents an exciting research opportunity. Moreover, while we designed and evaluated IPRM in the context of complex VQA, we believe it has the potential to operate as a general reasoning mechanism applicable to tasks beyond visual reasoning and question answering, such as language processing and embodied reasoning.A Appendix / supplemental material B Elaborated Experiments and Results Discussion B.1 Further comparisons on CLEVR-Humans, CLEVR-CoGenT, CLOSURE and NLVRv1Here, we provide further comparisons with benchmark-specific methods for CLEVR-Humans [35] , CLEVR-CoGenT [34] , CLOSURE [3] and NLVRv1 [63] (not reported in main paper due to space limitations). As mentioned in main paper, these benchmarks utilize synthetic images and are a test of pure visual reasoning capabilities that are minimally influenced by increased world knowledge or usage of stronger visual backbones.CLEVR-Humans as already mentioned in main paper evaluates a model's reasoning generalization capabilities to unseen scenarios or question forms. CLEVR-CoGenT studies compositional attribute generalization. Specifically, it has two conditions -i) cond.A wherein all cubes have color ∈ {gray, blue, brown, yellow} and cylinders ∈ {red, green, purple, cyan} (spheres can be any color), and ii) cond.B wherein color-sets are switched b/w cubes and cylinders. A model is then trained on one condition and evaluated on both the original and alternate condition. A higher accuracy on the alternate condition indicates that the model learns more 'compositionally' as it generalizes better to novel shape-color combinations with less feature/attribute combination overfitting. Finally, NLVRv1 evaluates language-grounded visual reasoning. Each sample of this benchmark comprises a set of three synthetic images and a composite natural language statement about the images which can evaluate to True or False and requires various visual-linguistic reasoning skills.As shown in table 5 , IPRM achieves state-of-art results across the three benchmarks and does not require pre-annotated bounding-boxes or functional programs as additional supervision. For CLEVR-Humans (table 5 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: Appendix provides details on hyperparameters and dataset specific settings for all experiments.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: Appendix mentions that results were averaged over atleast 3 seeds for primary experiments.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: Appendix mentions the type of GPU and its memory for all experiments along with batch size of experiments.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes]Justification: Authors reviewed code of ethics during submission.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: Paper mentions potential negative impacts of work in appendix.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? 
 
 Appendix: We provide results with frozen CLIP [61] visual backbones including CLIP VIT-L/14, CLIP VIT-B/16 and CLIP VIT-L/14@336px on GQA [29] , NLVRv2 [64] and CLEVR-Humans in table 8 . We empirically found utilizing a Distil-roberta language backbone to be more effective than the associated CLIP language backbone, and hence used the former for question processing. We compare with alternate prominent vision-language attention mechanisms including Cross-att and Concat-att blocks as well as a simple joint projection of vision and language pooled representations (referred as Wt-Proj-Att). As shown in the With regards to computational FLOPs, IPRM consumes 5.9GFLOPs which is marginally higher than Cross-Att 4-layer config (3.1GFLOPs) and lower than Concat-Att 4-layer config (8.9GFLOPs). Note, that the performance benefits of adding further layers of cross-or concat-att blocks are observed to be minimal after 4 layers, and can also depend on the amount of training data available. E.g. Both cross-and concat-att blocks of 2 layers had better performances on NLVRv2 (which has a limited set of training questions relative to GQA and CLEVR) in comparison to 4 layer config.We provide elaborate reasoning computation visualizations of IPRM showing the lang. and vis. attentions across parallel operations and computation steps during operation formation and operation execution stages. Fig. 8 shows a scenario wherein IPRM correctly utilizes parallel and iterative computations to compute intermediate operations of "find object close to front", "retrieve/compare shape and size", "find applicable objects with both same shape and size". Fig. 9 shows another correct prediction of IPRM, and this time, its intermediate reasoning visualization is useful to determine that the entailed reasoning appears sensible. Fig. 10 shows an incorrect prediction by IPRM and its intermediate reasoning visualizations also suggest that IPRM did not understand the question and thereby did not attend to relevant objects. Finally, Fig. 11 shows a scenario wherein while IPRM produces the correct answer, it's intermediate reasoning appears imprecise which makes the prediction (and underlying reasoning) less reliable. We provide further visualizations with a CLIP VIT-L/14 backbone on GQA samples in the supplemental jupyter notebook output (html format for easier viewing).We implement IPRM in PyTorch [58] as a generic vision-language module receiving a set of input vision (or scene-representation) tokens and input language (or task-representation) tokens. We provide Python-style pseudocode of IPRM in figs 12, 13 and 14. For all experiments, we set the internal dimension of IPRM to 512 and use the same configuration of num. parallel operations (N op )=6, num. computation steps (T)=9, reduction ratio (r)=2 and window size (W)=2. We follow benchmark-specific conventions for vision-language backbones that are detailed below in sec. C.1.For CLIP [61] , we utilize the official models from Huggingface [77] . All experiments are performed on a single NVIDIA A40 GPU with 46GB memory and averaged over 3 trials with different random seeds wherever possible (done for primary experiments on STAR, AGQA, CLEVRER-Humans, CLEVR-Humans). Unless otherwise specified, the learning rate is initialized to 1e-4 with Adam [38] optimizer and gradient clipping value of 8. The learning-rate is reduced based on validation acc. plateau with reduction factor 0.5, threshold 0.001 and patience 0. Further experiment hyperparameters and settings are provided below. Source code at: https://github.com/shantanuj/ IPRM_Iterative_and_Parallel_Reasoning_Mechanism.C.1 Benchmark-specific experiment details CLEVR-Humans. We use the CLEVR-Humans dataset from [35] which comprises images from original CLEVR dataset [34] and human crowdsourced questions. We use a batch size of 216 for training. We use the same language encoder (Distil-Roberta [48] from Huggingface [78] ) as in existing state-of-art MDETR [36] and frozen ResNet101 backbone layer 3 spatial features (as in [28, 52, 35] ). We perform all ablation experiments with 14x14x1024 visual features. Each ablation model is first pretrained for 10 epochs on the original CLEVR dataset (the initial learning rate for IPRM is 1e-4 and for language encoder is 1e-5) and then finetuned on CLEVR-Humans for 40 epochs with early stopping (learning rate of 1e-4 throughout). As observed in prior work [52] , we similarly found in multiple scenarios with occluded objects that visual attention only partially identified such objects. Hence, we simply resampled (bilinear sampling) visual input to obtain 16x16x1024 features and empirically found more complete visual attentions with a corresponding 1.1% improvement in accuracy. The final two best performing model configurations (Nop=6, T=9, W=2, R=2 and Nop=6, T=9, W=2, R=1) from ablations were then pre-trained for 35 epochs on CLEVR and finetuned on CLEVR-Humans. While we found that configuration Nop=6, T=9, W=2, R=1 obtains highest zero-shot (ZS) acc. of 65.6% and finetuned (FT) acc. of 86.3%, we adopt Nop=6, T=9, W=2, R=2 (with 63.3% ZS and 85.4% FT acc.) as our optimal model given its lesser parameters and FLOPs.GQA. We use the GQA compositional real-world image question answering dataset from [29] .Based on prior VQA methods on GQA [29, 25, 45, 33] , we utilize pre-extracted bounding-box object proposal features and object label predictions obtained from a pretrained object detector [19, 88] . The bounding box coordinates is normalized to range of 0 to 1 based on the original input image size, and the 4 coordinates are transformed to a distributed representation through a learned nonlinear projection. This representation is concatenated with a learned projection of the predicted object labels (initialized with glove [59] 300dim embeddings) to form the final visual input. We train IPRM for 25 epochs with a batch-size of 192 and same hyperparameters as before. We evaluate the final model on both the test-dev split and the official test evaluation server (https: //eval.ai/featured-challenges/225/evaluation).STAR-VideoQA. We use the STAR-VideoQA dataset for situational reasoning on real-world videos from [79] . Based on previous videoQA methods [79, 40, 45] for STAR, we utilize object bounding boxes, labels, human pose and human-object relations across frames (note: we do not use the situation hyper-graphs or functional programs). We first perform experiments with the provided ground truth object bounding boxes, labels and human-object relations as well as provided human pose predictions from Alphapose [16] as reported in main paper. Each of these is projected to a distributed representation through learned non-linear projections to obtain object token-wise representations. A further learnable positional embedding for each frame is added to these representations which are then flattened across frames to form the visual input to IPRM. For the language encoder, we found both a simple Bi-LSTM and Distil-roberta language encoder obtain similar performance, and hence choose the simpler Bi-LSTM as the language model. We evaluated models on both 16 sampled frames and 32 sampled frames, and empirically found using 32 frames has ∼1.3% higher performance and used it for our primary experiments. A batch size of 64 was used with learning rate 1e-4 over 20 epochs with early stopping. We evaluated the models on both the validation split and official test evaluation server https://eval.ai/web/challenges/challenge-page/1325/overview. For the all-predicted (no ground-truth) visual input setup, similar to [79] , we utilize a fasterRCNN [19] object extractor, ST-Trans scene graph extractor [10] and the same Alphapose predictor to obtain predicted object bounding boxes, labels, human-object relations and human poses. We observe a drop of ∼9% in predicted setup similar to observations in [79] , suggesting further performance can be achieved through better object and relationship detection backbones.AGQAv2. We use the AGQAv2 [20, 21] benchmark that comprises balanced training and test splits.We followed the same methodology as in STAR for language and visual backbones. Since AGQAv2 has a very large training and validation set, we found 8 epochs to be a sufficient number (with most performance increment observed in the 1st epoch itself).CLEVRER-Humans. We use the CLEVRER-Humans dataset introduced in [51] for temporal, physical and causal video reasoning which comprises videos from the original CLEVRER dataset [84] . Similar to in STAR-VideoQA and neurosymbolic models [84, 85] , we utilize a pretrained faster-RCNN based object localization and attribute prediction network from [84] . We again form object-level representations by concatenating learned projections of object-bounding box coordinates and predicted object attributes (i.e. color, shape and material). A frame-level learnable positional embedding is added and object-tokens across frames are flattened to form the final visual input to IPRM. For the language encoder, we used a simple bi-LSTM similar to existing methods. Note we do not use the functional programs or event causal graphs in our model. The batch size was 128 with a learning rate of 8e-5 and every 4 frames sampled (resulting on average 32 sampled frames). We evaluated models in the three setups -from scratch, zero-shot (CLEVRER-pretrained) and finetuned (CLEVRER-pretrained). Since the CLEVRER-Humans dataset is relatively small (comprising only 1076 questions ∼ 8 batches; ∼ 0.5% of original CLEVRER), for scratch training we trained for 250 epochs (with early stopping) while for finetuning we finetuned for 150 epochs (with 35 epochs for original CLEVRER training).CLEVR-CoGen. We use the CLEVR-CoGen dataset from [34] and follow the same setup as in CLEVR-Humans. We use a simpler bi-LSTM language encoder for experiments since the questions are synthetic program-generated unlike in CLEVR-Humans (crowdsourced free-form). We trained our model on condition A for 40 epochs (with early stopping) and used the best cond. A validation performance model to evaluate generalization performance on cond.B. For finetuning on cond.B we finetuned the best cond.A model for 20 epochs and used the best cond.B validation performance model to also evaluate on cond.A. All other hyperparameters are the same as mentioned for CLEVR-Humans.NLVR. We use the NLVRv1 and NLVRv2 datasets from [63, 64] . NLVRv1 comprises 3 synthetic images and a language statement while NLVRv2 comprises 2 real-world images and a lang. statement. For both datasets, the obtained visual tokens for each images was flattened to obtain the final visual input and an image-wise positional embedding was added to indicate image order. For the language encoder, we used a simple Bi-LSTM.In relation to VQA and deep-learning methods in general, the deployment of IPRM in real-world applications without thorough consideration of dataset or training distribution biases, could inadvertently reinforce existing vision, language and cultural biases present in the data, leading to erroneous outcomes or skewed answers. Further, the deployment of VQA methods such as IPRM in sensitive domains such as healthcare or scene/footage analysis could raise ethical concerns, including privacy violations, algorithmic reliability, and the potential for unintended consequences stemming from erroneous or biased predictions.Are there any objects that have both shape and size in common with the object that is found closest to the front? (GT: Yes; Pred: Yes) Reasoning Computation StepsWhat shape is the object closest to the gray object with the maximum occuring shape? Pred: cube GT: cube What shape is the object left of the blue small object with the maximum occuring shape? Pred: sphere GT: sphereFigure 11 : Example wherein IPRM produces correct answer but its visual attention trace suggests intermediate reasoning may be imprecise. The maximum occuring shape is cube; however both the blue small cylinder and blue small cube appear to be attended in the penultimate step as the "blue small object with max occuring shape" making the reasoning and prediction less reliable. • The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.38th Conference on Neural Information Processing Systems (NeurIPS 2024).Assuming "max" is applied after counts of all shapes are computed.pool_k = Lin_pk(op_state) 


Decision: Learning to

Summary of Review Process:
- 4 reviews.
- Leaning positive scores: 5,6,7,7
- Authors submitted rebuttals/responses
- 1 of 4 reviewers did not respond to rebuttal

Meta Review: 
In this work, LLMs and image generation models are leveraged to create synthetic image-text pairs for VLM training. An LLM generates captions and a T2I model uses these captions to generate synthetic images. 

Reviewers appreciated the following points about this paper:
- novelty of the framework, especially combining iterative and parallel reasoning processes
- extensive experiments and clear demonstration of performance improvement
- well written / well structured manuscript

Reviewer oSE3 asked about scalbility and efficiency, intution behind number of parallel operators, failure analysis, and notation confusion. The reviewer did not acknowledge the authors' response. The AC has read the responses and thinks that these address the questions. The AC recommends that it would be important to integrate these responses into the final version to improve readibility and the flow of the paper

Reviewer nZFw asked about the motivation behind using an MLP with nonlinear activattion, additional ablations, additional results on CLOSURE, additional baseline MAC, error analysis, and limitations section.  The reviewer deemed the authors' responses satisfactory and increased the rating to 7 (Accept).

Reviewer 3tYc suggested some improvements to the structure of the paper and a detailed discussion about IPRM, including scalability and sensitivity analysis.

Reviewer P4wf asked about generalization on multiple video dataset, multiturn conversations, comparison to recent work and improvements to the design of Fig 7.

AC: There is clear support from all reviewers to accept this paper. The questions raised by the reviewers were important and the authors were able to address those during the discussion phase. I strongly encourage the authors to incorporate these changes and integrate clarifications and discussions into the final paper.

Reviews: review: paper_summary: paper_summary: This paper introduces the Iterative and Parallel Reasoning Mechanism (IPRM), a neural architecture for complex visual reasoning and question answering tasks. IPRM involves a three-step iterative process: Operation Formation, Operation Execution, and Operation Composition. In Operation Formation, IPRM generates new parallel operations by extracting relevant language information based on previous operation states. During Operation Execution, these new operations are processed simultaneously, retrieving pertinent visual information guided by both newly formed operations and existing result states. Finally, in Operation Composition, IPRM integrates the new operations and their results into memory by dynamically combining them with each other and with previous operation states.  Experiments are performed on video reasoning like STAR, AGQAv2, visual reasoning datasets like GQA, CLEVR.
strength_weakness: strength_weakness: •	Some notations need further clarifications: The paper needs to improve consistency and clarity in its notation. For example, it is unclear if K in Figure 2 and its caption is equivalent to N_op used elsewhere. Some variables in equations are not properly defined or explained, such as U and H in equation (9), and u2 and v2 in equations (15) and (16). 

•	Model scaling:  How does the performance change if the number of parameters is increased/decreased? While the authors claim IPRM can be applied to different vision and language backbones, they don't provide evidence of its performance when scaling up. This omission leaves questions about the model's scalability and efficiency unanswered.

•	What is the intuition behind using fixed numbers of parallel operators and computation steps, given that different questions likely require different numbers of operations and reasoning steps? Is there any potential drawback of using fixed parameters in this context?

•	Can the author provide some analysis about failure cases of model?
questions: questions: Please see weaknesses.
limitations: limitations: N/A
paper_summary: paper_summary: The paper proposes an iterative and parallel reasoning mechanism (IPRM) for VQA tasks, combining step by step iterative computation with parallel processing of independent operations, and also maintaining an internal memory of operation states and results states. Firstly at each reasoning step, previous operation states attend to language features to form latent operations, then latent results are formed by attending to visual features based on the newly formed latent operations and prior result states. Finally new memory state is computed by combining the latent operation and result states alongwith prior memory states with a lookback window. IPRM outperforms previous methods on several complex image and video question answering tasks. IPRM is also lightweight and sample efficient. Ablation experiments demonstrate the importance of different components.
strength_weakness: strength_weakness: No major weaknesses, please see questions below.
questions: questions: 1. What is the reason behind applying a MLP with a nonlinear activation instead of just linearly transforming prior operations to get query in Eq 4, as is done in standard attention mechanisms?

2. Can the authors perform some ablations on operation execution component, regarding the importance of using previous result states and newly formed operations in forming attention key?

3. Based on the ablation plots, figure 6 (iii) it seems r=1 performs better than r=2, then why was r=2 chosen for the experiments? 

4. For the window length ablation it would be informative to perform the ablation for W>3, to make sure the performance doesn’t increase further with window length. Also it seems W=3 performs slightly better than W=2?

5. How does IPRM do on the CLOSURE [1] dataset (both zeroshot and finetuned)?

Suggestions:

1. It would be informative to also include the MAC model in the comparison of the parameters and sample efficiency in Figure 5.

2. Some of the recent prior work on visual reasoning [2,3,4] haven’t been cited.

3. The authors should include a discussion of limitations/future directions in the main paper.

[1] - Bahdanau, D., de Vries, H., O'Donnell, T.J., Murty, S., Beaudoin, P., Bengio, Y. and Courville, A., 2019. Closure: Assessing systematic generalization of clevr models. arXiv preprint arXiv:1912.05783.

[2] - Mondal, S.S., Webb, T. and Cohen, J.D., 2023. Learning to reason over visual objects. arXiv preprint arXiv:2303.02260.

[3] - Webb, T., Mondal, S.S. and Cohen, J.D., 2024. Systematic visual reasoning through object-centric relational abstraction. Advances in Neural Information Processing Systems, 36.

[4] - Mondal, S.S., Cohen, J.D. and Webb, T.W., 2024. Slot abstractors: Toward scalable abstract visual reasoning. arXiv preprint arXiv:2403.03458.
limitations: limitations: The authors have discussed the limitations in the appendix.
paper_summary: paper_summary: This paper introduces the Iterative and Parallel Reasoning Mechanism (IPRM), a novel neural architecture designed to enhance visual question answering (VQA) by combining iterative and parallel computation. The IPRM aims to address the limitations of existing methods that rely solely on either iterative or parallel processing. The iterative component allows for step-by-step reasoning essential for complex queries, while the parallel component facilitates the simultaneous processing of independent operations, improving efficiency and robustness.

The authors propose a lightweight, fully differentiable module that can be integrated into both transformer and non-transformer vision-language backbones. The IPRM outperforms current state-of-the-art methods across several benchmarks, including AGQA, STAR, CLEVR-Humans, and CLEVRER-Humans, demonstrating its capability to handle various complex reasoning tasks. Additionally, the mechanism’s internal computations can be visualized, enhancing interpretability and error diagnosis.
strength_weakness: strength_weakness: 1. While the paper is well-structured, certain sections, particularly those describing the technical details of the IPRM, can be dense and challenging to follow.
2. The paper could benefit from a more detailed discussion on the scalability of the IPRM, particularly in terms of computational resources and training time.
questions: questions: 1. Can the authors provide more insights into the scalability of IPRM? Specifically, how does the computational complexity of IPRM compare to other state-of-the-art methods?
2. It would be helpful to include an analysis of the sensitivity of IPRM’s performance to its hyperparameters, such as the number of parallel operations and the length of the reasoning steps.
limitations: limitations: The authors acknowledge the limitations of their work, particularly regarding the potential scalability challenges and the need for further evaluation on diverse datasets.
paper_summary: paper_summary: he paper addresses visual reasoning problems. It introduces a fully neural iterative and parallel reasoning mechanism (IPRM) that combines iterative computation with the ability to perform distinct operations simultaneously. Evaluation is performed on visual reasoning datasets such as CLVER, AGQA, CLEVR-Humans and STAR dataset.
strength_weakness: strength_weakness: * Generalization Results: The method shows zero-shot results only on CLEVRER-Humans. This is in contract to current SOTA video-language models such as LLaVA-Next or LLaMA-Vid which show zero-shot results across multiple datasets such as MSRVTT to TGIF-QA. Can the proposed model architecture be scaled to create more general purpose models similar to current video-language models?

* Multi-turn QA (related to the above point): The standard transformer architecture allows for multi-turn dialogue, where the user can ask the model multiple questions about a video (e.g. LLaVA-Next or GPT-4V). Can the proposed model architecture be extended to allow for multi-turn dialogue?

* Results on STAR: Recent works such as "Look, Remember and Reason: Grounded reasoning in videos with language models, ICLR 2024" outperform the proposed model.

* Results on GQA:  “Coarse-to-Fine Reasoning for Visual Question Answering, CVPR Workshops 2021” achieves 72.1% compared to 60.5% reported for IPRM. (Note that the use of scene graphs during training should not be a reason not to compare, instead the paper should discuss approaches to close the performance gap).

* The reasoning steps in Figure 7 are not very interpretable. In Figure 7 (left) the model does not seem to attend to the large cylinders in the first step. Similarly, in the Figure 7 (middle) the bottom reasoning chain seems to empty steps where the model does not look at any object. It would also be interesting to see the reasoning chains in case of real-world data such as STAR.
questions: questions: * Can the proposed architecture be scaled to provide zero-shot results on multiple datasets similar to current SOTA video-language models?
* Can the proposed architecture handle multi-turn conversations?
* The paper should compare/discuss SOTA approaches such as  "Look, Remember and Reason: Grounded reasoning in videos with language models, ICLR 2024" on STAR and “Coarse-to-Fine Reasoning for Visual Question Answering, CVPR Workshops 2021” on GQA.
limitations: limitations: Yes.

Comments: title: Reviewer oSE3: please respond to the authors' rebuttal
Thank you for your support!
Thank you!
Thank you for your support!
Official comment (2/2)
Official comment (1/2)
Great work!
Official comment by Reviewer nzfV
comment: Dear reviewer,

thanks for your participation in the NeurIPS peer review process.  We are waiting for your response to the rebuttal.  You gave a borderline rating (5). Is the response from authors satisfactory? Does it address weaknesses that you mentioned in the review?

- If yes, are you planning to increase your score?
- If no, could you help the authors understand how they can improve their paper in future versions?

Thanks,
AC
Thank you once again for your constructive feedback and useful suggestions, and for raising your score. We will include the CLOSURE results and failure mode analysis in the updated version of our paper.
Thank you for the detailed error analysis alongwith the examples for the CLOSURE dataset. It would be good to include the results on CLOSURE and some failure mode analysis in the final version of the main paper. The rebuttal has addressed my concerns. I have increased my rating to 7.
Dear reviewer 3tYc,

Thank you once again for the useful suggestions and for keeping your acceptance rating.
Dear reviewer P4wf, 

Thank you once again for your constructive suggestions and feedback, and for raising your score.
## Elaborated examples for ‘compare’: ##
_We use a list of objects instead of the image as we cannot update the rebuttal pdf / upload images._
### Example 1 ###

**Question**: there is another large metallic object that is the same shape as the big green shiny thing; does it have the same color as the tiny matte object?

**GT**: no **Pred**: yes

**Objects in scene**:
- small_rubber_green_cube -> (*‘tiny matte object’*)
- large_rubber_cyan_cylinder
- large_metal_brown_cube -> (*‘another large metallic object’*)
- large_rubber_brown_cylinder
- large_metal_blue_cylinder
- large_metal_green_cube -> (*‘big green shiny thing’*)
- large_rubber_blue_sphere'


**Remark**: The model appears to use large_metal_green_cube (i.e. the *‘big green shiny thing’*) and not 'large_metal_brown_cube' to perform the second clause (*‘same color as tiny matte object’*, i.e. small_rubber_green_cube) resulting in prediction of ‘Yes’ (same color) rather than ‘No’.

### Example 2 ###
**Question**: there is another yellow metallic object that is the same size as the yellow metal sphere; does it have the same shape as the red matte object?

**GT**: yes **Pred**: no

**Objects in scene:**
- small_metal_gray_cube
- large_rubber_red_cube  -> (*‘red matte object’*)
- large_metal_yellow_sphere
- large_metal_yellow_cube -> (*‘another yellow metallic object’*)


**Remark:** Model uses *“yellow metal sphere”* instead of “large_metal_yellow_cube” to perform *‘same shape’* comparison with 'large_rubber_red_cube'. This results in prediction of ‘No’ (not same shape).

### Example 3 ###
**Question:** there is another small gray thing that is the same material as the big yellow sphere; does it have the same shape as the large metallic object?

**GT:** no **Pred:** yes

**Objects in scene:**
- small_metal_cyan_sphere
- small_rubber_gray_sphere
- large_metal_yellow_sphere -> (*‘big yellow sphere’*)
- small_metal_gray_cube -> (*‘another small gray thing’*)


**Remark:** Model uses *“yellow metal sphere”* instead of 'small_metal_gray_cube' to perform *‘same shape’* comparison with 'large_metal_yellow_sphere' (i.e. the *“yellow metal sphere”* itself); results in prediction of ‘Yes’ (same shape) instead of ‘No’.



##  Elaborated examples for ‘or_mat’: ##
_We use a list of objects instead of the image as we cannot update the rebuttal pdf / upload images._

### Example 1 ###
**Question:** how many things are either large purple metallic blocks or large things that are the same shape as the large yellow metal object?

**GT**: 2   **Pred**: 3

**Objects in scene:**
- small_rubber_brown_sphere
- small_rubber_cyan_cylinder
- large_rubber_yellow_cylinder
- small_rubber_purple_cube
- large_metal_yellow_cube -> (*'large things that are same shape ..'*)
- large_rubber_yellow_sphere
- large_metal_blue_cube -> (*'large things that are same shape ..'*)
- large_metal_purple_cube -> (*‘large purple metallic block’*)

**Remark:** In counting objects with with *‘same shape’* as *‘large yellow metal object’*, the model includes 'large_metal_yellow_cube' (i.e. the 'large yellow metal' itself), resulting in a prediction of 3 total objects (instead of 2 as in ground truth). 


### Example 2 ###
**Question:** how many things are either small gray metallic balls or shiny things that are the same shape as the small purple metal object?

**GT**: 1 **Pred**: 2

**Objects in scene**:
- small_metal_brown_sphere
- small_metal_gray_sphere -> (*'small gray metallic balls'*)
- small_rubber_green_cylinder
- large_rubber_blue_sphere
- small_metal_purple_cube -> (*'small purple metal object'*)
- large_metal_gray_sphere

**Remark**: For the clause *"shiny things that are the same shape as the small purple metal object"*,  the model counts 'small_metal_purple_cube' (i.e. *"small purple metal object"* itself). As a result, its prediction is 2 (including small_metal_gray_sphere) instead of ground truth 1.
Dear reviewer nzfV,

As requested, we have conducted an error analysis on the *compare_mat* and *or_mat* settings of CLOSURE dataset by visualizing the intermediate attentions of the model and also analyzing the output predictions. Please note that for the ‘or_mat’ split IPRM achieves the highest performance amongst existing methods. For ‘or_mat_spa’ split IPRM achieves the second-highest performance behind the MAC method (although MAC has a much higher standard deviation of ± 15). 

On both the **compare_mat** and **compare_mat_spa** splits, our error analysis suggests that in many cases, IPRM uses the incorrect referent for ‘it’ to perform the ‘comparison’ operation in the second clause.
* E.g. given the 'compare_mat' question: *“There is another large metallic object that is the same shape as the big green shiny thing; does it have the same color as the tiny matte object?”*, based on its intermediate visual attention, the model appears to correctly perform and identify the first clause object (*‘large metallic object with same shape as big green shiny thing’*). However, in performing the second clause (*‘does it have same color as ..’*), the model appears to use *‘big green shiny thing’* as the referent for *‘it’* rather than the result of the first clause.
* For more clarity, we have elaborated some cases of this behavior in the comment (2/2) below. 

On the **or_mat** and **or_mat_spa** splits (which have numeric counting output labels), our error analysis suggests that IPRM includes the ‘comparison’ object in its count, while the ground truth count does not include it.
* E.g. given the ‘or_mat’ question: *'How many things are either large purple metallic blocks or large things that are the same shape as the large yellow metal object?'*, the model appears to identify i) large_purple_metal_cube, ii) large_metal_blue_cube  and iii) large_yellow_metal_cube (i.e. the comparison object *'large yellow metal object'* itself) . As a result, its prediction is 3 while ground truth is 2 (which does not include the ‘comparison’ object large_yellow_metal_cube).
* Please see the elaborated cases in the following comment (2/2) for more clarity.
The rebuttal has addressed most of my concerns. I will raise my score.
Thank you for your detailed rebuttal, as well as the places that I missed in the first read. I like the paper so I would like to keep my acceptance rating.
Thank you for the detailed rebuttal and the additional experiments. Can the authors do some failure mode analysis and provide some intuition why IPRM doesn't perform so well on the or_mat and compare_mat settings of the CLOSURE dataset?