Title: Learning to Reason Iteratively and Parallelly for Complex Visual Reasoning Scenarios
Venue: NeurIPS 2024
Accepted: yes
Keywords: iterative and parallel computation; complex visual reasoning and question answering; neural network based reasoning architectures
Abstract: Complex visual reasoning and question answering (VQA) is a challenging task that requires compositional multi-step processing and higher-level reasoning capabilities beyond the immediate recognition and localization of objects and events. Here, we introduce a fully neural Iterative and Parallel Reasoning Mechanism (IPRM) that combines two distinct forms of computation -- iterative and parallel -- to better address complex VQA scenarios.  Specifically, IPRM's "iterative" computation facilitates compositional step-by-step reasoning for scenarios wherein individual operations need to be computed, stored, and recalled dynamically (e.g. when computing the query “determine the color of pen to the left of the child in red t-shirt sitting at the white table”). Meanwhile, its  "parallel'' computation allows for the simultaneous exploration of different reasoning paths and benefits more robust and efficient execution of  operations that are mutually independent (e.g. when counting individual colors for the query: "determine the maximum occurring color amongst all t-shirts'"). We design IPRM as a lightweight and fully-differentiable neural module that can be conveniently applied to both transformer and non-transformer vision-language backbones. It notably outperforms prior task-specific methods and transformer-based attention modules across various image and video VQA benchmarks testing distinct complex reasoning capabilities such as compositional spatiotemporal reasoning (AGQA), situational reasoning (STAR), multi-hop reasoning generalization (CLEVR-Humans) and causal event linking (CLEVRER-Humans). Further, IPRM's internal computations can be visualized across reasoning steps, aiding interpretability and diagnosis of its errors.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nhe paper addresses visual reasoning problems. It introduces a fully neural iterative and parallel reasoning mechanism (IPRM) that combines iterative computation with the ability to perform distinct operations simultaneously. Evaluation is performed on visual reasoning datasets such as CLVER, AGQA, CLEVR-Humans and STAR dataset.\n\nSTRENGTHS:\n* The proposed method is novel and interesting. \n* The paper is well written and easy to understand. \n* The paper includes in-depth analysis of the proposed method.\n* The method shows promising zero-shot performance on CLEVRER-Humans,\n\nWEAKNESSES:\n* Generalization Results: The method shows zero-shot results only on CLEVRER-Humans. This is in contract to current SOTA video-language models such as LLaVA-Next or LLaMA-Vid which show zero-shot results across multiple datasets such as MSRVTT to TGIF-QA. Can the proposed model architecture be scaled to create more general purpose models similar to current video-language models?\n\n* Multi-turn QA (related to the above point): The standard transformer architecture allows for multi-turn dialogue, where the user can ask the model multiple questions about a video (e.g. LLaVA-Next or GPT-4V). Can the proposed model architecture be extended to allow for multi-turn dialogue?\n\n* Results on STAR: Recent works such as \"Look, Remember and Reason: Grounded reasoning in videos with language models, ICLR 2024\" outperform the proposed model.\n\n* Results on GQA:  “Coarse-to-Fine Reasoning for Visual Question Answering, CVPR Workshops 2021” achieves 72.1% compared to 60.5% reported for IPRM. (Note that the use of scene graphs during training should not be a reason not to compare, instead the paper should discuss approaches to close the performance gap).\n\n* The reasoning steps in Figure 7 are not very interpretable. In Figure 7 (left) the model does not seem to attend to the large cylinders in the first step. Similarly, in the Figure 7 (middle) the bottom reasoning chain seems to empty steps where the model does not look at any object. It would also be interesting to see the reasoning chains in case of real-world data such as STAR.\n\nQUESTIONS:\n* Can the proposed architecture be scaled to provide zero-shot results on multiple datasets similar to current SOTA video-language models?\n* Can the proposed architecture handle multi-turn conversations?\n* The paper should compare/discuss SOTA approaches such as  \"Look, Remember and Reason: Grounded reasoning in videos with language models, ICLR 2024\" on STAR and “Coarse-to-Fine Reasoning for Visual Question Answering, CVPR Workshops 2021” on GQA.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the Iterative and Parallel Reasoning Mechanism (IPRM), a novel neural architecture designed to enhance visual question answering (VQA) by combining iterative and parallel computation. The IPRM aims to address the limitations of existing methods that rely solely on either iterative or parallel processing. The iterative component allows for step-by-step reasoning essential for complex queries, while the parallel component facilitates the simultaneous processing of independent operations, improving efficiency and robustness.\n\nThe authors propose a lightweight, fully differentiable module that can be integrated into both transformer and non-transformer vision-language backbones. The IPRM outperforms current state-of-the-art methods across several benchmarks, including AGQA, STAR, CLEVR-Humans, and CLEVRER-Humans, demonstrating its capability to handle various complex reasoning tasks. Additionally, the mechanism’s internal computations can be visualized, enhancing interpretability and error diagnosis.\n\nSTRENGTHS:\n1. The combination of iterative and parallel reasoning within a single framework is novel, addressing specific limitations of existing VQA methods.\n2. The IPRM demonstrates superior performance across multiple challenging VQA benchmarks, demonstrated by extensive experimental results.\n3. The paper is generally well-structured.\n\nWEAKNESSES:\n1. While the paper is well-structured, certain sections, particularly those describing the technical details of the IPRM, can be dense and challenging to follow.\n2. The paper could benefit from a more detailed discussion on the scalability of the IPRM, particularly in terms of computational resources and training time.\n\nQUESTIONS:\n1. Can the authors provide more insights into the scalability of IPRM? Specifically, how does the computational complexity of IPRM compare to other state-of-the-art methods?\n2. It would be helpful to include an analysis of the sensitivity of IPRM’s performance to its hyperparameters, such as the number of parallel operations and the length of the reasoning steps.\n\nLIMITATIONS:\nThe authors acknowledge the limitations of their work, particularly regarding the potential scalability challenges and the need for further evaluation on diverse datasets.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes an iterative and parallel reasoning mechanism (IPRM) for VQA tasks, combining step by step iterative computation with parallel processing of independent operations, and also maintaining an internal memory of operation states and results states. Firstly at each reasoning step, previous operation states attend to language features to form latent operations, then latent results are formed by attending to visual features based on the newly formed latent operations and prior result states. Finally new memory state is computed by combining the latent operation and result states alongwith prior memory states with a lookback window. IPRM outperforms previous methods on several complex image and video question answering tasks. IPRM is also lightweight and sample efficient. Ablation experiments demonstrate the importance of different components.\n\nSTRENGTHS:\n1. The paper is well written and easy to follow.\n2. A novel way of combining iterative and parallel processing for VQA tasks.\n3. The approach demonstrates decent improvements on three video (STAR, AGQAv2, CLEVERER-Humans) and three image question answering tasks (CLVER-Humans, CLEVR-CoGenT, and GQA), alongwith the capability to generalize zeroshot and in out of domain settings (CLEVR-CoGenT).\n4. The approach also requires fewer training samples and parameters.\n\nWEAKNESSES:\nNo major weaknesses, please see questions below.\n\nQUESTIONS:\n1. What is the reason behind applying a MLP with a nonlinear activation instead of just linearly transforming prior operations to get query in Eq 4, as is done in standard attention mechanisms?\n\n2. Can the authors perform some ablations on operation execution component, regarding the importance of using previous result states and newly formed operations in forming attention key?\n\n3. Based on the ablation plots, figure 6 (iii) it seems r=1 performs better than r=2, then why was r=2 chosen for the experiments? \n\n4. For the window length ablation it would be informative to perform the ablation for W>3, to make sure the performance doesn’t increase further with window length. Also it seems W=3 performs slightly better than W=2?\n\n5. How does IPRM do on the CLOSURE [1] dataset (both zeroshot and finetuned)?\n\nSuggestions:\n\n1. It would be informative to also include the MAC model in the comparison of the parameters and sample efficiency in Figure 5.\n\n2. Some of the recent prior work on visual reasoning [2,3,4] haven’t been cited.\n\n3. The authors should include a discussion of limitations/future directions in the main paper.\n\n[1] - Bahdanau, D., de Vries, H., O'Donnell, T.J., Murty, S., Beaudoin, P., Bengio, Y. and Courville, A., 2019. Closure: Assessing systematic generalization of clevr models. arXiv preprint arXiv:1912.05783.\n\n[2] - Mondal, S.S., Webb, T. and Cohen, J.D., 2023. Learning to reason over visual objects. arXiv preprint arXiv:2303.02260.\n\n[3] - Webb, T., Mondal, S.S. and Cohen, J.D., 2024. Systematic visual reasoning through object-centric relational abstraction. Advances in Neural Information Processing Systems, 36.\n\n[4] - Mondal, S.S., Cohen, J.D. and Webb, T.W., 2024. Slot abstractors: Toward scalable abstract visual reasoning. arXiv preprint arXiv:2403.03458.\n\nLIMITATIONS:\nThe authors have discussed the limitations in the appendix.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the Iterative and Parallel Reasoning Mechanism (IPRM), a neural architecture for complex visual reasoning and question answering tasks. IPRM involves a three-step iterative process: Operation Formation, Operation Execution, and Operation Composition. In Operation Formation, IPRM generates new parallel operations by extracting relevant language information based on previous operation states. During Operation Execution, these new operations are processed simultaneously, retrieving pertinent visual information guided by both newly formed operations and existing result states. Finally, in Operation Composition, IPRM integrates the new operations and their results into memory by dynamically combining them with each other and with previous operation states.  Experiments are performed on video reasoning like STAR, AGQAv2, visual reasoning datasets like GQA, CLEVR.\n\nSTRENGTHS:\n-\tThe motivation behind the architecture is interesting and sound.\n\n-\tThe experiments show improved results comparing with other methods on several benchmarks.\n\n-\tThe visualization is interesting and somehow can explain intuition of the model.\n\nWEAKNESSES:\n•\tSome notations need further clarifications: The paper needs to improve consistency and clarity in its notation. For example, it is unclear if K in Figure 2 and its caption is equivalent to N_op used elsewhere. Some variables in equations are not properly defined or explained, such as U and H in equation (9), and u2 and v2 in equations (15) and (16). \n\n•\tModel scaling:  How does the performance change if the number of parameters is increased/decreased? While the authors claim IPRM can be applied to different vision and language backbones, they don't provide evidence of its performance when scaling up. This omission leaves questions about the model's scalability and efficiency unanswered.\n\n•\tWhat is the intuition behind using fixed numbers of parallel operators and computation steps, given that different questions likely require different numbers of operations and reasoning steps? Is there any potential drawback of using fixed parameters in this context?\n\n•\tCan the author provide some analysis about failure cases of model?\n\nQUESTIONS:\nPlease see weaknesses.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}