Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors reveal that Neural Module Networks (NMNs), i.e., question-specific compositions of modules that tackle a sub-task, achieve better or similar systematic generalization performance than the conventional Transformers, even though NMNs’ modules are CNN-based. 
To address this shortcoming of Transformers with respect to NMNs, in this paper, the authors investigate whether and how modularity can bring benefits to Transformers.
Namely, they introduce a Transformer Module Network (TMN), a novel NMN based on compositions of Transformer modules.

strength_and_weaknesses: Strength
- The observation about the comparison between the neural module networks and the Transformer models is novel and interesting.
- The authors provide a comprehensive ablation study to make the final model more solid.


Weakness
- It would be better to add qualitative results to see the behavior of the proposed method and the existing methods. Also, it would be better to discuss why the conventional Transformer models perform worse than the neural module networks by analyzing the samples.
clarity,_quality,_novelty_and_reproducibility: The proposed method is clear and novel. The quality of this paper looks favorable overall.
summary_of_the_review: The authors provide interesting observations, and the experiments are solid enough to show the effectiveness of the proposed method.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper mainly focuses on combining the strengths of Transformer and NMN by introducing a novel NMN based on compositions of Transformer modules named Transformer Module Network (TMN). The model is evaluated on CLEVR-CoGenT, CLOSURE, and GQA-SGL.
strength_and_weaknesses: *[Strength]*

The motivation that combining the strength of NMN and Transformer is contributing.

*[Weakness]*
1. In the approach, the paper directly substitutes the module in the NMN with the TRM encoder. However, it lacks ablation experiments and discussion about if the TRM structure outperforms all types of module in NMN-based method. (e.g., in n2mn[1] Table 1, the attention based modules like find, relocate, filter modules and the answer based modules like describe, compare and exist modules)
2. The Tree modular modular is not novel in NMN domain, [1] uses similar manner in Figure 3.
3. Lacks of SOA TRM-based and NMN-based model performance comparisons. In addition, the SOA method on GQA dataset is 72.1 from [2], and there is [3] with 69.46 on GQA-dev




[1] Hu, Ronghang, et al. "Learning to reason: End-to-end module networks for visual question answering." Pr oceedings of the IEEE international conference on computer vision. 2017.

[2] Nguyen, Binh X., et al. "Coarse-to-Fine Reasoning for Visual Question Answering." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.

[3] Kim, Eun-Sol, et al. "Hypergraph attention networks for multimodal learning." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.
clarity,_quality,_novelty_and_reproducibility: The paper clarified its contributions and approach clearly. 
In general, the paper can be reproduced and is of fair quality.
However, it lacks novelty since the tree structure and substituting the modular with the TRM encoder are not novel.
summary_of_the_review: The motivation that combining the strengths of the two structures is good. However, directly substituting the modular with TRM seems not novel to me. Besides, there are no experiments to prove that the TRM encoder outperforms all previous NMN modular in the task, thus, it does not convince me of its superiority by directly substituting all NMN modulars. In addition, it lacks comparisons of the SOA methods. And from my knowledge, the performance is not SOA (the last point in the weakness section). Overall, I'm leaning to reject it.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper focuses on the systematic generalization visual question answering where the test set contains novel combinations of training concepts. The authors propose a new modular network whose modules are transformers.
strength_and_weaknesses: Strength:

The paper focuses on the model robustness against VQA bias, which is a common problem in the VQA field.

The paper proposed a new systematic generalization dataset GQA-SGL based on the GQA. The new dataset can be a new challenge for future work in this field.

Weakness:

The novelty is limited. The proposed model is a modular network composed of more expressive transformer modules. 

The proposed model also suffers from the weakness of the previous modular network. It needs program supervision and relies on the performance of the question parser. Since all the experiments have a groundtruth program at test time, whether the question parser has the same systematic generalization ability is not verified.

The improvement on the CLEVR-CoGenT and CLOSURE dataset is not consistent. The proposed model performs worse than the baseline on the CLEVR-CoGenT but improves on the CLOSURE. I suppose the CLEVR-CoGenT and CLOSURE have novel images and questions respectively. However, the TMN can ignore the questions by accessing the groundtruth programs.
The authors suggest that the proposed model and baseline share the same visual feature extractor. Thus authors may verify this by training a new extractor from scratch similar to FiLM[1].

[1] Perez E., Strub F., de Vries H., Dumoulin V., Courville, A. FiLM: Visual Reasoning with a General Conditioning Layer. AAAI 2017
clarity,_quality,_novelty_and_reproducibility: The paper is well-organized and easy to understand.

The novelty is not enough. The proposed model only replaces the previous neural module with a transformer.
summary_of_the_review: The paper proposes a new dataset. But the proposed model is not novel and its effectiveness is not fully verified. Thus I give borderline.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper addresses the problem of systematic generalization in VQA. It proposes a new model that takes advantage of the learning capability of Transformers and the compositional modeling of Module Networks. The proposed model achieves state-of-the-art performance on three different VQA datasets including standard testbeds for VQA systematic generalization. 
strength_and_weaknesses: •	Strengths:

1.	The proposed method is technically sound and easy to understand. It is reasonable to have a model that makes use of the known advantages of Transformers and module networks.

2.	It shows better performance when compared against existing methods.

•	Weaknesses:

1.	While the proposed model is straightforward and easy to understand, it appears that the model brings very little insight into the systematic generalization task itself. It is well studied that module networks are a potential approach for systematic generalization but it is extremely brittle and ad-hoc which hinders its application in wider settings. Meanwhile, models with better representation learning such as Transformers are beneficial for all machine learning tasks, not just for systematic generalization. The proposed method does not offer any fundamental contributions toward solving systematic generalization.

2.	Experiments with original transformers do not sound reasonable as the original design of transformer is for translation tasks with having access to a more generous amount of training data. There is no reason to have to use 12 layers for transformers when applying it to a custom problem. The poor performance of transformers in the experiments might simply be due to overfitting.

clarity,_quality,_novelty_and_reproducibility: This paper is technically sound, with sufficient clarity. The details provided are fully explained and can be reproducible. However, it appears to be limited in scientific contribution.
summary_of_the_review: The paper is well written and easy to follow. However, it is limited in contribution to advance the systematic generalization task.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper aims to address the systematic generalization problem in VQA by combining the strengths of neural module networks and Transformers, and introducing a new Transformer Module Network (TMN). The TMN method is evaluated on three different VQA datasets. 

Strengths: 
- The systematic generalization problem is interesting and worth it to solve. 
- The proposed method is technically sound and easy to understand. 

Weaknesses: 
- Reviewers have common concerns regarding the novelty of this work compared with the other NMN methods. 
- It is unclear how the proposed method specifically addresses the systematic generalization problem. 
- Lack of result comparison and ablation study.
justification_for_why_not_higher_score: Reviewers have critical concerns regarding the method and the experiments of this work, as mentioned above, and the author's responses do not resolve them.
Decision: Reject