Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper introduces an benchmark for visual reasoning, or more specifically, the capabilities about abstraction of attributes and relations. It incorporates a compositionality prior to facilitate scene generation. It mainly explores how current models do in terms of generalization and sample efficiency. It conducts empirical studies on some baseline models, which compare the performances under different settings and indicate that current deep learning visual models still lag behind human performance a lot.
strengths: 1. Compositional generalization and sample efficiency are both significant topics in visual reasoning area. The paper focuses on the two issues and explores how performances on this benchmark are affected by different factors, e.g. architecture, pre-training, rule embedding.
2. The benchmark covers a wide range of tasks, most of which are clear and highly related to abstract reasoning, as well as multiple settings convenient for testing various levels of transferring or reasoning.
3. Incorporation of human performance helps construct a quantitative bar for reference, which further highlights the weakness and headroom for current visual reasoning models.
weaknesses: 1. The motivation of rule embedding is not explained clearly, making this design not convincing enough.
2. Though huge efforts are made in task integration and experiments, this work lacks in-depth analysis about different architectures and intuitive explanation for discrepancies observed in results
3. The implementation is not informative enough, attaching relevant schematics or illustrations of models would be better.
correctness: Yes
clarity: Clear as a whole
relation to prior work: Yes
additional feedback: 1. I am curious about why CNN-based models show different tendancy compared with Transformer-based models, e.g. between joint and individual rule embedding?
2. Similarly, the reason why CNN-based models outperform Transformer-based models seems to be the scale of ViT. On the other hand, ViT-Small is not commonly used in current visual reasoning tasks. So the selection of this baseline may not be quite reasonable.
review_title: A novel benchmark for evaluation on critical capabilities of visual reasoning model, as well as empirical results for reference to better understand current visual reasoning models

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: The paper proposes to create an  odd-one-out dataset  for visual reasoning, where each example has 4 images and 3 of the images obey one set of rules while one image does not. The paper describes a well-thought-out process to generate such data.

Experiments show that this dataset indeed presents a challenge to current models under a low-data setting (1000 images per task). However, I am not exactly clear how this dataset tests compositionality and how it relates to other work that also tests compositionality (e.g., CLEVR). I look forward to the rebuttal. 
strengths: 1. Interesting method for generating the odd-one-out dataset

The paper proposes a well-defined generation process of the odd-one-out dataset. In one example, three images will share a set of rules while one image will not; meanwhile, random parameters are used to insure all images are “dissimilar” on the surface. This also allows targeted test at models’ ability to reason over certain properties (e.g., Figure 7).


2. Interesting observations

The paper presents a study on how models with different architectures (ResNet v.s. ViT) learn compositional visual reasoning. They have made several interesting observations:

a. Under low-data regime, ViT is not as data-efficient as CNN.

b. ViT does not learn shapes well presumably due to the tokenization process. Such an issue is not well noticed on real-world benchmarks.



weaknesses: 1. A rather vague definition of compositionality
The word compositionality is used rather lightly. I would have appreciated more discussion on how this dataset presents a compositional challenge and how it compares to the compositional challenge in prior work (e.g., Compositionality decomposed: how do neural networks generalise?). 

For example, in CLEVR, compositionality means generalization to novel combinations of object categories and attributes. In this very paper, it means solving problems requiring multiple rules. Does this present a unique compositionality challenge? In CLEVR, it seems the compositionality challenge mainly lies in learning compositional representation (i.e., correctly identify and represent novel objects). It is the same in this paper?


2. Discussion with respect to related work
One of the selling points of the dataset is its limited size and focus on data efficiency. However, one could also simply downsample the training set of prior datasets such as SVRT. I am not sure the core novelty of this dataset compared to prior datasets (I guess this is also related to my question 1).


3. Limited size
Why stop at generating 1000 examples per rule? Would it be possible to generate more examples and test how many examples are needed for the models to reach high accuracy? This could be informative for future development.

correctness: Yes.
clarity: Yes.
relation to prior work: See weakness.
additional feedback: 1. Why join-ViT underperforms ind-ViT (Table 1)


2. How do we model 4 images with the model? Do we concat the 4 images into a large image and feed it into the model?


3. The title is a bit too broad
review_title: Review for A Benchmark for Compositional Visual Reasoning 

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: In this work, the authors present a benchmark for comparing the performance of models on visual reasoning with a focus on assessing sampling efficiency and compositionality. The authors contribute with a dataset, a method for generating visual reasoning problems, and an extensive comparison of different state-of-the-art vision models including a new model
strengths: - Besides the data splits provided (including a generalization split), the method presented for generating the dataset problems could be easily extended to incorporate different tasks to compare other characteristics of interest or a diverse set of visual relations.
- The thorough comparison of SOTA models includes sample efficiency for different training sizes and also evaluates the ability to use compositionality via contrasting the performance of training the models using individual or joint tasks and the capability of transferring the knowledge acquired on training to unseen tasks.
- The evaluation of the task difficulty according to the distinct elementary relations of the objects.
- The incorporation of a hybrid model that combined a convolutional backbone with a visual reasoning model against either purely deep learning or purely visual reasoning models.

weaknesses: - On the use of the self-supervised pre-training on ImageNet, the authors claim that this is performed to dissociate the feature learning from the reasoning learning. Nevertheless, the average human performing these tasks has almost surely already performed any other visual reasoning task different from discriminating objects. It could be interesting to see what happens when using another visual reasoning pretraining like Raven or PGM. 
- The human data to compare the results shown in this work is not provided. It would be interesting to include that data. Some minor population data like the age of the subjects could be also included.

correctness: The construction of the dataset is correct and the consequent evaluation of the models is also fine. Limitations of the work are included and discussed properly. The claim of 50 times more samples is needed for neural architectures to compete than a human seems a bit of an overstatement. 
clarity: The paper is well-written, with clear details of the limitations and contributions of the present work. Still, the number of characteristics compared between models made the work a little difficult to follow.
relation to prior work: The authors describe extensively how their work compares against previous work. They compare where this dataset stands on the visual reasoning benchmarks in relation to the number of tasks and number of samples.

additional feedback: Some minor comments and typos:
- Is the AUC reported equally averaged across each training scenario or is weighted like the SES metric?
- On L165, “... (the models are trained) jointly on several tasks”, do you mean all tasks or a sample of them?
- Figures 1 and 2 are never referenced in the text
- Figure 7, the last model I believe is supposed to be SCL-ResNet 18 instead of SCL-ResNet 10
- Supplementary L33 “Genralization”
- Supplementary Table 1 SCL-ResNet 18 line is missing the units (M)
review_title: A beneficial dataset alongside a generation method for comparing models on Visual Reasoning

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The paper proposes a novel benchmark for assessment of visual reasoning of modern neural net architectures. It proposes tasks by creation of basic abstract rules and use compositionality on them to create additional rules. In addition, they propose measures of sample efficiency and test ability of model generalization across different rules. Modern neural nets are evaluated and also compared against human baseline for data efficiency and overall accuracy of tasks.

strengths: The strengths of the paper are :-
1. The proposed method creates numerous tasks of varying levels of difficulty for assessment of visual reasoning using elementary rules and their composition. 
2. The dataset can be synthesized through a program set by rules and multiple levels of abstraction can be constructed.
3. They also conduct analysis of the sample efficiency and evaluate on tasks of composition which is significant.
4. The benchmark makes an important case to highlight the inability of modern neural nets in visual reasoning tasks and has potential to improve their interpretability.

weaknesses: The weakness are as follows :-
1. The proposed task is of one kind - odd-one-out rule. What about task such as visual completion of shapes based on pattern recognition? More varied tasks could be devised.
2. The reasoning evaluation is restricted to accuracy of the task only and there is no analysis on the representations learnt by the neural nets or visual explainablity of the task. 



correctness: Yes. 
clarity: The paper is well written.
relation to prior work: The comparison to prior work is shown.
additional feedback: The claim on learning of composition for transformers vs convolution architectures need further assessment, as both differ in optimization schemes under same data regime.
review_title: The paper proposes a new benchmark for visual reasoning tasks leveraging compositionality and motivates research into the investigation of the modern neural nets.  

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The authors provide a benchmark aimed at testing the ability for compositional generalization in visual reasoning based settings. To this end, they provide a wide variety of tasks with different underlying relations and perform individual as well as joint training over the different tasks. The work also covers a self-supervised pretraining baseline, which is more comparable to human performance as humans generally have a well developed visual cortex before they are tasked with such reasoning problems. Finally, analysis is done over task difficulties, sample efficiencies of models, ability for compositional generalization as well as trends in the presence and absence of curriculum over the tasks.
strengths: * The authors provide a benchmark that creates tasks with a wide variety of underlying abstract rule conditions constructed from some primitive rule conditions; thereby leading to a compositional prior in the data generating process.
* The authors show that models which are trained jointly on all tasks perform much better than those trained on tasks individually. It would be interesting to see an experiment that validates or invalidates this trend with increasing model sizes, whether this trend exists irrespective of the size of the model or only for models that are larger than a certain threshold.
* The work provides clear evidence of benefits of SSL-based pretraining for visual reasoning tasks.
weaknesses: * Maybe I misunderstood it, but the benchmark does not have any notion of out-of-distribution generalization based testing? Given the structure of the dataset, it would be nice to have this functionality present as well.
* Given that the models are trained in extremely low data regime in Table 1, it would be nice to have some estimate of standard error / deviation to understand if the difference between the algorithms is statistically significant since low data regime might lead to high deviation in results.
* There are a lot of details missing on how the dataset / tasks are generated. For example, an image of a few different tasks with compositional abstract underlying rule should be there in the main paper to make it easier to understand the kind of tasks that are being considered. There should also be additional details on how a scene is generated from the scene-graph, eg., how are two objects generated such that their contours are touching or how is it ensured that objects do not overlap, etc. Such details are missing from the draft.
correctness: * The claim that a lot of AI systems are less data efficient when compared to humans might be true but is not substantiated by the self-supervision based experiments. This is because while self-supervision does learn some knowledge about the visual semantics of the task, humans are introduced to much more complex visual stimuli as well as by actually acting in the world, they also a-priori develop knowledge about various kinds of reasoning. It could be that this a-priori abundance of information makes them more data efficient for specific tasks, and not the actual architecture itself. A closer step to this analysis might be using a large pre-trained network and finetuning it on the tasks presented; but even that might not be enough to substantiate this claim unless quantitative numbers demonstrate that the pre-trained system and humans both were "pre-trained" on the same amount and kind of information.
* The comparison between Figure 6a and 6b doesn't seem fair since in 6a the models are finetuned on difficult compositions while in 6b they are not finetuned on elementary relations and are just expected to perform well zero-shot. Does the same trend hold when you finetune in 6b or do 6a zero-shot?
clarity: * I could not understand Figure 3 completely. What is the reference and odd rule? How is the fourth frame violating the reference rule? Is there any need for the odd rule when any rule not equal to reference rule can be considered as an odd rule?
* Is it possible to have model performance without any curriculum on Figure 6b too to serve as a baseline to compare against?
* In Figure 7, what do the columns represent? What are the "group of composition rules?"? Can you provide an example on this to understand?
relation to prior work: * The idea of better sample complexity and inductive biases for better out-of-distribution generalization has been presented in a number of works and it would be nice to have a discussion along these lines to better understand how one can come up with systems that can do well in the provided tasks. Some examples of works that study these are:
  * Kerg et. al 2022 On Neural Architecture Inductive Biases for Relational Tasks
  * Mittal et. al 2021, 2022 Compositional attention: Disentangling search and retrieval and Is a Modular Architecture Enough?
  * Rahaman et. al 2021 Dynamic inference with neural interpreters
  * Goyal et. al 2019 Recurrent independent mechanisms
  * Bergen et. al 2021 Systematic Generalization with Edge Transformers
additional feedback: More details on how the dataset is actually created would be quite helpful!
review_title: Review on A Benchmark for Compositional Visual Reasoning

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: Proposed CVR dataset for Odd-One-Out visual reasoning that tests sample efficiency, generalization, transfer across task rules, and ability to leverage compositionality. CVR contains 103 unique task rules over distinct scene structures generated procedurally. The paper analyzes and evaluates convolutional and transformer-based models on the proposed task and stated axes. 

*EDIT: I thank the authors for clarifying some of the dataset details. I have raised my score slightly, though I still believe that the paper lacks clarity and is not ready for publication.* 
strengths: The authors propose an Odd-One-Out reasoning benchmark with more tasks (attributes) and samples that better test model performance on four axes -- sample efficiency, generalization, transfer across task rules, and ability to leverage compositionality. The four identified axes are important to the research community, and findings on the respective models tested are interesting. The dataset is accessible and released with no ethical and social implications. 
weaknesses: Several components of the paper are not defined rigorously or shown as claimed. For example, it is unclear what the “compositionality prior” relied upon in the paper is. All objects have attributes, and most procedurally generated datasets rely on object attributes for generation, such as CLEVR [1]. How is this unique to CVR?

It’s also unclear whether these additional attributes that lead to more tasks truly better test compositional generalization compared to prior visual reasoning tasks, which already test generalization in geometric shapes, color, etc. The setup for testing compositionality (starting L232) is confusing and seems to be overly complicated. A common method of testing compositionality is CLEVR CoGenT [1]. 

For evaluating models on benchmarks, is there a reason why the models are not pretrained on ImageNet? This would better disentangle the perception and reasoning challenge, and less hamper ViT’s performance. Could this evaluation be added?

It is not clear why the authors' proposed Sample Efficiency Score is better than that of common characterizations of model sample efficiency. Most ML works simply compare performance/accuracy vs. samples and presents numbers at cutoffs.

While the authors attempt to fairly  evaluate Resnet-50 and ViT-small with a similar number of parameters, there is evidence that ViTs suffer from poor initialization [2]. Furthermore, transformer architectures can recognize shapes effectively [3], countering some of the claims in the paper (L280, “We believe that ViT is not capable of learning shape based tasks because of its input processing which splits an image into patches…”). Several of the assertions made about model performance are too strong.

[1] Johnson, J., Hariharan, B., Van Der Maaten, L., Fei-Fei, L., Lawrence Zitnick, C., & Girshick, R. (2017). Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2901-2910).
[2] Chen, X., Hsieh, C. J., & Gong, B. (2021). When vision transformers outperform ResNets without pre-training or strong data augmentations. arXiv preprint arXiv:2106.01548.
[3] Naseer, M. M., Ranasinghe, K., Khan, S. H., Hayat, M., Shahbaz Khan, F., & Yang, M. H. (2021). Intriguing properties of vision transformers. Advances in Neural Information Processing Systems, 34, 23296-23308.

correctness: Yes.
clarity: Yes. However, parts of the paper are unclear, introduced insufficiently (ex. L74 -- compositionality prior is never defined or explained, L142 -- referring to specific tasks in a different dataset and paper), and confusing (ex. caption for Figure 2 in incorrect order, should move some explanations of the benchmarking process from the appendix into the main test). 

relation to prior work: The paper discusses prior visual reasoning benchmarks, but do not discuss prior works or datasets that measure model performance on the four axes that the benchmark tests (sample efficiency, generalization, transfer across task rules, and ability to leverage compositionality). Importantly, it seems that one of the main differences between CVR and visual reasoning benchmarks is the unique combination of the samples it contains and the tasks it captures. However, it is not convincing that this is a benchmark that needs to exist (in axes such as testing compositional generalization), instead of creating more tasks in the style of RAVEN & SVRT or more training samples in ARC. Paper also seems to miss some literature in odd-one-out tasks, for example G1-set & S1-set [1]. 

[1] Mańdziuk, J., & Żychowski, A. (2019, July). Deepiq: A human-inspired ai system for solving iq test problems. In 2019 International Joint Conference on Neural Networks (IJCNN)(pp. 1-8). IEEE.

additional feedback: The paper claims that the dataset is well-positioned to test sample efficiency, generalization, transfer across task rules, and ability to leverage compositionality. Though sample efficiency is convincing, the other three components --- generalization, transfer across task rules, and ability to leverage compositionality --- are not well defined and designed. More rigorous investigation of compositional generalization (compared to prior work such as C-VQA [1] , Winoground [2]) and transfer (ex. with pretrained models, meta-learning) would strengthen the paper. 

[1] Agrawal, A., Kembhavi, A., Batra, D., & Parikh, D. (2017). C-vqa: A compositional split of the visual question answering (vqa) v1. 0 dataset. arXiv preprint arXiv:1704.08243.
[2] Thrush, T., Jiang, R., Bartolo, M., Singh, A., Williams, A., Kiela, D., & Ross, C. (2022). Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 5238-5248).
review_title: Potentially interesting benchmark lacking rigorous design & experiments

Meta-review: metareview: Most reviewers appreciate the importance of the topic (compositional visual reasoning), the flexibility of the benchmark (easily extended to incorporate different tasks), the thorough comparison, insightful experiments. However, the reviewers also raise questions about the clarity of the presentation, the limited training set size. AC read reviews and response from the authors and found that the concerns are addressed fairly well. AC recommends to accept the submission as a poster. 
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept