Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a large annotated dataset for object concept learning. The proposed dataset contains annotation at two levels of granularity (category and instance level). The dataset also provides causal relations between object attributes and their affordances. The paper also provides a thorough insight into their annotation process, and introduces a baseline method called the Object Concept Reasoning Network (OCRN) based on causal intervention and instantiation. The proposed dataset is large-scale for similar ones in this space, and can be of great help to researchers working in the area of concept learning and compositionality.
main_review: +ves:
+ Present-day ML models lay a great emphasis on categorizing objects based solely on appearance. In case of distribution shift in a real-world dataset, such models find it difficult to generalize. A deeper understanding of objects based on their physical properties and affordances can avoid such problems, and the dataset introduced in this paper is a notable step in the right direction.
+ The constructed dataset is large-scale which will encourage the generalization of models towards real-world data and large-scale applications.

Concerns:
- The paper has adopted a causal intervention method to solve for category biasing. However, there are simpler methods to tackle such problems (undersampling/oversampling majority/minority classes, generating synthetic samples etc.). How is the proposed method to these approaches? Why is a causal approach necessary? For a dataset that may be of wide use, it may also be good to not make its description more complex than it should be.
- Surprisingly, for an image dataset paper, there were very few examples of the images in the paper. It may have been nice to see a few sample images in the supplementary material along with a clear description of their annotations.
- The paper presentation is very dense. While I appreciate the detailed description of every aspect, it at times seems one level of detail beyond what is required. Besides, for what seems like a dataset created with careful thought, it would have been good to see more examples from the dataset, or a running example to explain the decisions made for the dataset design.
- Similarly, in Sections 4 and 5, the many notations and inline equations make it difficult to follow. It may be a good idea to write problem formulation (or include them in the task overview), which lays down the assumptions, notations and objectives clearly and concisely, and then discuss the rest of the work.
- The notion of causal relations in real-world scenes can be tricky and complex (for e.g, the notion of context -- water can be causal for a swimming fish making context causal). In the few examples presented in the paper, it was not very clear whether the relationships between the attributes and the affordances in the image are really causal. The paper mentions that two annotators were used to label such causal relationships -- I am not sure if this may be sufficient.

Minor comments:
* Section 4.1 the sentence, “The ɑ, 𝛃 probabilities ...” seems incomplete.
* Section 4.2 “Attribute Estimation Debiasing” there is inconsistency in the notation of f_Oi.
summary_of_the_review: In general, the paper presents a dataset, which is a positive step towards concept learning tasks and will hopefully encourage future works to move beyond object categorization based only on appearance. The paper presentation seems too dense, and it’d be good to see some concrete directions of presenting the paper better to make it more accepted by the community.

I will wait to hear back from the authors and other reviewers, and accordingly adjust my score.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In short, the paper proposes a new task, that of object concept learning. Object concept learning is a basically a combination of object classification, attribute classification of said object, and affordance classification of said object, both at a category (affordance of any cup) and an instance (affordance of this cup) level. To this end, the paper proposes a dataset, which starts from existing ones and extends them accordingly. Further, it proposes a baseline method that is somewhat inspired by do-calculus (Pearl). Experiments show positive trends.
main_review: The strengths of the paper are as follows:

- (Very) Important problem.
- A new dataset is always a good contribution. Specifically, the work appears to have done the extra mile in combining lots of different and different data sources.

The overall weaknesses of the paper are as follows:
- The technical contribution is low. Basically, this is a dataset/task proposition paper. This is also the authors' summary.
- In that sense, it feels that the paper is not a direct fit to ICLR, which focuses more on technical contribution. 
- While it is claimed that the work is about causality, the causality in this context is rather learned with standard correlation-based approaches, which minimize standard learning objectives.
- In eq 3, the notation is not entirely clear to me, or I am misunderstanding something. If A stands for category-level attributes, their number is different than the number of object categories. How can then be that $P(A_j|O_i)=1$ $iff i=j$, since these two indices run over different sets?
- It is not entirely clear what is the baseline supposed to do? Basically, learn 'causal mechanisms' such that to attain 'perfect' recognition of the category-level, instance level attributes and affordances, and category label? And to do so, the do-operation intervention runs over all attributes/affordances in (3) and (6)? What exactly is the motivation of this intervention, as in selecting one attribute/affordance over another?
- The writing is not always very clear. For instance, I am not sure what is the x axis of figure 8 and 9, I could not find it in the caption? Is it the number of training data used to train the model, and if yes, why is nearly zero for a long period in the figure?
summary_of_the_review: In summary, I think the paper brings little technical innovation, and as such it should not be accepted.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new large-scale benchmark dataset for object concept learning, which consists of recognizing attributes, affordances, and their causal raltions about objects in input images. Detailed annotations of object categories, attributes and affordances on both category and instance levels, and their causal relations (on the instance level) are provided. This new dataset will be helpful for the community to advance the research of causal learning from visual data. 

A strong baseline method is also proposed that explicitly considers the causal structure and concept instantiation of object categories, attributes, and affordances. Although it shows better results than other baselines, there is still great room for future improvement.
main_review: Strengths:
1. Having such a large-scale dataset with detailed annotations and causal relationships of visual concepts (attributes and affordances in this paper) is valuable for the entire community. It will advance future research on causal learning from visual data.

2. Data and code will be publicly available (they are available for review). 

Weaknesses:

First of all, I have to admit I have limited knowledge about causal learning. I found some technical details are missing, particularly in Section 4 and 5, which make the technical part not easy to follow. Moreover, it would be great if more information could be provided in the paper (or in the supplementary material). Detailed comments can be found below.

1. For the data annotation, more annotators were hired if the first set of annotators could not reach an agreement. What is the percentage of such confusing cases?

2. What is the total cost of building such a dataset? This is useful as a reference if people would like to build a similar dataset in the future.

3. Although experts were involved in designing the data annotation process, for example, defining the object states, it is not clear whether there is any way to formally monitor the quality of the annotations.

4. Object states are used in the data annotation process for the instance-level affordance and causal relation parts. I would love to see the annotations of object states for such a benchmark dataset to be released as well. But they are intentionally excluded in the benchmark dataset and being used in the baseline model. Here are two questions:
    a) Since object states were used in the annotation process, is it possible that without using the states, some annotations of afforances and causal relations may be ambiguous or do not hold at all? For example, I may argue that both the apples in the middle column of Fig 4 can not be *directly* eaten as they need to be washed first. 
    b) According to the experimental results, the performance of the proposed baseline is still far from satisfactory. What if we explicity consider object states as part of the causal structure? Will it bring any improvement?


5. What are the context in Fig. 4? Are they attributes or object states?

6. For the task definition in Eq.(1), it is not clear about what the concrete tasks are. It would be helpful to explicitly explain/define them.

7. In Eq.(7) and Fig. 6, $f_{\alpha}'$ is introduced but without formal explanations. Particularly in Fig. 6, what are $f_{\alpha_1}$, $f_{\alpha_2}$, and $f_{\alpha_p}$? How are they different from $f_{\alpha}$? Where is the binary classifier $g_{\alpha_p}$? Is its loss part of $L_{\alpha}$?

8. Clarifications are needed for the definition of TDE. From my understanding, when $GT_{\beta_q}=1$, it means the affordance $\beta_q$ exists in the image. As a result, a high score is desired from the model's predicted score. Ideally, by explicitly considering the causal relationship of $\alpha_p$ and $\beta_q$, $P_{TDE}^{\alpha_p}(\beta_q)$ should be higher than $P(\beta_q)$. So TDE should be defined as $max(P_{TDE}^{\alpha_p}(\beta_q) - P(\beta_q), 0)$. Similarly, for the case where $GT_{\beta_q}=0$, it should be $max(P(\beta_q) - P_{TDE}^{\alpha_p}(\beta_q), 0)$.
summary_of_the_review: The issues mentioned in the weaknesses part above are minor. They do not affect the value of the proposed benchmark dataset and baseline algorithm. But it would be great if they could be addressed.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The primary contribution of this paper is two folds. First, the paper presents a new crowd-sourcing dataset that contains annotations for object categories, attributes, and affordances. Second, the paper presents a CNN training and inference pipeline that leverages "causal interference" to address potential biases.
main_review: Overall I found that the writing of this paper can still be improved. For example, Table 1 is too small; there are too many bold texts in Section 3; Section 4.2 contains many paragraph subtitles, it will be better if they can be grouped according to whether they are applied to attribute prediction or affordance prediction, etc.

My primary concern about this paper, however, lies in its clarity in formulating the causal story. First, I would suggest the authors state clearly where do these causal graphs come from. Why should we predict attributes based on the category (figure 1)? Why should the image be generated based on the Category (figure 5)? Specifically,

- is the causal graph representing the nature? E.g., physics.
- is the causal graph representing the data generation/collection pipeline? E.g., we first sample a category, and then search for the images.
- is the causal graph representing human's reasoning of the properties? This is quite tricky. For example, the arrow could be attribute -> category (e.g., a yellow, oval citrus fruit with thick skin and fragrant, acidic juice => lemon), or affordance -> category (an object that can be used to hold or transport something. => container).

I guess the primary reason for introducing a specific Bayes net is for debiasing purposes. The key idea is to apply "interference" during inference to compensate the biases during training (e.g., category frequency, etc.)

However, I feel that this part of the paper is not well compared with other approaches towards "debiasing". For example, 
https://openaccess.thecvf.com/content_CVPR_2020/papers/Wang_Towards_Fairness_in_Visual_Recognition_Effective_Strategies_for_Bias_Mitigation_CVPR_2020_paper.pdf
https://openaccess.thecvf.com/content/CVPR2021/papers/Ramaswamy_Fair_Attribute_Classification_Through_Latent_Space_De-Biasing_CVPR_2021_paper.pdf
These two papers discussed several methods for "remove biases." There are, of course, many other papers in this field, if the authors follow the pointers in these papers. I would like to see how the proposed model relate to these methods as well as how they compare empirically.

A minor thing about the metric: I am not sure how TDE/alpha-beta-TDE should be interpreted. why is this consistency score between network "weights" and "human labels" matter at all? This is especially tricky when all prediction networks take the image feature as their inputs.


summary_of_the_review: I think the dataset part is a solid contribution of the paper. However, for the proposed model, I think the authors are missing important baselines and discussions with related work.
review_title: null

Meta-review: This paper presents work on a dataset for object concept learning.  The main contributions include causal relations in the dataset and a method (OCRN).  The initial reviews pointed to concerns over the nature of the causal relations, the presentation of the paper, and the OCRN method and its motivations / use of the do-operator.  The reviewers engaged in significant discussions after considering the authors' responses and the others' reviews.  After this delibration, the concerns over the dataset, its annotations, and the presentation of the methods were deemed to be better served by a full revision and reconsideration of the paper.  As such, the paper is not recommended for acceptance at this time.
Decision: Reject