Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose a framework for studying the tendency of deep neural networks to preferentially adopt "cues". Specifically, they focus on settings where multiple cues are equally likely, though not all of them are equally exploited. To set up such a scenario, they introduce the WCST-ML task, in which the prevalence of cues can be parametrically controlled. They also conduct empirical studies on the more naturalistic UTKFace dataset. The authors introduce a set of metrics, such as path connectivity, attractor basin properties, etc. to analyze cue preferences from a loss landscape perspective. The authors also explain these observations based on the "complexity" of cues.   
main_review: Strengths:

Overall this is a well-written paper, clearly motivated and carefully constructed. The finding that the number of solutions (i.e. parameter configurations) that rely on preferred cues are also abundant is a good corroboratory result.

The motivations for this study are also exactly what the field needs at this point, given the abundant use of deep convolutional networks for a variety of applications with minimal understanding of its exact decision-making mechanisms.

Though the synthetic task (Wisconsin card sorting) has been widely adopted in the cognitive neuroscience community, their formulation for mainstream ML is nifty and allows for systematic empirical evaluation.

Weaknesses:

Having said that, this paper fails to deliver on its promise of intricate analysis.
(i) There is an inherent question that authors fail to sufficiently address. The selected "cues" for analysis have intrinsically different extraction demands from the stimuli, thus making the argument about them being "equally prevalent" moot. For example, accessing color is more direct (pixel-level information is directly available) than accessing, say, shape (for which a network needs to build sufficiently large receptive fields to understand the global geometry of a scene).

(ii) The experiments on the UTKFace dataset are not convincing. The authors themselves acknowledge that there might be other "shortcut cues" outside of the selected ethnicity and age cues. This brings up a subtle (yet important) question. In naturalistic datasets, how can one determine or interpret a basis set of cues that are truly orthogonal dimensions? The experiments on WCST-ML work because the shape, color, scale, and orientation are by definition orthogonal. For all we know, the "cues" in naturalistic datasets could be abstract and not human-interpretable too.

(iii) Treating the number of model parameters as a proxy for the Kolmogorov complexity of the input-output mapping seems ill-advised. If one trains a parameter-shared recurrent neural network it can perform the same effective computations as a ResNet, but with much fewer parameters (though the task per se has not changed, and by extension the cues). I would appreciate it if the authors can offer some insight into this.

Minor:
(i) Figures need to be made more legible. Axis labels and text insets are barely visible without zooming in.
(ii) Particularly, Figure 4 could be refined further. The caption isn't very descriptive, and the section on "Qualitative view on the loss surface" is a bit misleading. The ethnicity solution doesn't seem to be "characterized with a flatter and wider surface" as compared to age and gender solutions in Fig. 4. If there are more obvious examples, the authors should use them instead.
summary_of_the_review: This is a well-written paper trying to address an important problem. However, as I've expressed in the main review, some of the claims are unsupported and hence my initial assessment. I am willing to update my score if the authors are able to provide a convincing response!
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper conducts a study of which visual cues are preferred by current vision models. The paper designs a training setup with several cues where each cue is equally correlated with the image label. The paper shows that visual cues like color are much easier to be learned by a vision model, than other cues such as orientation and shape. The paper also provides evidence that easy-to-learn cues tend to converge to relatively flat minima and models that prefer these cues are more abundant in parameter space.
main_review: The paper is a very interesting read. It provides an interesting analysis of model preferences for various visual cues. The analysis shows the preference of visual models towards low complexity visual cues, such as color and ethnicity. 

Main comments:

- Averted solution or averted cues are not introduced, before being used in section 3.2. What is the difference between averted and preferred? Section 3.2 seems to imply that preferred solutions are computed by optimizing the model using D_diag and averted solutions computed using D_i for a specific property i that is not deemed preferred. Am I correct? If this is the case, it seems unreasonable to be comparing the two solutions directly (eg in Fig 4) since they are computed using datasets of different sizes.
- In Fig 4, it also is not clear what the loss landscape represents. Section 3.2 mentions that it's the loss around averted and preferred solutions, but Figure 4 does not mention which ones are averted and which ones are preferred. Also, Figure 4 says that the two directions of parameter variation were chosen at random. Given the high dimensionality of DNN parameters, it would be much more informative to depict several of these plots per solution, to cover more directions of variation.
- Section 4 provides evidence to the idea that certain cues have higher complexity than others. This is measured by searching for the smallest model that can memorized the training set. Color could be memorized by a model with only 1.2K parameters, and orientations needed 273K parameters. A natural question is then whether large models present less biases towards simpler cues. As the parameter count increases, the solution set for the orientation task must also increase. It would be interesting to see if the model preference for more complex cues would also increase, or if solution sets for simpler cues would still dominate for very large models.
- Although the diagonal dataset construct is appropriate to see which cues are preferred by a model, in practice this construct almost never applies. Real data may present correlations similar to those found in the diagonal construct, but will also contain off-diagonal samples. Thus, an interesting analysis would be to measure how much you'd need to deviate from D_diag to obtain an "averted solution". In other words, if we train a model with (100-p)% diagonal samples and p% off-diagonal samples labeled according to cue i (e.g. labeled for orientation), how likely is the model to correctly predict cue i for increasing values of p?
summary_of_the_review: The paper is a very interesting read. It provides an interesting and insightful analysis of model preferences for various visual cues. The analysis shows the preference of visual models towards low complexity visual cues, such as color and ethnicity. Given the potential impact of the analysis, I recommend the paper to be accepted. I would nevertheless strongly encourage the authors to address my main comments/concerns.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper discusses biases in inductive learning in deep neural networks that stem from pathologically sampled data. The authors pose a problem set where a trainer only sees samples where two or more latent values can only be observed in a fully correlated fashion (e.g. scale, color, shape). They then design various criteria and protocols to gain insight in the behavior of the learned model when the correlation does not hold any more (i.e. samples that have not been seen in the training data). They conclude that in this case of generalization to unseen data, the trained model has an implicit bias towards choosing (1) mostly single cues to determine the predicted label (e.g. color only), (2) that more preferred cues are simpler than less preferred cues and (3) that the underlying solution space of possible parameters has more solutions that prefer the simple cues. They demonstrate empirical results on variations of existing datasets (DSprites, UTKFace).   
main_review: The paper is written well, the claims are laid out clearly, and the method is largely well described. The approach seems methodically sound given the problem statement  (caveat in W2 below). The path of investigation is fundamentally valuable in the sense that models of biases and generalization of networks are helpful tools.

Weaknesses:
[W1] Some parts of the exposition need minor clarification:
(a) p6: "The trend is clear for ResNet20 and ViT, while the ethnicity and gender preferences are within the error bars for FFnet." This refers to Figure 3, and I cannot understand this from Figure 3, in the least it is confusingly stated. The ethnicity and gender graphs for FFNet in figure 3 seem separated by several standard deviations according to the error bars.
(b) Figure 6, three right columns: What does the dot and the black arch signify? I was not able to gain this from text or caption. I assume that the zero loss path is the dashed line?

[W2] A somewhat methodical weakness of the paper seems to be the problem statement. For instance in the introductory example of section one/figure 1, the authors imply that the model is biased towards scale (then shape, then color). I do agree that this is consistent with the model preferring the property of scale in making the decision. Is this bias, though? In order to evidence bias, I'd need a truth/label assignment on the off diagonals (e.g. indicating that blue small triangle is not supposed to be class 1). Without this truth assignment I do not see the bias of the model.

[W3] The main weakness of the paper seems to me that I am asking myself: What have I learned after reading the paper? I feel that the experiment setup is sound in the stated sense: what does the network learn if it only ever sees pathological samples (i.e. where all latent variables are exactly correlated). However, the presented answers that networks tend to learn simpler cues, where simpler can be measured by Kolmogorov complexity are not overly surprising or counter-intuitive. Perhaps the authors can rephrase the contributions more clearly towards the benefits? For instance: With the tools, analysis and exposure of this paper, what can I now see different, do different, etc. What are possible next steps where stronger impact is on the horizon?
summary_of_the_review: The paper addresses an important topic, but I feel that the overall impact, measured by depth of presented contributions is too shallow.
review_title: null

Meta-review: The reviewers were split, with one of them leaning towards rejection, primarily due the (perceived) limited impact of the study. I tend to agree with the other reviewers that this paper provides an interesting and original framework for analysis of learning models, and while there are substantial shortcomings, they are outweighed by the positives (including the promise this approach may hold for analysis of learning in more realistic scenarios). I therefore recommend acceptance, if space in the proceedings allows.
Decision: Accept (Poster)