Venue: ICLR 2018 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper aims at robust image classification against adversarial domain shifts. In the used model, there are two types of latent features, "core" features and "style" features, and the goal is to achieved by avoiding using the changing style features. The proposed method, which makes use of grouping information, seems reasonable and useful. 

It is nice that the authors use "counterfactual regularization". But I failed to see a clear, new contribution of using this causal regularization, compared to some of the previous methods to achieve invariance (e.g., relative to translation or rotation). For examples of such methods, one may see the paper "Transform Invariant Auto-encoder" (by Matsuo et al.) and references therein.

The data-generating process for the considered model, given in Figure 2, seems to be consistent with Figure 1 of the paper "Domain Adaptation with Conditional Transferable Components" (by Gong et al.). Perhaps the authors can draw the connection between their work and Gong et al.'s work and the related work discussed in that paper.

Below are some more detailed comments. In Introduction, it would be nice if the authors made it clear that "Their high predictive accuracy might suggest that the extracted latent features and learned representations resemble the characteristics our human cognition uses for the task at hand." Why do the features human cognition uses give an optimal predictive accuracy? On page 2, the authors claimed that "These are arguably one reason why deep learning requires large sample sizes as large sample size is clearly not per se a guarantee that the confounding effect will become weaker." Could the authors give more detail on this? A reference would be appreciated. 
review_title: The proposed methods seems useful but novelty seems limited

final_score: rating: 4: Ok but not good enough - rejection
review_content: Proposal is to restrict the feasible parameters to ones that have produce a function with small variance over pre-defined groups of images that should be classified the same. As authors note, this constraint can be converted into a KKT style penalty with KKT multiplier lambda.  Thus this is very  similar to other regularizers that increase smoothness of the function, such as total variation or a graph Laplacian defined with graph edges connecting the examples in each group, as well as manifold regularization (see e.g. Belkin, Niyogi et al. JMLR).  Heck, in practie ridge regularization will also do something similar for many function classes. 

Experiments didn't compare to any similar smoothness regularization (and my preferred would have been a comparison to graph Laplacian or total variation on graphs formed by the same clustered examples). It's also not clear either how important it is that they hand-define the groups over which to minimize variance or if just generally adding smoothness regularization would have achieved the same results.   That made it hard to get excited about the results in a vacuum. 

Would this proposed strategy have thwarted the Russian tank legend problem? Would it have fixed the Google gorilla problem? Why or why not?

Overall, I found the writing a bit bombastic for a strategy that seems to require the user to hand-define groups/clusters of examples. 

Page 2: calling additional instances of the same person “counterfactual observations” didn’t seem consistent with the usual definition of that term… maybe I am just missing the semantic link here, but this isn't how we usually use the term counterfactual in my corner of the field.

Re: “one creates additional samples by modifying…” be nice to quote more of the early work doing this, I believe the first work of this sort was Scholkopf’s, he called it “virtual examples” and I’m pretty sure he specifically did it for rotation MNIST images (and if not exactly that, it was implied).  I think the right citation is “Incorporating invariances in support vector learning machines
“ Scholkopf, Burges, Vapnik 1996, but also see Decoste * Scholkopf 2002 “Training invariant support vector machines.” 
review_title: Lacks sufficient comparisons to other similar regularizers

final_score: rating: 7: Good paper, accept
review_content: The paper discusses ways to guard against adversarial domain shifts with so-called counterfactual regularization. The main idea is that in several datasets there are many instances of images for the same object/person, and that taking this into account by learning a classifier that is invariant to the superficial changes (or “style” features, e.g. hair color, lighting, rotation etc.) can improve the robustness and prediction accuracy. The authors show the benefit of this approach, as opposed to the naive way of just using all images without any grouping, in several toy experimental settings.

Although I really wanted to like the paper, I have several concerns. First and most importantly, the paper is not citing several important related work. Especially, I have the impression that the paper is focusing on a very similar setting (causally) to the one considered in  [Gong et al. 2016] (http://proceedings.mlr.press/v48/gong16.html), as can be seen from Fig. 1. Although not focusing on classification directly, this paper also tries to a function T(X) such that P(Y|T(X)) is invariant to domain change. Moreover, in that paper, the authors assume that even the distribution of the class can be changed in the different domains (or interventions in this paper).
Besides, there are also other less related papers, e.g. http://proceedings.mlr.press/v28/zhang13d.pdf, https://www.aaai.org/ocs/index.php/AAAI/AAAI15/paper/view/10052/0, https://arxiv.org/abs/1707.09724, (or potentially https://arxiv.org/abs/1507.05333 and https://arxiv.org/abs/1707.06422), that I think may be mentioned for a more complete picture. Since there is some related work, it may be also worth to compare with it, or use the same datasets.

I’m also not very happy with the term “counterfactual”. As the authors mention in footnote, this is not the correct use of the term, since counterfactual means “against the fact”. For example, a counterfactual query is “we gave the patient a drug and the patient died, what would have happened if we didn’t give the drug?” In this case, these are just different interventions on possibly the same object. I’m not sure that in the practical applications one can assure that the noise variables stay the same, which, as the authors correctly mention, would make it a bit closer to counterfactuals. It may sound pedantic, but I don’t understand why use the wrong and confusing terminology for no specific reason, also because in practice the paper reduces to the simple idea of finding a classifier that doesn’t vary too much in the different images of the single object.

**EDIT**: I was satisfied with the clarifications from the authors and I appreciated the changes that they did with respect to the related work and terminology, so I changed my evaluation from a 5 (marginally below threshold) to a 7 (good paper, accept).
review_title: Potentially interesting, but fails to mention very related work

Meta-review: The paper proposes a method to robustify neural networks which is an important problem. They uses ideas from causality and create a model that would only depend on "stable" features ignoring the easy to manipulate ones. The paper has some interesting ideas, however, the main concern is regarding insufficient comparison to existing literature. One of the reviewers also has concerns regarding novelty of the approach.
Decision: Reject