Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: Summary:
(1) The paper makes vague and unsupported claims about causality.
   The notion of 'causal', as well as do() notation, is really hollowed out here. It is of limited significance. In Equation (3) do() disappears from one line to the next, illustrating how little it does.

(2) Lack of novelty: The paper references "Learning disentangled representations with Semi-supervised deep generative models", but fails to mention that their model is is nearly identical. 

(3) The quality is on the low side; confusing mistakes in figures, in some
 sections poor proof-reading.

Details:
(1) Exaggerated claims.
"We argue that the incapability for causal reasoning is the reason of DNN's vulnerability".  (This is stated twice in different words).

  Unfortunately, the term "causal reasoning" is used without making it precise.  The paper has a very weak relation to causality; one could replace "causal" with "translation invariant", and very little would change.  Their experiments try to learn translation invariance (appendix shows whitening invariance).

The paper makes gratuitous use of do() notation; for example, regular MNIST is reinterpreted as not being observational, but as arising from an intervention, namely do(translation=0).  But do() does not amount to much here. It reduces to mere conditioning here, as the do is always applied to a root node without parents.  This is clear in equation in 3, where do's are correctly just deleted from one line to the next; do(), and 'causal' are used mostly as a rhetorical device.

(2) The model in the paper is pretty much the same as "Learning disentangled representations with Semi-supervised deep generative models", figure 2 here corresponds to figure 4 there.  Both papers use partially supervised (or unsupervised) VAEs. There are some differences, e.g. the prior work considers a more general case and uses importance sampling.  This paper refers but fails to say how similar they are.  There should be a comparison between the difference inference methods used.


(3) The paper is sloppy in places.  

Figure 6: the dotted arrow between nodes A and Z should not be there.
It does not follow from the graphical model in Figure 5.

In Figure 12a),  the text contradicts the figure. The brown curve with fine tuning has lower accuracy than the green one without fine tuning, but the text says the opposite. 

Figure 11: there is a curve mislabelled as "Dis"

Results are mixed.  While results on vertical translation invariance and adversarial robustness seem good, the method fails to achieve horizontal translation invariance (Figure 7 b) and c)) - disappointing. But some of the translation invariance seems to be due to just using a generative model (the VAE), as it occurs without any fine-tuning.

review_title: Official Blind Review #4

final_score: rating: 8: Accept
review_content: In this paper, the author first assume the data generation process that sample X is generated by label Y, latent style (domain information) Z and other manipulations M and then propose the deep causal manipulation augmented model that use the do calculus to model the manipulations of data and further take it as the cause to the observed effect variables x. The author further elaborately devises experiment on the standard dataset and achieve good results.

Strong point:
(1)	The paper is well-written and the idea is motivated.
(2)	The experiment is convincing and the result is good.

Weak point:
(1)	Please provide the deducing process of the ELBO, such as how the ELBO is deduced in the training mode and the prediction process should be elaborated detailedly.
(2)	The symbolism is confusing, such as the \phi_1 and \phi_2 in Eq(6).
(3)	What is the difference between manipulations M and the latent style Z? Do they belong to the domain information? Please justify. 

review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: The paper tackles a very important problem, manipulation robustness of modern machine learning models, by applying a chain of design choices perfectly well:
  * VAEs as probabilistic generators
  * Manipulations as independent causes that can be generated by interventions
  * Causal inference for manipulation-corrected training
  * Bayesian inference for robust prediction

Sticking to the Bayesian point of view, the method can perform model averaging across potential manipulations at the test time. This is an extremely elegant property, which is also proven in the experiments to be very effective.

Being able to learn previously unseen types of manipulations only by proper application of causal inference tools is a very important news for the adversarial robustness community.

Figure 9 is a spectacular proof of concept to illustrate the disentanglement property of the proposed method.

I have only one point for improvement. I do not buy the argument that we should do q(m|x) but not q(m|x,y). Why should we exclude class-specific manipulations? This wouldn't affect the cause (the class) but only the outcome, so would be a valid manipulation. I would actually expect the model to work still well with q(m|x,y). Could the authors comment on what the concrete benefit of leaving out y from manipulations is?

The construction of the loss in Eq 5 is in spirit semi-supervised learning on m. We show the true m=0 cases to the model during training but assume not to know the labels of the manipulated samples. It could be beneficial to draw a link to semi-supervised learning here and even tie it to the arguments about the relationship between causal inference and semi-supervised learning:

Schölkopf et all., On Causal and Anticausal Learning, ICML, 2012.

review_title: Official Blind Review #1

Meta-review: This paper attempts to present a causal view of robustness in classifiers, which is a very important area of research.
However, the connection to causality with the presented model is very thin and, in fact, mathematically unnecessary. Interventions are only applied to root nodes (as pointed out by R4) so they just amount to standard conditioning on the variable "M". The experimental results could be obtained without any mention to causal interventions.
Decision: Reject