Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: ** Summary
The paper studies the problem of representation learning under invariance constraints (i.e., the representation should be invariant wrt some attributes). The authors first review the adversarial invariance induction (AII) approach and they point out its limitations and then they propose a novel variant, which introduces an explicit regularization to minimize pairwise divergence (i.e., different attributes should lead to the same conditional distribution over the learned representation). The authors support the modified objective function both from a formal point of view and with an extensive empirical validation

** Evaluation
The paper lies a bit outside my area of expertise. Although the paper tries to capture intrinsic limitations of the AII approach and build a more stable algorithm, my impression is that too many elements in the discussion and derivation remain too vague at the current stage and they would require better and clearer explanation.

Detailed comments:
1- In many parts of the paper the notation is not very rigorous and sometimes it may create confusion. In general, the writing needs to be improved in many parts:
- In eq. 1, it is not explained what the expectation is wrt. It should be x drawn from p(x). But it would be better to make it explicit.
- The setting defined in sect.2 should be more rigorous: it is not clear what y is and what are the attributes we would like to be invariant for. This notation is not fully consistent with eq.1.
- In the first paragraph of Sect.3, it would be very helpful to have a more complete sketch of the algorithm. In general, you mention in the Sect. 2 that you focus on the supervised case, but then it is not clear whether this is the case across the paper.
2- While the intuition behind using the divergence is sound, as it is mentioned, it seems to suffer from the same issue as the original AII: minimizing a lower bound does not guarantee that we are minimizing the actual objective. Having the support of 0, does not seem to make it much more sensible.
3- As the description of the algorithm advances in Sect.4.2, it is clear that many additional choices need to be made in order to have a full workable algorithm (e.g., how to estimate q_\phi^i). In the paper, the actual algorithm is never reported in detail and this makes the experiments very hard to reproduce in my opinion.
4- At the best of my understanding, the algorithm may become more and more intractable as the number of attribute values grows. In fact, you need to check the divergence for each pair of values and estimated distributions.
5- The empirical analysis seems well executed and a good level of detail is reported on how the datasets are managed and the experiments are set up.

Minor comments:
- The caption of Fig.1 is not very clear. Many elements at this stage of the paper are not defined yet (e.g., "our proposal minimize the proxy of divergence ...").
- p2: "as the assumption of ... is rarely holds", remove "is"
- p3: "(encoder) that parameterized", remove "that"
- last paragraph of Sect.2 is very confusing.
- p4: "updation" is not a word
- p4: "In general, minimizing the upper bound..." does not seem correct.
- p4: "even such a simple case" -> "even in such a simple case"
- Sect 4.1 "the above theoretical", I would not really say there is much theory behind the analysis in the previous section.
- p5 "an ttribute" -> "an attribute"
- p6 "the average discriminator's perception" what is the perception?
review_title: Official Blind Review #3

final_score: rating: 1: Reject
review_content: The paper points out that the practical behavior of AII assumes the optimality of the attribute classifier, which is rarely held in practice. And claims that the paper analyzes the practical behavior of AII both theoretically and empirically, indicating that AII has theoretical difficulty as it maximizes variational upper bound of the actual conditional entropy. Then it argues an ugly modification based on a wrong property of conditional entropy. 

- The paper says that it analyzes AII theoretically and empirically. But it only shows the practical drawback of AII intuitively without any theoretical proof.
- In Section 3, the paper says : 'In general, maximizing the upper bound of the function of interest $f$ does not guarantee the minimizing the $f$ '. 
- Also in the section 3, the paper says : 'Figure 2-(b) visualizes how distribution move during the optimization of AII on synthesized data'. And I want to ask why the caption of Figure 2-(b) is IIDM ?
- The proposition 1 on which the modification proposed in the paper is based will be not true when the distribution of attributes is not uniform by Bayses' s law which is used in the so called proof of the proposition 1. Which means that $p(z|a_i)=p(z|a_j) \Leftarrow p(a_i|z)=p(a_j|z)$ if and only if $p(a_i)=p(a_j)$.
- In the proof of equation 4 which is the main theorem of the total work. We can find $-\sum q_{\phi}^i(a)\log\mathbb{E}_{p_{\theta}^j(z)}[q_{\phi}(a|z)]\ge -\sum q_{\phi}^i(a)\mathbb{E}_{p_{\theta}^j(z)}[\log q_{\phi}(a|z)]$.  The inequality direction is reversed. 

review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: *Summary*

The paper proposes a new method to learn data-driven representations, being invariant to some specific nuisance factors which are detrimental for the selected (supervised) classification task.
Authors build upon the existing probabilistic framework termed Adversarial Invariant Induction (AII) from (Xie et al., 2017). 

They claim to explore it under a both theoretical and practical point of view, demonstrating the limitations of maximizing a variational upper bound on conditional entropy as a proxy to achieve invariance. 

Leveraging these observation, authors propose a novel method, called “invariance induction by discriminator matching” (IIDM) that is based on a regularized classification loss, penalized by a Kullbach-Leibler divergence between conditional distributions of the nuisance factor.

Extremely convincing experiments are carried out on a synthetic and a real benchmark in multi-source domain generalization (PACS).



*Pros*
1. The genesis of the proposed IIDM is extremely paced since smoothly derived from the AII framework.
2. Experimental results on a synthetic benchmark (a version of rotated MNIST) and on a popular benchmark for domain generalization (PACS) proved the effectiveness of IIDM



 *Cons*
1. The paper is hard to get, if the reader is not familiar with related literature
2. It is not fully clear from the paper which parts are original and which are inherited from prior work.
3. The structure of the paper needs to be improved (check my comments in the section beneath)
Some of the proposed methodologies are not clear (IIDM+)




*Detailed Comments*

The problem considered by authors is surely interesting and addressing a popular topic in computer vision and deep learning. 

1. Unfortunately, the paper, as it is is hard to get for scholars which are not expert of the AII formalism, which, in my opinion is not enough detailed. Therefore, in my opinion clarity is something that authors should try to work hard on: for instance, during the rebuttal time, authors can write from scratch an entire new Section in which they explain in plain terms the main outcomes of their paper, without entering too much into technical details.


2. Additionally, the structure of the paper needs, in my opinion a major re-styling, still for the sake of better readability:
2.a. A visualization of the proposed method (for instance, using flow-diagrams) in comparison with the existing AII should help in rapidly getting the factors of novelty of the proposed IIDM method. I would also encourage authors to add a pseudo-code
2.b. Since authors claim two major contributions (understanding AII + IIDM), I would like to see those two contributions thoroughly presented and dissected in two separated sections of the paper. I am not fully convinced with the actual writing style in which the two contributions seem to be intertwined together.


3. Although already convincing, the experimental part can be improved:
3.a. Instead of a version of rotated-MNIST, authors can test on the “digits-five” setting (MNIST, MNIST-M, SVHN, UPS, SYN) as done in several works like http://openaccess.thecvf.com/content_cvpr_2018/papers/Xu_Deep_Cocktail_Network_CVPR_2018_paper.pdf.
3.b. In addition to multi-domain generalization, authors could also have tried more classical unsupervised domain adaptation settings or, even, single-source domain generalization as in https://papers.nips.cc/paper/7779-generalizing-to-unseen-domains-via-adversarial-data-augmentation.



*Final Evaluation*

I think that the main aspect that authors should face during the rebuttal is to make the paper more easy to read and better separate the two contributions (understanding AII and IIDM). What I am not convinced at all about the writing style of the authors since when reading the paper I am not always capable of understanding what is novel (since proposed by authors) and what is inherited from prior work. But, maybe, the reason for this is that I am not an expert of the specific related field - but, even so, I think that the paper needs to be understood from the broadest audience possible.

Instead, I am familiar with multi-source/single-source domain generalization (and adaptation) and, after my careful analysis of the experiments, I see a lot of potential in the approach. I would me more than interested in checking the performance of the proposed method over some of the novel benchmarks that I have recommended. It would be nice if authors add more experiments, but I know that this is always a complicated request during a rebuttal period.
Globally, if I were asked to only rate the experimental part, I would have promoted for full acceptance. Unfortunately, the theoretical part of the paper is not fully clear to me and, therefore, I am not confident in calling for a full acceptance only based on the experiments. 

In brief, I would go for a weak reject, looking forward to the authors’ rebuttal and the opinion of the other Fellow Reviewers.
review_title: Official Blind Review #2

Meta-review: The paper proposes a modification to improve adversarial invariance induction for learning representations under invariance constraints. The authors provide both a formal analysis and experimental evaluation of the method. The reviewers generally agree that the experimental evaluation is rigorous and above average, but the paper lacks clarity making it difficult to judge the significance of it. Therefore, I recommend rejection, but encourage the authors to improve the presentation and resubmit.
Decision: Reject