Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper is concerned with contrastive disentanglement. The considered problem is interesting and important in the community, and the proposed method seems to be of practical use, according to the empirical results. My concern is that the contribution of the paper on the theoretical or methodological side seems a bit weak.

The proposed method relies on the VAE framework for contrastive disentanglement. The authors proposed two modifications: one is to explicit enforce the posterior of the content representation to be a Delta distribution, and the other is to make sure that the background representation has the same distribution across the background and the target. In the original cVAE framework for contrastive disentanglement, the two constraints were considered in a slightly more implicit way. It is not surprising to see that the proposed method seems to outperform cVAE only slightly. To see how much we can really gain by further explicitly enforcing the two constraints, the authors may do some further studies with varied sample sizes. BTW, in the result for CelebA in Table 1, cVAE seems to outperform the proposed method, denoted by 'Ours', according to Acc of z_x, but it was indicated in the text (as well as in by the bold font) that 'Ours' performed the best--is there a typo?

I acknowledge I read the authors' response and other reviews and would like to keep my original rating.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: Existing formulations for contrastive disentanglement ignore the information about the prior, i.e., the salient features of the background data should be zero, and moreover, additional KL-divergence-based losses which are hard to estimate in practice are introduced to improve disentanglement.

To resolve this issue, the authors propose new regularizations still based on VAE. Specifically, they propose to

Regularization 1. penalize the generated mean and standard deviation of background latent variable towards (0,0) which means Dirac-delta distribution on 0.

Regularization 2. match common latent variable distributions for background dataset and target dataset using Wasserstein distance.

While authors point out that existing formulation in Abid & Zou (2019); Ruiz et al. (2019) ignores about the prior, the effect of regularizing salient feature seems marginal. This can be found in experimental results in Table, proposed objective (7) gives similar or worse result on L MNIST(similar), CelebA(worse), and Affectnet(worse) dataset. Also, visualization of salient features in Figure 4 show that representation balancing effects of proposed regularization terms are marginal compared to cVAE.

Another cocern is proposed regularization 1,2 might hurt the original ELBO objective. This concern is also related to beta-VAE, the paper showed that large coefficient for prior regularization simply makes the disentanglement effect for VAE. On the other hand, proposed objective (7) requires 'large constant' for approximate quadratic loss. This might induce unintentional negative effects for the objective (7) and objective (8). It would be great if authors can show the proposed regularization does not induce such unintended numerical problem. A possible solution might be showing asymptotic theoretical guarantees as in semi-implicit variational inference.

The objective (8) seems to require additional computational cost to solve linear programming problem. It is not clear if it is effective enough to bear that extra cost; it is hard to conclude that proposed regularization performs better than previous works without confidence interval.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: Contributions:
1. This paper fixes two problems that appeared in common contrastive disentanglement methods.
2. The paper evaluates its method in multiple experiments.

Overall, this paper does a nice contribution to improving existing methods of contrastive disentanglement. My major concern is the novelty in this paper since all the contributions can be summarized into adding two additional loss terms, which I would like to discuss below.

1. [The missing term KL[q(s|y)||p_y(s)]]. One major claim of this paper is to add back the additional term KL[q(s|y)||p_y(s)] in order to push the encoder q(s|y) to converge to a point mass \delta\{s=0\} for background images y. According to the mathematical formula, this term is natural, but the major concern is the non-overlapping support between q(s|y) and p_y(s), since in the paper they assume p_y(s)=\delta\{s=0\}. This paper claims they fix this problem. But I do not think the solution is satisfactory, because they use another term E_q[\mu_{\phi,s}(y)^2+\sigma2_{\phi,s}(y)] to replace KL[q(s|y)||\delta\{s=0\}]. If I get it correct, the replacing term E_q[\mu(y)^2+\sigma2(y)]=H(q(s|y)) is the entropy of q(s|y). But if we check the true divergence term, KL[q(s|y)||\delta\{s=0\}]=-E_q[\log\delta\{s=0\}]-H(q(s|y)) that consists of a negative entropy term. This means the behavior of the proposed loss tries to minimize the entropy while the original term tries to maximize the entropy, regardless of it is ill-defined. Thus, the replacement of the loss term seems reasonable to me at first glance but does not quite directly fix the problem of an ill-defined KL-divergence. A more reasonable solution is to replace the KL divergence with the Wasserstein distance. Is there any difficulty if we do that?

2. [The additional distributional matching term W_2^2(q_t(z),q_b(z))]. This term makes perfect sense to me, even though it is just added artificially. I'm more curious about the way to evaluate the gradient. Normally, computing Wasserstein distance is hard due to a hard linear programming matching algorithm. In this paper, the authors seem to use an existing library to compute the optimal transportation matrix. Is that a heavy computational burden? Or one could resort to some entropy regularization methods with sinkhorn iterative solvers. I'm curious about how these two methods compare with each other.

3. Another concern is the numerical result seems not as good as the qualitative results.

Overall, I think this paper does make some changes regarding previous works on contrastive disentanglement. But their main contributions need some more explanation and justification.
review_title: Official Blind Review #1

Meta-review: The paper proposes new regularizations on contrastive disentanglement. After reading the author's response,  all the reviewers still think that the contribution is too limited and all agree to reject.
Decision: Reject