Venue: ICLR 2018 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: [====================================REVISION ======================================================]
Ok so the paper underwent major remodel, which significantly improved the clarity. I do agree now on Figure 5, which tips the scale for me to a weak accept. 
[====================================END OF REVISION ================================================]

This paper explores the problems of existing Deep variational bottle neck approaches for compact representation learning. Namely, the authors adjust deep variational bottle neck to conform to invariance properties (by making latent variable space to depend on copula only) - they name this model a  copula extension to dvib. They then go on to explore the sparsity of the latent space

My main issues with this paper are experiments: The proposed approach is tested only on 2 datasets (one synthetic, one real but tiny - 2K instances) and some of the plots (like Figure 5) are not convincing to me. On top of that, it is not clear how two methods compare computationally and how introduction of the copula  affects the convergence (if it does)

Minor comments
Page 1: forcing an compact -> forcing a compact
“and and” =>and
Section 2: mention that I is mutual information, it is not obvious for everyone

Figure 3: circles/triangles are too small, hard to see 
Figure 5: not really convincing. B does not appear much more structured than a, to me it looks like a simple transformation of a. 

review_title: An extension to DVIB

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper identifies and proposes a fix for a shortcoming of the Deep Information Bottleneck approach, namely that the induced representation is not invariant to monotonic transform of the marginal distributions (as opposed to the mutual information on which it is based). The authors address this shortcoming by applying the DIB to a transformation of the data, obtained by a copula transform. This explicit approach is shown on synthetic experiments to preserve more information about the target, yield better reconstruction and converge faster than the baseline. The authors further develop a sparse extension to this Deep Copula Information Bottleneck (DCIB), which yields improved representations (in terms of disentangling and sparsity) on a UCI dataset.

(significance) This is a promising idea. This paper builds on the information theoretic perspective of representation learning, and makes progress towards characterizing what makes for a good representation. Invariance to transforms of the marginal distributions is clearly a useful property, and the proposed method seems effective in this regard.
Unfortunately, I do not believe the paper is ready for publication as it stands, as it suffers from lack of clarity and the experimentation is limited in scope.

(clarity) While Section 3.3 clearly defines the explicit form of the algorithm (where data and labels are essentially pre-processed via a copula transform), details regarding the “implicit form” are very scarce. From Section 3.4, it seems as though the authors are optimizing the form of the gaussian information bottleneck I(x,t), in the hopes of recovering an encoder $f_\beta(x)$ which gaussianizes the input (thus emulating the explicit transform) ? Could the authors clarify whether this interpretation is correct, or alternatively provide additional clarifying details ? There are also many missing details in the experimental section: how were the number of “active” components selected ? Which versions of the algorithm (explicit/implicit) were used for which experiments ? I believe explicit was used for Section 4.1, and implicit for 4.2 but again this needs to be spelled out more clearly. I would also like to see a discussion (and perhaps experimental comparison) to standard preprocessing techniques, such as PCA-whitening.

(quality) The experiments are interesting and seem well executed. Unfortunately, I do not think their scope (single synthetic, plus a single UCI dataset) is sufficient. While the gap in performance is significant on the synthetic task, this gap appears to shrink significantly when moving to the UCI dataset. How does this method perform for more realistic data, even e.g. MNIST ? I think it is crucial to highlight that the deficiencies of DIB matter in practice, and are not simply a theoretical consideration. Similarly, the representation analyzed in Figure 7 is promising, but again the authors could have targeted other common datasets for disentangling, e.g. the simple sprites dataset used in the beta-VAE paper. I would have also liked to see a more direct and systemic validation of the claims made in the paper. For example, the shortcomings of DIB identified in Section 3.1, 3.2 could have been verified more directly by plotting I(y,t) for various monotonic transformations of x. A direct comparison of the explicit and implicit forms of the algorithms would also also make for a stronger paper in my opinion.

Pros:
* Theoretically well motivated
* Promising results on synthetic task
* Potential for impact
Cons:
* Paper suffers from lack of clarity (method and experimental section)
* Lack of ablative / introspective experiments
* Weak empirical results (small or toy datasets only).
review_title: A promising improvement to DVIB, but paper suffers from lack of clarity and limited experimentation.

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper presents a sparse latent representation learning algorithm based on an information theoretic objective formulated through meta-Gaussian information bottleneck and solved via variational auto-encoder stochastic optimization. The authors suggest Gaussianify the data using copula transformation and  further adopt a diagonal determinant approximation with justification of minimizing an upper bound of mutual information.  Experiments include both artificial data and real data. 

The paper is unclear at some places and writing gets confusing. For example, it is unclear whether and when explicit or implicit transforms are used for x and y in the experiments, and the discussion at the end of Section 3.3 also sounds confusing. It would be more helpful if the author can make those points more clear and offer some guidance about the choices between explicit and implicit transform in practice. Moreover, what is the form of f_beta and how beta is optimized?  In the first equation on page 5, is tilde y involved? How to choose lambda?

If MI is invariant to monotone transformations and information curves are determined by MIs, why “transformations basically makes information curve arbitrary”? Can you elaborate?  

Although the experimental results demonstrate that the proposed approach with copula transformation yields higher information curves, more compact representation and better reconstruction quality, it would be more significant if the author can show whether these would necessarily lead to any improvements on other goals such as classification accuracy or robustness under adversarial attacks. 

Minor comments: 

- What is the meaning of the dashed lines and the solid lines respectively in Figure 1? 
- Section 3.3 at the bottom of page 4: what is tilde t_j? and x in the second term? Is there a typo? 
- typo, find the “most orthogonal” representation if the inputs -> of the inputs 

Overall, the main idea of this paper is interesting and well motivated and but the technical contribution seems incremental. The paper suffers from lack of clarity at several places and the experimental results are convincing but not strong enough. 

***************
Updates: 
***************
The authors have clarified some questions that I had and further demonstrated the benefits of copula transform with new experiments in the revised paper. The new results are quite informative and addressed some of the concerns raised by me and other reviewers. I have updated my score to 6 accordingly. 



review_title: Interesting work . Both the clarity and the experimental results have been improved in the revised version

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposed a copula-based modification to an existing deep variational information bottleneck model, such that the marginals of the variables of interest (x, y) are decoupled from the DVIB latent variable model, allowing the latent space to be more compact when compared to the non-modified version. The experiments verified the relative compactness of the latent space, and also qualitatively shows that the learned latent features are more 'disentangled'. However, I wonder how sensitive are the learned latent features to the hyper-parameters and optimizations?

Quality: Ok. The claims appear to be sufficiently verified in the experiments. However, it would have been great to have an experiment that actually makes use of the learned features to make predictions. I struggle a little to see the relevance of the proposed method without a good motivating example.

Clarity: Below average. Section 3 is a little hard to understand. Is q(t|x) in Fig 1 a typo? How about t_j in equation (5)? There is a reference that appeared twice in the bibliography (1st and 2nd).

Originality and Significance: Average. The paper (if I understood it correctly) appears to be mainly about borrowing the key ideas from Rey et. al. 2014 and applying it to the existing DVIB model.
review_title: This paper improved on an existing latent variable model by combining ideas from different but somewhat related papers. Experimental results indeed show some improvements.

Meta-review: Observing that in contrast to classical information bottleneck, the deep variational information bottleneck (DVIB) model is not invariant to monotonic transformations of input and output marginals, the authors show how to incorporate this invariance along with sparsity in DVIB using the copula transform. The revised version of the paper addressed some of the reviewer concerns about clarity as well as the strength of the experimental section, but the authors are encouraged to improve these aspects of the paper further.
Decision: Accept (Poster)