Rate-Distortion Auto-Encoders
Authors: Luis G. Sanchez Giraldo, Jose C. Principe
ICLR 2014

Abstract: We propose a learning algorithm for auto-encoders based on a rate-distortion objective. Our goal is to minimize the mutual information between the inputs and the outputs of an auto-encoder subject to a fidelity constraint. Minimizing the mutual information acts as a regularization term whereas the fidelity constraint can be understood as a risk functional in the conventional statistical learning setting. The proposed algorithm uses a recently introduced measure of entropy based on infinitely divisible matrices that avoids the plug in estimation of densities. Experiments using over-complete bases show that the auto-encoder learns a regularized input-output map without explicit regularization terms or add-hoc constraints such as tied weights.


Peer review:
[Review]:
* Brief summary of paper:
The paper proposes a novel kind of regularization for learning autoencoders that is rooted in rate-distortion theory.
The regularization term is a kernel-based estimator of entropy (of the reconstructed data) proposed by the authors at last year's ICLR.
Experiments on 2D toy data show that it works as expected.

* Assessment:

The rate-distortion approach to autoencoders is interesting, and I beleive novel. I would however refrain from stating, as written in the paper, that it allows learning autoencoders without an explicit regularization terms: the conditional entropy clearly plays the role of a data-dependent regularization term (similar to most alternative approaches for learning overcomplete autoencders: they also don't define an explicit penalty on the parameters). 

The kernel-based entropy estimator seems however computationally very expensive, since it requires full eigendecomposition of Gram matrices. 

The main weakness of the paper is the very limited experimental evaluation of the method (2D toys), and the lack of comparison with any other regularized autoencoder approach, not even a discussion of what this new mathematically sophisticated and computationally heavy approach might offer as benefits. Similarly, the authors only use their own gram-matrix-based entropy estimator: a brief discussion of properly referenced alternative, more classical, nonparametric entropy estimators would have been in order (and ideally with experimental comparison). 

The paper is mostly well written. There is however a confusing proabably unintended notational shift happening after Eq. 14. In eq 14 and in ghe previous sub-section you use S_alpha, and in what follows you use H(K) which has nowhere been formally defined or related to S_alpha. Also where does the N arise from in eq 16. Please clarify.

Lastly I have a question/remark: conditional entropy H(X|X^) could also be rewritten as 
H(X|X^) = H(X) + H(X^|X) - H(X^)
Now since you are concerned with a deterministic autoencoder, X^ is a deterministic function of X, so it would seem that H(X^ | X) is a constant.
So it seems you might as well penalize only H(X^) rathe than trying to estimate and penalize the actual conditional entropy. Does this reasoning seem valid? 


* Pros and cons:

Pros:
+ original approach to autoencoder regularization

Cons:
- computationally very heavy approach
- very limited experimental evaluation (toy 2d data)
- lack of comparison (neither in discussion nor experimental) with anything related