Rate-Distortion Auto-Encoders
Authors: Luis G. Sanchez Giraldo, Jose C. Principe
ICLR 2014

Abstract: We propose a learning algorithm for auto-encoders based on a rate-distortion objective. Our goal is to minimize the mutual information between the inputs and the outputs of an auto-encoder subject to a fidelity constraint. Minimizing the mutual information acts as a regularization term whereas the fidelity constraint can be understood as a risk functional in the conventional statistical learning setting. The proposed algorithm uses a recently introduced measure of entropy based on infinitely divisible matrices that avoids the plug in estimation of densities. Experiments using over-complete bases show that the auto-encoder learns a regularized input-output map without explicit regularization terms or add-hoc constraints such as tied weights.


Peer review:
[Review]:
This paper proposes to regularize auto-encoders by minimizing the mutual information between input and output. The minimization of mutual information is based on an alternative definition of entropy (Sanchez et al. 2013). Although auto-encoders have been around for more than 20 years, the introduction of deep learning (Hinton et al. 2006,Bengio et al. 2006) has renewed the interest in these models and their regularization schemes (Vincent et al. 2008), as they can be stacked to achieve state-of-the-art performance.

A first contribution establishes a link between the proposed model (rate-distortion auto-encoders) and PCA in the case of a Gaussian input variable but does not discuss the already established proof that traditional linear auto-encoders are equivalent to PCA (Baldi and Hornik;1989).

The authors then derive a gradient training procedure for the rate-distortion auto-encoder and report the result of two experiments in dimension 2, namely they compare the input to the reconstruction :
1 - In the case where the input is a specific Gaussian variable.
2 - In the case where the input is a specific mixture of 3 Gaussian variables.

finally the authors  conclude that the reconstruction approximately fits the input distribution.

Although the use of rate distortion theory to regularize an auto-encoder is new, the paper suffers from several issues. First, the authors do not define the problem which they are trying to address. They present a cost function and their goal is to minimize it. To what end ? Accordingly, it is unclear what the authors are trying to prove in their experiments: a regularization property ? Since the end goad is not clearly defined, why is regularization important at all ? Additionally, the experiments are very insufficient as they only consider two very simple artificial datasets: the first with a single predefined Gaussian, and the second with three Gaussians. In these experiments, the proposed method is  not compared to any other model or baseline.

pros:
cons:
 - Goal is not defined.
 - experiments are in a very low dimensional space (dim=2) which is not very relevant for auto-encoders.
 - experiments do not compare the proposed model to a baseline (traditional auto-encoder) or to other forms of regularized auto-encoders (e.g. denoising auto-encoders or contracting auto-encoders).
 - experiments do not report any quantitative measure of performance (e.g. log-likelihood or classification accuracy).