Rate-Distortion Auto-Encoders
Authors: Luis G. Sanchez Giraldo, Jose C. Principe
ICLR 2014

Abstract: We propose a learning algorithm for auto-encoders based on a rate-distortion objective. Our goal is to minimize the mutual information between the inputs and the outputs of an auto-encoder subject to a fidelity constraint. Minimizing the mutual information acts as a regularization term whereas the fidelity constraint can be understood as a risk functional in the conventional statistical learning setting. The proposed algorithm uses a recently introduced measure of entropy based on infinitely divisible matrices that avoids the plug in estimation of densities. Experiments using over-complete bases show that the auto-encoder learns a regularized input-output map without explicit regularization terms or add-hoc constraints such as tied weights.


Peer review:
[Review]:
This paper proposes a new criterion to train auto-encoders: minimizing the
mutual information (MI) between the input and output distributions, under the
constraint that the output is close enough to the output. This constraint can
be seen as the 'risk' we want to keep small, while the minimization of the MI
adds regularization to prevent overfitting (i.e. learning an identity mapping).

This seems to be an interesting direction to investigate, however I find the
submitted paper to fall short in two important areas:

1. The motivation and intuition behind this algorithm are not very clear. At
the end of section 2, we do see that minimizing the MI 'can have the effect of
lowering the entropy of the output variable and thus, we can think of the
mapping f as a contraction' but that does not really explain what kind of
properties we can expect when minimizing eq. 2. For instance, if we wanted to
lower this entropy, why not just do that directly? Another point that confuses
me is that the problem is initially stated as a constrained optimization one,
but if I understand correctly, the actual algorithm is performing gradient
descent on the Lagrangian (eq. 8): this Lagrangian is the sum of two terms, one
being the risk to minimize, and one being the regularization, and the parameter
mu gives the trade-off between the two. Now my question is: why not start
directly from this criterion (which I personally find more intuitive) instead
of the constrained optimization formulation (whose added value is not obvious
to me)?

2. The experiments are extremely limited, being run only on two 2D toy datasets 
and without any comparison to other popular auto-encoder algorithms. I also
feel they are not enough to give additional intuition on the algorithm's
behavior. Here are some examples of topics which could have been investigated:
- How does the output change with mu?
- How does the output change with different kernels used in the entropy 
  estimations?
- How does the algorithm behave on real data?
- How does the algorithm behave as dimension increases? (when the data do not 
  lie on a low-dimensional manifold, local kernel methods tend to fail)
- How does the algorithm behave compared to the typical auto-encoders mentioned 
  in the introduction? (the goal would not necessarily be to show it works
  better, but to gain understanding of the differences between algorithms)

Overall, an interesting idea, but one that would deserve a more in-depth 
treatment (note that novelty seems limited, since the starting point is a
criterion already proposed for manifold learning, and the kernel-based entropy
estimation comes from a previous paper by the same authors).

A few more small points:
- There is a non negligible amount of typos, it could use a proofread pass.
- Notation inconsistencies (or not well explained): D instead of d in intro
  (not defined by the way), using both hat{x} and 	ilde{x} to denote the
  reconstructed data (and hat{x} is also used in the intro to denote the noisy
  input in the denoising auto-encoder), multiple P's in eq. 1, not clear if we
  are working in a continuous (eq.1)  or discrete (eq. 3) space, h not defined in
  1st paragraph of 2.1.
- The manifold learning algorithm described in Section 1 is not very clear. Eq.1
  seems to be only part of the cost, and it is not clear if it is minimized
  or maximized. Unfortunately I did not have time to read the corresponding
  reference, but I feel like it may deserver a more thorough description, given
  that it seems to be key to the algorithm presented here.
- Greek letters cannot be seen on some PDF readers (like an iPad), although it
  works under Windows.