Title: An Information-Theoretic Framework for Fast and Robust Unsupervised Learning via Neural Population Infomax
Authors: Wentao Huang
Kechen Zhang
Venue: ICLR 2017
Keywords: Unsupervised Learning
Theory
Deep learning

Abstract: A framework is presented for unsupervised learning of representations based on infomax principle for large-scale neural populations. We use an asymptotic approximation to the Shannon's mutual information for a large neural population to demonstrate that a good initial approximation to the global information-theoretic optimum can be obtained by a hierarchical infomax method. Starting from the initial solution, an efficient algorithm based on gradient descent of the final objective function is proposed to learn representations from the input datasets, and the method works for complete, overcomplete, and undercomplete bases. As confirmed by numerical experiments, our method is robust and highly efficient for extracting salient features from input datasets. Compared with the main existing methods, our algorithm has a distinct advantage in both the training speed and the robustness of unsupervised representation learning. Furthermore, the proposed method is easily extended to the supervised or unsupervised model for training deep structure networks.

TL;DR: We present a novel information-theoretic framework for fast and robust unsupervised Learning via information maximization for neural population coding.

Review discussion:
My main research interests are in computational neuroscience, information theory, machine learning and deep learning. For those of us who engage in AI related research, we all want to learn from the human brain how to process information to achieve intelligence.
For example, deep learning is now said to be brain-like, but also some researchers say that nothing to do with the neuroscience.
But one thing is for sure, and that is deep nets learning the brain processing information with hierarchical structure.
However, the current advances in neuroscience is of limited role on the inspiration for AI research.
How do we learn from the brain, learn what? We think we should learn the basic design principles for the information processing in our nervous system. What are the fundamental principles to guide the brain to design these complex structures and neural coding.
In fact, the principle on energy and information provides a fundamental constraint for the actual nervous systems. The efficient coding hypothesis proposed by Horace Barlow provides a good description of this principle.
But how to model this hypothesis, we will encounter great challenges. Because a direct calculation of Shannon's mutual information (MI) is generally a very difficult thing in many cases.
We first solve the problem of effective approximation for evaluating MI in the context of neural population coding, especially for high-dimensional inputs.
Then an fast and robust unsupervised learning algorithm is developed in this paper. With our methods, we have explained some interesting phenomena in neuroscience, which were previously unsolvable in other ways.
In machine learning and deep learning, this paper is a more fundamental part  that provides the basis for our other works. We have also got a lot of interesting results from other related works we have done or are doing.
For example, we use our method to train CNN, RNN and generative model, etc. We can unify supervised learning and unsupervised learning and can completely without the backpropagation algorithm for supervised learning of deep nets. We can also prove that the cost function of SVM (Hinge loss) is a special case of our neural population infomax. For the current deep nets, we need the big data for training, and in our framework, the small data we also can achieve good results.
The above is a brief introduction to the background of this paper. Since there is no similar work on this paper before, so there may be relatively more formulas and derivations and not very easy to follow.
Thanks!
This is an 18 page paper plus appendix which presents a mathematical derivation for infomax for an actual neural population with noise.  The original Bell & Sejnowski infomax framework only considered the no noise case.  Results are shown for natural image patches and the mnist dataset, which qualitatively resemble results obtained with other methods.
This seems like an interesting and potentially more general approach to unsupervised learning.
However the paper is quite long and it was difficult for me to follow all the twists and turns.
For example the introduction of the hierarchical model was confusing and it took several iterations to understand where this was going.  'Hierarchical' is probably not the right terminology here because it's not like a deep net hierarchy, it's just decomposing the tuning curve function into different parts.  I would recommend that the authors try to condense the paper so that the central message and important steps are conveyed in short order, and then put the more complete mathematical development into a supplementary document.
Also, the authors should look at the work of Karklin & Simoncelli 2011 which is highly related.  They also use an infomax framework for a noisy neural population to derive on and off cells in the retina, and they show the conditions under which orientation selectivity emerges.
Thanks so much for your suggestions!
In fact, the method of hierarchical infomax can be extended to a deep net hierarchy which is explained on page 5 of this paper. We have also used this method to train the deep nets. The work of Karklin & Simoncelli 2011 is really highly related, but they just solved a linear problem. I will try to condense this paper.
Thanks!
This paper proposes a hierarchical infomax method. My comments are as follows:
(1) First of all, this paper is 21 pages without appendix, and too long as a conference proceeding.
Therefore, it is not easy for readers to follow the paper. The authors should make this paper as compact as possible while maintaining the important message.
(2) One of the main contribution in this paper is to find a good initialization point by maximizing I(X;R).
However, it is unclear why maximizing I(X;\breve{Y}) is good for maximizing I(X;R) because Proposition 2.1 shows that I(X;\breve{Y}) is an “upper” bound of I(X;R) (When it is difficult to directly maximize a function, people often maximize some tractable “lower” bound of it).
Minor comments:
(1) If (2.11) is approximation of (2.8), “\approx” should be used.
(2) Why K_1 instead of N in Eq.(2.11)?
(3) In Eq.(2.12), H(X) should disappear?
(4) Can you divide Section 3 into subsections?
Thanks so much for your review!
(1) Since there is no similar work in the previous papers, so our paper is longer and there are relatively more formulas and derivations. We are now trying to condense this paper.
(2) Here notice that when noise variance \sigma^{2}\rightarrow 0, from Proposition 2.1. we can get I(X;Y)\simeq I(X;\bar{Y})=I(X;\breve{Y}), and I(X;R)=I(Y;R)\simeq I(\bar{Y};R). From (2.20) we know that maximizing I(X;R) is equivalent to maximizing I(Y;R) and I(Y;R)<=I(X;\breve{Y}) \simeq I(X;Y). We can see that as I(X;R) or I(Y;R) increases I(X;\breve{Y}) also increases.
While notice that there may be many transformation from X to Y (or \breve{Y}) to meet maximizing I(Y;R)(or maximizing I(X;R)) (see, for example, Section 2.2.1). A reasonable choice is this optimal transformation to meet maximizing I(X;Y) (or maximizing I(X;\breve{Y})), which also means that the minimum information loss in the first step transformation. We will explain these in detail in the revised version.
Minor comments:
(1) You are right. “\approx” should be used.
(2) Notice that N is very large, N >> K, and usually it can be clustered into K_1 classes, e.g. K_1 << N.
(3) In Eq.(2.12), I_G is equal to 0.5*<...> + H(X).
But here H(X) is a constant and does not affect the final optimization solution.
(4) It is a good suggestion. We will divide Section 3 into subsections in the revised version.
Thanks again!
Wentao Huang
This paper presents an information theoretic framework for unsupervised learning. The framework relies on infomax principle, whose goal is to maximize the mutual information between input and output. The authors propose a two-step algorithm for learning in this setting.
First, by leveraging an asymptotic approximation to the mutual information, the global objective is decoupled into two subgoals whose solutions can be expressed in closed form.
Next, these serve as the initial guess for the global solution, and are refined by the gradient descent algorithm.
While the story of the paper and the derivations seem sound, the clarity and presentation of the material could improve.
For example, instead of listing step by step derivation of each equation, it would be nice to first give a high-level presentation of the result and maybe explain briefly the derivation strategy. The very detailed aspects of derivations, which could obscure the underlying message of the result could perhaps be postponed to later sections or even moved to an appendix.
A few questions that the authors may want to clarify:
1. Page 4, last paragraph: "from above we know that maximizing I(X;R) will result in maximizing I(Y;R) and I(X,Y^U)".
While I see the former holds due to equality in 2.20, the latter is related via a bound in 2.21. Due to the possible gap between I(X;R) and I(X,Y^U), can your claim that maximizing of the former indeed maximizes the latter be true?
2. Paragraph above section 2.2.2: it is stated that, dropout used to prevent overfitting may in fact be regarded as an attempt to reduce the rank of the weight matrix. No further tip is provided why this should be the case. Could you elaborate on that?
3. At the end of page 9: "we will discuss how to get optimal solution of C for two specific cases". If I understand correctly, you actually are not guaranteed to get the optimal solution of C in either case, and the best you can guarantee is reaching a local optimum. This is due to the nonconvexity of the constraint 2.80 (quadratic equality). If optimality cannot be guaranteed, please correct the wording accordingly.
Thank you very much for review and suggestion! Your suggestion is very good and we will reorganize the article based on your suggestion and clarify your questions in the revised version.
Thanks again!
Wentao Huang
Although ultimately the questions should be taken care of in the final revision, could you provide some response here to those questions as well?
Sure. The following is the response to your questions:
(1) This question is similar to AnonReviewer3's question. From the Proposition 2.1 we can know that maximizing I(X;R) is equivalent to maximizing I(Y;R).
However, there may be many linear transformation from X to Y to meet maximizing I(Y;R) (or maximizing I(X;R)) (see, for example, Section 2.2.1), and we only need Y to satisfy certain conditions (see, for example, Eq. 2.60 in Section 2.2.2). A reasonable choice for the linear transformation is  to meet maximizing I(X;Y) (or maximizing I(X;\breve{Y})), which also means that the minimum information loss in the first step transformation. Ideally, this choice would satisfy maximizing I(X;Y) and maximizing I(Y;R) (or I(X;R)) at the same time.
On the other hand, if we do not choose the transformation to meet maximizing I(X;Y), obviously, this transformation will probably affect the maximum value of I(Y;R).
(2) Dropout can result in a sparse weights (low rank matrix) update. This means that the update is only concerned with more important parts, which is similar to the denoising process by the SVD (singular value decomposition) low rank approximation.
(3) Similar to question (3), if we want to maximize I(X;R), then we require that matrix C satisfies certain condition to maximize I(Y;R) (see Section 2.2 and Section 2.3).
However, actually we completely abandon the orthogonality constraint (Eq. 2.80) after the initial iteration t_0 epoches (see Section 2.3.1 and 2.3.2). The orthogonality constraint (Eq. 2.80) used by the initial iteration can accelerate the convergence rate.
Thanks!
Wentao Huang
We would like to thank the reviewers for their conscientious review and comments and for providing us with valuable feedback. We already updated the paper based on the reviewer comments.
Thanks!
Wentao
We have clarified some of the questions on paragraph 2 on page 4, on paragraph 2 on page 6 and on paragraph 2 on page 15, respectively. Thanks!

Meta-review:
role: PC
data: comment: The reviewers were not completely happy with the presentation, but it seems the theory is solid and interesting enough. I think ICLR needs more papers like this, which have convincing mathematical theory instead of merely relying on empirical results.

Decision: Accept (Poster)