Venue: ICLR 2017 conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper presents an information theoretic framework for unsupervised learning. The framework relies on infomax principle, whose goal is to maximize the mutual information between input and output. The authors propose a two-step algorithm for learning in this setting. First, by leveraging an asymptotic approximation to the mutual information, the global objective is decoupled into two subgoals whose solutions can be expressed in closed form. Next, these serve as the initial guess for the global solution, and are refined by the gradient descent algorithm.

While the story of the paper and the derivations seem sound, the clarity and presentation of the material could improve. For example, instead of listing step by step derivation of each equation, it would be nice to first give a high-level presentation of the result and maybe explain briefly the derivation strategy. The very detailed aspects of derivations, which could obscure the underlying message of the result could perhaps be postponed to later sections or even moved to an appendix.

A few questions that the authors may want to clarify:
1. Page 4, last paragraph: "from above we know that maximizing I(X;R) will result in maximizing I(Y;R) and I(X,Y^U)". While I see the former holds due to equality in 2.20, the latter is related via a bound in 2.21. Due to the possible gap between I(X;R) and I(X,Y^U), can your claim that maximizing of the former indeed maximizes the latter be true?
2. Paragraph above section 2.2.2: it is stated that, dropout used to prevent overfitting may in fact be regarded as an attempt to reduce the rank of the weight matrix. No further tip is provided why this should be the case. Could you elaborate on that?
3. At the end of page 9: "we will discuss how to get optimal solution of C for two specific cases". If I understand correctly, you actually are not guaranteed to get the optimal solution of C in either case, and the best you can guarantee is reaching a local optimum. This is due to the nonconvexity of the constraint 2.80 (quadratic equality). If optimality cannot be guaranteed, please correct the wording accordingly.

review_title: null

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes a hierarchical infomax method. My comments are as follows: 

(1) First of all, this paper is 21 pages without appendix, and too long as a conference proceeding. Therefore, it is not easy for readers to follow the paper. The authors should make this paper as compact as possible while maintaining the important message. 

(2) One of the main contribution in this paper is to find a good initialization point by maximizing I(X;R). However, it is unclear why maximizing I(X;\breve{Y}) is good for maximizing I(X;R) because Proposition 2.1 shows that I(X;\breve{Y}) is an “upper” bound of I(X;R) (When it is difficult to directly maximize a function, people often maximize some tractable “lower” bound of it).

Minor comments:
(1) If (2.11) is approximation of (2.8), “\approx” should be used. 

(2) Why K_1 instead of N in Eq.(2.11)?

(3) In Eq.(2.12), H(X) should disappear?

(4) Can you divide Section 3 into subsections?


review_title: null

final_score: rating: 7: Good paper, accept
review_content: This is an 18 page paper plus appendix which presents a mathematical derivation for infomax for an actual neural population with noise.  The original Bell & Sejnowski infomax framework only considered the no noise case.  Results are shown for natural image patches and the mnist dataset, which qualitatively resemble results obtained with other methods.

This seems like an interesting and potentially more general approach to unsupervised learning.  However the paper is quite long and it was difficult for me to follow all the twists and turns.  For example the introduction of the hierarchical model was confusing and it took several iterations to understand where this was going.  'Hierarchical' is probably not the right terminology here because it's not like a deep net hierarchy, it's just decomposing the tuning curve function into different parts.  I would recommend that the authors try to condense the paper so that the central message and important steps are conveyed in short order, and then put the more complete mathematical development into a supplementary document.

Also, the authors should look at the work of Karklin & Simoncelli 2011 which is highly related.  They also use an infomax framework for a noisy neural population to derive on and off cells in the retina, and they show the conditions under which orientation selectivity emerges.

review_title: Review of "information theoretic framework"

Meta-review: The reviewers were not completely happy with the presentation, but it seems the theory is solid and interesting enough. I think ICLR needs more papers like this, which have convincing mathematical theory instead of merely relying on empirical results.
Decision: Accept (Poster)