Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The paper provides an in-depth exploration of stochastic binary networks, continuous surrogates, and their training dynamics with some potentially actionable insights on how to initialize weights for best performance. This topic is relevant and the paper would have more impact if its structure, presentation  and formalism could be improved. Overall it lacks clarity in the presentation of the results, the assumptions made are not always clearly stated and the split between previous work and original derivations should be improved. 

In particular in section 2.1, the author should state what exactly the mean-field approximation is and at which step it is required (e.g. independence is assumed to apply the CLT but this is not clearly stated). Section 3 should also clearly state the assumptions made. That section just follows the “background” section where different works treating different cases are mentioned and it is important to restate here which cases this paper specifically considers. Aside from making assumptions clearer, it would be helpful to highlight the specific contributions of the paper so we can easily see the distinctions between straightforward adaptations of previous work and new contributions.

Specific questions:

It might be worth double checking the equation between eq. (2) and eq. (3) , the boundary case (l=0) does not make sense to me, in particular what is S^0 ?.

What does the term hat{p}(x^l) mean in the left hand side of eq.(3)? 

In eq. (7) (8) why use the definition symbol := ?

At the beginning of section 3.1, please indicate what “matcal(M)” precisely refers to. Using the term P(mathcal(M) = M_ij) does not make much sense if the intent is to use a continuous distribution for the means. 

Just after eq. (9), please explain what Xi_{c*} means. 

Small typo: Eq. (10) is introduced as “can be read from the vector equation 31”, what is eq. (31)?

In section 5.2, why reducing the training set size to 25% of MNIST?

review_title: Official Blind Review #5

final_score: rating: 6: Weak Accept
review_content: In this paper, the authors investigate the training dynamics of binary neural networks when using continuous surrogates for training. In particular, they study what properties the network should have at initialisation to best train. They do so via mean field approximations in the limit of very wide networks.

The authors provide concrete advice: the mean of the stochastic weights should be close to +/-1 at initialisation. Being able to give such advice to ML practitioners is of great value, but since this advice feels counter intuitive (naively, the mean of a binary +/-1 activation is typically the output of a tanh, and initialising this close to +/-1 means that gradient are going to be roughly 0, and can easily be exactly 0 in low precision computes). Right now, the justification of this initialisation is hard to find in the paper. This point would deserve a dedicated section, giving a summary of the argument, with references to more technical parts of the paper.

The presentation should also be improved as we can find the following issues:
* Missing letters, repeated words or even missing figures (Fig 5 and 7 in the appendix).
* Authors need to explain what is Edge of Chaos and give some reference (for example  C. G. Langton. Computation at the edge of chaos. Physica D, 42, 1990). This will make the paper more accessible to researchers less familiar with the theory but interested in its practical applications.
* Please, better explain figures, for example Figure 1: need to explain better what this plot is. Why are there both dotted and solid lines? What are the shaded regions: is that some confidence interval? But which one, and how many experiments were used for those plots?
review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: The paper addresses a very important and relevant topic of initialisation of weights of neural networks. It builds up on highly celebrated results of Poole, Schoenholz and others, using the language of mean field theory and an approach rooted in Dynamical Systems.
What the authors propose is an extension of the approach to other settings. The paper is very scientific and math-heavy. A good practice in such cases is to adhere to a format of a scientific mathematical paper and organise the material using Theorema, Lemmata, Propositions and Corollaries , Definitions and Proofs. Such language and framework exists for a reason - to structure the material and make the paper readable. The paper as is a stream of equations and discussion making it very unclear what the point is.
In order for this paper to be suitable for publication the reviewer would like to strongly suggest:
- Organise the material in a way that would make it clear what is claimed, what is proven etc.
- Make more specific what the added value of the paper is.
Also - for the contribution of the paper to be less incremental it would be valuable to add more formality to the original results, for example - Gaussian approximation is claimed without any sort of verification of assumptions of any version of CLT or Law of Large Numbers.
review_title: Official Blind Review #2

Meta-review: The authors study neural networks with binary weights or activations, and the so-called "differentiable surrogates" used to train them.
They present an analysis that unifies previously proposed surrogates and they study critical initialization of weights to facilitate trainability.

The reviewers agree that the main topic of the paper is important (in particular initialization heuristics of neural networks), however they found the presentation of the content lacking in clarity as well as in clearly emphasizing the main contributions. 
The authors imporved the readability of the manuscript in the rebuttal.

This paper seems to be at acceptance threshold and 2 of 3 reviewers indicated low confidence.
Not being familiar with this line of work, I recommend acceptance following the average review score.
Decision: Accept (Poster)