Critical initialisation in continuous approximations of binary neural networks
Authors: George Stamatescu
Federica Gerace
Carlo Lucibello
Ian Fuss
Langford White
Venue:  (2020)

Abstract: The training of stochastic neural network models with binary ($\pm1$) weights and activations via continuous surrogate networks is investigated. We derive new surrogates using a novel derivation based on writing the stochastic neural network as a Markov chain. This derivation also encompasses existing variants of the surrogates presented in the literature. Following this, we theoretically study the surrogates at initialisation. We derive, using mean field theory, a set of scalar equations describing how input signals propagate through the randomly initialised networks. The equations reveal whether so-called critical initialisations exist for each surrogate network, where the network can be trained to arbitrary depth. Moreover, we predict theoretically and confirm numerically, that common weight initialisation schemes used in standard continuous networks, when applied to the mean values of the stochastic binary weights, yield poor training performance. This study shows that, contrary to common intuition, the means of the stochastic binary weights should be initialised close to $\pm 1$, for deeper networks to be trainable.

Published as a conference paper at ICLR 2020
CRITICAL INITIALISATION IN CONTINUOUS APPROXI- MATIONS OF BINARY NEURAL NETWORKS
George Stamateseu, lan Fuss and Langford B. White
Federica Gerace
School Electrical and Electronic Engineering University of Adelaide
Institut de Physique Théorique
CNRS & CEA & Université Paris-Saclay
Adelaide, Australia
Saclay, France federicagerace @gmmall com
{george e st amaatescc @ @gma Sommall.o com [lang. white, ian. fuss ade wide.edu....
Carlo Lucibello
Bocconi Institute for DataScience and Analytics Bocconi University
Milan, Italy
carlo. hww.bo.Lcccbellleembboooo...
ABSTRACT
The training of stochastic neural network models with binary (+11) weights and activations via continuous surrogate networks is investigated. We derive rogates using novel derivation based on writing the stochastic neural network as Markov chain. This derivation also encompasses existing variants of the surro- gates presented in the literature. Following this, we theoretically study the surro- gates initialisation. We derive, using mean field theory, a set of scalar equations describing how input signals propagate through the randomly initialised networks. The equations reveal whether so-called critical initialisations exist for each surro- gate network, where the network can be trained to arbitrary depth. Moreover, we predict theoretically and confirm numerically, that common weight initialisa- tion schemes used in standard continuous networks, when applied to the mean values of the stochastic binary weights, yield poor training performance. This study shows that, contrary common intuition, the means of the stochastic binary weights should be initialised close to +1, for deeper networks to trainable.
INTRODUCTION
The problem of learning with low-precision neural networks has seen renewed interest in recent years, in part due the deployment of neural networks low-power devices. Currently, deep neural networks are trained and deployed on GPUs, without the memory or power constraints of such devices. Binary neural networks are promising solution to these problems. If one is interested in addressing memory usage, the precision of the weights of the network should be reduced, with the binary case being the most extreme. order to address power consumption, networks with both binary weights and neurons can deliver significant gains in processing speed, even making it feasible to run the neural networks on CPUs Rastegari et al. (2016). Of course, introducing discrete variables creates challenges for optimisation, since the networks are not continuous and differentiable.
Recent work has opted train binary neural networks directly via backpropagation on differen- tiable surrogate network, thus leveraging automatic differentiation libraries and GPUs. A key to this approach is in defining an appropriate differentiable surrogate network as an approximation to the discrete model. A principled approach is consider binary stochastic variables and this stochasticity to "smooth out"" the non-differentiable network. This includes the cases when (i) only weights, and (ii) both weights and neurons stochastic and binary.
In this work we study two classes of surrogates, both of which make use of the Gaussian central limit theorem (CLT)) the receptive fields of each neuron. In either case, the surrogates written

---

Published as a conference paper at ICLR 2020
differentiable functions of the continuous means of stochastic binary weights, but with more complicated expressions than for standard continuous networks.
One approximation, based on analytic integration, yiclds class of deterministic surrogates (ouuryy ict al. (2014). The other approximation is based on the local reparameterisation trick (LRT) Kingma & WWlling (201)) which yields class of stochastic surrogates Shayer al. (2017). Previous works have relied on heuristics to deal with binary neurons [Peters & Welling (2018), or not backpropa- gated gradients correctly. Moreover, none of these works considered the question of initialisation, potentially limiting performance.
The seminal papers of Saxe et al. (2013) Poole et al. 60166) Schoenholz al. (016)) used a mean field formalism to explain the empirically well known impact of initialization on the dynamics of learning in standard networks. From one perspective the formalism studies how signals propagate forward and backward in wide, random neural networks, by measuring how the variance and corre- lation of input signals evolve from layer to layer, knowing the distributions of the weights and biases of the network. By studying these moments the authors in [Schoenholz et al.] (2016) were able to explain how heuristic initialization schemes avoid the "vanishing and exploding gradients problem" Glorot & Bengio (2010), establishing that for neural networks of arbirary depth to be trainable they must be initialised at "criticality", which corresponds to initial correlation being preserved to any depth.
The paper makes three contributions. The first contribution is the presentation of new algorithms, with new derivation able to encompass both surrogates, and all choices of stochastic binary weights, or neurons. The derivation is based on representing the stochastic neural network as a Markov chain, a simplifying and useful development. As an example, using this representation we are easily able to extend the LRT to the case of stochastic binary neurons, which is new. This was not possible in Shayer et al. (2017). who only considered stochastic binary weights. As a second example, the deterministic surrogate of (fuudy et al. (2014) is easily derived, without the need for Bayesian message passing arguments. Moreover, unlike Souiryy et al. (2014) we correctly backprop- agate through variance terms, as we discuss.
The second contribution is the theoretical analysis of both classes of surrogate at initialisation, through the prism of signal propagation theory Poole et al. (2016), [Schoenholz et al. (2016). This analysis achieved through novel derivations of the dynamic mean field equations, which hinges on the use of self-averaging arguments Mezard et al. (1987). The results of the theoretical study, which are supported by numerical simulations and experiment, establish that for a surrogate of arbitrary depth to be trainable, it must be randomly initialised at "criticalityy. In practical terms, critical- ity corresponds to using initialisations that avoid the "vanishing and exploding gradients problem" Glorot & Bengio (010). We establish the following key results:
For networks with stochastic binary weights and neurons, the deterministic surrogate can achieve criticality, while the LRTT cannot.
For networks with stochastic binary weights and continuous neurons, the LRT surrogate can achieve criticality (no deterministic surrogate exists for this case)
In both cases, the critical initialisation corresponds to randomly initialising the means of the binary weights close to +1, counter intuitive result.
A third contribution is the consideration of the signal propagation properties of random binary net- works, in the context of training a differentiable surrogate network. We derive these results, which are partially known, and in order inform our discussion of the experiments.
This paper provides insights into the dynamics and training of the class of binary neural network models. To date, the initialisation of any binary neural network algorithm has not been studied, although the effect of quantization levels has been explored through this perspective BBlumenfeld ct al. (2019). Currently, the most popular surrogates are based on the so-called Setraight-Throupee estimator Bengio et al. (2013), which relies on heuristic definitions of derivatives in order to define a gradient. However, this surrogate typically requires the use of batch normalization, and other heuristics. The contributions in this paper may help shed light what is holding back the more principled algorithms, by suggesting practical advice on how to initialise, and what to expect during training.
2

---

Published as a conference paper at ICLR 2020
Paper outline: In section 2 we present the binary neural network algorithms considered. In sub- section 2.T we define binary neural networks and subsection 2.2 their stochastic counterparts. In subsection 2.3 we use these definitions to present new and existing surrogates in coherent frame- work, using the Markov chain representation of a neural network to derive variants of both the deterministic surrogate, and the LRT-hased surrogates. We derive the LRT for the case of stochastic binary weights, and both LRT and deterministic surrogates for the case of stochastic binary weights and neurons. In section 3 we derive the signal propagation equations for both the deterministic and stochastic LRT surrogates. This includes deriving the explicit depth scales for trainability, and solv- ing the equations to find the critical initialisations for each surrogate, if they exist. In section | we present the numerical simulations of wide random networks, to validate the mean field description, and experimental results to test the trainability claims. In section 5 we summarize the key results, and provide a discussion of the insights they provide.
2
BINARY NEURAL NETWORK ALGORIHTMM
2.1 CONTINUOUS NEURAL NETWORKS AND BINARY NEURAL NETWORKS
A neural network model is typically defined deterministic non-linear function. We consider
fully connected feedforward model, which is composed of N N weight matrices W' and bias vectors be in each layer l E (I,..., L.,, with elements WiJ E R and by e R. Given input vector 2. E R,No the network is defined terms of the following recursion,
x' o
he 1
W 1 + de
(1)
VN-
where the pointwise non-linearity is, for example, o-(-) = max(0,)) .). We refer to the input to a neuron, such as h², the pre-activation field.
deterministic binary neural network simply has weights W E [+1} and (²(-) sign(.). (.), and otherwise the same propagation equations. Of course, this is not differentiable, thus instead consider stochastic binary variables in order to smooth out the non-differentiable network. Ideally, the product of training surrogate of stochastic binary network is deterministic (or stochastic) binary network that is able to generalise from its training set.
2.2 STOCHASTIC BINARY NEURAL NETWORKS
In stochastic binary neural networks we denote the matrices as S with all weights s E {=1} be- ing independently sampled binary variables with probability is controlled by the mean M₁ = ES Neuron activation in this model are also binary random variables. due to pre-activation stochasticity and to inherent noise. We consider parameterised neurons such that the mean activation conditioned on the pre-activation is given by some function taking values [-1, 1]. i.e. E[x bh]] @(h)), for example <b-) tanh(-). We write the propagation rules for the stochastic network as follows:
s p(:; h 1 -
+6;
x² pl ((h)))
VNE-I
Notice that the distribution of xf factorizes when conditioning on 1 The form of the neuron's mean function o(-) depends on the underlying noise model. We can express binary random variable E {+1} with x p(x: 8) via its latent variable formulation x sign( aL). In this form is referred to as "natural" parameter, and the term L is a latent random noise, whose cumulative distribution function (-)) determines the form of the non-linearity since Φ(-) = (or(-) In general the form of Φf-) will impact on the surrogates' performance, including within and beyond the mean field description presented here. However, result from the analysis Section 3 is that choosing
deterministic binary neuron, ie. the sign(.) function, or stochastic binary neuron, produces the same signal propagation equations, up to a scaling constant.
We denote random variables with bold font. Also, following physics' jargon, we refer to binary +1 vari- ables Ising spins or just spins.
3

---

Published as a conference paper at ICLR 2020
2.3 DERIVATIONS OF NEW AND EXISTING SURROGATE NETWORKS
The idea behind several recent papers Soudry et al. (2014), Baldassi et al. (0118), Shayer et al. (2017). Peters & Welling (2018) is adapt the mean of the binary stochastic weights, with the stochastic model essentially used to "smooth out" the discrete variables and arrive at a differentiable function, open to the application of continuous optimisation techniques. We now derive both the de- terministic surrogate and LRT-based surrogates, in common framework. We consider a supervised classification task, with training set D for our surrogate model via
p D/ 3a)n== with Up the label. we define a loss function
C(M,b) 1 Σ log Es P(U, uu S.x,
For a given input Fn and realization of weights, neuron activations and biases in all layers, denoted by (S, x. b), the stochastic neural network produces a probability distribution over the classes. Expectations over weights and activations are given by the mean values, ES M and [[x*Ih'] <hh). This objective can be recognised method could be described as Type maximum likelihood, or empirical Bayes.
(minus) marginal likelihood, thus this
The starting point for our derivations comes from rewriting the expectation equation marginalization of a Markov chain, with layers indexes corresponding to time indices.
3 as the
Markov chain representation of stochastic neural network:
Es,x p(yul Iy. S.b,),,
plu xx") II p(
S.x: Ox:x==z,
8=1
Σ plunlst 81,x1-1
x 1-1 (p()))
Ex(x
--2 8h- )
Ep(x+lan S (sS) (4)
S 8L-1,xh-2
S
where in the second line we dropped from the notation p(S Md) the dependence on M for brevity. Therefore, for a stochastic network the forward pass consists in the propagation of the joint distribu- tion of layer activations, p(x (r,), according to the Markov chain. We drop the explicit dependence on the initial input xy from now on.
In what follows we will denote with o(h²) the average value of x according to p(x'). The first step to obtaining differentiable surrogate is to introduce continuous random variables. We take the limit of large layer width and appeal the central limit theorem to model the field h Gaussian, with mean h and covariance matrix E.
Assumption 1: (CLT for stochastic binary networks) In the large N limit, under the Lyapunov central limit theorem, the field h
VN-I S'xf-1 + bf converges to Gaussian random variable
with mean 1 -N-I E, Mr (h + of and covariance matrix Ef with diagonal Σ X E, (M (h 1))
While this assumption holds true for large enough networks, due to S and x independency, the Assumption 2 below, is stronger and tipically holds only at initialization.
Assumption 2: (correlations are zero) We assume the independence of the pre-activation field he between any two dimensions. Specifically, we assume the covariance Σ = Cov(h' h²) ) to be well approximated by ( with MF denoting the mean field (factorized) assumption, where
E (E) de NE-1 Σ1-(M,0000
(5)
This assumption approximately holds assuming the neurons in each layer are not strongly corre- lated. In the first layer this is certainly true, since the input neurons are not random variable In subsequent layers, since the fields hā and hf share stochastic neurons from the previous layer, this cannot be assumed to be true. We expect this correlation to not play significant role, since
In this case the variance is actually
N E, (M))))
4

---

Published as a conference paper at ICLR 2020
the weights act to decorrelate the fields, and the neurons are independently sampled. However, the choice of surrogate influences the level of dependence. The sampling procedure used within the lo- cal reparametrization trick reduces correlations since variables are sampled, while the deterministic surrogate entirely discards them.
We obtain either surrogate model by successively approximating the marginal distributions, p(x²) I dhh' p(x Ih') = p(x'), ), starting from the first layer. We can do this by either (i) marginalising over the Gaussian field using analytic integration, or (ii) sampling from the Gaussian. After this, use the approximation p(x;) to form the Gaussian approximation for the next layer, and so on.
Deterministic surrogate: We perform the analytic integration based on the analytic form of p(x++"|n Ih²) (x(h)). with a-)) sigmoidal function. In the case that (-)) is the Gaussian CDF, obtain p(x{) exactly[b the Gaussian integral of the Gaussian cumulative distribution function,
p(x)) - f dh (xx ((: EMrR)) = Φ
M
(6)
(1 Σ 1/2 MF
Since we start from the first layer, all random variables are marginalised out, and thus hς has no dependence random hī-1 via the neuron means (hh²) as in Assumption 1. Instead, we have dependence on means x² = E
E [x* hh] = En ((h)). Thus it is convenient to define the mean
under p(x{) as phh oh) = f dh o' (h(h) hh: o²). In the case that (-) is the Gaussian CDF, then vp (-) is the error function. Finally, the forward pass can be expressed as
xx =
he = (1 th
he
M
This is a more general formulation than that in Soudry al. (2014), which considered sign activa- tions, which we obtain in the appendices as a special case. Furthermore, in all implementations we
backpropagate through the variance terms ΣMF. which were ignored in the previous work of Soudry et al. (2014). Note that the derivation here is simpler well, not requiring complicated Bayesian message passing arguments, and approximations therein.
LRT surrogate: The basic idea here is rewrite the incoming Gaussian field h (W,, Σ) as h μ + VE where ~ (N, I). Thus expectations over h can be written as expectations over and approximated by sampling. The resulting network is thus differentiable, albeit not deterministic. The forward propagation equations for this surrogate are
1 VN
h
M x +b + + (
x² = of (h²).
(8)
The local reparameterisation trick (LRT) Kingma & Welling (0133) has been previously used to obtain differentiable surrogates for binary networks. The authors of Shayer et al. (2017) consid- ered only the case of stochastic binary weights, since they did not write the network as Markov chain. Peters Welling (2018) considered stochastic binary weights and neurons, but relied on other approximations to deal with the neurons, having not used the Markov chain representation.
The result of each approximation, applied successively from layer to layer by either propagating means and variances or by, produces differentiable function of the parameters M It is then possible to perform gradient descent with respect to the .M and b. Ideally, at the end of training obtain binary network that attains good performance. This network could be stochastic net- work, where we sample all weights and neurons, or a deterministic binary network. deterministic network might be chosen taking the most likely weights, therefore setting W sign(M?)), ), and replacing the stochastic neurons with sign(-) activations.
SIGNAL PROPAGATION THEORY FOR CONTINUOUS SURROGATES
Since all the surrogates still retain the basic neural network structure of layerwise processing, cru- cially applying backpropagation for optimisation, it is reasonable to expect that surrogates are likely
In the Appendices we show that other sigmoidal (-) can be approximated by a Gaussian CDF.
5

---

Published as a conference paper at ICLR 2020
to inherit similar "training problems" standard neural networks. In this section apply this formalism to the surrogates considered, given random initialisation of the means My and biases bf. We able to solve for the conditions of critical initialisation for each surrogate, which essentially allow signal to propagate forwards, and gradients to propagate backwards, without the effects such neuron saturation. The critical initialisation for the surrogates, the key results of the paper, provided in Claims and 3.
3.1 FORWARD SIGNAL PROPAGATION FOR STANDARD CONTINUOUS NETWORKS
We first recount the formalism developed in Poole et al. 2016). Assume the weights of a standard continuous network are initialised with W (U, a w. biases D ~ (0, of), and input signal za has zero mean Ex and variance E[z x Gha and with a denoting a particular input pattern. As before, the signal propagates via Equation Donuatiooofron from layer layer.
No E,(h from par- M M aris- ing from two different inputs xa and 2g with given covariance Lab- The mean field approxima- tion used here replaces each element in the pre-activation field h by Gaussian random variable whose moments are matched. Assuming also independence within a layer; Eh S Lad5y and E 130 = Sijo one can derive recurrence relations from layer to layer,
We are interested in computing, from layer to layer, the variance Lha = ticular input x: and also the covariance between the pre-activations Lno No E.
Lua di |DzO (V lua goalz) + of ( o
v2 e the standard Gaussian measure. The recursion for the covariance is given by with Dz
lub = 2 / (22))((())))())) + of = [ ) + of
(10)
where U. V Qaa 21. u, V
96 (C + v (cf and we identify co as the corre-
lation in layer f. The other important quantity is the slope of the correlation recursion equation or mapping from layer to layer, denoted X. which is given by:
det
= o" f DDa Dz2 Dzz (o
=
de--1
(11)
We denote X at the fixed point c* as X1. As discussed Poole al. (0016), when X1 1, correlations can propagate to arbitrary depth.
Definition 1: Critical initialisations are the points (of; ow) corresponding to X1 1.
Furthermore, X1 is equivalent to the mean square singular value of the Jacobian matrix for a sin- a
gle layer J4j カ.ー as explained in Poole et al. (2016). Therefore controlling X1 will prevent the gradients from either vanishing or growing exponentially with depth. We thus define critical initialisations as follows. This definition also holds for the surrogates which now study.
3.2 SIGNAL PROPAGATION THEORY FOR DETERMINISTIC SURROGATES
For the deterministic surrogate model we assume at initialization that the binary weight means M, are drawn independently and identically from a distribution P(M),, with mean zero and variance of the means given by o2. For instance, a valid distribution could be clipped Gaussiar or another stochastic binary variable, for example P(M) = 8 + am) ) + 1 am), whose variance oi The biases at initialization are distributed as i ~ AN(0 o
We show in Appendix B that the stochastic and deterministic binary neuron cases reduce to the same signal propagation equations, up to scaling constants. In light of this, we consider the deterministic
*That is, sample from Gaussian then pass the sample through function bounded the interval [-1, 1].

---

Published as a conference paper at ICLR 2020
sign(-) neuron case, since equation for the field is slightly simpler:
hf Σ, My ( + VN--I by
(12)
V Σ, (M3
which we can be read from the Eq. As in the continuous case we are interested in computing the variance aua N₂ (haa) and covariance Eh 1/: hi Sig via recursive formulae. The key
to the derivation is recognising that the denominator v Eheraa is self-averaging quantity Mezard et al. (987).. This means it concentrates in probability to its expected value for large N. Therefore safely replace it with its expectation. Following this self-averaging argument, can take expectations more readily as shown in the appendices. We find the variance recursion to be
t Exp² (1) + oo
Tan
(13)
Re (h)
Based on this expression, and assuming laa = qbb. the correlation recursion can be written as + gaaa ot, (h 1-1 (p +
Cab
(14)
gba
1 o
The slope of the correlation mapping from layer to layer, when the normalized length of each input is its fixed point dae
9bs 9" (mm: σb), denoted as X, is given by:
Oct
1 1+ --" om
/ Dzy (ua) (ua)p'(us)
x = aet11
= 1 1 1
(15)
where Ua and are defined exactly in the continuous case. Refer to the appendices for full details of the derivation.
3.2.1 CRITICAL INITIALISATION DETERMINISTIC SURROGATE
The condition for critical initialisation is XI since this determines the stability of the correlation map fixed point e* = 1. Note that for the deterministic surrogate this is always a fixed point. We can solve for the hyper-parameter (of, o that satisfy this condition, using the dynamical equations of the network.
Claim 1:
The points (ot, am) corresponding to critical initialisation are given by o
1/B[(('((VTT))) + 2 z)) and finding of that satisfies
(1
L o + (o
E [[(((((V
We solve for o numerically, as shown in Figure for different neuron noise models and hence coneaiiresss yo(-). We find that the critical initialisation for any of these design choices is close to the point (a of) (1,0). However, it is not just the singleton point, as for example in Hayou et al. (2019) for the ReLu case for standard networks. We plot the solutions in the Appendix.
This can be established by rearranging Equations 13 and 15
3.2.2 ASYMPTOTIC EXPANSIONS AND DEPTH SCALES
The depth scales, as derived in Schoenholz et al. (2016) provide a quantitative indicator to the number of layers correlations will survive for, and thus how trainable network is. Similar depth scales can be derived for these deterministic surrogates. Asymptotically in network depth e, we expect that lq - q*1 exp(- É,, and Ic/, -c1 exp(- E. the depth scales over which the variance and correlations of signals may propagate. We are most interested in the correlation depth scale, since it relates to X. The derivation is identical to that of Schoenholz et al. (0166). One can expand the correlation che = c* + é and assuming ofa = q", possible to write
- where the terms & and & define
1+ o 0b
#+1 e
(2sp( 0(()))
(16)

---

Published as a conference paper at ICLR 2020
The depth scale E-t are given by the log ratio log c+1
1 + qº om
/ (zup
= log
log x
(17)
1+
We plot this depth scale in Figure 2 We derive the variance depth scale in the appendices, since it is different to the standard continuous case, but not of prime practical importance.
3.3 SIGNAL PROPAGATION THEORY FOR LOCAL REPARAMETERIZATION TRICK SURROGATES
From Equation 8 the pre-activation field for the perturbed surrogate with both stochastic binary weights and neurons is given by,
M1a
E Mijo(h)aaa) VN
1
+b b2+ é. VN( - (1
1
(18)
where we recall that (0, 1). The non-linearity ((-) can of course be derived from any valid bi- nary stochastic neuron model. Appealing to the same self-averaging arguments used in the previous section, we find the variance map be
Le = (o +o +1 (1 1+ of
(19)
Interestingly, we see that the variance map does not depend on the variance of the means of the binary weights. This is not immediately obvious from the pre-activation field definition. In the covariance map we do not have such a simplification since the perturbation 6iaa is uncorrelated between inputs and b. Thus the correlation map is given by
of Bo(h 1-1 ja
)o(h(aa)) of
cgb
(20)
1 1+03
3.4 CRITICAL INITIALISATION: LRT SURROGATES
There is critical initialisation for the local reparameterissation trick based surrogate, for a network with binary weights and neurons.
Claim 2:
Proof: The conditions for critical initialisation are that c* 1 to be fixed point and Xi 1. No such fixed point exists. We have fixed point e* = if and only if o 1/E (1)_a))))))) Note that o < 1. For any (0z) which is the mean of the stochastic binary neuron, the expectation E[0² (=)] 1. 1. For example, consider ((2) = tanh(kz) for any finite kappa.
We also considered the LRT surrogate with continuous (lanh(-)) neurons and stochastic binary weights. The derivations are very similar to the previous case, we show in the appendix. The variance and correlation maps are given by
o Ed(h )o(h to E6 j, +
Le = Ed² of
d
(21)
This leads to the following result,
Claim 3: The critical initialisation for the LRT surrogate, for I the case of continuous Lanh(-) neurons and stochastic binary weights is the singleton (oī, o = (0, 1).
if and only if 0 1, by inspection. In turn, the critical initialisation condition X1 holds if (((('(1))
Proof: From the correlation map we have fixed point
o 1. Thus, find the critical initialisation, we need to find a value of 9aa = E (h (h1 + of that satisfies this final condition. In the case that Φ(-) tanh(-), then the function (o' (0 1-1
j.a 111) < 1, taking the value at the origin only, this requires Qea - 0. Thus we have the singleton (0 22) (0, 1) the solution.
8

---

Published as a conference paper at ICLR 2020
Dynamics c
3.
afe
3.0
0.000
::00
0.3
0.5
31
2
0.4
61
83
0.0
04
Figure 1: Dynamics of the variance and correlation maps, with simulations of network of width N 1000, realisations, for various hyperparameter settings: of E (0.2, 0.5, 0.99} (blue, green and red respectively). (a) variance evolution, (b) correlation evolution. (c) correlation mapping (cnn to Cout), with of = 0.001
4 NUMERICAL AND EXPERIMENTAL RESULTS
4.1 SIMULATIONS
We first verify that the theory accurately predicts the average behaviour of randomly initialised networks. We present simulations for the deterministic surrogate in Figure T We see that the average behaviour of random networks are well predicted by the mean field theory. Estimates of the variance and correlation are plotted, with dotted lines corresponding to empirical means and the shaded area corresponding one standard deviation. Theoretical predictions are given by solid lines, with strong agreement for even finite networks. Similar plots can be produced for the LRT surrogate. Appendix D we plot the depth scales as functions of om and db-
4.2 TRAINING PERFORMANCE FOR DIFFERENT MEAN INITIALISATION o
Here we experimentally test the predictions of the mean field theory by training networks to overfit a dataset in the supervised learning setting, having arbitrary depth and different initialisations. We consider first the performance of the deterministic and LRT surrogates, not their corresponding binary networks.
We use the MNIST dataset with reduced training set size (50%) and record the training performance (percentage of the training set correctly labeled) after 10 epochs of gradient descent over the training set, for various network depths I. < 70 and different mean variances o² E [o, 1). The optimizer used was SGD with Adam Kingma & Ba (2014) with a learning rate of 2 x 10 chosen after simple grid search, and batch size of 64. We see that the experimental results match the correlation depth scale derived, which are overlaid as dotted curves. A proportion of 36. was found to indicate the maximum attenuation in signal strength before trainability becomes difficult, similarly to previous works [Scheenollz et al. (2016).
reason we see the trainability not diverging in Figure ii that training time increases with depth, on top of requiring smaller learning rates for deeper networks, described Saxe al. (2013). The experiment here used the same number of epochs regardless of depth, meaning shallower networks actually had an advantage over deeper networks. Note that the theory does not specify for how many steps of training the effects of critical initialisation will persist. Therefore, the number of steps we trained the network for is an arbitrary choice, and thus the experiments validate the theory qualitative way. Results were similar for other optimizers. including SGD, SGD with momentum, and RMSprop. Note that these networks were trained without dropout, batchnorm or any other heuristics.
more
In Figure 2 we present the training performance for the deterministic surrogate and its stochastic bi- nary counterpart. The results for a deterministic binary network were similar to single Monte Carlo sample. Once again, we test our algorithms on the MNIST dataset and plot results after 5 epochs. We see that the performance of the stochastic network matches more closely the performance of the continuous surrogate as the number of samples increases, from N 5 to = 100 samples. We can report that the number of samples necessary to achieve better classification, at least for more shal- low networks, appears to depends on the number of training epochs. This is a sensible relationship, since during the course of training expect the means of the weights to polarise, moving closer

---

Published as a conference paper at ICLR 2020
the bounds =1. Likewise, expect that neurons, which initially have zero mean pre-activations, will also "saturate" during training, becoming either always "on" (+1) or "off" (-1). stochastic network being "closer" to deterministic would reauire fewer samples overall.
0.90
0.90
0.75
0.75
50
3E
wecen 25-
40
40
4
0.45
0.45
30
30
0.30
0.30
20
0.15
0.15
0.00
0.6
0.8
0.6
-55
deterministic surrogate (left) and LRT surrogate for stochastic binary weights and continuous neurons (right). The vertical axis represents network depth against the variance of the means of Both surrogates were trained with of approach criticality in both cases. Overlaid are curves proportional to the correlation depth scale E.. Bottom: Training performance of the deterministic surrogate and its binary counterparts after training the MNIST dataset for 5 epochs. Left: performance of the continuous surrogate. Centre: the performance of the stochastic binary network, averaged over Monte Carlo samples. Right: 100 Monte Carlo samples. The deterministic binary evaluation is similar single Monte Carlo sample, resembling the central figure.
Figure 2: Top: Training performance
Thus, as o +
DISCUSSION
This study of two classes of surrogate networks, and the derivation of their initialisation theories has yielded results practical significance. Based on results of Section in particular Claims 1-3, offer the following advice. If practitioner is interested in traiing networks with binary weights and neurons, one should use the deterministic surrogate, not the LRT surrogate, since the latter has critical initialisation. If practitioner is interested in binary weights only,the LRT in this case does have a critical initialisation (and is the only choice from amongst these two classes of surrogate). Furthermore, both networks are critically initialised when of 0 and by setting the means of the weights to #1.
seen that during training, when evaluating the stochastic binary counterparts concurrently with the surrogate, the performance of binary networks was worse than the continuous model, especially depth increases. We reported that the stochastic binary network, with more samples, outperformed the deterministic binary network, reasonable result since the objective optimised is the expectation over an ensemble of stochastic binary networks.
A study of random deterministic binary networks, included in the Appendices, and published re- cently Bllueenfeld et al.] (2019) a different problem, reveals unsurprisingly that binary networks are always in a chaotic phase. However binary network which is trained via some algorithm will of course have different signal propagation behaviour. makes sense that the closer one is to the early stages of the training process, the closer the signal propagation behaviour is to the randomly initialised case. We might expect that as training progresses the behaviour of the binary counter- parts approaches that of the trained surrogate. Any such difference would not be observed for heuristic surrogate as used in Courbariaux Bengio (2016) or Rastegari et al. (2016), which has no continuous forward propagation equations.
10

---

Published as a conference paper at ICLR 2020
REFERENCES
Carlo Baldassi, Federica Gerace, Hilbert J. Kappen, Carlo Lucibello, Luca Saglietti, Enzo
Tartaglione, and Riccardo Zecchina. Role of synaptic stochasticity in training low-precision ral networks. Phys. Rev. Lett. 120:268 03, Jun 2018. doi: 10.110s/hyyReevLet.. 120.262 103..
URL httptttss//.kk.aaat.t apsinkaapp... httpp//...r.....o...../..///errr
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation, 2013.
Yaniv Blumenfeld, Dar Gilboa, and Daniel Soudry. mean field theory of quantized deep networks:
The quanizztionn-eepth trade-off, 2019.
Matthieu Courbariaux and Yoshua Bengio. Binarynet: Training deep neural networks with weights and activations constrained to +1 or -1. CoRR, abs/1602.02830,, 2016. URL http:://axii
org/ hog/abs/1602 02830
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Yee Whye Teh and Mike Titterington (eds.), Proceedings of the Thirteenth Interna- lional Conference on Artificial Intelligence and Statistics, volume of Proceedings of Machine Learning Research, pp. 249-256, Chia Laguna Resort, Sardinia, Italy, 13-15 May 2010. PMLR. URL hatp:://proceedingg .owiinggmmiii mir wttp://psesee/////e/////
html
Soufiane Hayou, Arnaud Doucet, and Judith Rousseau. On the impact of the activation function on deep neural networks training. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.). Pro- ceedings of the 36th International Conference on Machine Learning. volume 97 of Proceedings of Machine Learning Research, pp.
Long Beach, California, USA, 09-10 15 Jun 2019. ngs. ml r...... pressesss//.... wtwphyyullhhaallo//e/aa/ 9a htm
PMLR. URI. htt.htttpp/pppoceeeeeo
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.6980,, 2014. URL
6980
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. CoRR, abs/1312.6114, 2013.
Marc Mezard, Giorgio Parisi, and Miguel Virasoro. Spin Glass Theory and Beyond, volume 9, 01 1987. doi: (/066//.2211677
Jorn W. T. Peters and Max Welling. Probabilistic binary neural networks. CoRR, abs/ wbss/889/ htw//1809.0000 889603668,
2018. URL (RRL:ttt://aaeee
1809. C3368
Ben Poole, Subhaneil Lahiri, Maithra Raghu, Jascha Sohl-Dickstein, and Surya Ganguli. Exponen- tial expressivity in deep neural networks through transient chaos. In D. D. Lee, M. Sugiyama, U. V. Luxburg, Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Sys- lems 29, pp. 3360-3368. Curran Associates, Inc., 2016.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.), Computer Vision - ECCV 2016, pp. 525-542, Cham, 2016. Springer International Publishing. ISBN 978-3-319-44493-0.
Andrew M. Saxe, James L. McClelland, and Surya Ganguli. Exact solutions to the nonlin- ear dynamics of learning in deep linear neural networks. CoRR, abs/1312.6120, 2013. URL http:://axxie org abs
6120
Samuel S. Schoenholz, Justin Gilmer, Surya Ganguli, and Jascha Sohl-Dickstein. Deep information propagation. CoRR, abs/161 1.01 2016. URI.nttp://arrsivv.rrr//bbb 611 01232
Oran Shayer, Dan Levi, and Ethan Fetaya. Learning discrete weights using the local reparameteri- zation trick. CoRR, abs/1710.07739, 2017.
Daniel Soudry, Itay Hubara, and Ron Meir. Expectation backpropagation: Parameter-free training of multilayer neural networks with continuous or discrete weights. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger (eds.), Advances in Neural Information Pro- cessing Systems 27, pp. 963-971. Curran Associates, Inc., 2014.
11

---

Published as a conference paper at ICLR 2020
A DERIVATION OF DETERMINISTIC SURROGATE NETWORKS
A.1 INTEGRATING OVER STOCHASTIC OR DETERMINISTIC BINARY NEURONS
The form of each neuron's probability distribution depends on the underlying noise model. We can express a stochastic binary random variable S E {+1} with S ~ p(S; 6) via its latent variable formulation,
S s0gn(0 (0 + a.)
(22)
In this form I is referred to as a "natural" parameter, from the statistics literature on exponential families. The term L is latent random noise, which determines the form of the probability distribu- tion. We also introduce a scaling to control the variance of the noise, so that as a - 0 the neuron becomes deterministic sign function. Letting a for simplicity, see that the probability of the binary variable taking a positive value is
p(S D / PL))L
(23)
where p(L) is the known probability density function for the noise L. The two common choices of noise models are Gaussian or logistic noise. The Gaussian of course has shifted and scaled erf(-) function its cumulative distribution. The logistic random variable has the classic "sigmoid" or logistic function as its CDF, (())
Ite-x.
Thus, the probability of the variable being positive a function of the CDF. the Gaussian case, the probability
this is (()) By symmetry, the probability of p(S -1) (--6). Thus, distribution for the binary random variable in general is the CDF of the noise L, and we write p(S) (S0)). In the logistic noise case, have p(S) (S0)
For the stochastic neurons, the natural parameter is the incoming field hy = S(,xx + bf. As- suming this is approximately Gaussian in the large layer width limit, can successively marginalise over the stochastic inputs to each neuron, calculating an approximation of each neuron's probability distribution, p(x{). This approximation is then used in the central limit theorem for the next layer, and so on.
For the case of neurons with latent Gaussian noise part of the binary random variable model, the integration over the pre-activation field (assumed to be Gaussian) is exact. Explicitly,
p(x))=
ΣΣ
p(x
S* (D((((")))
x--1
S
(x( /
WW (Ehrr)
h
x3) = p(x+)
(24)
2(5
where (-)) is the CDF of the Gaussian distribution. We have again EMr denoting the mean field approximation the covariance between the stochastic binary pre-activations. The Gaussian pectation of the Gaussian CDF is a known identity, which we state in more generality in the next section, where we also consider neurons with logistic noise.
This new approximate probability distribution p(x²) can then used as part of the Gaussian CLT applied at the next layer, since it determines the means of the neurons in the next layer,
Ex
2②p
(25)
If we follow these setps from layer to layer, we see that we are actually propagating approximate means for the neurons, combined non-ling non-linearly with the means of the weights. Given the approxi- mately analytically integrated loss function, it is possible to perform gradient descent with respect to the means and biases, M and if.
12

---

Published as a conference paper at ICLR 2020
In the case of deterministic sign() neurons we obtain partieularly simple expressions. In this case the "probability" of a neuron taking, for instance, positive is just Heaviside step function of the incoming field. Denoting the Heaviside with ( we have
p(x)) = ΣE x ".' ")(())))(((-)
/
(EMr)a)
h
~ ②
x
p(x))
(26)
(EMP)
We write the network forward equations for the case of deterministic binary neurons, since is a particularly elegant result. In general we have
मं <((n"), =²= EMph', R M +
(27)
where p(-) e-f(.) is the mean of the next layer of neurons, being scaled and shifted version of the neuron's 's noise model CDF. The constant is 17 v2 standard for the Gaussian CDF to error functin conversion.
A.2 EXACT AND APPROXIMATE GAUSSIAN INTEGRATION OF SIGMOIDAL FUNCTIONS
We now present the integration of stochastic neurons with logistic as well as Gaussian noise as part of their latent variable models. The logistic case is approximation built on the Gaussian case, motivated by approximating the logistic CDF with the Gaussian CDF. The reason we may be interested in using logistic CDFs, rather than just considering latent Gaussian noise models which integrate exactly, is not justified in any rigorous experimental way. Any such analysis would likely consider effect of the tails of the logistic versus the Gaussian distributions, where the logistic tails are much heavier than those of the Gaussian. One historic reason for considering the logistic function, we note, is the prevalence logistic-type functions (such as tanh(-) the neural network literature. The computational cost of evaluating either logistic eπrоr functions is similar, there no motivation from the efficiency side. Instead it seems a historic preference to have logistic type functions used with neural networks.
As we saw in the previous subsection, the integration over the analytic probability distribution for each neuron gave function which allows to calculate the means of the neurons in the next layer. Therefore, we directly calculate the expression for the means.
The Gaussian integral of the Gaussian CDF was used in the previous section derive the exact probability distribution for the stochastic binary neuron in the next layer. The result well known, and can be stated in generality follows,
(v 2
(aay
e
dy
+ V1 + 220
(28)
V2T02
We can integrate logistic noise binary neuron using this result well. The idea is to approximate the logistic noise with a suitably scaled Gaussian noise. However, since the overall network approx- imation results in propagating means from layer to layer, we equivalently need to approximate the tanh(-) with the with the crf. Specifically, if we have f(x; a) = tanh( (a) ). an approximation g(x; a) crf( VF .),, by requiring equality of derivatives the origin. order establish this, consider 2a
f'(0:a) (1 tanh tanh? (0/aa)
(29)
and
der((z;0)
2
2
(30)
dr ==
r=
V
13

---

Published as a conference paper at ICLR 2020
Equating these, gives o² 4a
thus
V=*
The approximate integral over the stochastic binary neuron mean is then
(x-2)
L f(y:a)
2c
VTT
20
dy
=
erf(
dy
(31)
v 2TT
-00
2a
VT
V
erf(
(32)
20r
mo²
with
(33)
2x²2
If we so desire, we can approximate this again with tanh(-) using the tanh(-) to erf(-) approxi- x
If as standard, mation reverse. The scale parameter of this tanh(-) will 02 20r." then
V x)) = tanh erf( 27
T2 47
(34)
B EQUIVALENCE OF DETERMINISTIC AND STOCHASTIC NEURONS FOR DETERMINISTIC SURROGATE
Assume stochastic neuron with some latent noise, as per the previous appendix, with mean i Ep(x,) (((--1). The field is given by
MC +
h
(35)
V2 1M
We that the expression for the variance of the field simplifies follows,
Mf
2 31+2 ( 2022 L = B(())
(36)
N(o2 E0² + +
(37)
21 No (h
0 E (h 1-1 + of
(38)
2 02 Edd (h) (h) ))
By similar steps, find that in the deterministic binary neuron case, would obtain the same expression, albeit with a different scaling constant. This easily seen by inspection the field term in the deterministic neuron case,
E, MG ) +b6
h V2 E111- (MI1-(M))) (02
(39)
which again was derived in the previous appendix.
C DERIVATION OF SIGNAL PROPAGATION EQUATIONS IN DETERMINISTIC SURROGATE NETWORKS
Here we present the derivations for the signal propagation in the continuous network models studied in the paper.
14

---

Published as a conference paper at ICLR 2020
C.I VARIANCE PROPAGATION
We first calculate the variance given signal:
Laa
N Σ (M),))= = E (M/,aa
(40)
Where for us:
E, m (11)/a) + b/
1/,0 =
(41)
V I (1 (mmm)) )² @² (hifaa))
and
mig N (0, b, ~ N (0,
(42)
E [E,, mbja (1 + by 2
E, mzgo (1177a1) +
(M /2] =E
Ni-1 E, (mly)² 2 0² (h/j/a²
E, (1 (mly) ) Φ² (11)22)
E, Eo² (1/jaa) + No
21 Eo² (1//aa) + NNi10
N111 1 1 E, )² 02 (M//2)
Ni-1 (1-oz Edd (hi
of
Ni-1
2 EΦ² (1/jae) + o
(43)
1 of EΦ² (1(-a))
Where, Ed² (1//20)) can be written explicitly, taking into account that h/ ~ N (0, (au):
(h/aa)
E 2 (h) = / 0 (h2aa = / dh V (1/aa))
(h/aa)
exp
2E
(1
(h
ff dh
=
j.a
V2Tdoa
exp
2
22 (h/aa)
(44)
We can now perform the following change of variable:
h1.
(45)
v Pea
Then:
(s/aa)
E [0 (N/,a)
Vdaa /
de dep,aepp
(Vetaztia)
V2
2
V2T / dzeep
22 o3
(v Voao=
(-2)
2
=DDzO0 (Vileez)
= 'zxo??
(46)
02 J Droz VVqa"' 2
+ +o
do E
(47)
1- 02 DrØ?
(V quaa
In the first layer, input neurons are not stochastic: they are samples drawn from the Gaussian distri- bution xº ~ N (0,
15

---

Published as a conference paper at ICLR 2020
C.1.1 CORRELATION PROPAGATION
To determine the correlation recursion start from its definition:
Lo.b
Cab
(48)
V
Taa Pbb
where Las represents the covariance of the pre-activations M1, and M related to two distinct input signals and therefore defined as:
Lab = N
h4 Mh, = [
(49)
Replacing the pre-activations with their expressions provided in eq. the self-averaging argument, we can then write:
(41) and taking advantage of
o
(1 Φ (11 ) + o
dab
(50)
Paa (1 o7,
2 (N )]) gib (1-0%)
022
hh-1 j.b
At this point, given that dea and dhe quite quickly approach the fixed point, we can conveniently as- sume Lea -os Moreover, exploiting eq. (47, we can finally write the expression for the correlation recursion:
o
(hl
(bb o3
+ of
Yoa
cgb
(51)
Pea
C.2 DERIVATION OF THE SLOPE OF THE CORRELATIONS AT THE FIXED POINT
To check the stability the fixed point, we need to compute the slope of the correlations mapping from layer to layer at the fixed point:
d
Xlg. =
dc-11 ab
1+ q.
0
④ (h(ta)) & (M)7T)) ln-
(52)
1 of de
1 +
d2
θ
D2,, (ua) (ub) lg-
q. of ab
de
where get rid of σr because independent from d Replacing the definition of u, and us provided in the continuous model, we can explicitly compute the derivative with respect e
1 1+ 9+
02
X
(A B), go 1 +00
(53)
where we have defined A and B
16

---

Published as a conference paper at ICLR 2020
/ Dasó VVeiial
20 20) ø' (v
c/b 2a + V (c)z))) 2b )) za
A Vq.
%bb
1-1
v
( DDiaDasb (Vehot 20 ø' (V
4bb
1-1
d-1
ab
ka + V (c2ol))
c
=b.
V
(c'T))
2
(54)
We can focus on B first. Integrating by parts over zb we get:
B= V B-VT.DDe.D DDDDDsD (Veta'zu ta Dz. ø' (Vat= (des za + V -1-5)) 20)) )) (55)
Then, integrating by parts over Za. the get:
B = V / DiaDiod (Veln' )o' of (Vits" cob Za + V (d 26 zat
(a) -9 / DinDioo (V Za
(56)
o' (V
J-1 9bb
1-1 20
+ V 1 ) 2b ))
Cab
Replacing and B in eq. fixed point, namely:
(53). we then obtain the closest expression for the stability at the variance
1 + q+ 2 of mm
/ Dii,Dzs'' (wa) ø' (ub)
Xlg.
(57)
C.3 VARIANCE DEPTH SCALE
As pointed out in the main text, it should hold asymptotically that:
(-4+.1)
d gol exp
(58)
En
with Eg defining the variance depth scale. To compute it we can expand over small perturbations around the fixed point, namely:
/ q: c
0 D26? (V - )2)) + of
(59)
1 oZ f DzØ? (Va. + +dz)
Expanding the square root for small e', we can then write:
2f DrØd (VTiz + d' 2 + of
(60)
1 02, Dro²
(V + e'
We can now expand the activation function & around small perturbations and then computing the square getting rid of higher order terms in e', thus finally obtaining:
17

---

Published as a conference paper at ICLR 2020
+ qu
o DzØ (vqu2) o (vT+2)
Uno qo +
(61)
Vq- 1- o ʃ DrØ² (v
Comparing this expression with the one in eq.
(59. can then write:
1 + qu o / D2o (v Ø (vTTz)
(62)
V%
1 on Dr (v4+2)
Integrating by parts over 2, we then obtain:
1+1 (1+ on
Dee' (vTx2) Ø' (V + DD2Ø" (vTu2) Φ (v C.
(63)
1 o f DzØd (v4-2)
Given that it holds eq. given by:
47 and noticing that evaluated at the correlation fixed point c, = is
1 o (1 + q.) / Dz (Vanz))?
o
xlc
(64)
can finally get:
dt xla=1
2n (1 g.)
f [DZD"(VT))) (
d
(65)
1+
1 1+
Given that expect (58) to hold asymptotically, that is:
11+1
1+1 exp
(66)
En
finally obtain the variance depth scale:
5-1 log (1 + q-) log Xla.=+ 22 (1 qo)
/ "zed ( ((Tr)))
(67)
1
D SUPPLEMENTARY FIGURES
D.1 CRITICAL INITIALISATION SIMULATIONS: DETERMINISTIC SURROGATE CASE
We Figure that the of critical initialisations exist in the plane, for but of 10-20 all the corresponding mean variances a - m² > which is not possible.
D.2 DEPTH SCALES
We see in Figure 4 the depth scales for the deterministic surrogate. Note the divergence as one expects following the simulations in Figure
18

---

Published as a conference paper at ICLR 2020
erf() stoch.
1.6
erf() det. tank() atock.
1.4
1.3
1.2
1.1
0,00
Figure 3: Plots the valid critical initialisations for the deterministic surrogate model, for stochastic binary weights and stochastic or deterministic binary neurons. Presented are the critical initialisa- tions the (o )f),, for both the a) stochastic neuron case with ((=) = erf( 2). b) the determinis- tic sign neuron case with (02) = erf(} +), and (c) the logistic based stochastic neuron, with tanh()
approximation. We see all lines are above o² for all but small of <<> 1.
Varfance depth &
Comeeaaion death 5,
175
15.
D001
42 -0.5
125
0.000
<3 3000
100
75
20
5.
30
2.5
0.
Figure 4: Depth scales as o2 is varied. (a) The depth scale controlling the variance propagation of signal (b) The depth scale controlling correlation propagation of signals. Notice that the correlation depth scale &. only diverges as o² + 1, whereas for standard continuous networks, there are an infinite number of such points, corresponding to various combinations of the weight and bias variances.
19

---

Published as a conference paper at ICLR 2020
D.3 JACOBIAN MEAN SQUARED SINGULAR VALUE AND MEAN FIELD GRADIENT BACKPROPAGATION
An alternative perspective on critical initialisation,, to be contrasted with the forward signal prop- agation theory, is that we are simply attempting to control the mean squared singular value of the input-output Jacobain matrix of the entire network, which we can decompose into the product of single layer Jacobian matrices. In standard networks, the single layer Jacobian mean squared singu- lar value equal to the derivative of the correlation mapping X as established in Poole et al. (2016). For the Gaussian model studied here this is not true, and corrections must be made to calculate the true mean squared singular value. This can be seen by observing the terms arising from denominator of the pre-activation field,
ah J
KL. Oh a (VE7)) = ('(N,,))
a
Ke (Eg) (M
+ (M,)²² VE
(68)
Since Ea a quantity that scales with the layer width Nz, it is clear that when we consider squared quantities, such the mean squared singular value, the second term, from the derivative of the denominator, will vanish in the large layer width limit. Thus the mean squared singular value the single layer Jacobian approaches X. We will proceed if is the exact quantity we are interested in controlling. The analysis involved determining whether the mean squared singular value is well approximated by X essentially takes through the mean field gradient backpropagation theory as described in Sccooenolzz et al. (2016). This idea provides complementary depth scales for gradient signals travelling backwards.
E REPARAMETERISATION TRICK SURROGATE
E. SIGNAL PROPAGATION EQUATIONS
We present, in slightly more detail, the signal propagation equations for the case of continuous neurons and stochastic binary weights yields the variance map,
Gaa (h)()) +
(69)
Thus, once again, the variance map does not depend on the variance of the means of the binary weights. The covariance map however does retain dependence on m.
du = Bo(h )o(h(,aa) of
(70)
with the same expression as before. The correlation map is given by
o Bo(h )o())))()))))))) oo
(71)
Ex² (1 + o
and we have the derivative of the correlation map given by
= oh, (n oo ('' (1)
(72)
E.2 DETERMINING THE CRITICAL INITIALISATION CONDITIONS
We recount the argument from the paper here. Since the mean variance of does not appear in the variance map, must once again consider different conditions for critical initialisation. Specifi- cally, from the correlation map we have a fixed point e = 1 if and only if
(73)
In turn, the condition X1 = 1 holds
Bl (h
(74)
Thus, to find the critical initialisation, we need to find a value qan = E0 + +o 2 that satisfies this final condition. In the case that (6-) tanh(-), then the function ( (0'(1==))) <<' taking the
20

---

Published as a conference paper at ICLR 2020
value 1 at the origin only, this requires 9aa - O. Thus the critical initialisation is the singleton point (o (m)) = (0,1) This is confirmed by experiment, as we reported the paper.
It is of course possible to investigate this perturbed surrogate for different noise models. For exam- plc, given different noise scaling K, as in the previous chapter, there will be corresponding of that satisfy the critical initialisation condition. We leave such investigation future work, given the case of binary weights and continuous neurons does not appear to be a particular interest over the binary neuron case.
F SIGNAL PROPAGATION OF BINARY NETWORKS
F.1 FORWARD SIGNAL PROPAGATION
In this neural network, it should be understood that all neurons simply sign(-) functions of their input, and all weights W E {=1} are randomly distributed according to
P(W = +1) = 0.5
(75)
(76)
thus maintaining zero mean.
The pre-activation field is given by
h
Σ Wt sign((f VN--1
(77)
So, the length map is:
Luu - 1 Dz(sign(
V dha 2)2) + of
(78)
=1
(79)
Interestingly, this the same value as for the perturbed Gaussian with stochastic binary weights and neurons.
The covariance evolves
Las Lao= -
D2 Dzy sign((a) sign(u)) + og
(80)
again have correlation map:
J D2g sign(u, (v.) (ug + of
Eg
(81)
c-1 1-1 1-1 V
90a
466
where as in the paper, ua = V qua
(-- 21 + 1 (cTT)))22)
21, ub
1bb
We can find this corrllation in closed form. First rewrite our integral with h, for joint density phha, ln),, and then rescale the he such that the variance so that dh,, VuaadUa
f dh sign(ha) (aggn(hassgnhuuuuuaa hn) = / dup sign( (v) sign(ve)p(va> 26)
(82)
(2P(v) O, U22 0) 2P( > 02 < 0))
(83)
where p(ua, 0b) is a joint with the same correlation Cas (which now equal to its covariance). and the capital P(v 02) corresponds to the (cumulative) distribution function. A standard result for standard bivariate normal distributions with correlation
sin (p) 1
cos cos cos-1(p)
P(v > 02 0)
+
P(u > 0, 0)
(84)
4
2T
2x
21

---

Published as a conference paper at ICLR 2020
So then have that
I
dhadhhgh(ha (oshs)hhhhhhh)) (ha, hv) = Vaagtb
1
+
sin
(c,o
cb
cos
(c
(85)
2
7
π
Thus the correlation map is:
cab ( + sin s (c
cos cos- (c 1)
+ +06
(86)
V Qiaa 8-1
466
sin (c of
(87)
2-1 6-1 Yua 4hb
V
Since, from before we have qaa
+ oi. we then obtain
2 sin
(c + of
abb
(88)
1 1+ of
Recall that sin (1) x. so have that is fixed point always.
dc but by first integrating over the @() sign() ac
We will now derive its slope, denoted as x non-linearities, and then taking the derivative.
Now we are in a place to take the derivative:
Oct
2
1
(89)
dc
π
V qaa 966
V1- (c
v (c
We see that the derivative x diverges at ces 1, meaning that there is critical initialisation for this system. This of course means that correlations will not propagate to arbitrary depth in deterministic binary networks, as one might have expected.
F.2 STOCHASTIC WEIGHTS AND NEURONS
We begin again with the variance map,
Luu = [1
(90)
where in this the field is given by
h4
Σ
W Sia
(91)
VN
where Tz--1 denotes stochastic binary neuron whose natural parameter is the pre-activation from the previous layer.
The expectation for the length map is defined in terms of nested conditional expectations, since wish to average over all random elements in the forward pass,
dua E2 F2 1 + o
(92)
= =1+0 of
(93)
Once again, this is the same value for the perturbed Gaussian with stochastic binary weights and neurons.
Similarly, the covariance map gives us,
d/u h4.. 1 Brnlh E Eng TA 1T to
(94)
= =Eo(h )o(h)) +
(95)
with phi(-) being the mean function, or a shifted and scaled version of the cumulative distribution function for the stochastic binary neurons, just in previous Chapters. This expression is equivalent to the perturbed surrogate for stochastic binary weights and neurons, with mean variance of x² 1. Following the arguments for that surrogate, critical initialisation exists.
22

---

Published as a conference paper at ICLR 2020
F.3 STOCHASTIC BINARY WEIGHTS AND CONTINUOUS NEURONS
In this case, as we show in the appendix, the resulting equations are
Laa = EB + oz
(96)
Lab E (()))))))))))))))))))))))))))))) of
(97)
which are, once again, the same as for the perturbed surrogate in this case, with o This means that this model does have critical initialisation, at the point (o of) (1,
F.4 CONTINUOUS WEIGHTS AND STOCHASTIC BINARY NEURONS
Similar arguments to the above show that the equations for this case are exactly equivalent to the perturbed surrogate model. This means that no critical initialisation exists in this case either.
G MISCELLANEOUS COMMENTS
G.1 REMARK: VALDITY THE CLT FOR THE FIRST LEVEL OF MEAN FIELD
A legitimate immediate concern with initialisations that send of stochastic weights Si are no longer stochastic, and that the variance of the Gaussian under the central limit theorem would no longer be correct. First recall the CLT's variance is given by Var(h") = (1- which the central limit variance the case of only stochastic binary neurons at initialisation. Therefore, the applicability of the CLT is invariant to the stochasticity of the weights. This is not so of course both neurons and weights are deterministic, for example if neurons are just tanh() functions.
may be that the binary
the means my + =1 then variance is equal in value to >; (1-23)
23


Reviews:
[{"review_id": "r1xtybOgir", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper provides an in-depth exploration of stochastic binary networks, continuous surrogates, and their training dynamics with some potentially actionable insights on how to initialize weights for best performance. This topic is relevant and the paper would have more impact if its structure, presentation  and formalism could be improved. Overall it lacks clarity in the presentation of the results, the assumptions made are not always clearly stated and the split between previous work and original derivations should be improved. \n\nIn particular in section 2.1, the author should state what exactly the mean-field approximation is and at which step it is required (e.g. independence is assumed to apply the CLT but this is not clearly stated). Section 3 should also clearly state the assumptions made. That section just follows the “background” section where different works treating different cases are mentioned and it is important to restate here which cases this paper specifically considers. Aside from making assumptions clearer, it would be helpful to highlight the specific contributions of the paper so we can easily see the distinctions between straightforward adaptations of previous work and new contributions.\n\nSpecific questions:\n\nIt might be worth double checking the equation between eq. (2) and eq. (3) , the boundary case (l=0) does not make sense to me, in particular what is S^0 ?.\n\nWhat does the term hat{p}(x^l) mean in the left hand side of eq.(3)? \n\nIn eq. (7) (8) why use the definition symbol := ?\n\nAt the beginning of section 3.1, please indicate what “matcal(M)” precisely refers to. Using the term P(mathcal(M) = M_ij) does not make much sense if the intent is to use a continuous distribution for the means. \n\nJust after eq. (9), please explain what Xi_{c*} means. \n\nSmall typo: Eq. (10) is introduced as “can be read from the vector equation 31”, what is eq. (31)?\n\nIn section 5.2, why reducing the training set size to 25% of MNIST?\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SkxNKaup9B", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors investigate the training dynamics of binary neural networks when using continuous surrogates for training. In particular, they study what properties the network should have at initialisation to best train. They do so via mean field approximations in the limit of very wide networks.\n\nThe authors provide concrete advice: the mean of the stochastic weights should be close to +/-1 at initialisation. Being able to give such advice to ML practitioners is of great value, but since this advice feels counter intuitive (naively, the mean of a binary +/-1 activation is typically the output of a tanh, and initialising this close to +/-1 means that gradient are going to be roughly 0, and can easily be exactly 0 in low precision computes). Right now, the justification of this initialisation is hard to find in the paper. This point would deserve a dedicated section, giving a summary of the argument, with references to more technical parts of the paper.\n\nThe presentation should also be improved as we can find the following issues:\n* Missing letters, repeated words or even missing figures (Fig 5 and 7 in the appendix).\n* Authors need to explain what is Edge of Chaos and give some reference (for example  C. G. Langton. Computation at the edge of chaos. Physica D, 42, 1990). This will make the paper more accessible to researchers less familiar with the theory but interested in its practical applications.\n* Please, better explain figures, for example Figure 1: need to explain better what this plot is. Why are there both dotted and solid lines? What are the shaded regions: is that some confidence interval? But which one, and how many experiments were used for those plots?", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "HkeufHm1cB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper addresses a very important and relevant topic of initialisation of weights of neural networks. It builds up on highly celebrated results of Poole, Schoenholz and others, using the language of mean field theory and an approach rooted in Dynamical Systems.\nWhat the authors propose is an extension of the approach to other settings. The paper is very scientific and math-heavy. A good practice in such cases is to adhere to a format of a scientific mathematical paper and organise the material using Theorema, Lemmata, Propositions and Corollaries , Definitions and Proofs. Such language and framework exists for a reason - to structure the material and make the paper readable. The paper as is a stream of equations and discussion making it very unclear what the point is.\nIn order for this paper to be suitable for publication the reviewer would like to strongly suggest:\n- Organise the material in a way that would make it clear what is claimed, what is proven etc.\n- Make more specific what the added value of the paper is.\nAlso - for the contribution of the paper to be less incremental it would be valuable to add more formality to the original results, for example - Gaussian approximation is claimed without any sort of verification of assumptions of any version of CLT or Law of Large Numbers.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors study neural networks with binary weights or activations, and the so-called "differentiable surrogates" used to train them.
They present an analysis that unifies previously proposed surrogates and they study critical initialization of weights to facilitate trainability.

The reviewers agree that the main topic of the paper is important (in particular initialization heuristics of neural networks), however they found the presentation of the content lacking in clarity as well as in clearly emphasizing the main contributions. 
The authors imporved the readability of the manuscript in the rebuttal.

This paper seems to be at acceptance threshold and 2 of 3 reviewers indicated low confidence.
Not being familiar with this line of work, I recommend acceptance following the average review score.
Decision: Accept (Poster)