Proper Straight-Through Estimator: Breaking symmetry promotes convergence to true minimum
Authors: Shinya Gongyo
Kohta Ishikawa
Venue: ICLR 2022 Submitted (2022)
Keywords: quantization
binary network
low bit network
Straight through estimator
STE

Abstract: In the quantized network, its gradient shows either vanishing or diverging. The network thus cannot be learned by the standard back-propagation, so that an alternative approach called Straight Through Estimator (STE), which replaces the part of the gradient with a simple differentiable function, is used. While STE is known to work well for learning the quantized network empirically, it has not been established theoretically. A recent study by Yin et. al. (2019) has provided theoretical support for STE. However, its justification is still limited to the model in the one-hidden layer network with the binary activation where  Gaussian generates the input data, and the true labels are output from the teacher network with the same binary network architecture. In this paper, we discuss the effectiveness of STEs in more general situations without assuming the shape of the input distribution and the labels. By considering the scale symmetry of the network and specific properties of the STEs, we find that STE with clipped Relu is superior to STEs with identity function and vanilla Relu. The clipped Relu STE, which breaks the scale symmetry, may pick up one of the local minima degenerated in scales, while the identity STE and vanilla Relu STE, which keep the scale symmetry, may not pick it up. To confirm this observation, we further present an analysis of a simple misspecified model as an example. We find that all the stationary points are identical with the vanishing points of the cRelu STE gradient, while some of them are not identical with the vanishing points of the identity and Relu STE.

Under review as a conference paper at ICLR 2022
PROPER STRAIGHT THROUGH ESTIMATOR: BREAK- ING SYMMETRY PROMOTES CONVERGENCE TO TRUE MINIMUM
Anonymous authors Paper under double-blind review
ABSTRACT
In the quantized network, its gradient shows vanishing except for non- differentiable points. The network thus cannot be learned by the standard back- propagation, so that an alternative approach called Straight Through Estimator (STE), which replaces the part of the gradient with simple differentiable func- tion, is used. While STE is known to work well for learning the quantized network empirically. it has not been established theoretically. A recent study by Yin al.) (2019) has provided theoretical support for STE. However, its justification is still limited the model in the one-hidden layer network with the binary activation where Gaussian generates the input data, and the true labels are output from the teacher network with the same binary network architecture. In this paper, we dis- cuss the effectiveness of STEs in more general situations without assuming the shape of the input distribution and the labels. By considering the scale symmetry of the network and specific properties of the STEs, we find that STE with clipped Relu is superior to STES with identity function and vanilla Relu. The clipped Relu STE, which breaks the scale symmetry, may pick up one of the local minima degenerated in scales, while the identity STE and vanilla Relu STE, which keep the scale symmetry, may not pick it up. To confirm this observation, we further present an analysis of simple misspecified model as an example. We find that all the stationary points are identical with the vanishing points of the cRelu STE gradient, while some of them are not identical with the vanishing points of the identity and Relu STE. Finally we have numerically confirmed the observation for the mixture Gaussian model with various teacher network.
INTRODUCTION
Quantization of the weights and the activations a promising technique to save the memory and accelerate the inference speed in deep neural networks (DNNs) which have been getting wider and deeper in recent years. There are two main approaches the quantization for DNNs (Krishnamoor [hi (2018)): Post-training quantization (PTQ) and quantization-awaa training (QAT). In the PTQ. the pre-trained networks are simply quantized without re-training the model. This approach allows us to achieve the nearly floating point accuracy at 8-bits, while below 8-bits, this results in significant accuracy degradation. Although there have been recent attempts to alleviate the accuracy degrada- tion in PTQ (Banner al. (2018); Choukroun et al. (2019) Zhao et al. (019);; Kryzhanovskiy al. (hou Hubara et al. (2017): Rastegari et al. (2016)), usually leads to better accuracy.
(202 ) the QAT, where the quantized weights and activations are trained (e.g.. /houu et al. (2016);
The difficulty in QAT is that the weights and the activations are discretized, and intrinsically non- differentiable. If we take the derivatives forcibly, they either vanish or diverge. To avoid this prob- lem, we replace them with the derivatives of some differentiable function in the backward pass only, called the Straight-Through Estimator (STE) (Hinton et al. (2012); Bengio et al. a (2013)). Since the replacement leads to bias, it is not always possible to learn the network successfully. However this approach can be applied to the low bits below 8-bits with tolerable accuracy degradation (Zhou et al. (2016); Choi et al. (2018) Esser et al. 2019); Bhalgat et al. (2020))

---

Under review as a conference paper at ICLR 2022
Originally, STE was introduced as the replacement with the derivative of the identity function by Hinton et al. 2012). Later, the term "STE" has been extensively used as the replacement with vari- et al.] (2013) studied the replacement with the derivative of sigmoid function as well as the original STE, and Hubara et al. (207)) used the derivative of the identity function clipped in the region of [a:] S 1. In low-bits network, /Zoo et al. (2016) used the quantized gradient after the replacement with the identity function, and [Cho et al. (2018). 8). [Eseer
ous functions. In the binary Sectwork,BBeggeee
ct al. (2019). and Bhalgat et al. (2020) used the the derivative of the clipped Relu (cRelu) for leaning the step size.
1.1 MAIN CONTRIBUTION
In this paper, refer the proxy of the gradient the STE gradient Although STE has been much success in training quantized DNNs empirically, the theoretical justification is very limited to the specific situations. It also remains unclear what differentiable functions should be used for the STE gradient. Our aim is to shed light on a new factor to determine the functions.
Without assuming the shape of the input distribution and the loss function, we first discuss the properties of the three STES, identity STE, Relu STE, and cRelu STE in one-hidden-layer network with binary activation from the perspective of the intrinsic symmetry. We find that because the identity STE and Relu STE keep the scale symmetry, the STE gradients should be zero for any scale to converge to the true (local) minimum of the loss function, while in the case of the cRelu STE, which breaks the scale symmetry, even if its gradient is not zero at the minimum at some scale and becomes zero at other scale, the cRelu STE converges to the minimum. Therefore, the back- propagation using the cRelu STE is most likely to converge to the minimum of the loss function among three STEs, This result reveals differences between Relu STE and cRelu STE, which could not be found in the models by Yin et al. (2019) and and Long et al. (2021).
Recently, Kumin et al. (2020) discussed the symmetry embedded in non- -una--uanzzedd neural networks, and the effect of its breaking such as the discretization, the weight decay, and the stochasticity during training. We discuss the new effect of breaking symmetry by the proxy of the gradient in training the quantized neural networks.
Next, to confirm this observation, we have studied a similar model of the Gaussian input as discussed in YYn et al.) (2019) as example. We have employed misspecified model suitable for most practical situations. Unlike the previous study, the true labels are presumed to be generated by the non-quantized network architecture. Consequently, we find that ignoring the scale degeneracy for the weight in front of the activation, all the stationary points are identical with the vanishing points in the cRelu STE gradient, while some of them are not identical with the vanishing points in the identity and the Relu STE gradients. In particular, at the global minimum point, both identity and Relu STE gradients do not vanish. Finally, to confirm more general case, we have numerically studied the model of the mixture Gaussian input.
1.2 RELATED WORKS
Recently, there have been a few theoretical studies on the justification of the STEs. Shekhovtsov & Yanush (2020) derived STE by the linearization of their proposed estimator in a stochastic binary deep network where the noises are injected before the binary activation be a smooth network. Cheng et al.] (2019) argued that STE can be interpreted as projected Wasserstein gradient flow under certain conditions. Yin et al. (2019) that inspired our study examined which of the three STE's, identity STE, Relu STE, and cRelu STE, converges to the true minimum in one-hidden- layer network with binarized activation and Gaussian input data. They clarified that if the labels obtained from teacher network of the same architecture (Du et al. (2018)) all the three STEs show the non-negative correlation with the population gradient, but only the identity STE gradient does not become zero in local minimum. This indicates that the propppropagttion using either Relu STE or cRelu STE may convergence to the local minimum, while it is impossible to show the convergence to the local minimum using the identity STE. Furthermore, they showed that all the three STES gradients vanish at the global minimum, indicating that they may achieve the global minimum independent of the choice of the STE, if we start with an appropriate initial value.
It is called "coarse" gradient Yin al. 2019
2

---

Under review as a conference paper at ICLR 2022
Very recently, Long et al. (2021) have discussed the justification of class of STEs with certain monotonicity, which generalizes the Relu STE, in one-hidden-layer network for the hinge loss, claiming that the STE gradients vanish at the global minimum.
1.3 NOTATIONS
I| denotes the Euclidean norm vector. A capital letter bold denotes the matrix, e.g., Z, and its component is given by the letter with lower indices, e.g., Zij. A small letter in bold denotes the vector, e.g.. !. and its component is given by the letter with lower indices, e.g., Fj- Imx is the m identity matrix, and Im is the m-dimennional vector of all ones.
2 GENERAL DISCUSSION ON STES IN ONE-HIDDEN LAYER CNN WITH BINARY ACTIVATION
2.1 PRELIMINARIES
The one-hidden layer convolutional neural network with binary activation function is realized as y v(Z,v, v" 33'o(22 [oo(
Σ Uio
Zizw)),
(1)
where (-) is the step function,
x > 0. 1
(x)
0
0,
Z Rm is constructed by deividing the input data, e.g. an image, into m patches with size n, and w E IR and v E Rmm are the trainable weights, i.c kernels, in the first and second layers, respectively (Du et al. )) We assume that Z generated by continuous distribution P()).
Due to the property of the step function, the network is invariant under the scale transformation for w, w + kw
(k > 0).
Using the loss for each sample, (v(v,, Z):y") ) with y" being the labels generated by the true distribution q(y") Z)) we can express the population loss function as L(v, "o""") ) = Ez [[(((v, w, Z);
(3)
2.2 PACK-PROPNGATION AND STE GRADIENTS
The gradients of the loss for each sample w.r.t and ur be formally expressed as Oi
de Ol dy
Oy a
Σ 7rywz);
(4)
du By 0v:
0e Σ ae dr:
Σ
dl dy 2 Zig Zij-
=
(5)
Owj Oz, Ow,
where x; (E, W, and we used the property that the derivative of the step function is Dirac's delta function, do(=)/ 8(2). The gradient w.r.t v can utilized for learning the network, while it is impossible use the gradient w.r.t due to the presence of the delta function. Remarkably, both population gradients w.r.t v and w are smooth functions due to its average over the continuous distribution,
EE z
/
II 8'
dipy
PC Zes })
Of Doo
(Zil (Z.1l00)
(6)
Ov:
By
08
1
E
(V w. 2n1 = 0);
Ow,
1001
(1 w. 2nn)
/
II
dZwz
P( (((Zwg)
00 Oh
Eo' Zu
176
Dy
3

---

Under review as a conference paper at ICLR 2022
where we used the orthogonal transformation, Zi.- Dky. Zoy, zyy=Ea Zue, with Oi = wy///a/ and E; O O = 6 We can easily see that the gradient w.r.t. ur does not have the component in w -driirct,,,, E, B, (buy)
O. Its vanishing originally follows from the scale symmetry the network: Because the loss has the scale symmetry, its gradient in the w-direction is obviously zero.
aE
Instead of the population gradient, we use the STE in practice, which replaces the delta function with the derivative of some differentiable function USTE. to train the kernel w.
STE (v, w. 2) = Σ
de By Us SSTTTE
Σ Zngruy 3'
Zig.
(8)
We consider three types of psTe(r): (i) the identity type (STE(()) = x (ii) vanilla ReLU type
(STE(#) z00(2> and (iii) the cReLU type (sTe(z) (z)or(( x) with the upper clipping
value r. The surrogate back-propagation using the STE is described in Algorithm (Yin et al.
2019
Algorithm Surrogate back-propagation using STE for learning one-hidden-layer CNN.
Input: initialization v" E R.m w" for 1 = 0, 1,, 1,,.... do
E I2", learning rate 1).
+1 v² 77 EZ
at (v' w"; Z)] Do
2+1 = w² -nEZ g
STE (v", w/; Z)]
end for
The average of the STE gradients over the input data is also expressed as
Ez g STE (v. w; 2) = Σ diausse (Zalwl) Pi w, Za).
Note that in general, the STE gradients have the non-zero component in the w-direction due their partial collapse of the form of the gradient, E, B, [9] TE (v, w; Z)] # O.
is obvious from Eq. that the only difference between the STE gradients with the identity func- tion, Relu, and cRelu the region of the integration over 2...
Ez [o (v, w Z)] = > dž, F. Fiy w. Zn1).
(10)
RA
where A id, Relu, eRelu, and the integration reigion RA given Rid (-00++00).
RRelu = [0, +00); and R.Relu 00,
Following the scale invariance of Fiy(v, (v, w. 2n) for w, (ey(U, kw, Ze1) = Fiy(D, w. Zp11) (k 0), the STE gradients have the following interesting features:
1. The identity and Relu STE gradients have scale invariance for
Ez
9j id/Relu
(v, kw; Z)
E
g
id/Res Belu
(v, w; 2)
(1I)
2. The cRelu STE gradient does not have scale invariance for due the clipping effect,
Ez [9] cetlll (w, kw, r; 2)] A Ez [9] (RRell(. (v. w. Z)]
(12)
where we have explicitly shown the dependence of the upper clipping value
3. Instead, the scale transformation of w for the cRelu STE gradient can be compensated by that of r::
Ez [e]elll( (v. w. kr; Z) Ez [g] Rei (v. w/kkr; 2)]
(13)
4

---

Under review as a conference paper at ICLR 2022
If we take kr with fixed large enough to cover most of the distribution P(Z). the left-hand side is identical with the Relu STE gradient. Here we define the spread of the distribution p. When we redefine w / as on the right hand, the cRelu STE gradient for |wo| r/p is approximately identical with the Relu STE gradient,
Ez [of w. r; 2)] Ez [offchu(e (v. w; Z)] for |u|| <r/p.
(14)
This result is intuitively obvious: when w is small enough to keep most of the pre-activation values in the clipped range, the cRelu gradient and the Relu gradient are approximately identical.
The important issue is that there no guarantee that the weights obtained by the STE back- propagation are the ones at the (local) minimum of the loss function. In other words, at the stationary points, (17, w) = (U,, w2), defined as the vanishing points of the population gradient,
0e
de
Ez
(v
Us. w w.) 0,
Ez
(v
Us w w.s) 0.
(15)
(ww,
the STE gradient is not zero in general. Here we consider the local minimum point that the STE gradient does not vanish,
STE (v.: ws; 2) + O. Ez 99
(16)
Note that (v, w) = (Ue, kww) for any k > 0 is also the local minimum point due to the scale invariance of the loss. In the case of the Relufidentity STEs, their gradient has the same finite value for any scales due to the scale invariance shown in Eq. I..
Ez Oj
id/nal id/Relu (v., kww; 2) E z
9g
3d/Relu (v., ws; Z) + 0.
(17)
In the case of the cRclu STE, however, the value of its gradient changes as the scale changes due to the breaking of the scale symmetry shown in Eq. 12). If it crosses zero function of scale,
Ez [g ( ki r; 2) = 0 for some ko-
(18)
the back propagation using cRelu STE can converge at the local minimum the population gradient. Therefore, we conjecture that due to breaking scale symmetry, the back- propagation using cRelu STE is the most likely achieve the (local) minimum of the loss function in the three STEs. This also implies that the cRelu STE is less biased than the other. We confirm the conjecture analytically for the Gaussian input with Relu-type teacher network in See and numerically for the mixture Gaussian input with various teacher network in Sec.
done by using
Since the above discussion is focused only on symmetry, the conjecture can be generalized to net- works with more symmetries in a straightforward way. In that case, if we employ an STE that breaks all of those symmetries instead of cRelu STE, it is more likely to achieve the minimum.
The similar mechanism can be also found in physical phenomena. For instance, this is the case of the ferromagnet, which has rotational symmetry in energy (or Hamiltonian). the system, if we impose the magnetic field, which breaks the rotational symmetry, the spin states, corresponding to minimum points, become aligned with the same direction.
ONE-HIDDEN LAYER CNN WITH BINARY ACTIVATION: GAUSSIAN INPUT AND LABELS GENERATED BY NON-QUANTIZED RELU NETWORK
We consider the simple model similar to Yin et al. (019)) given by the Gaussian input with the variance ,2, Z ~ P(Z) = V2na²
mn
Ei. We use the squared loss function,
(yy(v, w. Z):0" (v", w", Z)) (y "²) )²)² Unlike Yin et al. (2019), the true labels are as- sumed to be generated by the non-quantized Relu network.
(0 w., 2) Σ of fRetu Σ Ennu
(19)
5

---

Under review as a conference paper at ICLR 2022
with the Relu activation defined [ rz))..
Note that the models are misspecified. i.c., the population loss cannnot be zero even at the global minimum. In fact, it is inferred from the accuracy degradation in most practical situations (Hubara ctt al. (217)) that the binary networks are considered as misspecified models. On the other hand, the true labels generated by the same binarized architecture network becomes a specified model.
used in Yin et al. (2019), the
As shown below, this misspecified model leads to more striking difference between three STEs than the specified model. Particularly, the behaviors of three STE gradients at the global minimum completely different from each other.
3.1 POPULATION LOSS, ITS GRADIENT AND STATIONARY POINTS
The population loss L(v, ww v", w") ) is derived in Appendix A.I as
L(v, w; v". w²) Ez [e(2s w. Z; w=))
δ 8|w") T 2
8 v' T
Imxm + +1m 1m (n) v
(cos + 1m 1m111) v²
02
120
1I + 1m 1m) 8²
(20)
4x
where oo is the angle between un and w. Note that the loss is scale invariant for w, and thus its derivative satisfies e
Sw = 0. The stationary points are given by
öL
0L
20 [0/w"|
1 COS 4
cos cos$pmmxm
+
1m
1 2²
(21)
dv
22T
m + 1
0L
0 + v- 0² 0 & 0 =I
(22)
dw
where we used (mmxm + 1m 1m) Imxm
m 1m 1
There are three stationary points as classified in Appendix A.2
1. middle saddle point given by v =
28 m 2
cos Cos&Immx mmxm + 1-cos 2
m+11 1m 1/m 00
1m 1(m+1)(0' +(v 1
which satisfies σo = O. This exists if and only
arccos
if (m + 1) (v²)² (x 1m
2. local minimum if (m 1) (v=)² 2 2v TT )²
otherwise saddle point given by
Ur 20 20'w0" V2a
(-Imxm + m
2 1
1
0² = π.
m
3. global minimum given by v v0
28 w
ip 0. At the global minimum, the
22n
population loss does not become zero, L(v
Vo
w 5² t0
(5 2)
3.2 STE GRADIENT
The STE gradient is given by
STE gj
Σ USHSTE
Σ Ziyway
Zis
X
Σ Upo
Σ
Zeywy
Σ
op FRelu
Σ Zoyre
(23)
"The definition of the stationary points is not mathematically rigorous in our paper. Even if points are non-differentiable, but the gradient gives zero the points in the one-side limit, we call them the stationary points.

---

Under review as a conference paper at ICLR 2022
The converging points by the back-propagation in Algorithm 1 are characterized by vanishing the gradients,
de (v, w; Z) = 0, Ez [g] ST (v, w; Z)] = 0.
Ez
(24)
Do;
We discuss the explicit form of the three STE gradients and the associated vanishing points. The detailed derivation of the STE gradients is shown in Appendix A.3
A.5
3.2.1 IDENTITY STE
The identity STE gradient is given by
Ez 9 idl
w
(v' v)
δ²
2 w" (v v*)
(25)
2t
(w
Combined with Eq. (21). we find that the equation in Eq. (24) has no solution. Therefore, we use the back-proaaattion by identity STE, can converge to any points.
3.2.2 RELU STE
The Relu STE gradient is expressed as
82 |w"
Ez [relul
(Imxm + 1m (mm) u]
10_ (-Imxm Im Am17)) v")
2v 2n
w
2x
[wo
I sin to w
(DTx²)
6² ( p w
(26)
|w
2T
Combined with Eq. (T). the gradients vanish at the following points:
1. If vz satisfies (m + () 2 2 (v-2 1m)² :0
m
arccos
(mm+1)(e> +
(v- 1m
v ū
28 [l
(cos mm + 1-c0s
m+1 1,, 17 2²
V2
2. p T..
2 |
(-Imxm
m+1 1m 17
/2
We find the solutions are identical with two stationary points shown in item 1 and in Sec 3.1 leading to the the saddle point or local minimum, while the global minimum point s 0, v vo cannot be obtained. Therefore, by using the back pro-ppropgtttion with Relu STE, it always converges
o if (m + 1) (u²) (v*21 1m)" and and does not converge any points, otherwise.
3.2.3 CLIPPED RELU STE
The cRelu STE is given by
W
1 -1-6 e (db(rin)))) o² (Imxm + 1m 17r
Ez [geretu]
2n
2m
10
6²
lw"
( )²) w ToT (-Imxm + 1m 1mr v²1
e
2T
w
8² Clu, sp)
w
+ S(w, (p)
w
cot
to w
(D"e*),
(27)
[w0
sin yp
wo
Tw0*
where C(ww),g) and S([w), sp) are given in Eq. C([w]], 0) Iw"
106
At
= 0, they are simplified as C(wwl, = T)
2n
werfc
<28
S([w). 0) = O.
Remarkably, the cRelu STE gradient is proportional to the Relu STE gradient in the case of o. u 0 or n
Ez [gerelul I yor (1 - {)(T))) Ez [gretul - 0 or somer.
(28)

---

Under review as a conference paper at ICLR 2022
As shown in Sec 3.2.2 all the vanishing points in the Relu STE can be found at v' 0 o n, leading to the saddle point or the local minimum, so that they are also found the vanishing points of cRelu STE. On the other hand, the behavior of cRelu STE gradient at yp 0 is not related that of Relu STE gradient. The cRelu STE gradient at o = 0 is written as
wn (1 (1-c (1-c-A(afur) )) v²
Ez [gerelul
(Imxm + 1m 1m) e
2
|20
20
((-1 + X) Ixxm 1m
12 m 8²
(29)
2m
where
π 8/ml ( V2T werfe
<Zalwl/
A = x(120) =
(30)
c [((T)))
This becomes zero if
v
20
[w'
[Imxm + 1m) ((-1+
1 A) Imxm + 1m 1 v²
2
26
2 - A
(-1+ Imxm +
+ 1 17
v"
(31)
22T
To be consistent with (q. at 40 O, only X allowed. The solution provides the global minimum shown in item in 3.1
In fact, monotonically increases from to π as T| increases, and thus the solution at X 2, corresponding to the global minimum point, can be obtained for any clipping value r and the variance 8² by changing the scale |w]. Consequently, get all the stationary points found in cRelu STE:
(0' 1m
1. If v² satisfies (m + 1) (0²)) 2 2 22 :0 arccos
( 10 1m
v 1 28 wl
(cs8 pl + 1-008 1m 12 w²
22
2. 3 T, ū
2d 1
-maxm +
1m 12
v22x
m++1
3. p 0, w
wi
= co, v v0
26/ v2x
where wo (or oo) is given by A(wo)
Note that while the global minimum point is degenerated in scales for w, i.c. if (w, v) = (wo. vo) gives the global minimum, (ur, v) = (kww, 10) for any >> 0 also gives the global minimum, the cRelu STE picks up the point at the scale determined by A = 2.
EXPERIMENTS
To confirm our conjecture in more general case, we have numerically studied the Gaussian mixture input with various mean values. As teacher networks, we have tested tanh-type and sin-type as well as Relu-type. For all the examined setups, cRelu STE behaves like the population gradient, while id/Relu STES show qualitatively different behaviors as described below. To calculate the population loss and STE gradients, we have generated the mixture Gaussian samples and have taken their average. The population gradient has been obtained by calculating the finite difference of the population loss. We have demonstrated the back propagation with learning rate = 0.01 given in Algorithm
Shown Fig. are the results of ten mixture Gaussian input with random mean values for each components of Z. We employed
20, n 25, and the tanh-type teacher network. We find that the population gradient and cRelu STE show similar results, while id/Relu STEs are completely different. This indicates cRelu STE is less biased than id/Relu STEs.
8

---

Under review as a conference paper at ICLR 2022
In the case of id/Relu STEs, at early steps up to around 500 step. |w| decreases around the magnitude of the update quantity, so that it begins to oscillate. Then it escapes the oscillation, and the loss function shows the convergence a point different from the local minimum achieved by population gradient, while |w] becomes larger and larger due to their scale invariance. Interestingly, the values of the loss function small compared to the one obtained by population gradient, which implies that id/Relu STEs avoid being trapped in the local solution due to their large bias. However, note that even that point, the magnitude of continues to grow and eventually become numerically unstable.
10
pop. grad
pop. grad id.
400
Relu STE
Relu STE
cRelu STE
cRelu STE
6
300
loss
4.
200
2-
100
500
1000
1500
2000
500
1000
1500
2000
step
step
120
4.
pop. grad id. STE
pop. grad
100
id. STE
Relu
Relu
80
cRelu STE
cRelu STE
Σ
40
20
0
500
1000
1500
0
500
1000
1500
2000
step
step
Figure 1: Numerical results of the back-propagation by population gradient and three STEs. We generate 10000 samples which follow ten mixture Gaussian input with random mean values. We employ the tanh-type teacher network.
5 SUMMARY
We have found that breaking symmetry embedded in the network by STEs enhances the possibility of convergence to the true (local) minimum of the loss function. We have demonstrated that an STE breaks the scale symmetry embedded in the one-hidden-layee network with binary activation, is more likely to achieve the local minimum than the one which keeps the symmetry. The discus- sion can be generalized to the network with more symmetries in a straightforward way. The more symmetries embedded in the network an STE breaks, the more likely it is to converge. To confirm the mechanism, we have studied three STEs, identity STE, Relu STE, and cRelu STE, in sim- ple misspecified model with Gaussian input. We have found that the back-propagation by the cRelu STE, which breaks the scale symmetry, can converge to the global minimum, while identy/Relu STE cannot. Finally we have numerically confirmed the mechanism for the mixture Gaussian model with various teacher network.
REFERENCES
Ron Banner, Yury Nahshan, Elad Hoffer, and Daniel Soudry. Post-training 4-bit quantization of convolution networks for rapid-deployment. arXiv preprint arXiv:1810.05723, 2018.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: 130..332, 2013.

---

Under review as a conference paper at ICLR 2022
Yash Bhalgat, Jinwon Lee, Markus Nagel, Tijmen Blankevoort, and Nojun Kwak. Lsq+: Improving low-bit quantization through learnable offsets and better initialization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattemn Recognition Workshops, pp. 696-697,
2020.
Pengyu Cheng. Chang Liu, Chunyuan Li, Dinghan Shen, Ricardo Henao, and Lawrence Carin. Straight-throug estimator as projected wasserstein gradient flow. arXiv preprint arXiv:1910002176, 2019.
Jungwook Choi, Zhuo Wang. Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srini- vasan, and Kailash Gopalakrishnan. Pact: Parameterized clipping activation for quantized neural networks. arXiv preprint 180.5v688500608 2018.
Yoni Choukroun, Eli Kravchik, Fan Yang. and Pavel Kisilev. Low-bit quantization of neural net- works for efficient inference. In ICCV Workshops, pp. 3009-3018, 2019.
Simon Du, Jason Lee, Yuandong Tian, Aarti Singh, and Barnabas Poczos, Gradient descent learns une-hidden-layer enn: Don't be afraid of spurious local minima. In International Conference on Machine Learning, pp. 1339-1348, PMLR, 2018.
Steven K Esser, Jeffrey L. McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmen- dra S Modha. Learned step size quantization. arXiv preprint arXiv:1902.08153: 2019.
Geoffrey Hinton, Nitsh Srivastava, and Kevin Swersky. Neural networks for machine learning. Coursera, video lectures, 264(1):2146-213 2012.
Itay Hubara, Matthieu Courbariaux. Daniel Soudry, Ran El- El-Yaniv, and Yoshua Bengio. Quan- tized neural networks: Training neural networks with low precision weights and activations. The Journal of Machine Learning Research, (((1):6669-688 2017.
Raghuraman Krishnamoorthi. Quantizing deep convolutional networks for efficient inference: A 2018
whitepaper. arXiv preprint arXiv:
Vladimir Kryzhanovskiy, Gleb Balitskiy, Nikolay Kozyrskiy, and Aleksandr Zuruev. Qpp: Real- time quantization parameter prediction for deep neural networks. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 10684-10692, 2021.
Daniel Kunin, Javier Ergastuu-Brenng Surya Ganguli, Daniel LK Yamins, and Hidenori Tanaka. Neural mechanics: Symmetry and broken conservation laws in deep learning dynamics. arXiv preprint arXiv:2012.04725 2020.
Ziang Long, Penghang Yin, and Jack Xin. Learning quantized neural by coarse gradient method for nonlinear classification. Research in the Mathematical Sciences, 8(3):1-19, 2021.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In European conference on computer vision, pp. 525-542. Springer, 2016.
Alexander Shekhovtsov and Viktor Yanush. Reintroducing straight-through estimators as principled methods for stochastic binary networks. arXiv preprint arXiv:2006.06880 2020.
Penghang Yin, Jiancheng Lyu, Shuai Zhang, Stanley Osher, Yingyong Qi, and Jack Xin. Under- standing straight-througee estimator in training activation quantized neural nets. arXiv preprint www.1903305662.. 2019.
Ritchie Zhao, Yuwei Hu, Jordan Dotzel, Chris De Sa, and Zhiru Zhang. Improving neural network quantization without retraining using outlier channel splitting. International conference on machine learning, pp. 7543-7552. PMLR. 2019.
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Train- ing low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint arXiv:1606.06s 160 2016.
10

---

Under review as a conference paper at ICLR 2022
A DERIVATION OF SEC
In the Appendix, we provide the results of the Gaussian with variance 1, (0, 1). For some function (2), the expectation value of the Gaussian with the generic variance 8 (0, shown in Sec is written as
/ II i.
Σs, 2
Ez-NNo.o;)
dZ dZizf(Z)e 202
2Tō²
Er, 23
II 12
d2 dZiz)(
2m
(32)
where we have changed the integral variables from Z to Z = Z //. Therefore we get the results with the variance & by only changing from f(Z) Z)
A.1 DERIVATION LOSS FUNCTION
The expectaion of the loss is divided into the following three terms:
L( un; v², w=) Bz [o" a (Zw) vz a (Zw)] - Ez [v= o (Zw)o* fnela (zw"))
+ = Ez 1 [ Zw )u" *T [Relu (Zw")
(33)
The calculation of the first term and the third term are given in "Proof Lemma 1" in Yin et al. 2019) and "Proof of Section 3" in Ref. Du et al. (2018), respectively:
Ez ToT a "Zow o (Zw)] + Ezz TT fRelu (ww")=" fRetu fRelu(Zw")
2
1 o [Imx + 1m 1m) v +
( TT ((T - 1)4mx + 1m1 v²
(34)
8
4m
To discuss the second term, we evaluate the following quantity:
Fij = Ez ()) Zww"), 1ay
ma
II
dZni kd
2 En. Zio
Σ
Zim Win!
Σ jm win
Σ
Z
Zim w my
v 2T
m!
(35)
i 1 / case:
/ [ dZndZzp
Fij
e (2)+2))))))))
Σ Zim Win' Σ
Zimmwi wine
Σ m
Zime wiy
2m
/ (II dZzdZzz
e E
(Zilto)) (oulto))
(2z1loo1) (
2m
1
dž,1
(291+ (Za)
(Z)llo*) (z11
2
(36)
2/2T
where we used the orthogonal transformation in the second line:
Ž,1 Σ Orme Zime'
2, Σo tmm Zime
(37)
m

---

Under review as a conference paper at ICLR 2022
with O im Wmm Wm///w/ and O1 wwml/Ww"
(ii) i = case:
F
/II dZ
Σ
Zim Wm
Eim Zim ui
a Σ
Zem w'n )
(38)
2T
Without loss of generality, we can choose
(w) On-1);
w = (1i, W2, On
(39)
with w" > O, so that this can be written as
2
/
2
F6 =
dZ dZi2e +22))
Σ
Zim Wm
Zar' a (Zi1)
mm
2
d7. =²²² (cos(pp - 0)) cos Ow" (cos 0)
2
T/2
1
d7
d07.e 1 cos Ow"
2TT
Jo
4--
1
w (1 + cOs yp)
(40)
2T2T
where tp is the angle between w" and w. Eqs. 36 and 40
are combined as
1
Pg
2v2T [[1 + cos p)6es + 2 2m
w0'(1 81z),
(41)
so that the second term in Eq.
is expressed
|w0| TT (cos pl +1 1m1m) _ v² 2/2x
Ez [ o (Z fRelu (Zw")]
(42)
Consequently, the loss function is summarized
1 TT (Imxm 1m
|w"l T (cos + 1m (m) o² 2/2
L(v, wi v", w²)
8
+
1w0 Po ((T mmxm + 1m 1m11)) 2²
(43)
4x
The population gradient w.r.t and w is thus given by
aL
+ 1mm ō|w0*
dv 4 (Imxm
coos mmxm + Im 1 v",
(44)
2
aL 1 ØL
8 8/w01
B.p
sin (uTx*) 0*) ep:
(45)
dw
w
Og
2/2 |
where without loss of generality, (-sin cos y, On- Note that ew Pur
can take = (cos yo sin yo, 0n-2), C.p
9L
0 is satisfied due to the scale symmetry for the loss.
A.2 CLASSIFICATION OF STATIONARY POINTS
Using Eq.
21 the loss at the three stationary points is rewritten as
12 D*T
mm + (- (cos [ccos 1)2
L
cos"
+
1 11
1 12 x
4m
m
+
10
xース .T ((T 1) Imxm 1m 1 12) v².
(46)
2x
12

---

Under review as a conference paper at ICLR 2022
Obviously, this shows that yp is the global minimum point,
|
L 2 L|p=0
(I - 2)0 v.
(47)
4m
At s n, this becomes
1=11 rT
(-4
L
[mxm +
m m++1 + 1m 1 2²
4m
< 12 x
((T- Imxm + m 1::) 8²
4T
w 122 v² xT
(T 2) Imxm + m7 + 1
1m 1 m
(48)
4m
(v- m..) )
At & & arccos
this becomes
1mm+1](r> (1')² (w "m"""mm
1w0 12
(mmT1m))
L b|p=0
v rT
I m
4x
(mm (m 1) (1²)²²
(xrT
2
( (m (m +
(0¹)²
2
+
1m 1T
2²
-(m+ -m=+))(v") + (v+TIm) 1
m
12 22
+
((x - 10- I Imxm + +1 v
4T
|00
(vtT 19 (u+)
+ (v=T' mm)
4m
(-mm + 1) (0+)² + (M+T mm)
ww'
r ((T - 1) mxx + 1m11)) ) v².
(49)
4m
Using these results, we obtain
Llp=
L LIp=
1w
1mm) (o+)²
w=112 (0*)²_
1w=12
(o
1mm
4T
-(m + 1) (0+)2 (v+T 1 1mm)
4x
4x
m
+
((m+1)(u)) - TT 1m))
2)2
w
(50)
4m
(m + 1) - (m+1)
Note that because (m + 1) (v=)² 2 2 (v 1m is satisfied to exist the stationary point at yp = 4 the loss at y is smaller than the one 4 5,
7 110=0,pm 0.p
To| plo=,,..==x
(51)
satisfied. When (m + 1) (u*)² = (v TT 1mm) point at 4p is merged into the point at 4p =
2 Llpmp = Ll because the middle stationary
This implies that y saddle point or the local maximum if (m + 1) (x*)² 2 2 1mm)
the global minimum, 8 r the local minimum, and yp ç the
To fully clarify whether the stationary points are local minimum, maximum or saddle point, we have calculated the hessian matrix,
H =
Twai 8 diZt
8
[mxm + 1, TT
[ sin you" 2/3r
(52)
ww" sin TT
2 x* cos
2/24
2/2
13

---

Under review as a conference paper at ICLR 2022
ii v p
For = (x. ()((((( E R R).
w'1
2 712
4 x (mxxm + 1m x
yf sin & (x v*)
22
1 xx 1mm²
2²+
1 1
lw" ly 0²
2
| 12 sin (v*)²
(53)
2t
2T
where we used o"u² 0. Therefore, depending on a and y, the Hessian can be either positive or negative, which means the stationary point is saddle point.
(ii) v 2x
2/20"| (-Imxm V2x
2 + m+1 1m mm)) v² 4
$ (mmx + 1m 1m1
0 Iu D.Th 2//2
H=
(54)
Note that v- 0²
2) uw"
1" +T
(-I mm + m-1 1 0² If (m 1) (1²)² 2 (1
T11TT100 1m) ? then
22
of'" 0, so that the matrix becomes a positive definite matrix: For any E R E R, this satisfies
100
4 x=' (Imxm + 1m 1m x
2 e 3² 0,
(55)
2/²T
which means the stationary point is the local minimum. On the other hand, if (m + (v²) < 2 (u*T1, 1m) 2, the matrix becomes either positive negative, depending and y, that this stationary point becomes a saddle point.
(iii) v 40
2/w="
2
1 (I (Imxm 1m 12)
0
H
T v²
(56)
2v2n
The Hessian becomes a positive definite matrix: For any x E R"
E R, this satisfies
10"
- x T 4
(mmxm + +1m 1 x +
0i e0 2 0.
(57)
2
A.3 DERIVATION OF IDENTITY STE GRADIENT
By substituting (z) = into Eq. 23
we obtain
Ez igl,
II kdl
dZus
e A Zi
Σ ZyeUs
2
x
Σ Uko
Σ Zkewi
Σ
Ezsus
(58)
We calculate each term as follows.
A.3.1 CALCULATION OF THE FIRST TERM
We calculate the first term as was done Lemma 9 (and also Lemma 11) in Yin et al. (2019)
To discuss the first term, evaluate
Fisk
Ez [Z (o Zww)
(59)
14

---

Under review as a conference paper at ICLR 2022
Using Fijh. the first term can be expressed
first term
Σ
Fid Ukk.
(60)
J.k
ji) * k case:
Figk = 0,
(61)
because of the odd symmetry for i-th and j-th components in the integrand.
(ii) j = k case:
Aid 153
dZni
e +
231
Zzio
Σ Zyroi
22
1
d2 j
- Z3
Σ
Semi 2ime
(331)
2m
m
1
d2 f
E + E, 23 2 (251)
2m
1
dž
1 0 l' Z,10 (2)2)
22
1
O
T
0
2m
2T
W4
(62)
/2x
where we used the orthogonal transformation in the second line:
Eo Zim!
(63)
with O'm/ = //w/
The first term is reduced to
first term
1 Wi v 2T 1
(uT v).
(64)
A.3.2 CALCULATION OF THE SECOND TERM
To discuss the second term, we evaluate
Gid Ez Zzj [Seet [Zw"))]
(65)
Using Gid the second term can be expressed as
sccond tcrm
Σ
Gid
Gikk uy
(66)
j.k
(i)_ * & case:
Gld
(67)
because of the odd symmetry.
15

---

Under review as a conference paper at ICLR 2022
(ii j k case:
G
1
d7
e 3 Ei 23 Zzi/nchu
Σ Soaus
v
2m
1
n
/
dZzi
c - 2 23 Σ 0 O Oommi-zn
fReel (Ir0) 12,2)
v
2m
1
"
/
dZzz)
6 - 23 O ",,""",1RRRR 2j1 (20012z)
2m
dZ
+2.00 T Zz Relu (l221
2m
1
f d2 +23 0 T Z3, 2 la (lwe (2,1
2m
w"
w0
O
0
2 w
(68)
where we have used the orthogonal transformation in the second line:
Žμ1 E''m Zim
(69)
m
with Oim' ///////// and also have used
Jo dxxee-11121 -01/2a=
/2TT
(70)
2
The second term is written as
second term 2 wi (v² (D"O*) 0*)
(71)
A.4 DERIVATION OF RELU STE GRADIENT
By substituting (x) (xx) into Eq.
23
we obtain
))
Ez [grelul:
/
II k,1
dZzi
e
-.. Zia
Σ Eiooyo
Σ Zareer
Σ Zuzuz ))
x
Σ
Uso
Σ
Zeewr
Σ
Us fRotu
(72)
k
We calculate each term as follows.
A.4.1 CALCULATION OF THE FIRST TERM
We calculate the first term as was done in Lemma9 (and also Lemmall) 1) in Yin et al. (2019). We
evaluate
Freiu Ez
Zzz (a (ww)l, (a (Z[))
(73)
kjk
Using Fiyk the first term can be expressed
first term
Σ
Fiyh Ukuk.
(74)
j.k
16

---

Under review as a conference paper at ICLR 2022
ji + l case:
Frolu
1
[
dIdZZIdZa
dZ dZzidZai
6 - 23 +3,
Zysor
Σ Zzrun
Σ Zurws
2
1
2n
dZ dZ2gdZaas
E1E, 23
Σ
O Oa 3.mm
(2n1) a (3ea)
2
m
1
džşıdžRie +(2)++2h)) 0 T 2,10
(2z1) ((21)
2m
011
2 2mr
(75)
2 w
where we used the orthogonal transformation in the second line:
Zz1 = Eoi Zim!
(76)
with Oim' = Wmmm //ww.
(ii k case:
We get the same expression given in 62.
/
Frelu
II
- E 23 Zzj00 Σ B
e
2m
Wii
(77)
2iT
w
The first term is written as
first
Wi
8T (Imxm 1m 17)
(78)
2/2x w
A.4.2 CALCULATION OF SECOND TERM
To discuss the second term, we evaluate
Grelu Ez Zis [Zo)), fRela (Z (Zw*)) )
(79)
Using Grelu, the second term can be expressed
second term =
Σ
Crehu 1,
(80)
J.k
(i.j + 1 case:
2n
/
Grelu
II dZzdZZzs
e 3 23 +Zi Ziio
Σ Zyrno
fRolu Σ Zuwwz
2
2n
[[I dZ dZzdddza
E.
Σ OiT 7
a (Zzz) fnctu (10")
V 2m
2
/ dZ d2gidžnie +(2 ) 2,10 (2,1) 100 12112 (2ni)
1w"
O05 T
2m
1
1w"
Wis
(81)
2TT
17

---

Under review as a conference paper at ICLR 2022
where we used the orthogonal transformation in the second line:
Zzo=
Imme Zim''
ZHI
Σ
O
bm Ziem
(82)
m
with im Wmm W/////// and Oim' = /100
( = k case:
m
/
G3
II dZ
Σ Zgour
SRelu
Σ Zzono
Σz
2Tx/
/
2
dZ
e - 23 Zzzo
Zyrwi
SRetu (w² Zz2)
II dZz1
e
- Ei 23 Zzi (6:1 + +22) o
Σ Zrrur 1=1
fRelu (w" 2z1)
2m
1=1
dZ 1
(7 cos f6 + 7 sin 08:2) a (Zw cos (0 - -(p) fRelu (w* 7 cos 0)
2x
dZdAZe 1
(Z cos 05 Z sin 05.2) (Zw cos (0 pp) w² Z cos tr (cos 0)
(83)
w
dz 3 -
322
f d0 (cos² 00,1 cos 1 sin 00
2T
J
-x/2+p
w"
sin 24
+ 2
( - x)
8:1 + 4
cos 2,p 8z2
4
w"
sin 4
[T p) 6,1 +
(cos spo1 + sin (pdr2)
(84)
2
where we have chosen
w" (w" On-1),
(wi, W2, On-2)
(85)
with wn > 0 and to is defined the angle between w and w. We also have used the formula,
d (1/2)a² =2
f d8 cos 0 sin 20
+
4
d0 cos 1 sisnn 0 = /
cos 20.
(86)
4
Note that
w
81
w=*
W.
cos 90 + sin 2
(87)
w
This reduces to In summary, the second term is written as second term
sin yp
Grelu
(N
up) 20; +
(88)
π
1 100
wi T
(-Imxxn
1m 1mm) v²1
2n
1r
+ (
(0)
w
sin yp
w.
(oTe0)
(89)
2T
18

---

Under review as a conference paper at ICLR 2022
A.5 DERIVATION OF CLIPPED RELU STE GRADIENT
By substituting pl (x) (x))((r z) into Eq 23
obtain
Ez 1errllul
1
dZai
e
$ En, Zee
2t
(fr-
(r- (r- Σ Zgrron)
Σ
Σ Z-rrixx
x
a
3
Σ Exaurr) )
x
Σ
Uka
Σ Zarwo Σ of falu
(90)
h
We calculate each term follows.
A.5.1 CALCULATION OF THE FIRST TERM
To discuss the first term, we evaluate
Ferehu Ez ( Oo(r Zw)), 1o (o
(91)
Using perelu the first term can be expressed
first term
Σ
porelu UzUk- Uk- Kik
(92)
j.k
(i..+ k case:
2n
Ferelu Fijk
II dZzidZza
-1 23.+ Zzj
2m
(1-
Σ
Zan
r-- Σ Zyrur
Σ <>
1
2n /
[[ dZzzddus
c -1 23,t2 23,+
Σ sm Zjms
Zim
2m
1,1
r (r-lw | 251 o (Zx1)
x
1
2
/
dZ d2zıddZiee 1(23++20)00 T2,10 (2)1) σ o0r((- (r-lw|2,)
tw/2,1) a (Zu1)
2m
r//
1
d7,ne- -3 0/ o27 2
2
J
1
OJ T
61-6- (1-6-2(1617
2T2
1
(1
(1-e-
=b(mm) )) w
wi
(93)
2T
where we used the orthogonal transformation in the second line:
Zz1
Σ Orm Zijm
(94)
with O Wmm /|/0
19

---

Under review as a conference paper at ICLR 2022
(ii) j = k case:
perelu
/ dZzi
e E,
23 Zzio Σ Bywrn
o {r- - Eyewr)
?jj
2TT
/
dZzzi
E Zjm'
(251) a (r- 1-/1 1521)
22n
isss'
mm'
1
ʃ dž 1/2 1,,00, T Z110 (Zr1) σ (rr 1/-12/ Z,11
V2T
(") dZ 2 12 T Z₁1
1
2nT
OiT' (1 - E [(m)))
V2T
1 (1- - (1-c x-
(95)
w
2
In summary, the first term is written as
first term
2v2x (1 - { (
W
oJ (1mxm + 1m1 v.
(96)
A.5.2 CALCULATION OF THE SECOND TERM
To discuss the second term, we evaluate
Gerrelu
Ezz
Zzá [o(Z)) o Zwil, SRetu (Zen) [
(97)
tjk
Using Gerrea the second term can be expressed as
second termn Σ
Gikk v
(98)
j.k
(i) j + l case:
2n
/ II dZzidZuu
Gerelu Gjk
- 23. Zzj
e
2m
a (r- Σ
Σ Zzrur
Zyrurt SRelu Σ Zuaiot
x
2n /
džzzdZz
e - 22
Σ o
Eo!
Zime
2m
(111
rr w 1// FRelu (lw" (lw*|Z+a1)
2
/ dž dž dēzie= -123 ) 7
V2T
1111
r - w 12111 |w0 12100 (Zx1)
x
2
J d7
12 2,10 (2n1) (r-lurlz,,)
//lll/ 22,1) |w0*|
/2T
2
Jo dZ
2 0 T 3,31 |w=1
2m
[w"
0 T (1 - -
2n
|w"
(1-6 e- ((1ū)))
Wi w
(99)
20

---

Under review as a conference paper at ICLR 2022
where we used the orthogonal transformation in the second line:
Zu
> Din Zsm's
Σ Or Zakmm
(100)
with im = Wm/ //w/ and Omm- = u /1w/1
(ii) j k case:
/
Gerelu
II dZ
e 1/2 I 233 Zyi Zzj
2m
(r- (r-22 Σ Zyrwr/ fRelu
x o
Σ
Zzewn
Znoo
/
dZzi
e Ei Z3 Zzs
2
Trewr
Zyrwr
SRotu (w" Zzj)
1=1
1=1
1
II d2ga
-
$ Eiz 731 Zzi (8:1 + 8:2)
2x
1=1
2
(1- (r-) Σ Zyowr SRetu (w* 2z1)
ZrrWi
1
dZ.daZe 1/22 (Z cos cost611 0ō,1 + Z sin
2x
(Zw cos (0 (0-))) ( (1- Znwrr FRelu (w" cos
1=1
1
dZd0Ze- 1z² (Z cos 06,1 + sin 00.2)
2x
x (Zw cos (0 p)) a (r Zw cos (0 xp) w cos fo (cos 0)
(101)
w"
/ in woodlapp
d0 (cos² 0d.1 cos 0 sin 06:2)
2m
-/2+
wn
(x/2
dA (cos² 05,1 cos f sin 26
2n
-n/2+ip
(( cos (O sp)
r
x
2
+ 2
(102)
where we have chosen
w" = (w", 0n-1);
(w1, W2, On-2)
(103)
with an > O and yo is defined the angle between wn and un. We also have used the formula,
dxx 3 = =2 -t- 2
(104)
Note that
W.
cos 1oō,1 sin 10
(105)
21

---

Under review as a conference paper at ICLR 2022
Using C(w, (p), S(w, (p) defined
F/
((
w¹
C(w,ip)
dl cos 0
2
+
2T
1/2-
( cos (0 -p)
S(w,i))
w³
///
d0 sin 0 cos 0
2
r
+ 2
cost
2T 5/24pp
cos (0 (p)
(106)
If (p + 0, Gij is formally written
Cerelu {j)
C(w, (p)
W 1
+
S(w, sp)
wi
cot
W
(107)
sin 40 |wo|
If s 0r π, G is formally written
w w
Go C(w, 0)
for xp =0 0.
(108)
Gerelu for 4p = n
(109)
where we used S(w, 0) S(w, π) = C(w, π) = and
w"
//2
db cos $ {2 (( w cos
)²
2 +2) [(mm)))
C(w, 0)
2π
-/2
w
r (+)² 2n werfe
r
(110)
2x
V2w
with the following formula
dl
werfc
(111)
V22
1/2
//
dl
cos
De
2
1
ae $ 2t + (1- a²) Terfc
(112)
2
V2
x/2
In summary, the second term written as
second term
1 w=1
(1 (1-e- -
wi Fo" (-Imxm Im 1 w*1
2x wwl
Cn., sp) wn
1 wi cot y w )
w!! (0"00)
+ S(w,
sp)
(113)
|001
sin |m
22


Reviews:
[{"review_id": "carg3Auhj3F", "reviewer": "Reviewer_tphS", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths:**\nThe authors propose a clear, concise and closed form analysis of the true gradient (i.e. the gradient of the expected loss) and its stationary points. This is possible due to the simplicity of the model and the assumed data distribution. The same is done for each of the three STE variants. This proves that the first two STE variants are biased in general. Moreover, the authors prove in particular that none of the global optimisers of the expected loss is a stationary point for these STE variants. (Notice that the globally optimal weights of the hidden layer form a ray due to scale invariance). On the other hand, they show that at least one of these global optimisers is also a stationary point for the third STE variant.\n\n**Weaknesses**\nThe same model has been studied in the previous work Yin et al. (2019) for a different loss. It has been shown there that the first STE variant is biased, while the second and third where found to be unbiased in this case. The loss studied there was the MSE loss w.r.t. a ground truth model with the same architecture. The loss considered in the submitted manuscript is the MSE loss w.r.t. a ground truth model that uses ReLU activations in the hidden layer. The submitted manuscript thus also rules out the second STE variant as biased. \n\nThe authors cite the previous work Shekhovtsov & Yanush (2020) which derives STEs as a linearisation of an unbiased gradient estimator with low variance. In particular, it is shown there that the properly derived ST estimator is unbiased for models studied in the submitted manuscript, albeit for a different type of loss. It is clearly seen that their derived unbiased ST estimator breaks the scale invariance. I am pretty sure that the derivation given there can be generalised for the model & loss combination studied in the submitted manuscript.", "summary_of_the_paper": "The paper considers a network with one hidden layer with binary valued neurons sharing the same weight vector but having non-overlapping receptive fields. The authors analyse the stationary points of a particular MSE loss when a straight through estimator (STE) is used for the loss gradient. They analyse three variants for the STE: (1) derivative of the identity function, (2) derivative of ReLU and (3) derivative of a piecewise linear squashing function. Assuming a simple independent distribution over the inputs and a particular loss, the authors show that the first two variants are biased and only the third STE variant can be unbiased. The authors conjecture that the reason for this is that the third STE variant breaks the scaling symmetry w.r.t. the weights of the first layer.", "summary_of_the_review": "The paper provides a convincing in-depth analysis of STE properties for a particular simple combination of model and data distribution. However, the novelty is limited in view of the previous work Yin et al. (2019). Its significance is restricted due to the very specific assumptions on the model, data distribution and loss.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "WSbJsDudp-B", "reviewer": "Reviewer_1Ehv", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper contains little novelty compared to Yin et al. (2019)\nAlso, it is not well written. The claims are weak and it is not clear if they are specific to the specific setting chosen.\nThere is no numerical validation.\n\n- Claims such as\n\"\"\"\nConsidering the scale invariance of Relu STE gradient, this relation implies that the\nRelu gradient is approximately embedded in the cRelu STE gradient, so that if the back-\npropagation using Relu STE converges to a point, the back-propagation using cRelu STE\ncan also converge to that point.\n\"\"\"\nare not entirely clear nor sufficiently motivated.\n\n\n- The main claim reads\n\"\"\"\nIn other words, because of the breaking of the scale symmetry, the cRelu STE may pick up one of local minima,\nwhich degenerate in all scales, while because of keeping the scale symmetry, identity STE and Relu\nSTE may not. Therefore, we conclude that the back-propagation using cRelu STE is the most likely\nto achieve the true minimum in the three STEs.\n\"\"\"\nIn other words, the authors claim that some minima of the population loss of the original model *may* also be minima of the surrogate model with cReLU STE if the weights are appropriately rescaled. This claim has only the weak analytical support\ngiven by the analysis of the Gaussian input case of the next Section 3. No numerical support is provided for\nconvergence of the training dynamics in this toy model, let alone deeper networks and structured datasets.\n\n- In Section 3, the authors show that the cReLU STE has an additional stationary point (the global minimum) compared to the\nReLU STE. This is an interesting result, and also somehow natural given that a non-invariant gradient has more degree-of-freedoms\nfor satisfying the stationary point condition. The single setting explored, the one with the ReLU teacher, doesn't seem enough to support general claims though. Can the author check analytically a few other cases, e.g. tanh activation. Or at least experimentally (possibly also\nin deeper networks)? \n\n- Reviewing the paper was made more complicated by the poor writing, \nstarting from the very first sentence of the abstract  \"In the quantized network, its gradient shows either vanishing or diverging.\"\n", "summary_of_the_paper": "The paper investigates analytically the scale invariance properties of \ndifferent straight-through estimators used in training neural networks with binary activation and weights. \n\nThe main claim is that the cReLU STE, among the 3 considered, is the one that is more likely to give zero gradients in the minima of the population loss. This is (weakly) supported by theoretical analysis on a little variation of the shallow setting analyzed in Yin et al. (2019), where the teacher now contains ReLU activations.   \n", "summary_of_the_review": "While there is some novelty to the paper, I don't think the importance of the results and the novelty level\nare high enough for the ICLR standards.\nMoreover, there is a complete lack of experimental validation and some poor writing. \nI cannot recommend the manuscript for publication on ICLR. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "R1W6Xgdrite", "reviewer": "Reviewer_Tac4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strong points:\n* A novel approach on how to analyze various gradient estimators.\n* Interesting new theoretical insights into various flavors of the STE gradient approximation.\n* Detailed derivations in the appendix.\n\nWeak points:\n* No experimental results, even not on MNIST or a toy example to show that the theoretical findings correlate with empirical observations.\n* The straight through estimator (STE, identity function) is know to be a biased estimator. In the paper this discussion is not coming back, but I think it would be interesting to discuss whether the other estimators (ReLU, cReLU) are biased and whether one is more biased than the other. Is there a trade-off to be made between being more biased vs having a higher chance to converge to a local minimum?\n* The optimization problem defined in 2.1 (eq 1 & 2) is non-convex. For a non-convex optimization problem it is unclear to me based on the manuscript how the gradients in stationary points are important and increase the chance to get to the true global minimum (page 5, below eq 23). Some further explanation on the reasoning could be helpful to understand for the reader and to verify that these claims are true.\n* A discussion on whether or not this analysis also holds for 1) weight quantization and 2) multi bit quantization would be interesting. Especially for the latter case it is common practice that the gradient is clipped due to the clipping in quantization (STE is applied to the rounding operation, thus the gradient of the full quantization operation is clipped due to the gradient of the clipping, see Esser et al. 2019).\n* The clarity of the paper could improve and there are some misconceptions:\n    * The straight through estimator is clearly defined in literature (see Bengio et al. 2013) as back propagation through a function as it would have been the identity function (thus gradient of 1 in the backwards path). However, in the manuscript any approximate gradient is defined as a STE gradient. This misuse of terminology makes it extra difficult and confusing to understand the work, especially since it is of theoretical nature. (“In this paper, we refer to the proxy of the gradient as the STE gradient.”)\n    * It is a bit confusing that sigma is defined as step function. But based on the text (binary activations) and later equations it seems that it is a sign function. Would be good to be clear and maybe mathematically describe the exact function.\n\n", "summary_of_the_paper": "The paper does a theoretical analysis of various gradient approximations for the non differentiable sign function. They investigate 3 gradient estimators based on the STE assumption, namely (identity) STE, ReLU STE, and clipped STE (cReLU). They show theoretically that a gradient estimator which breaks the scale symmetry (clipped STE) in the network is more likely to achieve a local minimum than one that keeps the symmetry.", "summary_of_the_review": "Interesting and novel perspective on various gradient estimators for the sign function used in binary activations. The main drawbacks are that there is no empirical data to support any of their claims, several aspects are unclear based on the manuscript and the paper could improve in its overall clarity. In the current version and my understanding of it the drawbacks outweighs the strong points.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "IU4pWImT_st", "reviewer": "Reviewer_W5Me", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nI have read the rebuttal. I'd like to thank the authors for the revision, although it does not resolve my concerns.\n\n==========================\n\n## Summary\n\nTraining quantized neural network by the biased Straight-Through estimator (STE) is a common practice in the NN quantization literature. However, due to the biased nature of STE, there is no principled guidance in making algorithmic choices for such training methods. This work focuses on the choice of surrogate differentiable functions in STE, and looked at three common choices: identity, ReLU, and clipped ReLU. Through analysis of the scale invariance of gradients in STE, the paper suggests clipped ReLU might pick up local minima degenerated in scales, due to its property of breaking scale symmetry among the three. The authors further analyzed the stationary points of a simple misspecified model with Gaussian inputs to confirm their observations.\n\n## Comments\n\n* The clarity of the paper is very poor. Currently the paper is more like a technical note than a polished conference paper. There are deep stacks of equations everywhere and many of them lack clear explanation. To support my claim, here are a few examples:\n\nEquation (1) is confusing as a convolutional neural network because the hidden layer output is a simple matrix vector product, I figured it out after some thought that you restructured the input by replicating its features in a window to form each row of Z. However, you mentioned this nowhere and it is even not stated what $m$ and $n$ is.\n\n* The main claim is not rigorous, which makes it very difficult to understand the implications of the results and the potential impact of the paper.  Many arguments are hand-wavy and when approximation is used, there is no characterization of the error, to list a few:\n\n\"the cRelu STE gradient for |w| << 1 is approximately identical with the Relu STE gradient\"\n\n\"this relation implies that the Relu gradient is approximately embedded in the cRelu STE gradient\"\n\n\"may not be zero\" \n\n\"there is still a chance to be zero\"\n\n\"using cRelu STE is the most likely to achieve the true minimum in the three STEs\"\n\nAll these make the conclusion less convincing and the lack of empirical evidence makes it worse.\n\n## Overall suggestions\n\nI vote for rejection mainly due to the questionable analysis (lack of rigor) and the difficulty to interpret the relevance of the result. Also the work does not go beyond one-hidden layer network as it claimed in the abstract.\n\n## Questions and minor points\n\n* Is my understanding correct that the hard $g_j$ has no scale invariance (it only has scale invariance in the loss)? \n* I'm always curious, have the quantized neural network community ever looked into other unbiased gradient estimators developed by the Monte-Carlo gradient estimation people? Or is it true that STE empirically outperformed all of them so it is still the default practice?\n", "summary_of_the_paper": "Training quantized neural network by the biased Straight-Through estimator (STE) is a common practice in the NN quantization literature. However, due to the biased nature of STE, there is no principled guidance in making algorithmic choices for such training methods. This work focuses on the choice of surrogate differentiable functions in STE, and looked at three common choices: identity, ReLU, and clipped ReLU. Through analysis of the scale invariance of gradients in STE, the paper suggests clipped ReLU might pick up local minima degenerated in scales, due to its property of breaking scale symmetry among the three. The authors further analyzed the stationary points of a simple misspecified model with Gaussian inputs to confirm their observations.", "summary_of_the_review": "I vote for rejection mainly due to the questionable analysis (lack of rigor) and the difficulty to interpret the relevance of the result. Also the work does not go beyond one-hidden layer network as it claimed in the abstract.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Meta-review: I agree with the reviewers that this work is not well-presented, and it seriously lacks rigor and experimental support. The writing of this work also needs significant improvement. The authors made many claims without offering rigorous proofs, and hand-waved their argument throughout without strong empirical support. In the end, the authors' response did not address the reviewers' concerns satisfactorily and no one is excited enough to defend the current draft. Please consider revising your draft according to the reviewers' comments.
Decision: Reject