Question: The straight through estimator (STE, identity function) is know to be a biased estimator. In the paper this discussion is not coming back, but I think it would be interesting to discuss whether the other estimators (ReLU, cReLU) are biased and whether one is more biased than the other. Is there a trade-off to be made between being more biased vs having a higher chance to converge to a local minimum?
Evidence: Quantization of the weights and the activations is a promising technique to save the memory and accelerate the inference speed in deep neural networks (DNNs) which have been getting wider and deeper in recent years. There are two main approaches of the quantization for DNNs (Krishnamoorthi ()): Post-training quantization (PTQ) and quantization-aware training (QAT). In the PTQ, the pre-trained networks are simply quantized without re-training the model. This approach allows us to achieve the nearly floating point accuracy at 8-bits, while below 8-bits, this results in significant accuracy degradation. Although there have been recent attempts to alleviate the accuracy degradation in PTQ;;;), the QAT, where the quantized weights and activations are trained (e.g.,;;), usually leads to better accuracy.
The difficulty in QAT is that the weights and the activations are discretized, and intrinsically nondifferentiable. If we take the derivatives forcibly, they either vanish or diverge. To avoid this problem, we replace them with the derivatives of some differentiable function in the backward pass only, called the Straight-Through Estimator (STE);). Since the replacement leads to bias, it is not always possible to learn the network successfully. However this approach can be applied to the low bits below 8-bits with tolerable accuracy degradation;;;).
Originally, STE was introduced as the replacement with the derivative of the identity function by. Later, the term "STE" has been extensively used as the replacement with various functions. In the binary network, studied the replacement with the derivative of sigmoid function as well as the original STE, and used the derivative of the identity function clipped in the region of |x| ≤ 1. In low-bits network, used the quantized gradient after the replacement with the identity function, and, and used the the derivative of the clipped Relu (cRelu) for leaning the step size.
The gradients of the loss for each sample w.r.t v and w can be formally expressed as
where x i = σ( j Z ij w j ) and we used the property that the derivative of the step function is Dirac's delta function, dσ(z)/dz = δ(z). The gradient w.r.t v can be utilized for learning the network, while it is impossible to use the gradient w.r.t w due to the presence of the delta function. Remarkably, both population gradients w.r.t v and w are smooth functions due to its average over the continuous distribution,
where we used the orthogonal transformation,
We can easily see that the gradient w.r.t. w does not have the component in w-direction, j w j E Z ∂ ∂wj = 0. Its vanishing originally follows from the scale symmetry of the network: Because the loss has the scale symmetry, its gradient in the w-direction is obviously zero.
Instead of the population gradient, we use the STE in practice, which replaces the delta function with the derivative of some differentiable function µ STE , to train the kernel w,
We consider three types of µ STE (x): (i) the identity type µ STE (x) = x , (ii) vanilla ReLU type µ STE (x) = xσ(x) and (iii) the cReLU type µ STE (x) = xσ(x)σ(r − x) with the upper clipping value r. The surrogate back-propagation using the STE is described in Algorithm 1).
Algorithm 1 Surrogate back-propagation using STE for learning one-hidden-layer CNN.
The average of the STE gradients over the input data is also expressed as
Note that in general, the STE gradients have the non-zero component in the w-direction due to their partial collapse of the form of the gradient, j w j E Z g STE j (v, w; Z) = 0.
It is obvious from Eq.() that the only difference between the STE gradients with the identity function, Relu, and cRelu is the region of the integration over Zi1 ,
where A = id, Relu, cRelu, and the integration reigion R A are given as
Following the scale invariance of
, the STE gradients have the following interesting features:
1. The identity and Relu STE gradients have scale invariance for w,
2. The cRelu STE gradient does not have scale invariance for w due to the clipping effect,
where we have explicitly shown the dependence of the upper clipping value r. 3. Instead, the scale transformation of w for the cRelu STE gradient can be compensated by that of r:
If we take kr with fixed r large enough to cover most of the distribution P (Z), the left-hand side is identical with the Relu STE gradient. Here we define the spread of the distribution as ρ. When we redefine w/k as w on the right hand, the cRelu STE gradient for |w| < r/ρ is approximately identical with the Relu STE gradient,
This result is intuitively obvious: when w is small enough to keep most of the pre-activation values in the clipped range, the cRelu gradient and the Relu gradient are approximately identical.
The important issue is that there is no guarantee that the weights obtained by the STE backpropagation are the ones at the (local) minimum of the loss function. In other words, at the stationary points, (v, w) = (v s , w s ), defined as the vanishing points of the population gradient,
the STE gradient is not zero in general. Here we consider the local minimum point that the STE gradient does not vanish,
Note that (v, w) = (v s , kw s ) for any k > 0 is also the local minimum point due to the scale invariance of the loss. In the case of the Relu/identity STEs, their gradient has the same finite value for any scales due to the scale invariance shown in Eq.(),
In the case of the cRelu STE, however, the value of its gradient changes as the scale changes due to the breaking of the scale symmetry shown in Eq.(). If it crosses zero as a function of scale,
the back propagation using cRelu STE can converge at the local minimum as was done by using the population gradient. Therefore, we conjecture that due to breaking scale symmetry, the backpropagation using cRelu STE is the most likely to achieve the (local) minimum of the loss function in the three STEs. This also implies that the cRelu STE is less biased than the other. We confirm the conjecture analytically for the Gaussian input with Relu-type teacher network in Sec.3, and numerically for the mixture Gaussian input with various teacher network in Sec. 4.
Since the above discussion is focused only on symmetry, the conjecture can be generalized to networks with more symmetries in a straightforward way. In that case, if we employ an STE that breaks all of those symmetries instead of cRelu STE, it is more likely to achieve the minimum.
The similar mechanism can be also found in physical phenomena. For instance, this is the case of the ferromagnet, which has rotational symmetry in energy (or Hamiltonian). In the system, if we impose the magnetic field, which breaks the rotational symmetry, the spin states, corresponding to minimum points, become aligned with the same direction.
To confirm our conjecture in more general case, we have numerically studied the Gaussian mixture input with various mean values. As teacher networks, we have tested tanh-type and sin-type as well as Relu-type. For all the examined setups, cRelu STE behaves like the population gradient, while id/Relu STEs show qualitatively different behaviors as described below. To calculate the population loss and STE gradients, we have generated the mixture Gaussian samples and have taken their average. The population gradient has been obtained by calculating the finite difference of the population loss. We have demonstrated the back propagation with learning rate η = 0.01 given in Algorithm 1.
Shown in Fig. are the results of ten mixture Gaussian input with random mean values for each components of Z. We employed m = 20, n = 25, and the tanh-type teacher network. We find that the population gradient and cRelu STE show similar results, while id/Relu STEs are completely different. This indicates cRelu STE is less biased than id/Relu STEs.
In the case of id/Relu STEs, at early steps up to around 500 step, |w| decreases around the magnitude of the update quantity, so that it begins to oscillate. Then it escapes the oscillation, and the loss function shows the convergence to a point different from the local minimum achieved by population gradient, while |w| becomes larger and larger due to their scale invariance. Interestingly, the values of the loss function are small compared to the one obtained by population gradient, which implies that id/Relu STEs avoid being trapped in the local solution due to their large bias. However, note that even at that point, the magnitude of w continues to grow and eventually become numerically unstable.
Figure: Numerical results of the back-propagation by population gradient and three STEs. We generate 10000 samples which follow ten mixture Gaussian input with random mean values. We employ the tanh-type teacher network.
Since the replacement leads to bias, it is not always possible to learn the network successfully.****This also implies that the cRelu STE is less biased than the other. We confirm the conjecture analytically for the Gaussian input with Relu-type teacher network in Sec.3, and numerically for the mixture Gaussian input with various teacher network in Sec. 4.****We find that the population gradient and cRelu STE show similar results, while id/Relu STEs are completely different. This indicates cRelu STE is less biased than id/Relu
Answer: Thank you very  much for the useful suggestion. We have mentioned the points in Sec1 (Introduction), Sec2.2 (General discussion), and Sec.4(Experiments) in red as follows: Sec.1:  … Since the replacement leads to bias, it is not always possible to learn the network successfully. … Sec.2.2: … This also implies that the cRelu STE is less biased than the other. … Sec.4: ….We find that population gradient and cRelu STE show similar results, while id/Relu STEs are completely different. This indicates cRelu STE is the less biased estimator than the other estimators. …