Question: The optimization problem defined in 2.1 (eq 1 & 2) is non-convex. For a non-convex optimization problem it is unclear to me based on the manuscript how the gradients in stationary points are important and increase the chance to get to the true global minimum (page 5, below eq 23). Can you provide some further explanation on the reasoning ? It could be helpful to understand for the reader and to verify that these claims are true.
Evidence: We have found that breaking symmetry embedded in the network by STEs enhances the possibility of convergence to the true (local) minimum of the loss function. We have demonstrated that if an STE breaks the scale symmetry embedded in the one-hidden-layer network with a binary activation, it is more likely to achieve the local minimum than the one which keeps the symmetry. The discussion can be generalized to the network with more symmetries in a straightforward way. The more symmetries embedded in the network an STE breaks, the more likely it is to converge. To confirm the mechanism, we have studied three STEs, identity STE, Relu STE, and cRelu STE, in a simple misspecified model with Gaussian input. We have found that the back-propagation by the cRelu STE, which breaks the scale symmetry, can converge to the global minimum, while identy/Relu STE cannot. Finally we have numerically confirmed the mechanism for the mixture Gaussian model with various teacher network.
A DERIVATION OF SEC.3
In the Appendix, we provide the results of the Gaussian with variance 1, N (0, 1). For some function f (Z), the expectation value of the Gaussian with the generic variance σ, N (0, σ2 ), shown in Sec.3 is written as
where we have changed the integral variables from Z to Z = Z/σ. Therefore we get the results with the variance σ by only changing from f (Z) to f (σZ).
The gradients of the loss for each sample w.r.t v and w can be formally expressed as
where x i = σ( j Z ij w j ) and we used the property that the derivative of the step function is Dirac's delta function, dσ(z)/dz = δ(z). The gradient w.r.t v can be utilized for learning the network, while it is impossible to use the gradient w.r.t w due to the presence of the delta function. Remarkably, both population gradients w.r.t v and w are smooth functions due to its average over the continuous distribution,
where we used the orthogonal transformation,
We can easily see that the gradient w.r.t. w does not have the component in w-direction, j w j E Z ∂ ∂wj = 0. Its vanishing originally follows from the scale symmetry of the network: Because the loss has the scale symmetry, its gradient in the w-direction is obviously zero.
Instead of the population gradient, we use the STE in practice, which replaces the delta function with the derivative of some differentiable function µ STE , to train the kernel w,
We consider three types of µ STE (x): (i) the identity type µ STE (x) = x , (ii) vanilla ReLU type µ STE (x) = xσ(x) and (iii) the cReLU type µ STE (x) = xσ(x)σ(r − x) with the upper clipping value r. The surrogate back-propagation using the STE is described in Algorithm 1).
Algorithm 1 Surrogate back-propagation using STE for learning one-hidden-layer CNN.
The average of the STE gradients over the input data is also expressed as
Note that in general, the STE gradients have the non-zero component in the w-direction due to their partial collapse of the form of the gradient, j w j E Z g STE j (v, w; Z) = 0.
It is obvious from Eq.() that the only difference between the STE gradients with the identity function, Relu, and cRelu is the region of the integration over Zi1 ,
where A = id, Relu, cRelu, and the integration reigion R A are given as
Following the scale invariance of
, the STE gradients have the following interesting features:
1. The identity and Relu STE gradients have scale invariance for w,
2. The cRelu STE gradient does not have scale invariance for w due to the clipping effect,
where we have explicitly shown the dependence of the upper clipping value r. 3. Instead, the scale transformation of w for the cRelu STE gradient can be compensated by that of r:
If we take kr with fixed r large enough to cover most of the distribution P (Z), the left-hand side is identical with the Relu STE gradient. Here we define the spread of the distribution as ρ. When we redefine w/k as w on the right hand, the cRelu STE gradient for |w| < r/ρ is approximately identical with the Relu STE gradient,
This result is intuitively obvious: when w is small enough to keep most of the pre-activation values in the clipped range, the cRelu gradient and the Relu gradient are approximately identical.
The important issue is that there is no guarantee that the weights obtained by the STE backpropagation are the ones at the (local) minimum of the loss function. In other words, at the stationary points, (v, w) = (v s , w s ), defined as the vanishing points of the population gradient,
the STE gradient is not zero in general. Here we consider the local minimum point that the STE gradient does not vanish,
Note that (v, w) = (v s , kw s ) for any k > 0 is also the local minimum point due to the scale invariance of the loss. In the case of the Relu/identity STEs, their gradient has the same finite value for any scales due to the scale invariance shown in Eq.(),
In the case of the cRelu STE, however, the value of its gradient changes as the scale changes due to the breaking of the scale symmetry shown in Eq.(). If it crosses zero as a function of scale,
the back propagation using cRelu STE can converge at the local minimum as was done by using the population gradient. Therefore, we conjecture that due to breaking scale symmetry, the backpropagation using cRelu STE is the most likely to achieve the (local) minimum of the loss function in the three STEs. This also implies that the cRelu STE is less biased than the other. We confirm the conjecture analytically for the Gaussian input with Relu-type teacher network in Sec.3, and numerically for the mixture Gaussian input with various teacher network in Sec. 4.
Since the above discussion is focused only on symmetry, the conjecture can be generalized to networks with more symmetries in a straightforward way. In that case, if we employ an STE that breaks all of those symmetries instead of cRelu STE, it is more likely to achieve the minimum.
The similar mechanism can be also found in physical phenomena. For instance, this is the case of the ferromagnet, which has rotational symmetry in energy (or Hamiltonian). In the system, if we impose the magnetic field, which breaks the rotational symmetry, the spin states, corresponding to minimum points, become aligned with the same direction.
We have found that breaking symmetry embedded in the network by STEs enhances the possibility of convergence to the true (local) minimum of the loss function.****Since the above discussion is focused only on symmetry, the conjecture can be generalized to networks with more symmetries in a straightforward way. In that case, if we employ an STE that breaks all of those symmetries instead of cRelu STE, it is more likely to achieve the minimum.
Answer: Thank you very much for the comments. Our claim is that the cRelu STE is more likely to achieve the “true local” minimum of the loss function than id/Relu STEs. As you mentioned, all the STEs are biased estimators, so that there is no guarantee to show the convergence to the “true” local minimum. They may converge to “fake” minimum which is defined by vanishing point of STE gradient. To avoid confusion for readers, we have changed the sentences near the end of Sec.2.2 to make it clear. We also have added the discussion on biased/unbiased estimators you mentioned above.