Title: SPONTANEOUS SYMMETRY BREAKING IN DEEP NEU-RAL NETWORKS

Abstract: We propose a framework to understand the unprecedented performance and robustness of deep neural networks using field theory. Correlations between the weights within the same layer can be described by symmetries in that layer, and networks generalize better if such symmetries are broken to reduce the redundancies of the weights. Using a two parameter field theory, we find that the network can break such symmetries itself towards the end of training in a process commonly known in physics as spontaneous symmetry breaking. This corresponds to a network generalizing itself without any user input layers to break the symmetry, but by communication with adjacent layers. In the layer decoupling limit applicable to residual networks (He et al., 2015a), we show that the remnant symmetries that survive the non-linear layers are spontaneously broken based on empirical results. The Lagrangian for the non-linear and weight layers together has striking similarities with the one in quantum field theory of a scalar. Using results from quantum field theory we show that our framework is able to explain many experimentally observed phenomena, such as training on random labels with zero error (Zhang et al., 2017) , the information bottleneck and the phase transition out of it and the following increase in gradient variances (Shwartz-Ziv & Tishby, 2017), shattered gradients (Balduzzi et al., 2017) , and many more.

1. INTRODUCTION:

Deep neural networks have been used in image recognition tasks with great success. The first of its kind, AlexNet (Krizhevsky et al., 2012) , led to many other neural architectures have been proposed to achieve start-of-the-art results in image processing at the time. Some of the notable architectures include, VGG (Simonyan & Zisserman, 2015) , Inception (Szegedy et al., 2015) and Residual networks (ResNet) (He et al., 2015a) .

Understanding the inner workings of deep neural networks remains a difficult task. It has been discovered that the training process ceases when it goes through an information bottleneck (Shwartz-Ziv & Tishby, 2017) until learning rate is decreased to a suitable amount then the network under goes a phase transition. Deep networks appear to be able to regularize themselves and able to train on randomly labeled data Zhang et al. (2017) with zero training error. The gradients in deep neural networks behaves as white noise over the layers Balduzzi et al. (2017) . And many other unexplained phenomena.

A recent work (Anonymous, 2018) showed that the ensemble behavior and binomial path lengths (Veit et al., 2016) of ResNets can be explained by just a Taylor series expansion to first order in the decoupling limit. They found that the series approximation generates a symmetry breaking layer that reduces the redundancy of weights, leading to a better generalization. Because the ResNet does not contain such symmetry breaking layers in the architecture. They suggest that ResNets are able to break the symmetry by the communication between the layers. Another recent work also employed the Taylor expansion to investigate ResNets (Jastrzebski et al., 2017) .

In statistical terms, a quantum theory describes errors from the mean of random variables. We wish to study how error propagate through each layer in the network, layer by layer. In the limit of a continuous sample space, the quantum theory becomes a quantum field theory. The effects of sampling error and labelling error can then be investigated. It is well known in physics that a scalar field can drive a phase transition. Using a scalar field theory we show that a phase transition must exist towards the end of training based on empirical results. It is also responsible for the remarkable performance of deep networks compared to other classical models. In Appendix D, We explain that quantum field theory is likely one of the simplest model that can describe a deep network layer by layer in the decoupling limit.

Much of the literature on neural network design focuses on different neural architecture that breaks symmetry explicitly, rather than spontaneously. For instance, non-linear layers explicitly breaks the symmetry of affine transformations. There is little discussion on spontaneous symmetry breaking. In neural networks, the Goldstone theorem in field theory states that for every continuous symmetry that is spontaneously broken, there exists a weight with zero Hessian eigenvalue at the loss minimum. No such weights would appear if the symmetries are explicitly broken. It turns out that many seemingly different experimental results can be explained by the presence of these zero eigenvalue weights. In this work, we exploit the layer decoupling limit applicable to ResNets to approximate the loss functions with a power series in symmetry invariant quantities and illustrate that spontaneous symmetry breaking of affine symmetries is the sufficient and necessary condition for a deep network to attain its unprecedented power.

The organization of this paper is as follows. The background on deep neural networks and field theory is given in Section 2. Section 3 shows that remnant symmetries can exist in a neural network and that the weights can be approximated by a scalar field. Experimental results that confirm our theory is given in Section 4. We summarize more evidence from other experiments in Appendix A. A review of field theory is given in Appendix B. An explicit example of spontaneous symmetry breaking is shown in Appendix C.

2. BACKGROUND AND FRAMEWORK:

In this section we introduce our frame work using a field theory based on Lagrangian mechanics.

2.1. DEEP NEURAL NETWORKS:

A deep neural network consists of layers of neurons. Suppose that the first layer of neurons with weight matrix W 1 and bias b 1 takes input x 1 and outputs y 1

y 1 = W 1 x 1 + b 1 = W 1 x 1 ,

where x = (x , 1) and W 1 = (W 1 , b 1 ), where W 1 and b 1 are real valued. Now suppose that R 1 denotes a nonlinear operator corresponding to a sigmoid or ReLU layer located after the weight layer, so that

y 1 = R 1 W 1 x 1 .

For a neural network with T repeating units, the output for layer t is

y t = R t W t y t-1 = R t W t R t-1 W t-1 . . . R 1 W 1 x 1 = t-1 n=0 R t-n W t-n x 1 .

(1)

2.2. SYMMETRIES IN NEURAL NETWORKS:

We now show the necessary and sufficient conditions of preserving symmetry. We explicitly include symmetry transformations in Equation (1) and investigate the effects caused by a symmetry transformation of the input in subsequent layers. Suppose Q t ∈ G is a transformation matrix in some Lie group G for all t. Note that the Q t are not parameters to be estimated. We write y t = y t (Q t ), where the dependence on Q t is obtained from a transformation on the input, x t (Q t ) = Q t x t , and the weights,

W t (Q t ) y t (Q t ) = R t W t (Q t )x t (Q t ).

If G is a symmetry group, then y t is covariant with x t , such that y t (Q t ) = Q t y t . This requires two conditions to be satisfied. First,

W t (Q t ) = Q t W t Q -1 t , where Q -1 t Q t =

I and the existence of the inverse is trivial because G is a group and Q t ∈ G. The second is the commutativity between R t and

Q t , such that R t Q t = Q t R t .

For example, if g t ∈ Aff(D), the group of affine transformations, R t may not commute with g t . However, commutativity is satisfied when the transformation corresponds to the 2D rotation of feature maps.

Including transformation matrices, the output at layer t is

y t (Q t ) = t-1 n=0 R t-n Q t-n W t-n Q -1 t-n Q 1 x 1 .

2.3. THE LOSS FUNCTIONAL:

Statistical learning requires the loss function to be minimized. It can be written in the form of a mutual information, training error, or the Kullback-Leibler divergence. In this section we approximate the loss function in the continuum limit of samples and layers. Then we define the loss functional to transition into Lagrangian mechanics and field theory. Let z i = (X i , Y i ) ∈ X be the i-th input sample in data set X , (X i , Y i ) are the features and the desired outputs, respectively, and i ∈ {1, . . . , N }.

The loss function is

L = 1 N N i=1 L i (X i , Y i , W, Q),

where W = (W 1 , . . . , W T ), and

Q = (Q 1 , . . . , Q T ), Q t ∈ G

where G is a Lie group, and T is the depth of the network. Taking the continuum limit,

L X p(X, Y)L x (X, Y, W, Q)dXdY,

where p(X, Y) is the joint distribution of X and Y. Using the first fundamental theorem of calculus and taking the continuous layers (t) limit, we write

L x L x (t = 0) + T 0 dL x (X, Y, W(t), Q(t)) dt dt,

where L x (t = 0) is the value of the loss before training. We let L x,t = dL x /dt be the loss rate per layer. The loss rate L x,t is bounded from below. Therefore

min W L x (X, Y, W, Q) = L x (t = 0) + T 0 min W L x,t (X, Y, W(t), Q(t))dt.

Minimizing the loss rate guarantees the minimization of the total loss. We require L x,t to be invariant under symmetry transformations. That is, if

Q 1 (t), Q 2 (t) ∈ G. Then L x,t (X, Y, W(t), Q 1 (t)) = L x,t (X, Y, W(t), Q 2 (t)).

However if Q 1 (t) and Q 2 (t) do not belong in the same symmetry group, the above equality does not necessarily hold. Now we define the loss functional for a deep neural network

P[W, Q] = X ×{0,T } p(X, Y)L x,t (X, Y, W(t), Q(t))dXdYdt.

2.4. LAGRANGIAN MECHANICS AND FIELD THEORY:

Having defined the loss functional, we can transition into Lagrangian dynamics to give a description of the feature map flow at each layer. Let the minimizer of the loss rate be

W * (X, Y, Q(t), t) = arg min W p(X, Y)L x,t (X, Y, W(t), Q(t)).

From now on, we combine z = (X, Y) as Y only appears in W * in this formalism, each Y determines a trajactory for the representation flow determined by Lagrangian mechanics. Now we define, for each i-th element of W(t), and a non-linear operator R(t) acting on W(t) such that the loss minimum is centered at the origin,

EQUATION

We now define the Lagrangian density,

L = L(z, w, ∂ t w, ∂ z w, Q(t))

and L = T -V, where T is the kinetic energy and V is the potential energy. We define the potential energy to be

V = p(z)L x,t (z, W(t), Q(t)),

The probability density p(z) and the loss rate L x,t are invariant under symmetry transformations. Therefore V is an invariant as well.

Definition: Orthogonal Group The orthogonal group O(D) is the group of all D × D matrices such that O T O = I.

We now set up the conditions to obtain a series expansion of V around the minimum

w i (Q t ) = 0.

First, since V is an invariant. Each term in the series expansion must be an invariant such that

f (w i (Q t ), w i (Q t )) = f (w i , w i ) for all Q t ∈ G. Suppose G = O(D), the orthogonal group and that w T (Q t ) = w T Q T and w(Q t ) = Q t w. So w i w i is an invariant. Then f = w i w i is invariant

for all Q t where the Einstein summation convention was used

w i w i = D i=1 w i w i = w T w,

Now we perform a Taylor series expansion about the minimum w i = 0 of the potential,

V = C + w i H i j w j + w i w j Λ ij mn w m w n + O((w i w i ) 6 ),

where H i j = ∂ w i ∂ w j V is the Hessian matrix, and similarly for Λ ij mn . The overall constant C can be ignored without loss of generality. Because V is an even function in w i around the minimum, we must have

w i H i j w j = D i=1 w i H i i w i .

The O(D) symmetry enforces that all weight Hessian eigenvalues to be H i i = m2 /2 for some constant m 2 . This can be seen in the O(2) case, with constants a, b, a

= b, Q ∈ O(2) such that w 1 (Q) = w 2 and w 2 (Q) = w 1 , aw 1 (Q) 2 + bw 2 (Q) 2 = aw 2 2 + bw 2 1 , this does not equal aw 2 1 + bw 2 2 , so the O(2) symmetry implies a = b.

This can be generalized to the O(D) case. For the quartic term, the requirement that V be even around the minimum gives

w i w j Λ ij mn w m w n = D i,j=1 Λ ij ij w j w j w i w i .

Similarly the O(D) symmetry implies Λ ii ii = λ/4 for some constant λ and zero for any other elements, the potential is

V = m 2 2 w i w i + λ 4 (w i w i ) 2 ,

where the numerical factors were added for convention. The power series is a good approximation in the decoupling limit which may be applicable for Residual Networks.1 For the kinetic term T , we expand in power series of the derivatives,

T = 1 2 ∂w i ∂t ∂w i ∂t - c 2 2 ∂w i ∂z ∂w i ∂z + O((∂ t w) 4 , (∂ z w) 4 ),

where the coefficient for (∂ t w)2 is fixed by the Hamiltonian kinetic energy 1 2 (∂ t w) 2 . Higher order terms in (∂ t w) 2 are negligible in the decoupling limit. If the model is robust, then higher order terms in (∂ z w) 2 can be neglected as well. 2 The Lagrangian density is3 

L = 1 2 (∂ t w) 2 - 1 2 (∂ z w) 2 - m 2 2 w 2 - λ 4 (w 2 ) 2 ,

where we have set w 2 = w i w i and absorbed c into z without loss of generality. This is precisely the Lagrangian for a scalar field in field theory. Standard results for a scalar field theory can be found in Appendix B. To account for the effect of the learning rate, we employ results from thermal field theory (Kapusta & Gale, 2006) and we identify the temperature with the learning rate η. So that now m 2 = -µ 2 + 1 4 λη 2 , with µ 2 > 0.

2.5. SPONTANEOUS SYMMETRY BREAKING:

Spontaneous symmetry breaking describes a phase transition of a deep neural network. Consider the following scalar field potential invariant under O(D ) transformations,

V(w i , η) = m 2 (η) 2 w i w i + λ 4 (w i w i ) 2 ,

where m 2 (η) = -µ 2 + 1 4 λη 2 , µ 2 > 0 and learning rate η. There exists a value of η = η c such that m 2 = 0. In the first phase, η > η c , the loss minimum is at w * 0i = 0, where

w * 0i (η) = arg min w i V(w i , η).

When the learning rate η drops sufficiently low, the symmetry is spontaneously broken and the phase transition begins. The loss minimum bifurcates at η = η c into

w * i (η < η c ) = ± -m 2 (η) λ .

This occurs when the Hessian eigenvalue becomes negative, m 2 (η) < 0, when η < η c .

This phenomenon has profound implications. It is responsible for phase transition in neural networks and generates long range correlation between representations and the desired output. Details from field theory can be found in Appendix C. Figure 1 depicts the shape of the loss rate during spontaneous symmetry breaking with a single weight w, and the orthogonal group O(D ) is reduced to a reflection symmetry O(1) = {1, -1} such that w(Q) = ±w. At η > η c , the loss rate has a loss minima at point A. When the learning rate decreases, such that η < η c , the critical point at A becomes unstable and new minima with equal loss rate are generated. The weight must go through B to get to the new minimum C. If the learning rate is too small, the weight will be stuck near A. This explains why a cyclical learning rate can outperform a monotonic decreasing learning rate (Smith & Topin, 2017) .

Because the loss rate is invariant still to the sponteneously broken symmetry, any new minima generated from spontaneous symmetry breaking must have the same loss rate. If there is a unbroken continuous symmetry remaining, there would be a connected loss rate surface corresponding to the new minima generated by the unbroken symmetry. Spontaneous symmetry breaking splits the weights into two sets, w → (π, σ). The direction along this degenerate minima in weight space corresponds to π. And the direction in weight space orthogonal to π is σ. This has been shown experimentally by Goodfellow et al. (2014) in Figure 19 . We show the case for the breaking of O(3) to O(2) in Figure 2 . 

In this section we show that spontaneous symmetry breaking occurs in neural networks. First, we show that learning by deep neural networks can be considered solely as breaking the symmetries in the weights. Then we show that some non-linear layers can preserve symmetries across the nonlinear layers. Then we show that weight pairs in adjacent layers, but not within the same layer, is approximately an invariant under the remnant symmetry leftover by the non-linearities. We assume that the weights are scalar fields invariant under the affine Aff(D ) group for some D and find that experimental results show that deep neural networks undergo spontaneous symmetry breaking.

Theorem 1: Deep feedforward networks learn by breaking symmetries Proof: Let A i be an operator representing any sequence of layers, and let a network formed by applying A i repeatedly such that

x out = ( M i=1 A i )x in . Suppose that A i ∈ Aff(D)

, the symmetry group of all affine transformations. We have L = D i=1 A i , where L ∈ Aff(D). Then x out = Lx in for some L ∈ Aff(D) and x out can be computed by a single affine transformation L. When A i contains a nonlinearity for some i, this symmetry is explicitly broken by the nonlinearity and the layers learn a more generalized representation of the input. Now we show that ReLU preserves some continuous symmetries.

Theorem 2: ReLU reduces the symmetry of an Aff(D) invariant to some subgroup Aff(D ), where D < D. Proof: Suppose R denotes the ReLU operator with output y t and Q t ∈ Aff(D) acts on the input x t , where R(x) = max(0, x). Let x T x be an invariant under Aff(D) and let x T = (γ, ν), ν < 0 and γ > 0. Let a = Rx = (γ, 0). Then a T a = x T RRx = γ T γ. Then a T a is an invariant under Aff(D ) where D = dim γ. Note that γ i can be transformed into a negative value as it has passed the ReLU already.

Corollary If there exists a group G that commutes with a nonlinear operator R, such that QR = RQ, for all Q ∈ G, then R preserves the symmetry G.

Definition: Remnant Symmetry If Q t ∈ G commutes with a non-linear operator R t for all Q t , then G is a remnant symmetry at layer t.

For the loss function L i (X i , Y i , W, Q) to be invariant, we need the predicted output y T to be covariant with x i . Similarly for an invariant loss rate L x,t we require y t to be covariant with x t . The following theorem shows that a pair of weights in adjacent layers can be considered an invariant for power series expansion.

Theorem 3: Neural network weights in adjacent layers form an approximate invariant Suppose a neural network consists of affine layers followed by a continuous non-linearity, R t , and that the weights at layer t,

W t (Q t ) = Q t W t Q -1 t , and that Q t ∈ H is a remnant symmetry such that Q t R t = R t Q t .

Then w t w t-1 can be considered as an invariant for the loss rate.

Proof: Consider x(Q t ) = Q t x t , then y t (Q t ) = R t W t (Q t )x t (Q t ) = R t Q t W t Q -1 t Q t x t = R t Q t W t x t = Q t R t W t x t ,

where in the last line

Q t R t = R t Q t was used, so y t (Q t ) = Q t y t is covariant with x t . Now, x t = R t-1 W t-1 x t-1 , so that y t (Q t ) = Q t R t W t R t-1 W t-1 x t-1 . The pair (R t W t )(R t-1 W t-1

) can be considered an invariant under the ramnant symmetry at layer t. Let w t = R t W t -R t W * t . Therefore w t w t-1 is an invariant. In the continuous layer limit, w t w t-1 tends to w(t) T w(t) such that w(t) is the first layer and w(t) T corresponds to the one after. Therefore w(t) can be considered as D scalar fields under the remnant symmetry. The remnant symmetry is not exact in general. For sigmoid functions it is only an approximation. The crucial feature for the remnant symmetry is that it is continuous so that strong correlation between inputs and outputs can be generated from spontaneous symmetry breaking. In the following we will only consider exact remnant symmetries. We will state the Goldstone Theorem from field theory without proof.

Theorem (Goldstone) For every spontaneously broken continuous symmetry, there exist a weight π with zero eigenvalue in the Hessian m 2 π = 0. In any case, we will adhere to the case where the remnant symmetry is an orthogonal group O(D ). Note that W is a D × D matrix and D < D. We choose a subset Γ ∈ R D of W such that Γ T Γ is invariant under Aff(D ). Now that we have an invariant, we can write down the Lagrangian for a deep feedforward network for the weights responsible for spontaneous symmetry breaking.

The Lagrangian for deep feedforward networks in the decoupling limit Let

EQUATION

)

Now we can use standard field theory results and apply it to deep neural networks. A review for field theory is given in Appendix B. The formalism for spontaneous symmetry breaking is given in Appendix C.

4. MAIN RESULTS:

In this section we assume that the non-linear operator is a piecewise linear function such as ReLU and set R = I to be the identity and restrict our attention to the symmetry preserving part of R (see theorem 2). Our discussion also applies to other piecewise-linear activation functions. According to the Goldstone theorem, spontaneous symmetry breaking splits the set of weight deviations γ into two sets (σ, π) with different behaviors. Weights π with zero eigenvalues and a spectrum dominated by small frequencies k in its correlation function. 4 The other weights σ, have Hessian eigenvalues µ 2 as the weights before the symmetry is broken. In Appendix C, a standard calculation in field theory shows that the correlation functions of the weights have the form

EQUATION

where

ω 0 = |k| 2 + m 2 σ,π , m 2 σ = -µ 2 σ + 1 4 λη 2 and m 2 π = 1 4 λη 2 .

The correlation function of π approaches infinity as k → 0, as m 2 π → 0. As the loss is minimized, this corresponds to large correlation between representations and the desired output over all sample space and layers t.

Spontaneous symmetry breaking and the information bottleneck The neural network undergoes a phase transition out of the information bottleneck via spontaneous symmetry breaking described in Section 2.5. Before the phase transition, the weights γ have positive Hessian eigenvalues m 2 . After the phase transition, weights π with zero Hessian eigenvalues are generated by spontaneous symmetry breaking. The correlation function for the π weights is concentrated around small values of |k|, see Equation ( 4), with ω 0 = |k| for any t. This corresponds to a highly correlated representations across the sample (input) space and layers. Because the loss is minimized, the feature maps across the network is highly correlated with the desired output. And a large correlation across the sample space means that the representations are independent of the input. This is shown in Figure 2 of Shwartz-Ziv & Tishby (2017). After phase transition, I(Y ; T ) 1 bit for all layers T , and I(X; T ) is small even for representations in early layers.

Gradient variance explosion:

It has been shown that the variance in weight gradients in the same layer grow by an order of magnitude during the end of training (Shwartz-Ziv & Tishby, 2017) . We also connect this to spontaneous symmetry breaking. As two sets of weights, (σ, π) are generated with different distributions. Considering them as the same object would result in a larger variance.

Robustness of deep neural networks:

We find that neural networks are resilient to overfitting. Recall that the fluctuation in the weights can arise from sampling noise. Then (∂ z w i ) 2 can be a measure of model robustness. A small value denotes the weights' resistance to sampling noise. If the network were to overfit, the weights would be very sensitive to sampling error. After spontaneous symmetry breaking, weights at the loss minimum with zero eigenvalues obey the Klein-Gordon equation with m

2 π = 0, (∂ z π) 2 = (∂ z π * )(∂ z π) = |k| 2 , π = exp(iωt -ik • z).

The singularity in the correlation function suggests |k| 2 0. The zero eigenvalue weights provide robustness to the model. Zhang et al. (2017) referred to this phenomenon as implicit regularization.

5. CONCLUDING REMARKS:

In this work we solved one of the most puzzling mysteries of deep learning by showing that deep neural networks undergo spontaneous symmetry breaking. This is a first attempt to describe a neural network with a scalar quantum field theory. We have shed light on many unexplained phenomenon observed in experiments, summarized in Appendix A.

One may wonder why our theoretical model works so well explaining the experimental results with just two parameters. It is due to the decoupling limit such that a power series in the loss function is a good approximation to the network. In our case, the two expansion coefficients are the lowest number of possible parameters that is able to describe the phase transition observed near the end of training, where the performance of the deep network improves drastically. It is no coincidence that our model can explain the empirical observations after the phase transition. In fact, our model can describe, at least qualitatively, the behaviors of phase transition in networks that the decoupling limit may not apply to. This suggests that the interactions with nearby layers are responsible for the phase transition.

A VALIDATION OF PROPOSED FRAMEWORK FOR NEURAL NETWORKS IN THE LITERATURE:

In this section we summarize other experimental findings that can be explained by the proposed field theory and the perspective of symmetry breaking. Here Q ∈ G acts on the the input and hidden variables x, h, as Qx, Qh.

• The shape of the loss function after spontaneous symmetry breaking has the same shape observed by Goodfellow et al. (2014) towards the end of training, see Figure 1 . • The training error typically drops drastically when learning rate is decreased. This occurs when the learning rate drops below η c , forcing a phase transition so that new minima develop. See Figure 1 . • A cyclical learning rate (Smith & Topin, 2017) helps to get to the new minimum faster, see Section 2.5. • Stochasticity in gradient descent juggles the loss function such that the weights are no longer at the local maximum of Figure 1 . A gradient descent step is taken to further take the weights towards the local minimum. Stochasticity helps the network to generalize better. • When the learning rate is too small to move away from A in Figure 1 . PReLU's (He et al., 2015b) could move the weight away from A through the training of the non-linearity. This corresponds to breaking the symmetry explicitly in Theorem 1. • Results from Shwartz-Ziv & Tishby (2017) are due to spontaneous symmetry breaking, see Section 4. • Deep neural networks can train on random labels with low training loss as feature maps are highly correlated with their respective desired output. Zhang et al. (2017) observed that a deep neural network can achieve zero training error on random labels. This shows that small Hessian eigenvalues is not the only condition that determines robustness. • Identity mapping outperforms other skip connections (He et al., 2016) is a result of the residual unit's output being small. Then the residual units can be decoupled leading to a small λ and so it is easier for spontaneous symmetry breaking to occur, from m 2 = -µ 2 + 1 4 λη 2 . • Skip connection across residual units breaks additional symmetry. Suppose now an identity skip connection connects x 1 and the output of F 2 . Now perform a symmetry transformation on x 1 and x 2 , Q 1 and Q 2 ∈ G , respectively. Then the output after two residual untis is

Qx 3 = Q 1 x 1 + Q 2 x 2 + Q 2 F 2 . Neither Q = Q 1 nor Q = Q 2 can

satisfy the covariance under G. This is observed by Orhan & Pitkow (2017) . • The shattered gradient problem (Balduzzi et al., 2017) . It is observed that the gradient in deep (non-residual) networks is very close to white noise. This is reflected in the exponential in Equation ( 7). This effect on ResNet is reduced because of the decoupling limit λ → 0. This leads to the weight eigenvalues m 2 being larger in non-residual networks owing to m 2 = -µ 2 + 1 4 λη 2 . And so a higher oscillation frequency in the correlation function.

• In recurrent neural networks, multiplicative gating (Yuhuai et al., 2016) combines the input

x and the hidden state h by an element-wise product. Their method outperforms the method with an addition x+h because the multiplication gating breaks the covariance of the output.

A transformation Qx * Qh = Q(x * h), whereas for addition the output remains covariant Qx + Qh = Q(x + h).

B REVIEW OF FIELD THEORY:

In this section we state the relevant results in field theory without proof. We use Lagrangian mechanics for fields w(x, t). Equations of motion for fields are the solution to the Euler-Lagrange equation, which is a result from the principle of least action. The action, S, is

S[w] = L(t, w, ∂ t w)dt,

where L is the Lagrangian. Define the Lagrangian density

L(t) = L(z, t, w, ∂ z,t w)dz.

The action in term of the Lagrangian density is

S[w] = L(z, t, w, ∂ z,t w)dzdt.

The Lagrangian can be written as a kinetic term T , and a potential term V (loss function),

L = T -V

For a real scalar field w(x, t),

T = 1 2 ∂w ∂t 2 - 1 2 ∂w ∂z 2 = 1 2 (∂ t w) 2 - 1 2 (∂ z w) 2

where we have set the constant c 2 = 1 without loss of generality. The potential for a scalar field that allows spontaneous symmetry breaking has the form

V = m 2 2 w 2 + λ 4 w 4 .

In the decoupling limit, λ → 0, the equation of motion for w is the Klein-Gordon Equation

[(∂ t ) 2 -(∂ z ) 2 -m 2 ]w = 0.

In the limit of m 2 → 0, the Klein-Gordon Equation reduces to the wave equation with solution

w(z, t) = e i(ωt-k•z) ,

where i = √ -1.

One can treat w as a random variable such that the probability distribution (a functional) of the scalar field w(z, t) is p[w] = exp(-S[w])/Z, where Z is some normalizing factor. The distribution peaks at the solution of the Klein-Gordon equation since it minimizes the action S. Now we can define the correlation function between w(z 1 , t 1 ) and w(z 2 , t 2 ), w(z 1 , t 1 )w(z 2 , t 2 ) = 1 Z w(z 1 , t 1 )w(z 2 , t 2 )e -S[w] Dw,

where Dw denotes the integral over all paths from (z 1 , t 1 ) to (z 2 , t 2 ). In the decoupling limit λ → 0, it can be shown that exp(-S[w]) = exp -w(∂ 2 t -∂ 2 z + m 2 )w dzdt ,

where Stokes theorem was used and the term on the boundary of (sample) space is set to zero. The above integral in the exponent is quadratic in w and the integral over Dw can be done in a similar manner to Gaussian integrals. The correlation function of the fields across two points in space and time is w(z 1 , t 1 )w(z 2 , t 2 ) = G(z 1 , t 1 , z 2 , t 2 ), where G(z 1 , t 1 , z 2 , t 2 ) is the Green's function to the Klein-Gordon equation, satisfying

(∂ 2 t -∂ 2 z + m 2 )G(z 1 , t 1 , z 2 , t 2 ) = δ(z 1 -z 2 )δ(t 1 -t 2 )

. The Fourier transformation of the correlation function is

G(ω, k) = 1 ω 2 -|k| 2 -m 2 , m 2 > 0.

An inverse transform over ω gives

G(t, k) = i 2ω 0 e -iω0t ,

with ω 2 0 = |k| 2 + m 2 . 

EQUATION

)

When m 2 = -µ 2 + 1 4 λη 2 < 0, it can be shown that in this case the loss minimum is no longer at γ i = 0, but it has a degenerate minima on the surface such that i (γ i ) 2 = v, where v = -m 2 /λ. Now we pick a point on this loss minima and expand around it. Write γ i = (π k , v + σ), where k ∈ {1, . . . , D -1}. Intuitively, the π k fields are in the subspace of degenerate minima and σ is the field orthogonal to π. Then it can be shown that the Lagrangian can be written as

L = T π + T σ -V π -V σ -V πσ ,

where, in the weak coupling limit λ → 0,

EQUATION

:

The output of a residual unit is yt = xt + Ft(xt, W1t, W2t), with Ft xt and Ft(xt, W1t, W2t) = Rt2Wt2Rt1Wt1xt. After centering the loss minimum at the origin with Equation

and suitably normalizing, we can rewrite this as Ft = aw T w, with a 1 in the decoupling limit. SeePeskin & Schroeder (1995) for the normalization of a scalar field.

The kinetic term T is not invariant under transformation Q(t). To obtain invariance ∂tw i is to be replaced by the covariant derivative Dtw i so that (Dtw i ) 2 is invariant under Q(t)(Peskin & Schroeder, 1995). The covariant derivative is Dtwi (z, t, Q(t)) = ∂tw i (z, t, Q(t)) + αB(z, t, Q(t)) i j w j (z, t, Q(t)), with B(z, t, Qt) = Q(t)B(z, t)Q(t) -1. The new fields B introduced for invariance is not responsible for spontaneous symmetry breaking, the focus of this paper. So we will not consider them further.

Formally, the ∂zw term should be part of the potential V, as T contains only ∂tw terms. However we adhere to the field theory literature and put the ∂zw term in T with a minus sign.

The correlation functions πi(z, t)πi(z , t ) for i ∈ {1, . . . , D }, is a measure of similarity of the weight across at z and z . A singularity in frequency domain at |k| = 0 corresponds to a high correlation between the weights across all sample space z = (X, Y) and layers t. This can be interpreted as the correlation length.

Figures and Tables:

Figure 1: The characteristics of the loss rate with spontaneous symmetry breaking. The dashed line corresponds to the symmetric phase, while the solid line corresponds to the broken phase. Only at w = 0 the reflection symmetry w(Q) = ±w is respected. Here, w = σ.

Figure 2: The loss after spontaneous symmetry breaking from othogonal group O(3) to O(2). The symmetry transformations of O(2), which are 1-D rotations, form the degenerate minima.

Peer review (iclr2018):
 In this paper, an number of very strong (even extraordinary) claims are made: * The abstract promises "a framework to understand the unprecedented performance and robustness of deep neural networks using field theory." * Page 8 states that this is "This is a first attempt to describe a neural network with a scalar quantum field theory." * Page 2 promises the use of the "Goldstone theorem" (no less) to understand phase transition in deep learning * It also claim that many "seemingly different experimental results can be explained by the presence of these zero eigenvalue weights." * Three important results are stated as "theorem", with a statement like "Deep feedforward networks learn by breaking symmetries" proven in 5 lines, with no formal mathematics. These are extraordinary claims, but when reaching page 5, one sees that the basis of these claims seems to be the Lagrangian of a simple phi-4 theory, and Fig. 1 shows the standard behaviour of the so-called mexican hat in physics, the basis of the second-order transition. Given physicists have been working on neural network for more than three or four decades, I am surprise that this would enough to solve all these problems! I tried to understand these many results, but I am afraid I cannot really understand or see them. In many case, the explanation seems to be a vague analogy. These are not without interest, and maybe there is indeed something deep in this paper, but it is so far hidden by the hype. Still, I fail to see how the fact that phase transitions and negative direction in the landscape is a new phenomena, and how it explains all the stated phenomenology. Beside, there are quite a lot of things known about the landscape of these problems Maybe I am indeed missing something, but i clearly suspect the authors are simply overselling physics results. I have been wrong many times, but I beleive that the authors should probably precise their claim, and clarify the relation between their results and both the physics AND statistics litterature, or better, with the theoretical physics litterature applied to learning, which is ---astonishing-- absent in the paper. About the content: The main problem for me is that the whole construction using field theory seems to be used to advocate for the appearence of a phase transition in neural nets and in learning. This rises three comments: (1) So we really need to use quantum field theory for this? I do not see what should be quantum here (despite the very vague remarks page 12 "WHY QUANTUM FIELD THEORY?") (2) This is not new. Phase transitions in learning in neural nets are being discussed since aboutn 40 years, see for instance all the pionnering work of Sompolinky et al. one can see for instance the nice review in https://arxiv.org/abs/1710.09553 In non aprticular order, phase transition and symmetry breaking are discussed in * "Statistical mechanics of learning from examples", Phys. Rev. A 45, 6056 – Published 1 April 1992 * "The statistical mechanics of learning a rule", Rev. Mod. Phys. 65, 499 – Published 1 April 1993 * Phase transitions in the generalization behaviour of multilayer neural networks http://iopscience.iop.org/article/10.1088/0305-4470/28/16/010/meta * Note that some of these results are now rigourous, as shown in "Phase Transitions, Optimal Errors and Optimality of Message-Passing in Generalized Linear Models", https://arxiv.org/abs/1708.03395 * The landscape of these problems has been studied quite extensivly, see for instance "Identifying and attacking the saddle point problem in high-dimensional non-convex optimization", https://arxiv.org/abs/1406.2572 (3) There is nothing particular about deep neural net and neural nets about this. Negative direction in the Hessian in learning problems appears in matrix and tensor factorizaion, where phase transition are well understood (even rigorously, see for instance, https://arxiv.org/abs/1711.05424 ) or in problems such as unsupervised learning, as e.g.: https://journals.aps.org/prl/abstract/10.1103/PhysRevLett.86.2174 https://journals.aps.org/pre/pdf/10.1103/PhysRevE.50.1766 Here are additional comments: PAGE 1: * "It has been discovered that the training process ceases when it goes through an information bottleneck (ShwartzZiv & Tishby, 2017)". While this paper indeed make a nice suggestion, I would not call it a discovery yet as this has never been shown on a large network. Beside, another paper in the conference is claiming exacly the opposite, see : "On the Information Bottleneck Theory of Deep Learning". This is still subject of discussion. * "In statistical terms, a quantum theory describes errors from the mean of random variables. " Last time I studied quantum theory, it was a theory that aim to explain the physical behaviours at the molecular, atomic and sub-atomic levels, usinge either on the wave function (Schrodinger) or the Matrix operatir formalism (Hesienbger) (or if you want, the path integral formalism of Feynman). It is certainly NOT a theory that describes errors from the mean of random variables. This is, i beleive, the field of "statistics" or "probability" for correlated variables. It is certianly used in physics, and heavily both in statistical physics and in quantum thoery, but this is not what the theory is about in the first place. Beside, there is little quantum in this paper, I think most of what the authors say apply to a statistical field theory ( https://en.wikipedia.org/wiki/Statistical_field_theory ) * "In the limit of a continuous sample space, the quantum theory becomes a quantum field theory." Again, what is quantum about all this? This true for a field theory, as well for continous theories of, say, mechanics, fracture, etc... PAGE 2: * "Using a scalar field theory we show that a phase transition must exist towards the end of training based on empirical results." So it is a scalar classical field theory after all. This sounds a little bit less impressive that a quantum field theory. Note that the fact that phase transition arises in learning, and in a statistical theory applied to any learning process, is an old topic, with a classical litterature. The authors might be interested by the review "The statistical mechanics of learning a rule", Rev. Mod. Phys. 65, 499 – Published 1 April 1993 PAGE 8: * "In this work we solved one of the most puzzling mysteries of deep learning by showing that deep neural networks undergo spontaneous symmetry breaking." I am afraid I fail to see what is so mysterious about this nor what the authors showed about it. In any case, gradient descent break symmetry spontaneously in many systems, including phi-4, the Ising model or (in learning problems) the community detection problem (see eg https://journals.aps.org/prx/abstract/10.1103/PhysRevX.4.011047). I am afraid I miss what is new there... * "This is a first attempt to describe a neural network with a scalar quantum field theory." Given there seems to be little quantum in the paper, I fail to see the relevance of the statement. Secondly, I beleive that field theory has been used, many times and in greater lenght, both for statistical and dynamical problems in neural nets, see eg. * http://iopscience.iop.org/article/10.1088/0305-4470/27/6/016/meta * https://arxiv.org/pdf/q-bio/0701042.pdf * http://www.lps.ens.fr/~derrida/PAPIERS/1987/gardner-zippelius-87.pdf * http://iopscience.iop.org/article/10.1088/0305-4470/21/1/030/meta * https://arxiv.org/pdf/cond-mat/9805073.pdf