# Implicit Bias of Deep Learning in the Large Learning Rate Phase: A Data Separability Perspective
## 4 Theoretical Results
### Non-Convex Optimization
More precisely, when \(\eta\) belongs to this phase, \(T>0\) exists such that the output function \(f_{t}\) and the eigenvalue of NTK \(\lambda_{t}\) update in the following way:  
1. \(\mathcal{L}_{t}\) keeps increasing when \(t<T\). 2. _After the \(T\) step and its successors, the loss decreases, which is equivalent to:_ \[|f_{T+1}|>|f_{T+2}|\geq|f_{T+3}|\geq\ldots.\]
3. _The eigenvalue of NTK keeps dropping after the \(T\) steps:_ \[\lambda_{T+1}>\lambda_{T+2}\geq\lambda_{T+3}\geq\ldots.\]  
_Moreover, we have the inverse relationship between the learning rate and final eigenvalue of the NTK: \(\lambda_{\infty}\leq\text{lim}_{t\rightarrow\infty}\frac{4f_{t}}{\eta f_{ \text{exp}}}\) with exponential loss, or \(\lambda_{\infty}\leq\text{lim}_{t\rightarrow\infty}\frac{4f_{t}}{\eta f_{ \text{log}}}\) with logistic loss._  
We demonstrate that the catapult phase can be found in both degenerate and non-separable data through the following examples. The weight matrix is initialized by iid Gaussian distribution, i.e., \(w^{(1)},w^{(2)}\sim\mathcal{N}(0,\sigma_{w}^{2})\). For exponential loss, we adopt the setting of \(\sigma_{w}^{2}=0.5\) and \(m=1000\) while we set \(\sigma_{w}^{2}=1.0\) and \(m=100\) for logistic loss. Consider optimizing \(\mathcal{L}(w)\) using a one-hidden-layer linear network with dataset \(\{(x_{1}=[1,0],y_{1}=1)\) and \((x_{2}=[1,0],y_{2}=-1).\}\) and exponential (logistic) loss using gradient descent with a constant learning rate. Figure (a)a,c,e,g shows how the different choices of learning rate \(\eta\) change the dynamics of the loss function with exponential and logistic loss. Consider optimizing \(\mathcal{L}(w)\) using a one-hidden-layer linear network with dataset \(\{(x_{1}=[1,1],y_{1}=-1),\ (x_{2}=[1,-1],y_{1}=1),\ (x_{3}=[-1,-2],y_{1}=1)\) and \((x_{4}=[-1,1],y_{4}=1).\}\) and exponential (logistic) loss using gradient descent with a constant learning rate. Figure (b)b,d,f,h shows how the different choices of learning rate \(\eta\) change the dynamics of the loss function with exponential and logistic loss. As Figure 1 shows, in the catapult phase, the eigenvalue of the NTK decreases to a lower value than its initial point, while it remains unchanged in the lazy phase where the learning rate is small. For MSE loss, the lower value of the NTK indicates a flatter curvature given the training loss is low [16]. Yet, it is unknown whether the aforementioned conclusion can be applied to exponential and logistic loss. Through the following corollary, we show that the Hessian is equivalent to the NTK when the loss converges to a global minimum for degenerate data. Consider optimizing \(\mathcal{L}(w)\) with a one-hidden-layer linear network under Assumption 3 and exponential (logistic) loss using gradient descent with a constant learning rate. For any learning rate that loss can converge to the global minimum, the larger the learning rate, the flatter curvature the gradient descent will achieve at the end of training (see Corollary A1 in Appendix A for detail)._  
We demonstrate that the flatter curvature can be achieved in the catapult phase through Examples 3 and 4, using the code provided by [56] to measure the Hessian, as shown in Figure 3. In the lazy phase, both the curvature and eigenvalue of the NTK are independent of the learning rate at the end of training. In the catapult phase, however, the curvature decreases to a value smaller than that in the lazy phase. In conclusion, the NTK and Hessian have similar behaviours at the end of training on non-separable data. Finally, we compare our results from the catapult phase to the results with MSE loss and show the summary in Table 2.