SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Step 1. (Lemma 1 [26]) Giving a generic sequence  $u$  generated by AGD  $(v^k \in u)$ . As long as the initial point of  $u^k$  is close to saddle point  $\tilde{v}^k$ , the distance between  $u^k$  and  $\tilde{v}^k$  can be upper bounded by using the  $\rho$ -Hessian Lipschitz continuity property.
Step 2. Leveraging the negative curvature around the strict saddle point, we can project the  $u^k$  onto the two subspaces, where the first subspace is spanned by the eigenvector of  $M^{-1}G$  and the other one is spanned by the remaining eigenvectors. We use two steps to show  $\lambda_{max}(M^{-1}G) > 1$ : 1) we show that all eigenvalues of  $Q(\lambda) = [G - \lambda M]$  are real; 2)  $\exists \lambda > 1$ ,  $\det(Q(\lambda)) = 0$ .
Conclusion 1 illustrates that there exists a subspace spanned by the eigenvector of  $M^{-1}G$  whose eigenvalue is greater than 1, indicating that the sequence generated by AGD can still potentially escape from the strict saddle point by leveraging such negative curvature information (more can be found in supplementary materials).
Datasets. We evaluate our method on three tasks, i.e., image classification, objective detection, and semantic segmentation. For the image classification, we use three datasets: CIFAR-10 [20], CIFAR-100 [21], and ImageNet [33]. CIFAR-10 and CIFAR-100 contain 50000 training and 10000 testing images with 10-class and 100-class separately. ImageNet is a 1000-class task that contains 1281167 training and 50000 testing pairs. For the objective detection, we use MS-COCO dataset [22], which contains 82783, 40504, and 81434 pairs in the training, validation, and testing set separately. For the semantic segmentation, we choose PASCAL VOC 2012 [6]. Following the procedure of Zhao et al. [40], we use augmented data with the annotation of Hariharan et al. [11], resulting in 10582, 1449, and 1456 images for training, validation, and testing.
Evaluation metrics. In the classification, we use the accuracy on the testing set of each dataset [37, 23]. In the objective detection, we use the mean of Average Precision (mAP) [31] to evaluate the student model on the testing set of MS-COCO [22]. In the semantic segmentation, we use Mean Intersection over Union (mIoU) [40] to evaluate the student model on the testing set of VOC [6, 11].
Baseline methods. For the classification, we employ several popular loss functions, including fixed loss functions such as Cross Entropy loss (CE), the large-margin softmax loss (L-M softmax) [24], and the smooth 0-1 loss function (Smooth) [30] as well as dynamic loss functions, namely the adaptive robust loss function (ARLF) [3], the L2T-DFL loss function [37], stochastic loss function (SLF) [23] and ALA [16]. For objective detection, we compare our approach with the objective function set by YOLO-v3 [31]. For the semantic segmentation, we compare our approach with the objective function set by PSPNet [40].
Table 1: Results on datasets CIFAR-10 [20], CIFAR-100 [21] and ImageNet [33] for the classification task. All experiments are implemented with the same settings. The best results are highlighted in bold.
Method	CIFAR-10 [20]	CIFAR-100 [21]	ImageNet [33] length
ResNet8	ResNet20	ResNet32	WRN	ResNet8	ResNet20	ResNet32	NASNet-A
CE	87.6	91.3	92.5	96.2	60.2	67.7	69.6	73.5	-
Smooth [30]	87.9	91.5	92.6	96.2	60.5	68.0	69.9	-	-
L-M Softmax [24]	88.7	92.0	93.0	96.3	61.1	68.4	70.4	-	-
L2T-DLF [37]	89.2	92.4	93.1	96.6	61.7	69.0	70.8	-	1
ARLF [3]	89.5	91.5	92.2	95.9	60.2	67.8	69.9	-	-
SLF [23]	89.8	93.0	93.6	97.1	62.7	69.9	71.5	-	-
ALA [16]	-	-	93.2	96.7	62.2	69.5	70.9	74.6	200 [15]
Ours	90.7 ± 0.06	93.4 ± 0.18	93.8 ± 0.20	96.7 ± 0.09	63.5 ± 0.07	70.4 ± 0.03	72.0 ± 0.11	74.2	25
Implementation details. In all experiments, we optimize student models using standard stochastic gradient descent (SGD) with a learning rate of 0.1. The teacher model is trained with Adam, utilizing a learning rate of 0.001. The learning rate of DLN is set to 0.001. The teacher model is trained for 10 epochs, with redividing the training and validation data after each epoch. The validation errors in each task are explicitly reported. Our teacher model comprises a four-layer LSTM [14] with 64 neurons in the first three layers and 1 neuron in the final layer. We utilize a 1-vs-1 approach (details in supplementary materials) to process the student model's output in both classification and segmentation. We present DLN architecture for each task and ensure reliable evaluation by conducting 5 random restarts, using average results for comprehensive comparisons.
Image classification. For CIFAR-10 and CIFAR 100, we follow the SLF [23] and use architectures that include ResNet [12], and Wide-ResNet(WRN) [39] as the student model. For ImageNet, we follow the ALA [16] and use the identical NASNet-A [43]. In each experiment, the batch sizes for training and validation are set to 25 and 100, respectively. We perform a five-layer fully connected network, which contains 40 neurons in each hidden layer and 1 neuron in the output layer, as the DLN. The activation function for each hidden layer is set to Leaky-ReLU. The validation error is computed by CE.
Table 1 reports the performance of each loss function. Our approach achieves the best results on CIFAR-10 with ResNet8, ResNet20, and ResNet32, achieving  $90.70\%$ ,  $93.40\%$ , and  $93.81\%$ , respectively. On CIFAR-100, our method also outperforms baselines with an overall accuracy of  $63.50\%$ ,  $70.47\%$ , and  $72.06\%$  for ResNet8, ResNet20, and ResNet32, respectively. For WRN, our approach achieves the second-best performance, following the SLF method. The results on ImageNet illustrate that L2T-DLN improves the accuracy of the baseline by  $0.7\%$ . On ImageNet, our DLN demonstrates the second-best performance. The performance of ALA benefits from the larger length of a student learning stage ALA set (200) compared with ours (25). The ablation showed that the size of the length is positively correlated with the test accuracy and computational consumption (see Table 5).
Objective detection. In the task of objective detection, the YOLO-v3 model with a backbone of darknet-53 [31] is used in this experiment. The traditional loss in the YOLO model is a multi-part loss function, i.e.,  $\lambda_{cls}\ell_{cls} + \lambda_{conf}\ell_{conf} + \lambda_{loc}\ell_{loc}$ .  $\ell_{cls},\ell_{conf}$  and  $\ell_{loc}$  are detailed in supplementary materials. Redmon and Farhadi [31] set  $\lambda_{cls} = \lambda_{conf} = \lambda_{loc} = 1$ . In our experiment, our L2T-DLN learns to set these weights dynamically with a single-layer perceptron as DLN. The backbone of the YOLO is pre-trained on ImageNet, and we finetune the header of the YOLO. Specifically, the objective function of the student model is set to  $DLN([\ell_{cls},\ell_{conf},\ell_{loc}])$ . The validation error is computed by  $\ell_{cls} + \ell_{conf} + \ell_{loc}$ . The batch sizes for training and validation are set to 2 and 8, respectively. The length of student learning is set to 2. We take the training set and 35000 images of the validation set to train our L2T-DLN with an input size of 416*416. From Table 2, our L2T-DLN has more than  $1.6\%$  improvement with the baseline on mAP.
Semantic segmentation. The objective function of PSPNet [40] is set to  $CE(p, y) + 0.4 * CE(aux_p, y)$ , where  $p$ ,  $aux_p$ , and  $y$  denote the output of the master branch, the auxiliary branch of PSPNet, and the ground truth, respectively. For PSPNet with L2T-DLN, the objective function is set to  $DLN(p, y) + 0.4 * DLN(aux_p, y)$ , where the architecture of DLN is the one used in classification tasks. The validation error is computed by  $CE(p, y) + 0.4 * CE(aux_p, y)$ . The batch sizes for
Table 2: Objective detection on COCO [22]. DLN and original losses in YOLOV3.
Detectors	Size	mAP	FPS
YOLOV3 [31]	416	55.3	35
YOLOV3-ours	416	56.9	35
Table 3: Segmentation on VOC [6, 11]. DLN and original losses in PSPNet.
Method	mIoU
PSPNet [40]	82.6
PSPNet-ours	82.9
training and validation are set to 2 and 8, respectively. The length of student learning is set to 2. Table 3 shows that our L2T-DLN improves  $0.3\%$  compared with the baseline on mIoU.
In this subsection, we conduct ablation studies on CIFAR-10 [20] using ResNet8 to analyze the L2T-DLN synthetically. We specifically examine the proportion of training and validation data, the length of student learning stage  $(N)$ , the wrong learning rate setting, and the influence of the LSTM teacher. Furthermore, we provide the visualization of DLN at different learning stages in MNIST and CIFAR10 tasks. We assess the impact of each component by computing the test accuracy of the student model after optimizing the teacher model for 10 epochs.
The proportion of training and validation data. In L2T-DLN, the training dataset is divided into two sets: validation data and training data, with validation data serving as an unbiased estimator for model generalization. After each epoch, the dataset is redivided, allowing samples used in the validation data to be included in the training data, and vice versa. The validation ratio represents the fraction of training dataset samples exclusively used for validation. This study explores different training-validation data separations. Table 4 results indicate our performance remains stable across varying ratios due to the teacher's ability to capture short- and long-term dependencies. To make a trade-off between computational cost and accuracy, we set the ratio  $= 50\%$  for all our experiments.
Table 4: Results on different validation ratios ranging from  ${10}\%$  to  ${90}\%$  to show the impact of ratios.
Ratio	10%	25%	50%	75%	90%
Accuracy	90.41	90.53	90.70	90.68	90.35
The length of student learning. The computation of higher-order gradients in L2T-DLN (Eq. (6) and (8)) is computationally intensive and should be highlighted. Thus, this study explores the influence of the length of student learning  $(N)$  on the test accuracy and computational load in CIFAR-10 experiments using ResNet8. As shown in Table 5, the findings reveal that the test accuracy increases with the length of student learning. To make a trade-off between performance and computational cost, we suggest that a maximum length of 25 should be set for student learning. Overall, the study concludes that L2T-DLN has the potential to further improve the performance of student model with sufficient computing resources.
Table 5: Results on different lengths ranging from 1 to 75 to show the impact of the length. Time denotes the time consumption of a round of teacher learning.
Length	1	5	10	15	25	50	75
Accuracy	81.40	87.07	89.95	90.18	90.70	90.73	90.74
Time	1s	3s	5.8s	7.9s	13.4s	32s	76.7s
The influence of an LSTM teacher. As introduced above, the teacher model is similar to an optimization algorithm. Then we perform various optimizers, including Adam [19], SGD, RMSProp [13], and the LSTM teacher, to optimize the DLN and present the results in Table 6. Compared with ADAM, SGD, and RMSProp, our teacher can improve the performance of the student by  $0.48\%$ ,  $1.6\%$ , and  $0.53\%$ . We can conclude that 1) algorithms that can use the historical information, e.g., momentum, perform well; 2) the adaptability to capture and maintain short- and long-term dependencies can further enhance the loss function teaching, compared to handcrafted methods, e.g., exponentially weighted moving average [13] and moment estimation [19].
Table 6: Results on different optimizers to show the effectiveness of the LSTM teacher.
Optimizer	Adam [19]	SGD	RMSProp [13]	LSTM
Accuracy	90.17	89.05	90.12	90.65
(g)
Figure 3: Visualization of DLN loss value at different training stages in the MNIST-LeNet task (a-d) and CIFAR10-ResNet8 task (e-h). (a) & (e) initialized DLN, (b) & (f) DLN finished second teacher learning epoch, (c) & (g) DLN finished fifth teacher learning epoch, (d) & (h) DLN finished tenth teacher learning epoch (final). The X-, Y-, and Z-axis are the prediction for 0-category (denoted as prediction0), the prediction for 1-category (denoted as prediction1), and the loss value of DLN, respectively. We set the 0-category as the correct category and the 1-category as the wrong category. We can observe that the output value range of DLN initially expands and subsequently contracts. Specifically, the range shifts from  $(-0.3, 0.4)$  to  $(-2.25, -0.25)$  to  $(-0.3, -0.05)$  in MNIST, and from  $(0.8, 1.8)$  to  $(-2, 1)$  to  $(-0.175, 0)$  in CIFAR10.
Visualization. We visualize the loss value of DLN on MNIST and CIFAR-10 separately in Figure 3, which illustrates the capacity of L2T-DLN to adapt to the evolving states of students to attain improved performance. The DLN is initialized with the Kaiming normal initialization with LeakyReLU activations.
This paper introduces L2T-DLN, an adaptive model for various stages of student learning. Technically, We propose a differentiable three-stage teaching framework, asynchronously optimizing the student, DLN, and teacher. An LSTM teacher dynamically captures and retains experiences during DLN learning. Additionally, we assess L2T-DLN's ability to navigate strict saddle points using the negative curvature of their Hessian matrix. Experiments demonstrate our DLN outperforming specially designed loss functions. Nevertheless, our approach demands significant computational resources for high-order derivatives, which we aim to mitigate in future work.
This work was supported in part by National Natural Science Foundation of China (62302045, 82171965), Clinical and Translational Medical Research Fund of the Chinese Academy of Medical Sciences (2020-I2M-C&T-B-072), and Beijing Institute of Technology Research Fund Program for Young Scholars.
[43] Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8697-8710, 2018.