Title: L2T-DLN: Learning to Teach with Dynamic Loss Network
Venue: NeurIPS 2023
Accepted: yes
Keywords: Learning to teach

dynamic loss function

optimization
Abstract: With the concept of teaching being introduced to the machine learning community, a teacher model start using dynamic loss functions to teach the training of a student model. The dynamic intends to set adaptive loss functions to different phases of student model learning. In existing works, the teacher model 1) merely determines the loss function based on the present states of the student model, e.g., disregards the experience of the teacher; 2) only utilizes the states of the student model, e.g., training iteration number and loss/accuracy from training/validation sets, while ignoring the states of the loss function. In this paper, we first formulate the loss adjustment as a temporal task by designing a teacher model with memory units, and, therefore, enables the student learning to be guided by the experience of the teacher model. Then, with a Dynamic Loss Network, we can additionally use the states of the loss to assist the teacher learning in enhancing the interactions between the teacher and the student model. 
  Extensive experiments demonstrate our approach can enhance student learning and improve the performance of various deep models on real-world tasks, including classification, objective detection, and semantic segmentation scenario.

L2T-DLN: Learning to Teach with

Abstract

With the concept of teaching being introduced to the machine learning community, a teacher model start using dynamic loss functions to teach the training of a student model. The dynamic intends to set adaptive loss functions to different phases of student model learning. In existing works, the teacher model 1) merely determines the loss function based on the present states of the student model, i.e. , disregards the experience of the teacher; 2) only utilizes the states of the student model, e.g. , training iteration number and loss/accuracy from training/validation sets, while ignoring the states of the loss function. In this paper, we first formulate the loss adjustment as a temporal task by designing a teacher model with memory units, and, therefore, enables the student learning to be guided by the experience of the teacher model. Then, with a dynamic loss network, we can additionally use the states of the loss to assist the teacher learning in enhancing the interactions between the teacher and the student model. Extensive experiments demonstrate our approach can enhance student learning and improve the performance of various deep models on real-world tasks, including classification, objective detection, and semantic segmentation scenarios.

1 Introduction

In pedagogy study, teachers refine their teaching ability based on student feedback, e.g. , exam scores. Students benefited from the enhanced ability of teachers and then achieved high scores on the exam. Both teachers and students are developed in this interaction iteratively and constantly [27-29, 17]. The phenomenon is known as teaching-learning transaction [5], or learning to teach (L2T) in machine learning [7].

In L2T, a teacher model uses a dynamic loss function, which acts as an exam paper, to train and optimize the student model (as shown in Figure 1(a)). However, existing approaches adjust the loss functions by only employing a simple feedforward network as the teacher model, and neglecting the temporal nature of loss function adjustment. The disregarding of the experience accumulation for teachers ( e.g. , the ability to analyze all previous exam scores for a student), and, therefore, limits the potential of L2T. In addition, in previous works, the teacher model only focuses on the state of the student model, i.e. , training iteration number [37], training/validation accuracy [37, 16], training/validation loss[2], and the output of the student model[16, 2, 23]. However, the states of loss functions ( e.g. , the gradients concerning loss functions) are neglected, which dilutes the benefit of improving the exam paper. In other words, the teacher needs to consider that the question changes of an exam paper also influence the performance of a student.

S

θ

T

φ

τ

x

y

L m

→

L m+1

S

θ

x

y

L

ϕ

m

→

L

ϕ

m+1

T

φ

∇ϕ

m

e

m

ory

unit

(a)

(b)

In this paper, we propose an L2T framework with a Dynamic Loss Network (L2T-DLN), to address the above-mentioned issue (in Figure 1(b)). First, we adopt a Long-Short Term Memory (LSTM) model as the teacher and design a differentiable three-stage asynchronous optimization strategy. Then, to ensure the teacher model can be optimized with the state of loss functions, we design a Dynamic Loss Network (DLN) instead of using the dynamic loss function (DLF). Specifically, we start by optimizing the student model through backpropagation in the first step with a fixed DLN as the loss function. Then, compute the gradient of the validation error of the student model with respect to the DLN. Next, we input this gradient into the teacher model, and the output of the teacher model is used to update the DLN. To achieve the updating of the teacher model, we perform another round of student learning with the updated DLN and obtain the gradient of the validation error of the updated student with respect to the teacher model. Moreover, we analyze how L2T-DLN exploits the negative curvature by using a special alternating gradient descent (AGD) sequence, achieving a differentiable asynchronous optimization.

In summary, the usage of the gradient concerning DLN and the LSTM teacher model both ensure the teacher model captures and maintains short- and long-term temporal information, which can further improve the performance of loss function teaching, compared to feedforward teachers [37, 16, 23].

Our main contributions are 1) design a dynamic loss network-based teaching strategy to let the teacher model learn optimized by the gradient of DLN; 2) use LSTM as the teacher model to update the DLN with the temporal information of the teacher model; and 3) a convergence analysis of the approach, which is treated as a special AGD sequence and has the potential to escape strict saddle points. We conduct extensive experiments on a wide range of loss functions and tasks to demonstrate the effectiveness of our approach.

2 Related work

Recent work by L2T [7] provides a comprehensive view of teaching for machine learning, encompassing aspects such as training data teaching, loss function teaching, and hypothesis teaching. In contrast to previous literature on machine teaching [41, 25, 10, 38], L2T breaks the strong assumption regarding the existence of an optimal off-the-shelf student model [37]. Instead, L2T employs automatic techniques to reduce the reliance on prior human knowledge, aligning with principles such as learning to learn and meta-learning [34, 36, 42, 1]. The recent focus of L2T has been mainly on loss function teaching [37, 16, 23, 2] and training data teaching [7, 25, 35, 32, 8].

During the training of a student model, there is a variation in the distribution of predictions where earlier in the training the distribution tends to differ from that at convergence. Consequently, an adaptive loss function is crucial. Existing works [37, 16, 23, 2] formulate the loss adjustment as some independent tasks by performing a multi-layer perceptron (MLP) as the teacher model. The differences lie in the representation of dynamic loss functions and the input information of teacher models. The representation of the dynamic loss function includes the variation of handcrafted loss functions [37, 23] and neural network [16, 2]. The input information contains the training iteration number [37, 16], training/validation accuracy [37, 16], training/validation loss[2], and the output of the student model[16, 2, 23]. In detail, Wu et al. [37] trains a neural network with an attention mechanism to generate a coefficient matrix between the prediction of the student and the ground truth. Huang et al. [16] constructs the teaching-learning framework with reinforcement learning. Their teacher also employs an MLP and generates the policy gradient for a loss network. Liu and Lai [23] utilizes a teacher model to guide the selection and combination of handcrafted loss functions. Baik et al. [2] performs a teacher to generate two weights for each layer of a loss network, and then updates the parameters of the loss network by affine transformation with the two weights.

Assigning weights to different data points has been widely investigated in the literature, where the weights can be either continuous [18] or binary [7]. In detail, Fan et al. [7] proposed a learning paradigm where a teacher model guides the training of the student model. Based on the collected information, the teacher model provides signals to the student model, which can be the weights of training data. Liu et al. [25] leveraged a teaching way to speed up the training, where the teacher model selects the training data balancing the trade-off between the difficulty and usefulness of the data. Fan et al. [8] inputs internal states of the student model, e.g. , feature maps of the student model, to the teacher model and obtains the output of the teacher as the weight for corresponding training data.

3 Methodology

In this section, we overview our L2T-DLN in Section 3.1, introduce the corresponding framework for student learning in Section 3.2, describe the DLN learning framework in Section 3.3, and discuss teacher learning in Section 3.4.

3.1 Overview

Our L2T-DLN is a differentiable teaching framework that enhances the performance of a student model. The L2T-DLN contains stages: (I) student learning, which optimizes a student model S θ with parameter θ ; (II) DLN learning, which optimizes the DLN L ϕ with parameter ϕ ; (III) teacher learning, which optimizes the teacher model T φ with parameter φ .

Starting from S 0 θ , in stage (I) , we optimize the student model S 0 θ → S N θ on training data x train by leveraging the DLN L 0 ϕ as the loss function, where N denotes the number of iterations in a student learning stage. In stage (II) , we compute the error e val of the student model S N θ on validation data x val , and then determine the gradient ∇ ϕ 0 = ∂e val /∂ϕ 0 . This gradient is then given to the teacher model, and the DLN is updated as ϕ 1 = ϕ 0 + g 0 , where g 0 indicates the output of the teacher model T 0 φ . In stage (III) , we first train the student model S N θ → S 2 N θ with the updated DLN L 1 ϕ . Then, we obtain the validation error e val of the student model S 2 N θ and optimize the teacher model by backpropagation (BP) based on e val . The objective function of our L2T-DLN is:

Our goal is to achieve θ ( M + K ) N , ϕ M and φ K , where M and K denote the number of iterations of DLN and teacher learning, respectively (details are shown in Figure 2).

3.2 Student learning

For a given task, we define the input and output space as X and Y , respectively. The student model is then denoted by S θ : X → Y . Our student learning involves minimizing the output value of the

Teacher

learning

DLN

learning

S

θ

0

BP

T

φ

0

h 0

ϕ

0

θ

0

θ

N

+

S

θ

N

h 1

ϕ

1

θ

2N

S

θ

2N

T

φ

1

+

…

…

g 0

Student

learning

∇ϕ

0

∇ϕ

1

BP

φ

1

φ

0

…

θ

M+K N

…

φ

K

…

ϕ

M

L

ϕ

0

L

ϕ

1

L

ϕ

1

fix

BP

BP

x train . Note, w is a weight parameter of x , L m ϕ is the loss function, and m is the m th iteration for the DLN. During each stage of student learning, we iteratively train the student model N times with L m ϕ . The optimization of the student model during the current stage is:

where η denotes the learning rate of the student model. In our framework, the L m ϕ that with learnable parameters are optimized in different student learning stages for providing seemly guidance.

3.3 DLN learning

After the student learning stage ( e.g. , S 0 θ → S N θ with L 0 ϕ ), we use the teacher model T 0 φ to adjust the DLN parameters ϕ ( e.g. , ϕ 0 → ϕ 1 ). To enable the temporal property of DLN, we use an LSTM to transform ϕ dynamically:

where γ denotes the learning rate of DLN and ∇ ϕ 0 represents the gradient of the validation error e val of S θ with respect to ϕ 0 .

Considering the gap between training data and validation data, we employ the Reverse-Mode Differentiation (RMD) to calculate ∇ ϕ 0 . The RMD involves performing the SGD process in reverse order from N to 1 , as depicted by the black lines in Figure 2. According to Eq. (2), the gradient of the validation error e val ( S N θ ) with respect to θ N can be calculated as follows:

Then looping backward from N . At each step i = { N -1 , · · · , 1 } , the calculation of gradient is shown as:

At the same time, the gradient of e val ( S N θ ) with respect to ϕ 0 is accumulated as:

Reverting backward from N to 1 , we get ∇ ϕ 0 .

Taking N = 1 , we rewrite Eq. (6) by using the gradient, e.g., ∇ ϕ 0 , as:

where ∇ ϕ 0 contains the information of both training and validation data, ( ∂S 1 θ ( x val ) /∂θ 1 and ∂S 0 θ ( x ) /∂θ 0 ). As these pieces of information are dependent on each other, they are integrated into temporal changes of θ . The gradient concerning ϕ 0 achieves holistic information integration throughout the learning process, facilitated by prior knowledge (chain rule). Employing the gradient concerning the loss allows the teacher model to concentrate on capturing and preserving crucial information from gradients, negating the need for supplementary handling of dispersed states. Therefore, compared to the state of the student, e.g. , training/validation error, prediction, and numbers of iteration, ∇ ϕ 0 provides more information to promote deep interaction between DLN, the teacher, and the student.

3.4 Teacher learning

To retain the teaching experience, we utilize an LSTM to play the role of teacher. The LSTM utilizes a long-term memory unit, called the cell, to allow itself to maintain effective and long-term dependencies to make predictions, both in the current and future [9]. We employ the coordinate-wise manner [1] to process each value of the input information independently. Different behavior on each coordinate is achieved by using separate activations for each input value. This property leads the student to converge to a good solution.

To evaluate the teacher model, we perform another student learning ( e.g. , S N θ → S 2 N θ ) with the new DLN ( e.g. , L 1 ϕ ). The RMD is also utilized to calculate the gradient of validation error e val ( S 2 N θ ) with respect to the teacher model parameters φ 0 . We represent the computation using red lines in Figure 2. To obtain ∇ φ 0 , we loop the SGD process backward from 2 N to N +1 with the updated DLN L 1 ϕ according to Eq. (4), (5) and (6). At each step i = { 2 N -1 , · · · , N +1 } , the gradient ∇ φ 0 is updated as:

The process of L2T-DLN is summarized in Algorithm 1.

4 Convergence Analysis

Since our L2T-DLN is updated asynchronously, we can only access partial second-order information at each training stage. For example, given a quadratic objective function, while fixing one part of L2T-DLN, the problem is strongly convex with respect to the other part, but the entire problem is nonconvex. Even if the iterates converge for each part to the corresponding minimum points, the stationary point could still be a saddle point for the overall objective function [26]. Therefore, the analysis of how L2T-DLN exploits the negative curvature is necessary.

The Alternating Gradient Descent (AGD) [4] algorithm only updates partial variables of vector, e.g. , v = { θ, φ + ϕ } , which belongs to a subset of the feasible set. From the mean value theorem, we can

Algorithm 1 Obtaining the optimal student, DLN and teacher in L2T-DLN.

1: Hyperparameter: length of the student training N , total number of iterations for DLN learning M , total number of iterations for teacher learning K . 2: Input: Random initialization parameter θ 0 , ϕ 0 , and φ 0 . 3: m = 0 4: for k = (1 , · · · , K ) do 5: for i = (( m + k ) N +1 , · · · , ( m + k +1) N ) do ▷ A student learning stage 6: Conduct student model training step via Eq. (2) ( S i -1 θ → S i θ ). 7: end for 8: ∇ ϕ m = 0 , compute ∇ θ ( m + k +1) N via Eq. (4). 9: for i = (( m + k +1) N, · · · , ( m + k ) N +1) do ▷ Reversely calculating the gradient ∇ ϕ 10: Update ∇ ϕ m via Eq. (6). 11: Compute ∇ θ i via Eq. (5). 12: end for 13: update ϕ m → ϕ m +1 via Eq. (3). 14: m = m +1 ▷ Updating m 15: for i = (( m + k ) N +1 , · · · , ( m + k +1) N ) do ▷ A student learning stage 16: Conduct student model training step via Eq. (2) ( S i -1 θ → S i θ ). 17: end for 18: ∇ φ k = 0 , ∇ ϕ m = 0 , compute ∇ θ ( m + k +1) N via Eq. (4) 19: for i = (( m + k +1) N, · · · , ( m + k ) N +1) do ▷ Reversely calculating the gradient ∇ φ 20: update ∇ ϕ m via Eq. (6). 21: Update ∇ φ k via Eq. (8). 22: Compute ∇ θ i via Eq. (5). 23: end for 24: Update φ k → φ k +1 using ∇ φ k via gradient based optimization algorithm. 25: end for 26: output: θ ( M + K ) N , ϕ M , φ K

express the AGD rule of updating variables by assuming v (0) = 0 as follows, with e denoting the validation error.

The right-hand side of Eq. (9) not only contains the second order information of the previous point, i.e. , [ v 2 kN 1 , v k 2 ] , but also the one of the most recently updated point, i.e. , [ v 2( k +1) N 1 , v k 2 ] .

Different from traditional AGD, the dynamic system in L2T-DLN takes the first-order information to update the student and the second-order information to update the teacher. Specifically, ▽ 1 e ( v 2 kN 1 , v k 2 ) = ▽ e ( v 2 kN 1 , v k 2 ) and ▽ 2 e ( v 2( k +1) N 1 , v k 2 ) = ▽ 2 e ( v 2( k +1) N 1 , v k 2 ) . These represent the main challenges in understanding the behavior of the sequence generated by the AGD algorithm.

Although the higher-order information is divided into two parts, we can still characterize the recursion of the iterates around strict saddle points v ∗ . We can also split H as two parts, which are

and obviously, we have H = H u + H l .

Then recursion Eq. (9) can be written as

where △ k u ≜ ∫ 1 0 ( H k u ( v ) -H u ) dv , △ k l ≜ ∫ 1 0 ( H k l ( v ) -H l ) dv . However, it is still unclear from Eq. (11) how the iteration evolves around the strict saddle point. To highlight ideas, let us define

It can be observed that M is a lower triangular matrix where the diagonal entries are all 1s; therefore it is invertible. After taking the inverse of matrix M on both sides of Eq. (11), we can obtain

Our goal of analyzing the recursion of v k becomes to find the maximum eigenvalue of M -1 G . With the help of the matrix perturbation theory, we can quantify the difference between the eigenvalues of matrix H that contains the negative curvature and matrix M -1 G that we are interested in analyzing. With the gradient Lipschitz constants { ˜ C k } , we set L max ≜ max { C k , ˜ C k , ∀ k } ≤ C and give the following conclusion.

Conclusion 1. Let H ≜ ▽ 2 e ( x ) denote the Hessian matrix at an ϵ -second-order stationary solution (SS2) v ∗ where λ min ( H ) ≤ -γ and γ > 0 . We have

The proof of Conclusion 1 contains the following steps:

Step 1. (Lemma 1 [26]) Giving a generic sequence u generated by AGD ( v k ∈ u ). As long as the initial point of u k is close to saddle point ˜ v k , the distance between u k and ˜ v k can be upper bounded by using the ρ -Hessian Lipschitz continuity property.

Step 2. Leveraging the negative curvature around the strict saddle point, we can project the u k onto the two subspaces, where the first subspace is spanned by the eigenvector of M -1 G and the other one is spanned by the remaining eigenvectors. We use two steps to show λ max ( M -1 G ) > 1 : 1) we show that all eigenvalues of Q ( λ ) = [ G -λM ] are real; 2) ∃ λ > 1 , det ( Q ( λ )) = 0 .

Conclusion 1 illustrates that there exists a subspace spanned by the eigenvector of M -1 G whose eigenvalue is greater than 1, indicating that the sequence generated by AGD can still potentially escape from the strict saddle point by leveraging such negative curvature information (more can be found in supplementary materials).

5 Experiments

5.1 Experimental setup

Datasets. We evaluate our method on three tasks, i.e. , image classification, objective detection, and semantic segmentation. For the image classification, we use three datasets: CIFAR-10 [20], CIFAR-100 [21], and ImageNet [33]. CIFAR-10 and CIFAR-100 contain 50000 training and 10000 testing images with 10-class and 100-class separately. ImageNet is a 1000-class task that contains 1281167 training and 50000 testing pairs. For the objective detection, we use MS-COCO dataset [22], which contains 82783, 40504, and 81434 pairs in the training, validation, and testing set separately. For the semantic segmentation, we choose PASCAL VOC 2012 [6]. Following the procedure of Zhao et al. [40], we use augmented data with the annotation of Hariharan et al. [11], resulting in 10582, 1449, and 1456 images for training, validation, and testing.

Evaluation metrics. In the classification, we use the accuracy on the testing set of each dataset [37, 23]. In the objective detection, we use the mean of Average Precision (mAP) [31] to evaluate the student model on the testing set of MS-COCO [22]. In the semantic segmentation, we use Mean Intersection over Union (mIoU) [40] to evaluate the student model on the testing set of VOC [6, 11].

Baseline methods. For the classification, we employ several popular loss functions, including fixed loss functions such as Cross Entropy loss (CE), the large-margin softmax loss (L-M softmax) [24], and the smooth 0-1 loss function (Smooth) [30] as well as dynamic loss functions, namely the adaptive robust loss function (ARLF) [3], the L2T-DLF loss function [37], stochastic loss function (SLF) [23] and ALA [16]. For objective detection, we compare our approach with the objective function set by YOLO-v3 [31]. For the semantic segmentation, we compare our approach with the objective function set by PSPNet [40].

Implementation details. In all experiments, we optimize student models using standard stochastic gradient descent (SGD) with a learning rate of 0.1. The teacher model is trained with Adam, utilizing a learning rate of 0.001. The learning rate of DLN is set to 0.001. The teacher model is trained for 10 epochs, with redividing the training and validation data after each epoch. The validation errors in each task are explicitly reported. Our teacher model comprises a four-layer LSTM [14] with 64 neurons in the first three layers and 1 neuron in the final layer. We utilize a 1-vs-1 approach (details in supplementary materials) to process the student model's output in both classification and segmentation. We present DLN architecture for each task and ensure reliable evaluation by conducting 5 random restarts, using average results for comprehensive comparisons.

5.2 Results

Image classification. For CIFAR-10 and CIFAR 100, we follow the SLF [23] and use architectures that include ResNet [12], and Wide-ResNet(WRN) [39] as the student model. For ImageNet, we follow the ALA [16] and use the identical NASNet-A [43]. In each experiment, the batch sizes for training and validation are set to 25 and 100, respectively. We perform a five-layer fully connected network, which contains 40 neurons in each hidden layer and 1 neuron in the output layer, as the DLN. The activation function for each hidden layer is set to Leaky-ReLU. The validation error is computed by CE.

Table 1 reports the performance of each loss function. Our approach achieves the best results on CIFAR-10 with ResNet8, ResNet20, and ResNet32, achieving 90 . 70% , 93 . 40% , and 93 . 81% , respectively. On CIFAR-100, our method also outperforms baselines with an overall accuracy of 63 . 50% , 70 . 47% , and 72 . 06% for ResNet8, ResNet20, and ResNet32, respectively. For WRN, our approach achieves the second-best performance, following the SLF method. The results on ImageNet illustrate that L2T-DLN improves the accuracy of the baseline by 0 . 7% . On ImageNet, our DLN demonstrates the second-best performance. The performance of ALA benefits from the larger length of a student learning stage ALA set (200) compared with ours (25). The ablation showed that the size of the length is positively correlated with the test accuracy and computational consumption (see Table 5).

Objective detection. In the task of objective detection, the YOLO-v3 model with a backbone of darknet-53 [31] is used in this experiment. The traditional loss in the YOLO model is a multi-part loss function, i.e. , λ cls ℓ cls + λ conf ℓ conf + λ loc ℓ loc . ℓ cls , ℓ conf and ℓ loc are detailed in supplementary materials. Redmon and Farhadi [31] set λ cls = λ conf = λ loc = 1 . In our experiment, our L2T-DLN learns to set these weights dynamically with a single-layer perceptron as DLN. The backbone of the YOLO is pre-trained on ImageNet, and we finetune the header of the YOLO. Specifically, the objective function of the student model is set to DLN ([ ℓ cls , ℓ conf , ℓ loc ]) . The validation error is computed by ℓ cls + ℓ conf + ℓ loc . The batch sizes for training and validation are set to 2 and 8, respectively. The length of student learning is set to 2. We take the training set and 35000 images of the validation set to train our L2T-DLN with an input size of 416*416. From Table 2, our L2T-DLN has more than 1 . 6% improvement with the baseline on mAP.

Semantic segmentation. The objective function of PSPNet [40] is set to CE ( p, y ) + 0 . 4 ∗ CE ( aux p , y ) , where p , aux p , and y denote the output of the master branch, the auxiliary branch of PSPNet, and the ground truth, respectively. For PSPNet with L2T-DLN, the objective function is set to DLN ( p, y )+0 . 4 ∗ DLN ( aux p , y ) , where the architecture of DLN is the one used in classification tasks. The validation error is computed by CE ( p, y ) + 0 . 4 ∗ CE ( aux p , y ) . The batch sizes for

training and validation are set to 2 and 8, respectively. The length of student learning is set to 2. Table 3 shows that our L2T-DLN improves 0 . 3% compared with the baseline on mIoU.

5.3 Ablations

In this subsection, we conduct ablation studies on CIFAR-10 [20] using ResNet8 to analyze the L2T-DLN synthetically. We specifically examine the proportion of training and validation data, the length of student learning stage ( N ), the wrong learning rate setting, and the influence of the LSTM teacher. Furthermore, we provide the visualization of DLN at different learning stages in MNIST and CIFAR10 tasks. We assess the impact of each component by computing the test accuracy of the student model after optimizing the teacher model for 10 epochs.

The proportion of training and validation data. In L2T-DLN, the training dataset is divided into two sets: validation data and training data, with validation data serving as an unbiased estimator for model generalization. After each epoch, the dataset is redivided, allowing samples used in the validation data to be included in the training data, and vice versa. The validation ratio represents the fraction of training dataset samples exclusively used for validation. This study explores different training-validation data separations. Table 4 results indicate our performance remains stable across varying ratios due to the teacher's ability to capture short- and long-term dependencies. To make a trade-off between computational cost and accuracy, we set the ratio = 50% for all our experiments.

Table 4: Results on different validation ratios ranging from 10% to 90% to show the impact of ratios.

The length of student learning. The computation of higher-order gradients in L2T-DLN (Eq. (6) and (8)) is computationally intensive and should be highlighted. Thus, this study explores the influence of the length of student learning ( N ) on the test accuracy and computational load in CIFAR-10 experiments using ResNet8. As shown in Table 5, the findings reveal that the test accuracy increases with the length of student learning. To make a trade-off between performance and computational cost, we suggest that a maximum length of 25 should be set for student learning. Overall, the study concludes that L2T-DLN has the potential to further improves the performance of student model with sufficient computing resources.

The influence of an LSTM teacher. As introduced above, the teacher model is similar to an optimization algorithm. Then we perform various optimizers, including Adam [19], SGD, RMSProp [13], and the LSTM teacher, to optimize the DLN and present the results in Table 6. Compared with ADAM, SGD, and RMSProp, our teacher can improve the performance of the student by 0 . 48% , 1 . 6% , and 0 . 53% . We can conclude that 1) algorithms that can use the historical information, e.g. , momentum, perform well; 2) the adaptability to capture and maintain short- and long-term dependencies can further enhance the loss function teaching, compared to handcrafted methods, e.g. , exponentially weighted moving average [13] and moment estimation [19].

prediction 0

10.0

7.5

5.0

2.5

0.0

2.5

5.0

7.5

10.0

prediction 1

10.0

7.5

5.0

2.5

0.0

2.5

5.0

7.5

10.0

loss value

0.3

0.2

0.1

0.0

0.1

0.2

0.3

0.4

prediction 0

10.0

7.5

5.0

2.5

0.0

2.5

5.0

7.5

10.0

prediction 1

10.0

7.5

5.0

2.5

0.0

2.5

5.0

7.5

10.0

loss value

2.25

2.00

1.75

1.50

1.25

1.00

0.75

0.50

0.25

prediction 0

10.0

7.5

5.0

2.5

0.0

2.5

5.0

7.5

10.0

prediction 1

10.0

7.5

5.0

2.5

0.0

2.5

5.0

7.5

10.0

loss value

0.35

0.30

0.25

0.20

0.15

0.10

0.05

prediction 0

10.0

7.5

5.0

2.5

0.0

2.5

5.0

7.5

10.0

prediction 1

10.0

7.5

5.0

2.5

0.0

2.5

5.0

7.5

10.0

loss value

0.30

0.25

0.20

0.15

0.10

0.05

(a)

(b)

(c)

(d)

prediction 0

10.0

7.5

5.0

2.5

0.0

2.5

5.0

7.5

10.0

prediction 1

10.0

7.5

5.0

2.5

0.0

2.5

5.0

7.5

10.0

loss value

0.8

1.0

1.2

1.4

1.6

1.8

prediction 0

10.0

7.5

5.0

2.5

0.0

2.5

5.0

7.5

10.0

prediction 1

10.0

7.5

5.0

2.5

0.0

2.5

5.0

7.5

10.0

loss value

2.0

1.5

1.0

0.5

0.0

0.5

1.0

prediction 0

10.0

7.5

5.0

2.5

0.0

2.5

5.0

7.5

10.0

prediction 1

10.0

7.5

5.0

2.5

0.0

2.5

5.0

7.5

10.0

loss value

1.6

1.4

1.2

1.0

0.8

0.6

0.4

0.2

prediction 0

10.0

7.5

5.0

2.5

0.0

2.5

5.0

7.5

10.0

prediction 1

10.0

7.5

5.0

2.5

0.0

2.5

5.0

7.5

10.0

loss value

0.175

0.150

0.125

0.100

0.075

0.050

0.025

0.000

(e)

(f)

(g)

(h)

Visualization. We visualize the loss value of DLN on MNIST and CIFAR-10 separately in Figure 3, which illustrates the capacity of L2T-DLN to adapt to the evolving states of students to attain improved performance. The DLN is initialized with the Kaiming normal initialization with LeakyReLU activations.

6 Conclusions

This paper introduces L2T-DLN, an adaptive model for various stages of student learning. Technically, We propose a differentiable three-stage teaching framework, asynchronously optimizing the student, DLN, and teacher. An LSTM teacher dynamically captures and retains experiences during DLN learning. Additionally, we assess L2T-DLN's ability to navigate strict saddle points using the negative curvature of their Hessian matrix. Experiments demonstrate our DLN outperforming specially designed loss functions. Nevertheless, our approach demands significant computational resources for high-order derivatives, which we aim to mitigate in future work.

7 Acknowledgment

This work was supported in part by National Natural Science Foundation of China (62302045, 82171965), Clinical and Translational Medical Research Fund of the Chinese Academy of Medical Sciences (2020-I2M-C&T-B-072), and Beijing Institute of Technology Research Fund Program for Young Scholars.

References

Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent. Advances in neural information processing systems , 29, 2016.

Sungyong Baik, Janghoon Choi, Heewon Kim, Dohee Cho, Jaesik Min, and Kyoung Mu Lee. Meta-learning with task-adaptive loss function for few-shot learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 9465-9474, 2021.

Jonathan T Barron. A general and adaptive robust loss function. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4331-4339, 2019.

Dimitri P Bertsekas. Nonlinear programming. Journal of the Operational Research Society , 48(3):334-334, 1997.

Leland P Bradford. The teaching-learning transaction. Adult Education , 8(3):135-145, 1958.

Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision , 88:303-338, 2010.

Yang Fan, Fei Tian, Tao Qin, Xiang-Yang Li, and Tie-Yan Liu. Learning to teach. In International Conference on Learning Representations , 2018.

Yang Fan, Yingce Xia, Lijun Wu, Shufang Xie, Weiqing Liu, Jiang Bian, Tao Qin, and Xiang-Yang Li. Learning to reweight with deep interactions. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 7385-7393, 2021.

Felix A Gers, Jürgen Schmidhuber, and Fred Cummins. Learning to forget: Continual prediction with lstm. Neural computation , 12(10):2451-2471, 2000.

Mengqiao Han, Liyuan Pan, and Xiabi Liu. Astronet: When astrocyte meets artificial neural network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 20258-20268, June 2023.

Bharath Hariharan, Pablo Arbeláez, Lubomir Bourdev, Subhransu Maji, and Jitendra Malik. Semantic contours from inverse detectors. In 2011 international conference on computer vision , pages 991-998. IEEE, 2011.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky. Neural networks for machine learning lecture 6a overview of mini-batch gradient descent. Cited on , 14(8):2, 2012.

Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation , 9(8):1735-1780, 1997.

Chen Huang, Shuangfei Zhai, Walter Talbott, Miguel Angel Bautista, Shih-Yu Sun, Carlos Guestrin, and Josh Susskind. Addressing the loss-metric mismatch with adaptive loss alignment supplementary material.

Chen Huang, Shuangfei Zhai, Walter Talbott, Miguel Bautista Martin, Shih-Yu Sun, Carlos Guestrin, and Josh Susskind. Addressing the loss-metric mismatch with adaptive loss alignment. In International conference on machine learning , pages 2891-2900. PMLR, 2019.

JC Ihejirika. Teaching strategies for adult learners: Implications of learning characteristics for effective teaching-learning transaction. Academic Research International , 4(2):310, 2013.

Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels. In International conference on machine learning , pages 2304-2313. PMLR, 2018.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

A Krizhevsky. Learning multiple layers of features from tiny images. Master's thesis, University of Tront , 2009.

Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 and cifar-100 datasets. URl: https://www. cs. toronto. edu/kriz/cifar. html , 6(1):1, 2009.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13 , pages 740-755. Springer, 2014.

Qingliang Liu and Jinmei Lai. Stochastic loss function. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 4884-4891, 2020.

Weiyang Liu, Yandong Wen, Zhiding Yu, and Meng Yang. Large-margin softmax loss for convolutional neural networks. In International Conference on Machine Learning , pages 507-516. PMLR, 2016.

Weiyang Liu, Bo Dai, Ahmad Humayun, Charlene Tay, Chen Yu, Linda B Smith, James M Rehg, and Le Song. Iterative machine teaching. In International Conference on Machine Learning , pages 2149-2158. PMLR, 2017.

Songtao Lu, Mingyi Hong, and Zhengdao Wang. On the sublinear convergence of randomly perturbed alternating gradient descent to second order stationary solutions. arXiv preprint arXiv:1802.10418 , 2018.

Payal Mehra and Monika Mital. Integrating technology into the teaching-learning transaction: Pedagogical and technological perceptions of management faculty. International Journal of Education and Development using ICT , 3(1), 2007.

Thomas Howard Morris. Vocational education of young adults in england: A systemic analysis of teachinglearning transactions that facilitate self-directed learning. Journal of Vocational Education & Training , 70 (4):619-643, 2018.

Baraka Manjale Ngussa and Lazarus Ndiku Makewa. Constructivism experiences in teaching-learning transaction among adventist secondary schools in south nyanza, tanzania. American Journal of Educational Research , 2(11A):1-7, 2014.

Tan Nguyen and Scott Sanner. Algorithms for direct 0-1 loss optimization in binary classification. In International Conference on Machine Learning , pages 1085-1093. PMLR, 2013.

Joseph Redmon and Ali Farhadi. Yolov3: An incremental improvement. arXiv preprint arXiv:1804.02767 , 2018.

Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In International conference on machine learning , pages 4334-4343. PMLR, 2018.

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision , 115:211-252, 2015.

Jürgen Schmidhuber. Evolutionary principles in self-referential learning, or on learning how to learn: the meta-meta-... hook . PhD thesis, Technische Universität München, 1987.

Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, and Deyu Meng. Meta-weight-net: Learning an explicit mapping for sample weighting. Advances in neural information processing systems , 32, 2019.

Sebastian Thrun and Lorien Pratt. Learning to learn . Springer Science & Business Media, 2012.

Lijun Wu, Fei Tian, Yingce Xia, Yang Fan, Tao Qin, Lai Jian-Huang, and Tie-Yan Liu. Learning to teach with dynamic loss functions. Advances in neural information processing systems , 31, 2018.

Yan Yang, Liyuan Pan, and Liu Liu. Event camera data pre-training. arXiv preprint arXiv:2301.01928 , 2023.

Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In British Machine Vision Conference 2016 . British Machine Vision Association, 2016.

Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2881-2890, 2017.

Xiaojin Zhu. Machine teaching: An inverse problem to machine learning and an approach toward optimal education. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 29, 2015.

Barret Zoph and Quoc Le. Neural architecture search with reinforcement learning. In International Conference on Learning Representations , 2016.

Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 8697-8710, 2018.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces the concept of teaching in machine learning and proposes a framework called L2T-DLN (Learning to Teach with Dynamic Loss Network). The framework aims to address the limitations of existing approaches by incorporating the temporal nature of loss function adjustment and utilizing the states of the loss function. The authors formulate the loss adjustment as a temporal task using a teacher model with memory units and propose a dynamic loss network to enhance the interactions between the teacher and student model. Extensive experiments demonstrate the effectiveness of the approach on various real-world tasks.\n\nSTRENGTHS:\n1. The paper presents a novel framework, L2T-DLN, which integrates the concept of teaching with dynamic loss functions. This approach stands out by considering the temporal nature of loss function adjustment and utilizing the states of the loss function, setting it apart from existing works.\n2. The authors provide a thorough convergence analysis of the proposed framework, demonstrating its effectiveness and its ability to achieve convergence.\n\nWEAKNESSES:\n1. Unclear Motivation: The paper lacks a thorough discussion of the benefits of introducing the gradients concerning loss functions. It would be beneficial to include a clearer motivation.\n2. Marginal Improvement: The reported improvements of 0.4% lower than ALA on ImageNet for classification and only 0.3% improvement compared to the baseline on mIoU on the VOC dataset for segmentation are relatively small.\n\nQUESTIONS:\n1. Further clarification is needed in the paper on why it is necessary to consider the gradients concerning loss functions as a core idea. Besides experimental results, could there be theoretical analysis supporting this aspect?\n2. The authors introduce a Dynamic Loss Network instead of using a dynamic loss function. It would be helpful to explain if this choice leads to a larger number of parameters and potential unfairness in comparisons. Please clarify the impact of parameter quantity in this context.\n\nLIMITATIONS:\nThe implementation of this approach necessitates substantial computational resources for calculating high-order derivatives.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces an improvement towards the learning-to-teach framework. Compared with the previous works, the authors made an innovation that a dynamic loss network is added with LSTM acting as teacher model to enhance the temporal memorization. A three step optimization procedure is also proposed with an convergence analysis theoretically. Last extensive experiments are conducted on various computer vision tasks that demonstrated the effectiveness of the proposed approach.\n\nSTRENGTHS:\n1. The problem this paper studied is an important one yet insufficiently studied in the current literature, that is how to use a better teaching strategy in addition to merely conduct \"learning\". It's a very meaningful step forward of this paper that pushes forward the frontier of such a literature; Furthermore the design of dynamic loss network is a new perspective for this area as well, going beyond the dynamic loss function design.\n\n2. The paper is technically very sound with enough insights and depth. I particularly appreciate the convergence analysis which is done first time for the machine teaching/learning-to-teach area, with a good depth leveraging the negative curvature.\n\nWEAKNESSES:\nAs also stated in the paper, despite being theoretically sound and elegant, the proposed algorithm incurs much more computational overhead which will hinder its practical usefulness in real world applications. Particularly the compute of Hessian even three-order derivatives is a sign of its computational complexity. I agree with the author that this needs to be improved further in the future to enlarge the impact of this work.\n\nQUESTIONS:\nWhile the memory units of LSTM make sense in this scenario, how about use Transformer as another sequential model which may even enhance the temporal memorization compared with LSTM?\n\nLIMITATIONS:\nStated in the paper as well as the above point, that the computational complexity is one concern while it seems feasible and justifiable to further solve this, as the future potential value of this direction is likely large.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors state that existing methods only employ a simple feedforward network as the teacher model, which limits the potential of L2T. This issue motivates authors to propose a network with a memory unit to enhance the temporal analyzing ability of the teacher in the learning to teach (L2T) tasks. This proposed new network is combined with a dynamic loss network to address the issue of the existing works above. The experiment shows the superiority of the proposed method.\n\nSTRENGTHS:\n(+) This paper theoretically analyzes the convergence of the proposed method.\n\n(+) The proposed method shows the state-of-the-art performance compared with other methods.\n\nWEAKNESSES:\n(-) The novelty of this paper is not clear. In the title, abstract and Introduction, the authors repeatedly emphasize that they propose an L2T framework with a Dynamic Loss Network (L2T-DLN). However, these emphases can cause misleading since Dynamic Loss has already been introduced in previous work, as stated by the authors themselves in Lines 2, 23, and the caption of Figure 1. Consequently, the application of Dynamic Loss in L2T cannot be considered as the novelty or the main contribution of the paper. The continued highlighting of DLN in these sections adds to the confusion and may hinder the understanding of the paper. In my understanding, the primary contribution of this paper is a proposed network with a memory unit to enhance the temporal analyzing ability of the teacher in the learning to teach (L2T) tasks. By emphasizing this aspect, rather than the Dynamic Loss network, the paper will become more coherent and easier to comprehend.\n\n(-) The authors have not adequately demonstrated the effectiveness of their proposed method in both theory and practice. In the theoretical analysis, they solely focus on the convergence of the proposed L2T-DLN and fail to illustrate how the proposed network with a memory unit for the teacher can benefit the training of L2T task. Additionally, the experiments show the better performance of the proposed method compared with other SOTA methods. However, based on section 5.1, this better performance is based on many combinations of loss functions and tricks. It remains unclear whether the proposed network with a memory unit for the teacher contributes to the enhanced performance observed in the L2T task.\n\n(-) The paper lacks sufficient ablation studies to support its claims. In the ablation study section, the authors include some experiments related to selecting the best learning parameters for their proposed method; these parameters may primarily influence performance rather than directly relating to the main contributions of the proposed method. Therefore, it is crucial for the authors to conduct a series of ablation studies to demonstrate how the proposed network with a memory unit for the teacher can benefit the training of L2T tasks.\n\nQUESTIONS:\nSee weakness.\n\nLIMITATIONS:\nNo limitations", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a three-stage framework that dynamically adjusts the learning process of student model (i.e., target model). The loss value is calculated via the proposed Dynamic Loss Network (DLN), parameterized as neural network. And the DLN is updated by the teacher network implemented as LSTM. The authors also provide a convergence analysis. The model is evaluated on an array of tasks including image classification, object detection and semantic segmentation.\n\nSTRENGTHS:\n1. Compared to previous works, L2T-DLN leverages the state of loss function and the temporal information of student learning experience  to update the teacher.\n2. Experiments on various application including image classification, object detection and semantic segmentation showcase the effectiveness.\n3. The provided convergence analysis is helpful to understand the framework.\n\nWEAKNESSES:\n1. The proposed DLN is not unified. For image classification, it produces the loss value. For object detection, it generates the loss weights.\n2. Some technical details of the DLN are missed, as will discussed in next session.\n3. To access the temporal information of student learning, L2T-DLN would store the historical models and consume lots of memory.\n4. Since the DLN adjusts the loss weights of the objectives of YOLO-v3, it’s reminiscent of multi-task learning. In this regard, the authors should discuss the relationship with MTL.\n\nQUESTIONS:\n1. The output layer (e.g., activation function) of the DLN is not clearly described.\n2. It would be better to understand the DLN if the loss value/weights can be visualized.\n3. Figure 1 in the supplementary material indicates the DLN (blue line) would produces lower gradients applied on the student model. Is this because the baseline model uses a large learning rate (0.1 line 219)?\n4. The claimed effectiveness would be more convincing if an ablation study of learning rates is provided.\n5. Regarding the initial gradients produced by the DLN, which greatly exceeds the baseline, it may be related to the network initialization of the DLN. Since this detail is missing, I would suggest the author clarifying the initialization. \n6. For the DLN of object detection, is it initialized as identical mapping? If not, it’s helpful to investigate this manner.\n7. What’s the training epoch for baseline model and the L2T-DLN for image classification?\n\nLIMITATIONS:\n1. Some details (e.g., initialization and activation of the output layer) of the proposed DLN are missing. \n2. The used learning rate for image classification may not be optimal, an ablation study would help understand the framework.\n3. It would be helpful to visualize the output of DLN.  \n4. The framework may consume lots of GPU memory.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}