Iterative Teaching by Label Synthesis

1 Introduction . Machine teaching [ 103 , 106 ] studies the problem of constructing a minimal dataset for a target concept such that a learner can learn the concept based on this dataset . Machine teaching has diverse applications ranging from crowd sourcing [ 71 , 72 , 100 , 101 ] to model robustness [ 2 , 3 , 54 , 63 ] . Machine teaching also has nice connections with curriculum learning [ 7 ] and coresets [ 1 , 28 ] . Based on the learner , machine teaching can be performed in either batch or sequential fashion . The majority of prior work studies batch machine teaching [ 42 , 56 , 102 , 103 ] , where a teacher constructs a minimal batch set of training samples and provides it to a student in one shot without further interactions . Then the student keeps learning from this batch dataset for the target concept . The size of such a minimal batch set is called teaching dimension [ 22 ] . Differently , sequential machine teaching [ 38 , 43 , 44 , 61 ] bridges the gap between machine teaching and practical learning algorithms by studying the sequential ( i.e. , itera- tive ) learner such as neural networks . A typical example is iterative machine teaching ( IMT ) [ 43 , 44 ] where the teacher guides a learner to a target concept by interacting with the learner ( e.g. , feeding training samples ) in every iteration . The minimum number of such iterations is called the iterative teaching dimension . One of the largest challenges in sequential teaching is how to effectively and efficiently provide teaching examples to the iterative learner . Usually we are mostly interested in the pool-based teaching in IMT since it well matches the setting of modern machine learning . However , exploring all the possible teaching trajectories is computationally prohibitive . For example , there are ( m k ) n possible teaching trajectories ( n is the number of iterations ) if we select k samples per iteration from a pool of size m. Due to such a huge search space , the selection of teaching examples is a combinatorial problem that is inherently difficult to solve . IMT [ 43 ] performs the teaching sample selection with a greedy policy , but it could be substantially sub-optimal in certain cases [ 38 ] and its computational complexity also scales linearly with the size of the dataset . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . We propose a general teaching framework called LAbel Synthesis Teaching ( LAST ) , which in its standard version avoids the problems of sample selection – though we will later discuss how sample selection can be combined with our approach . In the standard version of LAST , teaching examples are randomly sampled ( similar to SGD ) in the pool and the teacher synthesizes their labels in order to quickly guide the learner to the desired model . A brief comparison between IMT and LAST is given in Fig . 1 . LAST restricts the teaching to the label space and bypasses the selection of teaching samples . Therefore , LAST avoids the high complexity of selecting teaching samples in a large pool . Intuition for why LAST is able to achieve promising teaching performance comes from the empirical success of knowledge distillation [ 26 ] and label smoothing [ 73 ] . Knowledge distillation shows that training neural networks with soft labels from a pretrained model can generally improve generalization . Label smoothing demonstrates that the ground truth labels are not necessarily the optimal supervision for training a model . Instead , smoothing the label with uniform distribution can calibrate the model and lead to better generalization . Both methods can be viewed as providing an alternative label ( instead of the ground truth ) to the learner in order to improve its generalizability . Moreover , [ 77 , 78 ] show that there exists privileged information beyond the ground truth labels that can significantly improve the convergence rate of learning algorithms . Therefore , now we can safely argue that the ground truth labels are not always optimal learning signals . Motivated by these work , we aim to construct a teacher model that can adaptively synthesize suitable labels for a learner ( with the hope to implicitly encode priviledged information ) in order to improve the learner ’ s convergence . Specifically , we study LAST primarily under the omniscient scenario where the teacher knows everything about the learner ( e.g. , the optimal learner parameters ) . To perform omniscient teaching , we consider a greedy teacher and a parameterized teacher . We show that greedy teaching can achieve exponential teachability ( ET ) [ 43 ] without selecting teaching examples . Additionally , we touch upon the black-box teaching scenario where the teacher knows less about the learner ( e.g. , the optimal learner parameters are unavailable ) , and discuss how to perform LAST in this case . LAST provides a unified view for understanding soft label methods , e.g. , knowledge distillation [ 26 , 53 ] , label smoothing [ 11 , 59 , 73 ] , and self-training [ 104 ] . All these methods can be interpreted as modifying the labels to achieve desirable learning behavior and outcome . With LAST , we can connect iterative machine teaching to many classic learning algorithms and shed novel light on them . 2 Related Work . Machine teaching . Batch machine teaching [ 42 , 56 , 102 , 103 ] has drawn increasing more attention recently . Different learners typically exhibit distinct behaviors during teaching . [ 10 , 74 ] focus on how to teach version space learners . The teaching dimension of linear learners [ 42 ] , kernel learners [ 36 ] and reinforcement learner [ 95 ] is extensively studied . Teaching a forgetful learner is explored in [ 29 , 44 ] . [ 94 , 105 ] consider how to teach multiple learners simultaneously . Sequential machine teaching [ 38 , 43 , 44 , 91 ] studies iterative learners by considering the specific optimization algorithm that the learner adopts . Unlike batch scenario , the teaching quality is evaluated by the learner ’ s convergence . [ 38 ] connects sequential teaching to optimal control and gains interesting insights , but it can not produce a practical teaching policy . [ 91 ] uses locality-sensitive sampling to scale IMT to largescale problems . Machine teaching is shown useful in reinforcement learning [ 24 , 32 , 63 , 76 ] , humanin-the-loop learning [ 9 , 31 , 55 ] , crowd sourcing [ 72 , 100 , 101 ] and cyber security [ 2 , 57 , 96 , 97 , 98 ] . [ 9 , 13 , 21 , 35 , 65 , 108 ] study machine teaching from a more theoretical point of view . Cooperative communication . Cooperative communication [ 69 ] is a mutual theory of mind reasoning between a teacher and a learner . The teacher selects data to convey a hypothesis and the learner infers a hypothesis given the selected data . Both agents have the shared goal of successfully transmitting beliefs . [ 84 ] formulates cooperative communication with the framework of optimal transport . In this framework , pedagogical reasoning [ 66 , 67 , 68 ] , cooperative inference [ 82 , 83 , 93 ] and Bayesian teaching [ 15 , 16 , 92 ] can all be viewed as special cases . Among this line of research , cooperative inference [ 82 ] shares many similarities to our work and also adopts an iterative teaching paradigm . While [ 82 ] formulates the teaching problem with a probabilistic inference framework , LAST addresses the iterative teaching with a nested optimization framework . Soft-supervised learning . It has been shown that supervising the model with soft labels can be beneficial to generalization [ 73 , 89 ] and robustness [ 75 , 107 ] . Label smoothing [ 73 ] replaces one-hot label vector with a mixture of itself and the uniform distribution , achieving stronger generalization in training neural networks . Knowledge distillation [ 26 ] uses the soft predicted label from a pretrained model to supervise the training of a student model , which can make the student model generalize better than a trained-from-scratch one . [ 89 ] perturbs the one-hot label vector with random noise and also observes better generalization . Large-margin softmax [ 45 , 50 , 51 , 52 ] can be approximately viewed as training learners with dynamic soft labels ( Appendix E ) . Self-training [ 8 , 90 , 104 ] also uses dynamic soft labels to iteratively train the learner . In constrast to the empirical nature of soft-supervised learning , LAST formulates it as a unified and principled iterative teaching framework . 3 Label Synthesis Teaching . Teaching protocol . We first consider the simplest teaching protocol following [ 43 ] . Both the teacher and the learner observe the same sample A and share the same feature space by representing A as x with the label y . The teacher has access to all the information about the learner such as the model parameter wi at the i-th iteration , the learning rate ηi , the loss function ` and the specific optimization algorithm ( e.g. , SGD ) . For interaction , the teacher can only communicate with the learner via examples { ( xij , yij ) } j=1 , ··· , m in the i-th iteration , where m denotes the batch size . Teaching objective . The goal of the teacher is to provide examples in each iteration such that the learner parameters w converge to desired parameters w∗ as fast as possible . For example , w∗ typically is arg minw E ( x , y ) { ` ( x , y|w ) } . Therefore , a general objective for any teaching algorithm is min { ( x1 , y1 ) , ··· , ( xT , yT ) } d ( wT , w∗ ) where T denotes the termination iteration for the teaching algorithm . We use batch size 1 here , but it is straightforward to extend the batch size to arbitrary m. d ( · , · ) denotes some discrepancy measure ( e.g. , Euclidean distance ) . This objective aims to find a teaching trajectory that is terminated at the T -th step such that the discrepancy between the current learner parameters and the desired learner parameters is the smallest . This is very challenging given the enormous search space . To simplify the problem , [ 43 ] proposes a greedy policy to generate the teaching curriculum , but it is computationally expensive when dealing with a large pool . 3.1 The LAST Framework Unlike the general teaching objective that optimizes the sequential sample selection , the LAST framework instead optimizes the teaching examples in the label space , and the corresponding objective becomes min { y1 , ··· , yT } d ( w T , w∗ ) where x1 , · · · , xT are sampled uniformly from the pool ( same as SGD ) . Such a teaching objective is general and does not make any assumption on the optimal structure in the teaching trajectory , making the problem intractable . To simplify the problem , we can assume that the teaching signal ( i.e. , label y in LAST ) only depends on the ran- domly sampled { x , ỹ } and the learner ’ s current and desired model ( i.e. , parameters ) . Therefore , we can formulate the corresponding label synthesis policy as y=πθ ( x , ỹ , wt , w∗ ) where π is optionally parameterized by θ and ỹ denotes the original ground truth label . Then the teaching objective becomes arg minπθ d ( w T , w∗ ) . An overview of LAST is given in Fig . 2 . Moreover , LAST provides a unified framework for knowledge distillation , label smoothing and self-training . Knowledge distillation [ 26 ] generates the label with the policy yi= exp ( zi ( x ) /ψ ) ∑ j exp ( zj ( x ) /ψ ) where ψ is the temperature and zi ( x ) is the i-th logit output of x from a pretrained neural network . Label smoothing [ 73 ] uses y=µỹ+ 1−µK 1 ( K is the number of classes ) as the label synthesis policy . Self-training [ 104 ] feeds the learner with the pseudo-label predicted by the learner from the last iteration . All these methods can be viewed as using some form of customized label synthesis policies .

Summary: This paper studies the problem of iterative machine teaching with a focus on synthesizing the labels.   Specifically, the authors deal with the white-box (w* is known, the target parameter) and black-box teaching settings (w* is unknown). For the white-box setting, the authors propose a simple greedy teaching algorithm LAST, which is myopic. To incorporate the long-horizon information, the authors further present two methods: 1) unrolling the gradient steps and 2) learn a teaching policy with RL. These two methods are also applied to deal with the black-box setting.   For the theoretical analysis, the authors cast the effect of label synthesis/teaching as smartly tuning the learning rates, which also enjoys the exponential teachability under the convex and Lipschitz assumptions on the loss functions.   Empirical results show that the proposed methods are effective and require fewer iterations to converge than standard training (SGD).