Iterative Teacher-Aware Learning

1 Introduction . Cooperative pedagogy is invoked across language , cognitive development , cultural anthropology , and robotics to explain people ’ s ability to effectively transmit information and accumulate knowledge [ 19 , 64 ] . As the usage of artificial intelligence and machine learning based systems ratchets up , it is foreseeable that extensive human-computer and agent-agent pedagogical scenarios will occur in the near future [ 49 ] . However , there is still a distance away from robots being able to directly teach or learn from humans as efficiently and effectively as humans do . One of the many difficulties is that machine learning and teaching are now usually studied in single-agent frameworks . Most of the prevailing machine learning methods focus on the improvement of individual learners and the explanations of how knowledge is obtained focus entirely on each learner ’ s unilateral experiences , either passive observations from a Markov decision process [ 45 , 55 ] , random samples from a data distribution [ 52 , 29 ] , responses of active queries provided by an oracle [ 3 , 53 ] , or demonstrations from an expert [ 4 ] . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Such machine learning framework is diametrically distinctive from human education , in whose context , learning often occurs sequentially instead of in batch , and from intentional messages given by a pedagogical teacher rather than random data from a fixed sampling process [ 62 ] . Recently , the advantage of pedagogical teachers over randomly sampled data or optimal task completion trajectories from experts has been shown in machine teaching [ 10 , 70 , 71 , 40 , 18 , 44 , 11 , 12 ] and in learning from demonstration ( LfD ) [ 27 , 31 ] . Nonetheless , compared with human pedagogy , these works still lack a sophisticated student model that can accommodate the teacher ’ s cooperation into his learning and acts differently than learning from passive data . Machine teaching algorithms model a cooperative teacher giving instructions in the format of data examples for continuous parameter [ 10 , 70 , 71 , 40 ] , Bayesian concept [ 18 , 44 ] or version space learning [ 11 , 12 ] , but seldom do they consider how learners may interpret differently between the data picked intentionally by the teacher and sampled randomly from the world . Standard LfD takes in demonstrations from an ( approximately- ) optimal expert to learn the underlying reward function [ 4 ] . Hadfield-Menell et al . [ 27 ] , Ho et al . [ 31 , 32 ] shows the advantage of using pedagogical rather than optimal demonstrations , yet , in either case , the learners are not aware of the teacher . Shafto et al . [ 54 ] , Yang et al . [ 68 ] , Wang et al . [ 62 , 64 ] move one step further and proposed recursive cooperative inference models having both the teacher and the student reasoning each other , an ability known as theory of mind ( ToM ) [ 50 , 8 ] . The first work modeled and predicted human behavior while the latter three managed to integrate ToM into machine learning . Despite the theoretical contribution , their approach [ 68 , 62 , 64 ] is confined to Bayesian concept learning with finite hypothesis space , in which the Sinkhorn scaling [ 57 ] is tractable . It is unclear how to apply their algorithms to settings involving continuous hypothesis spaces , such as learning neural networks . In this paper , we study how to integrate the cooperative essence of pedagogy into machine parameter learning and propose a teacher-aware learner who learns significantly faster than a naive learner , given an iterative machine teacher [ 40 , 41 ] . The learner estimates the teacher ’ s data selection process with distribution and corrects his likelihood function with this estimation to accommodate the teacher ’ s intention . Maximizing the new likelihood enables the learner to utilize both explicit information from the selected data and implicit information suggested by the pedagogical context . We theoretically proved the improvement brought by the learner ’ s teacher awareness and justified our results with various experiments . We believe that our work can provide insights into both human-machine interactions , such as online education , and machine-machine communications , such as ad-hoc teamwork [ 9 ] . Our main contributions are i ) modeling teacher-awareness for generic gradient optimization based parameter learning ; ii ) theoretically proving the improvement guaranteed by the teacher-aware learner over the naive learner under mild assumptions ; iii ) empirically illustrating the advantage of teacher-awareness learner when interacting with both machine and human teachers . 2 Related Work . Machine Teaching Our work is related to machine teaching as we used an iterative machine teacher in our framework . Machine teaching [ 70 , 71 , 40 , 49 ] solves the problem of constructing an optimal ( usually minimal ) dataset according to a target concept such that a student model can learn the target concept based on this dataset . Most of the machine teaching algorithms consider a batch setting , where the teacher designs a minimal dataset and provides it to a learning algorithm in one shot [ 10 , 70 , 47 , 71 , 11 ] . Iterative machine teaching has also been studied , where the teacher gives out data iteratively and checks the learner ’ s status before selecting the next data [ 34 , 6 , 40 , 41 , 43 ] , but previous works didn ’ t consider teacher-aware learners . There are also works applying machine teaching to inverse reinforcement learning ( IRL ) and LfD [ 4 , 10 , 27 , 31 , 32 , 28 ] . Our IRL experiments are different from those works as our data are provided iteratively and sequentially . Also , our learner is aware of the teacher ’ s intention . Ho et al . [ 31 , 32 ] integrated Bayesian rule in LfD to model mutual reasoning between the teacher and the learner , but they mainly used their model to explain human data . A theoretical study of the teaching-complexity of the teacher-aware learners was presented in [ 73 , 17 ] where the teacher and the learner are aware of their cooperation . Their analysis mainly attends to version-space learners which maintain all hypotheses consistent with the training data , and can not be applied to hypothesis selection via optimization , such as learning parameters . Peltola et al . [ 49 ] studied machine teaching for an active sequential multi-arm bandit learner . Although they also have a helpful teacher and a teacher-aware learner , their problem setting is different from ours . First , in every round of multi-arm bandit , the learner can actively choose an arm to pull , and then the teacher provides feedback for that arm , while in our cases , the data batch in each round is sampled randomly and independently from the learner ’ s current status . Second , as the teacher can only give binary ( success or not ) feedback to the learner , the counterfactual reasoning required for pedagogy is significantly simplified . Besides , they required that the same feature representation for the arms is shared between the teacher and the learner . Also , the learner doesn ’ t have to know the underlying parameter exactly to perform well in multi-arm bandit games , while in most of our cases , the learner tries to match the teacher ’ s model exactly . Fisac et al . [ 20 ] used similar formulation to model cooperative value alignment within a human-robot team . They assumed the human knows the robot ’ s value function during interactions , and parameters to be aligned are sparse and low dimensional . Learning to Teach Sharing the same goal as machine teaching , learning to teach ( L2T ) also seeks a teaching algorithm to improve the learning efficiency of AI . While machine teaching usually models the question as an optimization problem and solves for a closed-form teaching algorithm , works in L2T tend to train the teaching model in the process of the teacher-student interaction with gradient based optimization [ 67 ] or reinforcement learning ( RL ) algorithms [ 19 ] . Nevertheless , these works also focus mainly on the teacher algorithm and assume teacher-unaware learners . In some tasks , typically when the student aims to learn a concept in a discrete space , the teacher can track the learner ’ s status by modeling his belief over the concept [ 54 ] . As the beliefs within the learner ’ s mind are not usually known by the teacher , the teaching process can be modeled as a partially observable Markov decision process ( POMDP ) [ 46 ] , solving the optimal policy of which returns a teaching algorithm [ 51 , 65 ] . From the teacher ’ s perspective , the unknown part of the environment is the learner ’ s belief , a probability distribution over the concept space , so she has to form another belief over the learner ’ s belief . The intractable modeling of belief over continuous variables usually requires approximation methods such as particle filters [ 51 , 65 ] to solve , restricting the scope of these algorithms to naive learners and relatively simple learning tasks . Interactive POMDP [ 23 , 66 ] , a probabilistic multi-agent model , can also be used to model cooperative pedagogy with recursive teacher-learner reasoning . However , the nested belief over belief also suffers from intractability and is hard to scale up . If the concept space of the learner is continuous by itself , such as high dimensional continuous parameter spaces in our case , handling the belief over belief becomes difficult . Cooperative Bayesian Inference Shafto et al . [ 54 ] studies human pedagogy with examples using Bayesian learning models . The cooperative inference [ 68 , 63 , 64 ] in machine learning also formalizes full recursive reasoning from the perspectives of both the teacher and the learner . Distinctive from their concept learning settings , in this paper , we focus on parameter learning , in which the student has intractable posterior distribution and learns via gradient-based optimization . In addition , [ 54 , 68 , 63 , 64 ] only consider the problem of a single interaction between the teacher and learner . Wang et al . [ 62 ] proposed a sequential cooperative Bayesian inference algorithm and showed its performance advantage over naive Bayesian learner analytically and empirically . Nevertheless , they were still discussing concept learning with finite and usually small data and hypothesis sets . The Sinkhorn scaling [ 30 , 62 , 64 ] becomes infeasible when dealing with continuous parameter learning . Pragmatics Reasoning Our work is inspired by the study of pragmatics ( how context contributes to the meaning ) [ 26 , 27 ] and ToM [ 50 , 8 ] . The rational speech act ( RSA ) model raised by Golland et al . [ 24 ] and developed in [ 22 , 54 , 25 , 2 ] accommodates the idea of using not only the utterance but also the selection of the utterance to understand the speaker . Previous works in these areas are mainly from human action interpretation [ 21 , 60 , 36 ] , language emergence [ 69 , 35 ] , linguistics [ 33 , 2 , 13 ] and cognitive science [ 25 , 8 ] perspectives . To our knowledge , our work is the first to relate pedagogy and recursive reasoning to generic parameter learning and shows a provable improvement . Shafto et al . [ 54 ] proposed computational models for more diverse concept spaces , but mainly focus on modeling and predicting human behaviors .

Summary: **High-Level Summary**  This paper studies cooperative teacher and learner settings. In this setting, the teacher and learner are trying to solve a classification task with input $x \in \mathcal{X}$ and label $y \in \mathcal{Y}$. The setup works in an online fashion where at each time step, the teacher selects an example $(x, y)$ to present to the student from a large dataset and the student learns from this example. The teacher knows the optimal parameters. Previous work has focused on the setting where the teacher is aware of the learner and tries to select the example to ensure the student learns the fastest. This paper studies an extension where the learner is also aware of the teacher's feedback (teacher-aware learner). The key novelty is in the learner trying to maximize its parameters to fit the feedback provided by the teacher while also trying to predict the distribution the teacher is using to sample the data from. This has an additional benefit in that it can help the learner perform "pragmatic reasoning". This is related to previous work on Rational-Speech Act models (e.g., Golland et al., 2010). The key difference is that previous work on RSA provided limited theoretical analysis and was concerned with specific tasks e.g., predicting actions while this work focuses on a more abstract task of learning parameters for classification.