Iterative Teacher-Aware Learning
Authors: Luyao Yuan
Dongruo Zhou
Junhong Shen
Jingdong Gao
Jeffrey L Chen
Quanquan Gu
Ying Nian Wu
Song-Chun Zhu
Venue: NeurIPS 2021 Poster (2021)
Keywords: Dialog- or Communication-Based Learning
Pedagogy
Learning Theory

Abstract: In human pedagogy, teachers and students can interact adaptively to maximize communication efficiency. The teacher adjusts her teaching method for different students, and the student, after getting familiar with the teacher’s instruction mechanism, can infer the teacher’s intention to learn faster. Recently, the benefits of integrating this cooperative pedagogy into machine concept learning in discrete spaces have been proved by multiple works. However, how cooperative pedagogy can facilitate machine parameter learning hasn’t been thoroughly studied. In this paper, we propose a gradient optimization based teacher-aware learner who can incorporate teacher’s cooperative intention into the likelihood function and learn provably faster compared with the naive learning algorithms used in previous machine teaching works. We give theoretical proof that the iterative teacher-aware learning (ITAL) process leads to local and global improvements. We then validate our algorithms with extensive experiments on various tasks including regression, classification, and inverse reinforcement learning using synthetic and real data. We also show the advantage of modeling teacher-awareness when agents are learning from human teachers.

Iterative Teacher-Aware Learning
Luyao Yuan' yuanluyao@ucla.ee edu
Dongruo Zhou' drzhouêcs. ucla. edu
Juhong Shen jhshenOucla.cm edu
Jingdong Gao
mxuanQucla.com edu
Jeffrey L. Chen'
Quanquan Gu' qguccs ucla. edu
Ying Nian Wu? ywuōstat. ucla. edu
j plchenOeucla.com edu
Song-Chun Zhu sczhuβstat. lelt@asttttulla.... wula..llu...
Depaarmeent of Computer Science, "Department of Mathematies, *Department of Statistics
University of California, Los Angeles 'Beijing Institute for General Artificial Intelligence (BIGAI)
Abstract
human pedagogy, teachers and students can interact adaptively to maximize communication efficiency. The teacher adjusts her teaching method for different students, and the student, after getting familiar with the teacher's instruction mechanism, infer the teacher's intention to learn faster. Recently, the benefits integrating this cooperative pedagogy into machine concept learning in discrete spaces have been proved by multiple works. However, how cooperative pedagogy can facilitate machine parameter learning hasn't it been thoroughly studied. In this paper, we propose gradient optimization based teacher-aware learner who incorporate teacher's 's cooperative intention into the likelihood function and learn provably faster compared with the naive learning algorithms used in previous machine teaching works. We give theoretical proof that the iterative teacher-aware learning (ITAL) process leads local and global improvements. We then validate our algorithms with extensive experiments on various tasks including regression, classification, and inverse reinforcement learning using synthetic and real data. We also show the advantage of modeling teacher from human teachers.
reacher-awareness awareness when agents are learning
Introduction
Cooperative pedagogy invoked across language, cognitive development, cultural anthropology. and robotics to explain people's 's ability to effectively transmit information and accumulate knowledge [I9 As the usage of artificial intelligence and machine learning based systems ratchets up. is foreseeable that extensive cuman-comppter and agent-agent pedagogical scenarios will occur in the near future 49 However, there is still distance away from robots being able to directly teach or learn from humans as efficiently and effectively humans do. One of the many difficulties is that machine learning and teaching are now usually studied in single-agent frameworks. Most of the prevailing machine learning methods focus the improvement of individual learners and the explanations of how knowledge is obtained focus entirely on each learner's ' unilateral experiences, either passive observations from Markov decision process [45]] (55, random samples from data
64
distribution [52] (29), responses active queries provided by an oracle [3] 53), or demonstrations
from an expert
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Such machine learning framework is diametrically distinctive from human education, in whose context, learning often occurs sequentially instead of in batch, and from intentional messages given by pedagogical teacher rather than random data from fixed sampling process 62] Recently, the advantage of pedagogical teachers over randomly sampled data or optimal task completion trajectories from experts has been shown in machine teaching [TO] (70 771 40 [8] 44 12 and in learning from demonstration (LID) [27 31J. Nonetheless, compared with human pedagogy. these works still lack a sophisticated student model that can accommodate the teacher's 's cooperation into his learning and acts differently than learning from passive data. Machine teaching algorithms model cooperative teacher
giving instructions in the format of data examples for continuous parameter [T]] 70 71 40), Bayesian concept [8] 44 or version space learning [2), but seldom do they consider how learners may interpret differently between the data picked intentionally by the teacher and sampled randomly from the world. Standard Lff takes in demonstrations from an (approximately-) optimal expert to learn the underlying reward function 4 Hadfield-Menel et al. 27,, Ho et al.
[31 32 shows the advantage of using pedagogical rather than optimal demonstrations, yet, in either case, the learners are not aware of the teacher.. Shafto et al. (54), Yang et al. [68]. Wang et al.
[62 64] move one step further and proposed recursive cooperative inference models having both the teacher and the student reasoning each other, an ability known as theory of mind (ToM) [50 8 The first work modeled and predicted human behavior while the latter three managed to integrate ToM into machine learning. Despite the confined to Bayesian concept learning with finite hypothesis space, in which the Sinkhorn scaling [57] is tractable. It is unclear how to apply their algorithms to settings involving continuous hypothesis spaces, such as learning neural networks.
theoretical contribution, their approach [68 62 64
In this paper, we study how to integrate the cooperative essence of pedagogy into machine parameter learning and propose a teacher-aware learner who learns significantly faster than naive learner, given an iterative machine teacher 40 41 The learner estimates the teacher's data selection process with distribution and corrects his likelihood function with this estimation to accommodate the teacher's 's intention. Maximizing the new likelihood enables the learner to utilize both explicit information from the selected data and implicit information suggested by the pedagogical context. We theoretically proved the improvement brought by the learner's ' teacher awareness and justified our results with various experiments. We believe that our work can provide insights into both machine interactions, such as online education, and machine- communications, such as ad-hoc teamwork (X]
Our main contributions are i) modeling teacher-awareness for generic gradient optimization based parameter learning: ii) theoretically proving the improvement guaranteed by the teacher-aware learner over the naive learner under mild assumptions; tii) empirically illustrating the advantage of teacher-awareness learner when interacting with both machine and human teachers.
2
Related Work
Machine Teaching Our work is related to machine teaching as we used an iterative machine teacher in our framework. Machine teaching [70] 71 40 49) solves the problem of constructing an optimal (usually minimal) dataset according to target concept such that a student model can learn the target concept based on this dataset. Most of the machine teaching algorithms consider a batch setting, where the teacher designs a minimal dataset and provides it to learning algorithm in one shot [10 70 47 [T] Iterative machine teaching has also been studied, where the teacher gives
out data iteratively and checks the learner's s status before selecting the next data [34 6 40 41 43)
but previous works didn't consider teacher-aware learners. There are also works applying machine teaching to inverse reinforcement learning (IRI.) and L.f 10 27 K2] 28 Our IRI. experiments are different from those works as our data are provided iteratively and sequentially. Also, our learner is aware of the teacher's intention. Ho et al. [1 32 integrated Bayesian rule in LiD to model mutual reasoning between the teacher and the learner, but they mainly used their model to explain human data. A theoretical study of the
presented of the teacher-aware learners in [73 Ir where the teacher and the learner are aware of their cooperation. Their analysis mainly attends to version- space learners which maintain all hypotheses consistent with the training data, and cannot be applied to hypothesis selection via optimization, such as learning parameters.
Peltola et al. [49 studied machine teaching for an active sequential multi-arm bandit learner. Although they also have helpful teacher and teacher-aware learner, their problem setting is different from ours. First, in every round of multi-arm bandit, the learner can actively choose an arm to pull, and

---

then the teacher provides feedback for that arm, while in our cases, the data batch in each round is sampled randomly and independently from the learner's current status. Second, as the teacher can only give binary (success or not) feedback to the learner, the counterfactual reasoning required for pedagogy is significantly simplified. Besides, they required that the same feature representation for the arms is shared between the teacher and the learner. Also, the learner doesn't 't have to know the underlying parameter exactly to perform well in multi-arm bandit games, while in most of our cases, the learner tries to match the teacher's s model exactly. Fisac et al. [20] used similar formulation to model cooperative value alignment within a human-robot team. They assumed the human knows the robot's value function during interactions, and parameters to be aligned are sparse and low dimensional.
Learning to Teach Sharing the same goal as machine teaching, learning to teach (L2T) also seeks a teaching algorithm to improve the learning efficiency of AI. While machine teaching usually models the question as an optimization problem and solves for a closed-form teaching algorithm, works in I.2T tend to train the teaching model in the process of the teacher-student interaction with gradient based optimization [67 reinforcement learning (RL) algorithms [19]. Nevertheless, these works also focus mainly on the teacher algorithm and assume teacher-unaware learners, In some tasks, typically when the student aims to learn a concept in discrete space, the teacher can track the learner's status by modeling his belief over the concept [54]. As the beliefs within the learner's s mind are not usually known by the teacher, the teaching process can be modeled observable Markov decision process (POMDP) [46], solving the optimal policy of which returns a teaching algorithm [5T] 65]. From the teacher's perspective, the unknown part of the environment is the learner's 's belief, a probability distribution over the concept space, so she has to form another belief over the learner's belief. The intractable modeling of belief over continuous variables usually requires approximation methods such as particle filters [51] 65] to solve, restricting the scope of these algorithms naive learners and relatively simple learning tasks. Interactive POMDP [23] 66), a probabilistic multi-agent model, can also be used to model cooperative pedagogy with recursive teacher-learner reasoning. However, the nested belief over belief also suffers from intractability and is hard to scale up. If the concept space of the learner is continuous by itself, such as high dimensional continuous parameter spaces our case, handling the belief over belief becomes difficult.
partially
Cooperative Bayesian Inference Shafto et al. [54] studies human pedagogy with examples using Bayesian learning models. The cooperative inference [68] 63, 64 in machine learning also formalizes full recursive reasoning from the perspectives of both the teacher and the learner. Distinctive from their concept learning settings, in this paper, we focus parameter learning, in which the student has intractable posterior distribution and learns via gradient-based optimization. In addition, only consider the problem of single interaction between the teacher and learner. Wang et al. [62] proposed sequential cooperative Bayesian inference algorithm and showed its performance advantage over naive Bayesian learner analytically and empirically. Nevertheless, they were still discussing concept learning with finite and usually small data and hypothesis sets. The Sinkhorn scaling [30 62 64] becomes infeasible when dealing with continuous parameter learning.
(54] 68
63 64
Pragmatics Reasoning Our work is inspired by the study of pragmatics (how context contributes to the meaning) 26 27) and ToM [50
8. The rational speech act (RSA) model raised by Golland et al. 24 and developed in [2 54 25 accommodates the idea of using not only the utterance but also the selection of the utterance understand the speaker. Previous works in these areas are mainly from human action interpretation [21 660 36), language emergence [69] (35,, linguistics [33] 2 [3]
and cognitive science 255 g perspectives. To our knowledge, our work is the first to relate pedagogy
and recursive reasoning to generic parameter learning and shows a provable improvement. Shafto et al. [54) proposed computational models for more diverse concept spaces, but mainly focus on modeling and predicting human behaviors.
3 Background
Finding the optimal way of teaching parameters has been a challenging problem because of the
continuous state space and long horizon planning. One common framework is machine teaching [70]
71. Here, we adopt an iterative variation of machine teaching [40), consisting of three entities: the learner, the teacher and the world. The world is defined as parameter w", fixed and known
only by the teacher. Given a model y = h((;ww) parameterized by w, the world defined as
w" arg minwen ((a.)) [(h((;;)) v)], where P(z, v) is the data distribution in standard
3

---

machine learning. Here, 1 and h can vary across tasks, eg. 1 can be squared loss for regression, cross-entropy for classification, and negative log- likeeliidoodd for IRL 42 In this paper, we assume I to be a convex function and h(2;; w:) = h (r, w)) ). h can be an identity function for linear regression and softmax function for classification. Thus, we can omit in the loss function and write [((r, w). for short. I and h are common knowledge of the teacher and the learner.
Representation: The teacher represents an example as (z, y) while the student represents the same example as (7, 3/) (typically = Y/ and we use y when there is no ambiguity). The representation E X' and I E X can be different but deterministically related by an unknown mapping, ğ (z)). Suppose the teacher and the learner use model h (() and h((i,, v)) respectively, then w and v² are very likely in different spaces too. This is a common scenario when the teacher and the learner are a human and robot, or two robots from different factories. As the representation of examples can be complex, such as features extracted by deep neural networks [45] [22 29], using a linear model doesn't it impinge the expressive power of the overall model. the rest of the paper, we use w for the teacher's s parameter and y for the learner's s if they are from different spaces. Otherwise, we use w for both of them. We use x to refer to an example and its teacher representation. We use ž for its learner representation. Also, we don't 't specify the choice of G. Our only assumption about the teacher and the learner's 's representation will be discussed in Theorem !
Teacher: In general, the teacher can only communicate with the learner via examples. This restriction doesn't impinge the generality of the machine teaching framework, as the format of the data can In this paper, data are provided
be generic, such as demonstration used in the IRL. [72 5 61 42 iteratively. We use x' to denote the example used in the t-th iteration. The teacher aims to provide examples iteratively so that the student parameter y converges to its optimum 1² fast possible. Since the teacher doesn't 't know s² or v², we let the learner provide some feedback to her in each iteration so that she can track the pedagogy progress (details in Section 4.1
The learner has an initial parameter nd before learning. At time t, he has learning rate Th The learning algorithms for teacher-unawen learners are often simple. For iterative gradient based optimization, the learner usually uses stochastic gradient descent 44 learner receives (x', y²) from the teacher, his iterative update is:
Learner:
19 67 Suppose the
al((T ),r)) dve-1
D'= I/e
Mutual knowledge: We limit the mutual knowledge between the teacher and the learner, otherwise, the mutual reasoning between the two can theoretically become an infinite recursion. In this paper, consider teacher who assumes naive Iearner using Eq. ( to update his model. Mcanwhile, the learner knows the teacher selects data deliberately instcad of randomly (detailed in the next section). If we define a naive learner as having level-0 recursive reasoning, then the teacher and the teacher-aware learner have level-1 I and level-2 recursive reasoning respectively. This level of recursion is very close to human cognitive capability [14] [5]
and was also adopted by [49]
To summarize, the loss function 1, the model h, and the naive learner update function are common knowledge to the teacher and the learner. w* and the teaching mechanism are only known by the teacher, while of and the learning mechanism, i.e. how to update s² given data, are only known by the learner. He knows the teacher intentionally selects helpful data according to her estimation of himself, and the teacher assumes that the learner learns following Eq. (T). For our teacher-aware learner, this assumption is inaccurate, but we'll 'Il show how the proposed learner can learn much faster than a naive learner.
4 Teacher-Aware Learning
4.1 Cooperative Teacher
We first define the teacher whom the learner should be aware of. Let's consider a teacher using the same feature representation as the learner and knowing his parameter in each iteration. 440 termed
this kind of teacher the omniscient teacher, who, in the t t-th iteration, greedily chooses example

---

from a data batch D = [(Ti Us) ~ P(x, v)}:
al((2, 1o","""'"1 (o,,,w"1)), ).3))
w 2
(z',8") = arg min
Wh +/e
Ois
(x.
Oll (" 8) ? + 20p6
Øi((z, (1((,, )=
(nn(w= (w- w²
arg max (x.
Owt-1
Owi-1
The expression after arg max in Eq. (2) is defined as the teaching volume TV (z) y/w²), which represents the learner's progress in this iteration. It is a trade-off between the difficulty and the usefulness of example [see 40 sec. 4.1]. Notice that the teacher has control over D which is sampled from the data distribution P or from a large dataset. She only selects the best example from D'. Given D with mild batch size, c.g. 20, the arg max Eq. 2 can be exactly calculated.
Lessard et al. (88 has proved that, for an omniscient teacher, teaching greedily is sub-optimal. Yet, their findings cannot be directly applied to more practical teaching scenarios. Thus, we keep leveraging the greedy heuristic to model our cooperative teacher and generalize it to a non-omniscient teacher who doesn't fully know the learner in every iteration.
Suppose the teacher neither knows the learner's s/~ nor 2/* and they use different feature representa- tions of the data. To teach cooperatively, she has to imitate the learner's 's model in her own feature space and use w to guide the teaching. This can be done approximately if, in every round, the learner gives the inner products of μ²ー1 and the data to the teacher as feedback. Given the convexity
of the loss function 1, we have: w w*, (l( (xr,w"")))
2 1((x, w( 1((x, w"),)) Now, Eq. (2 can be approximated by inner products between the model parameter and the data [40]. Denote the learner's feedback ax = (i,v(-1), V(z D 7 G(x), then the teacher will teach as following:
w
(-ri² (
al v) Oos
2 + 21p. (1(ae, v) - (((, ()),,w"),,, ,))
arg max
x
It has been shown that cooperative teachers using Eq. S can substantially speed up the learning of a standard SGD learner [40). Nonetheless, only having cooperative teacher doesn't 'tt provide us the most effective interaction between the two agents, as the learner doesn't 't exploit the fact that the data come from a helpful teacher [54] In the next section, we introduce a teacher-aware learner.
4.2 Teacher-Aware Learner
Now we propose a learner who integrates teacher's pedagogy into his parameter updating process. Suppose we have a distribution p(z, y /²² v) x exp(-I((ã, v). y)). denoted as (1 y). Then, applying gradient descent to [((z, v), y) is equivalent to maximizing log (z, y) wrt. V. Hence, learner updating parameters with Eq. (T) can be considered as performing maximum likelihood estimation (MLE) when the data are randomly sampled from P(x., v)-
Nonetheless, in the machine teaching framework, data no longer randomly sampled from P(r, v)- tearher-aware learner should rectify his updating rule by considering teacher's 's helpfulness. Given the dataset De time the learner can postulate that the teacher more likely choose the example she thinks helpful following p(x, u/v² = =v, u,,l_1 D*), denoted as go( g D²) for short:
cXp(3.TV d (3. 1/2² )) S(DY./ exp( (3 v (5', , ((())
(x, 1/1' ,D')=
Be 200
(4)
(((,,,"')),,0 v) 2m ( (((=))))))))((((())))))) (5)
with TV (i, ((p = -n² gレー-1
(((,
The Boltzmann noisy rationality model 7 indicates that the teacher samples data according to the soft-max of their approximation of the teaching volumes, calculated wrt. her z² and the inner product feedback from the learner. Although, in practice, this estimation is usually different from the teacher's s actual example selection distribution, which is hard-max, corresponding to Br + oo, maximizing wrt. can still improve the learning.
5

---

The learner now wants to learn V, which not only makes y' more likely to be the correct label of x', but also (x', yy" more likely to be chosen from D Intuitively, given all data in D' are coherent with the true distribution, the teacher gives (a"; y²) but not other examples. With what v can the probability of this selection be maximized? So, at every time I, the student maximizes (r v²) and I (x²,, " D9) wrt. simultaneously. We can still use gradient descent. Omit when there is no confusion. Denote gx(()) @(((a))).)) then we have (derivation in supplementary Section A.1 87
= t- -17 aU a1((3', 1/ ). alog (,' v²)
dv -=wt-1) pi-l - m1re 1 223m (gat - (ot(v=
Out-1
(6)
Notice that y²= is constant in qu (a', ,e,,"10 301n²~1 D*) and the optimization is wrt. V, which is treated as v² in the calculation. The gradient is computed at v/ = v²=1. This is equivalent to maximizing a new log-likelihood function log (pu(x', y
d')g,(zt (x", y² y'|n'11 Dt).. an approximation of the log probability
that (x", v') being sampled in D' and then being selected by the teacher given V² v. The product is an approximation of p((x', .y'), D9 D9 1² = v) because the sampling of data in D' except (x* y²) regarded as deterministic. When 3, 0, i.e. the learner thinks the teacher uniformly samples data, and Eq. (6) becomes regular SGD.
An interpretation of the benefits brought by Eq. (6) that the learner not only learns from the literal meaning of the example selected by the teacher (The second term), but also compares that example with "also-rans" in D' (the third term), forming context incorporating additional information. This is a prevailing phenomenon in human communication, as messages often convey both literal In other words, we can acquire not only explicit information from what others said but also implicit information from what others didn't say. When the message space is finite and known, exact computations of the implicit information becomes tractable. Therefore, in scenarios like human-robot interactions, where robots usually provide predefined user interfaces with a fixed choice of instructions, our algorithm can easily conduct counterfactual reasoning by comparing the user's selected instruction with the others and deliver faster learning than only using the selected one. In Section 5.2 our experiment with humans as the teacher illustrates such an advantage.
meanings and pragmatic (contributed by the context) meanings 60 58
One nuance is that if we use y²ー as the v in Eq. (4) the second term of the teaching volume will (((""/"""))) That is, the be O. Thus, to better approximate v², in practice, we plug in D² T)- dyl-1 learner first updates -/11 just like naive learner. Then he calculates the gradient of log qu wrt. the new // and does an additional gradient descent corresponding to the last term in Eq. (6). Also, in supervised learning settings, the teacher needs to provide labels of the whole dataset for the learner to calculate the expectation. This is mild requirement easy to be satisfied in practice. In the iterative process, D' is mini- sampled from large dataset with small batch size, say 20 examples. Thus, Ezzg. [g [v)] can be calculated exactly, and, compared with the standard mini-batch gradient descent, the only additional information needed from the teacher is the index of (ar,, 31). In fact, we can further relax this condition by letting the learner estimate Ez~q. [ga (v)] with only a subset D' C Dt. In our experiments, we show that with only one random unchosen example provided, i.c. [Da| 1, the teacher-aware learner outperforms the naive learner. See Algorithm | for details.
We now prove the teacher-aware learner can always perform better than a naive learner given proper conditions.
Theorem (Local Improvement). Denote of = = (n For a specific loss function I, given the same learning status -²= and a teacher following Eq. (3. suppose x! satisfies that x' itself maximizes TV it (z, 11, Denote '' as the z E D which achieves the second largest vv", TV es (a. 1)v" Suppose that 11 (b*))l2 S for any x E D IV''"'vv 9z' (7²) ga> (i²)) > then with large enough 3r, the teacher-an aware learner using Eq. (6) is guaranteed to make no smaller progress than a naive learner using Eq. Α
One intuition for the assumption is that the best example selected by the teacher does bring more benefits to the learner than the other examples do. Suppose we have e = E 7ega (it) ), then
moving from e to vf follows m (gaa (7*) gae (ū²)). The assumption (7" -0--"-_ -0=,,=="0"" (z=) gas (b*)) =
(v' - gar (D*)- Ir* (Dt)) > 0 simply suggests that updating with xt gives the learner an advantage over updating with J.. The advantage points to v² (the two vectors v² 50 and g2 (7²) - g (D')
form an acute angle).

---

Algorithm 1: Iterative Teacher-Aware Learning
Input: Data distribution D, teacher parameter w", learning rate 1/t, teacher estimation scale 3e
Result: vTT)
Randomly initialize student model (0) ~ Uniform N);; Set t = and T as the maximum iteration number;
2 while t < T do
Teacher gets data batch D DD
Learner reports 0₂ = fylt-1),an a) for all æ E D³ to the teacher Teacher selects data for time t:
(a*, v²) arg (x,))
(-nè Ala..)))
x||
2m ((a2.)) 3) -1 [[WWW 2))
80.
Learner uses the selected data (zf,y") v²) and D' to calculate e(0 v(t-1) 172
al1(2",0 ,0 (1 11)'²²)
Bult-1) (0(("",(00))))
(((().0(0)).))
ult) (00) -23r1/²
06i0)
(a (E.ybltt 1,D))
P.TV aw (2,glutt-),,0))
exp
where qoie: (it, y/ul(t1),DD' DD)
Deeeee exp 8.TV oces ( (flee
with TV pco (2, ylult-)))), (,, Dd) defined Eq.
10
11
== t +
12 end
Corollary 2 (Global Improvement). Under the same condition of Theorem 1 suppose that [[a v) fovll2 and (((,,)). are L-Lipschitz for x with any Suppose the sample set Dt satisfies that for any x E D`, there exists x' Dt such that ||>' - =x112 S (/ + 4m)) for any t, where T the total number of iterations. Then if the inequality
|M
max v212 ----v112 ())
TV uz (x,, C0,ylun v()) < 11 - v²112 max (D-M)D
TV (l2))
(7)
holds for any Vv. 1/2 that satisfy ||2, - v2 S 1122 -v*12 then with the same parameter initialization. learning rate and teacher following Eq. slower than a naive learner to error.
3 leacher-aware learner can always converge not
To guarantee that ||z'- =zl12 (TL(( + 4n)) for any D, need the subset D C. D to be "uniform distributed' on D. To achieve this goal, we can uniformly sample point E D and let D to be the set of these points. It is easy verify that the 'uniform distributed" property holds with high probability when [T'| is large enough. Meanwhile, Eq. (7) in Corollary i defined as teaching monotonicity Liu et [40], and they proved that the squared loss satisfies teaching monotonicity given a dataset D = {a E Rd ||=||| <
R} [see 40) proposition 3J. The main difference between Eq. Z and that in Liu al. 40 that Eq. (7) works for the non-omniscient teacher setting, while Liu et al. 40 focuses on the omniscient teacher setting. Detailed proofs of the theories can be found in Section of our supplementary.
Experiments
5.1 Machine Teacher
To justify the effectiveness of ITAL, compared it with iterative machine teaching (IMT) with naive learner on regression, classification, and IRL tasks. The coverage of squared loss, cross-entropy loss, and negative likeliillo proves the robustness our algorithm on various selections of 1. For regression tasks, we measured the performance using the difference between ||w* w l| and the mean squared loss of the test set. For the classification task, we measured the difference, the cross-entropy loss, and the classification accuracy of the test set. For online IRL, problems, we measured the parameter difference, the total variance between the teacher's and the learner's policies, and the average rewards achieved by the leamnr. The feature dimension of the teachers can be
7

---

L2 Distance
Tock
150
1718
Linear regression. Divtaacce
(b) Gaussian data.
L2 Distance
Nap
Map
Map
2o:14
Hammn
90
1000
100
190
1956
2000
200
304
150
100
129 1004
199 2011
te-di tacts
(c) CIFAR-10.
(d) Tiny ImageNet.
12 Bietancs
L2 Distance
Terl ted?
Traniin Haratio
Training Iteration
Mop A
Map
Map C
Map
Map E
(c) Equation.
(f) Online IRL.
(g) Human study results.
Back
500
IDA--D
TAL-11
BAL-IE and ITAL-
IEAL2
Figure/ 1: Fig. Fig. If Cooperative teacher results. Our method always gives substantial
improvement over IMT, showing the effect of teacher-aware awareness. Within 2000 steps. ITAL already show convergence, while a naive learner only learns to limited extent in most tasks. Fig. Ig In the top plot, the height of each bar represents the decrease of the L2-distance between the learner's reward parameter and the ground-trute parameter. In the bottom, the height represents the accumulated reward. Paired t-tests were conducted between Human (Machine) ITAL & IMT respectively.
different from that of the learners in some experiments. In Fig. D we show the results of the teacher having smaller feature dimension than the learner does. We show the opposite in the supplementary Section B. Batch means the learner uses all the data in the mini- to calculate the mean gradient. SGD means the learner randomly selects an example in the
to calculate the gradient.
algorithm with M indicates | The mini-batch D' is randomly sampled at every step with batch size 20. The learning rate is 1e-3 for all the experiments. 3, is in the scale of 1e4, varying for different settings. We grid search 3, starting from 1e4 and use the largest one inducing Eq. (4) that is no longer a delta function. We ran each experiment with 20 different random seeds to calculate the mean and its standard error, shown in Fig. [I
ITAL-Ms represent
can be seen that using the full mini-batch gives almost identical learning performance as using only one random sample from it. IMT has noticeable but limited improvements comparing with Batch and SGD, suggesting not necessarily substantial advantage brought by the helpful teacher. ITAL, on the other hand, significantly outperforms all other baselines, even with only 2 data points as the approximation of the full mini-batch. The learner modeled by these baselines only lcarns from the examples, but when the examples are longer acquired randomly but from intentional teacher, the example selection of the teacher also conveys a large amount of information. In particular, the teacher-aware learner absorb information from not only the selected examples but also the unselected ones. As the learner has access to more unselected examples, he has a better approximation of the teaching process and learns more efficiently. Additional experimental details can be found in supplementary Section
5.1.1 Supervised Learning
Linear Models on Synthetic Data: In these experiments, we explored the convergence of our method in linear regression and multinomial logistic regression. For linear regression, we randomly generated a -ddimensional vector and bias term as the w", and E [xx((+1) as the training set, with the last column being all 1s. The labels are Y = =x=,. For the classification task, we randomly generated K points the d-dimensional space, each of which is used as the mean of a normal distribution. Then sampled N/K points from each Gaussian distribution together as the
8

---

training data. The labels are the indices of these distributions. With these data, we trained a logistic regression model using Scikit-learn 448, and used the coefficients as the teacher's w We used a random orthogonal projection matrix to generate the teacher's feature space from student's, At every step, a subset of the training data is randomly selected as the mini-batch. The data points in that mini-batch along with their labels and the index of the data selected by the teacher are sent to the student. Details of the data generation can be found in supplementary Section B.I
Linear Classifiers on Natural Image Datasets: We further evaluated our teacher-aware learner on image datasets, CIFAR-10 10 37) and Tiny ImageNet [ (an adaptation of ImageNet [16] used in Stanford 23 In with 200 classes and 500 images in each class). In these experiments, the teacher tried to teach the parameters of the last fully connected (FC) layer in a convolutional neural network (CNN) trained on the dataset. We trained 3 baseline CNNs independently to do CIFAR- 10 10-class and ImageNet 200-class classification. All of them achieved reasonable accuracy (2 82% for CIFAR-10 10 and 2 58% top-1, 2 85% top-5 for ImageNet). For CIFAR- 10, we trained three different types of CNNs, CNN-6/9/12. For ImageNet we used /GG-13/16/19 [56). The features fed into the last FC layers are extracted to be the teaching dataset. The learner's ' feature is from CNN-9/VGG- 16 and we set the teacher as either CNN-6/12 or VGG-13/19, Details about CIFAR-I0 and Tiny ImageNet experiments are in B.3 and Section B.4 respectively.
Linear Regression for Equation Simplification: In this experiment, we learned a linear value function that can be used to guide action selections. Given polynomial equations with fraction coefficients and unmerged terms, we want to simplify them into cleaner forms with all the terms merged correctly, all the coefficients rescaled to integers without common factors larger than 1, and all the terms sorted by the descending power. For example, equation - _ y + Jay = _ + J will be simplified to y + 10ry + 6y² = O. We defined a set of equation editing actions and a set of simplification rules. For a given equation, we applied the rules, recorded every editing action, and collected a simplifying trajectory. With all the trajectories of the training equations, we trained value function by assuming that the value monotonically increases in cach trajectory. Then the teacher tried to teach the student this value function. We used three different feature dimensions: 40D, 45D, and 50D. The learner always used 45D, and the teacher used 40D or 50D. Details can be found in Section B.5
5.1.2 Online Inverse Reinforcement Learning
In this experiment, we changed from labeled data in standard supervised learning to demonstrations in IRL. The learner wanted to learn the parameter for a linear reward function r(s, w²) so that the likelihood of the demonstrations is maximized [5] [/]6 [2]. One challenge is that the max function in [59] is non-diffcrentiable. Thus, we approximated max with soft-max, namely: Eo ko.) with is controlling the level of approximation and leveraged the online Bellman gradient iteration (39]. The IRL environment is an 8x8 8 map, with a randomly generated reward assigned to each grid. If we encode each grid using one-hot vector, then the reward parameter is a 64D vector with the i-th entry corresponding to the reward of the i-th grid. The agent can go up, down, left, or right in each grid. All demonstrations are in the format of (s, a), where s indicates grid and a an action demonstrated in that grid. The teacher uses a shuffled map encoding. For instance, if the first grid is [1, O,..... 0] to the learner, then it became [0, ..., 0, 1, 0, to the teacher. Details are included in Section B.6
Bellman equations
max(ag. ..., 0x,)
5.1.3 Adversarial Teacher
In addition to the cooperative setting that we assumed throughout the discussion above, we also explored if the learner can still learn given an adversarial teacher. An adversarial teacher doesn't 't mean that she gives fake data to the student, but she uses arg arg min in Eq. { instead of using arg max. That is, she always chooses the least helpful data for the learner. Hence, a learner, being aware of this unhelpful pedagogy, will adjust Eq. (3) accordingly by using 33 < O. We redid all previous experiments with an adversarial teacher and showed that our learner can still learn effectively given an adversarial teacher, while naive learner barely improves (see Fig. 14jin Section This experiment justifies the universal utility of modeling the teacher's 's intention regardless of the informativeness of the teaching examples.
9

---

5.2 Human Teacher
In the previous section, we showed that teacher-awareness substantially accelerates learning, given machine teacher. In this section, we further investigate if our teacher-aware learner can show an advantage in scenarios where humans play the role of the cooperative teacher. We hypothesize that despite the discrepancy between the pedagogical pattern of human and machine teachers, our learner can still benefit from his
modeled with Eq. (4)
We conducted a proof-of-concep human study with similar but simplified version of the IRL experiments in Section [.1.2 To better suit human participants, we first change the maps from 8 x 8 to 5 Second, instead of assigning random continuous rewards the grids, we color them with white, blue, and red, representing neutral, bad, and good tiles. In each teaching session, a participant is given one of the five reward maps as the ground truth and randomly initialized learner to be taught. Then, the participant will be asked to teach the learner about the ground truth reward in each grid by providing (s, a) examples as Section 5.1.2 In each time step, we construct the examples by randomly sampling set of 10 grids and drawing an arrow on each sampled grid indicating which direction the learner should go if in that grid. The human teacher is asked to choose the most helpful arrow given the learner's s current reward map. The map configurations are in Fig. [5] in Section B.8, similar map setup for reward teaching was used by Ho et al. [II. Every participant will teach both the naive and the teacher-aware learner about the same map. We then run paired sample t-test to compare the learning effect of the two types of learners. We show the improvement of the L2-distance between the learner's ' reward parameters and the ground-truth reward parameters and the accumulated reward Fig. 1g Comparison of policy total variance and learning curves are included in Section B.8 of the supplementary.
For all maps, the ITAL method has a significant (p-value 0) advantage over its IMT counterparts. The human ITALS all perform worse than machine ITALS. This is as expected as we directly reuse the machine teacher model to simulate humans. There no guarantee that all the participants follow the same teaching pattern as the machine teacher, or even have a consistent teaching pattern at all. Yet, we still manage to grasp human cooperation some extent. To illustrate the influence of the teacher model, we also teach the ITAL learner with random teacher, who samples the example uniformly every time and is not cooperative at all. As shown in Fig. Ig this combination doesn't benefit the learner, because the mismatch between the imagined cooperative teacher and the actual random teacher will very likely introduce iverrntterprttation results justify that human teachers do have cooperative (contrary uniform) pedagogy patterns and the current teacher-awe aware model can take advantage of them. Finding a comprehensive and accurate human-robot communication model will be open question for future works.
the examples. To summary, these
6 Discussion and Conclusions
Pedagogy has profound cognitive science background, but it hasn't received much attention in machine learning works until recently. In this paper, we integrate pedagogy with parameter learning and propose a teacher-aware learning algorithm. Our algorithm changes the model update step for the gradient learner to accommodate the intention of the teacher. We provide theoretical and empirical evidence to justify the advantage of the teacher-aware learner over the naive learner.
To be aware of the teacher, the learner needs an accurate estimation of the teaching model. In many cases, such model is not directly accessible, e.g. when there is human-in-thed in-the-loop. In this paper, we model the teacher in heuristic manner. Our human study proved the generality of this model, especially when the learner only assumes sub-optimal teacher with Boltzmann rationality. future work, more advanced teacher model should be investigated, acquired through task-specifi data and/or interactions between the agents. Another limitation of our work is that, our current setting, the learner's feedback is restricted to be inner products. A more generic message space can be leveraged to develop comprehensive learning as a bidirectional communication platform. We believe work illustrates the promising benefits of accommodating human pedagogy into machine learning algorithms and approaching learning as a multi-agent problem.
10

---

Acknowledgments and Disclosure of Funding
The work was supported by DARPA XAI project N66001-17-2-4029, ONR MURI project N 16-1-2007 and NSF DMS-2015577. We would like to thank Yixin Zhu, Arjun Akula from the UCLA Department of Statistics and Prof. Hongjing Lu from the UCLA Psychology Department for their help with human study design, and three anonymous reviewers for their constructive comments.
References
[1] httss//wwww.kugg....o////ttimmem
2017.
[2] J. Andreas and D. Klein, "Reasoning about pragmatics with neural listeners and speakers," arXiv preprint arXiv: 160.00052, 2016.
[3] D. Angluin, "Queries and concept learning," Machine learning, vol. 2, pp. 319-342, 1988.
[4] B. D. Argall, S. Chernova, M. Veloso, and B. Browning. "A survey of robot learning from demonstration," Robotics and autonomous systems, vol. 57, no. 5, pp. 469 469-483, 2009.
[5] M. Babes, V. Marivate, K. Subramanian, and M. L. Littman, "Apprenticeship learning about multiple intentions," in Proceedings of the 28th International Conference Machine Learning (ICML-11), 2011, pp. 897-904.
[6] H. Bak, Y. Choi, A. Akrami, Witten, and J. W. Pillow, "Adaptive optimal training of animal behavior," in Advances in neural information processing systems, 2016, pp. 1947-1955.
[7] C. Baker and B. Tenenbaum, "Modeling human plan recognition using bayesian theory of mind," Plan, activity, and intent recognition: Theory and practice, pp. 177-204, 2014.
[8] C. L. Baker, Jara-Etting Ettinger, R. Saxe, and B. Tenenbaum, "Rational quantitative attribution of beliefs, desires and percepts in human mentalizing," Nature Human Behaviour, vol. I, no. 4, pp. 1-10, 2017.
[9] S. Barrett, A. Rosenfeld, S. Kraus, and P. Stone, "Making friends the fly: Cooperating with new teammates," Artificial Intelligence, vol. 242, pp. 132-171, 2017.
[10] M. Cakmak and M. Lopes, "Algorithmic and human teaching of sequential decision tasks," in Twenty-Sixth AAAI Conference on Artificial Intelligence, 2012.
[11] Y. Chen, O. Mac Aodha, Su, P. Perona, and Y. Yue, "Near-optimal machine teaching via explanatory teaching sets," in International Conference on Artificial Intelligence and Statistics, 2018, pp. 1970-197 1978.
[12] Y. Chen, A. Singla, O. Mac Aodha, P. Perona, and Y. Yue, "Understanding the role of adaptivity machine teaching: The case of version space learners," " in Advances in Neural Information Processing Systems, 2018, pp. 1476-1486.
[13] R. Cohn-Gordon, N. Goodman, and C. Potts, "An incremental iterated response model of pragmatics," in Proceedings of the Society for Computation in Linguistics (SCiL) 2019, 2019, pp. 81-90. [Online]. Available: http::/wwww.aaddweooo///allogo//
[14] H. de Weerd, R. Verbrugge, and B. Verheij, "Higher-order theory of mind in the tacit communi- cation game," Biologically Inspired Cognitive Architectures, vol. 11, pp. 10-21, 2015.
[15]
"Negotiating with other minds: the role of recursive theory of mind in negotiation with incomplete information," Autonomous Agents and Multi-Agent Systems, vol. 31, no. 2, pp. 250-287, 2017.
[16] J. Deng, W. Dong. R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, "Imagenet: A large-scale hierarchical image database," in 2009 IEEE Conference on Computer Vision and Pattern Recognition, 2009, pp. 248-255.

---

[17] T. Doliwa, G. Fan, H. U. Simon, and S. Zilles, "Recursive teaching dimension, ve-dimension and sample compression," The Journal of Machine Learning Research, vol. 15, no. pp. 3107-3131, 2014.
[18) B. S. Eaves Jr, A. M. Schweinhart, and P. Shafto, "Tractable bayesian teaching," in Big Data in Cognitive Science. Psychology Press, 2016, pp. 74-99.
[19] Y. Fan, F. Tian, T. Qin, X.-Y. Li, and T.-.. Liu, "Learning to teach," in International Conference on Learning Representations, 2018. [Online]. Available: https: wttp://wwww..ttoooom/dd----euuue
[20] Fisac, M. A. Gates, J. B. Hamrick, C. Liu, D. Hadfield-Menell, M. Palaniappan, D. Malik, S. S. Sastry, T. L. Griffiths, and A. D. Dragan, "Pragmatic-pedaaggic value alignment," in Robotics Research. Springer, 2020, pp. 49-57.
[21] M. Frank, N. Goodman, P. Lai, and J. Tenenbaum, "Informative communication in word production and word learning," in Proceedings of the annual meeting of the cognitive science society, vol. 31, 2009.
[22] M. C. Frank and N. D. Goodman, "Predicting pragmatic reasoning in language games." Science, vol. 336, no. 6084, pp. 998-998, 2012.
[23] P. J. Gmytrasiewicz and P. Doshi, "A framework for sequential planning in multi-agent settings." Journal of Artificial Intelligence Research, vol. 24, pp. 49-79, 2005.
[24] D. Golland, P. Liang, and D. Klein, "A game-theoretic approach to generating spatial de- scriptions," in Proceedings of the 2010 conference on empirical methods in natural language processing. Association for Computational Linguistics, 2010, pp. 410-419.
[25] N. D. Goodman and M. C. Frank, "Pragmatic language interpretation probabilistic inference." Trends in cognitive sciences, vol. 20, no. 11, pp. 818-829, 2016.
[26] H. P. Grice, "Logic and conversation," in Speech acts. Brill, 1975, pp. 41-58.
[27] D. HadfieldMMenele S. Russell, Abbeel, and A. Dragan, "Cooperative inverse reinforcement learning," in Advances in neural information processing systems, 2016, pp. 3909-3917.
[28] L. Haug, S. Tschiatschek, and A. Singla, "Teaching inverse reinforcement learners via features and demonstrations," " in Advances in Neural Information Processing Systems, 2018, pp. 8464- 8473.
[29] K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition." in Proceedings of the IEEE conference computer vision and pattern recognition, 2016, pp. 770-778.
[30] D. Hershkowiz, U. G. Rothblum, and H. Schneider, "Classifications of nonnegative matrices using diagonal equivalence," SIAM journal on matrix analysis and applications, vol. 9, no. 4, pp. 455-460, 1988.
[31] M. K. Ho, M. Littman, MacGlashan, F. Cushman, and J. Austerweil, "Showing versus doing: Teaching by demonstration," in Advances in neural information processing systems, 2016, pp. 3027-3035.
[32] M. K. Ho, M. L. Littman, F. Cushman, and J. L. Austerweil, "Effectively learning from pedagogical demonstrations." in Proceedings of the Annual Conference of the Cognitive Science Society, 2018
[33] G. Jäger, "Game theory in semantics and pragmatics," Semantics: An international handbook of natural language meaning, vol. 3, pp. 2487-2425, 2012.
[34] E. Johns, O. Mac Aodha, and G. J. Brostow, "Becoming the expert-interactive multi-class machine teaching," " in Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, 2015, pp. 2616-2624.
12

---

[35] Y. Kang, T. Wang, and G. de Melo, "Incorporating pragmatic reasoning communication into emergent language," in Advances in Neural Information Processing Systems. H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, Eds., vol. 33. Curran Associates, Inc., 2020, pp. 10 348-10 10 359. [Online]. Available: 7520fa3 445544444455656655 a55fd Pap
[36] F. Khani, N. D. Goodman, and P. Liang. "Planning, inference and pragmatics in sequential language games," Transactions of the Association for Computational Linguistics, vol. 6, pp. 543-555, 2018.
[37] A. Krizhevsky, G. Hinton et al., "Learning multiple layers of features from tiny images," University of Toronto, 2009.
[38] L. Lessard, X. Zhang, and X. Zhu, "An optimal control approach sequential machine teaching." arXiv preprint arXiv: 1810.0665 2018.
[39] K. Li and J. W. Burdick, "Online inverse reinforcement learning via bellman gradient iteration," " arXiv preprini arXiv: 170..0007.9999. 2017.
[40] W. Liu, B. Dai, A. Humayun, C. Tay, C. Yu, L. B. Smith, J. M. Rehg, and L. Song, "Iterative machine teaching," " in Proceedings of the 34th International Conference on Machine Learning- Volume 70. JMLR. org, 2017, pp. 1149-2158.
[41] W. Liu, B. Dai, X. Li, Z. Liu, J. M. Rehg. and L. Song, "Towards black-box iterative machine teaching," " Proceedings of the 35th International Conference on Machine Learning, 2018.
[42] MacGlashan and M. L. Littman, "Between imitation and intention learning," "" in Twenty-Fourly International Joint Conference Artificial Intelligence, 2015.
[43] F. S. Melo, C. Guerra, and M. Lopes, "Interactive optimal teaching with unknown learners," in Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, 1JCAI-18. International Joint Conferences on Artificial Intelligence Organization, 7 2018, pp. 2567-2573. [Online]. Available:
http://doo.orr/11.2499/3//i//// 20188/336
[44] Milli, P. Abbeel, and I. Mordatch, "Interpretable and pedagogical examples," arXiv preprint 17X1v:1711.000 2017.
[45] V. Mnih, K. Kavukcuoglu. D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller, "Playing atari with deep reinforcement learning," arXiv preprint arXiv:1312.5600 2013.
[46] G. E. Monahan, "State of the art-a survey of partially observable markov decision processes: theory, models, and algorithms," " Management science, vol. 28, no. 1, pp. 1-16, 1982.
[47] K. R. Patil, J. Zhu, t.. Kopeé, and B. C. Love, "Optimal teaching for limited-capacity human learners," in Advances in neural information processing systems, 2014, pp. 2465-2473.
[48] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and F. Duchesnay, "Scikit-learn: Machine learning in Python," Journal of Machine Learning Research, vol. 12, pp. 2830, 2011.
[49] T. Peltola, M. M. Çelikok, P. Daee, and S. Kaski, "Machine teaching of active sequential learners," in Advances in Neural Information Processing Systems, 2019, pp. 11 202-11213.
[50] D. Premack and G. Woodruff, "Does the chimpanzee have a theory of mind?" Behavioral and brain sciences, vol. I, no. 4, pp. 515-526, 1978.
[51] A. N. Rafferty, F. Brunskill, T. Griffiths, and Shafto, "Faster teaching by pomdp planning." " in International Conference Artificial Intelligence in Education. Springer, 2011, pp. 280-287.
[52] O. Russakovsky, J. Deng, H. Su, Krause, S. Satheesh, Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei, "ImageNet Large Scale Visual Recognition Challenge," International Journal of Computer Vision (IJCV), vol. 115, no. 3, pp. 21 1-252, 2015.
13

---

[53] B. Settles, "Active learning literature survey," University of Sesconsin-Maddisee Department of Computer Sciences, Tech. Rep., 2009.
[54] Shafto, N. D. Goodman, and T. L. Griffiths, "A rational account of pedagogical reasoning: Teaching by, and learning from, examples," Cognitive psychology, vol. 71, pp. 55-89, 2014.
[55] D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre. G. Van Den Driessche, Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot et al., "Mastering the game of go with deep neural networks and tree search," nature, vol. 529, no. 7587, p. 484, 2016.
[56] K. Simonyan and Zisserman, "Very deep convolutional networks for large-scale image recognition," in International Conference on Learning Representations, 2015.
[57] R. Sinkhorn and P. Knopp, "Concerning nonnegative matrices and doubly stochastic matrices," Pacific Journal of Mathematics, vol. 21, no. 2, pp. 343-348, 1967.
[58] N. J. Smith, N. Goodman, and M. Frank, "Learning and using language via recursive pragmatic reasoning about other agents." " in Advances in neural information processing systems, 2013, pp. 3039-3047.
[59] R. S. Sutton and A. G. Barto, Reinforcement learning: An introduction. MIT press, 2018.
[60] A. Vogel, M. Bodoia, C. Pouts, and D. Jurafsky, "Emergence of gricean maxims from multi- agent decision theory," in Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, 2013, pp. 1072-1081.
[61] M. C. Vroman, "Maximum likelihood inverse reinforcement learning," Ph.D. dissertation, Rutgers University-Graduall School-New Brunswick, 2014.
[62] J. Wang, P. Wang, and P. Shafto, "Sequential cooperative bayesian inference," in Proceedings of the 37th International Conference Machine Learning (ICML-20), 2020.
[63] Wang, Paranamana, and Shafto, 'Generalizing the theory of cooperative inference." in The 22nd International Conference on Artificial Intelligence and Statistics. PMLR, 2019, pp. 1851-1850.
[64] P. Wang, J. Wang, P. Paranamana, and P. Shafto, "A mathematical theory of cooperative communication." 2020.
[65] Whitehill and J. Movellan, "Approximately optimal teaching of approximately optimal learners," IEEE Transactions on Learning Technologies, vol. 11, no. 2, pp. 152-164, 2017.
[66] M. P. Woodward and R. J. Wood, "Learning from humans as an i-pomdp," arXiv preprint arXiv: 0274, 2012.
[67] L. Wu, F. Tian, Y. Xia, Y. Fan, T. Qin, L. Jian-Huang, and T.-Y. Liu, "Learning to teach with dynamic loss functions," in Advances in Neural Information Processing Systems, 2018, pp. 6466-6477
[68] S. C.-H. Yang, Y. Yu, P. Wang, W. K. Vong, P. Shafto et al., "Optimal cooperative inference," in International Conference on Artificial Intelligence and Statistics. PMLR, 2018, pp. 376-385.
[69] L. Yuan, Z. Fu, J. Shen, L. Xu, J. Shen, and S.-C. Zhu, "Emergence of pragmatics from referential game between theory of mind agents," arXiv preprint arXiv:2001.07752, 2020.
[70] Zhu, "Machine teaching for bayesian learners in the exponential family," in Advances in Neural Information Processing Systems 26, C. J. C. Burges, L. Bot- tou, M. Welling, Z. Ghahramani, and K. Q. Weinberger, Eds. Curran Asso- ciates, Inc., 2013. pp. 1905-1913, [Online]. Available: http://papees.nissccopppprr 5042-machine- teaching- for baaessian-eaaneesss in- the- tite--epooential-faan familly.-ffmmylii
[71] X. Zhu, "Machine teaching: An inverse problem to machine learning and an approach toward optimal education," in Twenty-Ninth AAAI Conference Artificial Intelligence, 2015.
14

---

[72] B. D. Ziebart, A. L. Maas, J. A. Bagnell, and A. K. Dey, "Maximum entropy inverse reinforce- ment learning," in Aaai, vol. 8. Chicago, USA. 2008, pp. 1433-1438.
[73] Zilles, S. Lange, R. Holte, and M. Zinkevich, "Models of cooperative teaching and learning." " Journal of Machine Learning Research, vol. 12, no. Feb, pp. 349-384, 2011.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] We explicitly discussed the
assumptions of our models in our problem setup and limitations in Section [
(c) Did you discuss any potential negative societal impacts of your work? [No] Our work aims to enable human-like AI behaviors and facilitate human-robot collaboration. At this moment, we cannot think of any negative societal impacts of our work.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] Details are in Section [ Section B and supplementary codes.
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes] Details are in Section Section Bland supplementary codes.
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [Yes] We use 20 random seeds and report the standard deviation (67% confidence interval)
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See supplementary Section B
4. If you using existing assets code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] Cifar-10 has the MIT license and ImageNet data are downloaded from https://www.imagennee wrw.--nettor///woolaadddod
(c) Did you include any new assets either in the supplemental material or as a URL? [No] But we discussed how our data generated and included the source code in the supplementary codc.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [No] The data we used are open-source and usage comply with their terms.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [No] The data used doesn't 't include any personally identifiable information.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [Yes] See supplementary Section B.8 for screenshots and supplementary Jupyter Notebook for instructions.
(b) Did you deseribe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [Yes]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [No] Our data was collected from student volun- teers.
15


Reviews:
[{"review_id": "hgFYQL0aZ6", "reviewer": "Reviewer_LPK1", "summary": "The paper presents a gradient optimization-based teacher-aware learner who can incorporate a teacher's cooperative intention into its likelihood function, and learn faster in comparison to the existing naive learning algorithms for machine teaching. The authors evaluate the proposed algorithm empirically on several machine learning tasks such as classification/regression/IRL and also on a simple setting where instruction/advice is taken from human teachers. Their results show that learning is indeed faster with a teacher-aware learner.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Pros**\n\n* The modeling of a teacher-aware learner is interesting.\n* The experimental evaluation catered to different types of learning models and showed in all the diverse settings (classification/regression/IRL) that a teacher-aware learner has accelerated learning.\n\n\n**Cons**\n\n* Writing: The paper could have been made clearer, particularly equation (6) in Section 4.2. Since this is a crucial equation illustrating the gradient update rule for such a learner, it could have been better annotated, and upon annotation, refer to its different parts in the subsequent text that follows. There is a typo on line 202 where I think Eq.5 was mentioned, by it should have been Eq.3. \n\n* Evaluation: While the evaluation was exhaustive, one could elaborate further on the Online IRL setting in the main paper, and put the details of the Linear classifier on natural image datasets in the appendix instead. The Online IRL setting is a more interesting set-up for the problem. Furthermore, the experiments with the human teacher were overly simplified for this setting. The method has the potential for dealing with a more complex instruction set of a human teacher, given that the learner model is general enough and operates under the realistic assumption of a sub-optimal teacher.\n\n\n\n**General Comments**\nThe paper presents important theoretical guarantees for the proposed teacher-aware learner under a set of practical assumptions. The learner model has the potential of conducting counterfactual reasoning by comparing the teacher's selected instruction with that of others, in the setting that the teacher's instruction set is finite. The experimental evaluation also looks promising, particularly the verification with human teachers. It would be interesting to evaluate the performance of such a learner in a real-world setting and explore empirically its robustness to noise in teacher instruction.\n\n*Originality*: Novel Approach\n\n*Clarity*: Largely clear.\n\n*Quality*: Good\n\n*Significance*: High", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper is still in its nascent stages and far from deployment in the real world. Hence, negative societal impacts are not a concern just yet.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "Lk0EsRSG1-C", "reviewer": "Reviewer_WHuF", "summary": "The manuscript presents iterative teacher aware learning, an approach in which learners learn from teaching while incorporating the intent of the teacher for continuously parameterized models. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper presents iterative teacher aware learning (ITAL) which allows teaching in continuous spaces, and learners who take advantage of the teaching intent to augment learning. The approach builds on prior work in machine learning and cognitive science, and is distinctive in incorporating two features: learners who learn from teaching *and* the teacher's intent, and the ability to learn in continuous parameter spaces. Prior research has done each of these, but not together, which represents the core novel contribution of the paper. Update rules are derived that extend gradient descent methods to incorporate the teacher's selection process. Theoretical results prove that teacher aware learners learn no slower than non teacher aware. Empirical results confirm theoretical analysis for an array of problems including learning from human teaching. \n\nPositives: \n- The extension to learning from cooperative teaching in continuous parameter spaces is interesting. \n- The theoretical results are clear and relevant. \n- The experiments show ITAL learns faster than iterative machine teaching (IMT). \n\nWeaknesses:\n-  The heuristic nature of the teacher model was not analyzed or scrutinized, which limited possible insights provided by the paper. \n- The learner feedback as an inner product is a curious choice that was not well motivated or explained. \n- The connection to human learning is limited and over stated in some places. \n\nDetailed comments: \n- \"diametrically distinctive\" Please check the word usage here. \n- \"the learning mechanism\" What is the learning mechanism? \n- \" Yet, their findings cannot be directly applied to more practical teaching scenarios\" Why? Please explain. \n- \"Thus, we keep leveraging the greedy heuristic to model our cooperative teacher and generalize it to a non-omniscient teacher who doesn’t fully know the learner in every iteration\" I do not understand the logic here. \n- \"This can be done approximately if, in every round, the learner gives the inner products of ν t−1 and the data to the teacher as feedback.\" What does this mean intuitively? \n- \" It has been shown that cooperative teachers using Eq. (5) can substantially \" Do you mean equation 3? \n- \" to better approximate ν∗, in practice, we plug\" What is the justification for this? \n- Figure 1 is too small to read. \n- \"Pedagogy has a profound cognitive science background, but it receives limited attention in recent machine learning works\" This isn't quite accurate. Earlier in the paper a number of machine learning papers were cited.   \n\n~~~~~~\nBased on the reviewing discussions, I have decremented my rating to a 6. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No discussion of limitations or societal impact were included. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {"ethical_concerns": "The response to 5b is a bit curious. The authors report empirical results with human teachers, which would suggest they ran an experiment. I assume they obtained IRB approval. However, the response to question 5b is no? (Even when we think there is no harm, the IRB must be consulted to receive a determination. "}}, {"review_id": "54evhei0cmH", "reviewer": "Reviewer_djzf", "summary": "**High-Level Summary**\n\nThis paper studies cooperative teacher and learner settings. In this setting, the teacher and learner are trying to solve a classification task with input $x \\in \\mathcal{X}$ and label $y \\in \\mathcal{Y}$. The setup works in an online fashion where at each time step, the teacher selects an example $(x, y)$ to present to the student from a large dataset and the student learns from this example. The teacher knows the optimal parameters. Previous work has focused on the setting where the teacher is aware of the learner and tries to select the example to ensure the student learns the fastest. This paper studies an extension where the learner is also aware of the teacher's feedback (teacher-aware learner). The key novelty is in the learner trying to maximize its parameters to fit the feedback provided by the teacher while also trying to predict the distribution the teacher is using to sample the data from. This has an additional benefit in that it can help the learner perform \"pragmatic reasoning\". This is related to previous work on Rational-Speech Act models (e.g., Golland et al., 2010). The key difference is that previous work on RSA provided limited theoretical analysis and was concerned with specific tasks e.g., predicting actions while this work focuses on a more abstract task of learning parameters for classification. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Technical Summary**\n\nThe teacher and learner are aware of the input and output space, the model class and the loss function. The teacher and student receive the same input up to a fixed deterministic mapping ($x$ vs $\\tilde{x}$) and have a different mapping of parameters ($\\nu$ vs $\\omega$). The teacher has access to its optimal weight function $\\omega^\\star$, a dataset to pick examples from at time step $t$, and assumes that the learner is performing gradient descent on parameters wrt the loss function. The teacher picks an example such that a single step of gradient descent would bring the teacher model closest to the optimal values.  This is re-written in terms of the student model and is approximated by assuming that the student tells the teacher of the inner product between *every* data point in the dataset and its current parameter. In return, the student model does two things. It first updates its parameters to fit the example $(x, y)$ chosen by the teacher. However, it also updates it parameters to predict the distribution used by the teacher to chose the dataset. These two models use the same parameters. A subtle difference is that the teacher's model uses a point-mass distribution while the student approximates it using soft energy-based distribution. \nA result is presented showing that the proposed approach under some assumption does not worse than a naive learner where the student just do gradient descent without any pragmatic reasoning (but the teacher still selects samples carefully). \n\n**Questions and Concerns**\n\n1. I have some difficulty understanding the setup. Firstly, is there a deterministic mapping between $\\omega$ and $\\nu$ similar to $x$ and $\\tilde{x}$. Further, it is assumed that $\\langle x, \\omega \\rangle $ = $\\langle G(x), \\nu \\rangle $. If yes, then how is this $\\nu$ related to $\\omega$. This assumption seems to be necessary for the theory to work. E.g., in going from equation 2 to 3. Lastly, how is $\\omega$ evolving with time? Is the teacher performing this update or the world? Is $\\omega$ affected by student's progress? \n\n2. It is said that the teacher know's that student is following equation 1 (line 177). But then it is stated that the teacher does not the learning mechanism (line 176). \n\n3. If the teacher assumes that the student is following equation 1 but then the student does pragmatic reasoning then it breaks down this assumption. Shouldn't an infinite recursion occur that should be analyzed? \n\n4. The teacher has access to the dot product of every data point with the student's current parameter (line 200-203). This sounds quite strong and would increase the communication complexity of learning significantly. Further, the teacher even knows the learning rate scheduling of the learner. In real life, the learner might only appear as a black box with limited APIs. \n\n6. In my understanding, theoretical guarantees only show that under certain assumptions the proposal cannot do worse than a naive learner and does not demonstrate the superiority of the proposed approach. This is helpful but it also shows that the impact of novelty is primarily demonstrated via experiments. I'll request authors to clarify this point if my understanding is incorrect.\n\n**Presentation**\n\n1. Equation 4 has integration over the dataset which seems incoherent. I believe the authors meant summation.\n\n2. Perhaps consider using teacher and student instead of teacher and learner for reasons of symmetry. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "See above for a list of concerns some of which remark on limitations. ", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies a teacher-aware learning process in the context of iterative machine teaching. In particular, a new learner model is proposed that incorporates the teacher's intention into the likelihood function to learn faster.  Extensive experiments are performed on various tasks with synthetic and real data, including a user study on learning from human teachers. The reviewers acknowledged the importance of the studied problem setting and generally appreciated the results. However, the reviewers raised several concerns with the current version of the paper, including:  (a) the heuristic nature of the teaching model is not analyzed; (b) the learner feedback as an inner product is not well motivated or explained.; (c) the theoretical guarantees only show that under certain assumptions the proposal cannot do worse than a naive learner; (d) the learning mechanism referred to in the paper is not well-defined; (e) some of the important details of the experimental set-up for the online IRL experiments and the study with human teachers are missing from the main paper. I want to thank the authors for their detailed responses that helped in answering some of the reviewers' questions. While the overall assessment is positive, the paper still stands as borderline. Nevertheless, this is interesting and potentially impactful work. The reviewers have provided detailed and constructive feedback to the authors. We strongly encourage the authors to incorporate the reviewers' feedback when preparing a revision of the paper.
Decision: Accept (Poster)