Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The manuscript presents iterative teacher aware learning, an approach in which learners learn from teaching while incorporating the intent of the teacher for continuously parameterized models. 
main_review: The paper presents iterative teacher aware learning (ITAL) which allows teaching in continuous spaces, and learners who take advantage of the teaching intent to augment learning. The approach builds on prior work in machine learning and cognitive science, and is distinctive in incorporating two features: learners who learn from teaching *and* the teacher's intent, and the ability to learn in continuous parameter spaces. Prior research has done each of these, but not together, which represents the core novel contribution of the paper. Update rules are derived that extend gradient descent methods to incorporate the teacher's selection process. Theoretical results prove that teacher aware learners learn no slower than non teacher aware. Empirical results confirm theoretical analysis for an array of problems including learning from human teaching. 

Positives: 
- The extension to learning from cooperative teaching in continuous parameter spaces is interesting. 
- The theoretical results are clear and relevant. 
- The experiments show ITAL learns faster than iterative machine teaching (IMT). 

Weaknesses:
-  The heuristic nature of the teacher model was not analyzed or scrutinized, which limited possible insights provided by the paper. 
- The learner feedback as an inner product is a curious choice that was not well motivated or explained. 
- The connection to human learning is limited and over stated in some places. 

Detailed comments: 
- "diametrically distinctive" Please check the word usage here. 
- "the learning mechanism" What is the learning mechanism? 
- " Yet, their findings cannot be directly applied to more practical teaching scenarios" Why? Please explain. 
- "Thus, we keep leveraging the greedy heuristic to model our cooperative teacher and generalize it to a non-omniscient teacher who doesn’t fully know the learner in every iteration" I do not understand the logic here. 
- "This can be done approximately if, in every round, the learner gives the inner products of ν t−1 and the data to the teacher as feedback." What does this mean intuitively? 
- " It has been shown that cooperative teachers using Eq. (5) can substantially " Do you mean equation 3? 
- " to better approximate ν∗, in practice, we plug" What is the justification for this? 
- Figure 1 is too small to read. 
- "Pedagogy has a profound cognitive science background, but it receives limited attention in recent machine learning works" This isn't quite accurate. Earlier in the paper a number of machine learning papers were cited.   

~~~~~~
Based on the reviewing discussions, I have decremented my rating to a 6. 
limitations_and_societal_impact: No discussion of limitations or societal impact were included.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: **High-Level Summary**

This paper studies cooperative teacher and learner settings. In this setting, the teacher and learner are trying to solve a classification task with input $x \in \mathcal{X}$ and label $y \in \mathcal{Y}$. The setup works in an online fashion where at each time step, the teacher selects an example $(x, y)$ to present to the student from a large dataset and the student learns from this example. The teacher knows the optimal parameters. Previous work has focused on the setting where the teacher is aware of the learner and tries to select the example to ensure the student learns the fastest. This paper studies an extension where the learner is also aware of the teacher's feedback (teacher-aware learner). The key novelty is in the learner trying to maximize its parameters to fit the feedback provided by the teacher while also trying to predict the distribution the teacher is using to sample the data from. This has an additional benefit in that it can help the learner perform "pragmatic reasoning". This is related to previous work on Rational-Speech Act models (e.g., Golland et al., 2010). The key difference is that previous work on RSA provided limited theoretical analysis and was concerned with specific tasks e.g., predicting actions while this work focuses on a more abstract task of learning parameters for classification. 
main_review: **Technical Summary**

The teacher and learner are aware of the input and output space, the model class and the loss function. The teacher and student receive the same input up to a fixed deterministic mapping ($x$ vs $\tilde{x}$) and have a different mapping of parameters ($\nu$ vs $\omega$). The teacher has access to its optimal weight function $\omega^\star$, a dataset to pick examples from at time step $t$, and assumes that the learner is performing gradient descent on parameters wrt the loss function. The teacher picks an example such that a single step of gradient descent would bring the teacher model closest to the optimal values.  This is re-written in terms of the student model and is approximated by assuming that the student tells the teacher of the inner product between *every* data point in the dataset and its current parameter. In return, the student model does two things. It first updates its parameters to fit the example $(x, y)$ chosen by the teacher. However, it also updates it parameters to predict the distribution used by the teacher to chose the dataset. These two models use the same parameters. A subtle difference is that the teacher's model uses a point-mass distribution while the student approximates it using soft energy-based distribution. 
A result is presented showing that the proposed approach under some assumption does not worse than a naive learner where the student just do gradient descent without any pragmatic reasoning (but the teacher still selects samples carefully). 

**Questions and Concerns**

1. I have some difficulty understanding the setup. Firstly, is there a deterministic mapping between $\omega$ and $\nu$ similar to $x$ and $\tilde{x}$. Further, it is assumed that $\langle x, \omega \rangle $ = $\langle G(x), \nu \rangle $. If yes, then how is this $\nu$ related to $\omega$. This assumption seems to be necessary for the theory to work. E.g., in going from equation 2 to 3. Lastly, how is $\omega$ evolving with time? Is the teacher performing this update or the world? Is $\omega$ affected by student's progress? 

2. It is said that the teacher know's that student is following equation 1 (line 177). But then it is stated that the teacher does not the learning mechanism (line 176). 

3. If the teacher assumes that the student is following equation 1 but then the student does pragmatic reasoning then it breaks down this assumption. Shouldn't an infinite recursion occur that should be analyzed? 

4. The teacher has access to the dot product of every data point with the student's current parameter (line 200-203). This sounds quite strong and would increase the communication complexity of learning significantly. Further, the teacher even knows the learning rate scheduling of the learner. In real life, the learner might only appear as a black box with limited APIs. 

6. In my understanding, theoretical guarantees only show that under certain assumptions the proposal cannot do worse than a naive learner and does not demonstrate the superiority of the proposed approach. This is helpful but it also shows that the impact of novelty is primarily demonstrated via experiments. I'll request authors to clarify this point if my understanding is incorrect.

**Presentation**

1. Equation 4 has integration over the dataset which seems incoherent. I believe the authors meant summation.

2. Perhaps consider using teacher and student instead of teacher and learner for reasons of symmetry. 
limitations_and_societal_impact: See above for a list of concerns some of which remark on limitations.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper presents a gradient optimization-based teacher-aware learner who can incorporate a teacher's cooperative intention into its likelihood function, and learn faster in comparison to the existing naive learning algorithms for machine teaching. The authors evaluate the proposed algorithm empirically on several machine learning tasks such as classification/regression/IRL and also on a simple setting where instruction/advice is taken from human teachers. Their results show that learning is indeed faster with a teacher-aware learner.
main_review: **Pros**

* The modeling of a teacher-aware learner is interesting.
* The experimental evaluation catered to different types of learning models and showed in all the diverse settings (classification/regression/IRL) that a teacher-aware learner has accelerated learning.


**Cons**

* Writing: The paper could have been made clearer, particularly equation (6) in Section 4.2. Since this is a crucial equation illustrating the gradient update rule for such a learner, it could have been better annotated, and upon annotation, refer to its different parts in the subsequent text that follows. There is a typo on line 202 where I think Eq.5 was mentioned, by it should have been Eq.3. 

* Evaluation: While the evaluation was exhaustive, one could elaborate further on the Online IRL setting in the main paper, and put the details of the Linear classifier on natural image datasets in the appendix instead. The Online IRL setting is a more interesting set-up for the problem. Furthermore, the experiments with the human teacher were overly simplified for this setting. The method has the potential for dealing with a more complex instruction set of a human teacher, given that the learner model is general enough and operates under the realistic assumption of a sub-optimal teacher.



**General Comments**
The paper presents important theoretical guarantees for the proposed teacher-aware learner under a set of practical assumptions. The learner model has the potential of conducting counterfactual reasoning by comparing the teacher's selected instruction with that of others, in the setting that the teacher's instruction set is finite. The experimental evaluation also looks promising, particularly the verification with human teachers. It would be interesting to evaluate the performance of such a learner in a real-world setting and explore empirically its robustness to noise in teacher instruction.

*Originality*: Novel Approach

*Clarity*: Largely clear.

*Quality*: Good

*Significance*: High
limitations_and_societal_impact: The paper is still in its nascent stages and far from deployment in the real world. Hence, negative societal impacts are not a concern just yet.
review_title: null

Meta-review: The paper studies a teacher-aware learning process in the context of iterative machine teaching. In particular, a new learner model is proposed that incorporates the teacher's intention into the likelihood function to learn faster.  Extensive experiments are performed on various tasks with synthetic and real data, including a user study on learning from human teachers. The reviewers acknowledged the importance of the studied problem setting and generally appreciated the results. However, the reviewers raised several concerns with the current version of the paper, including:  (a) the heuristic nature of the teaching model is not analyzed; (b) the learner feedback as an inner product is not well motivated or explained.; (c) the theoretical guarantees only show that under certain assumptions the proposal cannot do worse than a naive learner; (d) the learning mechanism referred to in the paper is not well-defined; (e) some of the important details of the experimental set-up for the online IRL experiments and the study with human teachers are missing from the main paper. I want to thank the authors for their detailed responses that helped in answering some of the reviewers' questions. While the overall assessment is positive, the paper still stands as borderline. Nevertheless, this is interesting and potentially impactful work. The reviewers have provided detailed and constructive feedback to the authors. We strongly encourage the authors to incorporate the reviewers' feedback when preparing a revision of the paper.
Decision: Accept (Poster)