Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper studies the Teacher-Student Framework, in the case when the teacher is potentially sub-optimal. Learning here is based on an ensemble off-policy method. The core concept is to develop a teach intervention function that is based on the estimated sub-optimality of student actions with respect to the teacher's value function. The advantage of the proposed method is that it allows for continuous student improvement, to the point that the student takes complete control in the case of sub-optimal teacher. The authors provide theoretical bounds for the proposed method and evaluate it on a driving simulator. The proposed method 1) outperforms baselines across evaluation scenarios and 2) demonstrates the student capability to solve the task, even with a significantly sub-optimal teacher. 
strength_and_weaknesses: Strengths:

1. The proposed algorithm is intuitive and straight-forward
2. It asymptotically outperforms baselines
3. The proposed method is backed by theoretical analysis

Weaknesses

1. It seems that the algorithm takes some time before it starts improving and is initially substantially outperformed by EGPO. Is this because at the start of the training the switch allows most student actions, while EGPO mostly uses the teacher?

Question:

What is the effect of the optimization algorithm here? The proposed method uses Q-ensembles, which combined with more often training have demonstrated significant improvement in sample complexity (RedQ). What update frequency was used in this paper and is this directly comparable to the baseline methods? 

clarity,_quality,_novelty_and_reproducibility: The paper is well written and clear. The core concept of agent switches here is not new, but the particular implementation allows the student to learn and eventually outperform a sub-optimal teacher, unlike baseline methods. 
summary_of_the_review: Simple and intuitive method to allow student agent to learn and outperform a potentially sub-optimal teacher. Clear writing with good theoretical backing.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this work, the authors proposed a teacher-student framework (TSF), where a
teacher agent or human expert guards the training of a student agent by intervening and providing online demonstrations that the teacher policy may not be optimal. This is a more realistic setting than standard TSF in which the teacher policy is always assumed to be near-optimal. Specifically, they develop a new method that can incorporate arbitrary teacher policies with
modest performance and utilize an off-policy RL method known as Teacher-Student Shared Control (TS2C) to effectively update the student policy. In this method the main novelty is to incorporate teacher intervention based on trajectory-based value estimation. Theoretical analysis justifies also that the proposed algorithm attains efficient exploration and lower-bound safety guarantee without being affected by the teacher’s
own performance, and this algorithmic finding is also furthered validated with several benchmark experimentations (especially with autonomous driving simulation), showcasing that TS2C can exploit teacher policies at any performance level, maintain lower
training cost and the corresponding student policy can often outperform the imperfect teacher policy.
strength_and_weaknesses: Strengths:
The paper is well-written with concepts clearly explained. 
The problems studied in this work is important and the concerns of existing TSF work where the teacher is always optimal is a realistic one with good real-world implications. 
Experiment is thorough, validating the student policy can outperform the teacher in the TS2C algorithm. It is also backed with some theoretical analysis to justify this approach

Weaknesses:
Discussions of some experimental parameters are still missing. For example it is still unclear how the epsilon parameter is chosen. 
The theory part of this work is very straight-forward, the bound developed is pretty much a direct result from existing work.
especially lemma 1 and theorem 3.2 are pretty much standard results from occupancy distribution matching and policy-value differences in RL (e.g., TRPO-style proofs).
How tight is the bound in Theorem 3.3 (it seems with the (1-\gamma)^2 in the denominator the bound is rather loose), and how to effectively determine the average entropy in practice to validate the tightness of this bound?

clarity,_quality,_novelty_and_reproducibility: The paper is clearly written, with both algorithms and experimental setup quite clearly explained. In terms of contributions, the main novelty is TS2C, which is a flexible TSF framework that combines with off-policy RL student improvement and a value-aware methodology of deciding when a teacher policy should intervene the student.

TS2C can work with a wide range of teacher policies with different performance. Finally the authors evaluate the TS2C algorithm on a wide range of benchmark experiments and detailedly demonstrate the ability of TS2C in an autonomous driving scenario, which shows the algorithms' ability not only on imitation learning (of teacher) but also performance improvement guarantees. Contribution is slightly more on the empirical sense in which a new TSF framework is proposed to solve the problem of imperfect teacher. Algorithmically the novelty is moderately significant and most experimental hyperparameters and setup are explained in great details in the paper and its appendix to help readers better understand the experiment. However, the source code is not available reproducibility.
summary_of_the_review: The work is quite neat in terms of addressing the teacher-student framework in RL problem in which the teacher does not necessarily need to be near-optimal. The TS2C framework is also quite flexible and can be adapted to different off-policy RL algorithms. Experiments show the superiority of this algorithm in terms of imitating the teacher, deciding when the teacher should intervene the student during training and having the learned student. outperforming the teacher.

Novelty wise I think the underlying technique has merits in terms of problem formulation, algorithms and experimentation, albeit the theoretical analysis being a bit straight-forward, which is my main concern of the technicality of this work.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper considers the setting where one is provided a reference policy and we assume that we could be able to estimate or access the value function of the reference policy. The paper thus proposes the method called TS2C where they construct a mixture policy of the student policy and the reference policy and proposes a heuristic to decide which policy to use at run time. The experiment results show that the student policy indeed can overperform the reference policy in practice. 
strength_and_weaknesses: # Strength
* The paper is easy to follow, and the proposed algorithm is simple. 
* The experiment results indeed confirm that the student policy could eventually outperform the reference policy, validating the intuition of the algorithm.

# Weakness
* The paper should consider a more diverse set of benchmarks.
* The theory results in the paper are not very illuminating: for example, I am having a hard time understanding the significance of Thm. 3.3: $H$ should also depend on $\epsilon$ because $\epsilon$ defines $\pi_b$. Thus what exactly does the term $\sqrt{H-\epsilon}$ mean?
* The clearest issue of the paper is that the proposed algorithm seems like that the proposed algorithm is highly relevant to AggreVaTe [1] and LOLS [2]. The overall algorithm framework is very similar, and I would say the differences seem rather insignificant. I would assume that works such as [1,2] are overlooked in the literature search, so it would be great to compare and argue in the paper why the contribution of TS2C given the existence these previous works. 

[1] Ross, Stephane, and J. Andrew Bagnell. "Reinforcement and imitation learning via interactive no-regret learning."

[2] Chang, Kai-Wei, et al. "Learning to search better than your teacher." 
clarity,_quality,_novelty_and_reproducibility: * The clarity of quality of the paper is good.
* The novelty is greatly compromised given quite similar previous works that are not discussed in the paper.
* Reproducibility is good since the paper gives detailed experiment setups and the submission includes codes for the experiments. 
summary_of_the_review: The writing and the experimental results of the paper are good, but at the current stage, I am not convinced that the paper actually makes enough technical contribution given similar previous works. I would recommend a rejection.

======================
Increased the score after the rebuttal.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a novel algorithm to learn from imperfect teacher's policy and achieves super-teacher performance. It provides theoretical justification, performance bounds and experimental results with comparison against strong baselines.
strength_and_weaknesses: Strengths:
- The paper addresses a key problem of learning from an imperfect teacher policy
- It provides performance bounds and thorough empirical evidence

Weaknesses:
- The results are shown only on one environment
- Would be nice to have more details and experiments on how to choose epsilon and if it has to be manually chosen every time based on the environment?

Typos:
- There is a '5' at the beginning of the introduction section

clarity,_quality,_novelty_and_reproducibility: Well written paper, novel contribution and most details for reproducing the results are shared. 
summary_of_the_review: I recommend to accept the paper
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This work introduces a version of Teacher-Student online learning that allows for student actions to dominate when there is anticipated high value, even if they violate the behavior acceptable to the teacher.  The estimate of how valuable student actions may be comes from a learned Q function.  

The reviews for this paper were mixed, with two reviewers finding the proposed approach to be an intuitive solution to an important problem and to be theoretically well justified.  Reviewer oviQ, who gave a score of a 5, found the paper to be well written and tackle an important problem, but found the theoretical analysis to be too straightforward.  The authors replied that the theoretical analysis is merely an assurance and not a primary contribution.  I think this is a reasonable response.  Reviewer wVZY rated the paper a 3, and their review points out that the paper is clear, the algorithm is simple, and the experiments support the theory.  However, this reviewer believed that more diverse benchmarks were required and there is some relevant literature that is not compared that possibly is very similar.

Overall, in my opinion, there is some fundamental novelty to this approach, and the work is quite rigorously conducted.  However, there are a few high level points where I think things are slightly imprecise and I would encourage the authors to make a few additional changes before the camera-ready final version:
- The first sentence of the abstract asserts that the Teacher-Student Framework involves interventions that override student actions that are deemed unsafe.  As pointed out in the first paragraph of the main text, I'm not sure that interventions for safety is always assumed in Teacher-Student settings.  This is to say, there is a little more slack in how this framework is used and what it assumes than you imply in the abstract (e.g. a teacher inducing exploration is not "guarding" the student behavior).  I would encourage the authors to basically acknowledge that the version they compare against is one important teacher-student setting rather than the somehow primary or canonical one.
- Also, for example, when the authors state "This idea resembles imitation learning (IL) methods", they actually mean more precisely "online imitation learning" approaches.  As a related note, while I do think the present work is distinct from existing online imitation learning approaches, I share Reviewer wVZY's concern that the work does not appropriately discuss the connections to some of those previous papers.
- As a conceptual point, I'm not totally satisfied with there being a Q function from the experts that is always used to determine whether the student retains control versus the expert intervening.  As the Q function associated with the student learning develops, couldn't that be leveraged?  Basically, it just seems like there must be a large fraction of states for which the expert Q functions are non-informative (i.e. the ensemble has high variance) but where the student Q function has developed confidence.  In these regions, it seems like the expert will retain control, even though it perhaps should not. It would be worth discussing this design choice somewhere as it seems potentially a bit ad hoc.
justification_for_why_not_higher_score: I'm trying to gauge the potential impact based on reviewer feedback, and would be open-minded for the score to be higher (i.e., oral).
justification_for_why_not_lower_score: This paper is rigorous and innovative.  I think the community would be interested in this contribution.
Decision: Accept: notable-top-25%