Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: 
This paper proposes a framework that allows an AI agent to teach the human to do motor tasks, e.g., using a joystick for parking a car in simulation, or writing letters in an unfamiliar language. This framework's key advantage is that it will decompose the motor task into multiple skills, and compose those skills into individualized drills for the human student to practice.

The method works as follows:
1. Assuming that there are expert demonstrations of the given task. Then, the system first calls SKILLEXTRACTOR to extract skills from the demonstration. Each skill is represented as a sequence of actions and an interval where the skill is chosen to be executed.
2. Diverse scenario selection: choose a set of scenarios that can cover all skills via a greedy algorithm.
3. Identifying individual expertise in skills: ask the student to perform the task in those scenarios, and estimate her expertise.
4. Creating individualized drills: from step 3, identify the skills with low student expertise. Then, from the expert demonstration, identify N_drills n-grams involving the target skill with high frequency among all expert demonstrations.
Then, the student will practice each drill for N_rep times.

Results:
* Simulated student
  * Conclusion: Individualized drills (the proposed method) improve the total reward (statistically significant).
    * Compare among 3 methods
      * Ind.Drills: Individualized drills (the proposed method).
      * Drills: random drills.
      * FullTrajectory: the entire trajectory of expert demonstration.
    * Case 1: Simulated student as a half-trained behavior cloning agent
      * Ind.Drills > FullTrajectory's total reward (statistically significant).
      * Drills > FullTrajectory's total reward (statistically significant).
    * Case 2: Simulated student as a behavior cloning agent with reverse action difficulty
      * Ind.Drills > FullTrajectory's total reward (statistically significant).
      * Ind.Drills > Drills's total reward (statistically significant).

* Human student
  1. Conclusion: skill by SKILLEXTRACTOR improves the reward improvement (statistically significant).
    * Compare among 3 methods
      * Skills: SKILLEXTRACTOR (used by the proposed method).
      * TimeHeuristic: split the trajectory into skills evenly.
      * FullTrajectory: entire expert demonstration as a skill to practice.
    * Task 1: Parking
      * Skills > FullTrajectory's reward improvement (statistically significant).
      * Skills > TimeHeuristic's reward improvement (statistically significant).
    * Task 2: Writing
      * Skills > FullTrajectory's reward improvement (statistically significant).
  2. Conclusion: practicing with drills vs skills is not statistically significant.
    * Task 1: Parking
    * Task 2: Writing
  3. Conclusion: Individualized drills (the proposed method) improve the reward improvement (only marginal significance), and improves the subjective helpfulness (statistically significant)
      * Ind.Drills: Individualized drills (the proposed method).
      * Drills: random drills's reward improvement.
      * Skills: practice with skills's reward improvement.
    * Task 1: Parking:
      * Ind.Drills > Skills's reward improvement (only marginal significance)
    * Task 2: Writing
      * Ind.Drills > Skills's reward improvement (only marginal significance)

    * Task 1: Parking
      * Ind.Drills > Skills's helpfulness (statistically significant).
    * Task 2: Writing
      * Ind.Drills > Drills's helpfulness (statistically significant).
      * Ind.Drills > Skills's helpfulness (statistically significant).
strengths_and_weaknesses: 
* Strength
  * Very interesting and novel problem formulation.
  * Online simulation-based user study in 2 tasks.

* Weakness
  * The proposed method assumes that the skill extracted by SKILLEXTRACTOR matches the skills that the student would need. However, in reality, each student could have a different set of skills extracted.
  * The writing could be easier to follow by reducing the notations, especially those notations with many super / sub-scripts.
  * User study with real human successfully shows that skill extracted by SKILLEXTRACTOR (used in the proposed method) improves the reward improvement with statistical significance. However, the user study failed to show that the proposed individualized drills improve the reward improvement (only statistically marginal significance).
  * Minor: Line 145: the paragraph about the POMDP approach could be shortened, because the paper later says that POMDP is intractable and proposes another approach. So the POMDP approach is not very relevant to the paper.
  * Minor: Fig.4: label "No Skills" should be "Full Trajectory"?

questions: 
* Line 122: latent space M. Is the concept of latent space used in this paper? Latent space reminds me of VAE or graphical models. I was confused when seeing this in line 122.
* Line 314: does full trajectory mean that the student will just practice on the original expert demonstration?

limitations: N.A. All technical limitations are in `# Strengths And Weaknesses`.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents an approach to providing training instruction for motor control tasks automatically by extracting skills from expert trajectories and combining them into drills based on user expertise. The authors use an existing framework for skill extraction and their main formal contribution is an approach for user evaluation and for individualized training curriculum construction. The paper evaluates different training scenarios with synthetic and real human users to analyze and demonstrate the benefits of their training curricula.
strengths_and_weaknesses: The application of AI to individualized training of human motor skills is an interesting application with many potential uses. The paper is well written and clearly presents the concepts and algorithms used. The experiments are intuitively presented and show important results for understanding of the effects on human learning.
The main weaknesses of the paper some potentially limiting assumptions that should be discussed more. For example, the target motor control task is defined as an MDP with deterministic transitions. Some discussion why deterministic transitions are sufficient here or what limitations this introduces would be useful. 
Another issue is that in equation 5 the used indicator function II is not formally introduced.
In the experimental results, it would have been nice to also see some analysis of the effect of the amount of training on performance. In particular in the synthetic agent experiments, partially trained agents don’t seem to achieve high performance even after training. Would the proposed training approach actually get them to high performance or do they “plateau” ? In the human experiments, how close to the expert do users get ? This is hard to assess as no optimal performance reward estimate is provided.
questions: 
In the formalism, one question that remains is related to the limitations introduced by the underlying deterministic model.
Why is a deterministic skill model sufficing ?

In the experiments, a few issues would be important:
Does the training curriculum yield optimal behavior eventually or does it lead to an intermediate “plateau” where presented skills and drills do not address areas where the user could further improve ?
How realistic was the underlying synthetic agent model to capture human skill acquisition ?


limitations: While the paper discusses some limitations and social considerations, some of the underlying assumptions and resulting limitations are not fully discussed. Among these are the deterministic skill model and the used synthetic agents for the first set of experiments.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper presented an AI-assisted teaching algorithm that leverages skill discovery methods from reinforcement learning. The key idea of the proposed method is to decompose the skill to be taught into a few teachable sub-skills. By combining sub-skills into drill sequences, the algorithm then generates individualized curricula for different students.
strengths_and_weaknesses: Strengths:

1. The motivation of the paper is valid. I think the decomposition of a complex task into teachable skills is a simple and efficient strategy.

2. The paper is well-written.

3. The experiments are extensive. Both the PARKING and WRITING experiments are interesting. 

4. Extending the skill decomposition in RL literature to human learning seems promising.

5. The code is provided for reproducibility. 

Weaknesses:

1. My major concern is the discrepancy between the proposed approach and the target task (motor control). Particularly, I think the proposed approach does not capture the key challenge of teaching motor control skills. It seems that the success of the proposed method solely depends on human learning but ignores that humans may not be able to learn motor control skills without proper assistance. Let me explain. One of the characteristics of motor control skills is that it normally involves physical contact with the environment. In my opinion, to distinguish a teaching algorithm for a motor control task from the conceptual counterpart (e.g. learn classification), it is important to embed the notion of such physical interactions. For example, designing a practice sequence may not suffice in many rehabilitation tasks.  Without this, I don’t see a significant advancement over the existing Automatic Curriculum Learning (ACL) approaches made by the paper. 

2. I think the paper lacks an analysis of why the learned teachable skills from reinforcement learning literature can be directly applied to human learning, which makes the result less convincing. 

3. The paper investigates two tasks, PARKING, and WRITING. However, for both these tasks, the decomposition into teachable sub-skills can both be done by an expert. The approach will be more valuable if the author could show how the learned teachable skills compare with expert-defined teachable skills.  

4. In this paper, the objective is to align students’ trajectory with expert demonstrations. From my perspective, this is not a good way to specify the learning objective, especially for complicated tasks with multiple optimal solutions. 

questions: Questions:

1. I am not familiar with Balinese. However, I would like to ask: does the learned sub-skill have any semantic meaning? For example, in many writing systems, there are prescribed strokes or components in a character. Will CompILE be able to learn meaningful sub-skills in that case?

2. In the abstract, the paper mentions ‘individualize curricula to students with different capabilities’. What do the capabilities mean? Is it the current skill level of the student? In that case, is the training drill only generated based on the student’s initial performance? Is this caused by the lack of a human learning model?

3. In the supplementary materials, in figure 7, the performance improvement on writing before and after training is very subtle but the improvement over reward is significant, which is quite confusing to me. 

Overall, I still think the paper is interesting and well-written. Therefore, I lean towards acceptance for now. 
limitations: NA
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper introduced an assistive teaching system for improving human skills on continuous control tasks. The core idea of the paper is to decompose a complex task into multiple reusable skills, assess the human skill level for each decomposed skill, and compose new personalized drills that help improve the weakest skills of the user. The system is evaluated on two tasks: parking a car with joystick control, and writing characters from the Balinese alphabet. The authors tested the proposed method on both synthetic learners as well as real human subjects in a user study. The experimental results shows that the method achieve improved learning performance in both cases for both tasks.
strengths_and_weaknesses: Strengths:
1. The idea of leveraging skill discovery for teaching assistance is interesting and novel in my knowledge. 
2. The proposed method that combines skill discovery, personalized evaluation and drill design, is reasonable and well elaborated.
3. The experiments support the proposed method in improving human learning performance.

Weaknesses:
1. One assumption that the method is making is that the decomposed skills can be acquired through demonstration, which circumvents the challenges of modeling human learning dynamics as described in the ‘What should the teacher do’ section instead of solving it directly. As such the method may not work for more complex tasks such as performing stunts where individual skills can also be difficult to acquire.
2. The demonstrated tasks do not require complex and reactive decision making, compared to something like a real-time strategy game. It’s not clear for more complex problems like those whether the skill discovery technique can still find reasonable low-level skills and whether the pipeline can scale accordingly.
questions: 1. During the evaluation phase, if a user finds a different sequence of operations from the expertise but still accomplishes the task (for example in the parking task there might be multiple, equally good solutions), would the proposed system penalize that and try to make the user follow the expert trajectory?

2. What do the learned skills look like for both tasks? It would be helpful to have some form of visualization for both expert trajectories and student trajectories to better understand the discovered skills and the evaluation process. In addition, for the writing task, can we use each letter as a skill?

3. In the synthetic student case, how many epochs were used during fine-tuning? Why are half-trained ones so much worse than the reverse difficulty ones even after fine-tuning?

4. In line 342, the Skills approach picks the 3 most common skills, while for writing 8 skills were discovered. Does this mean 5 of them are not seen by the user in this version of the method?


limitations: The authors have addressed some limitations of the work. One additional discussion point that would be helpful is some additional example problems where the proposed method would succeed and fail, which would not only provide insights on when the method is applicable, but also help better judge the potential impact of the work.
review_title: null

Meta-review: metareview: The reviewers appreciated the extensive replies and the paper updates. Those managed to clear up most of the concerns of the reviewers that now all rate the paper cautiously positive. On the downside, now a lot of important material and discussion has been included in the appendix, while ideally the paper should be self-contained (i.e., no requiring the reader to also read the appendix).
The paper proposes a very interesting approach with a solid contribution and solid experiments. It is an important first step, while - like the authors point out - there are still quite a few limitations before this can be used practically, that will require separate papers to address.
confidence: Certain
Decision: Accept