Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a learning framework for predicting the labels of dynamic systems. Unlike existing model-based approaches and model-free approaches, the proposed model takes a middle ground and uses a knowledge distillation-based framework. It uses a teacher model to learn to interpret a trajectory of the dynamic system, and distills target activations for a student model to learn to predict the system label based only on the current observation.

Experimental results on both synthetic and simulated datasets confirm the effectiveness of the proposed framework.  

Pros:
1. The paper studies an important problem. Predicting the behavior of a dynamic system has many applications. 

2. The proposed model is interesting and may lead to a series of follow-up studies that leverage the strengths of both model-free and model-based methods using knowledge distillation techniques. 

Cons:
1. The baseline models are quite simple. There are stronger baselines as noted by the authors. While the proposal is a learning framework, it might still be worth customizing and comparing it with state-of-the-art models in specific problems. 

2. The presentation of the paper can be improved. It would be good to add a running example to explain the various concepts and definitions used in the paper. 

Additional comments:
Typo: "a teacher networks" => "a teacher network"; "using only using" => "using only"

**Update after author response:** I appreciate the authors' efforts to address my comments. The new version reads better. However, I am still not entirely convinced by the choice of the simple baselines. Since a positive rating is already given, I would keep it unchanged. 
review_title: A novel approach to predict labels of dynamical systems

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper presents a student-teacher framework, where the teacher network can be used to select and prioritize the relevant properties of the given dynamical system that should be learned by the student.

Pros:
- (significance) I think the presented framework is a powerful one, and has a potential to be applied broadly to many real-world problems. 
- (quality) The development of the method is sound and well-motivated. The method was tested on a toy example and then applied to tasks with varying degrees of challenges.

Cons: (mostly on clarity)
- What confuses me the most is the way the authors frames their work. From the current title "Learning to interpret trajectories", and the abstract, it was not really clear to me what the paper is about; I am not sure if the population of people who stop at the title would be the same as the population that finds the contents most interesting. Specifically:
    - Why is "trajectory" a central keyword for this work? The word trajectory can be used in many different contexts, I don't think it was made clear anywhere in the paper what is the defining features of the "trajectory-ness" that the authors want to emphasize.
    - What do you mean by "interpret"? This word is being used in a very loose fashion without a clear context; it is empty at best, and misleading at worst.
- I get that the proposed framework avoids the *problems* of model-based and model-free methods, but I am having difficulties identifying what *advantages* of the two methods that the framework is incorporating.
- One of the central questions that is raised in the introduction is this: "What is the right way to incorporate information from different trajectories?". But I am not sure how this work solves the problem of incorporating *different* trajectories specifically.

Additional comment:
- The core concepts from the previous works that the work is based on, such as *learning using privileged information* or the meta-gradient approach, are not clearly introduced. Even a brief, ~1 sentence description would be helpful.
- The ideas of model-based and model-free methods are reinforcement learning concepts, and may not be clear to people who are not in RL. Again some brief description would help.

Overall, I have a mixed feeling about this paper and I currently stand between scores 5 and 6. Whereas I find the proposed method interesting, I feel that the lack of clarity, and the confounding of messages in the framing, make this paper rather short of the standard for the conference. 

**UPDATE:** 
My major concerns were addressed in the revised version of the paper.

review_title: Interesting work, but the framing is confusing

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:
This work tries to find a compromise of model-based and model-free methods, using a teacher and student network . The teacher network is trained with meta-gradients. It interprets the trajectories and provides activations for a student network that is supervised for a given task using the current state. 

Strengths:
+ The paper is well motivated and solves interesting problem.
+ The related work is thoroughly reviewed. 

Weaknesses:
- Some claims made by authors are not validated. I suggest to add relevant citations in Sec.1. These claims support the motivation of this work but are not acceptable without proper references. For example, where is the evidence of deterioration on the tasks that are potentially relevant? Which task do model-free methods achieve substantially better performance?
- The evaluation is conducted using the self-generated baselines. Why don't you use existing methods shown in Sec.2 and compare? I cannot find the result of the model-based baseline (as a counterpart of MF).
- Typos: using only using model-free methods
review_title: well motivated 

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a teacher-student training scheme to incorporate the useful information of trajectory to improve the predictive performance of model-free methods. The teacher network tries to "guide" the student network at the training stage by presenting an interpretation of the trajectory. The guidance is implemented by adding to the loss function a regularization term that penalizes the "distance" between the teacher's output and the hidden states of the student. The proposed method was tested and compared to other model-free methods.

The study in this work is interesting and important in RL. It tries to tackle the weakness of model-free methods by introducing the dynamics while avoiding to build a full model like the model-based methods. Searching for an optimal tradeoff between the two would benefit the practical uses.

I'd vote for accepting the manuscript if the authors could address my concerns.

- As mentioned in the text, the student internal h and the supervision signal h* are not required to have the same dimensionality. Are they required to have the same number of layers or certain correspondence between layers? Why or Why not? Do the authors have general principles on the design of the teacher and the teaching loss?

- Is there any particular reason for using classification loss in all examples especially for a typically regression problem? Did regression loss lead to bad performance?

- In the toy XOR example, the particular design of x* seems to play the key role. It is not the typical trajectory of x. This hand-crafted data does not help with demonstrating the teacher learning the powerful interpretation itself.

- Though the authors have talked about why not to compare to a model-based method, I do not think it is convincing. As mentioned in the intro section, the model-based methods fail due to partial observability and etc, but I do not see the examples in this study have such issues. The computation of model-based methods depends on the complexity of internal model rather than the task. The argument should then be if the proposed method outperform model-based methods given the same complexity (e.g. the number of parameters) or same amount of data.


review_title: Recommendation to Accept

Meta-review: The paper proposes a new teacher-student framework where the teacher network guides the student network in learning useful information from trajectories of a dynamical system. The proposed framework is inspired by the Knowledge Distillation method. The teacher learns what information should be used from the trajectories and distills this information for the student in the form of target activations. In a nutshell, the framework allows the student to interpolate between model-based and model-free approaches in an automated fashion. Experimental evaluation on both the hand-crafted and simulated tasks demonstrate the effectiveness of the proposed framework. The reviewers had borderline scores in their initial reviews and raised several questions for the authors. The reviewers appreciated the rebuttal, which helped in answering their key questions -- I want to thank the authors for engaging with the reviewers during the discussion phase. The reviewers have an overall positive assessment of the paper, and believe that the proposed teacher-student framework is novel and potentially useful for many real-world problems. The reviewers have provided detailed feedback in their reviews, and I would like to strongly encourage the authors to incorporate this feedback when preparing the final version of the paper.
Decision: Accept (Poster)