Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: Overview:

The paper tackles the representation learning problem where the aim is to learn a generic representation that is useful for a variety of downstream tasks. A two-level optimization framework is proposed: an inner optimization over the specific problem-at-hand, and an outer optimization over other similar problems. The problem is studied in two settings of the imitation learning framework with the additional aim of providing mathematical guarantees in terms of sample efficiency on new tasks. An extensive theoretical analysis is performed, and some preliminary empirical results are presented. 

Decision:

In its current form, the paper should be rejected because (1) the empirical analysis is incomplete – the baseline isn't very appropriate, the results are not conclusive, details are scattered or not included, (2) the literature survey does not connect the proposed approach with existing approaches, and does not convince the reader why all the existing approaches have not been compared against empirically, (3) the paper is generally unpolished and needs more work before being considered for acceptance.

Details:

The paper makes both theoretical and empirical claims. I did not have the time to thoroughly verify the theoretical claims and took them at face value. I consider the theoretical guarantees associated with the proposed approach a welcome and valuable contribution to this field that has recently been relying primarily on limited empirical work to assess any method. 

The empirical results presented in the paper do not sufficiently support the claims of sample efficiency. One of the main issues with the empirical analysis is the choice of the baseline, which learns a policy from scratch. This does not help make conclusions about the sample efficiency of the proposed method on new tasks. A better baseline would be one that learns some representation from the T previous tasks, which would help infer if the proposed method to learn representations is actually more sample efficient on new tasks or not. There is also no comparison with existing approaches that are mentioned in the Related Work section. If those aren’t appropriate baselines for this problem, a small explanation of the reasons why would help readers understand why they haven’t been compared against. Additionally, an analysis of statistical significance of the results is missing and would significantly help in gauging the efficacy of the proposed approach. 

The paper notes that these are some preliminary experiments. The completion of the empirical analysis would definitely make a stronger case for this paper to be accepted.

Minor comments to improve the paper:

- Error bars in the plot, specification of number of runs, and other such experimental details would be very helpful in interpreting the results. 
- It would help a reader if the paper was more self-contained, e.g., if terms like supp(\eta), \bar{s}, \tilde{s} are defined more clearly.
- It would also help to say what the proofs intuitively mean, e.g., for a new task drawn from this particular distribution of tasks, the agent would achieve close-to-X performance within Y samples – something along those lines.
- There are some typos, e.g., 'possibility'->'possibly' on page 1, missing $H$ in specification of MDPs on page 2, 'exiting'->'exciting' on page 8, some latex symbols in Appendix D, etc.
- The bibliography has a lot of issues – some references are incorrectly parsed (e.g., Yan Duan, Marcin Andrychowicz, Bradly Stadie, Jonathan Ho, Jonas Schneider, Ilya Sutskever, Pieter Abbeel, and Wojciech Zaremba. One-shot imitation learning. 03 2017), others are inconsistent (e.g., "In NIPS" and "In Advances in Neural…”; the arXiv ones).

   
review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: This paper theoretically explores reinforcement/imitation learning via representation learning. The key theoretical question being investigated is the relationship between representation learning in a multi-task/meta learning setup and its dependence to the sample/task complexity. The paper sets up the problem in bilevel optimization framework, where the inner optimization learns/optimizes task specific losses, while the outer optimization learns the representation used in the inner level tasks. The main takeaway from the two theorems (which are the core contributions of this paper) are that when the number of tasks is higher than the number of samples, representation learning can reduce the sample complexity. The paper explores two scenarios in imitation learning, namely behavioral cloning and when only the states of the experts are available (and not their actions). Some experiments are provided to empirically validate the theory.

Pros: 
1. The paper presents a theoretical investigation into multi-task/meta learning for RL via learning representations. While, the main theoretical contributions are perhaps marginal with regard to prior work, the problem setting (RL) seems novel and the two theorems in this context are interesting.
2. The paper is well-written and appears to be very rigorous. I did not check for the correctness of all technical parts. There are several "abuse of notations" in the main text, which sometimes impact the otherwise smooth read of the technical parts. 
3. A good and concise review of RL concepts is provided.

Cons:
1. The paper lacks a good literature review to place this work in the right context. For example, while the paper refers to the works of Maurer 2016 and Sun et al. 2019 at several places, it is not formally and clearly mentioned anywhere what are the similarities to these prior works and what are the new contributions. For example, Maurer 2016 proposes a multi-task learning setup using representation learning, and most of Theorem 4.1 in this paper is taken from the results in that paper. While, the current paper uses bilevel optimization setting in an RL context, it is not clear to me if this (bilevel + RL) setting has any significant bearing against the theoretical results furnished by Maurer 2016. For example, the theorems in this paper (as far as I see) show that the bounds are scaled by a constant defined by H^2, the trajectory length? If there is something beyond this, then the paper needs to explicitly point it out. The same comment goes with the results against Sun et al. 2019 in Theorem 5.1.

2. Given that the main goal of this paper is to connect sample complexity with representation learning, it is important the paper provide a theorem stating this precisely. Theorems 4.1 and 5.1 provide a general bound, and the sample complexity is being described in the explanations of this theorem, which is very informal. Also, against what is claimed in the abstract, it appears that representation learning helps reduce sample complexity only when the number of tasks are larger, which perhaps needs to be explicitly mentioned. Also, note that there is a bearing of the bound on the trajectory length H (Theorem 4.1, and 5.1). Shouldn't this factor be also accounted for when explaining the sample complexity? 

3.  There has been several recent works on model-agnostic meta-learning (that also uses bilevel optimization and implicit gradients), however, older works on meta-learning (only for imitation learning) have been cited. The paper should include more recent works in this area and contrast against their theoretical findings. 

Apart from these, below are some minor comments that could help improve the reading of this paper:
a. Theorem 3.1 is not really a theorem, since it is very informal. Also, fix the Theorem numbers. 

b. Bullet 1. after Theorem 3.1, \ell^x(\pi) concentrates to \ell^x(\pu*). Also, the mention about sample complexity here should be backed with some reference/citation.

c. Assumption 4.2: The notation \pi_\mu(s)_{\pi*(\mu(s)) is unclear, shouldn't the subscript contain an argmax over the actions for \pi*? 

d. Theorem 4.1 and 5.1, what does it mean by "probability 1-\delta over the choice of the dataset X" ? Also, \mu^n seems undefined.

e. The first two terms in Theorem 4.1 are claimed standard, provide the citations?

f. Theorem 4.1, perhaps use some other notation for c, which is defined as the cost/reward in the RL setting.

Overall, the paper has some interesting theoretical results, and is mathematically rigorous, however lacks a clear distinction from prior and more recent works in this area. 

review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: (I bid on the paper thinking that bi-level optimisation would play a major role in the paper. Unfortunately, it does not, so my expertise in bi-level optimisation is not much use, I am afraid.)

The authors study applying policies learned on one task to another task, while considering practical finite-sample limitations. They call this the "representatition learning for imitation learning". Unfortunately:

The make extensive assumptions, summarised on page 3, but not formalised as Assumptions 1, 2, 3 anywhere, as far as I can tell. They assume: 
-- concentration of the loss, "which guarantees within-task sample efficiency" (but does not seem easy to support in practice? actually, one may observe samples from a stochastic process, rather than iid samples with any concentration what-so-ever?), 
-- that the "loss" they use with is somehow close to the optimum of the expected value function J (for which I again see no justification, empirical or otherwise).

Then they reuse results of Maurer et al (2016) and extend them to a case where the actions are not observable. The results are plausible, given the assumptions. Given the assumptions, they also do not seem to be particularly relevant to the practice of RL?

The empirical results involve only benchmarks of the authors own coinage, and hence are hard to evaluate. It seems plausible, again, however, that the approach may work in some cases. 


review_title: Official Blind Review #3

Meta-review: This paper proposes a methodology for learning a representation given multiple demonstrations, by optimizing the representation as well as the learned policy parameters. The paper includes some theoretical results showing that this is a sensible thing to do, and an empirical evaluation.

Post-discussion, the reviewers (and me!) agreed that this is an interesting approach that has a lot of promise. But there was still concern about he empirical evaluation and the writing. Hence I am recommending rejection.
Decision: Reject