Venue: NeurIPS 2022 Workshop Offline_RL

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The authors consider representation learning for RL in a multi-task context, where offline pretraining of a representation is done on some number of source tasks and the frozen representation is used for online policy learning in a target task. They assume generative (i.e. simulator) access to the source tasks, meaning the dynamics can be queried at any state-action pair. They introduce a novel notion of task relatedness they call the linear span assumption, and present a representation learning algorithm that performs provably well under this assumption.

Pros:
- The core contributions of the linear span assumption and the representation learning algorithm build well on prior work while providing significant novelty
- The theoretical results of the representation learning algorithm under the linear span assumption are strong
Cons:
- The generative source domain assumption is restrictive and limits practical impact
- Empirical evaluation is missing

Overall, the paper is clearly written and well-presented. The results are interesting enough to warrant further research.
review_title: Interesting theoretical results for representation learning in RL

final_score: rating: 7: Good paper, accept
review_content: The paper studies the RL transfer learning setting where an agent is pre-trained on a set of source tasks and then deployed on a target task. The authors analyze a new assumption on the relatedness of the source and target tasks. The assumption is that all the tasks are low rank MDPs and the source and target tasks share the same representation. More specifically, the target task dynamics are in the linear span of the source task dynamics. With this assumption and the assumption of generative access to the source tasks, the authors propose a transfer learning algorithm and analyze its regret. The regret bound matches that of a linear MDP with a known representation, but with an additional scaling factor that depends on how different the states seen in the target task are from those seen in the source tasks. 

The paper is well written and all points are explained, though it would have been helpful to have more detail on a few key parts (though understandable given the space constraints). I wasn’t sure how much the relatedness assumption differs from assumptions made in prior work. In the algorithm, it would be helpful to give more intuition for how the cross sampling procedure leads to the result.

Pros:
- The contribution is original, the authors study transfer learning under a new assumption on task relatedness. 
- The new theoretical results could inform practice in transfer learning.  

Cons:
- The algorithm assumes good reward free exploration in the source tasks, but this can be an arbitrarily hard problem. 
- The impossibility result (theorem 3.2) indicates that transfer learning is hard without the assumption of generative access to the source environments. However, are there other assumptions that are not as strong as generative access, but make transfer learning feasible? Is there an in-between assumption? Maybe pointing to some prior work here would be helpful. 

review_title: Novel analysis of representation transfer in RL

Decision: Accept