Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The paper attempts to give theoretical support for using shared representations among multiple tasks.  The architecture has already been proposed in another paper.  The main contribution of the paper is the theory that it claims to support this architecture.  However, I am dubious that the architecture achieves the claimed bound.  One, I do not see how the analysis of this paper is connected to this specific architecture.  Two, the analysis follows from an existing paper by Farahmand (2011), which has already established a bound on the difference Q* - Q^{\pi K}.  This paper considers the same difference, separately for each task, so that the same bound from Farahmand (2011) can be trivially used.  The shared layer h would affect the Q-values, but considering only the difference Q_t* - Q_t{\pi K} abstracts away how sharing representations is helpful.  Simply averaging the norm of the difference between Q_t* - Q_t{\pi K} as if they are independent ignores the fact that the Q_t’s are all dependent due to the shared layer h.  Also, I do not think that the proof of Theorem 2 and Theorem 6 can use the bound of Farahmand (2011).   Because the definition of approximation error epsilon_k in Farahmand (2011) is very different from epsilon_{avg,k} used in this paper, this warrants the analysis to start from the beginning, deriving the bounds from relating Q_t* - Q_t^{\pi K} to (\epsilon_{avg, k})_{k=0}^{K-1}.  The experiments compare known algorithms, and I am unsure how they support the theoretical bounds.   

Other comments:
Inside the definition of (T*Q)(s,a), the probability measure P(s’|s,a) would be P^{(t)}.  Thus, T* is not one optimality operator shared amongst all tasks,  but one for each task.  The notation should reflect this.  Stating the descriptions of k, n explicitly in section 2 would be helpful.  I had to read to section 3 to be sure that k stands for a sequence of number 0 to big K and i stands for 1 to n with n being the number of samples.   Also, giving an intuitive explanation of gaussian complexity (1) and its use would also be helpful.    
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper provides a theoretical justification for the benefit of multi-task deep RL (MTRL) with shared representations. By extending prior work (Farahmand (2011) and Maurer et al. (2016)), the authors demonstrate that the bound of MTRL can be improved if the cost of learning the shared representation at each AVI iteration can be reduced, which is mitigated as we increase the number of tasks. The author also empirically verify their theoretical results in a tabular Q-Fitted Iteration domain and also in challenging RL domains such as Mujoco. The results show that MTRL with shared representation can outperform their single task counterparts to some degree.

Overall this paper adapts the theory shown in Farahmand (2011) and Maurer et al. (2016) to the setting of MTRL and demonstrates the effectiveness of using shared layers, which seems intuitive. While the theory seems a bit incremental, it’s the first paper that theoretically validates the benefits of sharing knowledge, which is a contribution to the MTRL field. I would recommend a weak accept, though I have a few concerns on experimental results, and hope that the authors can clarify them during rebuttal.

Specifically, as the authors have noted, there is a wide range of prior works [1,2,3] that have empirically demonstrated the effectiveness of utilizing shared representations in MTRL. While the authors claim that the goal of the experiments is to show that MTRL with shared layers can outperform its sing task counterparts and thus they ignore other MTRL approaches. I believe that is not the main argument of the paper. The authors should provide empirical evidence on the claim that with an increasing number of tasks in MTRL, the error bound should improve and the performance of MTRL should also boost. Besides, I find the comparison where single-task training is initialized with shared representation a bit confusing. Training would definitely be improved when it’s initialized with some related pretrained features. Maybe the authors should compare this to some other methods such as initializing with single-task representation or even representation learned from training different tasks.

[1] M. Hessel, H. Soyer, L. Espeholt, W. Czarnecki,S. Schmitt, and H. van Hasselt. Multi-task deep reinforcement learning with popart.arXiv preprintarXiv:1809.04474, 2018.
[2] Teh, Y.W., Bapst, V., Czarnecki, W.M., Quan, J., Kirkpatrick, J., Hadsell, R.,Heess, N., Pascanu, R.: Distral: Robust multitask reinforcement learning. In: Ad-vances in Neural Information Processing Systems 30: Annual Conference on Neu-ral Information Processing Systems 2017 (2017)
[3] Wulfmeier, M., Abdolmaleki, A., Hafner, R., Springenberg, J. T., Neunert, M., Hertweck, T., ... & Riedmiller, M. (2019). Regularized Hierarchical Policies for Compositional Transfer in Robotics. arXiv preprint arXiv:1906.11228.
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: The submission derives bounds for approximate value and policy iteration for the multitask case in reinforcement learning. In addition, two common RL algorithms are adapted to demonstrate benefits of multitask RL given related tasks.

The paper is mostly well written but sometimes introduces potentially unnecessarily complex mathematical notation including missing and dual definitions which slow down the reader. Examples of missing definitions are given by K an n on top of page 3. 

Generelly, the paper is quite self consist thanks to minor changes in existing algorithms to show experimental results for the theoretical insights, but instead of requiring the reader to find other papers it would be better to define all terminology in the appendix (see equation 6).

The main contribution of the submission is the extension of an existing bound to the multitask case as the architectures are common across existing work. The extension is relevant given growing interest in meta and multitask RL but the changes in comparison to the single task case are minor and the experiment’s value purely lies in supporting this extension. 

review_title: Official Blind Review #2

Meta-review: This paper considers the benefits of deep multi-task RL with shared representations, by deriving bounds for multi-task approximate value and policy iteration bounds. This shows both theoretically and empirically that shared representations across multiple tasks can outperform single task performance.

There were a number of minor concerns from the reviewers regarding relation to prior work and details of the analysis, but these were clarified in the discussion. This paper adds important theoretical analysis to the literature, and so I recommend it is accepted.
Decision: Accept (Poster)