Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper looks into whether carefully making design choices for Offline RL training objectives can have similar power law scaling to supervised learning.
strength_and_weaknesses: Strength
- The paper is very well-written.
- Design choices are well-explained and motivated. 
- Experiments/ablations are abundant, and experimental results are convincing.
- The paper offers very convincingly results that offline RL algorithms can train very large models on a massive amount of data. This paves the way for a large unified decision-making foundation model (so to speak).

Weakness
- As thorough as this paper is, I find it relatively disappointing that despite many different offline RL algorithms having been proposed over the last 2-3 years, the authors only used CQL. This opens up several interesting empirical questions:
  1. The paper makes it seem that DR3 (feature normalization), categorical representation of return values, and Q-function architecture matter a great deal. Does the offline learning algorithm matter at all? Would TD3 + BC also scale? Would BCQ scale? It's definitely difficult to evaluate every single offline RL algorithm, but it would be great if at least one other offline RL algorithm is chosen to compare with CQL.
  2. With a huge amount of data (400M transitions and 2B transitions), one might wonder if pessimism is no longer needed since all possible (s, a) might have already been covered -- is the pessimistic penalty really necessary? An ablation study can train with TD3 directly or any off-policy Q-learning algorithm.
- "...making too many TD updates to the Q-function in offline deep RL is known to sometimes lead to performance degradation and unlearning" [1]. On such a large dataset with 40 games, is there an optimal stopping point for training? Is there performance degradation if you train longer? Or do you think because of DR3, performance degradation won't happen anymore? 

[1] DR3: VALUE-BASED DEEP REINFORCEMENT LEARNING REQUIRES EXPLICIT REGULARIZATION
clarity,_quality,_novelty_and_reproducibility: Very clear. High quality. Very novel. Hard to reproduce, but for large-scale studies like this, we can't ask for too much.
summary_of_the_review: I think this paper's quality far surpasses the current score I'm giving (which is a 5). I'm giving this score because I think the authors (with access to the amount of computing power and resource) should address some of the concerns I raised. A simple rebuttal would be, "investigating different kinds of offline RL training algorithms is out of the scope of this paper" -- this I fully understand, but the authors have a great experimental setup, and a massive amount of data, which I doubt many other people have access -- some preliminary investigations on my questions would benefit the Offline RL community tremendously.
review_title: null

final_score: rating: 10: strong accept, should be highlighted at the conference
review_content: summary_of_the_paper: This paper targets harnessing the potential of offline RL to allow high-capacity models to be trained on large datasets from a range of tasks, enabling generalization.  They propose modifications to standard offline Q-learning setups that enable performance to scale with model capacity and then demonstrate the effectiveness of these modifications through thorough experimentation.  They successfully train a single model on 40 Atari games and demonstrate that the model is able to outperform humans on most of the games, dramatically outperform other offline methods when trained on suboptimal data, and also provide a strong initialization for fine-tuning on harder variants of the individual games.

strength_and_weaknesses: Strengths:
- The obtained results are very impressive, from the effectiveness of the methods to enable strong scaling trends, to the ability to dramatically surpass the scores of the training trajectories, to the strong online and offline fine-tuning results.
- The proposed modifications were well motivated and the ablations illustrate the contribution of each change.
- Well written, with formatting that makes it easy to follow the key contributions and points.

Weaknesses:
- None of the results except Figure 8 have error bars despite the fact that the error bars in Figure 8 indicate there may be substantial variance.
- As far as I can tell, the MT Impala-DQN* setting is not described in/around Figure 5.

clarity,_quality,_novelty_and_reproducibility: The paper is well written, the experiments are high quality, the results are novel and the experiments can be reproduced thanks to the thorough descriptions in the appendix.  Due to the lack of error bars it is unclear, however, that the results of such a reproduction would be the same (though the size of the improvements of their proposed method are large enough that it is unlikely variance would substantially change the conclusions).
summary_of_the_review: This paper provides a major contribution towards being able to use large datasets and high capacity models, which have been very effective in other domains, to solve RL problems.  The proposed changes are clear and the results are very compelling, I strongly recommend acceptance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper aims to scale up model capacity and improve the generalization performance across tasks with offline Q-learning methods. In contrast, prior works mainly centered around small-scale, single-task problems where broad generalization and learning general-purpose representations are not expected. To this end, the paper proposes a framework Scaled Q-learning that combines the following three choices (1) make modifications to the ResNet family - utilize group normalization instead of batch normalization in ResNets, and utilize point-wise multiplication with a learned spatial embedding when converting the output feature map of the vision backbone into a flattened vector which is to be fed into the feed-forward part of the Q-function, (2) leverage a distributional representation of return values and a cross-entropy TD loss for training, and (3) use feature normalization to stabilize training. The experiments show the proposed framework outperforms the baselines (decision transformers). Ablation studies suggest that (1) C51 leads to much better performance for both ResNet 50 and ResNet 101 models in terms of mean squared error and (2) adding feature normalization significantly improves performance for all the models. I am leaning toward accepting this paper because it studies a promising research direction (i.e. improving offline Q-learning) and proposes a reasonable framework with supporting experimental results. My only concern is that the experiments can be improved by adding results from different domains such as robot manipulation, locomotion, or navigation.

strength_and_weaknesses: ## Paper strengths and contributions
**Motivation and intuition**
The motivation for addressing the problem of learning general-purpose representations in offline Q-learning is convincing.

**Novelty**
In my opinion, studying extrapolation beyond datasets even when trained entirely on a large but highly suboptimal dataset in order to increase model performance when the model capacity increases seem novel.

**Technical contribution**
- The idea of utilizing standard feature extractor backbones from vision (i.e., the Impala-CNN architectures) to improve performance as model capacity increases is intuitive and convincing.
- The outcomes of combining and empirically verifying the effectiveness of several techniques should be helpful for the research community.

**Clarity**
- The overall writing is clear. The authors utilize figures well to illustrate the ideas and framework. Figure 3 clearly shows an overview of the network architecture.
- The paper gives clear descriptions in both theoretical and intuitive ways. The notations and the formulations are well-explained. 

**Ablation study**
Ablation studies are comprehensive. The proposed framework consists of multiple components. The provided ablation studies are helpful for analyzing their effectiveness of them.

**Experimental results**
The presentation of the experimental results is clear. Particularly, Figure 7 provides clearly understandable results showing pre-trained representations from Q-learning enable positive transfer to novel games and lead to significant performance gain compared to return-conditioned supervised learning methods and representation learning approaches.

**Reproducibility**
Given the clear description in the main paper and the details provided in the appendix, I believe reproducing the results is possible. 

## Paper weaknesses and questions

**Experiment**
The experiments could be more diverse. While the claims and the results look promising, the proposed method is only evaluated in Atari games. I believe evaluating in robot manipulation domains, locomotion domains, or navigation domains would make this work a lot stronger.
clarity,_quality,_novelty_and_reproducibility: See above

summary_of_the_review: I am leaning toward accepting this paper because it studies a promising research direction (i.e. improving offline Q-learning) and proposes a reasonable framework with supporting experimental results. My only concern is that the experiments can be improved by adding results from different domains such as robot manipulation, locomotion, or navigation.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents a combination of learning techniques and model design choices that enable the training of a single policy in offline Q-learning settings that can generalize across tasks. The authors present extensive evaluations on the popular multi-task Atari test bed involving 40 different games. Authors show that the performance of their approach scales with capacity and even works when trained on suboptimal offline datasets. Lastly, models trained on diverse offline data show very favourable and fast adaptation to new game variants.
strength_and_weaknesses: Strengths:
- The paper shows that augmenting CQL by only three, rather simple but broadly applicable design choices enables learning a much improved single multi-task policy from suboptimal expert data.
- The method shows a nice scaling behaviour for models of increasing capacity
- Finetuning and adaptation experiments to new game variants are very compelling 
- Convincing and superior experimental results over baseline
- Ablation studies to assess the relevance of each model choice.
- The paper is well-structured and easy to follow.
- Nice presentation of results

Weaknesses:
- The paper makes a reference to an anonymous work from which the backbone architecture seems to be motivated. Unfortunately, I could not find this work in the supplementary material to gain further insights into the design decisions and the importance of this component.
- The importance of the backbone architecture is unclear as there are missing ablation studies.
- Minor: the policy still necessitates individual heads for each game and it would have been interesting to see how limiting this particular design choice is for obtaining the reported performances.

Minor:
- p.5: “[...] we found that this this speeds […]”


clarity,_quality,_novelty_and_reproducibility: The paper is very well-written and structured, easy to follow and the evaluation and technical implementation are sound. No codebase has been provided to judge reproducibility. While the individual components that are being introduced are not necessarily novel in themselves, it is the combination and claimed model performance that is novel and important for the field.

summary_of_the_review: Progress in offline RL is still often lagging behind online RL and I believe this paper represents an important step forward by augmenting CQL by a set of important techniques and model choices that allow the training of a single multi-task RL policy from suboptimal datasets with very favourable scaling and adaptation properties. The authors present very convincing and strong empirical evidence on the very challenging Atari test bed. I, therefore, recommend acceptance of this paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper aims to scale up model capacity and improve the generalization performance with offline reinforcement learning. In particular, the proposed design choices include making modifications to ResNets, leveraging cross-entropy based distributional backups and utilizing feature normalizations. Experimental results demonstrate that the effectiveness of the methods to enable strong scaling trends and the ability to dramatically surpass the scores of the training trajectories.
 
Overall, the paper is well written. The design choices are well explained and motivated. The experimental results and ablation studies are solid. All the reviewers agree with the novelty and solid intuition behind the proposed method and in-depth experimental evaluation. 

justification_for_why_not_lower_score: Since all the reviewers agree with the novelty and solid intuition behind the proposed method and in-depth experimental evaluation, I would like to choose Accept with Oral.
Decision: Accept: notable-top-5%