Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper considers the problem of offline reinforcement learning and investigates the performance of the residual gradient algorithm (RG) and an improved residual algorithm (R) on SAC. Empirically, the simple trick increases the score on D4RL gym tasks. By adding a residual component to SAC-N, the algorithm (when $N=10$) matches the state of the art performance on most of the tasks compared with SAC-N (which requires hundreds of ensembles). It reduces the computational cause dramatically.

Overall, this paper provides a clear explanation of the RAISIN algorithm (which is simply RA on top of SAC-N) and provides experimental results comparing the performance improvement and the ablation studies.
strength_and_weaknesses: The methodology is simple, and the only parameter to tune is the pessimistic, as everything else is based on top of SAC. The proposed algorithm achieves comparable performance with much less ensembles, which makes the algorithm promising.

The weaknesses of this paper are listed as follows:

- It seems that the novelty of this paper is limited, the methodology is a direct application of RA on SAC-N, and there are little theoretical discussion on how the performance get improved. 
- In Section 4.1 comparison with TD3+BC-10, the authors simply claim that Raisin outscores TD3+BC-10. However, in table 1, there are some tasks that RAISIN scores lower than TD3+BC, for example on hopper-expert, walker2d-expert, walker2d-medium-expert, hopper-medium-expert, hopper-medium, halfcheetah-random, etc. The authors do not provide sufficient explanation regarding what are the properties in these tasks that makes the performance of RAISIN worse.
- Since this paper does not provide new techniques or methodologies that shed lights on future researches in offline RL, but only provides experimental studies on a combination of two existing methods. It is important for the authors to provide more thorough empirical explanations and interpretations. The experimental results are good for general papers, but for a paper without enough technical novelty, higher standard in experiments should be expected.

clarity,_quality,_novelty_and_reproducibility: The paper is written clearly, the methodology lacks some novelty, the empirical results do provide some useful observations.
summary_of_the_review: This paper provide a novel observation that RA + SAC-N has good performance and reduced computational cost. The algorithm becomes more robust on a number of gym tasks. However, the performance of RAISIN does not outperform all SOTA results on all tasks (when compared with TD3+BC), and the experiments are not interpreted sufficiently.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper incorporates the resisual algorithm (RA) with SAC-N used in offline RL setting, and finds that the additional RA objective can not only improve the SOTA performance, but also reduce the number of ensemble functions for a more efficient computation.
strength_and_weaknesses: Strength
1. The paper empirically shows that adding RA to SAC-N is helpful in reducing the number of ensemble members. 

Weakness:
1. The proposed RAISIN method can be viewed as a combination of RA and SAC-N, but in some sense lacking adequate motivation: RA method is helpful to the stable policy training, and the main issue of SAC-N is the large number of ensemble functions, but why simply combining the two would be good?

2. The proposed method can be easily extended to most current popular offline rl algorithms, so it's better to add more experiments to verify its scalability.

3. About the pessimism settings in Appendix.A, when N=10, this paper chooses $\eta=0$ on half of tasks, but this choice means an actual equivalence with SAC-10, so I doubt whether the proposed method will work in this case.
clarity,_quality,_novelty_and_reproducibility: fair
summary_of_the_review: Although the proposed method can achieve significant performance improvement, there lacks adequate analysis and explaination on why combining the RA and SAC-N is useful.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the author integrated the Residual gradient algorithm RG into SAC and conducted extensive experiments to prove this new approach can achieve the same level of SAC-N’s performance and, at the same time, greatly reduce the critic number N. The newly proposed method is more “versatile,” which means it can converge under a diverse range of datasets.

The main contribution of this paper is it gives a comprehensive study of the RA approach’s effect on SAC in the offline learning framework. And through comparing to state-of-the-art algorithms, it proves that combing RA into SAC can achieve the same performance while significantly reducing the number of critics. 

strength_and_weaknesses: Strength: 

- The strength of this paper is that 1. It conducted comprehensive experiments to prove the effectiveness of RA with the SAC algorithm and its generality to different datasets.

- The paper gives a very clear motivation of why they investigated a relatively old and not proven effective algorithm and conducted experiments from different angles, such as the reduction of N compared to SG-based SAC methods and different value initialization. 

Weakness:

- Most of this paper’s efforts are on the experimental comparison of SG and RA SAC algorithms. And the difference between them is adding the gradient of the next states into the loss function. Though these two approaches have significant differences, the RA approach has already been investigated by previous researchers, as mentioned in the RA-DDPG work.

- In the baseline comparison, the author should also involve past work that extended RA to DDPG to prove their models' effectiveness and novelty. Since in the introduction, the author talked in detail and claimed RA is in nature beneficial to offline learning reinforcement learning, they should provide more information to prove it, not just compare between SAC methods. 

- This paper aims to prove that  RA-SAC is more versatile than the SG counterparts, and compared to traditional SAC, it needs fewer critics, thus significantly reducing the computing cost. However, as introduced in section 1, the RA method is slower to converge. So the author should also provide convergence curve comparison, not just RA-SAC’s. 


clarity,_quality,_novelty_and_reproducibility: The clarity of this paper is good and easy to follow. It provides a detailed discussion of SG and AG differences. However, some of the propositions are not clearly explained. For example, on page 2, the theoretical concerns are important but not mentioned though it seems this paper is focusing on experiments. 

Quality: this work’s quality is good but not perfect. The author conducted many experiments to support their claims, but in order to prove that RG could outperform SG, more investigations, such as different agents or tasks, should be involved. 

Novelty: the novelty is good since the investigation of combining RG and SG approaches is not enough. But the technical contribution is not that much. 

Reproducibility: the technical description is okay and not in detail. The producibility could be increased if the author releases their codebase. 

summary_of_the_review: This paper investigates the RA approach, which does not attract much attention in the RL field. Though the author provides many experiments to support their claim, the comparison is not enough to justify their claims( as described in the weakness and quality).
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper did an empirical investigation of residual algorithm, i.e., weighted averaging of residual gradient algorithm (RG) and semi-gradient algorithm (SG), for SAC. It showed empirically, the simple trick works great for SAC in offline setting, increasing the score of SAC on D4RL gym tasks, and achieving comparable performance for SAC-N with much less ensembles. All the reviewers expressed concerns on the limited novelty, i.e., applying a well-established technique. The scope of the work is quite limited. For example, does the conclusion generalizes to other RL algorithms involving bootstrapping (solving the Mean Squared Bellman Error) beyond SAC. In addition, the work missed comparison to more recent offline RL algorithms. The performance seems comparable at best or worse than CQL or decision transformer. It is thus hard to justify the cost of using ensemble of 10 critics. 
justification_for_why_not_higher_score: Limited novelty and scope of the work. Missed comparison to more recent offline RL algorithms which showed better or comparable performance with much lower compute.
Decision: Reject