Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper provides an algorithm for Differentially-Private offline reinforcement learning, for the tabular and linear MDP settings.
strength_and_weaknesses: Strengths
- the problem is interesting and has good motivation:  protection of sensitive information in the trajectories saved in offline reinforcement learning, and this is especially important in healthcare and financial contexts.

Weaknesses
1.  The paper is not clearly written.  Objects and variables are used without proper notation or introduction.  For instance I'm assuming $n$ is the number of trajectories in the dataset, but this doesn't seem to be stated.  What is $\phi(s,a)$, used in algorithm 2?
Also, the theorems can be stated more clearly.  For instance Theorem 3.1 has parentheses with phrases that distract from the reading of the theorem - this should be re-written.
2.  It's not obvious how the noise value $\sigma^2$ is derived for Algorithm 1.  Shouldn't the l2 sensitivity of the counts be $Hn\sqrt{n}$, and so $\sigma^2= H^2n^3/(2\rho)$ ?
3.  Too much information is relegated to the appendix, to the extent that it isn't easy to the read the paper without reading the appendix.
clarity,_quality,_novelty_and_reproducibility: Originality:  the work is original in that this particular setting for differential privacy has not been considered before.  The algorithms themselves are heavily based on previous work of Yin and Want 2021b.  
Clarity:  As mentioned above the paper lacks in clarity.
summary_of_the_review: While the topic of the paper is important, has good motivation, and is novel, the paper is very difficult to read and may contain errors.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes two privacy-preserved (i.e., DP Guaranteed) versions of existing offline RL algorithms (i.e., APVI and VAPVI) proposed in previous works . DP-APVI  is for tabular settings and  DP-VAPVI  for the case with linear function approximation (under linear MDP assumption). The authors provide theoretical analysis and also simulation results to confirm their theoretical analysis and show that their DP Guaranteed algorithms come with little drop in utility. 
strength_and_weaknesses: Strength:
* Addresses an important problem in offline RL
* The paper is well organized, theoretically grounded

Weakness:
* Lack of empirical evaluation such as experimentations on benchmark tasks such as D4RL. Additionally is there any reason that there is no simulation results for DP-APVI algorithm ? 
* Incremental Novelty. Would be nice to distinguish more clearly the novelty of this paper from previous offline works (i.e., VAPVI).

clarity,_quality,_novelty_and_reproducibility: * Quality: In the interest of time, I did not check the proof. But, according to a sketch of the analysis, it sounds correct.
* Clarity: Yes, it is clear
* Novelty: Incremental novelty over previous offline works by adding DP. It adds  to some extent  to the knowledge of the community.
* Reproducibility: Not clear to me, since source code and dataset were not available in supplementary materials. 
summary_of_the_review: Overall, the paper is well motivated, aiming to solve an important problem in the area of offline reinforcement learning. The work is theoretically well grounded, and well organized. As said the novelty is low, and I think it lacks enough experimental evaluation.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes offline RL algorithms with DP guarantees for tabular case and linear MDP. The key components are the private estimates of the visitation counts/conditional variance based on the Gaussian Mechanism, with some modifications to allow the use of existing pessimism-based value iteration algorithms. The paper provides privacy guarantees and utility guarantees for the proposed algorithms, and an empirical study to support the theoretical findings. 
strength_and_weaknesses: **Strength**
- The paper proposes a new offline RL algorithm with DP, which is claimed to be the first DP RL algorithm in the offline setting. I think the paper can have significant contribution to the offline RL community. 
- The paper in general is clearly-written and easy to follow. 
- The theoretical analysis is sound (although I acknowledge that I only check the proof for tabular case, and didn’t check the proof for linear MDP carefully).   

**Weaknesses**
- The definition of DP for the RL setting used in the paper is not well-motivated (see comments below).
- The presentation of the key components in Algorithm 1 and Algorithm 2 is not very clear to me (see the next part on quality and clarity).
- Major improvement is required for the empirical study.

**DP Definition**  
The paper defines neighboring datasets as two datasets that differ in one trajectory. However, I can think of other definitions in the RL setting. For example, two datasets that differ in one transition, or differ in one observed reward (if we want to preserve the privacy for reward only). The paper should provide more motivation for this particular definition used here, and discuss how existing work define DP in the RL setting. 

**Empirical study**  
I think the empirical study requires some improvements. The empirical study is supposed to support the theocratical findings, but I think the current results raise more concerns.  

First of all, the results do not have error bars, so I don’t know whether the difference are statistically significant. Please include the error bar in the results. Moreover, the paper mentions
> as the size of dataset goes larger, the performance of DP-VAPVI will converge to that of VAPVI

I don’t see the performance of DP-VAPVI converges to the non-private one in Figure 1. Maybe try to run the experiments with more episodes. Otherwise, I think this observation is not true. 

Finally, please explain how significant the difference in performance between the private and non-private ones are. For example, non-private gets a suboptimality gap ~= 1, while the private one with $\rho = 5$ gets a gap ~= 2. Is the difference (i.e., 1) significant? What is the range here? What value of $\rho$ is commonly used in practice?

clarity,_quality,_novelty_and_reproducibility: Overall, the paper is clearly-written, but some improvements are needed. 

The presentation of the key components can be improved. For example, it takes me some time to understand the optimization problem in (3), so I think the paper would be better if some descriptions are provided to explain what is happening in (3) and how is $E_P$ chosen (a short sentence suffices). I think this is the non-trivial part to combine common DP mechanisms in pessimism-based VI algorithm, so it is definitely worth spending some space to explain it more clearly. 
 
I have some other questions:
> Our algorithms involve substantial technical innovation over previous works on online DP-RL with joint DP guarantee. 

For the tabular case, is the main novelty how you obtain private visitation counts that satisfies $n’_{s,a} = \sum_{s’} n’_{s,a,s’}$? For the linear MDP case, the proposed algorithm seems like a straightforward application of Gaussian mechanism in the plug-in method. Am I missing something here? 

> Assumption 2.1 is the weakest assumption needed for accurately learning the optimal value.

"weakest" in what sense? Do you have any citation for that? 

For Algorithm 2, you add a noise with variance of the order of $H^4$. I am wondering if this can be an issue in environments with a higher horizon (e.g., 100 or 1000). Also where does the quadruple dependence come from? Is it improvable? 

summary_of_the_review: In summary, I am giving a weak accept recommendation, conditional on the empirical section can be improved as suggested. I think the paper can have a significant contribution to the offline RL community as a first step towards more practical DP offline RL algorithms, however, some improvements are required as mentioned in my comments on weakness and clarity. I would be happy to raise my score if the concerns are addressed. 

------Post-rebuttal update------  
After the author response and internal discussions with other reviewers and AC, I think my concerns about the presentation clarity and the empirical study have not been fully addressed. I think these issues are not difficult to address, however, the authors did not upload a revision. Therefore, I think the paper requires some improvements before being published, and I slightly lowered my score to 5. However, I encourage the authors to keep improving the paper and re-submit to a future conference.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper studies differentially private (DP) reinforcement learning (RL) in the offline setting. The authors provide DP algorithms for this problem and formally prove their privacy and utility guarantees. The utility guarantees are in terms of learning bounds under the tabular and linear MDP settings. 

Despite introducing some novel techniques such as privately computing the conditional variance, the analysis seems to largely follow existing works, e.g., [Chowdhury & Zhou 2021] and [Yin & Wang 2021]. 

The reviewers have also pointed out that the algorithmic novelty is limited as the main algorithm (Algorithm 1) relies heavily on a prior work [Yin & Wang 2021]. 

There have been also some concerns regarding the tightness of the upper bounds and the possibility of simplifying the algorithm by using the discrete Gaussian mechanism. Additionally, there is some criticism about the lack of motivation behind the notion of "neighboring datasets" in the proposed definition of DP in offline RL. 

Moreover, the presentation and writing could be improved in several parts of the paper, especially in Sec. 2. 

Finally, the paper could have benefited from a more convincing, extensive empirical evaluation on benchmark datasets for RL. 

All in all, the paper can benefit from a careful revision to address these issues. 
justification_for_why_not_higher_score: The paper has several shortcomings summarized in the meta review. There was a general consensus among reviewers that the paper is below bar for acceptance.
Decision: Reject