Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work presented State-Noisy Markov Decision Process (SN-MDP), where there is a noise generating mechanism (either from the environment noise or from the adversary), and the theoretical properties (such as convergence and contraction) for corresponding (expected) Bellman operator and distributional Bellman operator were proved. The theoretical analysis was done for both tabular and linear funcion approximation settings. Especially in function approximation setting, authors characterized the robustness blessing of distributional RL based on histogram distributional loss and analyzed how the noise factor affects TD learning by using influence function that utilizes the perburbation method. Empirical analysis was done for DQN and QRDQN by varying noise standard deviations and the position of noise (state/successor state or both), which aims to support the authors' intuition coming from their theorems. 
main_review: Authors extended State-Adversarial MDP to SN-MDP by considering more general noise generating mechanism. The theoretical contribution of this work seems clear, but I couldn't find out clear motivation and contribution from the experiments. The given experiments seems more focused on whether SN-MDP's Bellman operators and TD learnings can be used for various control problems or not and whether those are aligned with theoretical intuitions. However, I believe authors should have given experiments on the usefulness of this framework (e.g., authors may suggest more practical but simple problems where training observations are noisy as stated in the introduction). Also, only 3 runs are used for each experiments, and all results are reported without standard errors, which means that we cannot evaluate the statistical significance just by using the reported results. Therefore, I believe the empirical results should be refined for this work to be accepted. 
summary_of_the_review: Although the theoretical contribution seems clear, I believe we need additional experiments to support the authors' claim.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies the robustness of distributional reinforcement learning, in particular the robustness on state observations, which have been demonstrated in a few papers on adversarial attacks to deep reinforcement learning. Compared to existing works on robust reinforcement learning on state observations, the main difference in this work is that it considers the distributional RL setting and also considers noise during training time. Theoretically, the authors find that distributional RL can be more robust under this setting, via the lens of Lipschitz continuity of the loss function and the influence function. The findings are also verified empirically on 4 benchmarks.
main_review: Strengths:

1. This paper studies the robustness of distributional RL agents, which were never done by previous works. The results are quite promising and show that distributional RL might be more robust than expectation-based RL agents under noises on state observations.

2. The analysis includes both tabular and function approximation case, and an important theorem on the Lipschitz continuity of distributional RL is given.

3. Experiments are relatively comprehensive.

Weaknesses:

1. The proposed SN-MDP seems to be very similar to SA-MDP (Zhang et al., 2020). I understand the analysis under the distributional RL setting is new, but it is not clear to me what is the main difference in terms of SA-MDP formulation. Can you explain?

1. The paper is not well organized and writing can be improved. For example, there is no need to create 4 sub-sections for showing experimental results on 4 models. I think one subsection for random noise and one subsection for adversarial noise can be better. Theorems are not well motivated and the writing is hard to follow, especially in Section 4.2 and 4.3.

Questions:

1. In Figure 3 the DQN lines (dotted red lines) look better than QRDQN, which do not match texts ("QRDQN (solid lines) almost consistently outperforms
DQN"). Is it a plotting error?

2. Is it possible to empirically measure the Lipschitz continuity (Theorem 2), the positive definitiveness of the matrix $X^T DPE$ (Theorem 3), and the influence function (Theorem 4) on a toy example?
summary_of_the_review: I feel this paper does present an interesting study but my main concern on is its presentation. I feel this paper can become a good paper if the weakness and questions mentioned above can be addressed. I currently rate the paper at the borderline but I am willing to reevaluate the paper based on the authors' response.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper investigated the adversarial robustness of distributional RL against noisy/adversarial states. They proved the Lipschitz continuity of distributional RL , provided convergence condition for TD update under noisy states, and conducted sensitivity analysis
main_review: The idea of understanding the distributional robustness of distributional RL seems novel and has not been done before. Other than that, the technical contributions and insights are marginal, unclear, and disconnected: 
- Theorem 1 is a marginal extension of (Zhang et al., 2020): they impose a distribution $N$ on a disturbance function $v(s)$
- Function approximation: It was actually a simple linear model with KL loss as the distributional loss for distributional RL, thus the Lipschipts smoothness there is very straightforward and is not really helpful for the real setting of distributional RL. Moreover, In RL, we often consider bounded reward, so if we consider the expected RL with squared loss function, it still enjoys all the properties of KL loss arising from distributional RL 
- Disconnected results: How were the analysis in Sections 4.2 and 4.3 related to distributional RL? After the sensitivity analysis, the paper did not give any actionable insights from their analysis except that "the degree of sensitivity is heavily determined by the task" . This conclusion is not helpful and does not need any such sensitivity analysis. 
- The paper says that QRDQN is more robust than DQN in MountainCar but Figure 3 tells the opposite. Plus, three experiments are of the same nature; thus I think it is not necessary to split it into three subsections to make them sound comprehensive. 
- Section 5.3: "QRDQN eventually achieves similar performance as DQN, although QRDQN significantly reduces the sample efficiency [in Breakout]". From my own experience with this experiment, I don't think so. 


**Minor comments**:
- Eq (4) can be simplified 
- Section 3.1: "the state space go to infinity": unclear 

- 
summary_of_the_review: The idea of exploring adversarial robustness of distributional RL is interesting and novel, but the analysis presented in the paper are marginal, disconnected, and does not support the understanding of the robustness of distributional RL. 

===== AFTER REBUTTAL ===    
Increase the score to 3
review_title: null

Meta-review: Description of paper content:

A mixed theoretical and experimental paper that investigates the robustness of distributional RL to perturbations of state observations as compared to expectation-based value function learning. They provide sufficient conditions for TD’s convergence and prove the Lipschitz continuity of the loss of a histogram-based KL version of distributional RL with respect to the state features, whereas this is not true for expected RL. This continuity indicates a certain robustness of the loss with respect to perturbations of the state. The theory’s tie to experiment is weak in the sense that it is not predictive of the actual performance of any algorithm. The theoretical methods are based on a previously published paper SA-MDP.

Summary of paper discussion:
The reviewers raised concerns about the statistical significance of the experimental results, the clarity and organization of the writing, the novelty of the theoretical setting, and its usefulness for describing a real problem setting. The majority of reviewers rejected the paper and did not lift the scores after the rebuttal. 

(I personally wonder if the community would not benefit from conducting some of these kinds of theoretical analyses and experiments on LQR systems rather than Atari (etc.) environments.)
Decision: Reject