Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: summary: This work proposes a Cramer loss for the training process of distributional RL, and design a centered non-crossing architecture for distributional RL.
main_review: Strong points:
1.	Clarity: The paper is well structured.
2.	Technical novelty: A efficient algorithm is proposed to compute the Cramer distance.

Weak points:
1.	I am concerned about the validity of the experiment results, since the performance of QR-DQN in Figure 5 is much worse than the results in their original paper. I am not sure if the authors correctly implement this method and the conclusion seems to be unconvincing. 
2.	The advantages of Cramer loss over quantile (Huber) loss are not well presented. As mentioned in your paper, both losses achieve unbiased gradient and equivalence to 1-Wasserstein in minimization problem. What are the pros of Cramer loss? Computation efficiency? Or else? Corresponding ablation studies are lacked.
3.	Unfortunately, there is a nearly same network architecture called NDQFN (Zhou et al., 2021) with proposed centered non-crossing architecture (CNC), except that CNC focus on median while NDQFN aims at \tau = 0. However, NDQFN seems to achieve much better performance than CNC with Cramer loss.
4.	The experimental results in the present form is not strong. From the Figure 6 and Figure 10 in Appendix, it seems CNC-CR-DQN perform comparable with QR-DQN, even slightly worse.

Hence, I vote for rejection for current form.

Mentioned reference:
Zhou, F.,Zhu, Z., Kuang, Q., Zhang, L*. Non-decreasing quantile function network withefficient exploration for distributional reinforcementlearning. International Joint Conference onArtificial Intelligence (IJCAI), 2021. In press.

limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The authors provide theoretical analysis of distributional RL under Cramer distance. The authors further propose a neural architecture that yields monotonic quantiles and achieved improvements over prior works.
main_review: Originality: Low. Cramer distance in dsitributional RL has been studied in prior works, the authors provide some missing theoretical results. The proposed algorithm is slightly modified from NC-QR-DQN.

Quality: Medium. The theories are correct, but the experiment results are insignificant.

Clarity: Medium. The paper is easy to read, except for some minor issues.
 1. l.154 Formula...
 2. In l.92, l_p is the p-Wasserstein distance, but later used as the Cramer distance without clarification.
 3. Figure 3. is a little hard to interpret

Significance: Low.

Major concerns:
 1. After reading the paper, I find that while correct, the theoretical results are not likely to be able to improve dsitributional RL algorithms.
 2. From my opinion, NC-QR-DQN should be an important baseline of the work. But the authors didn't provide further comparison other than the three cases in Figure.5, therefore it is hard to tell the actual improvements of the proposed neural architecture.
 3. Missing score table, incomplete experiments (2 seeds) in Figure 10.


Questions & suggestions:
 1. If Cramer loss and Quantile Regression loss yields collinear gradients, why not use QR loss to save the trouble of computing Cramer distance? We only need the gradients to update network parameters.
 2. According to my understanding of the paper, the proposed neural architecture, CNC, have three major differences compared with NC-QR-DQN:
  1). For subnetwork 1, outputs median quantile value instead of minimum quantile value. Then, substract the cumsum of the first half per-quantile value differences instead of add.
  2). Remove the scaling factor, thereby using ReLU instead of Softmax for per-quantile value difference.
    I would suggest an ablation study on those two modifications to further investigate which one contributes more to the improvements.
limitations_and_societal_impact: The authors discuss the limitations adquately in their work.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper draws an equivalence between 1-Wasserstein distance and Cramer distance on staircase distributions, use Cramer distance in the place of 1-Wasserstein (and quantile regression) to avoid order constraints among the particles $\theta_i$ and consequently, proposes a non-crossing neural net for computing the Cramer distance in DRL.
main_review: **Strong points**: 
-	The connection between 1-Wasserstein and Cramer seems novel and interesting in DRL.  Especially, quantile regression in QR-DQN imposes order into the learnable particles $\\{ \theta_i \\}_{i=1}^N$ as each $\theta_i$ is an estimate of the $\hat{\tau}_i$-quantile which implies the monotonicity in the particles, i.e., $\theta_1 \leq \theta_2 \leq ... \leq \theta_N$. Cramer distance on the other hand is equivalent to 1-Wasserstein in terms of projection onto the space of a mixture of Heaviside step functions and in terms of gradients. Moreover, Cramer distance does not require order among $\theta_i $

**Weak points**: 
- Many parts of the writing are unclear and need more improvement (see below)
- I think the present paper lacks a discussion and comparison with related work that is directly related to the problem and proposed solution. 
- The motivation and purpose of the experimental section are unclear and there is no conclusion or discussion drawn from the experiment. The experimental result does not show that the proposed algorithm CNC-CR-DQN outperforms QR-DQN with statistical significance in the whole range of Atari games (Fig 6). This raises the question of whether the proposed algorithm is effective and necessary. 
- The theoretical results are quite simple and the connection between Cramer and Wasserstein is already made in [2]. 


**Comments and Questions**:  
-	I think the paper lacks relevant literature. Most distributional RL methods are based on quantile regression that requires ordered statistics in neural representation, creating challenges such as maintaining ordered statistics, monotonicity, and “crossing quantiles”. I think a recent work [Nguyen-Tang et al AAAI’21, “Distributional Reinforcement Learning via Moment Matching”] can completely avoid such problems in quantile-regression-based methods using moment matching; thus it is directly related to the problem and the proposed approach in the present paper. What is the difference between your approach and Nguyen-Tang's in addressing the order statistics constraint in QR-based DRL? 
-	The paper addresses the crossing quantile problem in DRL but it only briefly touches on crossing quantile in one sentence at line 47-48 and at line 155. I think it is inadequate. More discussion is needed and probably one dedicated paragraph for this is necessary.
-	Line 130-131: What exactly permutation in the symmetric group of size N? What **symmetric group** are you referring to? This part needs to be clearer. 
-	Eq. (14): What is $\theta’$? A typo? 
-	The experiment section lacks a clear explanation what is the purpose of the experiment? What conclusions can be drawn from the results? Currently, this section merely just describes the experimental procedure without insights drawn from it.
-	I think putting markers on Fig 6 would make it easier to read. Some colors are very similar. 
-	Fig 6:  CNC-CR-DQN outperforms QR-DQN in several games but in the entire Atari set, they perform very similarly (Fig. 6). How can you conclude that the proposed method that draws on Cramer distance without crossing quantiles works better in a statistical sense? 
- 1-Cramer distance ($ \int |F(z) - G(z)| dz $) is identical to 1-Wasserstein distance and using 1-Crammer distance also does not require order among $\theta_i$ as 2-Cramder distance. Why don't just use 1-Cramer distance? 
- What kind of games from the Atari set the proposed algorithm performs better than QR-DQN?

**Initial Recommendation**
While the idea of using Cramer distance as an alternative to the quantile regression loss is interesting, I do not find sufficient novelty and/or significant empirical performance in the current form, raising the question about the necessity and effectiveness of the proposed algorithm; moreover, the writing seems in rush and lacks related work. I vote for reject this time for the current form.  

limitations_and_societal_impact: The authors have not adequately addressed the limitations of their work.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposed to apply the Cramér distance to replace the 1-Wasserstein distance in the class distributional reinforcement learning. To justify the validity of this update, this paper proved two key properties: 1) Cramér projected distributional Bellman operator is a contraction and, 2) QR loss and Cramér gradients are collinear under non-crossing constraints. This paper also designed a low-complexity algorithm for computing the Cramér distance and a centered non-crossing architecture to implement the proposed algorithms.
main_review: The idea of this paper is novel and not incremental. It proposed a new framework for modeling the distance between distribution in the distributional Bellman operator. Given that most of the recent algorithms for distributional RL are based on QR, this paper is a significant plus to this field. The major issue of this paper is that the author(s) generally can not convince me that the Cramér-based models can outperform the QR-based models. The proofs generally guarantee that CR-DQN can learn reasonable distributions of Q values because it resembles QR-DQN, but none of them can help readers understand where is the improvement. This paper lists some benefits like 1) CR enables removing a constraint (line 172-174) and 2) extreme quantiles are hard to estimate in the classic QR-DQN (line 186), but none of them is fully defined or evaluated. The motivation is vague from my point of view. The empirical results do not show a clear win of CNC-CR_QDN and the results are not explained.

Detail comments:

1) Line 185 "As shown by the classical theory of extreme values", could you explain more to help readers understand it? since it is a major motivation for your network architecture. By the way, figure 4 in [31](NC-QR_DQN) has claimed NC-QR_DQN converged faster than QR_DQN, which make me believe the reduction of training efficiency (from the "symmetries" (line 177)) in CR is significant and adding the CNC structure can not resolve this issue. It explains why CR-DQN is generally more sample inefficient than the QR-based methods. Your results in Figure 7 and the appendix generally confirm this problem.

2) The Atari 57 experiment should include both NC-QR-DQN and NC-CR-DQN as baselines since the author(s) proposed replacing the original non-crossing network with the CNC architecture. Comparing both NC-QR-DQN and NC-CR-DQN can provide a form of ablation study that helps readers understand the influence of each component (CR and CNC).


limitations_and_societal_impact: Societal impact: The authors believe their work has no negative societal impact, I generally agree that their negative impact is limited, but I think the goal of RL is to develop autonomous agents that can replace humans in many positions, for example, drivers. The author might want to discuss the influence of autonomous agents on the job market.

Limitations: The discussion of limitations should be expanded, especially about why a clear win is not observed in the Atari benchmark.
review_title: null

Meta-review: As the authors raised concerns with the reviews, I have thoroughly read the paper and even discussed the situation with the senior area chair. While the reviews are all against acceptance, my individual assessment is much more positive. Not to burry the lead, but, after discussions I will be suggesting that the paper not be accepted. 

I very strongly encourage the authors to resubmit. Rather than viewing this work as a competitor with NC-QR-DQN, I see this paper as providing some very interesting theoretical connections between all QR-based methods and the Cramer distance. The reviewers are, perhaps rightly, less than confident in the impact of this work due to the empirical results in the paper. This is likely amplified by the discrepancy between the published NC-QR-DQN results (with exploration bonus) and those in this paper (with epsilon-greedy, and also using the Cramer loss?). 

However, the other point reviewers have raised, which I agree is less easily dismissed, is that if these losses are essentially equivalent, then how can this have real impact beyond what we already can be done with pure QR? However, most QR-DQN-like methods do actually use the Huber quantiles, while this method is equivalent to a true (non-crossing) QR. Perhaps there is a unique advantage to the non-crossing quantiles in the pure QR case. This seems like something that could be checked by keeping as much the same as in this paper but using the QR loss (without Huber). I think that comparing with the actual NC-QR-DQN algorithm (with the Huber quantiles and architecture, but without the exploration bonus) would be an important addition. I would also suggest improving the clarity of Figures 1 & 2 (right panel), but view this as a fairly minor issue.

Although perhaps worth discussing, I do not think that NDQFN is a necessarily empirical comparison. Obviously reviewers may disagree, and thus to buffer your chances of acceptance it may be worth attempting nonetheless. I would have very much liked to accept this paper, but at the same time I do not think it is quite strong enough to justify completely disregarding all four reviewers feedback. However, please know that your concerns with the reviews have been noted.
Decision: Reject