Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes SPL-DQN, a quantile-based distributional RL method that uses monotonic rational-quadratic splines to approximate the quantile function of the cumulative return (starting from each state-action pair), and the features of using this technique are: (i) The approximated quantile function is ensured to be monotonic; (ii) The monotonic rational-quadratic splines are continuously differentiable and can provide a more flexible class of smooth approximators for the quantile function (compared to piecewise linear or step functions).
Moreover, the feature (i) resolves the quantile crossing problem of several existing distributional RL methods (e.g. QR-DQN), under which the quantiles output from the neural network may not be monotonic. By conducting experiments with a toy example, this paper also empirically discovers a few potential issues with the relevant prior works that enforce monotonicity (e.g., NC-QR-DQN and NDQFN). Experimental results in both tasks with discrete (Cartpole) and continuous action spaces (Roboschool) are provided to demonstrate the performance of SPL-DQN.
main_review: Originality: SPL-DQN provides an alternative way of parameterizing the quantile function with monotonicity in distributional RL. Specifically, using an architecture (feature extractor + logit network + bin scale network) similar to NC-QR-DQN, SPL-DQN substantiates the non-crossing quantile logit network by using the rational-quadratic splines.

While the idea of using rational-quadratic splines is interesting, I do find that the proposed method needs to be better motivated:
- The benefits of using monotonic rational-quadratic splines as quantile approximators need to be better justified. In Section 3.3, empirical comparison of the four QR-DQN-based methods is provided, and it is mentioned that NC-QR-DQN could suffer from underestimation and degenerate quantile functions (e.g., quantile function being a straight line) in some states, and NDQFN could suffer from overestimation. While I appreciate the above observations, it remains unclear to me whether these observations are indeed fundamental systematic issues caused by the algorithm design. More empirical evidence or qualitative explanation (e.g., which design leads to the observed estimation bias) is required to strengthen these arguments.
- On the other hand, I think a more detailed comparison between IQN and SPL-DQN is needed, given that IQN could already represent a general class of quantile functions without the quantile crossing problem. Is there any qualitative benefit of using SPL-DQN compared to IQN (besides comparing them vis-a-vis in the experiments)?


Significance: The experimental results show that combining SPL with DDPG improves over the existing benchmark distributional RL methods in multiple tasks of Roboschool. Having said that, in many tasks (e.g., Walker2D, Halfcheetah, Hopper, and Reacher) the improvement of SPL over other monotonic QR-DQN counterparts appears somewhat marginal compared to the benchmarking results (e.g., https://github.com/araffin/rl-baselines-zoo/blob/master/benchmark.md). One possibility is that such performance differences could partially result from the additional stochasticity added to the environment. To clarify this, it could be helpful to add other stronger baselines like TD3 and SAC as a reference.  

On the other hand, to strengthen the empirical significance of SPL-DQN per se (as SPL as a critic can perform quite differently compared to SPL-DQN), it would be really helpful to provide an empirical comparison in more complex environments with discrete actions (e.g., Atari or MinAtar) in spite of the issue of stochasticity in the first paragraph of Section 4. This could better validate if the original SPL-DQN is indeed strong compared to other distributional RL methods.

Clarity: The overall writing and organization of the paper are good. The proposed method is in most places well-explained. There are only a few cases where additional details are provided in the appendix.

summary_of_the_review: This paper proposes a new variant of QR-DQN with ensured monotonicity in the quantile function. Given that the quantile crossing issue is not new and the design of SPL-DQN closely follows NC-QR-DQN, the overall technical novelty is somewhat limited. Despite that SPL appears to outperform other variants of QR methods in some control tasks, the amount of improvement is not very significant compared to the benchmarking results. Therefore, I lean towards rejection for now but am willing to change my score if the authors address the concerns.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper studies the problem of distributional reinforcement learning. It augments the traditional quantile-based algorithms with monotonic rational-quadratic splines. Such augmentation provides a natural solution to the quantile crossing issue, which exists for many other quantile-based algorithms. Extensive experiment results are performed to verify the effectiveness of the proposed method.
main_review: The paper is well written and clearly presented. Related works are surveyed in details. Although some existing works have already solved the quantile crossing problem, the paper carefully discusses the difference of such methods and the proposed algorithm, thereby justifying its novelty. In empirical evaluation, some of the standard experimental environment have been modified with randomness injected, and the paper provides clear motivation for such modification (more suitable for evaluating distributional RL algorithms). The extensive experiment results show that the proposed algorithm outperforms the existing ones in most of the test cases and therefore justify its effectiveness beyond novelty.
summary_of_the_review: The paper proposes a novel algorithm with strong empirical results.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to use monotonic rational-quadratic splines to interpolate an inverse cdf function for distributional RL. They proposed a neural architecture to implement such interpolation and achieved a simplified yet effective solution to the quantile crossing problem in distributional RL. 
main_review: **Novelty**: The idea of using monotonic rational-quadratic splines to interpolate an inverse cdf function is novel and interesting. I enjoyed reading this paper. 

**Correctness**: The methods are sound and derivations are correct. 

**Technicity**: Though the idea is novel, the technicity in the paper is quite low as the paper applied existing techniques to the quantile-crossing problem and combined several existing elements. The most significant technicity in the paper is perhaps the neural architecture to model the monotonic rational-quadratic splines, though this modelling is quite natural. 

**Clarity**: The paper is well written and organized. One comment for improving the clarity is that in defining $f_k$ in Eq. (11), it is more clearer to say that $h \in [x_k, x_{k+1}]$, and replacing $h$ in the RHS of Eq. (11) by $(h - x_k) / (x_{k+1} - x_k)$. The reason is that in the paragraph after Eq. (14), the paper talked about computing $f(\hat{\tau_i})$ and searching for the bin of $\hat{\tau_i}$. In the original form, $\hat{\tau_i} = h(x)$ which is not possible to search for a bin with this representation. 

**Experimental significance**: The paper demonstrated that their method, when used in actor-critic framework for continuous control, better approximates the inversed CDF and better performance than the other distributional RL methods in most experiments. What I am concerned about is that why didn't the authors test their algorithm on Atari as most distributional RL methods did, so that the comparison is more standard? Of course, the Atari testbed has its own problem as the paper also discussed but it is a nice testbed for distributional RL because only valued-based RL methods are compared, this performance is more reflective of whether the value estimation is good. In the actor-critic framework, the final performance also depends on the actor; thus if SPL+actor > IQN + actor, it is not clear if SPL alone is still better than IQN alone in value-based RL setting. 
summary_of_the_review: The paper makes a novel contribution in terms of using monotonic rational-quadratic splines to interpolate an inverse cdf function for distributional RL. The empirical result is very promising. I suggest weak acceptance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new neural network design to represent quantile functions for distributional reinforcement learning, based on smooth rational-quadratic splines. This representation has the advantage of being continuously differentiable.
The loss is computed by evaluating the quantile loss on a set of uniformly spaced quantile levels.
The benefits are shown empirically on standard continuous control environments, modified with noise to make them stochastic, by using the proposed distributional representation as critic in DDPG/SAC algorithms.  
main_review: Strengths:
- the proposed parameterization is continuously differentiable, in contrast to previous piecewise linear approaches
- paper is clearly written and well positioned with respect to prior work
- experimental results show significant improvements over previous approaches in many environments with added noise to assess robustness 

Weaknesses:
- novelty is low with respect to NDQFN, which uses piecewise linear interpolation instead
- it is a bit disappointing that it ends up considering a fixed set of quantile levels for the loss computation, which makes the whole approach equivalent to a fancy parameterization of a fixed level quantile network 
- it lacks some kind of analysis of the origin of the improvement over NDQFN. 


### Detailed comments:
It is claimed that this approach "offers greater accuracy in terms of quantile approximation" and is "a more general and precise approximation for quantile functions" .
These claims seem to be based on the visual assessment of Figure 2. I think they should be more strongly supported with some approximation result comparing splines to piecewise linear functions, with an equivalent number of degrees of freedom.
However, it is not clear, how a better capability to approximate an arbitrary function is useful since only the quantile values at the fixed quantile levels matter when optimizing the quantile loss.
Regarding the improvement over NDQFN: How much comes from differentiability ? How much comes from the representation itself (splines vs piecewise linear)? How much comes from the different choice of quantile levels  (fixed in SPL-DQN, and uniformly sampled in NDQFN)?  Also, a greater number of trainable parameters in SPL could also explain the increased performance. 
Also, why isn't  SPL always superior (cf. Reacher experiment)?


### Minor issues/comments:
Eq (4) : large parentheses should be used

"During training, ${\cal OU}(\mu';\sigma')$ noise..." : this notation should be introduced

The combination with DDPG/SAC should be described (at least in the appendix).

summary_of_the_review: The method is a natural "upgrade" of NDQFN and thus its novelty is relatively low. 
The experimental results on continuous control environments with noise show better results in most environments.
However, it lacks a real analysis of the causes of this improvement.

For these reasons, I tend to vote for rejection in this current form.


**================== After rebuttal ==================**

Although I find the spline approach and the experimental results interesting, 
I still feel that the paper lacks theory to explain the improvement and, also, that novelty is relatively low.
My concern regarding the fixed quantile levels used for training has been addressed with the uniform sampling and the new experiments.

Therefore, I increase my score to 6.
review_title: null

Meta-review: The paper proposes monotonic splines as an improvement on current approaches to parametrising quantiles in distributional RL. The idea is an obvious, natural improvement on what exists, and yields improved experimental results.
Decision: Accept (Poster)