Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: #######################################################################

Summary:

In this paper the authors propose a new RL method, spectral DQN, in which rewards are decomposed into different frequencies. This decomposition allow for the training loss to better balanced on certain tasks - in particular those with progressive rewards. The new method is shown to perform well on specially constructed tasks with extreme reward progressively, as well as on a selection of standard Atari tasks.

#######################################################################

Reasons for score:

I think a weak accept is appropriate here. I think the authors correctly identify a class of worth while tasks - namely those with progressive rewards, and reasonably establish that standard approaches struggle here. The new method is a strong implementation of a simple idea which is demonstrated to work, and does not require significant fine-tuning.

#######################################################################Pros:

1. The paper is well written and clearly presented. I'd commend the authors on their exposition and balanced motivation throughout 

2. I think this is an interesting direction of work more generally - exploring the performance of different methods against different reward distributions, and having agents predict quantities more flexible than the mean return. This would seem to be a contribution to that body of work.

3. I see no reason that the constructed domain ExponentialPong shouldn't join the benchmark for any subsequent general-purpose agent 🙂

#######################################################################

Cons:

1. While I believe the selected experiments are sufficient to demonstrate the claims in the paper, they do not fully explore the capabilities of the method and the intuitions that motivate it. It would have been good to see some more thinking here (even if it means experiments outside of Atari)

(1) It would seem that this approach would work for a parametrizable class of reward functions - why not test that? Perhaps ExponentialPong with reward $b^(\alpha N)$ evaluated on a grid for $b$ and $\alpha$?

(2) Similarly for failure modes

(3) Frequencies needn't be geometric, how might other choices have performed

2. I'd like to have seen a slightly richer discussion/motivation of the mixed Monte Carlo update. Clearly something is necessary here, but why this? What else was tried?

#######################################################################

Questions during rebuttal period:

Q1: you note the obvious limitation in the selection of the number of reward frequencies, but as you note this is reasonably easily ameliorated in practice. A more interesting question perhaps is whether an adaptive approach for b might work - was this tried?

#######################################################################

Some typos:

(1) 'Expoential Pong' → 'Exponential Pong', just above 4.1'

(2) Fig2, last row, "Fiilled proportion" → 'Filed proportion'
review_title: Review of 'Adapting to reward progressivity via spectral RL'

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes an extension to DQN, more generally applicable to value-based deep RL systems, that encodes the return using a thermometer encoding with exponentially-sized bins. This enables returns of vastly differing magnitudes to be learned without hurting performance. The authors propose an algorithm for learning these encode returns, including the use of a variance scaling term to speed up learning.

Overall, I enjoyed reading the paper and appreciated the clear exposition, which is sensible throughout. My main concern is whether this is solving a real problem, or a hypothetical one. The experiments don't support the former case, while I would argue that solving the problem hypothetically would require a more thorough development. For example, if progressivity is an issue, what form do we expect it to take? The two synthetic examples, Ponglantis and Exponential Pong, make for a fun case but not necessarily a realistic one. Training on all Atari games with a single network, for example, might be a better case. 

I have a few technical questions for the authors:

4.1: Why use a thermometer encoding versus a binary encoding? I.e., write the reward in binary, and encode its bits. 

4.2: Why not use base gamma instead of base 2? That seems like a more natural way to encode returns.

4.3: "impossible to train over full, infinite spectrum" -- given that returns are bounded for gamma < 1, what do you mean? It seems like you need log(VMax) bins at most.

5: "We do not simply set ..." I would have expected some ablation studies here, regarding the role of the parameter. The choice w_i = 1/sigma_i seems particularly ad-hoc. In particular, I'm disappointed that you did not include the experiment w_i = 1. It seems natural to me that exponential weights should fail, whereas it's not clear that w_i = 1 should, and the use of sigma complicates things (and adds a moving part).

The results of Fig. 4 are not terribly exciting. Sometimes spectral DQN works better, sometimes it does not. Can you comment on this? Should we be concerned? Also, you should include published DQN results (and if possible, other value-based methods) as I would expect these to be worse for games that have rewards of different magnitudes. 

8: "the phenomenon of reward progressivity". Is the issue really progressivity? Or is it that value-based learners struggle to deal with rewards of varying magnitude (the progression doesn't matter)?

Distributional RL algorithms replace the L2 loss with probabilistic losses. For example, the C51 algorithm uses the KL divergence. That could help deal with reward scale also, by creating a loss that is insensitive to scale. In particular, I can imagine the binary cross-entropy loss making sense since most of your outputs are binary. 


Minor points

- Condition on gamma < 1/L_h L_h-1 -- doesn't that mean that Lh must be almost 1? Not much slack. 
- Intro, 2nd paragraph. A concrete example would help the reader here.
- Table 1 might want to be left justified, as it's a little jarring to read.


==== Updated review

In light of the authors' revisions, I'm happy to raise my score to 6. I think the paper is better, although I still wish the presentation was more compelling -- as given, this seems more like an exercise than a contribution with a demonstrated impact. On the other hand, this level of contribution seems relatively on par with e.g. other deep RL papers.

Regarding the revisions, I would encourage the authors to integrate them with the main text. For example, Figure 3 really wants the weight=1 result (maybe as two separate panels -- comparison to other algorithms, a); ablation on weights, b)).
review_title: An intriguing idea, well-developed, somewhat lacking a use case

final_score: rating: 7: Good paper, accept
review_content: ## Summary

This paper details the problems that might arise in value-based reinforcement learning methods in domains where reward progressivity is present. To show that current methods do not handle reward progressivity, the authors introduce two domains, _Exponential Pong_ and _Ponglantis_.

After showing that current methods do not work well in these domains, the paper then goes on to propose a solution, spectral decomposition of rewards. The paper shows that returns learned separately on decomposed rewards can be composed to get the original return. The paper then presents spectral Q-learning and spectral DQN, with experiments on the domains presented earlier as well as experiments on 6 Atari games.

On the two domains where progressive rewards were shown to be problematic, Spectral DQN is shown to work better than current approaches. On Atari games, spectral does as well as current approaches, doing better in 3 out of 6 domains.

## Positives
+ The setting of progressive rewards is interesting.
+ The domains proposed for testing these rewards are clear.
+ The spectral reward decomposition is a simple idea and is explained well.
+ The effectiveness of spectral DQN on the two domains introduced in the paper is clear.
+ Experimental details are clear and additional steps taken to stabilize learning are included.

## Negatives/ Questions
- One question that does not seem to be satisfactorily addressed is whether there is a commonly used benchmark domain or one which was not specifically engineered for progressive rewards where value based deep reinforcement learning would fail or not perform well unless it was using Spectral DQN.
- The spectral DQN objective (eqn. 8) includes target compression. It is unclear how much benefit the spectral decomposition is having in the presence of target compression. While it is fine to require target compression for the full benefit of spectral DQN, it would be good to see an ablation of how well spectral DQN does without target compression.
- Another useful ablation could be to remove the monte carlo mixing and show how unstable the updates get.
- How does the van Seijen et al. paper on logarithmic mappings (Using a Logarithmic Mapping to Enable Lower Discount Factors in Reinforcement Learning) compare to the related work? It is motivated by a different problem, but since a possible logarithmic mapping might mitigate the problems that come up with progressive rewards their method might be a possible solution to be compared against.

## Other Comments
* Figure 2 has a typo (Fiilled instead of filled).

## Summary
Overall, I find the idea in this paper clear, simple, and effective. There are some additional questions and comments that if addressed would make the paper a more well-rounded submission.


review_title: Review for Spectral RL

final_score: rating: 7: Good paper, accept
review_content: This work describes and addresses the issue of _reward progressivity_ in reinforcement learning, where as the task progresses the scale of the reward changes. The authors argue that reward progressivity harms Q-learning when training signals arising from large rewards interfere with those arising from smaller rewards. They propose a form of reward decomposition with an analogous modification to the Q-network output, which together help to ensure that training losses from small and large rewards are similarly scaled. The authors present a handful of experimental results demonstrating that their proposed method outperforms two other reward re-scaling baselines when reward progressivity is an issue and maintains good performance in more standard tasks.
  
  
### Clarity
This paper is exceptionally clear. There are a couple areas where techincal details are somewhat missing: namely, their implementation of the mixed Monte Carlo update. Otherwise, the ideas are very clearly presented and the authors are careful to spell out caveats and details, which will improve reproducibility and downstream adaptation.

### Originality and Quality
_Spectral DQN_ is, to my knowledge, original. However, comparison to other forms of reward decomposition is lacking, both experimentally and in the discussion. The authors mention several examples of works that enable non-scalar representation of rewards for handling variability but there is not much attention given to how  these prior works are different and/or why they are not included as baselines.
  
_Reward progressivity_ as a motivating issue is also, to my knowledge, original. Indeed, the authors point out the 2 main baselines do not consider _intra-task_ reward variability. This brings up one of my main issues with the paper: there is very little demonstration that reward progressivity is the disruptive force that the authors describe it as. The authors show that baselines suffer/fail on tasks that are engineered towards this problem, but this doesn't offer enough to convince a sceptical reader to believe the paper's underlying intuitions. My understanding of these intuitions is that reward progressivity causes RL to stagnate because it biases the Q-network to be accurate around large rewards at the expense of being accurate around low rewards -- the issue being that this latter inaccuracy leads to poor policies in important but low-reward regions of the task.
   
If this is indeed the intuition, then it would be very useful to actually demonstrate some sort of accuracy trade-off induced by reward progressivity. This would also provide a useful lens through which to validate Spectral DQN, by perhaps being able to characterize how it balances Q-network accuracy for different reward scales compared to other baselines.
  
It would also be valuable to discuss whether reward progressivity is ultimately an issue of discounting. The authors seem to touch on this when describing the _Mrs. Pacman_ results, but a bit more exposition would be useful.
 
There are numerous other examples where the authors add interpretation around a particular result to help characterize where Spectral DQN does/doesn't succeed and what a practitioner might want to look out for.These interpretations improve the paper's quality somewhat, but the level of analysis is ultimately lacking. For instance, the authors offer speculation over the results for _Centipede_ and _Bowling_ but don't back up this speculation with any meaningful analysis. I would not want to punish the authors for offering these insights (that commentary is valuable and honest) but I do think the paper would have been better served by spending more time on analysis to back up a couple key insights.
  
### Significance
If we take the authors' claims at face value (which I am still inclined to do even though the paper lacks more detailed validation), this paper presents a simple technique to handle reward progressivity. This contribution is significant from a practical standpoint but perhaps only when solving tasks with this particular reward structure. For this to have broader signficance, it would be necessary for the authors to provide a more detailed characterization of the potential pitfalls associated with the technique's heuristics. Again, these are discussed (which is appreciated) but only speculatively.


**Pros**
- Adds a simple technique for balancing losses associated with rewards of different
 scales.
- Follows a straightforward intuition.
- Overcomes extreme examples of the problem it is designed to address.
- No obvious, systematic drawbacks of the proposed modification.
- Exceptionally clear and well-written.

**Cons**
- Analysis is limited to a handful of training curves with only one minor ablation (in Figure 3)
- The paper tends to emphasize speculation over more concrete demonstration.
- Unclear whether the motivating problem presents a significant obstacle.

### Suggestions for improvement
I would be happy to increase my score if the authors are able to address my criticisms. In particular, I think an analysis demonstrating the how learning dynamics are negatively impacted by reward progressivity (via its measurable effects on the Q-network) would go a long way. In addition, a more thorough ablation study and/or sensitivity analysis would add very useful clarity around the impact of, for example, the mixed Monte Carlo update strategy.


### Post-discussion feedback
Thank you for the valuable discussion and revisions. I believe the paper has improved and I've updated my score to reflect that.

The new analysis in Figure 4 and its surrounding discussion offer some useful insights. I think the authors could still improve this analysis a bit in a final version of the paper (if only just to make the plots a bit easier to parse visually), but the TD-error instability of the baselines seems reasonably clear and I'm inclined to agree with how the authors frame this as an example of the problem they solve. In addition, the added discussion around discounting is also valuable; and the simplification of the algorithm and added ablation experiments improve the quality of the contributions.
review_title: Simple, effective, and intuitive -- but lacking analysis

Meta-review: This paper presents a deep RL algorithm to handle tasks where rewards can differ greatly in magnitude.  The proposed solution decomposes the reward into a set of exponentially sized bins with a thermometer encoding, and computes a weighted sum of the value functions learned for each bin.  The approach addresses the common tactic of reward clipping and value rescaling in deep RL algorithms.  The experiments demonstrate the potential utility of this approach on artificially constructed Atari games, and the experiments also show the approach remains competitive on six standard Atari games.  

The reviewers found both strengths and weaknesses in the paper.  The overall approach was viewed as a clear and sensible (R1, R2, R4) approach to handling widely varying reward scales in a domain.  It may be a useful contribution in a manner similar to other methods that make deep RL algorithms more robust to scaling issues encountered in practice (R4).  The main concerns were whether this was solving a real problem or not (R2, R3), and the lack of a theoretical development for the multiple heuristics (R2,R3).  

The author response then simplified the algorithm, which also served to clarify which aspects of the algorithm were relevant to the performance improvements. The response removed some of the heuristics (mixing in Monte Carlo returns) and changed other choices to be more principled ($1/\sigma^2$). The author response also described how the proposed algorithm addressed different scaling concerns from those handled by earlier methods. The author response also provided clarifications to many minor questions raised by the reviewers.  In the ensuing discussion, the reviewers were happy with the revised paper.  Though some minor theoretical reservations remained, the reviewers agreed this paper was a useful contribution.

The reviewers indicate to accept the paper as a useful contribution in deep RL to address certain reward scaling issues.  The paper is therefore accepted. 
Decision: Accept (Poster)