Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new RL algorithm that contains two principles of value function mapping and reward decomposition. This proposal generalizes many existing RL frameworks such as classical Q-learning, Logarithmic Q-learning, and Q-Decomposition. The paper also provides generic theoretical results to backup the theory. The paper also demonstrates this idea on the suite of Atari 2600 games. 
main_review: The strength: 

- New general value mapping that generalizes from previous work e.g. Log Q-learning
- The orchestration of value mappings and decomnposed reward that can allow the above general value mapping. 
- Theoretical and experimental results to backup the proposed idea

Weakness:

- Contribution on value mapping and reward decomposition can be incremental.
- Experiment results on average are still worse than Rainbow.

In overall, the paper is well written and pursues an interesting research problem. Though the proposed idea of value mapping and reward decomposition is incremental given existing work, it's worth trying and has showed the benefit.

The reward decomposition that is based on fixed and hand-designed configuration will limit the novelty and application. It's too technical comparing to the contributions in Sections 2 and 4.

It will be more useful if there are more choices of $f$. The current evaluation looks limited. Though there is improvement over LogDQN, it's unclear how and where the proposed ideas contribute to the score improvement. More ablation studies will also be helpful.
summary_of_the_review: See above
=====================
After rebuttal: The author response has addressed my concerns. I am happy to increase the overall rating.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a framework for estimating the Q-value function by decomposing the reward into a linear combination of reward signals or channels. These individual channels are then mapped into a new space, similarly to the log Q-learning approach of van Seijen et al (2019). However, here the framework is extended to a more general class of function (convex, etc). Additionally, some of the theoretical assumptions in the original log Q-learning paper are softened. The method (with a particular instantiation of reward mapping) is tested on the Atari benchmark, and demonstrates improved performance compared to DQN and log Q-learning.

main_review: The main contribution of the paper is a framework for mapping multiple reward channels to a "nicer" space in parallel. I find this general concept and framework to be both interesting and novel. I also thought the paper was well presented: the ideas are clearly described and easy to follow. The framework does build upon the work of van Seijen et al (2019), which somewhat hinders the novelty rating, but overall I believe this to be a useful contribution to the literature. In particular, I could see future work that seeks to discover good mappings or pick from a library of potential functions, and this paper provides the platform for that.

Other positives include the softening of theoretical assumptions, which means we can get away with using a fixed learning rate (and have the other handled by Adam etc). This removes the need for yet another hyperparameter and will make implementing this framework (or even the original log Q learning work) easier. While I did not spend too much time on the appendix, I also liked the inclusion of Figure 3 in lemma 2, which made it slightly easier to follow than if it were just writing or mathematical notation. 

The only downside for me to this paper is the misalignment between the theory and the experimental results. The theory and discussion talk at length about the advantages of various mappings. This is done with the Pac-Man example, as well as in the section "Slope of Mappings". While these discussions are interesting and highly relevant, the experiment itself only serves to show that the chosen mapping outperforms DQN and log Q learning across the Atari tasks. And while a reason for that performance is provided, this behaviour is never demonstrated experimentally. Put another way, the experiments could conceivably have come from *any* paper that improves upon DQN --- they don't really speak to the power or specifics of the framework here. 

I realise that space is an issue, but perhaps the Pacman example can be removed to make more room for an additional experiment? In particular, I could imagine something like a toy domain setup that has a top down view of the reward function and the effect that various choices of mappings have on it, as well as the resulting behaviour when the value functions are learned in the mapped space compared to the original reward function. Obviously, this is a very rough idea, but the main thing I'm looking for here would be to provide empirical support for some of the claims, such as "As a result, when learning on a game which often has a large return, LogDQN operates mostly on areas of $f$ where the slope is small, and it can incur significant error compared to normal DQN."

Minor comments:

1. The second last paragraph in Section 4.2 that talks about boundedness. Do you mean here that if we know $r_{min}$ and $r_{max}$, then we know that the max return is $\dfrac{r_{max}}{1 - \gamma}$ and so we can use this to clip the values if necessary?
2. How are c and d selected in Section 5?
3. Bottom of page 3: "bellow"

summary_of_the_review: The paper proposes an interesting extension of prior work that offers great potential for expansion in future work. I'm very positive about the framework and theoretical results, but slightly down on the empirical ones, since they do not speak to the particulars of the framework.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper describes a generic class of algorithms that decompose the reward signal into multiple channels and also map the value function into another generic space via arbitrary functions.  They argue such a class of function is useful to specify certain properties of the learned value on a specific reward signal, also decomposed into channels. They show that known algorithms from the literature are instances of this convergent class of methods.
main_review: This paper applies the observations from [1] to describe a general algorithm that separates the Q-learning update into two steps: (i) averaging due to environment stochasticity, and (ii)  averaging over different states and actions, moving the latter into the other space. The paper presents these in a clear and simple way, which is very nice. It further formalizes these principles and incorporates them in a generic class of algorithms. It is interesting to see instances of these class in known existing algorithms. The paper is motivated from the view of small agent, big world needing to learn about many things at the same time, which is a promising avenue for autonomous agents. The paper is fairly clear, though it could benefit from some improvements in terms of structure. For instance, the section presenting the first idea describes a proof for a theorem it has not yet been stated, so it is a bit confusing. The authors could maybe move this discussion further down, after the theorem is presented. The assumption of having access to the minimum and maximum return is a bit alarming, how would one have access to those in an \emph{unknown} environment, this seems to imply that the algorithm is not applicable in the same way to all instances of the same problem. The paper feels a bit disconnected, as one idea is presented after the next, and they are presented in a disconnected manner. The ideas are only at the end connected into an algorithm. In the reward decomposition section, there is a mention "SARSA-like update" as common knowledge, without introducing it. The reward decomposition is linear. Why? What are the limitations or assumptions that this is making, and why do we think that channels with separate properties will emerge in the reward space, such that they can be mapped to value functions with different properties, beyond the game-like artificial environments? Where do the values of the weightings applied to the reward channels come from ($\lambda_i$)? The section discussing the "slope of mappings" makes insightful observations, but is a bit hard to follow, and would greatly benefit an illustration. It is only here that the reader is explained why "Assumption 2" was introduced all the way at the beginning, along with a lay word interpretation. The experimental section does not explain the results. What is $d$, where does it come from?
summary_of_the_review: The paper presents an interesting idea of viewing value decomposition by mapping in a different space after separating the reward functions. Although some parts of the paper still leave open questions, most is clear and very simply written. This framework appears to be useful at least in the Atari domain, but the specifics of the experiment and interpretation of results would benefit some more clarity. The paper contains interesting insights.
review_title: null

Meta-review: Description of paper content:

The paper provides a framework to develop a family of algorithms that decompose rewards into linear combinations of several reward channels. The value functions per channel are estimated in a new space using an invertible function transformation, f. The framework encompasses several previously published algorithms, including Log Q-Learning. Conditions are provided for acceptable choices of f. Convergence to the optimal Q function in the tabular case is proven for a special learning update.

Summary of paper discussion:

All review scores were above the acceptance threshold. Overall, the reviewers found the idea interesting, the theoretical results satisfying, and the writing and presentation clear. Initial concern about the directedness of the experiments in showing the usefulness of this particular theoretical framework to explain performance improvements was allayed when some of the results in the paper (e.g. reward density in Atari Skiing) were re-emphasized. Generally, all reviewers felt that this was a nice, thorough contribution with the demerit that the framework lacked “a killer application” experimentally.
Decision: Accept (Poster)