Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper provides a framework to integrate value decomposition into a broad class of actor-critic algorithms, and applied it to SAC to derive the SAC-D algorithm. Lots of experiments are conduct to verify SAC-D's performance over SAC. This paper introduces the influence metric, and demonstrated its use in measuring each reward component's effect on an agent's decisions. Finally, they provides several examples of how value decomposition can diagnose and remedy agent learning problems.
strengths_and_weaknesses: Originality:
The operation of integrating value decomposition into reinforcement learning algorithms seems to be common sense, which has been proposed by many existing literature. And the performance of value decomposition has been explored in a large body of literature. The work of this paper seems to be just another confirmation of previous findings. Therefore, this paper is less original.

Quality:
This paper is well written and easy to understand. But, I have two concerns as follows. (1) Since SAC-D is developed by incremental, heuristic refinement  to SAC, it is not clear whether the performance gain obtained by this refinement is limited to a few experiments in this paper. (2) The iterative design of RL agents shown in Section 6 is quite interesting. However, it still seems to require strong task-specific priors from experts.

Clarity:
This paper is well written and easy to understand.

Significance:
The work is a bit significant, but does not give new insights into value decomposition.
questions: (1) Since SAC-D is developed by incremental, heuristic refinement  to SAC, it is not clear whether the performance gain obtained by this refinement is limited to a few experiments in this paper. 
(2) Value decomposition in this paper is artificially divided. How does value decomposition with different divisions affect the effect of the algorithm?
limitations: Yes.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes *value decomposition* for diagnosing and iteratively designing actor-critic algorithms. The key idea of value decomposition is to treat the reward signal as constituted by a number of distinct components that together yield value. However, rather than just take some simple operation of these distinct signals, value decomposition proposes to learn a $Q$ function for each separate component of the reward: a Q-network, then, outputs $m$ predictions (for $m$ reward components) rather than just $1$. This relatively simple idea is explored as a mechanism for diagnosing a variety of characteristics of an RL algorithm, and ultimately give rise to the design and improvement of new algorithms. As the paper points out, the core idea of value decomposition is not new, and has been applied to Q-learning variants in the past. Indeed, value decomposition is conceptually similar to typical linear assumptions on the reward or value, but encourages algorithm design to explicitly learn these distinct value components. Moreover, to my knowledge, this is the first extension to actor critic algorithms: the paper proposes SAC-D, a version of Soft Actor-Critic with value decomposition in the mix. Section 4 presents findings from experiments contrasting various forms of SAC-D in different continuous Gym environments, such as Bipedal walker or Ant. 

Arguably the most significant aspect of value decomposition is that it unlocks new perspectives for diagnosing agent failures, and for iteratively improving the design of these agents in light of these failures. Section 5 presents a series of experiments and insights showcasing how dropping out one of the reward components can impact aspects of the agent. Figure 2, for example, illustrates the the influence each reward component exerts on the Q function through contour maps over different mixes of the reward components $Q_1, Q_2$ and $Q_3$, while Figure 3 explores the _fractional influence_ of each component. I found Figure 3b to be particularly illuminating. These insights culminate in Section 6 that provides a pragmatic view on how to incrementally improve agent design using the diagnostic tool of value decomposition. For instance, in Lunar Lander, certain reward components should be negative, while others should be non-negative. Observing that an agent is incorrectly predicting certain values to be outside their possible bound allows for incremental adjustment of the agent. 
strengths_and_weaknesses: [STRENGTHS]

Overall, this paper proposes a novel perspective for diagnosing agent failures and iteratively improving agents as a result. As the introduction of the paper points out, this new viewpoint directly engages with the practical difficulties of deploying RL agents: they often fail for unknown reasons, and staring at a monolithic neural network and chaotic learning curves can often be daunting to debug and react to. The method of value decomposition is simple, easy to understand, and can be broadly applicable. The paper is well written and takes care in motivating many of its claims. It also simply lays out its core contributions, and limitations. The experiments are broad, interesting, and reveal the considerable potential behind the proposed method.

I believe this paper possesses many virtues, and is sure to be of interest to the community. For this reason, pending any issues uncovered by other reviewers, I recommend accepting this paper.

[WEAKNESSES]

There are several small things that could improve the paper, but none are major. First, the scope of the paper is quite ambitious, but I believe ultimately that sections 5 and 6 and the perspectives therein really constitute the core contributions. In this sense, it is not clear the new algorithm SAC-D or the experiments in Figure 1 are strictly necessary to the core of the work. A mild suggestion, but perhaps they can be moved to the appendix. Otherwise, I have a few minor writing suggestions and typos to fix (see below).

Writing Suggestions:
- In Algorithm 1, consider using the \texttt{} wrapper for the boolean "use_cagrad"
- I believe in Equation 2b $Q_i$ should be changed to $Q_j$.
- The pseudocode for Algorithm 1 might be cleaned up. For instance, I find the added text to detract from the readability of the algorithm. There are a few too many comments as well.
- Another minor point, but I found the y-axis of Figure 1b to be counter intuitive. The first few times I read this plot, I assumed the "Normalized Expected Reward" in Figure 1a was applying its y-axis to both plots. I wonder if there is some visual adjustment to be made that can prevent this collision.
- Small point of consistency: the subfigure labels differ throughout the paper in style.
questions: Q1: At first I found the extension to SAC quite natural, but reading it through a few times, I find it less clear why SAC is needed for the rest of the story. To me the main claim of the work is that value decomposition can be a powerful diagnostic tool for understanding agent limitations, and fixing them. This perspective applies just as easily to the existing Q-learning approaches to value decomposition. So, I am left wondering: why is SAC the focus, rather than Q-learning? Another way to put it: do you anticipate that the pragmatic suggestions from Section 5 and 6 extend to Q-learning based value decomposition as well?

Q2: Do you anticipate the linearity of the reward composition to be problematic or overly limiting?

Q3: One reaction I have to the work is that linear functions (and approaches focused on linearity) in general are praised for their interpretability. In this way, the idea of value decomposition is very nearly just making a linearity assumption. How, in general, should we think about this approach as different from linear RL, or just assuming that the reward admits a linear decomposition with some unknown weights as in Jin et al. (2020)?

Jin, Chi, et al. "Provably efficient reinforcement learning with linear function approximation." Conference on Learning Theory. PMLR, 2020.
limitations: N/A
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a tool to diagnose and design the reward function for training the reinforcement learning (RL) agent. In practice, the reward function in RL consists of multiple terms (e.g., collision penalty and velocity bonus). The proposed method assumes the reward is the linear combination of these terms and learn a Q-function for each reward term. Such a decomposition enables the reward function designer to inspect the influence of each reward on the agent's decision, thus reducing the efforts of tuning the reward function without guidance. Several diagnosis case studies showcase how RL practitioners use this tool to design the reward function.
strengths_and_weaknesses: Originality:

Though value decomposition is not a new idea, using value decomposition to guide the reward function design and the proposed influence metric are an original contribution, to my best knowledge. 

Quality and clarity:

This paper is easy-to-follow. The experiments are well-designed.

Significance:

The proposed reward design scheme is important in RL.
questions: - The current formulation needs the reward function to be a linear combination of multiple rewards. Can the author comment possibility of extending the proposed method to non-linear rewards?
- In section 6, the results of some diagnosis (e.g., weight scheduling, markov features) can be applied to SAC without value decomposition. Have the authors tried applying weight scheduling and markov features to SAC without value decomposition? This might help answer whether the diagnosis results can be applied back to non-decomposed version. 
limitations: The author acknowledged the limitations in the paper.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: The paper proposes to use value decomposition for the critic of generic actor-critic (AC) methods, and then applies it to soft-actor-critic (SAC) as a particular case. The idea of value decomposition is not new and has been studied in various prior work over past 20 years or so. Thus, the actual contribution of the paper is simply to use value decomposition in the context of AC methods. 
strengths_and_weaknesses: Strengths:
Nothing in particular.

Weaknesses:
- As the main idea of the paper is quite simple, I expected a neat mathematical analysis. For example, what is the impact of value decomposition on AC methods and why they should be used? When and how they change bias/variance of the base AC method? Does using value decomposition influence convergence? etc...

- Re-weighting the reward components may make the algorithm completely unstable and even non-convergent. A clear analysis is required, beyond the few presented experimental results. 

- The presentation may also be improved. This paper mostly looks like writing a report around some experimental results. 

- Minor: some missing key references for value decomposition.
questions: - The paper lacks any thorough analysis of possible issues and meaningful solutions with formal ground. This is essential. 

- L127-128: "Because the component weights are factored out of the component Q-functions, they may be varied without changing the component prediction target." 

This is not correct. First, changing the weights will change Q, hence, it may change j at line 4 of Algorithm 1. Second, changing Q induces a change of policy. Notice that your value learning is on-policy and changing Q can beaks that because the policy that collects the transition becomes different from the learning policy. At the very least this can make the learning process unstable/slow depending on how much and how fast the weights are changing. This is not a trivial issue. Note also that if Q-learning is used instead of SARSA to make it off-policy, then the entire algorithm will NOT converge to optimality due to Jensen’s inequality. See reference [3] bellow for a detailed analysis. 

- L186: Not clear why norm is used (also not clear what norm it is). Is your action-space multi dimensional? If yes, you may need to describe it properly in your section 2.1.

Some references:

[1] van Seijen, H., Fatemi, M., Romoff, J. & Laroche, R. Separation of Concerns in Reinforcement Learning. arXiv [cs.LG] (2016)

[2] Fatemi, M. & Tavakoli, A. Orchestrated Value Mapping for Reinforcement Learning. in International Conference on Learning Representations (2022).

[3] Laroche, R., Fatemi, M., Romoff, J. & van Seijen, H. Multi-Advisor Reinforcement Learning. arXiv [cs.LG] (2017)

limitations: The authors counted four limitations, which I agree with in general. However, my rejection of the paper is not grounded on these limitations.
review_title: null

Meta-review: metareview: The reviewers carefully analyzed this work and agreed that the topics investigated in this paper are important and relevant to the field. However, reviewers expressed different opinions on the merits and contributions of this work. 

One reviewer pointed out that similar ideas have been explored before and that the paper does not necessarily give new insights into value decomposition. The authors counter-argued by saying that the proposed method is the first concrete value-decomposition framework (with these particular properties) to be proposed in the literature. 

Another reviewer had a more pessimistic view of the merits of this work. They argued that the paper lacks novelty and simply applies the idea of value decomposition to a different setting—in the context of actor-critic (AC) methods. They also argued that they expected a more formal discussion about the impact of value decomposition on AC methods and on whether re-weighting reward components could cause instability. The authors responded that off-policy DRL algorithms typically do not have convergence guarantees and that their goal was orthogonal: to provide more insight into the design of agents. The authors also argued that standard convergence guarantees apply if the reward function stops changing. Ultimately, this reviewer's two main points of contention were: 
1. From their point of view, methods that help implement agents via iterative design could, in principle, be interesting for discussion in other venues, such as a workshop paper, but that this level of contribution does not meet the bar for a full conference paper; and
2. Having dynamic weights could cause the method to become unstable and the paper did not study whether this could be a problem in practice. 

Two other reviewers, by contrast, expressed very positive views on this work. 

One reviewer argued that even though the general idea has been explored before, using it to guide the reward function design is an original contribution. They also argued that the experiments were well-designed. After reading the authors' rebuttal, this particular reviewer said that the authors addressed all their technical questions. 

Another reviewer expressed similarly strong positive views on this paper, arguing that:
1. It explores an important direction that ultimately leads to the design and improvement of new algorithms. 
2. Even if the underlying idea is not new, this is the first extension of that idea to actor-critic algorithms. 
3. The paper proposes a novel perspective for diagnosing agent failures and iteratively improving agents, and that this new viewpoint directly engages with the practical difficulties of deploying RL agents. 
4. This reviewer further supported their positive view of this paper by stating that the experiments are broad, interesting, and reveal the considerable potential behind the proposed method. For these reasons, they believe this paper will likely be of general interest to the community.

Ultimately, the main point of contention between the opinion of these two reviewers seems to result from their opposite views on what, precisely, is the contribution of this work:
- One reviewer sees this paper as one that merely applies the idea of value decomposition of actor-critic methods, without providing thorough theoretical and formal analyses. This reviewer argued that this contribution does not meet the bar for publication as a full conference paper.
- The other reviewer views this paper as one that introduces novel diagnostic tools for understanding learning agents. They argued that they found the cleanliness around the proposed diagnostic tools to be appealing and that the resulting proposal for iterative design, based on these diagnostics, is broadly useful for RL. This reviewer highlighted three kinds of diagnostic unlocked by the perspective introduced in the paper: **(1)** identifying insufficient state features; **(2)** value prediction errors; and **(3)** reward and exploration. They believe these avenues are new and useful when viewed from the perspective of iteratively identifying and correcting failure modes of agents. Ultimately, they disagreed with the reviewer above by arguing that they did not find the attachment to actor-critic algorithms all that critical.

Overall, thus, it seems like there is disagreement regarding the merits of this paper primarily due to two reviewers not agreeing on whether the proposed iterative design method is (in principle) a sufficient contribution to a conference paper. Regarding this matter, a fourth reviewer expressed a positive view of this work by stating that the questions studied here address an important aspect missing in the literature: how to diagnose agents. They argued that—given this paper's contributions towards that goal—the lack of formal analyses should not be considered a critical reason for rejection.

All reviewers encourage the authors to update their work based on their constructive criticisms and, in particular, in a way that tackles the points of contention mentioned in the original reviews and their post-rebuttal comments.
confidence: Less certain
Decision: Accept