Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper explores an alternative loss function for fitting critic in Reinforcement Learning. Instead of using the standard mean squared loss between critic predictions and value estimates, the authors propose to use a loss function that also incorporates a variance term. The authors dub the approach AVEC. The authors combine their approach with popular RL algorithms such as SAC and PPO and evaluated on the standard benchmarks for continuous control.

Although the paper demonstrates interesting empirical results, I think that the current experimental evaluation has a number of flaws that prevent me from recommending this paper for acceptance. The paper provides basic motivation but it is lacking thorough theoretical investigation of the phenomena. Also the proposed loss is biased in the stochastic mini batch optimization due to the expectation under the squared term that is not addressed in the paper either. Finally, I have major concerns regarding the experimental evaluation. The set of OpenAI mujoco tasks is different from commonly used tasks in literature. In particular, Hopper and Walker2d, which are used in the vast majority of the literature, are ignored in table 1 and figure 2. This fact raises major concerns regarding generality of the approach.

In conclusion, the paper presents interesting results on some tasks for continuous control. However, the paper requires more thorough experimental evaluation to confirm the statements. Also a deeper theoretical analysis will greatly benefit this work. I strongly encourage the authors to continuous working this approach and revise the paper to improve the theoretical and empirical analysis. This paper presents a very interesting idea but in the current form it is not ready for acceptance.
review_title: Interesting approach but not sufficient empirical and theoretical evidence to confirm the effectiveness of the approach

final_score: rating: 7: Good paper, accept
review_content: ### Strengths

The paper proposes a simple and elegant idea for changing the value function objectives in deep RL and demonstrates reasonable empirical evidence of it's potential usefulness.  The authors also provide a clearly articulated intuitive motivation and provide experiments to support the proposal.  The idea complements several other algorithms and is therefore quite widely applicable (and easy to try). The analysis of the experiments is also quite interesting and clearly presented. 

### Weaknesses

The paper is mostly well written and has interesting theoretical insights as well as empirical analysis. Here are a some weaknesses.

* The theoretical justification for the variance reduction while technically correct, seems like it should be miniscule in theory. For the $T$ independent RV case being analyzed, the condition required for the improvement is that $\Delta  \triangleq 2 \mathbb{V}(X_i) - \frac{1}{T} \sum_{j=1}^T \mathbb{V}(X_j) > 0$, which seems reasonable unless the sample in question is an outlier with a very small variance to begin with. However, the overall reduction itself has another $\frac{1}{T}$ scaling, i.e. the variance reduction over the squared error case is equal to $\frac{\Delta}{T}$, which seems to be vanishingly small as the number of samples $T$ is large even if $\Delta \gg 0$. Note that for the situation where this core idea is being applied, the parameter $T$ is approximately, the number of samples in the expectation over $(s, a)$, which is large in practice.
* The improvements are a good sanity check, but somewhat marginal in many cases (especially given the error bars).

### Additional comments/feedback

* In Section 4.2 paragraph on State-value function estimation line 3, should the targets be $\widehat{V}^\pi$ rather than $V^\pi$?
* In Figure 1, some additional detail on the claims seems necessary (e.g. what parameterization is being considered?)
* In the discussion below the specification for $\mathcal{L}^1_{AVEC}, \mathcal{L}^2_{AVEC}$, the authors say "the reader may have noticed that these equations  slightly differ from Eq. 3", but I am not able to see what difference is being alluded to.
* Figure 4 looks quite surprising in terms of the large qualitative difference between the baseline and AVEC-baseline graphs. Just to be sure, do you measure the fit with respect to $f_\phi$ or the bias corrected version, $g_\phi$? (obviously, the latter makes more sense?). 
* The Ablation study in Section 5.4 seems intriguing, but what the conclusions imply seems unclear. It appears the authors were expecting to see some non-zero value of $\alpha$ to improve over $\alpha=0$ (AVEC), but this isn't the case? Some additional clarification here would be useful. Also, it is a bit confusing to separate the plots into two depending on whether the weighting is less than one; as I'm guessing the exact same plot is used for the non-alpha versions in each pair of these graphs?
* In Figure 5, the distance to the true value function seems to be relatively flat (or even mildly increasing) through the entire horizon in both graphs. Is this simply due to the resolution, as I'd expect there to be a drop at least in the initial phase over time.





review_title: A simple and widely applicable alternative to the squared loss objectives in RL with some evidence of empirical benefits.

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper presents AVEC, a new critic loss for model-free actor-critic Reinforcement Learning algorithms. The AVEC loss can be used with any actor-critic algorithm, with PPO, TRPO and SAC being evaluated in the paper. The loss builds on the mean-squared-error, and adds a term that minimizes $E_s [f_{\\phi}(s) - \\hat{V}^{\\pi_{\\theta_k}}(s) ]$. The addition of that extra term is motivated by recent research on the stability of actor-critic algorithms, and the benefits obtained by the AVEC loss are empirically demonstrated in numerous environments, with AVEC+PPO, AVEC+SAC and AVEC+TRPO.

Quality: the paper presents an interesting idea, that is simple but well-motivated, and leads to encouraging empirical results. Both the theoretical and empirical motivations are strong.

Clarity: the paper flows well and is quite clear. However, an intuition for what the added term in the AVEC loss is missing. Section 4.2 motivates the added term in a mathematical way, but a few sentences explaining what the added term does, in simple terms, may help the readers understand why AVEC is a better loss than simple MSE.

Originality: the contribution of this paper seems original. It builds on recent work, but the recent work identifies problems while this paper offers an original solution to these problems.

Significance: the fact that AVEC provides good empirical results, and can be used as the critic loss of any actor-critic Reinforcement Learning algorithm, points at the high significance of this work. Many actor-critic implementations can easily be improved by using the AVEC loss. Another positive point is that the paper discusses how to implement the AVEC loss in algorithms that fit a neural network on batches of samples. This really helps implementing the proposed loss, that contains an expectation in an expectation and is therefore not trivial to properly implement.

In general, I like this paper and recommend acceptance.

A few questions/issues:

- An explicit mention of the gradient of the loss, or at least a discussion of where to stop back-propagating gradients, would have been interesting. $f_{\phi}$ appears two times in the AVEC loss, and it is unclear whether the loss contributes to gradients in $f_{\phi}$ two times, or if the expectation over states is first computed (without computing any gradients), and then used as a constant in the rest of the evaluation of the loss.
- As mentioned in "clarity", an intuition of what the added term of the AVEC loss does, especially since it is "inserted" in the mean-squared-error (inside the square), would help the less mathematics-savvy readers. It is not crucial to understand the paper, but the generality of the approach proposed in the paper may lead it to be used often by students, and so an intuition of why AVEC works and what it does would greatly help.

Author response: the authors clarified my questions, so I maintain my recommendation for acceptance.
review_title: New critic loss with good theoretical and empirical motivations

Meta-review: This paper is accepted, however, it could be much stronger by addressing the concerns below.

The theoretical analysis of the proposed methods is weak.
* As far as I can tell, the proposition has more to do with the compatible feature assumption than their method. Furthermore the compatible feature assumption is very strong and not satisfied in any of their experiments.
* Sec 4.2 does not provide strong support for their method. R2 points out issues with their statements about variance and the next subsection argues from an overly simplistic diagram.

The experimental results are promising, however, R3 brought up important issues in the private discussion:
* Their implementation of SAC systematically produces results worse than reported in the original paper (they use a version of SAC with automatically tuned temperature https://arxiv.org/pdf/1812.05905.pdf); 1a) Their SAC gets average returns of 2.5k at 500k steps while the original implementation gets 3k at 500k steps; 1b) Their SAC on HalfCheetah 10k at 1M steps, original paper - 11k at 1M steps; 1c) The same applies to Humanoid, there is no improvement with respect to the original SAC;
* Their approach degrades performance on Hopper. 
* They use non-standard hyper parameters for SAC. 0.98 instead of 0.99 for the discount and 0.01 instead of 0.005 for the soft target updates. That might be the main reason why their SAC works worse than the original implementation. 
* The authors use the hyper-parameters suggested for HalfCheetahBulletEnv for all continuous control tasks. For HalfCheetah, however, the authors of the stable-baselines repository (which this paper uses) suggest to use the hyper parameters from the original SAC paper (https://github.com/araffin/rl-baselines-zoo/blob/master/hyperparams/sac.yml#L48). Nonetheless, the results for the unmodified SAC reported in this work for HalfCheetah/Hopper/Walker/Ant are subpar to the original results, suggesting that the hyper-parameters for HalfCheetahBulletEnv are suboptimal for these tasks.

Given the simplicity of the change and the promising experimental results (with some caveats), I believe the community will find this paper interesting and will lead to followup work that can patch the theoretical gaps.
Decision: Accept (Poster)