Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors studied robust losses for learning value functions in reinforcement learning. The main contribution of this paper lies in the development of two novel robust loss functions for reinforcement learning and a saddle point reformulation based on the Huber and the absolute Belleman error and the biconjugates. 
main_review: Overall, I think this paper is interesting and is well motivated. My main concern is about its novelty. The authors highlighted that their main contribution is the introduction of two novel robust losses which are reformulated based on the least absolute loss and the Huber loss. This leads to the problem: why specifically the two losses? It seems to me that the comments and the analysis of this paper may also apply to other convex Lipschitz losses. It seems to me that it is the Lipschitz constant of the loss that "change the solution quality". I would expect more comments in this regard in the paper. 

In addition, as a minor comment, the presentation of the paper could be further improved. For instance, the abbreviations, MHBE and MABE, were used without being defined beforehand.    
summary_of_the_review: See above.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed using Huber Bellman error to robustify the loss function in learning value function and proposed using conjugate function to avoid double sampling. It also conducted experiments to justify its algorithm.
main_review: Strengths.  The main contribution of this paper is using Huber Bellman error instead of mean squared Bellman error, since using the conjugate function to solve double sampling has been proposed in [1].
Weakness. In the last paragraph of page 8, it said that it applied QRC without a target network. Would the performance of QRC become better and outperform QRC-Huber when combined with the target network?
summary_of_the_review: This paper is novel in the sense that it cooperates Huber loss with value evaluation to robustify value evaluation. It also conducts experiments to justify this improvement. Therefore, I think this paper is marginally above the acceptance threshold.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper starts with the premise that squared error minimization, despite its wide use, might not be the most effective option for learning value functions. The authors hypothesize that this might be because of squared error's emphasis on outlier states where the bellman error is large at the expense of accuracy on other states. To address this, they consider the absolute value and huber errors alongside the squared error objective and propose a saddle-point reformulation for these objectives that requires learning an auxiliary learned state function which, essentially, attempts to predict the residual error at each state.

Based on this motivation, starting with Section 4 the authors then make a connection of the proposed robust loss framework involving a learned auxiliary state function with prior work on algorithms for improving TD learning, namely, GTD2 and TDC.  Empirical analysis covers both prediction tasks as well as control tasks. In the prediction experiments, evaluations are conducted over carefully designed synthetic problems with linear function approximation in order to highlight particular challenges for each objective. These demonstrate that the Huber objective can often improve the prediction accuracy over squared error over a wide range of step sizes (learning rates).

For the empirical investigation on non-linear control problems, the authors consider a Huber-extension of the previously proposed QRC algorithm (which itself extends the TDC update to be considered in conjunction with the DQN target definition) and demonstrate that QRC-huber can improve on baselines in certain environments while being competitive elsewhere. These algorithms are also validated on a mini version of the Atari domain called Minatar. 
main_review: Strengths:
* A nicely executed paper that starts from a simple premise and connects several prior ideas together -- namely the TDC/GTD2 algorithms with the conjugate formulation of Bellman error minimization and uses this to derive a more robust versions of the aforementioned algorithms.
* Empirical results investigate both a simpler setup with linear prediction experiments as well as a more end to end non-linear control experiment to validate the practical benefits for policy optimization. 
* One of the novel conceptual contributions in this paper, appears to be making a connection between the auxiliary model $h$ used in the conjugate formulations of the Bellman error with the very differently motivated "secondary weights" model in GTD2 and TDC algorithms, where the additional model $h$ in both cases can be viewed as attempting to predict a residual error corresponding to the "primary weights" of the value function. 


Request for clarifications / other comments for improvement:
* For the empirical results on non-linear control problems (Figures 4, 5 and Table 1), it is somewhat unclear how much of a role is played by the auxiliary variable learning mechanism versus simply changing the loss from squared error to huber loss. For example, it would be interesting to see how a more naive baseline like semi-gradient update corresponding to the DQN + with Huber loss metric instead of DQN + squared error performs alongside the other three curves in each of the environments. 
* The main algorithmic proposal is based on the closely related prior works TDC, GTD2 and QRC, yet this aspect is somewhat buried until fairly late in the paper. 
* For the last term in Theorem 3.2, $p_\tau$ is applied to the vector $\mathcal{T} v - v$, but without specifying any reduction operation -- looking at the proof, it seems to me that there might be a need to add a max operation over the states to this bound. Could you please confirm? The combination of the multiplicative term involving a matrix inverse norm and the max operation over the states above makes this a potentially fairly weak bound in practice.
* Another case of using an auxiliary model that learns to predict the TD errors has also been proposed for improving learning in actor-critic models, e.g. "Characterizing the gap between AC and Policy gradient", Wen et. al., ICML 2021.


Minor comments
* In Equation (4), the notation for state seems inconsistent between t and t+1 across $s, S'$.
* In the equation for QRC-Huber updates for the auxiliary variable, there is some inconsistent notation across the references $\theta_h, \theta_{ht}, \theta_{h, t+1}$. Also, the variable $\beta$ in the secondary variable update needs a definition or reference of some sort.

summary_of_the_review: This is a nice paper that makes a novel connection between the secondary variable update in GTD2/TDC with the conjugate formulation of Bellman errors involving an auxiliary state function (both of which involve predicting the residual error with a separate model). While the technical contributions in Section 3 (e.g. Theorem 3.2) are not particularly significant, I believe the main value in the paper is the conceptual linking of two different lines of work to derive an improved algorithm over well motivated baselines. The empirical evaluation is well motivated and quite thorough, even if only for a limited set of benchmarks.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a mean Huber value error in the TD-learning. And the paper demonstrates the robustness under such loss.
main_review: The robustness is important in RL learning. The MHBE defined in this paper is inituitive. And the authors also develop reformulation to solve it in practive.
summary_of_the_review: The major contribution is to develop a new type of loss, which is not sufficient according to the ICLR standards. 

My main concern is that the contribution is not sufficient. I think the main contribution of this paper is the introduction of a robust loss. The conjugate reformulation is standard. And the bound developed in Theorem 3.2 is not surprising.

The writing is also not clear. For example, MHBE and MABE are never defined.
review_title: null

Meta-review: The paper proposes to use the Huber and absolute loss for value function estimation in reinforcement learning, and optimizes it by leveraging a recent primal-dual formulation by Dai et al. 

This is a controversial paper. On one hand, it is a well motivated idea to apply robust loss on RL; the paper implemented the idea well by leveraging the saddle point formulation, and empirically demonstrate its advantages in practice. 

On the other hand, the technical novelty of this paper is limited. The idea of Huber and standard conjugate formulation are straightforward application of existing techniques (despite being well motivated). 

The authors seem to think that there has been no application of Huber loss on RL. But existing implementations of RL already uses Huber loss. For example, in the openAI baselines (https://openai.com/blog/openai-baselines-dqn/), they said the following: 

"Double check your interpretations of papers: In the DQN Nature paper the authors write: “We also found it helpful to clip the error term from the update [...] to be between -1 and 1.”. There are two ways to interpret this statement — clip the objective, or clip the multiplicative term when computing gradient. The former seems more natural, but it causes the gradient to be zero on transitions with high error, which leads to suboptimal performance, as found in one DQN implementation. The latter is correct and has a simple mathematical interpretation — Huber Loss. You can spot bugs like these by checking that the gradients appear as you expect — this can be easily done within TensorFlow by using compute_gradients." 

The authors discussed the first approach above on in the rebuttal, but I am not sure if the authors have considered the second method. If not, it would be worthwhile to discuss and compare with it.  

See also "Agarwal et al. An Optimistic Perspective on Offline Reinforcement Learning" and "Dabney et al. Distributional Reinforcement Learning with Quantile Regression."

On the other hand, I have not seen the application of saddle point approach by primal-dual method of Dai on Huber specially. 

It seems that the proposed algorithm is in the end equivalent to MSBE+primal-dual+ (h with softmax output). If it is that simple, I think it would help the readers to explicitly point this out upfront in the beginning (which is an interesting conceptual connection).  Because the primal-dual approach need to be approximate h with a neural network, the difference of the two methods is vague in the primal-dual space. 

A side mark:  when we say "an objective for which we can obtain *unbiased* sample gradients", i think that the gradient estimator of the augmented Lagrange is unbiased; the gradient estimates of MHBE and MABE are still biased.  

Overall, it is a paper with a well motivated and valuable contribution, but limited in terms of technical depth and novelty.
Decision: Reject