Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper tries to argue that the Bellman error cannot be trusted as a good metric or objective for off-policy evaluation. To support the argument, the paper provides theoretical analysis as well as extensive empirical analysis. Theoretical analysis gives a few example on the relation between the true value error and the Bellman error, which could be arbitrary especially under finite sample cases. Empirical results show that even when the Bellman residual is minimized well, the value error could be large in various of settings, in comparison, FQE consistently outperform BRM even when the Bellman error is much larger.
main_review: ## Strong Points
Significance: I agree with the authors that the empirical demonstration of comparing Bellman error (training loss) vs value error (testing loss) is less considered in off-policy evaluation community and should receive higher emphasis. This paper may bring potential attention to the community to rethink of how to view Bellman error into the design of the algorithm.

Counterexample Construction: The paper construct several simple MDP as counterexample that the Bellman error is not with aligned with the value error.

Empirical evaluation: The empirical evaluation of this paper is extensive and provides useful insight. Reader can save their time to try BRM on TD3 (which is extensively studied in this paper and is failed)

## Weak Points
Novelty: The direct use of BRM fails to solve the off-policy evaluation has been widely known in the community. The main conclusion in this paper tries to emphasis on the fail of generalization of the Bellman error because of the finite dataset. However, I believe there are a lot of off-policy evaluation papers, especially those with high confidence interval estimation papers are trying to take the finite sample into account. Without any model assumption on either the value function, dynamics, rewards or density function, no algorithm can learn the region we have not observed and the value error can be arbitrary (like what you said in the example in Fig. 1) and therefore we need to make reasonable assumption in order to generalize the existing region's information. If we can assume the value function to be in a linear class, or smooth or with a known Lipschitz constant we can definitely be able to generalize (check theoretical papers for off-policy evaluation).

Quality: I like the motivation of the paper, but when I go deep into the main body of the paper, I do not find too many new things to me and feel disappointed. I would expect a new algorithm can be provided to alleviate the difference between Bellman error and the value error by either under reasonable assumption or a new way of using the Bellman information. Another thing I would expect the explanation of the comparable success of FQE. Pro. 3 looks weak to me and only said that if the sign is the same, we can expect improvement. however, since the target network $Q_{\bar \theta}$ is also randomly initialized the same as $Q_\theta$, we don't know much about the sign. Another point is that in real practice the target network can also updated in each iteration with a slower rate, so we cannot think this is similar to a fixed regression version in Eq. (13).
summary_of_the_review: In sum, I think this is an interesting paper and the topic worth further investigating. But so far the paper is not ready enough for the bar of the top conference like ICLR. I would vote for rejection at this moment and encourage the authors to further improve on the idea and submit to another venue.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies the Bellman equation commonly used in reinforcement learning (RL) algorithms. The typical motivation behind using the Bellman equation to design RL objectives is that uniformly driving the Bellman error to 0 implies that the true value function has been learned, i.e., the value error is 0. However, as this paper demonstrates both theoretically and empirically, a number of issues may arise before this point in practical regimes. For example, Bellman error and value error may not be well correlated, as shown via experiments evaluating minimizing the Bellman error directly versus fitted Q-evaluation. Furthermore, when evaluated using off-policy data, Bellman error may not even be well correlated with value error for different runs of the same algorithm. These findings may be useful for a more complete understanding of when and why RL algorithms break down, especially in paradigms such as offline RL.
main_review: Strengths
---
The paper is of relatively high quality. Experiments are carried out with a clear purpose and the methodology seems sound.

The paper is well-written and well-structured. Theoretical results are presented cleanly and with useful exposition. I did not carefully check through the proofs in the appendix. Experimental results are also presented well and generally easy to follow.

And, as mentioned previously, these findings may be useful for a more complete understanding of when and why RL algorithms break down, especially in paradigms such as offline RL.


Weaknesses
---
I am not an expert in this area, so I am unsure as to whether all of the results (the theoretical results in particular) are fully novel or if prior work has discovered similar or roughly the same results. Additional discussion of this in the related works section would be appreciated. In particular, it seems like results showing that Bellman error can be optimized without optimizing the true objective have probably appeared in prior work, though again, I am not sure.

A potentially relevant line of recent work may be Q-learning and dynamic programming approaches to offline RL. [1] may serve as a useful survey here.

Some greater analysis of the experiments would also be generally helpful. In Fig 2, are there potentially some conclusions missing due to incomplete training, e.g., some of the learning curves still exhibit a downward trend at 1 million steps? And explanations for some of the outliers, e.g., FQE HalfCheetah value error and BRM Hopper Bellman error, could prove insightful. A similar comment holds for Fig 3.

More vaguely, the tables and Fig 4 are harder to follow. Adding just a couple of sentences in the main text to more explicitly describe the tables and Fig 4 would likely fix this.

Even more vaguely, but perhaps also most importantly, I still have lingering concerns regarding the significance of this work. Some examples of how this work may be significant are: the work provides insight for how to devise a better RL algorithm, or the work sheds light on why some prior RL algorithms work better than others. Currently in the text, I do not see clear arguments for either of these or other examples of what may be the immediate impact of this work. Should the authors have additional thoughts here regarding how the work may influence additional work (or provide insight into prior work), that would be appreciated.


[1] Levine et al, "Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems." arXiv 2005.01643.
summary_of_the_review: Primarily due to concerns about significance, I am initially recommending a weak accept of this paper. I am happy to engage in discussion with the authors and other reviewers in order to reach a more confident final recommendation.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper examines the role of Bellman error as an objective function in offline reinforcement learning. The Bellman error measures how “closely” the estimated value function $Q$ satisfies the Bellman equation, while the value error measures how close the estimated value function $Q$ is to the true value function $Q^\pi$. In offline RL and OPE, we’re interested in value error, but since the true value function is unknown, common practice is to look at Bellman error. 

The paper provides both theoretical analysis, and empirical experiments using two algorithms related to Bellman error (BRM and FQE). The main takeaways are: (i) zero Bellman error implies zero value error, but (ii) low Bellman error doesn’t imply low value error, making it a poor objective to optimize and a poor metric to compare using, (iii) the success of practical iterative value-based methods like FQE that optimizes Bellman error-like losses relies on generalization during training to reliably predict values for unseen state-action pairs. 

main_review: ### **Strengths**
+ Exposition is clear and easy to follow. 
+ The paper includes theoretical analyses that are simple and to the point, and makes good use of illustrative examples. 
+ Empirical experiments test various settings to support the main claims, and results are nicely summarized in figures/tables. 

### **Weaknesses**
- Could you comment on the connection between the notion of “generalizability during training” and “realizability/completeness of function approximator” [1]
- There are two ways Bellman error can be used: (i) as the loss function to optimize, and (ii) as a metric to compare two different value functions. The authors provided a good review of (i) in Sec 5, which is more on the learning aspect. More recently, researchers looking at the model selection aspect, which is more related to (ii), have made similar observations that the TD error loss does not correlate well with policy value/performance [2][3][4]. Therefore, I think adding a discussion on model selection can strengthen the work and potentially resonate with more researchers. 


### **References**
1. Jinglin Chen, Nan Jiang. Information-Theoretic Considerations in Batch Reinforcement Learning. ICML 2019. https://arxiv.org/abs/1905.00360
2. Tom Le Paine et al. Hyperparameter Selection for Offline Reinforcement Learning. 2020. https://arxiv.org/abs/2007.09055
3. Shengpu Tang, Jenna Wiens. Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settings. MLHC 2021. https://arxiv.org/abs/2107.11003
4. Justin Fu et al. Benchmarks for Deep Off-Policy Evaluation. ICLR 2021. https://arxiv.org/abs/2103.16596

summary_of_the_review: This paper brings renewed attention to the role of Bellman error in offline RL, beyond the more widely known double sampling issue. In my opinion, Bellman error (or relatedly, TD error) is commonly thought of as the equivalent of *loss function* in supervised learning but for offline RL, and such analysis is much needed for the community and a good step at understanding the empirical success of deep RL and its fundamental difference compared to supervised learning. I would be strongly in favor of acceptance if the authors can aptly discuss how this contribution relates to the past work provided above.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies the relation between the Bellman equation and the accuracy of the value functions. This paper shows that Bellman error is not a good measure for comparing value functions and thus is an ineffective objective function for off-policy evaluation.

main_review: Strength:
The message sent by this paper is very interesting and important. This paper challenges one broadly accepted method of using Bellman error as the objective function. It proves that minimizing empirical Bellman error can be far from minimizing the value error, especially for the off-policy evaluation problems. Their theoretical results are corroborated by experimental evidence.

Weakness:
The theoretical results of the paper are not deep enough It basically only gives some existence result which is good but not helpful enough. It actually feels a little bit obvious.
Besides, the case considered in the paper are limited. It only considers the discounted MDP setting, while ignoring many other important settings such as finite-horizon MDPs and goal-oriented MDP (i.e. shortest path). Therefore I wonder if similar results can hold for a broader class of models/problems.

summary_of_the_review: Despite the weakness mentioned above, I still think this paper is good. The message is important and the results are novel and interesting.

My current recommendation is (weak) accept.
review_title: null

Meta-review: This paper studies whether the Bellman error is a good metric to reflect the quality of value function estimation, focusing on finite-sample off-policy data sets. Both theoretical analyses and empirical experiments have been provided, showing that the Bellman error is often not the right metric to consider. However, while I appreciate the authors' theoretical attempts, the current theoretical contributions are not deep/significant enough. As the reviewers mentioned, the failure of the direct use of BRM is not surprising given the insufficiency of data (namely, no algorithm can make predictions on completely unseen regions unless further modeling structure is present). The authors might want to further strengthen their theory along this important direction.
Decision: Reject