Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a new method called GenTD for off-policy evaluation of GVFs. The algorithm can further estimate multiple intercorrelated GVFs at once under some reasonable (causal) assumptions. The contributions are mainly theoretical (convergence proofs). Some empirical evaluation on a small MDP is provided.
strengths_and_weaknesses: # Strengths
Overall, the paper tackles an interesting problem, identifies a shortcoming of previous algorithms and proposes a solution with theoretical guarantees. I found it overall pretty clear.

# Weaknesses

## Scalability
I have some concerns about the scalability of the method and whether it could extent beyond the linear regime. In particular, Alg 1 requires some projections step that may not make sense in the non-linear regime. Could authors comment on whether they think it is an issue?

## Experiments
The experimental validation is weak in my opinion. I understand this is a theory paper and the main contribution is the algorithm and its guarantees, however I believe it is also in the authors interest to showcase that the algorithm performs well on environments more complex. In particular, GenDICE which this work builds upon had some empirical evaluation on non-trivial gridworlds as well as a simple control (Half-Cheetah) environment.
As far as I know, methods based on min-max problems (DualDICE etc) are known to be quite unstable, so I think it is important that you showcase the robustness of your algorithm.

questions: - L259 could you comment on the choice and importance of psi?
- L260, you assume the matrix A to be non-singular, however if psi' is intended to be P_pi \Psi, then shouldn't this matrix be singular because it contains I - P_\pi which is singular?
- I am puzzled by the strength of the convergence results you have using only A1 and A2. For instance, why doesn't it suffer from deadly-triad-like issues? I had expected to see some assumptions on the spectrum of some matrices related to phi, P and D.

Minor:
- What is R_\theta L241? I may have missed the definition
- I may also have missed the definition/intuition behind m(x) L251
- I may also have missed the definition of the F-norm L296
limitations: Limitations are mentioned above. Beyond the questions on the theory, the limitations appear most on the practical side. In the paper the empirical evaluation may be lacking and thus making it harder to convince the community that this algorithm would be a safe and efficient one. More broadly, it is not clear to me how this algorithm would be extended to the non-linear setting.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The problem formulation this paper is concerned with is the General Value Function (GVF) approximation task, where the algorithm's task is to estimate a vector which is defined as an expectation of some algebraic manipulation of the signals received. This vector is named a GVF.  On this task, the authors mainly compared their approach with another existing method called Gradient Temporal Difference (GTD). The main observation of this paper is that the goal of GTD is to minimize the empirical Mean Squared Projected Bellman Error (MSPBE), whereas a more legit goal would be the expected mean-squared projected Bellman error which they name as the Mean Squared Projected General Bellman Error (MSPGBE). To account for the shift from the empirical (data-dependent) mean to the expected mean, they applied an existing density ratio estimation method. 
strengths_and_weaknesses: Strengths:\
    Originality: The motivation is legit.\
    Quality: The logic chain works through, it is clear what they are doing. \
    Clarity: The story is easy to read. (I didn't check any proof.)\


Weakness:\
    Significance: \
        1. Seems A+B, where A = GTD, B = density ratio estimation.\
        2. More precisely, I put this as a weakness because personally, I don't quite get the logic why learning density ratio would be more accurate. The authors put as a counter-example where GTD fails for Complete F_\Phi. However, if for the same example, the density ratio is not learned very well, why would GenTD work?
   

questions: I understand that I might be asking a question that might have been addressed by other papers, but again, why is density ratio the right thing to estimate? There are other types of divergences too, is there something special about RL that makes density ratio easy to estimate?

Could you explain why for the counter-example you proposed for GTD, why would GenTD work? Or is the counter-example merely evidence that GTD is not perfect?
limitations: I guess assumptions count for limitations. A better limitation from my point of view would be to specify for which interesting classes of RL tasks, the algorithm GenTD wouldn't work.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper considers evaluating multiple interrelated general value functions using offline data. A generalized TD learning algorithm is developed and its theoretical properties are analyzied in detail. The strengths and weaknesses of the paper are given in the next section. The contributions of the paper are mainly theoretical and include (1) establishment of the contraction property for both forward and backward value functions; (2) development of a generalized TD learning algorithm to overcome the limitations of previous baseline methods; (3) convergence analysis of the proposed algorithm; (4) identification of sufficient conditions under which the proposed algorithm would converge. 
strengths_and_weaknesses: Strengths:

1. The paper evaluates both forward and backward general value functions. Despite the richness of the literature on off-policy evaluation, these general value functions are less studied. In that sense, the paper targets an interesting research problem. 
2. The proposed method considers multiple interrelated general value functions jointly rather than on a case-by-case basis. 
3. A generalized TD learning algorithm is proposed to address the limitations of the gradient TD methods.
4. Convergence of the model parameter and the estimated value functions are investigated in detail.

Some comments: 

1. There is a typo in the title of the pdf file. In addition, there is a question mark in the checklist. 
2. The linearity assumption is quite strong. The algorithm might perform poorly in nonlinear systems with high-dimensional state information.
3. It remains unclear to me if the variance of "the reward to go" (mentioned on L23) can be represented in the form of the general value function. In particular, the variance of the cumulative reward would involve the interaction term that measures the covariance of the rewards at different time points. 
4. Can you allow $B_j$ to be unobservable? The gradient of the value function (mentioned on L26) does not seem to be an observable quantity. It might be more useful to cover cases where $B_j$ needs to be estimated from the data as well. 
5. The literature on off-policy evaluation with a general scalar value function is not thoroughly reviewed. In addition, it was mentioned on P2, L92 that these methods are not directly comparable. However, I do not agree with this argument. Suppose the initial state-action distribution concentrates on a particular state-action pair, then the value function is reduced to the state-action value. Combined with a kernel-type estimator, existing OPE methods such as (marginal) importance sampling, double robust estimation (double reinforcement learning) can be potentially applied to this setting. It remains unclear to me whether the proposed estimator is better. 
6. The numerical study is oversimplified. It considers a simple toy example with 7 states and 2 actions. More extensive simulations based on e.g., OpenAI gym environments are needed to test the empirical performance of the proposed algorithm.
questions: Please refer to **Strengths and Weaknesses**
limitations: The main limitation includes the linearity assumption as well as a lack of extensive empirical studies. The former was partly discussed in the discussion where nonlinear function approximation was mentioned.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a new off-policy algorithm to evaluate forward and backward general value functions that have the property of causal filtering. They show that the existing algorithm (GTD) fails in this (for example, failing to compute the ground truth GVF) and propose an algorithm to overcome its shortcomings.

Thanks to the authors for putting in the effort in doing this work!

strengths_and_weaknesses: Strengths:
- I think this is an important area to research. GVFs are relatively under-researched and it's important to see this type of work.
- The 1) use of the concrete example on line 140 was helpful to communicate the idea of GVFs; 2) the comparisons between GenTD and GTD was also nice.

Weaknesses:
- Given that the concept of GVFs is unique and somewhat niche, I think the authors should be clearer on what GVFs exactly are (see suggestions below).
- The experimental section seems rather limited. The cited paper on GVFs General value function networks, 2021, seems to have a pretty detailed experimental section. It would be interesting to see how the proposed algorithm scales to harder domains.

questions: Questions:
- I’m a bit wary of the dependence that GenTD has on the function class being expressive enough given that performance degrades a lot if this assumption is not met (CFT vs. INCFT). Three subpoints: a) Even in the case presented in Section 5 (the CFT case), they don’t seem to achieve perfectly 0 error in the left graph of Figure 1. I’d imagine that in this relatively simplified setting, zero error could be achieved. b) The plots of INCFT suggest that performance drastically deteriorates when the assumption of expressiveness of the function class does not hold, which is a bit concerning since we wouldn't want an algorithm that has such a strong dependence on this function class expressiveness assumption. If performance degraded only slightly, then it would be another thing. c) It may also be the case that the CFT example used in the experiments is an extreme worst case. If that is so, then I recommend the authors vary the degree of failure of the assumption holding and see how sensitive GenTD is at different levels of the assumption holding true.

Suggestions:
- Title has a spelling mistake “evaluatio”
- For Figure 1, could the y-axis labels be made more clearer. The labeling of only 0 and 30 doesnt make it clear what exactly the errors are.
- The concept of GVF is relatively niche and so it must be made clear what GVFs actually are, for example in the intro. Its difficult to follow the intention of the work and what is exactly trying to be solved until Section 2 is reached.


limitations: No, they do not address it, but this seems to be more fundamental work with no direct societal impact.
review_title: null

Meta-review: metareview: The paper proposes a new algorithm called GenTD for the estimation of multiple general value functions (predictive and retrospective) from off-policy data. The paper shows convergence guarantees for this algorithm to the ground truth for a certain class of general value functions with causal filtering. 

The initial reviews were mixed. On the positive side, the reviewers found the writing to be clear overall, found the studied problem important and appreciated the theoretical results. On the negative side, several reviewers voiced concerns regarding the experimental evaluation. Other concerns are the limitation of the linear setting and possible extensions to the non-linear setting as well as the significance, specifically, whether this work is merely a combination GTD and density ratio estimation. The authors' response could alleviate these concerns, further clarifying the contributions of the paper as well as adding additional experimental results. After the discussion with the authors, all reviewers view the paper positively and the AC agrees. All in all, this paper is recommended to be accepted.
confidence: Less certain
Decision: Accept