Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper proposes a new algorithm to the off-policy evaluation problem in reinforcement learning, based on stationary state visitation. The proposed algorithm does not need to know the behavior policy probabilities, which provide a broader application scenario compared with previous work. It is based on directly solving the fixed point problem of the resulting target stationary distribution, instead of solving a fixed point problem about the stationary ratios.

In general, this paper is looking at an interesting problem with a lot of increasing focus these days. I think this paper contributes great idea and good theoretical analysis. I like the idea of matching the resulting target distribution instead of minimizing a loss over the ratio. However, several unclear places in the current version hurt the presentation of results. I would like to see them get improved and will increase my score accordingly if so.

Detailed comments:

1) The algorithm part could be presented more clearly. So far I did not see where the empirical operator \hat{B} is formally defined. The word *empirical* is also confusing to me in "B is approximated by empirical data" because B is not an expectation, but an *integral* which has no *empirical* opposite of it. For the equations on top of page 5, shouldn't they be k[, ] about empirical operator instead of the expected operator since the RHS is already in sample-based form?

2) Related with the last one, B has an integral. To approximate the integral, we only have one sample from the transition probability actually, and the sample state is not uniformly distributed. It needs some explanation of why that would not cause a problem to approximate the integral.

3) The current loss function is invariant to the scale of w at all. Since the w is normalized, this is not a problem for the resulting estimator, ideally. However, that can be a numerical issue for float numbers. It's possible that the output from function approximator w goes to 0 or \infty. Both cases can lead to NaN of the output/function approximator update eventually. I'd like to hear if the author has met this problem in the experiment or not, and how can that be fixed. 

4) I have to point out, as just a slight con of this paper, the technique used in this paper is not that much different from Liu et al 2018. Since it minimizes a loss function which is a supremum over an RKHS, and the resulting empirical loss also has a similar form. It's nice to see the author provide some details of making it work with mini-batch. These details are important for function approximator as NN.

Minor point:
 - On page 12 the equations after "We have by the definition of D_k", I did not follow the second step of the equations.

Suggestion:
This paper study the similar settings (behavior-agnostic OPE), using similar method (on the stationary distribution) came out several months ago: https://arxiv.org/pdf/1906.04733.pdf. I knew it's unfair to ask the author to compare it with a very recent prior/parallel work. However since they are in such a similar case, and they have code available, is it possible to directly compare with the result from their code? https://github.com/google-research/google-research/tree/master/dual_dice


======= After rebuttal =======
The author's feedback clarified some of my concerns in the initial review. After reading the author's feedback and other reviews, I think this paper has enough show contribution to the related work. Some of my previous concerns (point 2 and 3) seems true for many related works in this area in general. I partly agree that it is not very fair to ask this paper to fix them. The updated version also presents the theory section in a more clear way. So I'd like to raise my score. I've no problem with acceptance, but I won't fully heart argue for acceptance.



review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: Summary : 

This paper proposes an approach for long horizon off-policy evaluation, for the case where the behaviour policy may not be known. The main idea of the paper is to propose an estimator for off-policy evaluation, that computes the ratio of the stationary state distributions (similar to recent previous work Liu et al.,) but when the behaviour policy may be unknown. This is an important contribution since in most practical off-policy settings, the behaviour policy used to collect the data may not necessarily be known. 


Comments and Questions : 

	- This is an important and novel contribution where the authors propose an approach for OPE that comes down to solving a fixed point operator. This is simlar to solving for fixed point Bellman operators, but where their problem can be formulated as a backward recursive problem. 
	- The operator defined as in equation 3 is in terms of the backward view of time - this allows the operator to capture the visitation from the previous s,a to the current s. This is the backward flow operator with which a fixed point equation can be described. Although the authors note that similar operators have appeared in the literature before - their main contribution is in terms of using such operators for the OPE problem, which seems novel to me and is an interesting approach with potential benefits as demonstrated in this paper. 
	- The core idea comes from equation 9 which tries to minimize the discrepancy between the empirical distribution and the stationary distribution. This can be formulated as an optimization problem, and the paper uses blackbox estimators, as described in section 4.2 for solving this problem.
	- The next interesting part of the paper comes from solving the optimization problem in equation 9 with Maximum Mean Discrepancy (MMD) - this is a popular approach that has recently become well-known, and the authors make use of it minimize the differences between the empirical and the stationary state distribution. 
	- Section 4.2 appears a bit confusiing to me with some details missing - it would perhaps be useful if the authors could include more details for 4.2, especially with some explanations of how they arrived at the expression in equation 10. This would also make the paper more self-contained, for readers in the RL community perhaps not so well-read with literature on MMD.  Appendix C contains the detailed derivation, but more intuitive explanations might be useful for the context of the paper. 
	- The proposed black box estimator seems quite useful as demonstrated in figures 2 and 3. Although the authors evaluate their approach of few simple domains - it would have been useful if there were more extensive experiments performed for OPE problems. This would be useful since from fig 2, it appears that the proposed method only outperforms in 3 out of 4 evaluated problems. 
	- For experiments, it would also be useful to demonstrate the significance of not knowing the behaviour policy and what are the usefulness of it. The paper is motived in terms of unknown behaviour policies that generated the data - so few experiments that explicitly shows the benefit of it would perhaps strengthen the paper more. 
	- I am curious to know more about the bias-variance trade-off of the proposed OPE estimator as well. Ie, does the proposed method introduce any bias, or has significance in terms of lower variance for the long horizon problem? Experimentally, would it be possible to demosntrate whether the approach has lower variance compared to existing baselines?

Score : 

- Overall, I think the paper has useful contributions. It is a well written paper, but some additonal details in section 4.2 might be useful, especially on the appriach with MMD. Experimentally, I think there are some experiments missing and doing those can significantly strengthen the paper as well. The proposed method seems simple and elegant, and I would recommend for a weak acceptance of the paper.  

review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: Summary

This paper proposes using a black-box estimation method from Liu and Lee 2017 to estimate the propensity score for off-policy reinforcement learning. They suggest that, even when the full proposal distribution is available (as is the case in much of the off-policy RL literature), that estimating the propensity score can allow for a much lower variance off-policy correction term. This reduced variance allows the proposed method to "break the curse of horizon" as termed by Liu 2018.

Review

The proposed fixed point formulation for learning a parameterized off-policy state distribution allows for lower variance off-policy corrections decreasing the impact of the curse of horizon. This approach appears both theoretically sound and empirically well-supported.

I am concerned with the consistency argument made in section 4.3. It appears from equation (10) and the final statement of theorem 4.1 that it is necessary to have two independent samples of x' given a single state x. In the general case, without access to the environment model, it is not possible to obtain two samples of x'. If the environment is continuing, then the probability of returning to state x to obtain a second sample is 0. Am I misunderstanding the requirements specified by the objective function in equation (10)? An additional concern with the consistency argument is that it appears to assume that the approximator for d_w can achieve 0 error according to the MMD. If d_pi is not representable by the approximator, which could reasonably be the case in more challenging domains, it is unclear from this analysis or empirical results what the behavior of the system will be. It is difficult to assess how large of an assumption this is for consistency claim; for difficult problems will d_pi never be representable, or is this a fairly low concern?

The empirical results look to have high enough variance in the final outcomes that it is difficult to consistently assess the performance of each algorithm (looking specifically at figure 3). However, the primary competitor algorithm is IPS which the proposed algorithm handily beats in three of the four problems. In the fourth problem, it is unclear that the competitor algorithm is winning, and could in fact be better only due to chance given the size of the respective error bars. It is worth noting that, because the parameter settings were tuned only for 50 trajectories, it is important to primarily assess performance based only on that point. It is likely that, given more trajectories to learn from, each algorithm would have chosen a smaller stepsize and effectively performed similarly.

Additional comments (did not affect score)

I would be interested in seeing the scalability of this approach empirically. Given the additional parameterized function to learn, I am unsure if this method would scale reasonably to much larger problems. However, I recognize that the scalability question is largely outside the scope of this paper.
review_title: Official Blind Review #2

Meta-review: This paper addresses an important and relevant problem in reinforcement learning: learning from off-policy data, taking into account the offsets in the visitation distribution of states. This has the promise of lowering variance even with long horizon roll-outs. Existing methods have required access to the behavior policy (or have required data from the stationary distribution). The novel proposed approach instead uses an alternative method, based on the fixed point of the "backward flow" operator, to calculate the importance ratios required for policy evaluation in discrete and continuous environments. 

In the initial version of the submission, several concerns were expressed regarding both the quality of the paper and clarity. The authors have updated the paper to address these concerns to the satisfaction of the reviewers, who are now unanimously in favor of acceptance. 
Decision: Accept (Poster)