Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: This paper proposes a new estimator to infer the stationary distribution of a Markov chain, with data from another Markov chain. The method estimates the ratio between stationary distribution of target MC and the empirical data distribution.  It is based on the observation that the ratio is a fixed point solution to certain operators. The proposed method could work in the behavior-agnostic and undiscounted case, which is unsolved by the previous method.

This paper tackles an interesting problem with an increasing number of studies in the reinforcement learning community and gives a practical algorithm with strong empirical justification, as well as theoretical justification. I think this paper should be accepted.

Detailed comments:
1) This paper provides experiment results in multiple domains, including two continuous control domains which are more complex than experiments in previous OPE methods. The paper also provides many details about the learning dynamics and ablation studies, which is very useful for the reader to understand the result of the paper.
2) The theoretical result is as same strong as previous work DICE and DualDICE, under similar assumptions.
3) I appreciate this paper formalizes the two difficulties of degeneration and intractability, and then explain how those are addressed in a principled way. Degeneration is important and is at least ignored in two similar works on this topic.
review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: In this paper the authors proposed a framework for off-policy value estimation under the scenario of infinite horizon RL tasks. The new proposed method utilize the variational representation of $f$-divergence, which quantifies the difference between $\mathcal{T}\tau$ and $\tau p$, where $\tau$ is the parametric density ratio between the unknown behavior policy data and the target policy. If only if $\tau$ is the true density ratio, the loss $\mathcal{D}_{f}(\mathcal{T}\tau || \tau p) = 0$. 

Compared with prior work (Nachum et. al 2019), the new proposed framework can generalize the undiscounted case $\gamma = 1$, and the derivation for the new algorithm is quite simple and easy to follow. The experimental results show the advantage of the proposed methods over baseline methods such as model-based, DualDice etc, for both discrete and continuous cases. Moreover, I have two specific questions:

- The choice of $f$-divergence. Although the author mentioned the difficulty of using the dual representation of KL divergence, it would be nice to have an ablation study that shows the effectiveness of various $f$-divergence (Personally I think Jensen-Shannon Divergence may be also a good choice).

-The authors should also have a discussion that similar idea can be generalized to more general  distribution metrics such as Integral Probability Metrics, specifically wasserstein-1 distance (similar to wasserstein-gan) or maximum mean discrepancy (Maybe it is unnecessary to conduct experiments, some discussion should be enough to clarify the relationship. I think there is a concurrent submission using MMD metrics).
 

Overall I think this is a good paper and I recommend for acceptance. 

Reference Papers:
- Nowozin, Sebastian, Botond Cseke, and Ryota Tomioka. "f-gan: Training generative neural samplers using variational divergence minimization." Advances in neural information processing systems. 2016.
- Arjovsky, Martin, Soumith Chintala, and Léon Bottou. "Wasserstein gan." arXiv preprint arXiv:1701.07875 (2017).
- Nachum, Ofir, et al. "DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections." arXiv preprint arXiv:1906.04733 (2019).
- Anonymous, “Black-box Off-policy Estimation for Infinite-Horizon Reinforcement Learning”, submitted to ICLR 2020.

review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: Main contributions:
This paper generalizes the recent state-of-the-art behavior agnostic off-policy evaluation DualDice into a more general optimization framework: GenDice. Similar to DualDice, GenDice considers distribution correction over state, action pairs rather than state in Liu et al. (2018), which can handle behavior-agnostic settings. The optimization framework (in equation (9)) is novel and neat, and the practical algorithm seems more powerful than the previous DualDice. As a side product, it can also use to solve offline page rank problem.

Clarity:
This paper is well established and written. 

Connection of theory and experiment:
I have a major concern for the theory 1 about the choice of regularizer $\lambda$. For infinite samples case, the derivation of theory 1 is reasonable since both term is nonnegative. However, in practice we will have empirical gap for the divergence term, thus picking a suitable $\lambda$ seems crucial for the experiment. I think a discussion on $\lambda$  for average case in experiment part should be added. And compared to Liu et al. (2018) which normalized the weight of $\tau$ in average case, which one is better in practice?

Overall I think this paper is good enough to be accepted by ICLR. The optimization framework can also inspire future algorithm using different divergence.
review_title: Official Blind Review #3

Meta-review: The authors develop a framework for off-policy value estimation for infinite horizon RL tasks, for estimating the stationary distribution of a Markov chain. Reviewers were uniformly impressed by the work, and satisfied by the author response. Congratulations! 

Decision: Accept (Talk)