Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: 
Strengths:

1) The experiments are extensive, and clearly demonstrate the merits as compared to prior benchmarks for off policy RL.

2) The contextual discussion is clear, well-motivates the proposed approach, and gives a nice overview of how importance sampling and off policy RL intersect.



Weaknesses:


1) Theorem 1 seems vacuous. The proof is a simple exercise in elementary calculus -- one may easily show the minimizer of a quadratic is the least squares estimator. The authors need to better explain what is the technical novelty of this statement, and why it is meaningful. Upon inspection it does not seem to qualify as a theorem. This is also true of Theorems 2 and 3. Therefore, I feel the conceptual contribution is not enough to warrant acceptance.

2) The notion of successor representation seems identical to the occupancy measure, which in order to estimate, requires density estimation, which is extremely sample inefficient. Can the authors comment about how to estimate the successor representation efficiently? There is very little discussion of sample complexity throughout, which is somewhat alarming because a key selling point of off-policy schemes for RL is that they alleviate the need for sampling from the MDP transition dynamics.

3) The actual algorithm pseudo-code is missing from the body of the paper, which is permissible because it is in the appendix. However, the structural details of how the algorithm works iteratively and how it departs from previous works are also not explained. That is, while derivation details are presented, iterative details are not. In my opinion this should be strictly required in the body of the paper, as well as contextual discussion of what is similar/different from previous works, but all I could find was high level presentation of objectives minimized at various sections, but not how they are interlinked.

4) The background discussion is disjointed. There is a preliminaries section on page 5, as well as a background section 3.


Minor Comments:


1) References missing related to the sample/parameterization complexity issues associated with importance sampling: 

Koppel, A., Bedi, A. S., Elvira, V., & Sadler, B. M. (2019). Approximate shannon sampling in importance sampling: Nearly consistent finite particle estimates. arXiv preprint arXiv:1909.10279.
review_title: Theorems are vacuous, and do not seem to correspond to what the experiments demonstrate.

final_score: rating: 6: Marginally above acceptance threshold
review_content: ***Summary***
The paper proposes an approach to employ successor representation combined with marginalized importance sampling. The basic idea exploited in the paper consists of expressing the occupancies in terms of the successor representation and to model it via a linear combination of some features. This allows handling, although approximately, continuous state-action spaces. After having derived the objective function, an experimental evaluation on both Mujoco and Atari domains is presented, including an ablation study.

***Major***
- (About the linearity of the weight) Linear representations expressed in terms of a feature function are common in RL as the reward function can be often seen as a trade-off of different objectives encoded in the features. However, the choice of the linear representation in Equation (7) is based on the assumption that the marginalized weight is linear in the feature function. This assumption seems to me less justified compared to the one for the reward function. Clearly, a suitable feature design could overcome this limitation. Can the authors explain how the features \phi are selected or learned? 
- (Experimental evaluation) The results presented in the experimental evaluation are partially unsatisfactory, as also the authors acknowledge. It seems that there is no clear benefit in employing the marginalized importances sampling (both the baselines and the proposed approach) compared to standard deep temporal difference approaches. The authors suggest that this phenomenon can be ascribed to the fact that the quality of the marginalized weights is affected by the successor representation learned. I don't think this is the main weakness of the paper, but a reflection of the usefulness of the method in complex scenarios is necessary. Alternatively, it would be interesting to compare the proposed approach with DualDICE and GradientDICE on simpler tasks (maybe toy ones) in which DualDICE and GradientDICE work well.


***Minor***
- The related work section should be moved later in the paper, maybe after Section 4
- Pag 2, two lines above Equation (2): the transition model is here employed as a distribution over the next state s' and the reward r, but the reward function is considered separately in the definition of MDP presented before
- Figures 2, 3, and 4: the plots are not readable when printing the paper in grayscale. I suggest using different linestyles and/or markers

***Typos***
- Pag 2: isn't -> is not
- Pag 2: doesn't -> does not
- Pag 8: the the -> the

***Overall***
The paper can be considered incremental compared to DualDICE. I did not find any fault, but I feel that the significance of contribution is currently insufficient for publication at ICLR. In particular, for a paper that proposes a practical variation of a theoretically sound algorithm, the experimental evaluation is essential. I think that the results are currently unable to clearly show the advantages of the proposed method.
review_title: Practical Marginalized Importance Sampling with the Successor Representation

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes SR-DICE, which uses a successor representation to compute DICE (discounted stationary distribution correction term).
* I am worried about both the technical and experimental qualities of this work. The theorems presented are either obvious or previously presented in other works. While the authors argue that the marginalized importance ratio is independent of the horizon (I assume that they are talking about the variance), MIS only alleviates the estimator variance's exponential dependence on the horizon to become the polynomial dependence on the horizon (as proved in Xie et al., Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Sampling, 2019). In the experiments, it is hard to believe that the GradientDICE and DualDICE perform that poorly having Log MSE larger than 0 while the GenDICE paper reports Log MSE less than -4 (HalfCheetah).
* The paper uses $\phi$ and $\psi$ learned by the previous deep successor representation learning algorithm, which is not meant to be used to learn marginal importance ratio. In particular, $\phi$ is learned by minimizing state, action, and reward reconstruction error and $\psi$ is the discounted sum of $\phi$. If we consider a case where $\pi$ only exploits a very small subset of state-action space, it is easy to see that the reconstruction error minimization in the dataset is not an optimal representation for the marginal importance ratio learning. In this sense, only the linear vector $w$ is used for the learning of marginal importance ratio.
* The experiment setting is not fair. Direct-SR and SR-DICE in their implementation have effectively 2 hidden layers, where DualDICE and GradientDICE in their implementation have a single hidden layer.
* The paper is hard to follow. Especially, notation abuse between the real reward and the virtual reward which is optimized to give a marginal importance ratio is very confusing (abuse between real Q and minimizer Q as well). Section 4.2 is also confusing because the authors imposes the problem of DualDICE that is not actually handled by SR-DICE.
* The idea of adopting successor representation for learning marginal importance ratio seems quite novel.
* Some people will be interested in this work, but I think the paper would not have much impact on the field.

Overall,

PROS:
* The idea of using successor representation for learning marginal importance ratio is novel.
* Avoids minimax formulation of other DICE algorithms, which makes the optimization very hard.

CONS:
* Not very meaningful theoretical results are presented, which mostly just confuse readers.
* Uses the representation that is not learned for marginal importance ratio learning
* Questionable experiment results


Minor details:
* y axis label is "Log MSE" for figures although the y axis is log scaled MSE.


--------------------------------------
Most of the concerns are addressed by the authors, and I raised my score accordingly.


review_title: Not convinced by both the theoretical and empirical results

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors propose SR-DICE based on deep SR for density ratio learning. Empirical advantages are observed in tested domains. Overall I think the idea is interesting and theoretically sound, but the experiments are not fully convincing.

It looks the main claim is that SR-DICE is better than other MIS methods because SR-DICE delegates the update propagation over the MDP to SR, while other MIS methods consider update propagation and density ratio learning together. To me this claim is not coupled with function approximation at all, so I would like to first see some experiments in the tabular setting. SR-DICE is a two-stage learning algorithm, i.e., SR learning + density ratio learning, both have hyperparameters to be tuned. GradientDICE and DualDICE are one-state learning algorithm. If in the tabular setting, we can empirically verify that under the best hyperparameter configuration of each algorithm (guaranteed by a thorough grid search), SR-DICE is more data efficient (counting the samples used in both stages) than GradientDICE and DualDICE, in terms of the density ratio prediction error, then the argument can be well backed. Well-controlled experiments like this, however, do not appear in the current submission.

Once deep networks are used for function approximation, we run into the problem of representation learning. The authors should at least include one more experiment, where MIS methods run directly on the pretrained deep SR features \psi_\pi(s) and/or \phi(s). In this way, we can distinguish whether the empirical advantage of SR-DICE comes from SR-DICE itself or the improved representation learning.

I'm also interested in seeing experiments for larger gammas, e.g., 0.999, 0.9999. I'm wondering if SR-DICE can consistently outperform GradientDICE with increasing discount factors.

Overall, I'm happy to increase the score if I have any misunderstanding or more convincing results are presented. I appreciate that the authors include deep TD and behavior R(\pi) as baselines. The empirical study has independent interest beyond SR-DICE. Moreover, deep TD is also referred to as Fitted-Q-Evaluation (FQE) in [x].

x: Voloshin, Cameron, et al. "Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning." arXiv preprint arXiv:1911.06854 (2019).

=======================

(Nov 24) The author response addressed my concerns and I therefore raised my score from 5 to 6. I particularly like the idea of using successor representation for density ratio learning.
review_title: More experiments are necessary to understand how and why SR-DICE works

Meta-review: The paper is about an approach that combines successor representation with marginalized importance sampling.
Although the reviewers acknowledge that the paper has some merits (interesting idea, good discussion, extensive experimental analysis) and the authors' responses have solved most of the reviewers' issues, the paper is borderline and the reviewers did not reach a consensus about its acceptance. In particular, the reviewers feel that the contributions of this paper are not significant enough.
I encourage the authors to modify their paper by taking into consideration the suggestions provided by the reviewers and try to submit it to one of the forthcoming machine learning conferences.
Decision: Reject