Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper introduces the problem of active offline policy selection which consists of selecting the best policy from a set of candidates given a small budget to interact with the environment. The paper proposes a solution (A-OPS) based on Bayesian optimization using a kernel function to relate candidate policies and generalize between them. The proposed solution is thoroughly evaluated on simulated robotic control environments using benchmark datasets from the off-policy evaluation literature and several ablations are included to rationalize the choices of the proposed method.
main_review: ### Strengths:

1. The proposed problem seems important and not yet formalized in the community. Indeed much of the recent focus on offline RL and off-policy evaluation has neglected the possibility for a small budget of interaction with the environment. Thinking about how to effectively exploit this small budget is an important area for future work, and hopefully this paper can spur research in this area.
2. The proposed solution is elegant. While not ground-breaking, the proposed solution does seem to be an intuitive and elegant way to approach the problem. With such a small budget of interaction, bayesian optimization is often a very strong approach, as this paper bears out.
3. The experimental evidence is substantial. The paper runs on many environments with large datasets of policies. The advantage over the baselines is very evident. The ablations seem to do a comprehensive job of rationalizing each of the major components of the proposed approach. 

---

### Weaknesses:

1. The formalism is often imprecise. Here are a few examples:

   (a) In Section 2.1, the notion of a budget which is often discussed informally in the text is never formally defined

   (b) As a result $ \hat k$ is used in the main definition of regret but is never defined. This makes it somewhat unclear if the paper is concerned only with the regret of the final policy or if it also cares about the cumulative regret during the evaluation in the environment (my impression was the former, but slightly unclear). 

   (c) In section 2.2 $ N_k $ is never defined, but often used. (It seems to be used to indicate the number of return observations of policy $ k$)

2. Some important details are omitted in the experiments section.

   (a) In Figure 7 the labels on the x-axis are never explained. The reader can try to guess (is MB-AR maybe model-based auto-regressive?), but these labels really ought to be explained in the text.

   (b) For the plots in e.g. Figure 5, are the algorithms re-run from the start given each budget or are the algorithms just progressively evaluated after each trajectory is added? It seems to be the second, but this was a bit unclear. 

3. This is perhaps a more minor point, but based on the results in the appendix, it would seem useful to discuss the variability of the performance across tasks. While in aggregate the advantage of A-OPS over simple OPE is obvious, the advantage is much larger on some tasks than others (Fig 9, Fig 12). Moreover, for some tasks in the MPG domain, it seems that A-OPS performs worse than OPE even after 400 trajectories (Fig 12). Even if this discussion is just put in the appendix, it would be useful to discuss rather than just repeating that A-OPS is outperforming the other methods. 

---

### Recommendation:

Accept.

I think this is a strong paper that introduces an interesting problem and provides a reasonable solution with substantial experimental evidence. The paper is sloppy about details in a few places, but these should be easily fixed. 

---

### Minor comments/questions:

1. Is there a way for the algorithm to incorporate knowledge of the size of the budget? Now the budget just determines the length of the loop, but it seems that the algorithm could modify its strategy with knowledge of the budget.

2. Have you thought about how to select the set of candidate policies to best leverage the strengths of A-OPS? I understand that for this paper 

   


limitations_and_societal_impact: Space is already a bit tight, but the paper would benefit from a more substantial discussion of the limitations in the current methods and the potential directions of improvement. I do not think a discussion of societal impact is necessary for this paper.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper introduces a problem of active offline policy selection, where the goal is to select the best-performing policy among the fixed K policies using an abundant offline dataset and a limited additional interaction with the environment. This work proposes to solve this problem by Bayesian optimization (BO) with a Gaussian process (GP), where the input and output of the objective function are a policy and its expected return respectively, and function queries render noisy observations. To facilitate GP in this problem, a new GP kernel is introduced, which takes two policies as input, and its output is computed based on the similarities of policy distribution for each state in the offline dataset. Also, to make BO more data-efficient, off-policy evaluation (OPE) estimates through the offline dataset are used for a warm start of BO. Finally, UCB acquisition function is used for active policy selection. Experimental results show that the proposed BO with OPE-based warm start significantly outperforms the baselines. Also, ablation studies confirm that UCB selection is better than uniform selection, GP is better than methods that do not consider policy correlation, and using OPE is better than not using it.

main_review: - Overall, the paper is well-written and easy to follow. The introduced active offline policy selection is well-motivated by practical situations such as A/B tests, and the proposed method is sound with a suitable combination of Bayesian optimization and OPE estimates. I found introducing this setup is novel. The proposed solution itself is not surprising, but the experiments are thorough and the results are also convincing. The ablation studies are solid and appreciated.
- The additional data collected during online interactions could be used to improve the OPE estimates since they are particularly on-policy samples for a candidate policy?
- In Eq (3), the generative assumption may not be consistent with the actual episodic return distribution since it is likely that return distribution may not be Gaussian-distributed (e.g. highly skewed distribution). Would this inconsistency make a problem?
- In Figure 3 (middle), why are the similarities between D4PG policies very low, even with the same hyperparameters?
- It seems the proposed policy distance (Eq (6)) may not be optimal since it considers state distribution by the offline dataset, not the state distribution induced by the target policy. For example, if the target policies visit a very limited region of states, to measure the policy similarity, we only need to consider those states that will be visited while ignoring other states that will not be visited. The distance between stationary distributions induced by policies could be a more suitable distance measure?
- What if the OPE estimates are significantly underestimated (or overestimated), compared to the true expected returns? Pure exploitation (or pure exploration) might happen during active policy selection?
- In Figure 5, why doesn't the performance of A-opt at (# trajectories=0) match the performance of OPE (even worse than OPE)?

limitations_and_societal_impact: I think the authors have discussed the limitations and potential negative societal impact properly.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper works on the policy selection problem in a setting characterized by features: available logged data and allowing a limited number of online interactions with the environment. To do so, this work proposes to use Bayesian optimization to combine existing OPE techniques and online interactions. For the setting with large number of policies, a kernel function is introduced to take the correlation between policies, which indicate the performances of different policies. 
main_review: The problem formulation is not typical. It is a mix of the common offline and online settings. On the one hand, it aims to leverage the logged datasets and OPE; on the other, it is not totally offline as the setting in the paper allows online interactions, although the number such interactions are limited. Given this, this work proposes a Bayesian optimization solution. While such a setting is possible in real-world, I doubt the importance of such a non-typically setting itself. 
limitations_and_societal_impact: Please also refer to the main review section. For high-stakes decision-making scenarios that offline RL aims to address, further online interactions with the environment is not available for reasons including ethical or cost concerns. Take healthcare and education for example, logged dataset can be large, but online interaction is typically infeasible. This work fails to show the importance of the proposed setting itself.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper considers a novel problem setting of "Offline Policy Selection", whereby there is access to an initial set of policies trained offline, and an algorithm must choose a policy to deploy in the online simulated environment. I am giving this paper the score of weak reject because despite introducing an important problem setting, I am not convinced the method is actually particularly effective. Hopefully this can be clarified during the rebuttal phase. As such, the emphasis of this review is on areas for improvement.
main_review: Strengths:

1) Novel problem setting, definitely well motivated. Could be important in using Offline RL more broadly.

2) The policy kernel idea seems intuitive, and the discussion around choice of kernels (A.5) is interesting. I think it would improve the paper if some of this discussion/additional experiments are included in the main body (more below).

3) The authors did make efforts to include multiple ablations, and discussed limitations.

4) The use of OPE + online is obvious in hindsight which is a common trait of valuable contributions.

Weaknesses/Questions:

1) The actual number of trials used is much larger than I would expect, with BO not converging until >100 trajectories. For DM control suite, *this is enough to train a near optimal policy from scratch using online data*. Why would we not just do this? I think a key baseline is OPE+online, since OPE always performs reasonably well zero-shot, we could then use the online data to fine tune the best OPE policy. 

2) Given 1), it seems that formulating this problem as blackbox optimization may not be the best way to go, since it is not making use of the structure of the task at all. Aside from using the episode returns, A-OPS does not leverage any of the trajectory data (as far as I am aware) which ends up being 200k timesteps worth of online samples by the end.

3) What is the actual variance of the policy returns? The aggregate regret is definitely an effective overall metric to use, but if for example the min and max polices for a task are only 5-10% apart it is rather meaningless.

4) It would be useful to include a plot showing the *min return* for each method. Using an exploratory acquisition function (such as UCB) likely leads to also testing some policies far away from the current best in the behavioral space, which could lead to very poor returns. For practical use this may be infeasible since if deploying on real hardware you could break the robot. This is alluded to in the Limitations section in the Conclusion, but it is not shown anywhere. I have a minor concern this may be because it looks bad for A-OPS. 

5) Looking at the *individual* task performance in the Appendix, it seems to be very hit and miss. Do the authors have any intuition why none of the methods or ablations work well for the manipulator dm control task? Similarly with MPG, for 2/4 of the tasks the OPE approach actually does better than A-OPS despite using zero online data. It seems only really the slide task which makes A-OPS outperform overall due to the poor performance of OPE, what happened there?

6) How did you tune the UCB exploration parameter? 

7) I assume the answer to 6) and also other ablations is that the method was tuned on the testing data. Is this true? 

Minor comments/typos:

-l.97: "We can use GP" → "we can use a GP"

-l.204: "We perform 4 several manipulation tasks" (not clear what this means)
limitations_and_societal_impact: Limitations are discussed sufficiently well but of course could always be expanded. There is no expected negative societal impact from this work.
review_title: null

Meta-review: The reviewers felt this paper would be a nice contribution to the conference. It introduces a novel problem setting of active offline policy selection problem formulation, which relies both on logged data and limited online interactions with the environment.  The authors have done a good job replying to all concerns during the discussion.  To strengthen the contribution, authors are encouraged to open source components of the approach.
Decision: Accept (Poster)