Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new method for ranking of offline RL policies with off-policy evaluation (OPE). The ranking is produced with a model that 1) learns a pairwise policy representation with a transformer architecture, 2) uses a crowd layer to aggregate OPE scores of other methods. In the experimental results the authors show that their method is able to outperform the other baselines. The ablation studies show the importance of various components of the proposed method.
strength_and_weaknesses: Strengths:

- The paper is well written and easy to follow. The details of the method and the experiments are clearly explained. The figures are informative and well explained.
- The idea of using pairwise policy representation sounds interesting and novel.
- The experimental results where the proposed method outperformed the other baseline is very encouraging.
- The experimental results studies the problem with different settings.

Weaknesses:

- To my mind, the experimental results are lacking an adequate baseline that is comparable with the proposed method. A baseline would be comparable if it also uses other existing OPE methods to aggregate the results. For example, I can imagine several easy baselines in this case: 1) take the average OPE scores of all methods and produce a ranking out of them, 2) use a majority voting scheme to aggregate the rankings, 3) there are many rank aggregation methods that could be considered, for example, [1]. 
- As the proposed method aggregates the ranks from the existing policies, the computational cost for it is much higher than for any other method and it includes the costs of all other methods. This should also be discussed. 
- Another limitation in the current experiments is that as far as I understand the method needs to be trained for every new set of policies and the environment from scratch. Then, it is tested on its own training set (no validation or test set). Do I understand the setting correctly? Would the policy representations generalize across different sets of policies? Suppose a new policy is added to the set of policies, can the previous results be re-used? To me, the method would be useful in practice if it can show signs of such generalization.
- I do not understand why transformers are the best architecture in the given policy representation design. As the states (equation 1) are chosen as just a set (not ordered), what is the advantage of using a transformer which is known to be the best suited for sequential data? Did the authors consider other architectures (possibly simpler, e.g., MLP) here?

Other comments:
- Several times the authors mention that in practice finding the best policy is the main objective. In that case, it would be more logical to consider off-policy policy selection (OPS) problem formulation and as the quality metric measure the regret @1. How would the method perform in that case? 
- I still do not understand the role and training of the "aggregation token" very well, maybe this could be explained further.
- In the first part of section 4.2 the authors say that they "show how the policy ranking can be reduced to binary classification". I think this is a common way to approach the ranking problem (but the text sounds now like this is one of the contributions). Some work could be references here, for example, [2].

[1] Fast and Accurate Inference of Plackett–Luce Models. Lucas Maystre, Matthias Grossglauser. NIPS 2015.

[2] Preference Learning with Gaussian Processes. Wei Chu, Zoubin Ghahramani. ICML 2005.

clarity,_quality,_novelty_and_reproducibility: Clarity: good.

Quality: good.

Novelty: the paper combines several existing components and aggregates the results of the existing methods, but the idea of the aggregation is reasonably novel. Also, using pairwise instead of direct policy representation sounds novel to me.

Reproducibility: the paper provides sufficient details on the methodology as the space permits. Are the authors planning to open source the code?
summary_of_the_review: I am leaning toward rejecting this paper mainly because I find the experiments lacking comparable baselines that would benefit from aggregating the results of the existing methods in the same way as the proposed method. Also, I would also like to see some generalization of the method or policy representation to the unseen policies that would make the method scalable to real world problems.

---
Updated my score after rebuttal in the light of new empirical results.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: To fulfill OPE task, this paper simplifies OPE into OPE and proposes an end-to-end method, namely SOCCER. SOCCER is compared with several baselines in two environments. Experimental results show that the proposed method achieves high accuracy with certain hyper-parameters. An ablation experiment is also performed.
strength_and_weaknesses: Strength:
1. This paper simplifies the OPE into OPR tasks.
2. This paper proposes an end-to-end method to solve the OPR.
3. The proposed method is relatively technical sound.

Weaknesses:
1. The experiment settings lack some explanations, e.g. the reason for the specific value of hyper-parameters.
2. The figures need to be improved, e.g. sub-figures in Fig.4 are not aligned.
3. Ablation experiments as well as baselines need to be considered more carefully. From my point of view, the essence of the proposed method is similar to a label aggregation method with deep learning. Thus, some other structure-like method should also be added, e.g. SpeeLFC. The ablation experiment introduces an extra strategy to obtain ‘truth’ to train the method. This setting lowers the convincing performance of the proposed method.

clarity,_quality,_novelty_and_reproducibility: The overall quality of this paper as well as the reproducibility of the proposed method is fair. The proposed method is technically sound. However, the novelty is relatively limited. From my point of view, the proposed method is like a combination of several existing methods in a simplified manner.
summary_of_the_review: This paper simplifies the OPE task into an OPR task and proposes an end-to-end method, namely SOCCER, to tackle the problem. SOCCER contains policy representation (pairwise), a feature extractor (MLP and MSA), and a training strategy (Crowd Layer). SOCCER is compared with several baselines in two environments with three modes. The process of the experiment is relatively complete. An ablation experiment is also performed, while some settings make the results not very convincing. Experimental results show that the proposed method can achieve good performance in some conditions. Although this work lacks theoretical innovation and the representation of this paper has some flaws, the entire work is relatively complete and the OPE problem is relatively solved.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper studies the problem of offline policy evaluation. The key idea is that instead of predicting the value of the target policy, the authors proposed to evaluate the relative rank of the policy, among a set of candidate policies. When there is no sufficient historical observations of different behavior policies, the authors proposed to leverage the idea of crowd-sourcing to integrate prediction from multiple offline policy evaluators. The final ranking/scoring of each target policy is obtained by aggregating the pairwise comparisons among all candidate policies. Experiments on a set of simulated offline trajectory data were performed against a rich collection of offline policy evaluation methods. 
strength_and_weaknesses: Strength:
+ The idea of off-policy rank evaluation is reasonable and valid, which to my best knowledge is first time discussed in literature. Although only high-level intuitive arguments are provided regarding the advantage of this new objective, it is still nice to observe its effectiveness in the reported empirical studies. 
+ The reported empirical studies provide a comprehensive picture about the comparison between the proposed solution and a rich set of baseline methods.  

Weakness:
- The proposed method lacks necessary theoretical justification, with unknown properties about the provided estimations, e.g., there is no way to quantify its bias and variance. 
- The solution boils down to a binary classification problem, under which the crowd-sourcing idea is a natural extension when only noisy labels exist. Following this logic, methods for learning from noisy labels can also be leveraged to address the problem. 
- The empirical nature of the proposed solution creates a large set of hyper-parameters, which make the tuning and comparison hard to exhaust. For example, presumably the number of states where we execute the policies to obtain their actions, the number of pairwise policy comparisons, and the number of OPE workers are important for the performance of the proposed solution. But there is no experiment evaluating the impact from such hyper-parameters.   

Questions:
- In Section 5.2, it is mentioned that in each epoch, 5 OPE workers are randomly selected from the baseline models. I am not sure why we should sample different OPE workers every epoch. Shouldn’t they be the same throughout the training and testing stages for the crowd layer to learn the (equivalent) confusion matrix?  
- What’s the principle to determine the number of states where we execute the policies to obtain their actions, the number of pairwise policy comparisons, and the number of OPE workers in practice? Are they the more the better?
- Why not have a dedicate policy encoder to represent each policy and then compare their embeddings for binary classification? Or we can simply follow the way we use transformer to encode two sentences into one embedding to embed the two policies. The conventional position embedding in transformer can help us realize the corresponding positions are actions from two policies but under the same states.
- My understanding about $e_\alpha$ and $e_\beta$ is that they are fixed one-hot vectors. But Section 5.2 described them as random vectors at initialization. Clarification is necessary here.  

clarity,_quality,_novelty_and_reproducibility: The paper is generally well written and easy to follow, especially its core idea of converting off-policy evaluation into a binary classification problem. Every step in the proposed solution is standard, e.g., the crowd-layer for integrating different off-policy evaluators, and transformer to encode a sequence of actions. Hence, it should not be difficult to reproduce the algorithm pipeline. The authors also provided details about most of the hyper-parameter settings in the paper, which should help ensure the reproducibility of the reported results.  
summary_of_the_review: The idea of studying offline policy ranking, instead of evaluation, is an interesting and practical idea. But the proposed solution is overly simplified and lacks necessary theoretical justification or analysis. It is hard to know when the algorithm would work better than standard off-policy evaluation methods.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors present an Off-Policy Evaluation (OPE) method geared toward ranking policies w.r.t their estimated performance, as compared to the standard task of predicting the estimated performance absolutely. It is more aligned with the OPE end goal, i.e. ranking policies among a set of candidates. 

Previous work on off-policy ranking SOPR-T (Jin et al., 2022) used supervised learning for ranking policies, but it assumes access to multiple deployed policies with reward signals available. It is argued by the authors that this assumption is impractical, as it assumes multiple deployed policies along with their rewards in the log data. 

Authors instead propose a self-supervised learning method, based on the 'learning from crowd' paradigm, where they assume different OPE methods as workers in a crowd-sourcing setting, with each worker generating noisy labels. A previous work, 'Crowd Layer (CL)' from learning via crowdsourcing literature is used to learn with the noisy label setting. 
strength_and_weaknesses: Strengths of the paper:

- The problem of OPE via policy ranking is very practical and bypasses the requirement for estimating the absolute policy value (expected reward), which is known to be a difficult problem. 
- In practical settings, like recommender systems, ultimately we need a ranked list of policies, according to their estimated 'online' performance. 
- The method can be used in a setting where access to multiple deployed policies with corresponding rewards is not feasible, for ex: in a clinical trial. 
- Existing work from crowd-sourcing is used to learn from existing OPE methods while assuming they generate noisy labels, which is a fair assumption. 

Some questions to the authors: 
- How is a transformer-part important to the off-policy ranking pipeline? For ranking policies, have you compared with existing Learning-to-rank methods like LambdaMART on top of features extracted from trajectories of the policies, two-tower network (like Neural Matrix Factorization) with say RNN based feature extraction for both heads? For a given trajectory, a listwise method like lamdaMART can rank all the policies at a time, instead of a pairwise comparison. 
- A clarification question (sorry if I missed something obvious): What is the dimensionality of $x_k$s (Eq. 2)? Is it a scaler, or a vector? Since you are adding a 2-d $e_\alpha$ vector to it, I am assuming it's a 2-D vector?  
- In a setting like a recommender system/search, where you have access to a large log data with multiple policies and their rewards, how does this method compare with the baseline "SOPR-T"?
clarity,_quality,_novelty_and_reproducibility: - The paper is very well-written, and it's very easy to understand. 
- The method proposed is novel and very practical, especially in a setting where access to deployed logging policies is limited, like clinical trials. 

summary_of_the_review: - Authors propose an off-policy ranking method, which directly predicts the ranking between two policies, using a ranking method. 
- Assuming a setting where access to multiple logging policies is restricted, the proposed method can leverage existing OPE methods in a pseudo-crowd-sourcing setup, with each OPE method as a worker in the crowd, and learning from the workers in a noisy label setting.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper builds on a 2022 work where a policy comparator is trained in a supervised learning fashion on pairs of policies. This work makes two additional advancements:

* A better representation for pairs of evaluated policies.

* A crowdsourcing framework where the performance of the policies is estimated using existing off-policy estimators. The logic behind this approach is that if most off-policy estimators are pretty good, it is possible to pool them to learn even a better comparator.

This paper received 3 borderline reviews and one reject. The borderline reviews slightly improved after the rebuttal but the reject review remained. The main reason for this review is that the approach is not principled, because it is a result of self-training from potentially biased and correlated off-policy estimates. At the last minute (December 9), the authors submitted a formal analysis. The most critical reviewer checked it and found strong assumptions, such as independence of off-policy estimators. This is obviously false because the estimators use the same dataset.

To break the impasse between the authors and the reviewer, I read the paper and also the new analysis. In summary, Theorem 1 is proved for

* Any two fixed policies $\pi_i$ and $\pi_j$

* A fixed OPE worker

The random quantities are the biases in the OPE of $\pi_i$ and $\pi_j$. The proof of Theorem 1 relies on an assumption that the biases are independently distributed. However, how can this be if $\pi_i$ and $\pi_j$ are very similar? One approach, as the most critical reviewer pointed out, is that the OPE of $\pi_i$ uses a different dataset from that of $\pi_j$. This is not done in the paper. Interestingly, this is also the main issue that the most critical reviewer had since the beginning. Everything is done on a single dataset and it is unclear how the biases average out.

After reading the paper, I also wanted to share one observation. If you check Figure 4, SOCCER without CL seems to be better overall than SOCCER. Thus the crowdsourcing idea may not work. Therefore, I believe that the main lift in this paper is due to a new way of learning the representation for pairs of policies.
justification_for_why_not_higher_score: This work may not be sound.
Decision: Reject