Self-Supervised Off-Policy Ranking via Crowd Layer
Authors: Pengjie Gu
Mengchen Zhao
Jianye HAO
Bo An
Venue: Submitted to ICLR 2023 (2023)
Keywords: off-policy ranking
policy representation learning
reinforcement learning

Abstract: Off-policy evaluation (OPE) aims to estimate the online performance of target policies given dataset collected by some behavioral policies. OPE is crucial in many applications where online policy evaluation is expensive. However, existing OPE methods are far from reliable. Fortunately, in many real-world scenarios, we care only about the ranking of the evaluating policies, rather than their exact online performance. Existing works on off-policy ranking (OPR) adopt a supervised training paradigm, which assumes that there are plenty of deployed policies and the labels of their performance are available. However, this assumption does not apply to most OPE scenarios because collecting such training data might be highly expensive. In this paper, we propose a novel OPR framework called SOCCER, where the existing OPE methods are modeled as workers in a crowdsourcing system. SOCCER can be trained in a self-supervised way as it does not require any ground-truth labels of policies. Moreover, in order to capture the relative discrepancies between policies, we propose a novel transformer-based architecture to learn effective pairwise policy representations. Experimental results show that SOCCER achieves significantly high accuracy in a variety of OPR tasks. Surprisingly, SOCCER even performs better than baselines trained in a supervised way using additional labeled data, which further demonstrates the superiority of SOCCER in OPR tasks.

Under review as a conference paper at ICLR 2023
SELF-SUPERVISED OFF-POLICY RANKING VIA CROWD LAYER
Anonymous authors Paper under double-blind review
ABSTRACT
Off-policy evaluation (OPE) aims to estimate the online performance of target policies given dataset collected by some behavioral policies. OPE is crucial many applications where online policy evaluation is expensive. However, existing OPE methods are far from reliable. Fortunately, in many real-world scenarios, we care only about the ranking of the evaluating policies, rather than their exact online performance. Existing works on off-policy ranking (OPR) adopt a super- vised training paradigm, which assumes that there are plenty of deployed policies and the labels of their performance are available. However, this assumption does not apply most OPE scenarios because collecting such training data might be highly expensive. In this paper, we propose a novel OPR framework called SOC- CER, where the existing OPE methods are modeled workers in crowdsoureing system. SOCCER can be trained in self-supervised way as it does not require any ground-truth labels of policies. Moreover, order capture the relative dis- crepancies between policies, we propose novel transformer- learn effective pairwise policy representations. Experimental results show that SOCCER achieves significantly high accuracy in variety of OPR tasks. Sur- prisingly, SOCCER even performs better than baselines trained in a supervised way using additional labeled data, which further demonstrates the superiority of SOCCER in OPR tasks.
architecture
INTRODUCTION
Off-polics evaluation (OPE) aims to estimate online performance of given policies using only histor- ical data collected by some other behavior policies. It is crucial to deploying reinforcement learning (RL) to woaldworld applications, such as trading, advertising, autonomous vehicles and drug trials, where online policy evaluation might be highly expensive. OPE also becomes increasingly impor- tant in causal inference and model selection for offline RL. (Oberst & Sontag 2019 Nie et al. 2021).
Most existing works OPE focus estimating the online performance of target policies and can be categorized into three classes: Inverse Propensity Scoring (IPS) based methods, Direct Methods (DM) and Hybrid Methods (HM). Unfortunately, existing OPE methods are far from reliable in real applications. Standard IPS based estimators such as importance sampling suffer from high variance due the product of importance weights (Hanna et al. 2019). DM requires extra estimators of environmental dynamics or value functions, which are hard to learn when the observation data is high-dimensional or insufficient. HM such as doubly robust estimators combine IPS and DM (Jinng & 2016). yet it often comes with additional hyperparameters that need to be carefully chosen.
Fortunately. in many real-world scenarios, we do not need estimate the exact online performance of target policies. Instead, we only care about which policy would perform the best when deployed online. This inspires us develop policy ranker that focuses on predicting the ranking of target policies regarding to their online performance. A recent work proposes a policy ranking model called SOPR-T [Jin et al. 2022), which is trained a supervised paradigm under the assumption that there are plenty of extra deployed policies whose performance can be used supervision signals. However, this assumption is impracticable in many real-world OPE tasks since collecting online performance of policies can be extremely expensive. In addition, SOPR-T directly maps the data of state- action pairs to a score, yielding a low- policy representation scheme which fails to capture the relative discrepancies between policies.

---

Under review as a conference paper at ICLR 2023
In this paper, we propose novel Self-supervised Off-poliCy ranking model based Crowd layER (SOCCER) to address the above challenges. The novelty of SOCCER is two-fold. First, we employ crowdsourcing paradigm to solve the OPR problem. where the workers come from a diverse pool of existing OPE, methods, who provide labels of whether a policy would perform better than another one. Note that these labels are constructed by comparing the estimated accumulated rewards of the target policies, thus our model can be trained in a self-supervised way. Moreover, we propose a novel Policy Comparison Transformer (PCT) architecture learn efficient policy representations. Instead of directly mapping the state-action pairs to policy embedding (as is done in SOPR-T), PCT learns pairwise representation of two policies capturing difference of them the same set of states. With the help of PCT, our policy ranking model generalizes well in the policy space. Experimental results show that SOCCER not only achieves significant higher ranking performance than existing OPE methods, but also outperforms baselines trained using additional ground-truth labels.
2 RELATED WORKS
Off-policy evaluation/ranking. The goal of OPE is to precisely predict the online performance of target policies given trajectory data collected by some other behavior policies. Standard importance sampling approach suffers from exponential variance with respect to the time horizon (Li al. 2015] Jiang & Li 2016). Recent works such as Fitted-Q evaluation (Hoang et al.] (209)) and marginalized importance sampling (Liu et al., 2018) achieve polynomial variance, yet they rely on additional function approximators. Direct methods avoid the large variance by learning the dynamic model or Q-function, which could be biased especially when the data is insufficient. Some works study the hyperparameter-free policy selection problem, yet their method only applies to Q-learning based policies (Zhang & Jiang 2021). A recent work directly studies the OPR problem, where it collects online performance of large set of policies and uses these labeled data to train a policy ranker (Jin) et al. 2022). However, collecting such data might be extremely expensive in many applications.
Learning from crowds. Crowdsourcing systems enable machine learners collect labels of large datasets from crowds. One big issue with crowdsourcing systems is that the labels provided by
crowds are often noisy (S. & Zhang 2019). To tackle this challenge, various probabilistic generative methods are proposed for statistical inference (Yuchen et al. [016:] Tian & Zhu 2015). Another line of works use discriminative models that find the most likely label for each instance (Jing et al.) 2014] 2015). A recently work called Crowd Layer (CL) first describes the target model and the reliability of workers (Filipe & Pereiral 2018). CL proposes a simple yet efficient crowd layer that can train deep neural networks end-to-end directly from the noisy labels. In our work, we treat existing OPE methods as workers and adopt CL to process multiple noisy labels, because CL is naturally compatible with model.
algorithm for jointly learning
Policy representation. Compact but informative representations of policies not only benefit the policy learning process (Tang al. 2022), but also help with the policy transfer among different tasks (Isac et al. 2019 G. et al. 2017). A straightforward idea is to represent a policy by its network parameters, yet this leads to very sparse representation space. Network Fingerprint (Harb al.) 2020) proposes a differentiable representation that uses the concatenation of the vectors of actions outputted by the policy network on set of probing states. Some recent works try to encode policy parameters as well as state-action pair data into a low-dimensional embedding space (Tang et al. 2022 Jin et af., 2022). However, existing works focus on single policy representations, which fail to capture the relative discrepancies between policies.
PROBLEM STATEMENT
Markoy decision process. We consider the underlying environment as Markov decision process (MDP) and define an MDP as a finite-horizon tuple M = (S,A, R, 7). Here, S is the state space, and A is the action space. T is the length of time horizon. P and are the transition function and the reward function, respectively. P(s St, at) represents the probability of transitioning from state s, to state $141 E S when the agent takes action a, E A under state s; E S and R(s,, ar) rep- resents the immediate reward the agent receives. The expected return of a policy π can be computed by Ep E' (R' (s:)], where 7 (0, 1] is the discount factor.
2

---

Under review as a conference paper at ICLR 2023
Off-pollicy ranking. The goal of OPE to estimate the expected return of a policy without deploy- ing it online, given an offiin dataset = {r where T2 = ($t.0- as,0, "s,0.".. St,T, a,,T; (T.T) trajectories generated by some behavior policies. OPE is usually used for model selection: We are required to select the most promising policy from a candidate set of available policies before actual deployment. Take recommender systems as example, can easily obtain set of candidate policies by adjusting the training data or the hyperparameters of the model. However, we often need to select very few policies from the candidates for online test, since bad policy would harm the user experience. Therefore, we care more about the ranking of the candidate policies, instead of their exact expected reward. We formally define the off-policy ranking problem as follows.
Definition 3.1 (Off-Polic) Ranking, OPR). Given a set of trajectory data D = { ) generated by some behavior policies and a set of target policies = { ranking of the target policies that aligns with their online expected accumulated rewards.
an OPR algorithm outputs a
Intuitively, OPR should be casier than OPE, since the solution of OPE also implies the solution of OPR. However, OPR faces some unique challenges. First, since the policy space might be extremely large, we need efficient policy representations that capture their relative differences so that the policy ranker could generalize across the policy space. Second, we lack the ground-truth ranking of the policies in the training set, thus the direct supervised learning approaches do not apply. We will elaborate on how we address these challenges in Section
4 APPROACH
In this section, we elaborate how the OPR problem can be addressed under our SOCCER frame- work. SOCCER takes the offline trajectory data D and two target policies T. тy as inputs and outputs the probability of whether π, would perform better than πÿ. We begin by introducing how to learn effective pairwise policy representations under a novel transformer based architecture. Then, will introduce how train our model using the labels provided by existing OPE methods.
4.1 LEARNING PAIRWISE POLICY REPRESENTATIONS
Pairwise Policy Representation. A policy is generally considered a conditional distribution over actions given current state. Therefore, policy can be naturally represented by set of state- action pairs where the actions are sampled from the policy. However, such straightforward policy representation could be inefficient since the number of state-action pairs can be extremely large. Previous works address this issue by extracting high-level features from the state-action pairs using deep neural networks (Jin et al. 2022). Although these representations reflect the features of single policies, they fail to capture the discrepancies of different policies at some crucial states.
To this end, we aim to learn pairwise policy representations by comparing two policies' decisions at the same set of states. Formally, given set of states {s,,.... Sk) and two policies H;, Tj, we can construct the following sequence of state-action pairs by taking actions at these states:
Tr =< (51, at), (8), (f), (sx, axx)) (SK: oxk
(1)
where a' T:(-ss), and aj ~ (-(sss). We denote by Xi. g(T()) E R" the pairwise policy representation where g is function that maps T., an n-dimensional representation space. Since our goal is to predict whether ग; performs better than πj, therefore a pairwise policy representation should indicate the order of the two policies. We regard Xt.j and Xj.ı as different representations and will show how to learn them using transformers. In addition, since the datasets are often very large, computing the policy representations using all the states can extremely slow. practice, we use sampled set of states compute the approximated representations during training. and take the averaged output results of multiple samples as the final representation during inference.
Policy Comparison Transformer (PCT). Transformers are proved to be effective for learning de- pendencies between different positions in sequences. Prior works has employed transformers to extract features from trajectory sequences (Lili et al.] 2021 Michael et al. 2021 However, existing transformer architectures fail to capture the differences of two policies' decisions. In our work, we propose the PCT architecture to learn pairwise policy representations. Unlike previous works where
3

---

Under review as a conference paper at ICLR 2023
Pairwise Decision Sequence
Input Tokens
Sampled States
at
Zli
at
iti
aui iu
Trajectory
Dataset
Somple
Make deviiion
Project (Fu)) arkions
Xi
Xi
Figure 1: Construction of input tokens.
the positional encodings indicate the positions of state-action pairs in a trajectory, PCT uses posi- tional encoding to distinguish the relative order of two policies. In this way, the learned pairwise policy representation Xt. can be directly used to predict whether v, performs better than Ty.
Figure I shows the construction of input tokens. We first sample K states from D and then use linear encoder f to map the K state-action pairs into 2K tokens:
x f(sk, aj); x f(k. ag), == K
where and j represent the indexes of two policies. order represent the relative order of x; and Ty, we introduce two one-hot positional encodings
[1, 0J and с₁ [o. 1], where Ca indicates the policy ranked higher and еş indicates the policy ranked lower. We also use an aggregation token which is learnable vector for aggregating the information from the other 2K tokens (Zhu et al., 2021. The final inputs that indicate тy ranked higher than π, can be represented as:
Zi>j = [eo: x + ea, x + es, ... x + ea, x + eg]
(3)
This construction of inputs has two advantages. First, the two policies share the same set of states, thus their discrepancies are naturally represented by the different actions taken at these states. Sec- ond, we can easily get mirrored representation 2jzi by simply exchange the positional encoding Ca and og used in 2jsi-
We adopt widely used transformer architecture as our encoder (Dosovitskiy et al. 2021 It con- tains alternating layers of multi-head self-attention (MSA) and multi-layer perception (MLP) blocks. Layernor (LN) and residual connections are applied to the outputs of each block. For brevity, we re-write the inputs in Equation as z (0)) And the computations at each block can be represented as:
z" MSA (1-11) + (1-1)
1= L
z = MLP(LN(22) (1 (1) + +
1=1....
Xt, LN(2)).
The final pairwise policy representation Xi,j is the corresponding outputs of the aggregation token en taken from z (4)) Note that Xi,j changes to Xj when exchange the positional encodings and eg, but they are permutation invariant to the order of inputted state-action pairs.
4.2 CROWDSOURCING APPROACH TO OPR
In this section, will introduce how to train the PCT in cases regarding to the existence of ground-truth ranking labels of policies. First, we show that the policy ranking problem can be reduced to binary classification problem since pairwise policy representations used to predict the ranking of two policies. Second, we will introduce novel crowdsourcing system where multiple OPE methods modeled as workers. We will also show how to train the PCT leveraging the inaccurate labels provided by the workers.
be directly
Reducing OPR to binary classification. We first consider the case when there is a training set R consisting of T deployed policies Tद as well as their real expected accumulated
[ = rewards R. this case, we can directly construct binary labels by comparing the performance of the two policies. We use an indicator 1 >R, to represent the label of a pair of policies (æ,, nT)). The PCT can be trained by minimizing the following binary cross entropy loss:
Cxup =
E -
(1 (1,,, log(Des) (10,,59,) loog(1 - -(,))}
(5)
4

---

Under review as a conference paper at ICLR 2023
Estimated Noisy Binary Labels
Noisy Binary Labels
(m, m
Crowd Layer
Ler
Trajectory Dataset
Pairwise Policy
Representation
y
Probubility of whether n perfoms better than
OPE Worker Set
Linear Projection
Policy Comparison Transfrormer Encoder
Co
X
X ep
X! ea
x!
X ea
X ep
Aggregation Token
Policy Positional Embeddings
Input Tokens
Figure 2: The framework of SOCCER.
where Di.j sigmoid represents the predicted probability that o; performs better than Tj. @ is a function that projects Xi.j to a real number. The final ranking of test policies is based on their scores computed by:
1 N
score; =
Σ J L,..,
(6)
jzt
which can be interpreted as the expected probability that n, performs better than other test policies Rodrigo et al. (2019).
Learning from crowds by Crowd Layer. Supervised training is efficient when the dataset II' = (( ( contains enough policies. Unfortunately, collecting such training data can be extremely expensive in many real applications. Meanwhile, note that although existing OPE methods are not robust enough, they actually provide candidate solutions to the OPR problem. To this end, aim to borrow ideas from crowdsourcing domain OPR problem. Specifically, suppose that there exists a set OPE algorithms estimating the policy performance, can treat these algorithms as crowd workers for generating inaccurate labels and make use of these labels to train our models. The intuition is that the inaccurate labels generated implicitly conditioned on the ground-truth performance of policies. If
alternative way approach the
by OPE annotators can take advantage of these labels and learn their relationships with the ground-trute labels, our prediction fi.j would be more close the ground truth labels.
In the framework of SOCCER, we adopt Crowd Layer (CL, (Filipe & Pereira 2018 8 as our back- bone for learning from crowd labels. CL. is able to automatically distinguish the good from the unreliable annotators and capture their individual biases in many other domains, such as image an- notation (Guan et al., 2018 Li et al. 2022) and music genre classification (Rodrigues et al. 2013). In addition, CL is naturally compatible with deep learning approaches since it simply adds crowd layer to the deep neural networks and can be trained in an end-to-end way. As shown Figure 2 add CL to the top of our predicted probability It,.. During training, CL adjusts the gradients coming from these noisy labels according to its reliability by scaling them and adjusting their bias. The adjusted gradients then backpropagated to PCT according the chain rule.
5

---

Under review as a conference paper at ICLR 2023
Formally, assumee that there are R annotators of OPE methods. For each annotator r, its estimation about the expected return of rm is denoted as Rm The goal of CL is to train a mapping function Hi ((i.) to predict the noisy binary label generated by annotator r: Vi Log The overall objective can be written
Les.
E
Wh, log(ūT,) (1 vl.z) log(1 [T,))-]
r=1,- R
The complete training procedures of SOCCER is summarized in Algorithm|] T In practice, to reduce the computational cost brought by CL. function. Therefore, the number of additional parameters only grows linearly with the number of annotators. Note that the CL is only available during training since we still use Di, generate the predicted ranking of test policies.
( linear projection followed by a sigmoid
Algorithm 1: SOCCER Training
Data: Offline Trajectory set D, Policy set II, OPE worker set W. Policy comparison transformer g
Result:
N
The number of trainning epochs
K
The number sampled states Initialize the transformer
K
Initialize the aggregation token ep:
Initialize the policy positional embeddings and e3;
Initialize the token encoder f:
Initialize the linear projection function d; Initialize the crowd layer E # Initialize parameters while N * 0 do
Sample 2 policies: n4 FJ ~ II;
Sample K states: 81, 82. ... 88K D;
Take actions by both policies: al ~ n.(-1sk), ay ()(())));
Construct pairwise policy decision sequence:
Ti. =< ($1, ai),, ($1. ).),. ... (SK o'x), (SK, oxx) >;
Encode state-action pairs into input tokens: x [(sk, ) x², (sk,
Add policy positional embeddings to tokens:
Zo = =ieo. x + ea. x + eg, X + ea, Xxx teg
Get the pairwise policy representation: Xi., fZo);
Get the probability of whether performs better than aj: Di,j = o(xi,)):
for each OPE worker r W do
Generate the noisy labels: vī.j Imyr
Get the estimated noisy labels by crowd layer e": = (br.)):
Compute the loss according to Equation 7
Backpropagate gradients to all parameters for minimizing the loss;
end N
1;
end
EXPERIMENTS
In this section, we compare SOCCER with widely-used OPE methods on various tasks. We also present ablation studies with respect to PCT and CI., which are the main components of SOCCER.
5.1 EXPERIMENTAL SETTINGS
Trajectory Set. We evaluate SOCCER and all baseline OPE methods D4RL dataset consisting of various trajectory sets (Fu et al. 2020). These sets of trajectory data are generated by different behavioral policies in different simulated environments. Overall, we adopt trajectory sets collected

---

Under review as a conference paper at ICLR 2023
2 replay
Walker
oxpert
oxper7
fuil replay
Rank Corretaton
0666
63
0.00
1:91
0.2
0.4
0.25
0.23
0.
-0.00
-0.3
Sct11
3e1
5411
56111
Sec1
Set
2.5
0.4
0.5
6.5
3.4
03
Regret&
0.3
0.2
02
61
6-
540.1
0.0
561
545.1
50
D4l1
Set1
0.0
900
3.0
5411
501
68
5601
541
SOCCER
DualDICE
IW
FOE
MB
DR
Figure 3: Comparing SOCCER with other OPE baselines.
from environments of Mujoco games: Hopper-v2 and Walker2d-v2. Besides, there are different types of trajectory sets for each environment: expert, full-replay and medi um. The difference between them is that the behavioral policies collecting these 3 types of trajectories show different performance in the simulated environment. And these behavioral policies are trained by the Soft Actor-Critic (SAC) algorithm online (Haarnoja et al. 20188).
Policy Set. To evaluate the abilities of all methods to correctly rank a of policies in offline way. We use the policy set released by Jin et al. (2022) as the candidate set of policies. For each trajectory set mentioned above, there are 2 types of policy sets referred as Set and Set II) in which the expected return of policies are evenly spaced in the performance range of them. As mentioned in Jin et (2022). , Set and Set II aim to simulate two kinds of OPE cases. Set I consists of policies trained by offline RL algorithms (CQL (Kumar et al. (020), BEAR (Kumar) jet al. 2019), CRR (Wang et al. 2020)). Since these algorithms have different network architec- tures and the generated policies are snapshots of models that stop training at different epochs, the policies contained in Set I show diverse behavioral performance. This is aligned with practical cases where the sources of policies are diverse and unknown. On the other hand, set II con- tains policies trained by SAC, which is the same as the algorithm of training the behavioral policies. Therefore, Set. II corresponds the practical OPE cases, such as production development and update. The updated policies share many common properties with the policies generating the tra- jectory data. In the experiments of ablations, we also compare some models that use extra training policy sets for providing supervised labels. These training sets are also released by (Jin et al. 2022 and policies in them are trained by SAC online.
We compare SOCCER with six state-of-the-ae baselines. i) Fitted Q-Evaluation (FQE (Hoang al. 2019). is a value-based OPE method, which learns a neural network to approximate the functioo of the evaluated policy by temporal difference learning on the trajectory set. ii) Model-based estimation (MB (Paduraru/ (013)). It learns the dynamics model of environment, and estimates the expected return of evaluated policies by computing their average returns of Monte- Carlo rollouts the model environment. iii) Weighted importance sampling (IW (Mahmood et al.) 2014 It leverages weighted importance sampling correct the weight of the reward, regarding the collected trajectory data distribution to the data distribution of the evaluated policy. iv) Dua aualDe ICE (Nachum et al.) 2019). It also aims to achieve distribution correction yet without directly using importance sampling. It learns an estimation of the state-action stationary distribution for achieving distribution correction. v) Doubly Robust (DR (Jiang & Li, 2016). utilizes unbiased estimator that leverages an estimated environment model to decrease the variance of the unbiased estimates
Baselines I
We leverage a popular implementation of OPE algorithms: goog 1 resea http:/oooooeeesseeeeeereeeeeee// baselines used in our paper
https://github com/
icc__va It contains the first 5

---

Under review as a conference paper at ICLR 2023
HalfCheetah-r2 juti-mplay
Walker2d-v2 full-replay
modfum
riger
mediuu
1.00
1.50
1.00
1.0
Rank Corretation
0.75
0.75
0.5
0.8
0.6
0.50
0.5
04
0.6
25
0:25
0.00
0.00
0.4
0.4
-5
-0
02
-3.50
-0.50
3011
Set
0.6
0.23
0.5
0.125
Regredie
0.100
0.25
0.100
0.075
0.15
0.000
0.050
01
0.1
0.025
0.05
0.0
Set.11
0.
94t
Setll
0.0
Sec1
Sei
Set
Set
0.00
5et
5ct
0.000
50%te
50%6T
Figure 4: Comparing SOCCER with other ablations.
produced by importance sampling techniques. vi) SOPR-T Jin et al. 2022). This transformer- based model learns to achieve OPR tasks by learning a score function to score policies according to their generated state-action pairs. However, it relies on strong assumption that there are extra deployed policies well as these policies' true performance. Therefore, only compare it when there are extra supervised information.
Evaluation Metrics. We evaluate all models according two widely-used metrics. i) Spea rpea'man' rank correlation. It is the Pearson correlation between the ground truth rank sequence and the evaluated rank sequence of the evaluated policies. ii) Normalizes Regretkk. the normalized difference between the actual value of the best policy in the policy set, and the actual value of the best policy in the estimated top-k set. Mathematically. it can be computed by
It is
Vmax
Viopk
regret@k
Vmaz
Vinin"
where Vman and Vinnn the expected return the best and the worse policies, respectively, in the entire set, while Viops is the estimated top k policies.
5.2 HYPERPARAMETERS.
In the implementation of our model, we control the scale of the learning objective function in our models by controlling the optimization procedure. It is conducted using Adam with learning rate of 10-2, and with no momentum or weight decay. PCT is set as a 8-layer and 8-head transformer, and the dimensions of representations and all tokens are 128. The aggregation token eg and the policy positional embeddings e, and eg are initialized as random vectors. The token encoder f is a 3-layer MLP followed by ReLU () activation function, and its feedforward dimension is 128. The linear projection & is a single-layer MLP. During training, we set the number of training epochs as 100, and use the model snapshots at the final epoch to achieve OPR tasks in experiments. For each epoch, we sample 1280 pairwise policy decision sequences for training, and each sequence contains 256 sampled states. Besides, in each epoch, we sample 5 models from the baseline models as the OPE worker to generate noisy labels. In the inference stage, we sample 32 pairwise sequences for each policy pairs to compute the average score of each policy. All experiments in this paper are carried out with 3 different seeds, and they are 1, 2,3, respectively.
We use the official implementation of SOPR-T: http:://github.cmmm///////////
8

---

Under review as a conference paper at ICLR 2023
5.3 EXPERIMENTAL RESULTS
Comparison with Other OPE Baselines. We first compare SOCCER against 5 baselines across environments. For each environment, there are 3 different trajectory sets in which each set contains 2 distinct policy sets. So there are 12 policy sets required to correctly ranked by all models. We show the rank correlation and regret@k values of the estimated rank sequences generated by each model Figure 3
Overall, we can find SOCCER shows superior performance compared with other baselines. Further- more, among all policy sets, SOCCER shows the highest rank correlations in 10 sets and the lowest @egret@k k in 8 sets, indicating that it can achieve robust and effective performance in diverse sets of policies. By contrast, the results of other baselines shows high variance across different policy sets. On the other hand, SOCCER shows great performance on learning noisy labels generated by other OPE workers. Note that in these experiments, all OPE workers which generate noisy labels for SOC- CER are directly sampled from the trained models of baselines. We can find that SOCCER could still perform well despite of the low quality of OPE workers. For example, looking at the results in Set. of the expert trajectory set in HalfCheetah-v2, all OPE baselines show poor performance (the average rank correlation of them is about -0.27), indicating that labels generated by them are very noisy. However, SOCCER shows a high performance (about 0.62) while its supervised in- formation is from such low-quality labels. Intuitively, we believe that there are two main reasons causing this phenomenon. First, the effective representation capability of our proposed policy com- parison transformer could help to reduce the biases of noisy annotators. Second, the crowd layer could automatically distinguishes the good from the unreliable OPE annotators and captures their individual biases. In conclusion, SOCCER shows highly effective and robust OPR results across diverse kinds of policies. Besides, it can largely reduce the biases induced by OPE annotators and thus gets better performance than these OPE workers.
Ablations. To figure out the importance of each component in framework. We perform several ablations on the same policy sets as the experiments mentioned above. All results are illustrated in Figure 4 Specifically, we compare SOCCER with 3 different models. The first referred to SOCCER without CL is the model using our PCT architecture but discarding the crowd layer. Since it has module learn from crowds, so we make train it by providing extra sets of deployed policies released by (Jin et al.) 2022). The second one is SOPR-T (Jin et al., 2022). It is also a transformer- based model learn policy representations. The difference between it and SOCCER is that it learns an individual policy representation, while SOCCER learns pairwise policy representa- tions which aim to capture subtle differences between any two policies. Furthermore, SOPR-T also cannot learn from crowds, so we train it in the same supervised way. As shown in Figure SOCCER without CL shows better results than SOPR- on 10 policy sets, and SOCCER performs better than SCPR-T on 8 policy sets. This indicates that in the OPR tasks, our proposed pairwise policy representation which aims to capture relations between intra-policy decisions has stronger represen- tation ability than the pointwise policy representation, which regards each policy as individual point, without considering the subtle decision differences between policies. On the other hand, SOCCER and SOPR-T with CL both show comparable performance with SOCCER wi chout CL and SOPR-T, respectively, while SOCCER and SOPR-T with CL cannot get any supervised information from extra deployed policies. This indicates that regarding results estimated by other OPE methods as noisy labels and using a crowd layer to learn from such labels is an effective way when there no sufficient extra policies providing supervised labels.
CONCLUSIONS
This paper proposes SOCCER, a novel self-supervised learning framework for addressing OPR problems. SOCCER is the first framework that combines crowdsourcing with OPR problems. In order to capture the relative discrepancies between policies, SOCCER employs a policy compari- transformer with a novel positional encoding to learn effective pairwise policy representations. Experimental results show that SOCCER not only outperforms baselines on various tasks but also generalizes well in diverse policy sets.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
Rodrigues Filipe and Francisco Pereira. Deep learning from crowds. In Proceedings of the AAAI Conference Artificial Intelligence, pp. 161 1-1618, 2018.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: Datasets for deep data-driven reinforcement learning. arXiv, 2020.
Bellemare Marc G., Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. International Conference on Machine Learning, pp. 449-458, 2017.
Melody Guan, Varun Gulshan, Andrew Dai, and Geoffrey Hinton. Who said what: Modeling in- dividual labelers improves classification. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 3109-3118 18, 2018.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Confer- ence on Machine Learning, pp. 1861-1870, 2018.
Josiah Hanna, Niekum Scott, and Peter Stone. Importance sampling policy evaluation with an estimated behavior policy. In International Conference on Machine Learning, pp. 2605-2613, 2019.
Jean Harb, Tom Schaul, Doina Precup, and Pierre-Lue Bacon. Policy evaluation networks. arXiv, 2020.
Le Hoang, Cameron Voloshin, and Yisong Yue. Batch policy learning under constraints. In Inter- national Conference on Machine Learning, pp. 3703-3712, 2019.
Ameeviit Isac, Danica Kragic, and Johannes A. Stork. VPE: Variational policy embedding for transfer reinforcement learning. In ICRA, pp. 42-42,2222 2019.
Nan Jiang and Lihong Li. Doubly robust off-policy value evaluation for reinforcement learning. In International Conference Machine Learning, pp. 652-661, 2016.
Yue Jin, Yue Zhang, Tao Qin, Xudong Zhang, Jian Yuan, Houqiang Li, and Tie- Yan Liu. Supervised off-policy ranking. In International Conference on Machine Learning, pp. 10323-10339, 2022.
Zhang Jing, Xindong Wu, and Victor S. Sheng. Imbalanced multiple noisy labeling. Transactions on Knowledge and Data Engineering, 27(2):8995503, 2014.
Zhang Jing, Victor S. Sheng, Jian Wu, and Xindong Wu. Multi-class ground truth inference in crowdsourcing with clustering. Transactions on Knowledge and Data Engineering, 2884):1080- 1085, 2015.
Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. Stabilizing off-policy Q-Learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems, pp- 11761-11771, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-Learning for offline reinforcement learning. In Advances in Neural Information Processing Systems, pp. 1179-1191, 2020.
Junbing Li, Changqing Zhang, Joey Tianyi Zhou, Huazhu Fu, Shuyin Xia, and Qinghua Hu. Deep- lift: Deep specifs feature learning for image annotation. IEEE Transactions on Cybernetics, 777327777777 2022.
Lihong Li, Rémi Munos, and Csaba Szepesvári. Toward minimax off-policy value estimation. In Artificial Intelligence and Statistics, pp. 608-616 2015.
10

---

Under review as a conference paper at ICLR 2023
Chen Lili, Lu Kevin, Rajeswaran Aravind, Lee Kimin, Grover Aditya, Laskin Michael, Abbeel Pieter, Srinivas Aravind, and Mordatch Igor. Decision transformer: Reinforcement learning via sequence modeling. In Advances in Neural Information Processing Systems, pp. 15084-15097, 2021.
Qiang Liu, Lihong Li, Ziyang Tang, and Dengyong Zhou. Breaking the curse of horizon: Infinite- horizon off- off-policy estimation. In Advances in Neural Information Processing Systems, pp. 5361-5371, 2018.
A. Rupam Mahmood, Hado P van Hasselt, and Richard S Sutton. Weighted importance sampling for off-policy learning with linear function approximation. Advances in Neural Information Processing Systems, 2014.
Janner Michael, Qiyang Li, and Sergey Levine. Offline reinforcement learning as big sequence modeling problem. In Advances in Neural Information Processing Systems, pp. 1273-1286, 2021.
Ofir Nachum, Yinlam Chow, Bo Dai, and Lihong Li. DualDICE: Bchavior-agnostic estimation of discounted stationary distribution corrections. In Advances in Neural Information Processing Systems, 2019.
Xinkun Nie, Emma Brunskill, and Stefan Wager. Learning when-to-treat policies. Journal of the American Statistical Association, 116(533):392-409, 2021.
Michael Oberst and David Sontag. Counterfactual off-policy evaluation with gumbel-max structural
causal models. In International Conference on Machine Learning, pp. 4881-4890, 2019.
Cosmin Paduraru. Off-policy evaluation in Markov decision processes. PhD thesis, McGill Univer- sity Libraries, 2013.
Nogueira Rodrigo, Wei Yang, Kyunghyun Cho, and Jimmy Lin. Multi-stage document ranking with bert. arXiv preprint arXiv: 14424, 2019.
Filipe Rodrigues, Francisco Pereira, and Bernardete Ribeiro. Learning from multiple annotators: Distinguishing good from random labelers. Pattern Recognition Letters, 34(12):1428 14211114286666,- 2013. ISSN 0167-8655.
Sheng Victor S. and Jing Zhang, Machine learning with crowdsourcing: A brief summary of the past research and future directions. Proceedings of the AAAI Conference on Artificial Intelligence, pp. 9837-9843, 2019.
Hongyao Tang, Zhaopeng Meng. Jianye Hao, Chen Chen, Daniel Graves, Dong Li, Changmin Yu, Hangyu Mao, Wulong Liu, Yaodong Yang, Wenyuan Tao, and Li Wang. What about inputting policy in value function: Policy representation and policy-extended value function approximator. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 8441-8449 2022.
Tian Tian and Jun Zhu. Max-margin majority voting for learning from crowds. In Advances in Neural Information Processing Systems, pp. 1621-1629, 2015.
Ziyu Wang, Alexander Novikov, Konrad Zolna, Josh S Merel, Jost Tobias Springenberg. Scott Reed, Bobak Shahriari, Noah Siegel, Caglar Gulcehre, Nicolas Heess, and Nando de Freitas. Critic regularized regression. In Advances Neural Information Processing Systems, pp. 7768- 7778, 2020.
Zhang Yuchen, Xi Chen, Dengyong Zhou, and Michael Jordan. Spectral methods em: A provably optimal algorithm for crowdsourcing. JMLR, 17(1):3537-3580 2016.
Siyuan Zhang and Nan Jiang. Towards typerparameter-ffee policy selection for offline reinforce-
ment learning. In Advances in Neural Information Processing Systems, 2021.
Chen Zhu, Wei Ping, Chaowei Xiao, Mohammad Shoeybi, Tom Goldstein, Anima Anandkumar, and Bryan Catanzaro. Long-short transformer: Efficient transformers for language and vision. In Advances in Neural Information Processing Systems, pp. 17723-17736, 2021.
11


Reviews:
[{"review_id": "swAOZMHVuK", "reviewer": "Reviewer_jfpk", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new method for ranking of offline RL policies with off-policy evaluation (OPE). The ranking is produced with a model that 1) learns a pairwise policy representation with a transformer architecture, 2) uses a crowd layer to aggregate OPE scores of other methods. In the experimental results the authors show that their method is able to outperform the other baselines. The ablation studies show the importance of various components of the proposed method.", "summary_of_the_review": "I am leaning toward rejecting this paper mainly because I find the experiments lacking comparable baselines that would benefit from aggregating the results of the existing methods in the same way as the proposed method. Also, I would also like to see some generalization of the method or policy representation to the unseen policies that would make the method scalable to real world problems.\n\n---\nUpdated my score after rebuttal in the light of new empirical results.", "strength_and_weaknesses": "Strengths:\n\n- The paper is well written and easy to follow. The details of the method and the experiments are clearly explained. The figures are informative and well explained.\n- The idea of using pairwise policy representation sounds interesting and novel.\n- The experimental results where the proposed method outperformed the other baseline is very encouraging.\n- The experimental results studies the problem with different settings.\n\nWeaknesses:\n\n- To my mind, the experimental results are lacking an adequate baseline that is comparable with the proposed method. A baseline would be comparable if it also uses other existing OPE methods to aggregate the results. For example, I can imagine several easy baselines in this case: 1) take the average OPE scores of all methods and produce a ranking out of them, 2) use a majority voting scheme to aggregate the rankings, 3) there are many rank aggregation methods that could be considered, for example, [1]. \n- As the proposed method aggregates the ranks from the existing policies, the computational cost for it is much higher than for any other method and it includes the costs of all other methods. This should also be discussed. \n- Another limitation in the current experiments is that as far as I understand the method needs to be trained for every new set of policies and the environment from scratch. Then, it is tested on its own training set (no validation or test set). Do I understand the setting correctly? Would the policy representations generalize across different sets of policies? Suppose a new policy is added to the set of policies, can the previous results be re-used? To me, the method would be useful in practice if it can show signs of such generalization.\n- I do not understand why transformers are the best architecture in the given policy representation design. As the states (equation 1) are chosen as just a set (not ordered), what is the advantage of using a transformer which is known to be the best suited for sequential data? Did the authors consider other architectures (possibly simpler, e.g., MLP) here?\n\nOther comments:\n- Several times the authors mention that in practice finding the best policy is the main objective. In that case, it would be more logical to consider off-policy policy selection (OPS) problem formulation and as the quality metric measure the regret @1. How would the method perform in that case? \n- I still do not understand the role and training of the \"aggregation token\" very well, maybe this could be explained further.\n- In the first part of section 4.2 the authors say that they \"show how the policy ranking can be reduced to binary classification\". I think this is a common way to approach the ranking problem (but the text sounds now like this is one of the contributions). Some work could be references here, for example, [2].\n\n[1] Fast and Accurate Inference of Plackett–Luce Models. Lucas Maystre, Matthias Grossglauser. NIPS 2015.\n\n[2] Preference Learning with Gaussian Processes. Wei Chu, Zoubin Ghahramani. ICML 2005.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "TKlTZz6cE08", "reviewer": "Reviewer_5DpY", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors present an Off-Policy Evaluation (OPE) method geared toward ranking policies w.r.t their estimated performance, as compared to the standard task of predicting the estimated performance absolutely. It is more aligned with the OPE end goal, i.e. ranking policies among a set of candidates. \n\nPrevious work on off-policy ranking SOPR-T (Jin et al., 2022) used supervised learning for ranking policies, but it assumes access to multiple deployed policies with reward signals available. It is argued by the authors that this assumption is impractical, as it assumes multiple deployed policies along with their rewards in the log data. \n\nAuthors instead propose a self-supervised learning method, based on the 'learning from crowd' paradigm, where they assume different OPE methods as workers in a crowd-sourcing setting, with each worker generating noisy labels. A previous work, 'Crowd Layer (CL)' from learning via crowdsourcing literature is used to learn with the noisy label setting. ", "summary_of_the_review": "- Authors propose an off-policy ranking method, which directly predicts the ranking between two policies, using a ranking method. \n- Assuming a setting where access to multiple logging policies is restricted, the proposed method can leverage existing OPE methods in a pseudo-crowd-sourcing setup, with each OPE method as a worker in the crowd, and learning from the workers in a noisy label setting. ", "strength_and_weaknesses": "Strengths of the paper:\n\n- The problem of OPE via policy ranking is very practical and bypasses the requirement for estimating the absolute policy value (expected reward), which is known to be a difficult problem. \n- In practical settings, like recommender systems, ultimately we need a ranked list of policies, according to their estimated 'online' performance. \n- The method can be used in a setting where access to multiple deployed policies with corresponding rewards is not feasible, for ex: in a clinical trial. \n- Existing work from crowd-sourcing is used to learn from existing OPE methods while assuming they generate noisy labels, which is a fair assumption. \n\nSome questions to the authors: \n- How is a transformer-part important to the off-policy ranking pipeline? For ranking policies, have you compared with existing Learning-to-rank methods like LambdaMART on top of features extracted from trajectories of the policies, two-tower network (like Neural Matrix Factorization) with say RNN based feature extraction for both heads? For a given trajectory, a listwise method like lamdaMART can rank all the policies at a time, instead of a pairwise comparison. \n- A clarification question (sorry if I missed something obvious): What is the dimensionality of $x_k$s (Eq. 2)? Is it a scaler, or a vector? Since you are adding a 2-d $e_\\alpha$ vector to it, I am assuming it's a 2-D vector?  \n- In a setting like a recommender system/search, where you have access to a large log data with multiple policies and their rewards, how does this method compare with the baseline \"SOPR-T\"?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "MBQw9k_4C8p", "reviewer": "Reviewer_KHEq", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "To fulfill OPE task, this paper simplifies OPE into OPE and proposes an end-to-end method, namely SOCCER. SOCCER is compared with several baselines in two environments. Experimental results show that the proposed method achieves high accuracy with certain hyper-parameters. An ablation experiment is also performed.", "summary_of_the_review": "This paper simplifies the OPE task into an OPR task and proposes an end-to-end method, namely SOCCER, to tackle the problem. SOCCER contains policy representation (pairwise), a feature extractor (MLP and MSA), and a training strategy (Crowd Layer). SOCCER is compared with several baselines in two environments with three modes. The process of the experiment is relatively complete. An ablation experiment is also performed, while some settings make the results not very convincing. Experimental results show that the proposed method can achieve good performance in some conditions. Although this work lacks theoretical innovation and the representation of this paper has some flaws, the entire work is relatively complete and the OPE problem is relatively solved.", "strength_and_weaknesses": "Strength:\n1. This paper simplifies the OPE into OPR tasks.\n2. This paper proposes an end-to-end method to solve the OPR.\n3. The proposed method is relatively technical sound.\n\nWeaknesses:\n1. The experiment settings lack some explanations, e.g. the reason for the specific value of hyper-parameters.\n2. The figures need to be improved, e.g. sub-figures in Fig.4 are not aligned.\n3. Ablation experiments as well as baselines need to be considered more carefully. From my point of view, the essence of the proposed method is similar to a label aggregation method with deep learning. Thus, some other structure-like method should also be added, e.g. SpeeLFC. The ablation experiment introduces an extra strategy to obtain ‘truth’ to train the method. This setting lowers the convincing performance of the proposed method.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "1IHDfOUEXlL", "reviewer": "Reviewer_RMFP", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the problem of offline policy evaluation. The key idea is that instead of predicting the value of the target policy, the authors proposed to evaluate the relative rank of the policy, among a set of candidate policies. When there is no sufficient historical observations of different behavior policies, the authors proposed to leverage the idea of crowd-sourcing to integrate prediction from multiple offline policy evaluators. The final ranking/scoring of each target policy is obtained by aggregating the pairwise comparisons among all candidate policies. Experiments on a set of simulated offline trajectory data were performed against a rich collection of offline policy evaluation methods. ", "summary_of_the_review": "The idea of studying offline policy ranking, instead of evaluation, is an interesting and practical idea. But the proposed solution is overly simplified and lacks necessary theoretical justification or analysis. It is hard to know when the algorithm would work better than standard off-policy evaluation methods.  ", "strength_and_weaknesses": "Strength:\n+ The idea of off-policy rank evaluation is reasonable and valid, which to my best knowledge is first time discussed in literature. Although only high-level intuitive arguments are provided regarding the advantage of this new objective, it is still nice to observe its effectiveness in the reported empirical studies. \n+ The reported empirical studies provide a comprehensive picture about the comparison between the proposed solution and a rich set of baseline methods.  \n\nWeakness:\n- The proposed method lacks necessary theoretical justification, with unknown properties about the provided estimations, e.g., there is no way to quantify its bias and variance. \n- The solution boils down to a binary classification problem, under which the crowd-sourcing idea is a natural extension when only noisy labels exist. Following this logic, methods for learning from noisy labels can also be leveraged to address the problem. \n- The empirical nature of the proposed solution creates a large set of hyper-parameters, which make the tuning and comparison hard to exhaust. For example, presumably the number of states where we execute the policies to obtain their actions, the number of pairwise policy comparisons, and the number of OPE workers are important for the performance of the proposed solution. But there is no experiment evaluating the impact from such hyper-parameters.   \n\nQuestions:\n- In Section 5.2, it is mentioned that in each epoch, 5 OPE workers are randomly selected from the baseline models. I am not sure why we should sample different OPE workers every epoch. Shouldn’t they be the same throughout the training and testing stages for the crowd layer to learn the (equivalent) confusion matrix?  \n- What’s the principle to determine the number of states where we execute the policies to obtain their actions, the number of pairwise policy comparisons, and the number of OPE workers in practice? Are they the more the better?\n- Why not have a dedicate policy encoder to represent each policy and then compare their embeddings for binary classification? Or we can simply follow the way we use transformer to encode two sentences into one embedding to embed the two policies. The conventional position embedding in transformer can help us realize the corresponding positions are actions from two policies but under the same states.\n- My understanding about $e_\\alpha$ and $e_\\beta$ is that they are fixed one-hot vectors. But Section 5.2 described them as random vectors at initialization. Clarification is necessary here.  \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject