Title: Uncertainty-Aware Instance Reweighting for Off-Policy Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: off-policy learning

uncertainty
Abstract: Off-policy learning, referring to the procedure of policy optimization with access only to logged feedback data, has shown importance in various important real-world applications, such as search engines and recommender systems. While the ground-truth logging policy is usually unknown, previous work simply takes its estimated value for the off-policy learning, ignoring the negative impact from both high bias and high variance resulted from such an estimator. And these impact is often magnified on samples with small and inaccurately estimated logging probabilities. The contribution of this work is to explicitly model the uncertainty in the estimated logging policy, and propose an Uncertainty-aware Inverse Propensity Score estimator (UIPS) for improved off-policy learning, with a theoretical convergence guarantee. Experiment results on the synthetic and real-world recommendation datasets demonstrate that UIPS significantly improves the quality of the discovered policy, when compared against an extensive list of state-of-the-art baselines.

Uncertainty-Aware Instance Reweighting for Off-Policy Learning

Abstract

Off-policy learning, referring to the procedure of policy optimization with access only to logged feedback data, has shown importance in various real-world applications, such as search engines and recommender systems. While the ground-truth logging policy is usually unknown, previous work simply employs its estimated value for the off-policy learning, ignoring the negative impact from both high bias and high variance resulted from such an estimator. And such impact is often magnified on samples with small and inaccurately estimated logging probabilities. The contribution of this work is to explicitly model the uncertainty in the estimated logging policy, and propose an Uncertainty-aware Inverse Propensity Score estimator (UIPS) for improved off-policy learning, with a theoretical convergence guarantee. Experiment results on the synthetic and real-world recommendation datasets demonstrate that UIPS significantly improves the quality of the discovered policy, when compared against an extensive list of state-of-the-art baselines.

1 Introduction

In many real-world applications, including search engines [2], online advertisements [35], recommender systems [8, 22], only logged data is available for subsequent policy learning. For example, in recommender systems, various complex recommendation policies are optimized over logged user interactions (e.g., clicks or stay time) with items recommended by previous recommendation policies (referred to as the logging policy ) [51, 14]. However, such logged data is often known to be biased, since the feedback on items where the logging policy did not take is unknown. This inevitably distorts the evaluation and optimization of a new policy when it differs from the logging policy.

Off-policy learning [41, 27] thus emerges as a preferred way to learn an improved policy only from the logged data, by addressing the mismatch between the learning and logging policies. One of the most commonly used off-policy learning methods is the Inverse Propensity Scoring (IPS) [8, 25], which assigns per-sample importance weight (i.e., propensity score) to the training objective on the logged data, so as to get an unbiased optimization objective in expectation. The importance weight in IPS is the probability ratio of taking an action between the learning and logging policies.

Unfortunately, the ground-truth logging policy is oftentimes unavailable to the learner in practice, due to reasons like legacy issues, i.e., it was not recorded in the data. Additionally, in specific situations like the healthcare domain [28] or two-stage recommender systems [8], access to the ground-truth logging policy is not feasible. One common treatment by many previous studies [35, 22, 8, 24] is to first estimate the logging policy using a supervised learning method (e.g., logistic regression,

1

1

2

3

4

5

6

7

8

9

0

1

2

3

4

5

·

10

-

2

Log(item freq)

Estimated Logging Probability

(a) Estimated Logging Probability

1

2

3

4

5

6

7

8

9

0

2

4

6

8

·

10

-

3

Log(item freq)

Uncertainty

(b) Uncertainty of Estimation

neural networks, etc.), and then employ the estimated logging policy for off-policy learning. In this work, we first show that such an approximation results in a biased estimator which is sensitive to data with small estimated logging probabilities. Worse still, small estimated logging probabilities usually suggest there are limited related samples in the logged data, whose estimations can have high uncertainties, i.e., being wrong with a high probability. Figure 1 shows a piece of empirical evidence from a large-scale recommendation benchmark KuaiRec dataset [12], where items with lower frequencies in the logged dataset have lower estimated logging probabilities (via a neural network estimator) and higher uncertainties at the same time. The high bias and variance caused by these samples can greatly hinder the performance of subsequent off-policy learning. We defer detailed discussions of this result in Section 2.

In this work, we explicitly take the uncertainty of the estimated logging policy into consideration and design an Uncertainty-aware Inverse Propensity Score estimator (UIPS) for off-policy learning. UIPS reweighs the propensity score of each logged sample to control its impact on policy optimization, and learns an improved policy by alternating between: (1) Find the optimal weight that makes the estimator as accurate as possible, based on the uncertainty of the estimated logging policy; (2) Improve the policy by optimizing the resulting objective function. The optimal weight for each sample is obtained by minimizing the upper bound of the mean squared error (MSE) to the ground-truth policy evaluation, with a closed-form solution. Furthermore, UIPS ensures that off-policy learning converges to a stationary point where the true policy gradient is zero; while convergence may not be guaranteed when directly using the estimated logging policy. Extensive experiments on a synthetic and three realworld recommendation datasets against a rich set of state-of-the-art baselines demonstrate the power of UIPS. All data and code can be found in https://github.com/Xiaoyinggit/UIPS.git .

2 Preliminary: off-policy learning

We focus on the standard contextual bandit setup to explain the key concepts in UIPS. Following the convention [16, 29, 36], let x ∈ X ⊆ R d be a d -dimensional context vector drawn from an unknown distribution p ( x ) . Each context is associated with a finite set of actions denoted by A , where |A| < ∞ . Let π : A×X → [0 , 1] denote a stochastic policy, such that π ( a | x ) is the probability of selecting action a under context x and ∑ a ∈A π ( a | x ) = 1 . Under a given context x , the reward r x ,a is only observed when action a is chosen, i.e., bandit feedback. Without loss of generality, we assume r x ,a ∈ [0 , 1] . Let V ( π ) denote the expected reward of the policy π :

We look for a policy π ( a | x ) to maximize V ( π ) . In the rest, we denote E x ∼ p ( x ) ,a ∼ π ( a | x ) [ · ] as E π [ · ] .

In off-policy learning, one can only access a set of logged feedback data D := { ( x n , a n , r x n ,a n ) | n ∈ [ N ] } . Given x n , the action a n was generated by a stochastic logging policy β ∗ , i.e., a n ∼ β ∗ ( a | x n ) , which is usually different from the learning policy π ( a | x ) [24, 40, 8]. The actions { a 1 , . . . , a N } and their corresponding rewards { r x 1 ,a 1 , . . . , r x N ,a N } are generated independently given β ∗ . The main challenge is then to address the distributional discrepancy between β ∗ ( a | x ) and π ( a | x ) , when optimizing π ( a | x ) to maximize V ( π ) with access only to the logged dataset D .

One of the most widely used methods to address the distribution shift between π ( a | x ) and β ∗ ( a | x ) is the Inverse Propensity Scoring (IPS) [8, 25]. One can easily get that:

yielding the following empirical estimator of V ( π ) :

where π ( a n | x n ) /β ∗ ( a n | x n ) is referred to as the propensity score. Various algorithms can be readily used for policy optimization under ˆ V IPS ( π ) , including value-based methods [33] and policy-based methods [19, 31, 42]. In this work, we adopt a well-known policy gradient algorithm, REINFORCE [42]. Assume the policy π ( a | x ) is parameterized by ϑ , via the 'log-trick', the gradient of ˆ V IPS ( π ϑ ) with respect to ϑ can be readily derived as,

Approximation with an unknown logging policy . In many real-world applications, the ground-truth logging probabilities, i.e., β ∗ ( a | x ) of each observation ( x , a ) in D , are unknown. As a typical walk-around, previous work employs supervised learning methods such as logistic regression [30] and nerural networks [8] to estimate the logging policy, and replaces β ∗ ( a | x ) with its estimated value ˆ β ( a | x ) to get the following BIPS estimator for policy learning:

However, as shown in the following proposition, inaccurate ˆ β ( a | x ) leads to high bias and variance in BIPS. Worse still, smaller and inaccurate ˆ β ( a | x ) further enlarges this bias and variance.

Proposition 2.1. The bias and variance of ˆ V BIPS ( π ϑ ) can be derived as follows:

As a result, with high bias and variance caused by inaccurate ˆ β ( a | x ) , it is erroneous to learn π ϑ ( a | x ) by simply optimizing ˆ V BIPS ( π ϑ ) . Furthermore, this approach may also hinder the convergence of off-policy learning, as discussed later in Section 3.2.

However, smaller ˆ β ( a | x ) usually implies less number of related training samples in the logged data, and thus ˆ β ( a | x ) can be inaccurate with a higher probability. To make it more explicit, let us revisit the empirical results shown in Figure 1. We followed the method introduced in [8] to estimate the logging policy on KuaiRec dataset [12] and plotted the estimated ˆ β ( a | x ) and its corresponding uncertainties on items of different observation frequencies in the logged dataset. We adopted the method in [45] to measure the confidence interval of ˆ β ( a | x ) on each instance. A wider confidence interval, i.e., higher uncertainty in estimation, implies that with a high probability the true value may be further away from the empirical mean estimate. We can observe in Figure 1 that as item frequency decreases, the estimated logging probability also decreases, but the estimation uncertainty increases. This implies that a smaller ˆ β ( a | x ) is usually 1) more inaccurate and 2) associated with a higher uncertainty.

3 Uncertainty-aware off-policy learning

Our idea is to consider the uncertainty of the estimated logging policy by incorporating per-sample weight ϕ x ,a , and perform policy learning by optimizing the following empirical estimator:

Intuitively, one should assign lower weights to samples whose ˆ β ( a | x ) is small and far away from the ground-truth β ∗ ( a | x ) . We then divide off-policy optimization into two iterative steps:

Deriving the optimal instance weight: Find the optimal ϕ x ,a to make ˆ V UIPS ( π ϑ ) approach its ground-truth V ( π ) as closely as possible, so as to facilitate policy learning. The derived optimal weight is denoted as ϕ ∗ x ,a (see Theorem 3.2).

Policy improvement : Update the policy π ϑ ( a | x ) using the following gradient:

The whole algorithm framework and its computational cost, as well as important notations are summarized in Appendix 7.1.

3.1 Derive the optimal uncertainty-aware instance weight

We expect to find the optimal weight ϕ x,a to make the empirical estimator ˆ V UIPS ( π ϑ ) as accurate as possible, taking into account the uncertainty in estimated logging probabilities. Intuitively, a high accuracy of the estimator is crucial for determining the correct direction of policy learning. We follow previous work [36, 29] and measure the mean squared error (MSE) of ˆ V UIPS ( π ϑ ) to the ground-truth policy value V ( π ϑ ) , which captures both the bias and variance of an estimator. A lower MSE indicates a more accurate estimator.

In UIPS, instead of directly minimizing the MSE, which is intractable, we find ϕ x ,a to minimize the upper bound of MSE. As we show later, the optimal ϕ x ,a has a closed-form solution which relates to both the value of π ϑ ( a | x ) / ˆ β ( a | x ) and the estimation uncertainty of ˆ β ( a | x ) .

Theorem 3.1. The mean squared error ( MSE ) between ˆ V UIPS ( π ϑ ) and ground-truth estimator V ( π ϑ ) is upper bounded as follows:

As the first expectation term E π ϑ [ r 2 x ,a π ϑ ( a | x ) β ∗ ( a | x ) ] is a non-negative constant, we denote it as λ ∈ [0 , ∞ ) when searching for ϕ x ,a . To minimize this upper bound of MSE, the optimal ϕ x ,a for each sample ( x , a ) should minimize the following,

An interesting observation is that setting ϕ x ,a = ˆ β ( a | x ) β ∗ ( a | x ) , i.e., turning π ( a | x ) ˆ β ( a | x ) ϕ x ,a into π ( a | x ) β ∗ ( a | x ) does not result in the optimal solution of Eq.(6). This is because such a setting only reduces bias (i.e., the first term of Eq.(6)), but fails to control the second term, which is related to the variance. Moreover, we cannot directly minimize Eq.(6) due to the unknown β ∗ ( a | x ) . But it is possible to obtain a confidence interval which contains β ∗ ( a | x ) with a high probability, when ˆ β ( a | x ) is obtained via a specific estimator, e.g., (generalized) linear model or kernel methods.

Following previous work [23, 16, 22], we adopt the realizable assumption that β ∗ ( a | x ) can be represented by a softmax function applied over a parametric function f θ ∗ ( x , a ) . Moreover, the universal approximation theorem [18] states that a parametric function with sufficient capacity, when combined with a softmax function, can approximate any distribution. Then we have:

where f θ ( x , a ) is an estimate of f θ ∗ ( x , a ) . Following the conventional definition of confidence interval [20], we define γ and U x ,a such that | f θ ∗ ( x , a ) -f θ ( x , a ) | ≤ γU x ,a holds with probability

at least 1δ , where γ is a function of δ (typically the smaller δ is, the larger γ is). Then γU x ,a measures the width of confidence interval of f θ ( x , a ) against its ground-truth f θ ∗ ( x , a ) . As derived in Appendix 7.2, with probability at least 1δ , we have β ∗ ( a | x ) ∈ B x ,a and

where Z ∗ = ∑ a ′ exp( f θ ∗ ( a ′ | x )) and ˆ Z = ∑ a ′ exp( f θ ( a ′ | x )) .

As β ∗ ( a | x ) can be any value in B x ,a with high probability, we aim to find the optimal ϕ x ,a that minimizes the worst case of Eq.(6), thereby ensuring that ˆ V UIPS ( π ϑ ) approaches its ground-truth V ( π ϑ ) under the sense of MSE, even in the worst possible scenarios. This ensures the subsequent policy improvement direction will not be much worse with high probability. Thus, we formulate the following optimization problem:

The following theorem derives a closed-form formula for the optimal solution of Eq.(8).

Theorem 3.2. Let η ∈ [exp( -γU max x ) , exp( γU max x )] , where U max x = max a U x ,a . The optimization problem in Eq.(8) has a closed-form solution:

The following corollary demonstrates the advantage of UIPS. The detailed proof of Theorem 3.2 and Corollary 3.3 can be found in Appendix 7.8.

Corollary 3.3. With ϕ ∗ x ,a derived in Theorem 3.2, ˆ V UIPS ( π ϑ ) in Eq.(4) achieves a smaller upper bound of MSE than ˆ V BIPS ( π ϑ ) in Eq. (3).

Insights about ϕ ∗ x ,a . The detailed analysis of the effect of ϕ ∗ x ,a can be found in Lemma 7.1 in Appendix 7.8. In summary, we have the following key findings,

For samples whose largest possible propensity score is under control: i.e., π ϑ ( a | x ) min B x ,a < √ λ , higher uncertainty implies smaller values of π/ ˆ β . This suggests samples of this type with positive rewards are underestimated, and the extend of underestimation increases with the estimation uncertainty. UIPS thus chooses to increase ϕ ∗ x ,a with uncertainty, to emphasize these long-tail positive samples.

Conversely, for samples with large propensity scores, UIPS decreases ϕ ∗ x ,a as the uncertainty increases, so as to prevent their distortion in policy learning.

Uncertainty estimation. Now we describe how to calculate U x ,a , i.e., the uncertainty of the estimated ˆ β ( a | x ) . In this work, we choose to estimate β ∗ ( a | x ) using a neural network, because 1) its representation learning capacity has been proved in numerous studies, and 2) various ways [11, 45] can be leveraged to perform the uncertainty estimation in a neural network. We adopt [45] due to its computational efficiency and theoretical soundness. Following the proof of Theorem 4.4 in [45], given the logged dataset D , we can get with a high probability that there exists γ such that:

where g ( x n , a n ) is the gradient of f θ ( x n , a n ) with respect to the neural network's last layer's parameter θ w ⊂ θ , i.e., g ( x n , a n ) = ∇ θ w f θ ( x n , a n ) . And M D = ∑ N n =1 g ( x n , a n ) g ( x n , a n ) ⊤ , implying U x n ,a n = √ g ( x n , a n ) ⊤ M -1 D g ( x n , a n ) .

3.2 Convergence of policy learning under UIPS

The following theorem provides the convergence result for UIPS, which converges to a stationary point of the expected reward function. The proof is provided in Appendix 7.9.

Theorem 3.4. Denote G max and Φ as the maximum value of ∥ ∂π ϑ ( a | x ) ∂ϑ ∥ and E β ∗ [ π 2 ϑ ( a | x ) ˆ β 2 ( a | x ) ( ϕ ∗ x ,a ) 2 ] respectively, i.e., ∥ ∂π ϑ ( a | x ) ∂ϑ ∥ ≤ G max and E β ∗ [ π 2 ϑ ( a | x ) ˆ β 2 ( a | x ) ( ϕ ∗ x ,a ) 2 ] ≤ Φ . And denote V max as the finite maximum expected reward that can be achieved, and φ max = max x ,a {∣ ∣ ∣ β ∗ ( a | x ) ˆ β ( a | x ) ϕ ∗ x ,a -1 ∣ ∣ ∣ } . Assume that the expected reward of π ϑ , i.e., V ( π ϑ ) , is a differentiable and L-smooth function w.r.t ϑ . Denote the policy parameters obtained by Eq. (5) at iteration k ∈ [ K ] as ϑ k , then φ max ∈ (0 , 1) and

where ∇ V ( π ϑ ) is the true policy gradient under ground-truth logging probability, i.e., ∇ V ( π ϑ ) = E β ∗ [ π ϑ ( a | x ) β ∗ ( a | x ) r x ,a ∇ ϑ log( π ϑ ( a | x ))] .

Theorem 3.4 shows that, as K → ∞ and with 1 / (1 -φ max ) and Φ being controlled, UIPS leads policy update to converge to a stationary point where the true policy gradient ∇ V ( π ϑ k ) is zero. And fortunately, UIPS is effective in controlling both 1 / (1 -φ max ) and Φ . Specifically, we denote φ x ,a = ∣ ∣ ∣ β ∗ ( a | x ) ˆ β ( a | x ) ϕ ∗ x ,a -1 ∣ ∣ ∣ and Φ x,a = π 2 ϑ ( a | x ) ˆ β 2 ( a | x ) ( ϕ ∗ x ,a ) 2 . It is clear to note that λφ 2 x,a +Φ x,a corresponds to the objective in Eq.(6) for deriving ϕ ∗ x ,a for each sample ( x , a ) . In other words, UIPS selects { ϕ ∗ x ,a } to minimize φ max = max { φ x,a } and Φ = E β ∗ [Φ x ,a ] , which directly accelerate the policy converge to a stationary point with the true policy gradient being zero.

In the case of BIPS in Eq.(3), we have ϕ x ,a ≡ 1 . Although Φ may be large due to small logging probabilities, the more concerning issue is that the requirement φ max ∈ (0 , 1) is no longer satisfied when β ∗ ( a | x ) ≥ 2 ˆ β ( a | x ) , which may happen with a non-negligible probability. Hence, the convergence of policy learning under ˆ V BIPS is no better than that under UIPS.

4 Empirical evaluations

We evaluate UIPS on both synthetic data and three real-world datasets with unbiased collection. We compare UIPS with the following baselines, which can be grouped into five categories:

Cross-Entropy (CE) : A supervised learning method with the cross-entropy loss over its softmax output. No off-policy correction is performed in this method.

BIPS-Cap [8]: The off-policy learning solution under the BIPS estimator in Eq.(3). The estimated propensity scores are further suppressed to control variance, i.e., taking min ( c, π ϑ ( a | x ) ˆ β ( a | x ) ) as the propensity score. Setting c to a small value can reduce variance, but introduces bias.

MinVar & stableVar [46], Shrinkage [36]: This line of work improves off-policy evaluation by reweighing each sample. For example, MinVar and stableVar reweigh each sample by h x ,a ∑ a ′ h x ,a ′

methods simply treat ˆ β ( a | x ) as β ∗ ( a | x ) , and none of them consider the uncertainty of ˆ β ( a | x ) .

with h x ,a = ˆ β ( a | x ) π ϑ ( a | x ) 2 and h x ,a = √ ˆ β ( a | x ) π ϑ ( a | x ) respectively, since they find that π ϑ ( a | x ) 2 / ˆ β ( a | x ) is directly related to policy evaluation variance. Su et al. [36] propose to shrink the propensity score by λ/ ( λ + π ϑ ( a | x ) 2 ˆ β ( a | x ) 2 ) , which is a special case of our UIPS with U x ,a = 0 and η = 1 . All these

SNIPS [39], BanditNet [16], POEM [38], POXM [23], Adaptive [22]: This line of work aims for more stable and accurate policy learning. For example, SNIPS normalizes the estimator by the sum of propensity scores in each batch. BanditNet extends SNIPS and leverages an additional Lagrangian term to normalize the estimator by an approximated sum of propensity scores of all samples. POEM jointly optimizes the estimator and its variance. POXM controls estimation variance by pruning samples with small logging probabilities. Adaptive proposes a new formulation to utilize negative samples.

ApproxKNN [5] and IPS-C-TS : The line of work improves off-policy learning by applying calibration to estimated logging probabilities. ApproxKNN utilizes the K-Nearest Neighbor algorithm for calibration, which exhibits the lowest calibration error in [5]. IPS-C-TS, on the other hand, employs temperature scaling, a widely recognized and effective calibration method for probability distribution [13].

τ

= 0

.

5

τ

= 1

τ

= 2

Algorithm

P@5

R@5

NDCG@5

P@5

R@5

NDCG@5

P@5

R@5

NDCG@5

IPS-GT

0.5589

±

1

e

-

3

0.1582

±

6

e

-

4

0.6093

±

1

e

-

3

0.5526

±

2

e

-

3

0.1565

±

6

e

-

4

0.6007

±

1

e

-

3

0.5531

±

2

e

-

3

0.1557

±

7

e

-

4

0.6037

±

1

e

-

3

CE

0.5553

±

6

e

-

4

0.1573

±

2

e

-

4

0.6037

±

5

e

-

4

0.5510

±

6

e

-

4

0.1561

±

2

e

-

4

0.5995

±

4

e

-

4

0.5386

±

2

e

-

3

0.1524

±

7

e

-

4

0.5874

±

2

e

-

3

BIPS-Cap

0.5515

±

2

e

-

3

0.1553

±

8

e

-

4

0.6031

±

2

e

-

3

0.5526

±

2

e

-

3

0.1561

±

6

e

-

4

0.6016

±

1

e

-

3

0.5409

±

3

e

-

3

0.1529

±

9

e

-

4

0.5901

±

2

e

-

3

MinVar

0.5340

±

2

e

-

3

0.1509

±

6

e

-

4

0.5857

±

2

e

-

3

0.5282

±

2

e

-

3

0.1491

±

7

e

-

4

0.5791

±

2

e

-

3

0.5036

±

4

e

-

3

0.1415

±

1

e

-

3

0.5543

±

3

e

-

3

stableVar

0.4577

±

5

e

-

3

0.1310

±

1

e

-

3

0.5111

±

2

e

-

3

0.5373

±

3

e

-

3

0.1523

±

9

e

-

4

0.5866

±

3

e

-

3

0.5279

±

3

e

-

3

0.1492

±

8

e

-

4

0.5781

±

3

e

-

3

Shrinkage

0.5526

±

2

e

-

3

0.1562

±

7

e

-

4

0.6024

±

1

e

-

3

0.5499

±

4

e

-

3

0.1545

±

1

e

-

3

0.6040

±

3

e

-

3

0.5347

±

2

e

-

3

0.1513

±

6

e

-

4

0.5824

±

2

e

-

3

SNIPS

0.2616

±

6

e

-

2

0.0749

±

2

e

-

2

0.3150

±

7

e

-

2

0.3538

±

5

e

-

2

0.0987

±

1

e

-

2

0.4144

±

6

e

-

2

0.4379

±

3

e

-

2

0.1226

±

9

e

-

3

0.5177

±

3

e

-

2

BanditNet

0.4011

±

3

e

-

2

0.1131

±

8

e

-

3

0.4830

±

2

e

-

2

0.3894

±

4

e

-

2

0.1095

±

1

e

-

2

0.4741

±

3

e

-

2

0.4122

±

3

e

-

2

0.1153

±

8

e

-

3

0.4934

±

3

e

-

2

POEM

0.5480

±

2

e

-

3

0.1539

±

8

e

-

4

0.6008

±

2

e

-

3

0.5502

±

2

e

-

3

0.1551

±

6

e

-

4

0.6000

±

2

e

-

3

0.5399

±

2

e

-

3

0.1526

±

8

e

-

4

0.5893

±

2

e

-

3

POXM

0.4006

±

3

e

-

2

0.1130

±

8

e

-

3

0.4828

±

2

e

-

2

0.3616

±

4

e

-

2

0.1019

±

1

e

-

2

0.4522

±

4

e

-

2

0.3816

±

4

e

-

2

0.1069

±

1

e

-

2

0.4680

±

4

e

-

2

Adaptive

0.3831

±

2

e

-

2

0.1050

±

4

e

-

3

0.4382

±

2

e

-

2

0.4734

±

4

e

-

3

0.1325

±

1

e

-

3

0.5326

±

3

e

-

3

0.3936

±

1

e

-

2

0.1097

±

4

e

-

3

0.4368

±

2

e

-

2

ApproxKNN

0.5576

±

1

e

-

3

0.1580

±

4

e

-

4

0.6059

±

2

e

-

3

0.5527

±

9

e

-

4

0.1567

±

1

e

-

4

0.6010

±

1

e

-

3

0.5409

±

2

e

-

3

0.1532

±

6

e

-

4

0.5890

±

1

e

-

3

IPS-C-TS

0.5565

±

1

e

-

3

0.1577

±

3

e

-

4

0.6048

±

7

e

-

4

0.5517

±

6

e

-

4

0.1563

±

2

e

-

4

0.6002

±

6

e

-

4

0.5393

±

1

e

-

3

0.1526

±

5

e

-

4

0.5879

±

1

e

-

3

UIPS-P

0.4019

±

3

e

-

2

0.1131

±

1

e

-

2

0.4831

±

3

e

-

2

0.3904

±

4

e

-

2

0.1096

±

1

e

-

2

0.4749

±

3

e

-

2

0.4109

±

3

e

-

2

0.1149

±

1

e

-

2

0.4922

±

3

e

-

2

UIPS-O

0.4135

±

4

e

-

2

0.1167

±

1

e

-

2

0.4954

±

4

e

-

2

0.3896

±

4

e

-

2

0.1096

±

1

e

-

2

0.4739

±

3

e

-

2

0.4519

±

3

e

-

2

0.1268

±

8

e

-

3

0.5296

±

2

e

-

2

UIPS

0.5608

±

2

e

-

3

0.1589

±

8

e

-

4

0.6113

±

3

e

-

3

0.5572

±

2

e

-

3

0.1571

±

8

e

-

4

0.6074

±

2

e

-

3

0.5432

±

3

e

-

3

0.1534

±

8

e

-

4

0.5946

±

2

e

-

3

p-value

3

e

-

3

1

e

-

2

4

e

-

5

2

e

-

5

2

e

-

1

4

e

-

10

1

e

-

1

5

e

-

1

4

e

-

2

UIPS-P and UIPS-O : These are two variants of our UIPS with different ways of leveraging uncertainties. UIPS-P directly penalizes samples whose estimated logging probabilities are of high uncertainty, i.e., taking ϕ x ,a = 1 . 0 / exp( γU x ,a ) , which follows previous work on offline reinforcement learning [43, 4]. UIPS-O adversarially uses the worst propensity score for policy learning, i.e., ϕ x ,a = 1 . 0 / exp( -γU x ,a ) .

4.1 Synthetic data

Data generation. Following previous work [24, 23], we generate a synthetic dataset by a superviseto-bandit conversion on Wiki10-31K dataset [7], which is an extreme multi-label classification dataset. The Wiki10-31K dataset contains approximately 20K samples. Each sample is associated with a feature vector ˜ x of 101,938 dimensions and a label vector y ˜ x of 31K classes with more than one positive class. Let y ˜ x ,a denote the label of class a under ˜ x , and we take each class as an action. The huge action space creates great challenges in off-policy learning, e.g., sparse observations, and therefore better evaluates different methods.

We split the dataset into train, validation and test sets with size 11K:3K:6K. The test set is from the official split. Since the original feature vector ˜ x is too sparse, for ease of learning, we first embedded it to dimension d by x = W ˜ x , and synthesized the ground-truth logging policy β ∗ ( a | x ) by:

where W and { θ ∗ a } are pre-trained parameters by applying a logistic regression model on the train set, τ is a positive hyper-parameter that controls the skewness of logging distribution. A small value of τ leads to a near-deterministic logging policy, while a larger τ makes it flatter. More implementation details can be found in Appendix 7.3.

Evaluation metrics . To evaluate the learned policy π ϑ ( a | x ) , we calculate Precision@K (P@K), Recall@K (R@K) and NDCG@K following previous work [23, 24]. Higher P@K, R@K and NDCG@K imply a better policy.

Effectiveness of policy learning. Table 1 shows the average performance and standard deviations of all algorithms under 10 random seeds on three synthetic datasets generated under different τ . As the ground-truth logging policy is accessible on the synthetic datasets, we included a new baseline IPSGT, which uses the IPS estimator with the ground-truth logging probabilities. We calculated p -value under t-test between UIPS and the best baseline to investigate the significance of improvement.

First, we can observe that UIPS achieved similar and even better performance than IPS-GT when τ = 0 . 5 and τ = 1 , but performed worse than IPS-GT when τ = 2 . Despite using ground-truth

logging probabilities, IPS-GT still suffered from high variance caused by samples with small logging probabilities, which is the main cause of its worse performance when τ = 0 . 5 and τ = 1 . In contrast, UIPS effectively controlled the negative impact of these high-variance samples, resulting in a better bias-variance trade-off.

With an increasing τ , suggesting a decrease in the probability of selecting positive actions, most algorithms experienced a drop in performance. However, UIPS consistently outperformed all other algorithms across all three datasets and metrics. Interestingly, as τ decreases, the performance improvement of UIPS became even more pronounced, despite SNIPS, BanditNet, and POXM being designed to handle small logging probabilities of positive actions.

ApproxKNN and IPS-C-TS generally achieved better performance than BIPS-Cap, implying the effectiveness of calibration of estimated logging probabilities. However, UIPS still consistently outperformed both ApproxKNN and IPS-C-TS. The main reason is that calibration primarily focuses on adjusting the estimated probabilities to ensure on average the model's predictions are reliable and accurate. In contrast, UIPS specifically handles the impact from each individual sample in policy learning.

UIPS also consistently outperformed Shrinkage (a special case of UIPS with uncertainties always being zero) on all three datasets, demonstrating the benefits of considering the estimation uncertainty. Finally, blindly reweighing through uncertainties, regardless of their impact on the accuracy of the resulting estimator and the learned policy, ultimately resulted in poor performance, as demonstrated by UIPS-P and UIPS-O.

Performance under different uncertainty levels. As shown in Figure 1, low-frequency samples in the logged dataset suffer higher uncertainties in their propensity estimation. Thus, we divided the test set into two subsets according to the average frequency of associated actions, where the uncertainty in the subset associated with low-frequency actions is on average 8% higher than that in high-frequency actions. Table 2 shows the results on these two subsets when τ = 0 . 5 . In addition, we include the results of the top three baselines that directly utilize the estimated logging policy. Table 2 clearly demonstrates that only UIPS performed better than CE on the test set with low-frequency actions, implying the distortion of inaccurately estimated logging probabilities and the effectiveness of UIPS in efficiently handling them.

Off-policy Evaluation. We further inspected whether ˆ V UIPS in Eq.(4) leads to more accurate offpolicy evaluation. Following previous work [29, 46, 36], we evaluated the following ϵ -greedy policy: π ( a | x ) = 1 -ϵ | M x | · I { a ∈ M x } + ϵ/ |A| , where M x contains all positive actions associated with instance x . For each x in the test set, we randomly sample 100 actions following the logging policy in Eq.(9) to generate the logged dataset. Table 3 shows the MSE of the estimators to the ground-truth policy value under 20 different random seeds. From Table 3, one can observe that: 1) IPS-GT with a skewer logging policy (i.e., smaller τ ) leads to higher MSE, consistent with previous findings [29, 46, 36]; 2) inaccurate logging probabilities result in high bias and variance, leading to much larger MSE of BIPS compared to IPS-GT. Furthermore, this distortion is particularly pronounced when the ground-truth logging policy is skewed ( τ = 0 . 5 ); and 3) although all using the estimated logging policy, ˆ V UIPS yields the smallest MSE, comparing to other baselines that are designed to improve over BIPS.

Hyper-parameter Tuning. Discussions about hyper-parameter tuning and performance of UIPS under different hyper-parameters can also be found in Appendix 7.3.1.

4.2 Real-world data

To demonstrate the effectiveness of UIPS in real-world scenarios, we evaluate it on three recommendation datasets: (1) Yahoo! R3 1 ; (2) Coat 2 ; (3) KuaiRec [12], for music, fashion and short-video recommendations respectively. All these datasets contain an unbiased test set collected from a randomized controlled trial where items are randomly selected. The statistics of the three datasets and implementation details, e.g., model architectures and dataset splits, can be found in Appendix 7.3.2.

Following [10], we take K = 5 on Yahoo! R3 and Coat datasets, and K = 50 on KuaiRec dataset. The p -value under the t-test between UIPS and the best baseline on each dataset is also reported to investigate the significance of improvement.

Yahoo

Coat

KuaiRec

Algorithm

P@5

R@5

NDCG@5

P@5

R@5

NDCG@5

P@50

R@50

NDCG@50

CE

0.2819

±

2

e

-

3

0.7594

±

6

e

-

3

0.6073

±

7

e

-

3

0.2799

±

5

e

-

3

0.4618

±

1

e

-

2

0.4529

±

7

e

-

3

0.8802

±

2

e

-

3

0.0240

±

8

e

-

5

0.8810

±

6

e

-

3

BIPS-Cap

0.2808

±

2

e

-

3

0.7576

±

5

e

-

3

0.6099

±

8

e

-

3

0.2758

±

6

e

-

3

0.4582

±

7

e

-

3

0.4399

±

9

e

-

3

0.8750

±

3

e

-

3

0.0238

±

7

e

-

5

0.8788

±

5

e

-

3

MinVar

0.2843

±

4

e

-

3

0.7685

±

1

e

-

2

0.6168

±

1

e

-

2

0.2813

±

3

e

-

3

0.4668

±

9

e

-

3

0.4414

±

8

e

-

3

0.8827

±

1

e

-

3

0.0240

±

5

e

-

5

0.8886

±

2

e

-

3

stableVar

0.2787

±

2

e

-

3

0.7499

±

7

e

-

3

0.5919

±

7

e

-

3

0.2840

±

3

e

-

3

0.4662

±

5

e

-

3

0.4393

±

7

e

-

3

0.8524

±

7

e

-

3

0.0231

±

2

e

-

4

0.8570

±

4

e

-

3

Shrinkage

0.2843

±

3

e

-

3

0.7654

±

8

e

-

3

0.6204

±

7

e

-

3

0.2790

±

5

e

-

3

0.4636

±

4

e

-

3

0.4464

±

1

e

-

2

0.8744

±

3

e

-

3

0.0238

±

9

e

-

5

0.8771

±

6

e

-

3

SNIPS

0.2222

±

4

e

-

3

0.5828

±

1

e

-

2

0.4357

±

1

e

-

2

0.2643

±

7

e

-

3

0.4287

±

1

e

-

2

0.4009

±

9

e

-

3

0.8411

±

6

e

-

3

0.0228

±

2

e

-

4

0.8431

±

6

e

-

3

BanditNet

0.2413

±

8

e

-

3

0.6442

±

2

e

-

2

0.4988

±

2

e

-

2

0.2781

±

8

e

-

3

0.4527

±

1

e

-

2

0.4251

±

1

e

-

2

0.8758

±

5

e

-

3

0.0239

±

2

e

-

4

0.8810

±

4

e

-

3

POEM

0.2732

±

3

e

-

3

0.7357

±

1

e

-

2

0.5880

±

1

e

-

2

0.2791

±

4

e

-

3

0.4566

±

6

e

-

3

0.4375

±

6

e

-

3

0.7785

±

1

e

-

2

0.0210

±

2

e

-

4

0.7779

±

6

e

-

3

POXM

0.2250

±

5

e

-

3

0.5940

±

1

e

-

2

0.4542

±

2

e

-

2

0.2663

±

6

e

-

3

0.4308

±

9

e

-

3

0.4006

±

1

e

-

2

0.8962

±

1

e

-

2

0.0245

±

4

e

-

4

0.9041

±

1

e

-

2

Adaptive

0.2762

±

3

e

-

3

0.7451

±

9

e

-

3

0.5919

±

8

e

-

3

0.2830

±

3

e

-

3

0.4634

±

5

e

-

3

0.4217

±

5

e

-

3

0.8375

±

1

e

-

2

0.0227

±

4

e

-

4

0.8460

±

1

e

-

2

ApproxKNN

0.2697

±

2

e

-

3

0.7225

±

5

e

-

3

0.5760

±

6

e

-

3

0.2755

±

2

e

-

3

0.4594

±

5

e

-

3

0.4490

±

4

e

-

3

0.8839

±

2

e

-

6

0.0240

±

5

e

-

5

0.8895

±

2

e

-

3

IPS-C-TS

0.2816

±

2

e

-

3

0.7582

±

5

e

-

3

0.6114

±

5

e

-

3

0.2799

±

3

e

-

3

0.4625

±

7

e

-

3

0.4462

±

6

e

-

3

0.8781

±

3

e

-

3

0.0239

±

1

e

-

4

0.8749

±

3

e

-

3

UIPS-P

0.1829

±

8

e

-

3

0.4560

±

3

e

-

2

0.3300

±

1

e

-

2

0.2685

±

7

e

-

3

0.4364

±

9

e

-

3

0.4087

±

7

e

-

3

0.8638

±

8

e

-

3

0.0235

±

3

e

-

4

0.8685

±

7

e

-

3

UIPS-O

0.1947

±

3

e

-

3

0.4959

±

1

e

-

2

0.3600

±

8

e

-

3

0.2657

±

5

e

-

3

0.4306

±

9

e

-

3

0.4146

±

9

e

-

3

0.8651

±

8

e

-

3

0.0235

±

2

e

-

4

0.8697

±

7

e

-

3

UIPS

0.2868

±

2

e

-

3

0.7742

±

5

e

-

3

0.6274

±

5

e

-

3

0.2877

±

3

e

-

3

0.4757

±

5

e

-

3

0.4576

±

8

e

-

3

0.9120

±

1

e

-

3

0.0250

±

5

e

-

5

0.9174

±

7

e

-

4

p-value

4

e

-

2

1

e

-

2

3

e

-

2

2

e

-

2

6

e

-

4

5

e

-

5

6

e

-

4

6

e

-

4

1

e

-

3

We can first observe from Table 4 that on all three datasets, the proposed UIPS achieved the highest precision, recall and NDCG. Apparently, accurate estimation of logging probabilities in real-world scenarios with large action spaces and sparse interactions is challenging to achieve, causing BIPSCap to underperform CE. Additionally, calibration poses difficulties in such scenarios, leading to poor performance of ApproxKNN and IPS-C-TS across all three real-world datasets. BanditNet, POEM and POXM performed better in problems with a larger action space, while MinVar, stableVar and Shrinkage as well as Adaptive suited better for scenarios with a smaller action size. Again, UIPS outperformed Shrinkage, highlighting the importance of handling uncertainty in the estimated logging policy. But simply reweighing based on uncertainties, without considering their impact on the accuracy of the resulting estimator and the learned policy, led to poor performance, as shown by UIPS-P and UIPS-O.

4.3 Comparisons against other lines of off-policy learning

We discuss about the difference between UIPS and recent work with direct propensity estimation [34, 21], the DICE line of work [26, 47, 9] as well as the work on the distributionally robust off-policy learning [32, 17, 44] in Appendix 7.6, Appendix 7.5 and Appendix 7.7 respectively. Our empirical evaluations on recent work [26, 44, 34] from all three lines suggest that these lines of work cannot properly handle inaccuracies in the estimated logging probabilities that hinder policy improvement. Moreover, we also integrated the proposed UIPS with the doubly-robust (DR) estimator and our results suggest the accuracy of the imputation model greatly affected policy learning under DR, but UIPS still provides benefits. More details can be found in Appendix 7.4.

5 Related work

Our work is the first of its kind to account for the uncertainty of logging policy estimation in off-policy learning. The following two lines of work are most related to this work.

Off-policy learning. In many real-world applications, such as search engines and recommender systems, interactive online model update is expensive and risky [15]. Off-policy learning has therefore attracted great interest, since it can leverage the already logged feedback data [2, 8, 22]. The main challenge in off-policy learning is to address the mismatch between the logging policy and the learning policy. One common and widely-applied approach is to leverage the Inverse Propensity Scoring (IPS) method to correct the discrepancy between the two policies. And various methods are proposed to enhance IPS for more stabilizing learning [39, 37, 38] and improved variance control [23, 22], as well as extensions to more complex problems [40, 24].

However, all these solutions directly use the estimated logging policy for off-policy correction, leading to sub-optimal performance as shown in our experiments. A recent study on causal recommendation [10] also argues that propensity scores may not be correct due to unobserved confounders. They assume the effect of unobserved confounder for any sample can be bounded by a pre-defined hyperparameter, and adversarially search for the worst-case propensity for learning. Mapping to off-policy learning, their solution is a special case of our UIPS-O variant with uncertainty as a pre-defined constant.

There were existing studies [34, 21, 26, 47, 9] also explore direct estimation of the propensity ratio to bypass estimating the logging policy. However, as discussed in Appendix 7.6 and Appendix 7.5, they demonstrate inferior performance compared to UIPS. This is primarily due to either the lack of consideration for the accuracy of the estimated propensity ratio, similar to the limitations of existing IPS-type algorithms in handling inaccurately estimated logging probabilities, or the degeneration to a specific IPS estimator that suffers high variance.

Recent work on distributionally robust off-policy evaluation and learning [32, 17, 44] also addresses uncertainty in off-policy learning. However, their approach to handling uncertainty and the underlying motivation differ significantly from ours, resulting in distinct techniques employed. Further details can be found in Appendix 7.7. Additionally, experiments conducted in Appendix 7.7 demonstrate that directly adapting methods from distributionally robust off-policy learning to handle inaccurately estimated logging probabilities leads to poor performance.

Off-policy learning can also be directly built on off-policy evaluation. Several work [36, 46] also propose to control the variance of the estimator caused by small logging probabilities through instance reweighing. Again, they directly use the estimated logging policy for correction, and thus performed worse than UIPS as observed in our experiments. A recent study [29] assumed additional structure in the action space and proposed the marginalized IPS. Instead, our work considers the uncertainty in the estimated logging policy and thus does not add any new assumptions about the problem space.

Uncertainty-aware learning. Estimation uncertainty has been extensively studied [45, 50, 1]. In the context of on-policy reinforcement leanring and bandits [1, 48, 49], the use of uncertainty aims to strike a balance between exploration and exploitation by adopting an optimistic approach (i.e., UCB in bandits). One the other hand, most reseach on offline reinforcement learning/bandits [43, 4, 6] tends to be more conservative, employing techniques such as Lower Confidence Bounds (LCB) or penalizing out-of-distribution states and actions based on uncertainty to address extrapolation errors. However, these principles differ fundamentally from UIPS, which directly minimizes the mean square error of off-policy evaluation. The closed-form solution of the resulting per-instance weight in UIPS reflects how uncertainty contributes to the policy evaluation error. Moreover, Our UIPS-O and UIPS-P baselines leverage uncertainties using the two aforementioned general principles respectively. However, empirical findings indicate that blindly penalizing or boosting samples based on uncertainty is problematic. Proper correction depends on both uncertainty in logging policy estimation and the actual value of estimated logging probabilities.

6 Conclusion

In this paper, we propose a Uncertainty-aware Inverse Propensity Score estimator (UIPS) to explicitly model the uncertainty of the estimated logging policy for improved off-policy learning. UIPS weighs each logged instance to reduce its policy evaluation error, where the optimal weights have a closedform solution derived by minimizing the upper bound of the resulting estimator's mean squared error (MSE) to its ground-truth value. An improved policy is then obtained by optimizing the resulting estimation. Extensive experiments on synthetic and three real-world datasets as well as the theoretical convergence guarantee demonstrate the efficiency of UIPS.

As demonstrated in this work, explicitly modeling the uncertainty of the estimated logging policy is crucial for effective off-policy learning; but the best use of this uncertainty is not to simply downweigh or drop instances with uncertain estimations, but to balance it with the actually estimated logging probabilities in a per-instance basis. As our future work, it is promising to investigate how UIPS can be extended to value-based learning methods, e.g., actor-critics. And on the other hand, it is also important to analyze how tight our upper bound analysis of policy evaluation error is; and if possible, find new ways to tighten it for improvements.

References

Yasin Abbasi-Yadkori, Dávid Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. In Advances in Neural Information Processing Systems , pages 2312-2320, 2011.

Aman Agarwal, Ivan Zaitsev, Xuanhui Wang, Cheng Li, Marc Najork, and Thorsten Joachims. Estimating position bias without intrusive interventions. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining , pages 474-482, 2019.

Ahmad Ajalloeian and Sebastian U Stich. On the convergence of sgd with biased gradients. arXiv preprint arXiv:2008.00051 , 2020.

Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-based offline reinforcement learning with diversified q-ensemble. Advances in neural information processing systems , 34:7436-7447, 2021.

Raghu Aniruddh, Gottesman Omer, Liu Yao, Komorowski Matthieu, Faisal Aldo, Doshi-Velez Finale, and Brunskill Emma. Behaviour policy estimation in off-policy policy evaluation: Calibration matters. arXiv preprint arXiv:1807.01066 , 2018.

Chenjia Bai, Lingxiao Wang, Zhuoran Yang, Zhihong Deng, Animesh Garg, Peng Liu, and Zhaoran Wang. Pessimistic bootstrapping for uncertainty-driven offline reinforcement learning. arXiv preprint arXiv:2202.11566 , 2022.

K. Bhatia, K. Dahiya, H. Jain, P. Kar, A. Mittal, Y. Prabhu, and M. Varma. The extreme classification repository: Multi-label datasets and code, 2016.

Minmin Chen, Alex Beutel, Paul Covington, Sagar Jain, Francois Belletti, and Ed H Chi. Top-k off-policy correction for a reinforce recommender system. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining , pages 456-464, 2019.

Bo Dai, Ofir Nachum, Yinlam Chow, Lihong Li, Csaba Szepesvári, and Dale Schuurmans. Coindice: Off-policy confidence interval estimation. Advances in neural information processing systems , 33:9398-9411, 2020.

Sihao Ding, Peng Wu, Fuli Feng, Yitong Wang, Xiangnan He, Yong Liao, and Yongdong Zhang. Addressing unmeasured confounder for recommendation with sensitivity analysis. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 305-315, 2022.

Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning , pages 1050-1059. PMLR, 2016.

Chongming Gao, Shijun Li, Wenqiang Lei, Jiawei Chen, Biao Li, Peng Jiang, Xiangnan He, Jiaxin Mao, and Tat-Seng Chua. Kuairec: A fully-observed dataset and insights for evaluating recommender systems. In Proceedings of the 31st ACM International Conference on Information and Knowledge Management , CIKM '22, 2022.

Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning , pages 1321-1330. PMLR, 2017.

Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. Deepfm: a factorization-machine based neural network for ctr prediction. arXiv preprint arXiv:1703.04247 , 2017.

Nan Jiang and Lihong Li. Doubly robust off-policy value evaluation for reinforcement learning. In International Conference on Machine Learning , pages 652-661. PMLR, 2016.

Thorsten Joachims, Adith Swaminathan, and Maarten De Rijke. Deep learning with logged bandit feedback. In International Conference on Learning Representations , 2018.

Nathan Kallus, Xiaojie Mao, Kaiwen Wang, and Zhengyuan Zhou. Doubly robust distributionally robust off-policy evaluation and learning. In International Conference on Machine Learning , pages 10598-10632. PMLR, 2022.

Patrick Kidger and Terry Lyons. Universal approximation with deep narrow networks. In Conference on learning theory , pages 2306-2327. PMLR, 2020.

Sergey Levine and Vladlen Koltun. Guided policy search. In International conference on machine learning , pages 1-9. PMLR, 2013.

Lihong Li, Yu Lu, and Dengyong Zhou. Provably optimal algorithms for generalized linear contextual bandits. In International Conference on Machine Learning , pages 2071-2080. PMLR, 2017.

Xu-Hui Liu, Zhenghai Xue, Jingcheng Pang, Shengyi Jiang, Feng Xu, and Yang Yu. Regret minimization experience replay in off-policy reinforcement learning. Advances in Neural Information Processing Systems , 34:17604-17615, 2021.

Yaxu Liu, Jui-Nan Yen, Bowen Yuan, Rundong Shi, Peng Yan, and Chih-Jen Lin. Practical counterfactual policy learning for top-k recommendations. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 1141-1151, 2022.

Romain Lopez, Inderjit S Dhillon, and Michael I Jordan. Learning from extreme bandit feedback. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 8732-8740, 2021.

Jiaqi Ma, Zhe Zhao, Xinyang Yi, Ji Yang, Minmin Chen, Jiaxi Tang, Lichan Hong, and Ed H Chi. Off-policy learning in two-stage recommender systems. In Proceedings of The Web Conference 2020 , pages 463-473, 2020.

Rémi Munos, Tom Stepleton, Anna Harutyunyan, and Marc Bellemare. Safe and efficient off-policy reinforcement learning. Advances in neural information processing systems , 29, 2016.

Ofir Nachum, Yinlam Chow, Bo Dai, and Lihong Li. Dualdice: Behavior-agnostic estimation of discounted stationary distribution corrections. Advances in Neural Information Processing Systems , 32, 2019.

Doina Precup. Eligibility traces for off-policy policy evaluation. Computer Science Department Faculty Publication Series , page 80, 2000.

Aniruddh Raghu, Omer Gottesman, Yao Liu, Matthieu Komorowski, Aldo Faisal, Finale Doshi-Velez, and Emma Brunskill. Behaviour policy estimation in off-policy policy evaluation: Calibration matters. arXiv preprint arXiv:1807.01066 , 2018.

Yuta Saito and Thorsten Joachims. Off-policy evaluation for large action spaces via embeddings. arXiv preprint arXiv:2202.06317 , 2022.

Tobias Schnabel, Adith Swaminathan, Ashudeep Singh, Navin Chandak, and Thorsten Joachims. Recommendations as treatments: Debiasing learning and evaluation. In international conference on machine learning , pages 1670-1679. PMLR, 2016.

John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning , pages 1889-1897. PMLR, 2015.

Nian Si, Fan Zhang, Zhengyuan Zhou, and Jose Blanchet. Distributionally robust policy evaluation and learning in offline contextual bandits. In International Conference on Machine Learning , pages 8884-8894. PMLR, 2020.

David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature , 529(7587):484-489, 2016.

Samarth Sinha, Jiaming Song, Animesh Garg, and Stefano Ermon. Experience replay with likelihood-free importance weights. In Learning for Dynamics and Control Conference , pages 110-123. PMLR, 2022.

Alex Strehl, John Langford, Lihong Li, and Sham M Kakade. Learning from logged implicit exploration data. Advances in neural information processing systems , 23, 2010.

Yi Su, Maria Dimakopoulou, Akshay Krishnamurthy, and Miroslav Dudík. Doubly robust off-policy evaluation with shrinkage. In International Conference on Machine Learning , pages 9167-9176. PMLR, 2020.

Adith Swaminathan and Thorsten Joachims. Batch learning from logged bandit feedback through counterfactual risk minimization. The Journal of Machine Learning Research , 16(1):1731-1755, 2015.

Adith Swaminathan and Thorsten Joachims. Counterfactual risk minimization: Learning from logged bandit feedback. In International Conference on Machine Learning , pages 814-823. PMLR, 2015.

Adith Swaminathan and Thorsten Joachims. The self-normalized estimator for counterfactual learning. advances in neural information processing systems , 28, 2015.

Adith Swaminathan, Akshay Krishnamurthy, Alekh Agarwal, Miro Dudik, John Langford, Damien Jose, and Imed Zitouni. Off-policy evaluation for slate recommendation. Advances in Neural Information Processing Systems , 30, 2017.

Sebastian Thrun and Michael L Littman. Reinforcement learning: an introduction. AI Magazine , 21(1):103-103, 2000.

Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning , 8(3):229-256, 1992.

Yue Wu, Shuangfei Zhai, Nitish Srivastava, Joshua Susskind, Jian Zhang, Ruslan Salakhutdinov, and Hanlin Goh. Uncertainty weighted actor-critic for offline reinforcement learning. arXiv preprint arXiv:2105.08140 , 2021.

Da Xu, Yuting Ye, Chuanwei Ruan, and Bo Yang. Towards robust off-policy learning for runtime uncertainty. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 10101-10109, 2022.

Pan Xu, Zheng Wen, Handong Zhao, and Quanquan Gu. Neural contextual bandits with deep representation and shallow exploration. In International Conference on Learning Representations , 2021.

Ruohan Zhan, Vitor Hadad, David A Hirshberg, and Susan Athey. Off-policy evaluation via adaptive weighting with data from contextual bandits. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining , pages 2125-2135, 2021.

Ruiyi Zhang, Bo Dai, Lihong Li, and Dale Schuurmans. Gendice: Generalized offline estimation of stationary values. arXiv preprint arXiv:2002.09072 , 2020.

Xiaoying Zhang, Hong Xie, Hang Li, and John CS Lui. Conversational contextual bandit: Algorithm and application. In Proceedings of the web conference 2020 , pages 662-672, 2020.

Xiaoying Zhang, Hong Xie, and John CS Lui. Heterogeneous information assisted bandit learning: Theory and application. In 2021 IEEE 37th International Conference on Data Engineering (ICDE) , pages 2135-2140. IEEE, 2021.

Dongruo Zhou, Lihong Li, and Quanquan Gu. Neural contextual bandits with ucb-based exploration. In International Conference on Machine Learning , pages 11492-11502. PMLR, 2020.

Guorui Zhou, Xiaoqiang Zhu, Chenru Song, Ying Fan, Han Zhu, Xiao Ma, Yanghui Yan, Junqi Jin, Han Li, and Kun Gai. Deep interest network for click-through rate prediction. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining , pages 1059-1068, 2018.

Algorithm 1 UIPS

7 Appendix

7.1 Notations and Algorithm Framework.

For ease of reading, we list important notations in Table 5 and summarize the main framework of the proposed UIPS in Algorithm 1.

Computation Cost. The additional computation cost of UIPS over IPS comes from two parts:

Pre-calculate uncertainties (line 1-5 in Algorithm 1) : This part calculates uncertainty of the logging probability for each ( s, a ) pair, and it only needs to be executed once . The computational cost of this step is O ( Nd 2 + d 3 ) , where O ( Nd 2 ) is for calculating uncertainties in each ( s, a ) pair and O ( d 3 ) is for matrix inverse.

Calculate ϕ ∗ x ,a during training (line 8 in Algorithm 1): It only takes O(1) time, the same computational cost as calculating IPS score.

Note that calculating the logging probability for each sample, which is essential for both UIPS and IPS, takes O ( Nd |A| ) time. Since the dimension d is usually much less than action size |A| and sample size N , UIPS does not introduce significant computational overhead compared to the original IPS solution.

7.2 Derivation of confidence interval of logging probability.

Given that | f θ ∗ ( x , a ) -f θ ( x , a ) | ≤ γU x ,a holds with probability at least 1 -σ and

where Z ∗ = ∑ a ′ exp( f θ ∗ ( a ′ | x )) and ˆ Z = ∑ a ′ exp( f θ ( a ′ | x )) , we can get that with probability at least 1 -σ :

The step labeled as (1) is due to the modelling of ˆ β ( a | x ) . And the step labeled as (2) is because Z ∗ is a positive constant independent of ˆ β ( a | x ) . Thus with probability at least 1δ , we have β ∗ ( a | x ) ∈ B x ,a and

7.3 Experiment details

7.3.1 Synthetic Data

Data generation. Given the ground-truth logging policy β ∗ ( a | x ) , we generate the logged dataset as follows. For each sample in train set, we first get the embedded context vector x from its original feature vector ˜ x . We then sample an action a according to β ∗ ( a | x ) , and obtain the reward r x ,a = y ˜ x ,a , resulting bandit feedback ( x , a, r x ,a ) , where y ˜ x ,a is the label of class a under the original feature vector ˜ x . We repeat above process N times to collect the logged dataset. In our experiments, we take d = 64 , N = 100 .

We model the logging policy as in Eq.(9), where { θ a } are the parameters to be estimated. To train the logging policy, we take all samples in the logged dataset D as positive instances, and randomly sample non-selected actions as negative instances as in [8].

7.3.2 Real-world Data

Statistics of datasets. The statistics of three real-world recommendation datasets with unbiased data can be found in Table 6.

All these datasets contain a set of biased data collected from users' interactions on the platform, and a set of unbiased data collected from a randomized controlled trial where items are randomly selected. As in [10], on each dataset, the biased data is used for training, and the unbiased data is for testing, with a small part of unbiased data split for validation purpose (5% on Yahoo R3 and Coat,

and 15% on KuaiRec). We take the reward as 1 if : (1) the rating is larger than 3 in Yahoo! R3 and Coat datasets; (2) the user watched more than 70% of the video in KuaiRec. Otherwise, the reward is labeled as 0.

We adopted a two-tower neural network architecture to implement both the logging and learning policy, as shown in Figure 2. For the learning policy, the user representation and item representation are first modelled through two separate neural networks (i.e., the user tower and the item tower), and then their element-by-element product vector is projected to predict the user's preference for the item. We then re-use the user state generated from the user tower of the learning policy, and model the logging policy with another separate item tower, following [8]. We also block gradients to prevent the logging policy learning interfering the user state of the learning policy. In each learning epoch, we will first estimate the logging policy, and then take the estimated logging probabilities as well as their uncertainties to optimize the learning policy.

β(alx)

πg(a|x)

MLP

MLP

block

gradient

itememb.

userstatex

neural net

item

neural net

item

tower

tower

itemfeature

userfeature

7.3.3 Implementation details.

To facilitate hyper-parameter tuning, we disentangled two η s in two the terms of ϕ ∗ x ,a in Theorem 3.2, and introduce η 1 and η 2 to represent η in the first and second term respectively in our implementation. This is due to the scale of η in the first term is closely related to the scale of λ , with λ/ √ η as the truly effective hyper-parameter. But the scale of η in the second term is independent from λ .

Moreover, while λ = E π ϑ [ r 2 x ,a π ϑ ( a | x ) β ∗ ( a | x ) ] depends on π θ as discussed in Eq.(6), we cannot adaptively set the value of λ since the ground-truth logging policy β ∗ ( a | x ) is unknown. However, we have:

where the first inequality is due to the Cauchy-Schwarz inequality and the second inequality is because ∑ a π θ ( a | x ) 4 ≤ ∑ a π θ ( a | x ) = 1 with π θ ( a | x ) ∈ [0 , 1] . We denote ˜ λ = ( ∑ a r 4 x ,a β ∗ ( a | x ) 2 ) , which is dataset-specific constant and independent from π θ . By replacing λ in Eq.(6) with ˜ λ , we can still minimize an upper bound of MSE ( ˆ V UIPS ( π ϑ )) , which ensures that the result of our analysis still holds. Thus, considering ease of computation and efficiency, we take a fixed λ during our policy learning.

We then use grid search to select hyperparameters based on the model's performance on the validation dataset: the learning rate was searched in { 1 e -5 , 1 e -4 , 1 e -3 , 1 e -2 }; λ, γ, η 1 were searched in {0.5, 0.1, 1, 2,5, 10, 15, 20, 25, 30, 40, 50}. And η 2 was searched in {1, 10, 100, 1000}. For baseline algorithms, we also performed a similar grid search as mentioned above, and the search range follows the original papers.

Ablation study: hyper-parameter tuning. Although UIPS has four hyperparameters ( λ , γ , η 1 , and η 2 ), one only needs to carefully finetune two of them, i.e., γ and η 2 1 /λ , to obtain good performance of UIPS. This is because:

(a) Precision@5

(b) Recall@5

(c) NDCG@5

P@5

R@5

NDCG@5

1

0.5532

0.5529

0.5516

0.5510

0.5505

0.560

1

0.1561

0.1557

0.1551

0.1548

0.1547

1

0.6033

0.6035

0.6037

0.6038

0.6039

- 0.610

10

0.5525

0.5547

0.5546

0.5546

0.5553

- 0.558

1

0.1561

0.1565

0.1563

0.1563

0.1564

0.158

1

0.6024

0.6048

0.6058

0.6065

0.6070

0.608

- 0.556

0.606

2

0.5505

0.5552

0.5575

0.5587

0.5599

- 0.554

2

0.1556

0.1569

0.1576

0.1579

0.1583

0.157

2

0.6002

0.6057

0.6083

0.6098

0.6112

- 0.604

5

0.5495

0.5558

0.5589

0.5600

0.5608

- 0.552

5

2

0.1554

0.1573

0.1583

0.1586

0.1589

- 0.156

25

0.6004

0.6063

0.6095

0.6106

0.6113

- 0.602

3

0.5481

0.5544

0.5580

0.5591

0.5602

- 0.550

3

0.1550

0.1569

0.1580

0.1584

0.1588

3

0.5994

0.6054

0.6087

0.6094

0.6102

- 0.600

- 0.548

- 0.155

- 0.598

0.5457

1

1

0.5512

5

T

0.5541

10

0.5554

15

T

0.5564

20

- 0.546

4

0.1542

1

0.1558

5

0.1567

10

0.1571

15

0.1574

20

4

0.5960

1

0.6012

5

T

0.6043

10

0.6058

15

0.6068

20

- 0.596

入

入

入

η 2 acts as a capping threshold to ensure ϕ ∗ x ,a ≤ 2 η 2 holds even when the corresponding propensity scores are very low. Hence, it should be set to a reasonably large value (e.g., 100).

The key component (i.e., the first term) of ϕ ∗ x ,a can be rewritten in the following way. While all ( x , a ) pairs will be multiplied by ϕ ∗ x ,a , η 1 in the numerator will not affect final performance too much, and the key is to find a good value of η 2 1 /λ to balance the two terms in the denominator:

Thus with η 1 and η 2 fixed, the effect of hyper-parameter γ and λ on precision, recall as well as NDCG can be found in Figure 3. We can observe that to make UIPS excel, B x ,a needs to be of high confidence, e.g., γ = 25 performed the best on the dataset with τ = 0 . 5 . Moreover, the threshold √ λ/η 1 cannot be too small or too large.

7.4 Experiments on the doubly robust estimators.

The doubly robust (DR) estimator [15], which is a hybrid of direct method (DM) estimator and inverse propensity score (IPS) estimator, is also widely used for off-policy evaluation. More specifically, let ˆ η : X × A → R be the imputation model in DM that estimates the reward of action a under context vector x , and ˆ β ( a | x ) be the estimated logging policy in the IPS estimator. The DR estimator evaluates policy π based on the logged dataset D := { ( x n , a n , r x n ,a n ) | n ∈ [ N ] } , by:

where ˆ V DM ( π ) is the DM estimator:

Again assume the policy π ( a | x ) is parameterized by ϑ , the REINFORCE gradient of ˆ V DR ( π ϑ ) with respect to ϑ can be readily derived as follows:

The imputation model ˆ η ( x , a ) is pre-trained following previous work [22] with the same neural network architecture as the logging policy model. Besides the standard DR estimator, we also adapt

UIPS and the best two baselines on off-policy evaluation estimator (i.e., MinVar and Shrinkage based on the results in Table 1 and Table 4) to doubly robust setting using the same imputation model.

Table 7 and Table 8 report the empirical performance of the learned policy on the synthetic datasets and three real-world datasets respectively. For ease of comparison, we also include the experiment results of BIPS-Cap and UIPS on each dataset in these two tables. Two p -values are also provided: (1) p -value (UIPSDR): The p -value under the t-test between UIPSDR and the best DR baseline on each dataset; (2) p -value (UIPS): The p -value under the t-test between UIPS and the best DR baseline on each dataset. From Table 7 and Table 8, we can first observe that DR cannot consistently outperform BIPS-Cap: It outperformed BIPS-Cap on the Coat and KuaiRec dataset, while achieving much worse performance on the synthetic datasets and Yahoo dataset. This is because the imputation model also plays a very important role in gradient calculation as shown in Eq.(12). Its accuracy greatly affects policy learning. When the imputation model is sufficiently accurate, for example, on the Coat dataset with only 300 actions, incorporating the DM estimator not only led to better performance of DR over IPS, but also improved performance of UIPSDR over UIPS. And in particular, in this situation UIPSDR performed better than DR with the gain being statistically significant. When the imputation model is not accurate enough, for example, on the KuaiRec dataset with a large action space but sparse reward feedback, DR is still worse than UIPS, and UIPSDR also performs worse than UIPS due to the distortion of the imputation model.

7.5 Difference against DICE-type algorithms.

The DICE line of work [26, 47, 9] is proposed for off-policy correction in the multi-step RL setting with the environment following the Markov Decision Process (MDP) assumption. Although the DICE line of work does not require the knowledge of the logging policy, it is fundamentally different from our work. Given a logged dataset D := { ( s t , a t , r s t ,a t , s t +1 ) } collected from an unknown logging policy β ∗ ( a t | s t ) , DICE-type algorithms propose to directly estimate the discounted stationary distribution correction w s , a d π ( s t ,a t ) to replace the product-based off-policy correction weight ∏ t t t β ∗ ( a t | s t ) , which suffers high variance due to the series of products. While the estimation of w π/D ( s t , a t ) is agnostic to the logging policy, it highly depends on two assumptions: (1) Environment follows an MDP, i.e., s t +1 ∼ T ( s t , a t ) with T ( · , · ) denoting the state transition function; (2) Each logged sample should be of a state-action-next-state tuple ( s t , a t , r s t ,a t , s t +1 ) that contains state transition information.

π/D ( t t ) = d D ( s t ,a t ) π ( a | s )

We further inspected how DICE-type algorithms would work in the contextual bandit setting, which can be taken as a one-state MDP. We take DualDICE [1] as an example for illustration, and similar conclusions can be drawn for other DICE-type algorithms. We can easily derive that in the contextual bandit setting, DualDICE degenerates to the IPS estimator that approximates the unknown ground-truth logging policy β ∗ ( a | s ) with its empirical estimate from the given logged dataset D := { ( s i , a i , r s i ,a i ) | i ∈ [ N ] } , which inherits all limitations of IPS estimator, such as high variance on instances with low support in D .

More specifically, recall that DualDICE estimates the discounted stationary distribution correction by optimizing the following objective function (Eq.(8) in [26]):

with the optimizer x ∗ ( s, a ) = w π/D ( s, a ) . In a multi-step RL setting, one cannot directly calculate the second expectation as d π ( s, a ) is inaccessible, thus DualDICE takes the change-of-variable trick to address the above optimization problem. However, in the contextual bandit setting, let p ( s ) denote the state distribution, d π ( s, a ) = p ( s ) π ( a | s ) and d D ( s, a ) = p ( s ) β ∗ ( a | s ) . The optimization problem can be rewritten as :

With the logged dataset D , the empirical estimate of above optimization problem is :

yielding x ∗ ( s, a ) = π ( a | s ) N s /N s,a , where N s and N s,a denote the number of logged samples with s i = s and ( s i = s, a i = a ) respectively. This is actually equivalent to the IPS estimator with N s,a /N s as the estimate of β ∗ ( a | s ) . We refer to the above estimator as DICE-S, and Table 9 and Table 10 show the empirical performance of the policy learned through DICE-S on the synthetic and real-world datasets respectively. Similar as BIPS-Cap described in Section 4, we also clip propensity scores to control variance. One can observe from Table 9 and Table 10 that DICE-S performs worse than BIPS-Cap in all datasets except the KuaiRec dataset. Additionally, DICE-S underperforms compared to UIPS in all datasets. This is due to the fact that although DICE-S is unbiased, it only becomes accurate with numerous logged samples, and suffers high variance when logged samples are limited, resulting in its poor performance in our experiments.

7.6 Comparison against work on direct propensity estimation.

In addition to DICE, several other works [34, 21] propose methods to directly estimate the propensity ratio without requiring a behavior policy. These methods then use the propensity estimates to prioritize instances in the replay buffer for better TD learning. To compare its effectiveness, we also included a new baseline called IPS-LFIW, which implements the approach proposed in [1] to directly estimate the propensity ratio for off-policy learning.

The average performance and standard deviations of IPS-LFIW and UIPS on three synthetic datasets with different τ are reported in Table 11. Recall that smaller τ indicates a more skewed groundtruth logging policy. The p-value under the t-test between UIPS and IPS-LFIW is also provided to

investigate the significance of improvements. Notably, UIPS consistently outperformed IPS-LFIW with statistically significant improvements. One major reason for the worse performance of IPS-LFIW is that it does not consider the accuracy of the estimated propensity ratio, in a direct analogy to failing to handle uncertainty in the estimated logging probabilities in existing IPS-type algorithms.

7.7 Comparison against distributionally robust off-policy evaluation and learning.

Our work is fundamentally different from the line of work on distributionally robust off-policy evaluation and learning [32, 17, 44]. This results in different objectives that guide the use of min-max optimization.

Specifically, the work in [32, 17, 44] assumes unknown changes exist between their training and deployment environments, such as user preference drift or unforeseen events during policy execution. Thus they choose to maximize the policy value (e.g., ˆ V IPS ( π ) ) in the worst environment within an uncertainty set around the training environment,

As a result, their uncertainty set is created by introducing a small perturbation to the training environment. For example, the work in [32, 17] searches the worst environment P ∞ in the σ -close perturbed environments around the training environment P 0 (Eq.(1) in [17]):

And the work in [44] adversarially perturbs the known ground-truth logging policy π 0 ( ·|· ) in searching for the worst case (Eq.(4) in [44]):

In contrast, UIPS assumes the training and deployment environments stay the same, but the groundtruth logging policy is unknown. To control the high bias and high variance caused by inaccurately and small estimated logging probabilities, UIPS explicitly models the uncertainty in the estimated logging policy by incorporating a per-sample weight ϕ x ,a as discussed in Eq.(4). In order to make ˆ V UIPS ( π ϑ ) as accurate as possible despite the unknown ground-truth logging policy β ∗ ( ·|· ) , UIPS solves a min-max optimization problem in Eq.(8). This optimization problem seeks to find the optimal ϕ x ,a that minimizes the upper bound of the mean squared error (MSE) of ˆ V UIPS to its ground-truth value, within an uncertainty set of the unknown ground-truth logging policy β ∗ ( ·|· ) . The closed-form solution for the min-max optimization is also derived as in Theorem 3.2.

Furthermore, observing that work in [44] also performs optimization over an uncertainty set of the logging policy, we further adapted their method to handle the inaccuracy of the estimated logging policy, by taking π 0 ( ·|· ) as the estimated logging policy, i.e., π 0 ( ·|· ) = ˆ β ( ·|· ) . We name the adapted methods as IPS-UN. Table 12 demonstrates the performance of the learned policy under IPS-UN on three synthetic datasets. The results suggest that directly applying IPS-UN to handle inaccurately estimated logging probabilities is not be a feasible solution to our problem. One important reason for the worse performance of IPS-UN is that it strives to optimize for the worst potential environment, which might not be the case in our experiment datatsets. On the other hand, UIPS assumes the training and deployment environments stay same and strives to identify the optimal policy with an unknown ground-truth logging policy.

7.8 Theoretical Proofs.

Proof of Proposition 2.1:

Proof. Because of the linearity of expectation, we have E D [ ˆ V BIPS ( π ϑ ) ] = E β ∗ [ π ϑ ( a | x ) ˆ β ( a | x ) r x ,a ] , and thus:

Since samples are independently sampled from logging policy, the variance can be computed as:

By re-scaling, we get:

This completes the proof.

Proof of Theorem 3.1:

Proof. We can get:

We first bound the bias term:

Step (1) follows the linearity of expectation and step (2) is due to the Cauchy-Schwarz inequality. We then bound the variance term:

Combining the bound of bias and variance completes the proof.

Proof of Theorem 3.2:

Proof. We first define several notations:

˜ T ( ϕ x ,a ) = max β x ,a ∈ B x ,a T ( ϕ x ,a , β x ,a ) denotes the maximum value of inner optimization problem.

T ∗ = min ϕ x ,a ˜ T ( ϕ x ,a ) = min ϕ x ,a max β x ,a ∈ B x ,a T ( ϕ x ,a , β x ,a ) denote the optimal minmax value. And ϕ ∗ x ,a = arg min ϕ x ,a ˜ T ( ϕ x ,a ) .

We first find the maximum value of the inner optimization problem, i.e., ˜ T ( ϕ x ,a ) for any fixed ϕ x ,a . And there are three cases shown in Figure 4:

Case I: When ˆ β ( a | x ) ϕ x ,a ≥ B + x ,a , , ˜ T ( ϕ x ,a ) achieves the maximum value at β x ,a = B -x ,a . In other words, ˜ T ( ϕ x ,a ) = T ( ϕ x ,a , B -x ,a ) when ϕ x ,a ≤ ˆ β ( a | x ) B + x ,a .

Case II: When B -x ,a ≤ ˆ β ( a | x ) ϕ x ,a ≤ B + x ,a , i.e., Z ∗ exp( -γU x ,a ) ˆ Z ≤ ϕ x ,a ≤ Z ∗ exp( γU x ,a ) ˆ Z , then ˜ T ( ϕ x ,a ) will be the maximum between T ( ϕ x ,a , B -x ,a ) and T ( ϕ x ,a , B + x ,a ) .

More specifically, when ˆ β ( a | x ) ϕ x ,a ≤ B + x ,a + B -x ,a 2 , i.e., ϕ x ,a ≥ 2 ˆ β ( a | x ) B + x ,a + B -x ,a , ˜ T ( ϕ x ,a ) = T ( ϕ x ,a , B + x ,a ) . Otherwise when ϕ x ,a < 2 ˆ β ( a | x ) B + x ,a + B -x ,a , ˜ T ( ϕ x ,a ) = T ( ϕ x ,a , B -x ,a ) .

1

1

Bxa

Bxa

B(alx)

Φxa

Bxa

β(alx)

中x.a

Bxa

β(alx)

中x.a

Bxa

x,a

Case I

CaseII

CaseIII

Case III: When ϕ x ,a ≥ ˆ β ( a | x ) B -x ,a . implying ˆ β ( a | x ) ϕ x ,a ≤ B -x ,a , ˜ T ( ϕ x ,a ) = T ( ϕ x ,a , B + x ,a ) .

Overall, we get that:

Next we try to find the minimum value of ˜ T ( ϕ x ,a ) . We first observe that without considering constraint on ϕ x ,a , when

T ( ϕ x ,a , B + x ,a ) achieves the global minimum value. However, ϕ + x ,a ≤ ˆ β ( a | x ) B + x ,a ≤ 2 ˆ β ( a | x ) B + x ,a + B -x ,a , which implies when ϕ x ,a ∈ [ 2 ˆ β ( a | x ) B + x ,a + B -x ,a , ∞ ) , the minimum value of T ( ϕ x ,a , B + x ,a ) is achieved at 2 ˆ β ( a | x ) B + x ,a + B -x ,a .

On the other hand, without considering any constraint on ϕ x ,a , the global minimum value of T ( ϕ x ,a , B -x ,a ) is achieved at:

Thus if ϕ -x ,a ≤ 2 ˆ β ( a | x ) B + x ,a + B -x ,a , ϕ ∗ x ,a = ϕ -x ,a , since T ( ϕ -x ,a , B -x ,a ) ≤ T ( 2 ˆ β ( a | x ) B + x ,a + B -x ,a , B -x ,a ) = T ( 2 ˆ β ( a | x ) B + x ,a + B -x ,a , B + x ,a ) . Otherwise, when ϕ -x ,a > 2 ˆ β ( a | x ) B + x ,a + B -x ,a , the minimum value of T ( ϕ x ,a , B -x ,a ) is also achieved at 2 ˆ β ( a | x ) B + x ,a + B -x ,a , implying ϕ ∗ x ,a = 2 ˆ β ( a | x ) B + x ,a + B -x ,a .

Overall,

Let η = Z ∗ ˆ Z , we can get

Note that η ∈ [exp( -γU max s ) , exp( γU max s )] , since ˆ Z exp( -γU max s ) ≤ Z ∗ = ∑ a ′ exp( f θ ∗ ( a ′ x )) ˆ Z exp( U max s ) .

| ≤ This completes the proof .

Proof of Corollary 3.3:

Proof. Following the similar procedure as in Theorem 3.1, we can derive the mean squared error ( MSE ) between ˆ V BIPS ( π ϑ ) and ground-truth estimator V ( π ϑ ) is upper bounded as follows:

When β ∗ ( a | x ) ∈ [ B -x ,a , B + x ,a ] , with B -x ,a := ˆ Z exp( -γU x ,a ) Z ∗ ˆ β ( a | x ) and B + x ,a := ˆ Z exp( γU x ,a ) Z ∗ ˆ β ( a | x ) , MSE ( ˆ V BIPS ( π ϑ ) ) can be further upper bounded as follows.

For ease of illustration, we set λ = E π ϑ [ r 2 x ,a π ϑ ( a | x ) β ∗ ( a | x ) ] and

Let T ∗ BIPS denote the upper bound of MSE ( ˆ V BIPS ( π ϑ ) ) , then we can derive that

Recall that in Theorem 3.2, we show that the upper bound of MSE ( ˆ V UIPS ( π ϑ ) ) is as follows:

where ϕ -x ,a is defined in Eq.(16). Next we show that T ∗ UIPS ≥ T ∗ BIPS .

When ˆ β ( a | x ) > B + x ,a + B -x ,a 2 , we can get T ∗ BIPS = T (1 , B -x ,a ) and 2 ˆ β ( a | x ) B + x ,a + B -x ,a > 1 . If ϕ -x ,a < 2 ˆ β ( a | x ) B + x ,a + B -x ,a , then T ∗ UIPS = T ( ϕ -x ,a , B -x ,a ) ≤ T (1 , B -x ,a ) , since ϕ -x ,a is a global minimum. Otherwise when ϕ -x ,a ≥ 2 ˆ β ( a | x ) B + x ,a + B -x ,a > 1 , T ∗ UIPS = T ( 2 ˆ β ( a | x ) B + x ,a + B -x ,a , B + x ,a ) = T ( 2 ˆ β ( a | x ) B + x ,a + B -x ,a , B -x ,a ) < T (1 , B -x ,a ) .

In both cases, we have T ∗ UIPS ≤ T ∗ BIPS , thus completing the proof.

Lemma 7.1. Under fixed π ϑ ( a | x ) and ˆ β ( a | x ) , and α x ,a = √ λ 2 η 2 -λ (1 -η ) η 2 exp( -2 γU x ,a ) , we have the following observations:

If π ϑ ( a | x ) ˆ β ( a | x ) ≤ α x ,a , ϕ ∗ x ,a = 2 η/ [exp( γU x ,a ) + exp( -γU x ,a )] . Otherwise ϕ ∗ x ,a = λ/ [ λ η exp( -γU x ,a ) + ηπ ϑ ( a | x ) 2 ˆ β 2 ( a | x ) exp( -γU x ,a ) ] ] . In other words, ϕ ∗ x ,a ≤ 2 η always holds.

Otherwise ϕ ∗ x ,a decreases as U x ,a increases.

If α x ,a ≤ π ϑ ( a | x ) ˆ β ( a | x ) and π ϑ ( a | x ) B -x ,a < √ λ , larger U x ,a brings larger ϕ ∗ x ,a .

Proof. The following inequality validates the first observation:

For the second and third observations, α x ,a ≤ √ λ √ 2 η ≤ √ λ η . Let L ( u ) = λ η exp( -γu ) + ηπ ϑ ( a | x ) 2 ˆ β ( a | x ) 2 exp( -γu ) , we can have:

To make ∇ u L ( u ) ≥ 0 , we need u ≥ 1 γ log ( √ λ ˆ β ( a | x ) ηπ ϑ ( a | x ) ) . This implies when U x ,a ≥ 1 γ log ( √ λ ˆ β ( a | x ) ηπ ϑ ( a | x ) ) , ϕ ∗ x ,a will decrease as U x ,a increases; otherwise as U x ,a increases, ϕ ∗ x ,a also increases.

More specifically, when α x ,a ≤ π ϑ ( a | x ) ˆ β ( a | x ) , we have:

In other words, for these samples, higher uncertainty implies smaller value of π/ ˆ β , and UIPS tends to boost such safe sample with higher ϕ ∗ x ,a .

In other cases, ϕ ∗ x ,a decreases as U x ,a increases. This completes the proof.

7.9 Convergence Analysis

Next we provide the convergence analysis of policy improvement under UIPS.

Definition 7.2. A function f : R d → R is L -smooth when ∥∇ f ( x ) -∇ f ( y ) ∥ 2 ≤ L ∥ x -y ∥ 2 , for all x, y ∈ R d .

We first state and prove a general result, which serves as the basis to complete our convergence analysis of policy improvement under UIPS. The proof is a special case of convergence proof of stochastic gradient descent with biased gradients in [3]. Suppose we have a differentiable function f : R d → R , which is L -smooth, and attains a finite minimum value f ∗ := min x ∈ R d f ( x ) .

Let b ( x ) = ∇ ˜ f ( x ) -∇ f ( x ) denote the difference between ∇ ˜ f ( x ) and ∇ f ( x ) , and δ ( x ) = ζ ( x ) -∇ ˜ f ( x ) denote the noise in gradients. We assume that:

Suppose we cannot directly assess the gradient ∇ f ( x ) . Instead we can only assess a noisy but unbiased gradient ζ ( x ) ∈ R d at the given x of the function ˜ f ( x ) .

where the constants φ and M satisfy 0 < φ < 1 and M ≥ 0 respectively. When running stochastic gradient descent algorithms, i.e. x k +1 = x k -η k ζ ( x k ) , we have the following guarantee on the convergence of x k to an approximate stationary point of f .

Theorem 7.3. Suppose f ( · ) is differentiable and L -smooth, and the assessed approximate gradient meets the conditions in Eq. (21) with parameters ( σ k , φ k ) at iteration k . Denote σ max = max k σ k and φ max = max k φ k . Set the stepsizes { η k } to η k = min { 1 ( M +1) L , 1 / ( σ max √ K ) } , after K iterations, the stochastic gradient descent satisfies :

Proof. With η k ≤ 1 ( M +1) L , we have:

By taking expectations on both side, we have:

where the inequality labeled as (1) is due to E [ δ ( x )] = 0 , inequality labeled as (2) is due to η k ≤ 1 ( M +1) L , and inequality labeled as (3) is due to ∥ b ( x k ) ∥ ≤ φ k ∥∇ f ( x k ) ∥ .

By summing over iterations k = 1 , 2 , . . . , K and re-arranging the terms, we obtain:

where the last inequality follows from f ( x K +1 ) ≥ f ∗ . Since η k = min { 1 ( M +1) L , 1 σ max √ K } , ∀ k = 1 , . . . , K , we can obtain:

Dividing both sides of the above inequality by Kη 1 (1 -φ max ) , we obtain the following,

Given this general result, we can now prove Theorem 3.4 by showing that the gradient in UIPS meets the requirements in Theorem 7.3.

Proof of Theorem 3.4:

Proof. UIPS aims to maximize the expected return V ( π ϑ ) . Therefore, we can utilize Theorem 7.3 by setting f = -V ( π ϑ ) . Since f ∗ = -V max and the expected reward is always non-negative, it follows that f ( x 1 ) -f ∗ ≤ V max .

We first introduce some additional notations. Let ρ ∗ ϑ ( x , a ) = π ϑ ( a | x ) β ∗ ( a | x ) denote the propensity score under ground-truth logging policy, and ˆ ρ ϑ ( x , a ) = π ϑ ( a | x ) ˆ β ( a | x ) ϕ ∗ x ,a represet the propensity score of UIPS. Recall that ϕ ∗ x ,a is derived through solving the optimization problem in Eq (8). Let g ϑ ( x , a ) = ∂π ϑ ( a | x ) ∂ϑ . The true off-policy policy gradient is computed as follows:

For UIPS, the approximate policy gradient in each batch with batch size as B is:

To utilize Theorem 7.3, we set ∇ f = -∇ V ( π ϑ ) , ∇ ˜ f = -∇ V UIPS ( π ϑ ) , and ζ = ∇ ˆ V UIPS ( π ϑ ) . We will now demonstrate that the assumptions in Eq. (21) can be satisfied.

Let φ ϑ = max {∣ ∣ ∣ ˆ ρ ϑ ( x ,a ) ρ ∗ ϑ ( x ,a ) -1 ∣ ∣ ∣ } , We first have:

And next we show that 0 < φ ϑ < 1 .

Recall from Eq.(17) in proof of Theorem 3.2, we have :

where B -x ,a := ˆ Z exp( -γU x ,a ) Z ∗ ˆ β ( a | x ) , and B + x ,a := ˆ Z exp( γU x ,a ) Z ∗ ˆ β ( a | x ) . Thus we have:

Since B + x ,a ≥ β ∗ ( a | x ) , thus 0 ≤ β ∗ ( a | x ) ˆ β ( a | x ) · ϕ ∗ x ,a ≤ 2 , implying 0 < φ ϑ < 1 .

Also, since ∇ ˆ V UIPS ( π ϑ ) is an unbiased estimate of ∇ V UIPS ( π ϑ ) , we have:

which is an unbiased estimate of :

Finally, we have the following,

where the equality labeled as (1) is due to E [ ∥ Y -E [ Y ] ∥ 2 ] = E [ ∥ Y ∥ 2 ] -∥ E [ Y ] ∥ 2 . Hence, by applying Theorem 7.3, we have:

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper delves into the issue of off-policy learning, the objective of which is to devise a new action selection policy based solely on the logged feedback derived from a logging policy. The paper pays particular attention to scenarios in which the logging policy remains unidentified and its estimation proves challenging. Under these conditions, common estimators, like IPS, may lose beneficial attributes such as unbiasedness. To address this complication, the paper introduces a new off-policy learning (OPL) approach called Uncertainty-aware off-policy learning. This new framework aims to optimize the uncertainty-aware objective function by employing a novel weighting scheme that is tuned by minimizing an upper bound of MSE in estimation. A local convergence based on the proposed method is also shown. Experimental results indicate that this proposed framework outperforms a range of benchmark methods on both semi-synthetic and real-world recommendation datasets.\n\nSTRENGTHS:\n- The paper addresses the practically relevant problem of dealing with uncertainty in logging policy estimation in off-policy learning.\n\n\n- The paper proposes a reasonable and conceptually straightforward method to handle the issue of uncertain logging policies, providing theoretical guarantees regarding estimation and local convergence.\n\n\n- The paper presents comprehensive experiments, not just basic performance comparisons, but also experiments on off-policy evaluation (OPE) and critical hyperparameters (some of which are included in the appendix).\n\nWEAKNESSES:\n- Given that several papers already exist on the topic of distributionally robust off-policy learning (OPL), as discussed in the paper, the formulation of a problem addressing the uncertainty of logging policies may not be groundbreaking, even though I understand that their motivations differ somewhat.\n\n\n- In the experiments, the issue might also be tackled by simply applying calibration during the estimation of the logging policy, as seen in the following paper:\n\nAniruddh Raghu, Omer Gottesman, Yao Liu, Matthieu Komorowski, Aldo Faisal, Finale Doshi-Velez, and Emma Brunskill. Behaviour Policy Estimation in Off-Policy Policy Evaluation: Calibration Matters. https://arxiv.org/pdf/1807.01066.pdf \n\n\n- In most of the experiments, CE performs quite well and is not substantially outperformed by UIPS. Therefore, considering the current experiment results, I may not use UIPS in practice and would rather rely on CE, which is much easier to implement (there is no need to estimate the logging policy when using CE), and does not require the tuning of additional hyperparameters as with UIPS.\n\n\n- Related to the previous point, in most experiments, the second-best methods for each metric and dataset perform very similarly to UIPS. I am not sure how essential it truly is to address uncertainty in logging policy estimation. I understand that the results are statistically significant, but results can be deemed significant even with a slight performance difference if the sample size is sufficient. In this context, my focus is on the performance difference.\n\nQUESTIONS:\n- How do baseline methods perform in the experiments when they are combined with a calibrated logging policy estimator? Some additional results about this would be useful.\n\n\n- Could you provide the results relative to the performance of the (true) logging policy? This enables us to see how much improvements the methods bring compared to the logging policy.\n\n\n- When does UIPS become really crucial? That is, are there any situations where UIPS performs well while all other methods do not work satisfactory. In the current experiments, the second-best methods perform very similarly to UIPS in all datasets and metrics. Moreover, CE performs reasonably and stably for a range of metrics and datasets, which makes it a really good choice in practice indeed.\n\nLIMITATIONS:\nThe paper touches on the tightness of the bound as a limitation and future work in the last section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers a scenario in off-policy evaluation where we don't have access to the action probabilities of the logging policy, which we need to compute the propensities in IPS. Prior work would estimate these probabilities from data, but would ignore the uncertainties associated with these estimates. In this work the authors propose to re-weight the propensities based on thess uncertainties. The exact form of the weights is derived by minimizing an upper bound on the MSE. The resulting method consistently beats SOTA baselines on both toy and real datasets.\n\nSTRENGTHS:\nA novel IPS variant that is grounded in and backed by theory (i.e. is derived by minimizing the MSE of the estimator), and is designed to solve a concrete problem in existing methods. \n\nSolid experimental methodology, sufficient details on the experimental setup to enable reproducibility.\n\nStrong results: the proposed method beats a broad list of SOTA baselines on both toy and real datasets.\n\nWEAKNESSES:\nI have doubts re. the importance of the problem that the method is solving: we can simply log the propensities, right? How common is it in practice that these propensities are not available?\n\nAt the same time, I see that in some experiments the method improves upon using GT propensities. This would be a good selling point of the method, but it goes against the initial motivation. How can a method that takes uncertainty into account be better than having _no_ uncertainty? This warrants more discussion in the paper, and makes me wonder if there are alternative explanations to why the method works.\n\nIn general, I've found the manuscript to be rather dense and hard to read in places. I would've liked to see more intuition and/or pedagogical examples to understand what the method does exactly. E.g. on line 158: \"UIPS assigns them with an increasing weight [..] as uncertainty increases\" -- why does it make sense?\n\nThe quality of the write-up could be improved: a few grammar mistakes, figure captions could be more informative, some opaque references to prior work could be expanded to make the paper more self-contained. \n\nThe related work could be stronger: lines 301 - 320 largely repeat the introduction. What about other methods in on-policy/off-policy that exploit the uncertainties? UCB in bandits, for example?\n\nQUESTIONS:\nHow sensitive is the method to the uncertainty estimation approach? Does something simple like MC-Dropout produce similar results?\n\nAlso see a few questions in the weaknesses section.\n\nLIMITATIONS:\nLittle discussion of limitations: e.g. what about additional cost due to having to estimate uncertainties? Limited societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an Uncertainty-aware Inverse Propensity Score estimator (UIPS) for off-policy learning, taking into account the uncertainty in the estimated logging policy. The authors demonstrate that the commonly used method of estimating the logging policy can lead to biased estimators, particularly for samples with small estimated logging probabilities. UIPS addresses this issue by reweighting the propensity scores based on the uncertainty of the estimated logging policy. The paper provides a theoretical analysis of the convergence properties of UIPS and presents experimental results on synthetic and real-world recommendation datasets, comparing against state-of-the-art baselines.\n\nSTRENGTHS:\n● The paper addresses an important problem in off-policy learning and proposes a novel method, UIPS, to improve the quality of the discovered policy.\n● The authors provide a comprehensive theoretical analysis of UIPS, including a convergence guarantee.\n● The experimental results demonstrate the effectiveness of UIPS compared to some baselines on both synthetic and real-world datasets.\n\nWEAKNESSES:\n● There remain some issues unsolved in the paper, such as the availability of the logging policy. See the questions for details.\n● There are some related works that are not mentioned in this paper. In off-policy RL, several papers work on behavior-agnostic instance reweighting [1,2]. They compute the prioritization weight without the need of obtaining a behavior policy. There are also papers that discuss the importance ratio term when applying RL to recommendation systems [3,4].\n● Introducing another neural network to estimate $\\beta^*$ will increase the system complexity and the computational cost during training and testing. This may hinder the practical application of the algorithm.\n● The synthetic dataset and the offline evaluation can give biased evaluation results of the algorithms.\n\n[1] Sinha, Samarth, et al. \"Experience replay with likelihood-free importance weights.\" Learning for Dynamics and Control Conference. PMLR, 2022.\n[2] Liu, Xu-Hui, et al. \"Regret minimization experience replay in off-policy reinforcement learning.\" Advances in Neural Information Processing Systems 34 (2021).\n[3] Cai, Qingpeng, et al. \"Reinforcing User Retention in a Billion Scale Short Video Recommender System.\" arXiv preprint arXiv:2302.01724 (2023).\n[4] Chen, Minmin, et al. \"Off-policy actor-critic for recommender systems.\" Proceedings of the 16th ACM Conference on Recommender Systems. 2022.\n\nQUESTIONS:\n1.How is this paper related to behavior-agnostic methods [1,2,5]? With a GAN-like estimator, these methods no longer reconstruct all those behavior policies. They may also be regarded as baselines to compare with.\n2.Why are the probabilities $\\beta^*(a|x)$ not recorded in the data? With stochastic logging policies, it is easy to store probabilities together with state and actions when generating data. With deterministic logging policies, a common practice is to sample actions from a Gaussian distribution, with policy output as mean and a certain standard deviation. The action probability will also be available.\n3.Why do UIPS-O and UIPS-P lead to poor performance?\n\n[5] Nachum, Ofir, et al. \"Dualdice: Behavior-agnostic estimation of discounted stationary distribution corrections.\" Advances in neural information processing systems 32 (2019).\n\nLIMITATIONS:\nThe usage of off-policy correction and uncertainty-based reweighting is limited to policy-based techniques based on the REINFORCE trick. Such techniques can have higher variance than value-based techniques such as TD3 and SAC, and may lead to unstable training.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes UIPS, a method that models the uncertainty of the estimated logging policy to improve off-policy learning. It assigns weights to each observation instance instead of simply dropping those with high uncertainty. The paper deduces the optimal form of weights from minimizing the upper bound of the resulting estimator’s MSE. Then it gets the improved policy by a two-step iterative optimization. This method is evaluated on both synthetic data and three real-world datasets in which UIPS outperforms multiple baselines.\n\nSTRENGTHS:\nThe paper addresses an important problem in policy optimization and proposes an innovative way to handle the uncertainty in the logged data. It derives a closed-form solution for the upper bound of MSE and proves the convergence of the method. It also conducts extensive evaluation on synthetic and real-world dataset to demonstrate the effectiveness of UIPS.\n\nWEAKNESSES:\n1. For both synthetic and real-world data, although UIPS achieves the best results, it does not outperform the second-best method by a significant extent. It's not clear how this method will be useful in application.\n2. The paper lacks a more detailed study and comparison with existing work on handling uncertainty. More explanation is needed on why UIPS is better compared to other methods.\n3.  Some deduction is not clear. For example, in-between steps are needed to show the \"log trick\" used in formula (2). The figure illustration is not very clear either. For example, it's confusing to use ``Log(item freq)\" on the X-axis. In Tables 3 and 4, the metrics are not very clear.\n\nQUESTIONS:\n1. How many iterations and how much time does the algorithm run in the evaluation? Does the result verify Theorem 3.4?\n2. For Figure 1, the conclusion \"items with lower frequencies in the logged dataset have lower estimated logging probabilities\" is not a universal trend but only applies for frequency less than 7. How do you explain this observation, and does this affect your other results?\n\nLIMITATIONS:\nThe authors are upfront about their limitations and listed future directions to address them.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-16"}