Counterfactual Learning of Stochastic Policies with Continuous Actions
Authors: Houssam Zenati
Alberto Bietti
Matthieu Martin
Eustache Diemert
Pierre Gaillard
Julien Mairal
Venue: Accepted by TMLR ()

Abstract: Counterfactual reasoning from logged data has become increasingly important for many applications such as web advertising or healthcare. In this paper, we address the problem of learning stochastic policies with continuous actions from the viewpoint of counterfactual risk minimization (CRM). While the CRM framework is appealing and well studied for discrete actions, the continuous action case raises new challenges about modelization, optimization, and~offline model selection with real data which turns out to be particularly challenging. Our paper contributes to these three aspects of the CRM estimation pipeline. 
First, we introduce a modelling strategy based on a joint kernel embedding of contexts and actions, which overcomes the shortcomings of previous discretization approaches. Second, we empirically show that the optimization aspect of counterfactual learning is important, and we demonstrate the benefits of proximal point algorithms and smooth estimators. Finally, we propose an evaluation protocol for offline policies in real-world logged systems, which is challenging since policies cannot be replayed on test data, and we release a new large-scale dataset along with multiple synthetic, yet realistic, evaluation setups.

Published in Transactions on Machine Learning Research (03/2025)
Counterfactual Learning of Stochastic Policies with Continuous Actions
Houssam Zenati'
h.semati@ual..c.a
Gatsby Computational Neuroscience Unit, University College London
Alberto Bictti
abiettilfffdiirnnnnnsiitttoorrrr
Center for Computational Mathematics, Flatiron Instâtute, New York, NY, USA
Matthieu Martin
mat. wadt.martin@eciteo..oom com e.diemert@critco.cm com
Eustache Diemert
Criteo AI Lab
Pierre Gaillard
pierveggaillard@ingg Dulien.mairaldnn @inria..f
Julien Mairal
Univ. Grenoble Alpes, Inria, CNRS, Grenoble INP, LJK, 38000 Grenoble, France
Reviewed on OpenReview: https: // optpe/veee....e.e..mmee/e/ net/ formm? id= fC4bh1Pm2r
Abstract
Counterfactual reasoning from logged data has become increasingly important for many applications such web advertising or healtheare. In this paper, address the problem of learning stochastic policies with continuous actions from the viewpoint of counterfactual risk minimization (CRM). While the CRM framework is appealing and well studied for discrete actions, the continuous action case raises new challenges about modelization, optimization, and offline model selection with real data which turns out to be particularly challenging. Our paper contributes to these three aspocts of the CRM estimation pipeline. First, we introduce modelling strategy based on joint kernel embedding of contexts and actions, which overcomes the shortcomings of previous discretization approaches. Second, we empirically show that the optimization aspect of counterfactual learning is important, and we demonstrate the benefits of proximal point algorithms and smooth estimators. Finally, we propose an evaluation protocol for offline policies in real-world logged systems, which challenging since policies cannot be replayed on test data, and we release a new large-scale dataset along with multiple synthetic, yet realistic, evaluation setups.
1 Introduction
Logged interaction data is widely available in many applications such as drug dosage prescription [Klluss & Zhou 2018), recommender systems Li et al. 2012), or online auctions (Bottou et al. 2013). An important task to leverage past data in order to find good policy for selecting actions (e.g., drug doses) from available features (or contexts), rather than relying on randomized trials or sequential exploration, which may be costly to obtain subject to ethical concerns.
More precisely, we consider offline logged bandit feedback data, consisting of contexts and actions selected by a given logging policy, associated to observed rewards. This is known as bandit feedback, since the reward is only observed for the action chosen by the logging policy. The problem finding a good policy thus requires a form of counterfactual reasoning to estimate what the rewards would have been, had we used a different policy. When the logging policy is stochastic, one may obtain unbiased reward estimates under a
*Most of this work was done while the author the Criteo Lab and Tnria.

---

Published in Transactions on Machine Learning Research (03/2025)
new policy through importance sampling with inverse propensity scoring (IPS, Horvitz & Thompson 1952
One may then use this estimator or its variants for optimizing new policies without the need for costly experiments (Bottou et al. 2013 Dudik et al.) 2011 Swaminathan & Joachims 2015a b), an approach also known as counterfactual risk minimization (CRM). While this setting is not sequential, we assume that learning a slochastic policy is required so that one may gather new exploration data after deployment.
In this paper, we focus on stochastic policies with continuous actions, which, unlike the discrete setting, have received little attention in the context of counterfactual policy optimization (Demirer et al.] 2019, Kallus & Zhou 2018 Chen et al., 2016). As noted by Kallus & Zhou 2018) and as experiments confirm, addressing the continuous case with naive discretization strategies performs poorly. Our first contribution is about data modeling: we introduce a joint embedding of actions and contexts relying on kernel methods, which takes into account the contimuous nature of actions, leading to rich classes of estimators that prove to be effective
in practice.
In the context of CRM, the problem of estimation is intrinsically related to the problem of oplimization of non-convex objective function. In our second contribution, we underline the role of optimization algorithms (Bottou et al.) 2013; Swaaminthhan & Joachims 2015b). We believe that this aspect was overlooked, as previous work has mostly studied the effectiveness of estimation methods regardless of the optimization procedure. In this paper, we show that appropriate tools can bring significant benefits. To that, effect, we introduce differentiable estimators based on soft-clipping the importance weights, which are more amenable to gradient-based optimization than previous hard clipping procedures (Bottou et al. 2013 Wang et al., 2017). We provide a statistical analysis of our estimator and discuss its theoretical performance with regards to the literature. We also find that proximal point algorithms (Rockafellar) 1976) tend to dominate simpler off-the-shell optimization approaches, while keeping reasonable computation cost.
Finally, an open problem in counterfactual reasoning is the difficult question of reliable evaluation of new policies based on logged data only. Despite significant, progress thanks to various IPS estimators, we believe that this issue is still acute, since we need to be able to estimate the quality of policies and possibly select among different candidate ones before being able to deploy them in practice. Our last contribution is a small step towards solving this challenge, and consists of a new offline evaluation benchmark along with new large-scale dataset, which we call CoCoA, obtained from a real-world system. The key idea is to introduce importance sampling diagnostics (0wen] 2013 to discard unreliable solutions along with significance tests to assess improvements to a reference policy. We believe that this contribution will be useful for the research community as we are unaware of similar publicly available large-scale datasets for continuous actions.
Our findings, outlined below, are based on experimental results and validated through diverse lasks, demon- strating the effectiveness of counterfactual learning for stochastic policies with continuous actions.
1. Discrete methods are not effective enough for continuous action problems. Discretization strategies tend to underperform in applications involving continuous actions. To address this, we propose a modeling approach that jointly embeds contexts and continuous actions, demonstrating superior empirical performance compared to the linear models of Kallus & Zhou (2018); Chen et al. (2016)
and the free-based method of (Majzoubi et al. 2020 This approach, termed the Counterfactual Loss Predictor (CLP), is detailed in Section 3.2
2. Optimization perspectives of counterfactual learning problems matter. Our work introduces a novel differentiable estimator based on soft-clipping of importance weights. Empirically, the soft-clipping strategy outperforms existing weight transformation methods (Wang et al. 2017 Su et al. 2019
Metelli et al. 2021 Additionally, our findings underscore the effectiveness of proximal point methods
[Rockafellar 1976 which have been shown to be particularly beneficial for optimizing nonconvex objective functions Paquette et al. 2018). The soft-clipping estimator and the associated optimization
tools are introduced in Section 4 In Section 5 we analyze the excess risk of the proposed estimator and compare to existing methods in the literature.
3. Real-world counterfactual risk ménimization requires a standard offline evaluation protocol. To address this, we release CoCoA, a large-scale dataset designed for counterfactual learning with continuous actions, and propose offline evaluation procedure for model selection in counterfactual
2

---

Published in Transactions on Machine Learning Research (03/2025)
risk minimization tasks. Our protocol is empirically validated for assessing and identifying unreliable solutions, making it a crucial tool for real-world deployment of policies learned through counterfactual risk minimization. The dataset and evaluation protocol are presented in Section 6
2 Related Work
A large effort has been devoted to designing CRM estimators that have less variance than the IPS method, through clipping importance weights (Boltou et al., 2013; [ann et al., (017)) variance regularization (Swaminathan & Joachims 2015a). or by leveraging reward estimators through doubly robust methods (Dudik
ct al., 2011 Robins& & Rotnitzky 1995). In order to tackle an overfitting phenomenon termed "propensity
overfitting" Swaminathan & Joachims (0015b) also consider self- seff-normalized estimators (Owen 2013). Such
estimation techniques also appear in the context of sequential learning in contextual bandits (Agarwal et al.
2014 Langford& & Zhang 2008), as well as for off-policy evaluation in reinforcement learning (Jiang & Li
2016 In contrast, the setting we consider is not sequential. Moreover, unlike direct approaches (Dudik je al. 2011) which learn a cost predictor to derive a deterministic greedy policy, our approach learns a model indirectly by rather minimizing the policy risk. In discrcte actions scttings, several approaches have been proposed for large action spaces (Majzoubi et al., 2020 Sachdeva et al. 2023 Saito et al., 2023).
While most approaches for counterfactual policy optimization tend to focus on discrete actions, few works have tackled the continuous action case, again with focus on estimation rather than optimization. In particular, propensity scores for inntis actions were considered by Hirano & Imbens (2004). More recently, evaluation and optimization of continuous action policies were studied in a non non- parametric context by Kallus & Zhou| (2018). Lee et al. (2022). and by Demirer et al. (2019) in a semi-parametric setting. In causal inference, recent works have addressed the question of treatment effect estimation with continuous
actions (Colangelo & Lee 2023 Bockel-Rickermant et al. 2023
In contrast to these previous methods, (i) we focus on stochastic policies while they consider deterministic ones, even though the kernel smoothing approach of Kallus & Zhou (2018) and Lee et al. (222)) may be interpreted as learning a deterministic policy perturbed by Gaussian noise. (ii) The terminology of kernels used by Kallus& & Zhou (2018) refers to a different mathematical tool than the kernel embedding used in our work. We use positive definite kernels to define a nonlincar representation of actions and contexts in order to model the reward function, whereas Kallus & Zhou (2018) use kernel density estimation to obtain good importance sampling estimates and not model the reward. Chen et al.] (2016) also use kernel embedding of contexts in their policy parametrization, while our method jointly models contexts and actions. Moreover, their method requires computing an n X n Gram matrix, which does not scale with large datasets; in principle, it should be however possible to modify their method to handle kernel approximations such as the Nyström method (Williams & Seeger 2001). Besides, their lcarning formulation with a quadratic problem is not compatible with CRM regularizers introduced by (Swaminathan & Joachimss 2015a b) which would change their optimization procedure. Eventually, we note that Krause & Ong (2011) use similar kernels to ours for jointly modcling contexts and actions, but in the different sctting of scquential decision making with upper confidence bound strategies.
(iii) While Kallus& & Zhou (2018) and Demirer et al. (2019) focus on policy estimation, our work introduces a new contimuous-action data representation and incompasses optimization: in particular, we propose a new contextual policy parameterization, which leads to significant gains compared to baselines parametrized policies on the problems we consider, as well as further improvements related to the optimization strategy. We also note that, apart from Demirer et al. (2019) that uses an internal offline cross-validation for model selection, previous works did not perform offline model selection nor evaluation protocols, which are crucial for deploying methods on real data.
Optimization methods for learning stochastic policies have been mainly studied in the context of reinforcement learning through the policy gradient theorem (Ahmed et al. 2019 /utton et al., 2000 Williams) 1992). Such methods typically need to observe samples from the new policy at each optimization step, which is not possible in our setting. Other methods leverage a form of ool-po estimates during optimization (Kakade & Langford 2002) Schulman et al. (017), but these approaches still require to deploy learned policies at cach step, while we consider objective functions involving only a fixed dataset of collectod data. In the context of CRM, (Su et al. (2099) introduce estimator with a continuous clipping objective that achieves an improved
3

---

Published in Transactions on Machine Learning Research (03/2025)
bias-variance trade-off over the doubly-robust strategy. Nevertheless, this estimator is non-smooth, unlike our soft-clipping estimator.
3 Modeling of Continous Action Policies
We now review the CRM framework, originally introduced for discrete action spaces, and then present our modeling approach for policies with continuous actions.
3.1 Background
For stochastic policy æ over a set of actions A, contextual bandit environment generates i.i.d. context features I ~ Px in X, actions a ~ x(-[x)) and losses Py(llz, a), which may be seen as negative rewards, for some conditional probability distribution Py. We denote the resulting distribution over triplets (r, a,y) by Px. Then, the risk of a policy ı defined as:
L(r) = B(,,,))) [u].
(1)
For the logged bandit problem, the enviromment provides an offline logged dataset (I4, 04 Vi)==.....m where assume (zi, az yi) ~ Pno ii.d. for given stochastic loggin&ppolic policy To, and assume the propensities To,: (a)(a))))) to be known Then, the task lies in using the logged data to determine policy Ω in set of stochastic policies II with small risk Unlike reinforcement learning problems, where data can be gathered iteratively, here the data is collected at once using To. As a result, the quality of a new policy Ω can only be assessed counterfactually: "How would á have performed if it had been used during data collection?"
To minimize L, we typically have access to an empirical estimator 7 allowing perform counterfactual reasoning and solve the following regularized problem:
E arg min {f(m) ((())
#Ell
where 52 a regularizer. With appropriate counterfactual estimators for f., the framework of 2 has been called counterfactual risk minimization (Swaminathan Joachims 2015a).
In this paper, motivated by real-world problems involving continuous actions, such as those presented in Section 7 which requires addressing three aspects:
Modeling: Which policy class is suitable for continuous actions, particularly when it is important lo capture potential nonlinear interactions between actions and contexts? To address this, we propose the Counterfactual Loss Predictor (CLP), a method that serves as smooth approximation of a greedy policy, leveraging kernel embedding of actions and contexts.
Oplimization: We propose counterfactual loss functions L that disferentiable (Section 4. 11 Then, discuss optimization strategies to deal with the non-convexity of L and Ω (Section 4.2
offline protocol to evaluate the performance of the solution of 2 and to perform model selection on the choice (II, L, S2).
Evaluation: In Section 6 design
Before, we present our contributions, we recall below existing choices for L and Ω that we will evaluate later, and another classical estimator (Direct method) which will motivate our new policy class IT.
The terminology associated to 'logged" data and 'logging" policy from Swaminathan & Joachims 2016a is often referred to trajectory data and behavior policy in the importance sampling literature (Owen 2013) 2In the contimuous action case, density function of evaluated the actions given context.
that the probability distributions admit densities and thus propensitics denote the
Note that this definition may also include deterministic policies by allowing Dirac measures, unless includes specific constraint e.g., minimum variance, which may desirable order to gather data for future offline experiments.
4

---

Published in Transactions on Machine Learning Research (03/2025)
Counterfactual policy learning. The counterfactua counterfactua approach approcch tackles the distribution mismatch between the logging policy (ar)) and policy x in IT via importance sampling. The TPS method (Horvitz & Thompson 1952) relies on correcting the distribution mismatch using the well-known relation
y π T(aax)
( ((r,).))
To [o(a
assuming To has non-zero mass on the support of æT This allows us to derive an unbiased empirical estimate ((ag)aa))
(TPS(T) = Σ y
(4)
To,
However, the empirical estimator (TP)(T) has large variance and is subject to various overfitting phenomena. First, this estimator can be prone to loss overfitting which refers to overweighting losses y/ of rare sample points which were highly unlikely under тo. For this, regularization ion stralegies have been proposed, such as clipping the importance sampling weights (cIPS, Bottou et al.) 2013 Swaminathan & Joachims 2015ap. Second, such estimators also suffer from propensity overfitting (Swaminathan & Joachims 20155b) where the minimization of the latter estimator can lead to learn policies that artificially overlit training data for negative losses Vi (to maximize x(a,(zz)) or conversely avoid training data when losses are positives. Subsequently, self-normalized estimators (SNIPS, Swaminathan & Joachims 2015b Owen 2013) have been presented to
circumvent this issuc, see Appendix A for a review and Appendix B.1 for a motivating example.
The direct. method DM and the optimal greedy policy. For this class of methods, an important. quantity is the expected loss given actions and context, denoted by n (x, = Ely|,, a]. If this expected loss was known, an optimal (deterministic) greedy policy x' would simply select actions that minimize the expected loss
π (x) = arg min 17" (z,
55)
dEA
It is then tempting to use the available data to learn an estimator f(2 a) of the expected loss, for instance by using ridge regression to fit y f( a;) the training data. Then, we may use the deterministic greedy policy
( = arg min ((x aeA
This approach, termed direct method (DM), has the benefit of avoiding the high-variance problems of IPS- based methods, but may suffer from large bias since it ignores the potential mismatch between fDM and To- Specifically, the bias is problematic when the logging policy provides unbalanced data samples (e-g., only samples actions in a specific part of the action space) leading to overfitting (Bottou et al. 2013 Dudik et al. 2011: Swaminathan & Joachims) 2015b). Conversely, counterfactual methods re-balance these generated data samples with importance weights and mitigate the distribution mismatch to better estimate reward function on less explored actions (sec explanations in Appendix Cl. Nevertheless, such loss estimators can be sometimes effective in practice and may be used to improve IPS estimators in the so-called doubly robust (DR) estimator (Dudik et al. 2011) by applying TPS to the residuals y; Ma,, az); thus using o as control variate to decrease the variance of IPS.
3.2 The Counterfactual Loss Predictor (CLP) for Continuous Actions Policies
We recall that our estimator is designed by optimizing 2 over a class of policies II. In this subsection, we discuss how to choose II to model continuous actions problems. We emphasize that when considering continuous action spaces, the choice of policies II is more involved than in the discrete case. One may indeed naively discretize the action space into buckets and leverage discrete action strategies, but then local information within each bucket gets lost and it is non-trivial to choose an appropriate bucketization of the action space based on logged data, which contains non-discrete actions.
"The 'connterfactual term comes from the causal inference literature and refers in this policy learning context all estimators
using the inverse propensity scoring (IPS) method Horvitz & Thompson 1952h
5

---

Published in Transactions on Machine Learning Research (03/2025)
We focus stochastic policies belonging to certain classes of continuous distributions, such as Normal or log-Normal with context-dependent means and variances. Specifically, we consider set of policies of the form
TT s.t. for any x E E Ta(+2)) = D(us(z), with (B
(6)
where D(a, b) is a distribution of probability with mean and variance b, such as the Normal distribution, and @ is parameter space. Here, the parameter space O can be written θ O x G.. The space O, is either singleton (if a is considered as fixed parameter specified by the user) or I (if a is parameter to be optimized). The space 6 is the parameter space which models the contextual mean x + uo(2), which will be specified shortly. Note that in our policy set, the variance parameter a is a parameter that calibrates the stochasticity of the policy. For applications where a learned policy needs to be deployed to collect data in later rounds, a minimal variance can be enforced to ensure future exploration.
3.2.1 Examples of simple context-dependent policies
Before introducing a more expressive model for θ3 we consider the following simple counterfactual basclines for Og that only consider contexts and that will be compared to our model in the experimental Section Given a context x X' C TR::
constand: p=(x)) = 8 (context-independenl)) linear: (z(z)) = (x. B1) + Bo with 8 (So, 31) Rd+1;
poly: (z( (xx",Bi) 9o with 31) E B
These baselines require learning the parameters 8 by using the CRM approach (2 Intuitively, the goal is to find a stochastic policy that is close to the optimal deterministic one from Eq. 5 Yet. these approaches do not model potential interactions between actions and contexts. While these approaches, adopted by Chen et al. (2016), Kallus & Zhou (0188) can be effective in simple problems, they may be limited in more difficult scenarios where the expected loss 17" (1(x, a) has a complex behavior as a function of a. To address this issue, we introduce the Counterfactual Loss Predictor (CLP) model in the next section.
3.2.2 The counterfactual loss predictor (CLP) model
The CLP model relies on two ideas: (i) setting (z(z) to be a smooth approximation of the greedy policy F and (ii) using parametric model næ( a) of the loss which is expressive enough to capture complex interactions between actions and contexts. More precisely, assuming that we are given such a parametric model na (z, a), which we call loss predictor, we parametrize the mean of a stochastic policy by using soft-argmin operator
with temperature 7 >>0 0:
m
exp(-7713( a.))
CLP:
CLP (x) = Σ ai 2j=1 exp( (0p(0mmsa,,, ag))
where d1y-..,am, Um A are anchor points (e.g., a regular grid quantiles of the action space). Then, P4 may be viewed smooth approximation of the greedy policy (greedy(x) arg min, 71I This allows CLP policies to capture complex behavior of the expected loss as function of a. The motivation for introducing soft-argmin operator over finite mumber of anchor points is to build a policy that avoids the explicit optimization over the continuous action set, while making the resulting CRM problem differentiable.
Next, we need of course to define the loss predictor 17(2, a). We choose it of the form
na(a, a) = (3, (xx a)),
for some parameter 3 RP, which norm controls the smoothness of nj, and feature map (xx, a) E RP that we detail in two parts: a joint kernel embedding between the actions and the contexts and a Nyström approximation. The complete modeling of na(2, a) is summarized in Figure 1
1. Joint kernel embedding. In continuous action problem, a reasonable assumption is that losses vary smoothly as function of actions. Thus, good choice is to take n) in space of smooth functions such as the

---

Published in Transactions on Machine Learning Research (03/2025)
contexts
Joint kernel embedding
Nyström approximation
Loss predictor
(x',a))
((z,a) I'
ns(a, = (8, 1/( o))
actions E A
Figure 1: Illustration of the joint kernel embedding for the counterfactual loss prodictor (CLP) and loss estimator.
reproducing kernel Hilbert space H (RKHS) defined by a positive definite kernel (Schölkopl & Smola 2002 so that one may control the smoothness of r through regularization with the RKHS norm. More precisely, we consider kernels of the form
K(xx, a), (I', a')) = (x), ux (2'))e 8 aa.-
(8)
where, for simplicity, (x) is either linear embedding (x) = quadratic one vxx(x) (xz"" ax), while actions are compared via a Gaussian kernel, allowing to model complex interactions between contexts and actions.
2. Nyström melhod and erplicil embedding. Since Iraditional kernel methods lack scalability, we rely on the classical Nyström approximation (Williams & Seeger 2001) of the Gaussian kernel, which provides us inite-dimensions approximate embedding va((a) in R such that e ||a--'|| = (Wala), vala')) for all actions a, a' This allows us to build finiteddimensional embedding
<xx, a) = (x((x) @ va(a),
(9)
where denotes the tensorial product, such that
K( a), (x', ,²)) ve (fx(()),
= (v(2, (('',
More precisely, Nyström's approximation consists in projecting each point from the RKHS to a m-dimensional subspace defined as the span of anchor points, representing here the mapping the RKHS of actions 01, .2,...,,mm of the Nyström dictionary Z. In practice, we may choose a; to be equal to the a, in 7 since in both cases the goal is to choose a set of "representative" actions. For one-dimensional actions (A S R), it is reasonable to consider a uniform grid, or a non-umiform ones based on quantiles of the empirical distribution of actions the dataset. In higher dimensions, one may simply use a K-means algorithms and assign anchor points to centroids. The number of anchor points directly affects the quality of the approximation: a larger number enhances representational power but reduces computational efficiency for large-scale applications, and vice versa. In practice, this number is treated hyperparameter, selected using the oflline protocol described in Section 6 Its impact is further illustrated through comparison between continuous and discrete strategics in Appendix G.1
From an implementation point of view, Nyström's approximation considers the embedding (a((a) = K-1/2 Kz(a), where Kzz [K (ū;, ai)lis and Kz(a) [Ka(a, āz)]; and KA is the Gaussian kernel. The whole procedure to design the CLP policy set HoLP is given in Algorithm [
The anchor points that we use can be seen as the parameters of an interpolation strategy defining a smooth function, similar to knots in spline interpolation. Naive discretization strategies would prevent us from exploiting such a smoothness assumption on the loss with respect to actions and from exploiting the structure of the action space. Note that Scction provides comparison with naive discrctization strategies, showing important benefits of the kernel approach. Our goal was design a stochastic, computationally tractable, differentiable approximation of the optimal (but unknown) greedy policy (

---

Published in Transactions on Machine Learning Research (03/2025)
Algorithm 1: Construction of the CLP Policy Set
Input: Temperature y > 0, kernel K, Nyström dictionary S A, parametric distribution (such as Normal or log-Normal).
Output: Policy IlCLP.
1. Define the d-dimensional feature map & as in Eq. 9 by using K and Z.
2. For any B R and (z,0) E X x A, define
exp(->115zz, a))
ns(2, a) = (3, (x)
and
MB (z)
(z)
aEZ Laez exp( (xp(-773(2, a'))
3. Define the policy set
HCLP {IT s.t. Vo X, x(+/x) D(ug (x), with (8, 0).
4 On Optimization Perspectives for CRM
Because models yield non-convex CRM problems, it is crucial to study optimization aspects. Here, we introduce a differentiable clipping strategy for importance weights and discuss optimization algorithms.
4.1
Soft Clipping IPS
The classical hard clipping estimator
LAPS(() n Σ y min (T(a:l=:)/To.. M}
(10)
makes the objective function non-differentiable, and yields terms the objective with clipped weights have zero gradient. In other words, a trivial stationary point of the objective function is that of stochastic policy that differs enough from the logging policy such that all importance weights clipped. To alleviate
this issue, propose differentiable logarithmic soft-clipping strategy. Given threshold parameter M 2 0 and an importance weight =(aa)=)) no.i. consider the soft-clipped weights:
w/s
if w/ S M
(ww
(11)
(MM) log (w, + a(M) - M) otherwise,
where a(M) is such that a(M) log(a(M)) = M, which yields a differentiable operator. We illustrate in Fig. the expression of the logarithmic clipping
10
c(w) min(w,
10
Figure 2: Soft Clipping of importance sampling weight

---

Published in Transactions on Machine Learning Research (03/2025)
We give further explanations about the benefits of clipping strategies in Appendix B and compare soft-clipping strategy to other methods in Appendix G.2 Then, the soft clipping TPS (scIPS) estimator is defined as:
क (aalaz)) M)
LECPs (T))
Σ ya
(12)
%0,i
Note that our soft-clipping estimator differs from existing importance weight transformation strategics such as the power-mean correction of importance sampling (PMCIS) (Metelli et al.) 2021), the SWITCH (Wang et al. 2017) and CAB (Su et al. 2019 estimators, which were not motivated by differentiability issuesas (see empirical comparison in Appendix G.2
We now provide similar generalization bound to that of Swaminathan & Joachims
(015a)) (for the hard-
clipped version) for the variance-regularize objective of this soft-clipped estimator, justifying its use as a good optimization objective for minimizing the expected risk. Writing X²(T)) = y.C
M). we recall mo(a 22)
(a, las)
the empirical variance with scIPS that is used for regularization:
VToIPs(T)=
1 n 1 E x(()))
with X(T) = Σ (i(T)..
(13)
We assuine that costs ys E [-1,0] almost surely, as in (Swaminathan & Joachims 2015a), and make the additional assumption that the importance weights x (aa xx))/ (o(a (4)24)) are upper bounded by constant W almost surely for all π E II. This is satisfied, for instance, if all policies have a given compact support (c.g., actions are constrained to belong to a given interval) and To puts mass everywhere in this support.
Proposition 4.1 (Generalization bound for (T))) Let II be a policy class and no be a logging policy, under which an input-action-cost triple follows Dro- Assume that -1 I y I O a.s. when (x, a, y) ~ Dro and that the importance weights are bounded by W. Then, with probability at least 1 - 8, the IPS estimator with soft clipping (12) on samples satisfies
(
+ log (II, log
VT E I1,
+0
n
where = <(W, M) Olog W), VeOIPS(T) denotes the empirical variance of the cost estimates 200 and CII, M) is complerity measure of the policy class defined in [24
We prove Proposition 4.1 in Appendix E This generalization error bound motivates the use the empirical variance penalization as in (Swaminathan & Joachims, 2015a) and shows that minimizing both the empirical risk and penalization of the soft clipped estimator minimizes an upper bound of the true risk of the policy.
Note that while the bound requires importance weights bounded by constant W, the bound only scales logarithmically with W when W > M, compared to lincar dependence for IPS (Swaminathan & Joachims) 2015a). However we gain significant benefits in terms of optimization by having a smooth objective.
Remark 4.1. If losses are in the range [-c,0], the constand S can be replaced by eS, making the bound homogeneous in the scale (indeed, the variance term is also scaled by c).
Remark 4.2. For fixed parameter M, the scIPS estimator is less biased than the cIPS. Indeed, s(alx) motala))
bound the importance weights as min <(alx)) [o(a bound:
M <
s(alx) and subsequently derive the M) < [o(a|z]
क (alæ) To(alz)
0|12
[
TT(aa (
т T(alx)
M -
- 20
E (g
min
M
2 Bx.awoo(
We note however that the parameter M may have different optimal values for both methods, and that the main motivation for such a clipping strategy is to provide differentiable estimator which is not the case for cIPS in areas where all point are clipped.

---

Published in Transactions on Machine Learning Research (03/2025)
4.2 Proximal Point Algorithms
Non-convex CRM objectives have been optimized with classical gradient-based methods (Swaminathan & Joachims 2015a by such as L-BFGS (Liu & Nocedal 1989), or the stochastic gradient descent ap- proach (Joachims ct al. 2018 Proximal point methods are classical approaches originally designed for convex optimization Rockafellar 1976), which were then found to be useful for nonconvex functions (Fukushima & Paquette et al. 2018). In order to minimize a function L, the main idea is to approximately solve sequence of subproblems that are better conditioned than C, such that the sequence of iterates converges towards better stationary point of L. More precisely, for our class of parametric policies, the proximal point method consists of computing sequence
Mine 1981
0g x min (E(TT) + 1 2110- -110-00--00
02 02-1
(14)
where ((TT) = (Ta) + (Ta) and is constant parameter. The regularization term 92 often penalizes the variance (Swaminathan & Joachims 2015b), see Appendix F.2 The role of the quadratic function in (14) is to make subproblems "less nonconvex" and for many machine learning formulations, it is even possible to obtain convex sub-problems with large enough K (see Paquette et al.) 2018
In this paper, we consider such a strategy 14 with a parameter K, which we we to zero only for the last itcration. Note that the effect of the proximal point algorithm differs from the proximal policy optimization (PPO) strategy used in reinforcement learning (Schulman al. 2017), even though both approaches are related. PPO encourages new stochastic policy be close previous one in Kullback-Leibler distance. Whereas the term used in PPO modifies the objective function (and changes the set of stationary points), the proximal point algorithm optimizes and finds a stationary point of the original objective L, even with fixed x.
The proximal point algorithm (PPA) introduces an additional computational cost as it leads to solving multiple sub-problems instead of a single learning problem. In practice for 10 PPA iterations and with the L-BFGS solver, the computational overhead was about 3x in comparison to L-BFGS without PPA. This overhead seems be the price to pay to improve the test reward and obtain better local optima, as we show in the experimental section 7.2.2 Nevertheless, would like to emphasize that computational time is often not critical for the applications consider, since optimization is performed offline.
5 Analysis of the Excess Risk
In the previous section, we have introduced new counterfactual estimator L 12 of the risk, which satisfies good optimization propertics. Motivated by the gencralization bound in Proposition 4.1 for any policy class II, we associate LECIPS with the dala-dependent regularizer and define the following CRM estimator
{ [LLOTPs(T)) + x Vir TEll
CCRM arg min
(15)
n
where (TIIP((T) is the empirical variance defined in 13 In this section, provide theoretical guarantees on the excess risk of #CRM first for any general policy class I1, then for our newly introduced policy class HCLP (Section 3.2 We provide the following high-probability upper-bound on the excess-risk.
Proposition 5.1 (Excess risk upper bound). Consider the notations and assumptions of Proposition 4.1 Let &CRM be the solution of the CRM problem in Eq. (15) and n" Il be any policy. Then, with well chosen parameters X and M, denoting the variance of Vaa TT" (</x=//To((]]]] with probability at least - 8:
/ (1+ log(W + (I) + log log log log(W + +e) (II, M) + log 4
L(ACRM) -
L(TT) -
n
n
where S hides universal melliiplicative constants. In porticular, assuming also that no (2(aa) are uniformly bounded, the complerity of the class Herr described in Section 3.2. applied with a bounded kernel and θ {3 RRT s.t ||3||| < C) {o}, is of order
(ITLL))) 0(m
10

---

Published in Transactions on Machine Learning Research (03/2025)
where m is the size of the Nyström dictionary and O() hides multiplicative constonts independent of and m.
The proof and the exact definition of (11, M) are provided in Appendix E Our analysis relies on Theorem
15 of Maurer & Pontil 2009).
The closest works are the ones of Chen et al. (2016) and Kallus & Zhou (2018). Chen et al.] (2016) analyze their method for Besov policy classes B [K4). When a - 00, they obtain a rate of order On O(n-1/4) In this case, their setting is parametric and their rate can be compared to our On 1/2) when m is finite. Kallus & Zhou (2018) provide bounds with respect to general deterministic classes of functions, whose complexity is mcasured by their Rademacher complexity. For parametric classes, their excess risk is bounded (up to logs) by Rs x 2 + h h-1n-11/ + h2, where h is smoothing parameter. By optimizing the bandwidth h- O(n O(n-1/6), their method also yields rate of order O(n
Comparison with related work
Yet, a key difference between their setting and ours explains the gap between their rate O(n-1/4) and O(n-1/2) of Proposition 5.1 Both consider deterministic policy classes, while we only consider stochastic policies. Indeod, W and a" would be unbounded for deterministic policies in Proposition 5.1 Therefore, to leverage deterministic policies, they both need to smooth their predictions and suffer an additional bias that we do not incur. This is why there is difference between their rate and ours. For instance, for stochastic classes with variance o², Kallus & Zhou (2018) would satísfy Rs S a - the other hand, our approach would satisfy a
- -1/2 for h = o, which
would also entail rate of order rale Onn-1/3) for deterministic policies, i.e., o² + 0 (see Appendix E.2). This may be explained by the fact that, contrary to KKllus& & Zhou (2018); Chen et al. (2016) who only use it in practice, we consider variance regularization and clipping in our analysis.
172) Interestingly,
Another related work is (Demirer et al. 2019). They obtain an excess risk rate of O(n-1/2) when learning deterministic continuous action policies with a policy space of finite and small VC-dimension. Under a margin condition, as in bandit problems, their rate may be improved to Ollo(n)/nn). However, their method significantly differs from ours and Chen el al.] 20166 Kallus & Zhou (2018) because it relies on a two steps plug-in procedure: first estimate a nuisance function. then learn a policy using with a value function using this estimate. Eventually, we note that Majzoubi et al. (2020) also enjoys a regret of O(11//2 (up to logarithmic factors) but learns tree policies that are hardly comparable to ours. Both approaches turn out to perform worse in all our benchmarks, as seen Section. 7.2.2
6 On Evaluation and Model Selection for Real-World Data
The CRM framework helps finding solutions when online experiments are costly, dangerous or raising ethical concerns. As such it needs a reliable validation and evaluation procedure before rolling-out any solution in the real world. In the continuous action domain, previous work have mainly considered semi-simulated scenarios (Bertsimas& & McCord 2018 Kallus & Zhou 2018). where contexts are taken from supervisod datasets but rewards are synthetically generated. To foster research on practical continuous policy optimization, we release new large-scale dataset called CoCoA, which to our knowledge is the first to provide logged exploration data from a real-world system with continuous actions. Additionally, we introduce a benchmark protocol for reliably evaluating policies using off-policy evaluation.
6.1 The CoCoA Dataset
The CoCoA dataset comes from the Criteo online advertising platform which ran experiment involving randomized, continuous policy for real-time bidding, Data has been properly anonymized disclose any private information. Each sample represents a bidding opportunity for which multi-dimenssinal context x in Rd is observed and a continuous action a in IR+ has been chosen according to a stochastic policy To that is logged along with the reward -y (meaning cost y) in R. The reward represents an advertising objective such as sales or visits and is jointly caused by the action and context (a) Particular care has been taken to guarantee that each sample (zx, a, mo(a,)a (2)) ys) is independent. The goal is to learn a contextual, continuous, stochastic policy x (afær) that generates more reward in expectation than To, evaluated offline,
to not
11

---

Published in Transactions on Machine Learning Research (03/2025)
while keeping some exploration (stochastic part). As seen in Table 1 typical feature of this dataset is the high variance of the cost (V[Y)), motivating the scale of the dataset N to obtain precise counterfactual estimates.
N
d E[-Y]
V[Y]
V[A]
P(Y + 0)
120.106
3
11.37
9455
.01
.07
Table 1: CoCoA dataset summary statistics.
6.2 Evaluation Protocol for Logged Data
In order to estimate the test performance of a policy on real-world systems, off-policy evaluation is needed, only have access to logged exploration data. Yet, this involves in practice number of choices and difficulties, the most documented being i) potentially infinite variance of TPS estimators (Bottou et al. 2013 and ii) propensity over-fitting (Swaminathan & Joachims 2015a b) where actions in the logged data can be overfitted or avoided (see Section 3.1p. The former implies that it can be difficult to accurately assess the performance of new policies due to large confidence intervals, while the latter may lead to estimates that reflect large importance weights rather than rewards.
A proper evaluation protocol should therefore guard against such outcomes.
Algorithm 2: Evaluation Protocol
Input: 1- ō: confidence of statistical test (def: 0.95); v: max deviance ratio for effective sample size (def: 0.01);
Output: counter-factual estimation of L(T) and decision to reject the null hypothesis (Ho:
I(x) 2 [.To)..
1. Split dataset D + Dirain Dvalid, Diest
2. Train π on Darain and tune policy class and optimization Syper-parameters on Dvalid (for e.g by internal cross-vale vanss-valiiaiion)
3. Estimate effective sample size (et(TT) on Dvalid
if Madl > then
Estimate (SNIPS(T) on Deest and test [sNirs(T) < ((TO on Drest with confidence 1-6 5. the test is valid, reject Ho, otherwise keep it.
else
Keep Ho, consider the estimate to be invalid.
end
A first, structuring choice is the IPS estimator. While variants of IPS cxist to reduce variance, such clipped
IPS, we found Sell-Normalized TPS (SNIPS, Swaminathan & Joachims 2015b: Lefortier et al. 2016 Owen 2013 Nedelec et al. 2017) to be more effective in practice. Indeed, it avoids the choice of a clipping threshold, generally reduces variance and is equivariant with respect to translation of the reward.
A second component is the use of importance sampling diagnostics to prevent propensity over-fitting. Lefortier ct al. (2016) propose to check if the empirical average of importance weights deviates from 1. However, there is no precise guideline based on this quantity to reject estimates. Instead, we recommend the effective sample size Nel (E ///3 wf, which measures how many samples diagnostic usable to perform estimation of the counterfactual estimate; we follow Owen (2013). who recommends to reject the estimate when the relative effective sample size neg/n is less than 1%. A third choice is a statistical decision procedure to check if I(TT) < [(ma)). In theory, any statistical test. against null hypothesis Hn: L(T) 2 L(TT) with confidence level 1 - can be used.
actually
Finally, we present our protocol in Algorithm 2 Since we cannot evaluate such a protocol on purely offline data, performed an empirical evaluation on synthetic setups where could analytically design true
12

---

Published in Transactions on Machine Learning Research (03/2025)
positive (L(T) < [(TT)) and true negative policies. We discuss in Section 7 the concrete parameters of Algorithm 2 and their influence on false (non-)discovery rates in practice.
Model selection with the offline protocol In order to make realistic evaluations, hyperparameter selection is always conducted by estimating the loss of a new policy in a counterfactual manner. This with propensitics obtained from the logging policy xo of
roquires using a validation set (or pross-validatiin the training set. Such estimates are less accurate than online ones, which would require to gather new data obtained from n, which we assume is not feasible in real-world scenarios.
To solve this issue, we have chosen to discard unreliable estimates that do not pass the effective sample size test from Algorithm 2 When doing cross valssatii,,,,n,, it implics discarding folds that do not pass the test, and averaging estimates computed on the remaining folds. Although this induces a bias in the vrossvvalidation procedure, we have found it to significantly reduce the variance and dramatically improve the quality of model selection when the number samples is small, especially for the Warfarin dataset in Section 7
7 Experimental Setup and Evaluation
We now provide an empirical evaluation of the various aspects of CRM addressed in this paper such policy class modelling (CLP), estimation with soft-clipping, optimization with PPA, offline model selection and evaluation. We conduct such a study on synthetic and semi-synthetic datasets and on the real-world CoCoAdataset.
7.1
Experimental Validation of the Protocol
First, we study the ability of Algorithm 2 to accurately decide if a candidate policy æ is better than a reference logging policy mo (condition L(x) < ((xo)) on synthetic data. Here we simulate logging policy to being a lognormal distribution of known mean and variance, and an optimal policy x" being a Gaussian distribution. We generate a logged dataset by sampling actions a ~ To and trying to evaluate policies with costs observed under the logging policy. We compare the costs predicted using IPS and SNIPS offline metrics to the online metric the setup synthetic, is then easy to check that indeed they are betler or worse than mo- We compare the IPS and SNIPS estimates along with their level of confidences and the influence of the effective sample size diagnostic.
Our first result. is that SNIPS estimates are highly correlated to the true (online) reward (average correlation p p= .968, 30% higher than IPS, see plots in Appendix D.I). Then, we have shown that with a proper choice of the and 8 parameters, it possible to control the False Discovery Rate (FDR) (< 10-4) and False Non- Son-Discovery Rate (FNDR) (< 5.10-4) on the same setup. We observed that on simple synthetic setups the effective sample size criterion For policies which were not close to the logging policy we found that standard statistical significance testing at 1 - 5 level was by itself not enough to guarantee a low FDR which justified the use of v. Adjusting the effective sample size can therefore influence the performance of the protocol (see Appendix D.2.2 for detailed results, D.3 for further illustrations of importance sampling diagnostics).
nentnn is seldom necessary for policies close to the logging policy (ir R mo)-
7.2 Experimental Evaluation of the Continuous Modelling and the Optimization Perspectives
In this section we introduce our empirical settings for evaluation and present our proposed CLP policy parametrization, and the influence of optimization in counterfactural risk minimization problems.
7.2.1
Experimental Setup
We present the synthetic potential prediction setup, a semi-synthetic setup as well as our real-world setup.
Synthetic potential prediction. We introduce simple synthetic environments with the following generative process: an unobserved random group index g in g drawn, which influences the drawing of a context æ and of an unobserved "potential" p in R, according to a joint conditional distribution Px Px.P Intuitively, the
13

---

Published in Transactions on Machine Learning Research (03/2025)
potential p may be compared to users priori responsiveness to treatment. The observed reward -y is then function of the context x; action a and potential p. The causal graph corresponding to this process is given in Figure 3
X
context
action
outecome
group label
Figure 3: Causal Graph of the synthetic setting. A denotes action, X context, G unobserved group label, Y outcome and unobserved potentials. Unobserved elements are dotted.
Then, we generale three datasets ("noisymoons, noisycircles, and anisotropic", abbreviated respect. "moons, circles, and GMM" in Table and illustrated in Appendix F.1 with two-dimensional contexts 2 or 3 groups and different choices of Px,Plg.
The goal is then to find policy Taaa)) that maximizes reward by adapting to an unobserved potential. For our experiments, potentials are normally distributed conditionally on the group index, plg ~> (Ng.. 0²). As many real-world applications feature a reward function that increases first with the action up to peak and finally drops, we have chosen a piecewise linear function peaked at = p (see Appendix F.1 Figure that mimics reward over the CoCoAdataset presented in Section G In bidding applications, a potential may represent unknown true value for advertisement, and the reward is then maximized when the bid (action) matches this value. In medicine, increasing drug dosage may increase treatment evfectiveness but if dosage exceeds a threshold, secondary effects may appear and eclipse benefits (Barnes & Eltherington 1966).
Semi-synthetic setting with medical data. We follow the setup of Kallus & Zhou 2018 using a dataset on dosage of the Warfarin blood thinner drug (War 2009). The dataset consists of covariates about patients along with dosage treatment prescription by medical expert, which is scalar value and thus makes the setting useful for continuous action modelling. While the dataset is supervised, Appendix F.1 a contextual bandit, environment by using a hand-crasted reward function that is maximal for actions that are within 10% of the expert's therapeutic drug dosage, following Kallus & Zhou (201)).
simulate (see details in
Evaluation methodology For synthetic datasets, we generate training, validation, and test sets of size 10000 each. For the CoCoA datasct, we consider a 50%-25%-25% training-validatinn- run each method with 5 different random intializations such that the initial policy is close the logging policy. Hyperparameters such as the regularization parameter X, the number of anchor points m, etc. are selected on validation set with logged bandit feedback as explained in Algorithm 2 The variance a of the policy classes that we consider are always set to the original variance of the logging policies. We use an oflline SNIPS estimate of the oal we consibtained policies, while discarding solutions deemed unsafe with the importance sampling diagnostic. On the semi-synthetic Warfarin dataset we used a cross-vard validation procedure to improve model selection due to the low dataset size. For estimating the final test performance and confidence intervals on synthetic and on semi-synthetic datasets, we use an online estimate by leveraging the known reward function and taking Monte Carlo average with 100 action samples per context: this leverage accounts for the randomness of the policy itself over given fixed samples. For offline estimates the randomness across samples to build confidence intervals: we use 100-fold bootstrap and take percentiles of the distribution of rewards. For the CoCoA dataset, we report SNIPS estimates for the test metrics.
sets. We then
7.2.2 Empirical Evaluation
We now evaluate our proposed CLP policy parametrization and the influence of optimization in counterfactural risk minimization problems. The code to reproduce our experiments can be found al Stttps://githun com/
criteo- crteerrreerrca//rrmmmaatttooooo. catpoouussactio----e crm
Continuous action space requires more than naive discretization. In Figure 4 we compare our continuous parametrization to discretization strategies that bucketize the action space and consider stochastic
14

---

Published in Transactions on Machine Learning Research (03/2025)
Test reward
0.8
04
Discrete
IPS Contiuous
5GM Contrinus
10
20
30
50
Nb anchor points
Figure 4: Continuous vs discretization strategies. Test rewards NoisyMoons dataset with varying numbers of anchor points for our continuous parametrization for IPS and SDM, versus naive discretization with softmax policies. Note that few anchor points are sufficient to achieve good results on this dataset; this is not the case for more complicated ones (e.g., Warfarin requires at least 15 anchor points).
discrete-action policies on the resulting buckets, using IPS and SDM. We add minimal amount of noise the deterministic DM in order pass the net/n > validation criterion, and experimented different hyperparamcters and models which were selected with the offline evaluation procedure. On all synthetic datasets, the CLP continuous modeling associated to the IPS perform significantly better than discrete approaches (see also Appendix G.1,, across all choices considered for the number of anchor points/buckets bbckets. To achieve reasonable performance, naive discretization strategies require a much finer grid, and are thus also more computationally costly. The plots also show that our (stochastic) direct method strategy, where we use the same parametrivation is overall outperformed by the CLP parametrization combined TPS, highlighting benefit of using counterfactual methods compared to direct fit to observed rewards.
Counterfactual cost predictor (CLP) provides competitive parameterization for continuous- action policy learning. We compare CLP modelling approach to other parameterized modelings (constant, linear and non-lincar described in Section 3.2p on our synthetic and semi-synthetic setups described
in Section 7.2.1 as well as the CoCoAdataset presented in Section 6.1
Noisycircles
NoisyMoons
Anisotropic
Warfarin
CoCoA
Logging policy mg
0.5301
0.5301
0.4533
-13.377
11.34
Constant
0.6115 + 0.0000
0.6116 + 0.0000
0.6026 + 0.0000
-8.964 + 0.001
11.36 + 0.13
Linear
0.6T13
0.0001
0.7326 + 0.0001
0.7638 + 0.0005
12.857
0.002
11.35 + 0.02
scIPS
Poly
0.6959 + 0.0001
0.7281 + 0.0001
0.7448 + 0.0008
10.36 + 0.11
CLP
0.7674 = 0.0008
0.7805 = 0.0004
0.7703
0.0002
-8.720
0.001
11.44 0.10
Constant
0.6115 1 0.0001
0.6115 1 0.0001
0.5930 1 0.0001
-9.511 1 0.001
11.32 1 0.13
Linear
0.6115
0.0001
0.7360 + 0.0001
0.7103 + 0.0003
-10.583
0.005
10.34 + 0.12
SNIPS
Poly
0.6969 + 0.0001
0.7370 + 0.0001
0.5801 + 0.0002
11.13 + 0.08
CLP
0.6972 + 0.0001
0.74091 + 0.0004
0.7899 0.0002
-9.161
0.001
11.48 +
Table 2: Test rewards (higher the better) for several contextual modellings (sce main text for details).
In Table we show a comparison of test rewards for different contextual modellings (associated to different. parametric policy classes). We show the performance and the associated variance of the best policy obtained with the offline model selection procedure (Section 6.2 Specifically, we consider grid of hyperparameters and optimized the associated CRM problem with the PPA algorithm (Section 4.2h. We report here the performances of scIPS and SNIPS estimators. For the Warfarin dataset, following Kallus & Zhou (2018), we only consider the linear context parametrization baseline, since the dataset has calegorical features and higher- dimensoonll contexts. Overall, we find our CLP parameterization to improve over all other contextual modellings, which highlights the effectiveness of the cost predictor exploiting the continuous action structure. As all the methods here have the same sample efficiency, the superior performance of our method can be imputed to the richer policy class we use and which better models the dependency of contexts
15

---

Published in Transactions on Machine Learning Research (03/2025)
and actions that may reduce the approximation error. We can also draw another conclusion: unlike syuthetic setups, it is harder to obtain policies that beat the logging policy with large statistical significance on the CoCoAdataset where the logging policy already makes a satisfactory baseline for real-world deployment. Only CLP passes the significance test on this dataset. This corroborates the need for offiinc evaluation procedures, which were absent from previous works.
Soft-clipping improves performance of the counterfactual policy learning. Figure $ shows the improvements in test reward of our optimization-driven strategies for the soft-clipping estimator for the synthetic datasets (see also Appendix G.3 The points correspond to different choices of the clipping parameter M, models and initialization, with the rest of the hyper-paramettrs optimized on the validation set using the offline evaluation protocol. This plot also shows that soft clipping provides benefits over hard clipping, perhaps thanks to a more favorable optimization landscape. Overall, these figures confirm that the optimization perspective is important when considering CRM problems.
NosyMoors Dataset
No syCriiiss Datase:
Antotrop c Dataust
820
M 2.6
0.54
0
24
1.
23
215
21.3
593
0405
6454
Mote lng
Mackiling
Mocel
Uheer
Clasiic cloping
clpplig
Clsss clop mg
Figure 5: Influence of soft-clipping Relative improvements in the test performance for soft- vs hard-clipping on synthetic datasets. The points correspond to different choices of the clipping parameter, models and initialization.
In Appendix G.2 we also illustrate how soft-clipping improves or competes with other clipping strategies Meeteli et al. 2021 Wang ct al. 2017 Su et al. 2019
Proximal point algorithm (PPA) influences optimization of non-convex CRM objective functions and policy learning performance. We illustrate in Figure t the improvements in test reward and in training objective of our optimization-driver strategies with the use of the proximal point algorithm (see also Appendix G.3 Here, each point compares the test metric for fixed models as well as initialization seeds, while optimizing the remaining hyperparameters on the validation set with the offline evaluation protocol. Figure (left) illustrates the benefits of the proximal point method when optimizing the (non-convex) CRM objective in wide range of hyperparameter configurations, while Figure G (center) shows that in many cases this improves the test reward well. In our experiments, have chosen L-BFGS because it was performing best among the solvers we tried (nonlinear conjugate gradient (CG) and Newton) and used 10 PPA iterations. As for computational time, for 10 PPA iterations, the computational overhead was about 3x in comparison to I--BFG without PPA. This overhead seems to be the price to pay to improve the test reward and obtain better local optima. Overall, these figures confirm that the proximal point algorithm improves performance in CRM optimization problems.
The scIPS estimator along with CLP parametrization and PPA optimization improves upon previous state of the art methods. We also provide a baseline comparison to stochastic direct methods, to (O-learn) of Chen et al. (2016) using their surrogate loss formulation for continous actions, to kernel smoothing (KS) (Kallus& & Zhou 2018) who propose a counterfactual method using kernel density estimation. Their approach is based on an automatic kernel bandwidth selection procedure which did not perform well on our datasets except Warfarin; instead, we select the best bandwidth on a grid through cross- cross-validatee validation and selecting through our offline protocol. We also investigate their self-normalized (SN) variant, which is presented in their paper but not used in their experiments: it turned out to have lower performances in practice. Moreover, we experimented using the generic doubly robust method from Demirer et al. (2019) but
16

---

Published in Transactions on Machine Learning Research (03/2025)
Moons Passoee
NoleyMoons Data
[aoggpod
44
16.5
Ertinati
Prosimal Print Alc
22
Fm
soff
1.5
Botter local minima 75%
0.00
Mudei ing
0.5
0.3
Inea pleceotaceertttt
0.00
Figure 6: Influence of proximal point optimization. Relative improvements in the training objective and w/o using the proximal point method (left) and relative improvements in the test performance w and w/o using the proximal point method (right).
could not reach satisfactory results using the parameters and feature maps that were used in their empirical section and with the specific closed form estimators for their applications. Nevertheless, by adapting their method with more elaborated models and feature maps, we managed to obtain performances beating the logging policy; these modifications would make promising directions for future research venues. We eventually also compare to (CATS) (Majzoubi et al. 2020) who propose an offline variant of their contextual bandits algorithm for continuous actions. We used the code of the authors and obtained poor performances with their offline variant. We do not provide results on their method for the CoCoAdataset as we could not access the logged propensities to use the offline evaluation protocol. For the SDM on CoCoA, we did not manage to simultancously pass the ESS diagnostic and achieve statistical significance, probably due the noise and variance of the dataset.
Noisycircles
Moosy
Anisotropic
Warfarin
CoCoA
Stochastic Direct Method
0.6205
0.0004
0.7225 + 0.0006
0.6383 + 0.0003
-9.714 + 0.013
O-learn
0.608
0.0002
0.645
0.0003
0.754
0.0002
99.407 + 0.004
11.03 0.15
KS
0.612 0.0001
0.731 # 0.0001
0.785 = 0.0002
100.99
11.38 + 0.07
SN-KS
0.609 0.0001
0.595
0.0001
0.652
0.0001
-2.569
0.001
9.14
CATS offline
0.589 0.0011
0.592 + 0.0011
0.569 = 0.0012
12.236 + 0.2548
Ours
0.767 = 0.0008
0.781 = 0.0004
0.770 + 0.0002
-8.720 + 0.001
11.44 + 0.10
Table Test rewards (higher the better) for previous methods for the logged bandit problem with continuous
actions
8 Conclusion and Discussion
In this paper, we address the problem of counterfactual learning of stochastic policies on real data with continuous actions. This raises several challenges about different steps of the CRM pipeline such as (i) modelization, (ii) optimization, and (iii) evaluation. First, we propose a new parametrization based on a joint kernel embedding of contexts and actions, showing competitive performance. Second, underline the importance of optimization in CRM formulations with soft-clipping and proximal point methods. We provide statistical guarantees of our estimator and the policy class we introduced. Third, we propose an oflline evaluation protocol and new large-scale dataset, which, to the best of our knowledge, is the first with real-world logged propensities and continuous actions. For future research directions, we would like to investigate the doubly robust estimator with the CLP parametrization of stochastic policies with continuous actions: this estimator achieves the best results in the discrete action case but requires further tochniques be adapted to continuous actions.
17

---

Published in Transactions on Machine Learning Research (03/2025)
Acknowledgments
HZ acknowledges support from the Gatsby Charitable Foundation. HZ also acknowledges support by the KARAIB AT chair MANR-20-CHIA-00255 and the H2020 Research Infrastructures Grant EBBAAN-Health 101058516. JM was supported by ERC grant number 101087696 (APHELEIA project) and by ANR 31A MInI@Grenoble Alpes (ANR-19-P31A-0003))
References
Estimation of the Warfarin dose with clinical and pharmacogenetic data. New England Journal of Medicine, 360(8):753-76) 2009.
Alekh Agarwal, Daniel Hsu, Satyen Kale, John Langford, Lihong Li, and Robert Schapire. Taming the monster: A fast and simple algorithm for contextual bandits. In International Conference on Machine Learning (ICML), 2014.
Zafarali Ahmed, Nicolas Le Roux, Mohammad Norouzi, and Dale Schuurmans, Understanding the impact of entropy on policy optimization. In International Conference on Machine Learning (ICML), 2019.
Charles Dee Barnes and Lorne George Eltherington. Drug dosage in laboratory animals: handbook. Univ of California Press, 1966.
Dimitris Bertsimas and Christopher McCord. Optimization over continuous and multi-dimensional decisions with observational data. In Advances in Newral Information Processing Systems (NewrIPS), 2018.
Christopher Bockkl--iickrrmann Toon Vanderschueren, Jeroen Berrevoets, Tim Verdonck, and Wouter Verbeke. Learning contimuous-valuel treatment effects through representation balancing. https://arziv..rr pwtrziiioorg//b//./ his/2309.03777 2023.
Léon Bottou, Jonas Peters, Joaquin Quiñonero Candela, Denis X. Charles, D. Max Chickering, Elon Portugaly, Dipankar Ray, Patrice Simard, and Ed Snelson. Counterfactual reasoning and learning systems: The example of computational advertising. Journal of Machine Learning Research (JMLR), 14(1)33207-3260, 2013.
Guanhua Chen, Donglin Zeng, and Michael R. Kosorok. Personalized dose finding using outcome weighted learning. Journal of the American Stalistionl Association, 111(516):1500-155 1521,, 2016.
Kyle Colangelo and Ying- Ying Lee. Double debiased machine learning nonparametric inference with continuous treatments. htlps://rrriv.rg/////// htrgg/////00...//////// hatp//0000.00066, 2023.
Mert Demirer, Vasilis Syrgkanis, Greg Lewis, and Victor Chernozhukov, Semi Demi-parametric efficient policy learning with contimuus actions. In Advances in Neural Information Processing Systems (NewrIPS), 2019.
Miroslav Dudik, John Langford, and Lihong Li. Doubly robust policy evaluation and learning. In International Conference Machine Learning (ICML), 2011.
Masao Fukushima and Hisashi Mine. A generalized proximal point algorithm for certain non-convex minimization problems. International Journal of Systems Science, 12(8):989 1000, 1981.
Keisuke Hirano and Guido W Imbens. The propensity score with continuous treatments. Applied Bayesian modeling and causal inference from incomplete-dale perspectives, 226164:73 84, 2004.
D. G. Horvitz and D. J. Thompson. A generalization of sampling without replacement from finite universe. Journal of the American Statislical Association, 47(260):663 685, 1952.
Nan Jiang and Lihong Li. Doubly robust off- off-policy value evaluation for reinforcement learning. In International Conference on Machine Learning (ICML), 2016.
Thorsten Joachims, Adith Swaminathan, and Maarten de Rijke. Deep learning with logged bandit feedback. In International Conference on Learning Representations (ICI.R), 2018.
18

---

Published in Transactions on Machine Learning Research (03/2025)
Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In Interna- tional Conference Machine Learning (TCMT.), 2002.
Nathan Kallus and Angela Zhou. Policy evaluation and optimization with continuous trealments. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2018.
Andreas Krause and Cheng S Ong. Contextual gaussian process bandit optimization. In Advances in Neural Information Processing Systems (NIPS), 2011.
John Langford and Tong Zhang. The epoch-greedy algorithm for multi-armed bandits with side information. In Advances in Newral Information Processing Systems (NIPS), 2008.
Haanvid Lec, Jongmin Lee, Yunscon Choi, Wonscok Jcon, Byung-Jun Lee, Yung-Kyun Noh, and Keo-Eung Kim. Local metric learning for olf-policy evaluation in contextual bandits with continuous actions. Thirty-Sizth Conference on Neural Information Processing Systems, 2022.
Damien Lefortier, Adith Swaminathan, Xiaotao Gu, Thorsten Joachims, and Maarten de Rijke. Large-scale validation of counterfactual learning methods: test-bed. arXiv preprint orXiv:6612.0037,, 2016.
Lihong Li, Wei Chu, John Langford, Taesup Moon, and Xuanhui Wang. An unbiased offline evaluation of contextual bandit algorithms with generalized linear models. In Proceedings of the Workshop on On-line Truding of Exploration and Exploilation 2, 2012.
Dong C Liu and Jorge Nocedal. On the limited memory bigs method for large scale optimization. Mathemalioal programmingg (15-3):500 528, 1989.
Maryam Majzoubi, Chicheng Zhang, Rajan Chari, Akshay Krishnamurthy, John Langford, and Aleksandrs Slivkins. Efficient contextual bandits with continuous actions. In Advances in Neural Information Processing Systems (NewrIPS), 2020.
Andreas Maurer and Massimiliano Pontil. Empirical bernstein bounds and sample variance penalization. In Conference on Learning Theory (COLT), 2009.
Alberto Maria Metelli, Alessio Russo, and Marcello Restelli. Subgaussian and differentiable importance sampling for off-policy evaluation and learning. In Advonces in Neural Information Processing Systems (NeurIPS), 2021.
Thomas Nedelec, Nicolas Le Roux, and Vianney Perchet. comparative study of counterfactual estimators. arXiv preprint arXiv: 11l4:7704.00 704.00773, 2017.
Art Owen. Monte Carlo theory, methods and eramples. 2013.
Courtney Paquette, Hongzhou Lin, Dmitriy Drusvyatskiy, Julien Mairal, and Zaid Harchaoui. Catalyst for gradient-based nonconvex optimization. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2018
James Robins and Andrea Rotnitzky. Semiparametric efficiency in multivariate regression models with missing data. Journal of the American Statistical Association, 90(429):122-112) 1995.
Ralph Tyrrell Rockafellar. Monotone operators and the proximal point algorithm. STAM journal on control and optimization, 14(5):877 898, 1976.
Noveen Sachdeva, Lequn Wang, Dawen Liang, Nathan Kallus, and Julian McAuley. Off-policy evaluation for large action spaces via policy convolution, 2023.
Yuta Saito, Qingyang Ren, and Thorsten Joachims. Off-policy evaluation for large action spaces via conjunct effect modeling. In International Conference on Machine Learning (ICML), 2023.
Bernhard Schölkopf and Alexander Smola. Learning with kernels: support vector machines, regularization, oplimization, and beyond. MIT press, 2002.
19

---

Published in Transactions on Machine Learning Research (03/2025)
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. Xiv preprint orXiu:170700633 2017.
Yi Su, Lequn Wang, Michele Santacatterina, and Thorsten Joachims. Cab: Continuous adaptive blending for policy evaluation and learning. In International Conference on Machine Learning (ICMT.), 2019.
Richard S Sutton, David McAllester, Satinder P Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Advances in Neural Information Processing Systems (NIPS), 2000.
Adith Swaminathan and Thorsten Joachims. Counterfactual risk minimization: Learning from logged bandit feedback. In International Conference on Machine Learwing (ICML), 2015a.
Adith Swaminathan and Thorsten Joachims. The self-normalized estimator for counterfactual learning. In Advances in Newral Informalion Processing Systems (NTPS). 2015b.
Yu-Xiang Wang, Alekh Agarwal, and Miro Dudík. Optimal and adaptive off-policy evaluation in contextual bandits. In International Conference on Machine Learning (ICML); 2017.
Christopher KT Williams and Matthias Seeger. Using the nyström method to speed up kernel machines. Ade. Neural Information Processing Systems (NIPS), 2001.
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229-2)) 256, 1992.
20

---

Published in Transactions on Machine Learning Research (03/2025)
This appendix is organized follows: in Appendix A present review of off-policy estimators, then in A Appendix B we present discussions and toy experiments to motivate the need for clipping strategies on real datasets. In Appendix C we present motivations for counterfactual estimators using importance sampling. Next, we provide motivations in Appendix D for the offline evaluation protocol with experiments justifying the need for appropriate diagnostics and stat lesting for importance sampling. Next, we provide the details of our analysis and the proofs of Propositions
in Appendix E Then, Appendix is devoted to experimental details that were omitted from the main paper for space limitation reasons, and which are results (sce also the code provided with the submission). Finally, in Appendix G we present additional experimental results to those in the main paper.
important for reproducing
The CoCoA dataset is available at the following link https://árive.goop www.g/owooo......oooooooooo oomnooennid-
IGCCYFNNqx htw.:///wesS11bbbbbbuuuuueeeee2
A
Review of Off-policy Estimators
Clipped eslimalor. In Equation [3), the counterfact.al approach lackles the distribution mismatch wetween the logging policy (orr and the evaluated policy in II via importance sampling and uses inverse propensity scoring (IPS, Horvitz & Thompson (1952)). However, the empirical (IPS) estimator has large variance and may overfit negative feedback values for samples that are unlikely under To (see motivation for clipped
estimators in B.1). resulting in higher variances. Clipping the importance sampling weights in Eq. 16
as Bottou et al. (2013) mitigates this problem, leading to a clipped (cIPS) estimator
{ [as] {
Lars(T))
Σ ==1
yb min
M
(16)
To.
Smaller values of M reduce the variance of ((T)) but induce a larger bias. Swaminathan & Joachims 2011 5a
also propose adding empirical variance penalty term controlled by a factor À to the empirical risk L(x) Specifically, they write (T(T)) = min [(a,(as)) Ro(e;|z;)) M) and consider the empirical variance for regularization:
(TPS((T) n
Σ - with ((T) (T(
1
(17)
n
which is subsequently used to obtain regularized objective L with hyperparameters M for clipping and X for variance penalization, respectively, so that QMPS((T) = >v Vapa(=) and:
((T) (APs((T) + +Xp Vaps(T)
(18)
n
The self-normalized estimator: /waminathan & Joachims 2015b) also introduce regularization mechanism for tackling the so-called propensity overfitting issue, occuring with rich policy classes, where the method would focus only on maximizing (resp. minimizing) the sum of ratios (a,,/2:)/m Fo, for negative (resp. positive)
costs. This effect is corrected through the following self-normalized importance sampling (SNIPS) estimator (Own 2013 see also), which is equivariant to additive shifts in cost values:
EL yow
with W T(a)kr)))
(SNNIS((T)
(19)
En
To
The SNIPS estimator is also associated to QSNIPS(TT) estimator that writes
AV (xN n
which uses an empirical variance
Ei= ui
(v: (T))))(T)))
(SNIPS(T)
(20)
EE wF))
21

---

Published in Transactions on Machine Learning Research (03/2025)
Direct methods. Such direct methods (DM) fit the loss values over contexts and actions in observed data with an estimator ((,aa) a); for instance by using ridge regression to fit yi R ( a;); and to then use the deterministic greedy policy fzM = arg min, fi(z,) a). These may suffer from large bias since it focuses on estimating losses mainly near actions that appear in the logged data but have the benefit of avoiding the high-variance problems of TPS-based methods. While such greedy deterministic policies may be sufficient. for exploitation, stochastic policies may be needed in some situations, for instance when one wants to still encourage some exploration in a future round of data logs. Using stochastic policy also allows us to obtain more accurate off off-policy estimates when performing valsdaaiio on logged data, Then, it may be possible to define a stochastic version of the direct method by adding Gaussian noise with variance o²:
ASDM(-1x) (t),
(21)
In the context of offline evaluation on bandit data, such smoothing procedure may also be seen as form of kernel smoothing for better estimation (Kallus & Zhou 2018).
Doubly robust eslimalors. Additionally, such direct loss estimators can be effective when few samples are available, and may be combined with IPS estimators in the so-called doubly robust estimator (DR, see, e.g. Dudik al. (2011)). This approach consists of correcting the bias of the DM estimator by applying IPS to the residuals 1/- (22, at), thus using v as control variate to decrease the variance of IPS. For discrete actions, the DR. estimator takes the form
(T)(T)= = 1 x(aspam)) (y: Ma,
1
Σ
T0,i
TI i= aEA
We provide a more in depth discussion on a doubly robust adaptation of our estimator in Appendix G.4
B Motivation for Clipped Estimators
In this section we provide an illustration of the logarithmic soft clipping and a motivation example for clipping strategies in counterfactual systems.
B.1 A Toy Example for Soft Clipping
Here we provide a motivation example for clipping strategies in counterfactual systems in a toy example. In Figure 7 we provide an example of large variance and loss overfitting problem.
We recall the data generation: a hidden group label in G is drawn, and influences the associated context distribution and of an unobserved potential p in It, according to joint conditional distribution Px, The observed reward is then a function of the context x, action a, and potential p. Here, we design one outlier (big red dark dot on Figure left). This point has a noisy reward r, higher than neighbors, and a potential p high as its neighbors have a low potential. We artificially added a noise in the reward function f that can be written as:
r r= f(, + 8 g~~ 100,
As explained in Section 7.2.1 the reward function is a linear function, with its maximum localized at the point x = p(x), Le.. at the potential sampled. The observability of the potential p only through this reward function f. Hercafter, we compare the optimal policy computed, using different types of estimators.
The task is to predict the high potentials (red circles) and low potentials (blue circles) in the ground truth data (left). Unfortunately, rare event sample with high potential is put in the low potential cluster (big dark red dot). The action taken by the logging policy is low while the reward is high: this sample is an outlier because it has a high reward while being a high potential that has been predicted with a low action. The resulting unclipped estimator biased and overfits this high reward/low propensity sample. The rewards of the points around this outlier are low as the diamcter of the points in the middle figure show. Inversoly, clipped estimator with soft-clipping succeeds learn the potential distributions, does not overfit the outlier, and has larger rewards than the clipping policies as the diameter of the points show.
22

---

Published in Transactions on Machine Learning Research (03/2025)
data
Actions sampled unclipped estimator
Actions sampled soft clipped astimator
1.0
r-X
p-P
1.0
r-X
1.0
(-X
--A
0.8
High
Low
Low
Medium
0.8
High
Low
Low
Medium High
0.8
Low
High
Low
Medium
gh
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
Figure 7: High variance and loss overfitting. Unlikely (mo. R 0) sample (II, (0.6, 0.) with high reward (left) results in larger variance and loss overfitting for the unclipped estimator (middle) unlike clipped estimator (right).
C Motivation for Counterfactual Methods
Direct methods (DM) learns reward/cost prodictor over the joint context-action space X A but ignore the polential mismalch between the evaluated policy and the logging policy and тo- When the logged data does not cover the joint context-action space X' x A sufficiently, direct methods rather fit the region where the data has been sampled and may therefore lead to overfitting (Bottou et al. 2013 Dudik et al. 2011 Swaminathan Joachims 2015b). Counterfactual methods instead learn probability cistributions directly with a re-weighting procedure which allow them to fit the context-action space even with fewer samples.
In this toy setting we aim to illustrate this phenomenon for the DM and the counterfactual method. We create synthetic 'Chess' environment of ini-dimensional contexts and actions where the logging policy purposely covers only small area of the action space, as illustrated in Fig, g The reward function is either 0, 0.5 or 1 in some areas which follow a chess pattern. We use lognormal logging policy which is peaked in low action values but still has a common support with the policies we optimize using the CRM or the DM.
Chessboard
1.0
0.8
0.6
0.6
Action
3-
0.4
2
0.2
0.2
1
0.0
0.2
0.4
0.2
0.6
1.0
Figure 8: 'Chess' toy synthetic setting (left) and lognormal logging policy (right).
Having set that enviromment and the logging policy, we illustrate Fig. the logging dataset, the actions sampled by the policy learned by Direct Method and eventually the actions sampled by counterfactual IPS estimator. To assess fair comparison between the two methods, we use the same continuous action modelling with the same parameters (CLP parametrization with m = 5 anchor points).
This toy example illustrate the mentioned phenomenon in how the counterfactual estimator learns a re- balanced distribution that maps the contexts to the actions generating higher rewards than the DM. The latter only learns mapping that is close the actions sampled by the logging and only covers smaller set of actions.
23

---

Published in Transactions on Machine Learning Research (03/2025)
Chesshoard
Chessboard
Chevsloard
1.0
28
04
04
04
144
04
32
00
0.
04
06
08
10
8.0
62
00
1.
00
0.
04
06
08
1.0
Conteot
Content
Figure 9: Logged data (left), action sampled by direct method (middle) and action sampled by counterfactual policy.
D Motivation for the Offline Evaluation Protocol
In this part we demonstrate the dflline/online correlation of the estimator we use for real-world systems and for validation our methods even syuthetic and semi-syyuhettcc setups. We provide further explanation ions of the necessity of importance sampling diagnostics and we perform experiments to empirically assess the rate of false discoveries of our protocol.
D.1 Correlation of Self-Normalized Importance Sampling with Online Rewards
We show in Figures T0 12 comparisons of IPS and SNIPS against an on-policy estimate of the reward for policies obtained from our experiments for linear and non-linear contextual modellings on the synthetic datasets, where policies can be directly evaluated online. Each point represents an experiment for model and a hyperparameter combination. We measure the R² score to assess the quality of the estimation, and find that the SNIPS estimator is indeed more robust and gives better fit to the on-policy estimate. Note also that overall the IPS estimates illustrate severe variance compared to their SNIPS estimate. While SNIPS indeed reduces the variance of the estimate, the bias it introduces does not deteriorate too much its (positive) correlation with the online evaluation.
These figures further justify the choice of the self-normalized estimator SNIPS Swaminathan & Joachims 2015b for oflline evaluation and validation to estimate the reward on held-out. logged bandit data. The SNIPS estimator is indeed more robust to the reward distribution thanks its equivariance property additive shifts and does not require hyperparameter tuning.
Dataset anisotropic offline/Online correlation, linear modelling
Dataset anisotropic Offline/Online correlation, non-linear modelling
5
672
R² = 0.60
R = .9.5
Above
SNIPS
Above
Above
Below
2
Below
0.0
02
0.4
0.6
0.8
0.0
0.2
0.4
0.6
0.8
0.0
0.6
0.0
0.2
0.6
R(n)
A(n)
R(m)
R(m)
Figure 10: Correlation between offline and online estimates Anisotropic synthetic data. Linear (left) and non-lineat (right) contextual modellings. Ideal fit would be y =x
D.2 Experimental validation of the protocol
We empirically evaluate our offline evaluation protocol on a toy setting where we simulate logging policy To being a lognormal distribution of known mean and variance, and an optimal policy T² being Gaussian distribution. We generate logged dataset by sampling actions mo and trying to evaluate policies & with
24

---

Published in Transactions on Machine Learning Research (03/2025)
Dataset noisycircies Offline/Online correlation, linear modelling
Dataset noisycircles Offline/Online correlation, non-linear modelling R² = 964
R?
962
R = 0 697
Above Delove
Above
Above no
Above ng
Below
Below m0)
0.6
0.2
0.6
Rim)
R(m)
Rtm)
Atm)
Figure 11: Correlation between offline and online estimates on NoisyCircles synthetic data. Linear (Ieft) and non-lineat (right) contextual modellings. Ideal fit would be y
Dataset noisymoons offline/Online correlation, linear modelling
Dataset noisymoons Offline/Online correlation, non-linear modelling
R² 0 716
R² 0. 949
=²=00. 0.719
A 1.943
Above n
Above rB
Above
SIIIS
Below no
Below mo
Below
Below
0.0
0.2
0.4
0.6
0.0
0.2
0.4
0.6
0.0
0.2
0.4
0.6
0.0
0.2
0.4
0.6
R(m)
Figure 12: Correlation between offline and online estimates on NoisyMoons synthetic data. Linear (Ieft) and non-linear (right) contextual modellings. Ideal fit would be y = x.
costs observed under the logging policy. We compare the costs predicted using IPS and SNIPS offline metrics to the online metric as the setup is synthetic, it is then easy to check that indeed they are better or worse than Th. We compare the IPS and SNIPS estimates along with their level of confidences and the influence of the effective sample size diagnostic. Offline evaluations of policies illustrated Figure 13 are estimated from logged data (zi, ais Mis mo) where a; To(-)r;) and where the policy risk would be optimal under the oracle policy n².
0.8
Logging polkc mo
0.7
Optimal policy n
Evaluated policy
0.8
0.5
0.4
0.3
0.
0.1
0.0
Actions sampled
Figure 13: Illustration of policies: logging policy To, optimal #² and example policy f.
While the goal of counterfactual learning is to find policy á which is as close a possible to the optimal policy x", based on samples drawn from a logging policy mn, it is in practice hard to assess the statistical significance policy that is too 'far' from the logging policy. Offline importance sampling estimates are indeed limited when the distribution mismatch between the evaluated policy and the logging policy (in terms of KL divergence (*ol(a)) large. Therefore we create setup where we evaluate the quality of offline estimates for policies (i) "close" to the logging policy (meaning the KL. divergence D ( is low) and
25

---

Published in Transactions on Machine Learning Research (03/2025)
(ii) "close' to the oracle optimal policy (meaning the KL divergence (T= is low). In this experiment, we focus on evaluating the ability of the offline protocol to correctly assess whether I(a) I [(mo) or not by comparing to online truth estimates. Specifically, for both setups (i) and (ii), we compare the number of False Positives (FP) and False Negatives (FN) of the two offiine protocols for N = 2000 initializations, by adding Gaussian noise to the parameters of the closed form policies. False negatives are generated when the offline protocol keeps Ho L(x) 2 [(mo) while the online evaluation reveals that [(TT) S [(mo), while false positives are generated in the opposite case when the protocol rejects Ho while it is true. We also show histograms of the differences between online and offline boundary decisions for (L(ft) < [(To)), using bootstrapped distribution of SNIPS estimates to build confidence intervals.
D.2.1 Validation of the use of SNIPS estimates for the offline protocol
To assess the performance of our evaluation protocol, we first compare the use of TPS and SNIPS estimates for the offline evaluation protocol and discard solutions with low importance sampling diagnostics laf < v with the recommended value v 0.01 from Owen 2013). In Table 4 we provide an analysis of false positives and false negatives in both setups. We first observe that for setup (i) the SNIPS estimates has both fewer false positives and false negatives. Note that is setup is probably more realistic for real-world applications where we want to ensure incremental gains over the logging policy. In setup (ii) where importance sampling is more likely to fail when the evaluated policy is too "far' from the logging policy, we observe that the SNIPS estimate has a drastically lower number of false negatives than the IPS estimate, though it slightly has more false positives, thus illustrating how conservative this estimator is.
Setup (i)
Setup (ii)
Offline Protocol
IPS
SNIPS
IPS
SNIPS
# M To
Keep Ho
To
Keep Ho
f > mo
Keep Ho
# h To
Keep Ho
# 2 WD
1282
24
1296
10
1565
67
1631
1 362
"Truth"
Keep Ho
19
675
0
694
368
6
Table 4: Comparison of false positives and false negatives: Perturbation to the logging policy To (setuu (i)) and perturbation to the optimal policy (setup (ii).. The SNIPS estimator yields less FN and FP on setup (i), while being more effective on sctup (ii) as well by inducing a drastically lower FP rate than IPS and low FN rate. The effective sample size threshold is fixed at 0.01
We then provide in Fig. 14 histograms of the differences of the upper boundary decisions between online estimates and bootstrapped offline estimates over all samples for both setups (i, left) and (ii, right). Both histograms illustrate how the IPS estimate underestimates the value of the reward with regard to the online estimate, unlike the SNIPS estimates. In the setup (ii) in particular, the IPS estimate underestimates severely the reward, which may explaín why IPS has lower number of false positives when the evaluated policy is far from the logging policy. However in both sctups, IPS has higher number of false negatives. We also observed that our SNIPS estimates were highly correlated to the true (online) reward (average correlation = ,968 30% higher than IPS, see plots in Appendix D.1 for the synthetic setups presented in section 7.2.1 which therefore confirms our findings.
D.2.2 Influence of the effective sample size criteria in the evaluation protocol
In this sctup we vary the effective sample size (ESS) threshold and show in Fig. 15 how it influences the performance of the offline evaluation protocol for the two previously discussed setups where we consider evaluations of (i) perturbations of the logging policy (left) and (ii) perturbations of the optimal policy (right) in our synthetic setup. We compute precision, recall and Fl scores for each threshold values between 0 and 1. One can see that for low threshold values where no policies are filtered, precision, recall and F1 scores remain unchanged. Once the ESS raises above a certain threshold, undesirable policies start being filtered but more false negatives are created when the ESS is too high. Overall, ESS criterion is relevant for both setups. However, we observe that on simple synthetic setups the effective sample size criterion is seldom necessary for policies close to the logging policy (it A To). Conversely, for policies which are not close to the logging policy the standard statistical significance testing at - 8 level was by itself not enough
net/n
26

---

Published in Transactions on Machine Learning Research (03/2025)
Histogram (online reward offline reward)
Histogram (online reward offline reward)
ips
snips
snips
Number of samples
Number of samples
30
25
20
10
10
5
-0.05
Figure 14: Histogram of differences between online reward and offline lower confidence bound. Perturbation to the logging policy (left), perturbation to the optimal policy x² (right). Effective sample size threshold v = 0.01
to guarantee low false discovery rate (FDR) which justified the use of v. Adjusting the effective sample size can therefore influence the performance of the protocol (see Appendix D.3 for further illustrations of importance sampling diagnostics in what-if simulations).
Effective Sample Size influence 17p
Effective Sample Size influence
Perfomence
Performance
0.8
Precision
0.7
Precision
Recal
Recall
0.2
0.4 ESS Ihreshold
0.3
0.7
0.1
0.2
0.3
0.8
0.7
ESS threshold
Figure 15: Precision, recall and F1 score varying with the ESS threshold synthetic setups (i) and (ii). Sctup (i) perturbation of the logging policy (left.) and setup (ii) perturbation to the optimal policy (right). The ESS threshold can maximize the F1 score.
D.3 Importance Sampling Diagnostics in What-lf simulations
Importance sampling estimators rely on weighted observations to address the distribution mismatch for offline evaluation, which may cause large variance of the estimator. Notably, when the evaluated policy differs too much from the logging policy, many importance weights are large and the estimator is inaccurate. We provide here a motivating example ratustrate the effect of importance sampling diagnostics in a simple scenario.
When evaluating with SNIPS, consider an "effective sample size" quantity given in terms of the importance weights w; <aa,(a;)/m (a/ (z;)) by n. (Σ w,)²/ E wf. When this quantity is much smaller than the sample size n, this indicates that only few of the examples contribute to the estimate, that the obtained value is likely a poor estimate. Apart from that, we note also that IPS weights have an expectation of 1 when summed over the logging policy distribution (that is ((r,))= [T(((>))/ TO (a[=)) = 1.). Therefore, another sanity check, which is valid for any estimator, is to look for the empirical mean 1/n Σn T(a, [x;])/ To, and compare its deviation 1. In the example below, we illustrate three diagnostics: (i) the one based on effective sample size described in Section 0 (ii) confidence intervals, and (iii) empirical mean of IPS weights. The three of them coincide and allow remove test estimates when the diagnostics fail.
27

---

Published in Transactions on Machine Learning Research (03/2025)
Example D.1. What-if simalation: For in RR, let max(æ) ma wish to estimate E(max(X)) for X Lid Ft o), where samples are drawm from logging policy zo (No, ao) (d 3, (Ao, 00) = (1,1/2)) and analyze parameters around the mode of the logging policy Pro with fixed variance o 1/2. In this parametrized policy ezample, we see in Fig. 16 that ne/n < 1, confidence interval range increases and Σ m (ms) + 1 when the parameter μ of the policy being evaluated is far away from the na,*
logging policy mode PAa
Effective sample rignn
Estimation
Importance sampling weights mean A5
10a
10
0.6
10
Threshold
Theoretical mean
10
Pe: made
Hey
0.2
He, Empirical mean
10-1
2
3
0
1
2
3
5
0
1
2
3
4
5
Figure 16: Importance sampling diagnostics. Ideal importance sampling: i) effective sample na/n close 1, ii) low confidence intervals (C.L.s) for m, iii) empirical mean Σ: w, close to 1. Note that when p differs too much from Mooo importance sampling fails.
Note that this example, the parameterized distribution that learned (multivariate Gaussian) is not the same as the parameterized distribution the logging policy (multivariate Lognormal) which skewness may explain the asymmetry of the plots. This points out another practical problem: even though different parametrization of policies is theoretically possible, the probability density masses overlap is in practice what is most important to ensure successful importance sampling. This observation is of utmost interest for real-life applications where the initialization of policy to be learned needs to be "close' to the logging policy; otherwise importance sampling may fail from the very first iteration of an optimization in learning problems.
E Analysis of the Excess Risk
In this appendix, we provide details and proofs on the excess risk guarantees that are given Section 4 and 5 We start by recalling the definitions of e covering and the one of our soft-clipping operator Ç provided in Eq. 11
Definition E.1 (epsilon covering and metric entropy). An e-covering a subset Ao C. such that is contained in the union of balls of radius e centered in points in Ag, in the metric induced by a norm -l The cardinality of the smallest e-covering is denoted by H(e, A, I]) and its logarithm is called the metric entropy.
For any threshold parameter M 2 and importance weight w 0, the soft-clip operator Ç is defined by
if w M ((w, M) = aaM log (w a(M) M) he
w
where a(M) is such that a(M) log(a(M)) = M.
E.1 Ommitted Proofs
We start by defining our complexity measure C₂ (11, M), which will be upper-bounder by the metric entropy in sup-norm at level g 1/n of the following function sct,
T(a|x) FI.M = { (I, (, + 5 To(az))
M)
for some E T
(22)
28

---

Published in Transactions on Machine Learning Research (03/2025)
where S= M). The function set corresponds to clipped prediction errors of policies л normalized into [0, 1.. More precisely, to define rigorously (TI, M), we denote for any 2 and >> 0, the complexity of a class F by
sup where F(fxs Ci v.. (1. (o...,,,.................. Vn)) f F) CR. Then, C, (II, is defined by CII, log Hoc (1/n, FI,M, 2n).
Ho, (1s ((
F(Ex, ai vet). Ilae):
(23)
(24)
We are now ready to prove Proposition 4.1 that we restate below.
Proposition 4.1 Generalivation bound for (TT)) Let II be a policy class and no a logging policy, under which an input-action-cost triple follows DTo- Assume that -1 -1 S y S a.s. when (x, a,y) "Do and that the importance weights are bounded by W. Then, with probability at least 1-6, 6, the IPS estimator with soft clipping 12 samples satisfies
( + log ((,, log })
Vx E II,
I(x) ((t)stuerrs(n +
where S= ((W, (II denotes the empirical variance of the cost estimates 20 and (II, is complezity measure (24) of the policy class.
Proof. Let IT be a policy class, To be logging policy, and 5 >> O. Tet M 2 0 be threshold parameter, W 2 supa.- (=(l////oo(aaa)))) 2 0 a bound on the importance weights, and $ <(W,
Let first consider the finite setting, which case (IT, M) S log [T] Since all functions in Fn,M defined Eq. (22) take values in [0, 1], we can apply the concentration bound of Maurer & Pontil (2009 Corollary 5)
to FI,M, which yields that with probability at least 1 8, for any π E II
1 Σ
(T) 11)8)
7 7log(2
B(r,ay) If (x,
(a as (a u)
(25)
n
3(n 1)
where (T) is the sample variance defined in 13 Furthermore, note that by construction of the fa, for any π I1,
E ((z.an) (f(( Ff 1+ (m)
x fa (Zt, ai 9i) 1 ++ LECIPS (T)
and
Σ n
where L (TT) ((,,a)) [ M)] denotes the clipped expected risk of the policy and LEaIPS(M) (T) is defined in (12). Thus, multiplying 25 by S and using that L(T)) < L (T) (since y S 0 and ((w,AM M) I w for all w), we get that with probability 1- 8,
7 (2 2 2ianss log logg2/111) S 3(n (---1)
L(T) < (TC +
VT 11.
"
The finite setting may finally be extended to infinite policy classes by leveraging Maurer & Pontil (2009 Theorem 6) as in (Swaminathan & Joachims 2015a). This essentially consists in replacing [II] above with an empirical Eo covering number of Fu,M of size Hox (1/n. 2n). Note that the number of empirical samples 2n due to the double-sample method used by Maurer Pontil 2009).
We now state the excess risk upper-bound Proposition E.1 and provide the proof. The following proposition is intermediate result. that will allow us to derive the Proposition 5.1
29

---

Published in Transactions on Machine Learning Research (03/2025)
Proposition E.1. Consider the notations and assumptions of Proposition 4.1 Let RCRM be the solution of
the CRM problem in Fg. (15) and π²" II be any policy. Then, the choice À = =333(Ca (TI, M) log(30/6))
implies with probability at least 1 - å the following upper-bound on the excess risk
30 5 (Cn M) + log 225 (nn(Cn M) + log 5 + har har(m"),
30
) -
V
where VM (T
and hM (T)) are the variance and bias of the is the clipped estimator of x- and respectively
defined in 26 and 29
4.1 Fix n² II. Applying, Theorem 15 of Maurer & Pontil 2009 to the function set FILM defined in 22 get w.p. -
Proof. We consider the notations of the proof of Proposition
((r, [sRCRM (a, a, v)] - ((r Pro [f (I, a, v)]
V2 [ (x, a, v)) (II, .M + log ) 22 (II, + log 30
Using the definition of fx(x, a. 22 we have
L M (T) P( (I, =1
(2((T)
and (a [fn (x,
82
where
(T) (T) (a, (uS x(a|2))
M
(26)
(o(a)
Substituting into the previous bound, this entails
32V VVM log 30
22, M) log 30)
IM (MCRRM IM (TT) $
(27)
n -
To conclude the proof, it only remains to replace the clipped risk LM with the true risk L. On the one hand, since the costs y take values into [-1, 0), we have y< (ar (alx)/mo(a(sj) M) 2 ==(al=)/=((a(t) which yields
L(*CRM) S L ((CRMM)
(28)
On the other-hand, by defining the bias
has(T") = B(maaw)P. us (M, n" (alz))
(ala)
(o(a)x))
y (a(a)
(29)
we also have -( hm S MMT)), which together with (27 and 28 finally concludes the proof
L(*CRM) - L(xx)= S (C, M) log
225 (C, M) + log ) + (xr(
We can now use the latter to prove Proposition 5.1 that is restated below.
Note that their notation, Mx(x) equals CI M) + log(10), X is the dataset (f where (11,0190) vi) Paus and P() the expectation with respect. one test sample F( [.].
30

---

Published in Transactions on Machine Learning Research (03/2025)
Consider the notations and assumptions of Proposition 4.1 Let #CRM be the solution of the CRM problem in Fg.. 15 and x²" ET be any policy. Then, with well chosen parumeters X, denoling the variance 0²= = Var.o TT" (a x /o(aaa))
Proposition 5.1
with probability at least 1 1
- -((T) 1 + ( M) + log log log(W + cC, (II, M) log !)
n
where I hides universal maltiplicative constants. In porticular, assuming also that To (2)aa) are uniformly bounded, the complexity of the class Hor described in Section 3.2 applied with a bounded kernel and θ { Rmm s.t 1|11 SC {o},
is of order
CI M) S O(mlogn)
where m is the size of the Nyström dictionary and O(-) hides melllppliaative conslants independent of n and m (see 38
Proof. We first consider a general policy class II and some ग" E II. In this proof, to ease the notation, we write [ Var [-], and Po (-) to respectively refer ((a.a,)) [.], (a. [.] and ((..a))PPP)) (.).
We consider the notation of the proof of Proposition E.1 and start from its risk upper-bound
(C (II,
32VM(TT*
log )) 22 (II, M) log 30)
L(*CRM)- - L( $
+ hmr("',
(30)
n
where we recall, for any threshold M, the definitions of the bias and the variance of the clipped estimator of n²,
hM hM(TT) = Ero [v<<(T)(
(alzz) (ake M) To(aa (T(())))
xt (a|xx) and V (1T) Var
n (a(zz)
.21) ]
yC
M
(a(a)z)
To (ro(alz))
Step 1: For any threshold M, upper-bound of the variance Var (T²") and the bias has (ar( By assumption, the (unclipped) variance x /To is bounded and we write
x (a|x) [a(a)))
(
n (ala) - -1) ))
o.
Var
E (m.0,))-P7
First, we bound the clipped variance as
(=(("*)= = Varz. US
M <o(a|x=))
= Ero
v²c n (akor
M )
27
Er uc
To x)) M
To(a)x)
(T' (alt)) 27
E
T, M (x ( = = +1+1 - I, (TT. 0² +1. (31)
To(a
Then, by writing X = ===<(a/=)=o(4d) the bias may be upper- upper-bounder bounded as
hM(TT) s E X - ((X,
EsD (X - M))1( MH
Pεo 1-X-M1 (X) M} > t) t)at
100 P.. (XX 1)2 (t M 1)2)
100
P (X - M > t)dt =
Ex (X 1)2]
Em
[[X - 1)2
a²
di
(32)
Jo
(t M 1)²
M 1
M
31

---

Published in Transactions on Machine Learning Research (03/2025)
Furthermore, if W M then S <(W, W M, else, using ((M) M/ M/log(a(MM) & max(M,c) v
M +
S = M) ((M) log W + a(MM) M) - (M + e) log(W +e).
(33)
Therefore, substituting
31
32
and
into 30
yields the following upper-bound the excess risk
L(#CNM) L(TT)
30 a 32(1 + (C,( M) + log
22(M + log( + (C, + log 30 o
n-1
M
We now choose M such that
22(M - log (W + (I, + log 30
o²
(34)
n -
M
which is possible since the left term grows from 0 to infinity and the right term decreases from infinity to 0 for M > 1. Therefore, from the last two terms we eventually have
(1+ (1+ log( + (II, + log +
log(W + (C, + log })
( S
(35)
n
where $ hides universal multiplicative constants. This concludes the first part of the proof.
Step 2: Evaluating the policy class complerity C. (HIL,, M).
In this part, we provide a bound on the metric entropy (lICL... M) = log Hoo (1/n, FiELL 2n). We recall that Fner is defined in 22 and NeLL is described in Section 3.2 More preciscly, let 2 C A be a Nyström
dictionary size m 2 and 7 > O. Since we use Gaussian distributions, we have
ICL {T s.t. for any X, Ta(-1=) N44gg (z), (z), with $ e e
where
θ = {B E ET s.t ||1| c
where
exp( Ma a))
CLP 13
aE2 Ex'ez (Mx and no(x, = (B, $( ()),
for some embedding va described in Section 3.2 which satisfies (1, S for any (x, a). Fix x E X. Let us show that B (x)) is Lipschitz. Denote by Zz(x) = EaEz exp(->ng(x, a)) the normalization factor. We consider the gradient of ug(F(x) with regards /
(1 a) exp ((3, (1 a)))
0.3 (x) (x)= = Σe
(z)
aEZ
exp (13, (b, a)))
Enc (b exp (13, w( ((,))))))) (x(x)²
Taking the norm, and upper-bounding [],( a)l I e and ||a|| s az, this yiclds
@ugi
(m) s 2vaz.
03
(z) is 2va Lipschit, which implies that Therefore, 8 + ug (x)
11
B + [
exp
ov2
2
32

---

Published in Transactions on Machine Learning Research (03/2025)
are also Lipschitz with parameter
2 vaz En x²
(36)
We recall that the metric entropy Cn,(( log No (1/n, Fue 2n) is applied to the function class
TT((az)) FILLL = fs {fs: (2, - 1+ 5 (
M M) for some vp E HCLLP
By assumption, the inverse of the logging policy weights are bounded mo(alz) S Mo for any (x, a) E X A (as in Kallus & Zhou (2018)). Therefore, together with (36]. for any (z, X A x Vi the function G + (a v) is Lipschitz with parameter
(37)
So²
Let 6 > 0. Because there exists an e-covering of the ball {3 E Rd || z C} of size (C/e),, together
with 37 : the latter provides covering FilLLe in nurm with parameter
2 va
eV eπ So²
Equalixing this with n=1 and taking the log of the size of the covering entails
(V2 C.
Cn (ULLP, S dlogg
et
So²
Now, we recall that is the dimension the embedding v which we model as
(xr = vx(x) & (a(
where (a) E Rn is the embedding obtained by using the Nyström dictionary of size m on the action space and (x) E Rdx is the embedding the context space X' C Rdd, Typically dx = de + dg + or dxx = dz respectively with the polynomial and linear maps considered in practice. Thus, d mdx. Substituting the latter into the complexity upper-bound and using S S and M, we finally get.
(V2z 2 CMCuvaz a²
(H M) S mdx log
(38)
where recall that dx is the dimension of the contextual feature map, C' a bound on the parameter norm B, Mo bound na =1 v² a bound on the kernel, o² the variance of the policies, and bound on the action norms ||a]]-
E.2 Discussion: on the Rate Obtained for Deterministic Classes
Consider the deterministic CLP class that assigns any input x to the action 10 (x)( defined in 7 Although the paper focuses on stochastic policios, this appendix provides an excess-risk upper-bound with respect to this deterministic class.
The latter corresponds to the choice = 0 in the CLP policy set defined in Section 3.2 and therefore, Proposition 5.1 cannot be applied directly. Fix some o² > to be optimized later. For any B E R" we denote by L(ugh) the risk associated with the deterministic policy a = (x) and define T3 3-1z) ~ N (2) o²). Then, let œCRM be the counterfactual estimator obtained by Proposition 5.1 on the class
HCLP (-g (+(x)) with probability 1 5
L(#CRM)- - L(u CL. < ((#CRM) - L(TTTP)) + L(T
- L(u CLP
2
< L(#CRM) - L(T + Loo
2log
8
33

---

Published in Transactions on Machine Learning Research (03/2025)
where assumed that the risk is Lo-Lipschtitz and used that P([X) < o I 8 for X ~ N(), From Proposition 5.1 this yields, with probability - 28
(1+ log + (C, (II... M) + log 1) + Loo 22og
2
L(*CRM) L(g
n
Now, note that C. M) and log(W) only yield logarithmic dependence o² and and will thus not impact the rate of convergence. The variance o² has stronger dependence on o² but can be upper-bounded as follows
(a (a/x) To(a|2)
- 1
TB 1 (alz) To(alz)) 2 To no(a|z)da
0,- = Var
Tu(a(z)
33 f
CUP (a|z))
Mo / "gt (a)x)) Moya
1
1
da (a(a
where the last equality is because T T (+1x) is Gaussian distribution with variance o². Therefore, keeping only the dependence on a and and neglecting log- log-oactors, we get the high-probability upper-bound
[(*CRM) - L(ug S 0
1 Von ( +0)
The choice a = -1/3 entails a rate of order
F
Details on the Experiment Setup and Reproducibility
In this section we give additional details on syuthetic and semi-syuthetic datasets, we provide details on the evaluation methodology and information for experiment reproducibility.
F.1 Synthetic and Semi-Synthetic setups
As many real-world applications feature a reward function that increases first with the action, then plateaus and finally drops, we have chosen piecewise linear function as shown in Figure I7 that mimics reward buckets over the CoCoA dataset presented in Section B
Synthetic setups
Synthetic
rja.p
f/p
Reward
Roward
-1
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
0.6
0.8
1.0
1.2
1.4
Action taken
Action taken
Figure 17: Synthetic reward engineering. buckets (right).
The synthetic reward (left) is inspired from real-dataset reward
We provide an illustration of the three synthetic datascts Figure
18
Semi-synthetic medical setup The semi-synthetic cost inputs prescriptions from medical experts to obtain y(a, x) max((a- 1 0.1f*, 0), to mimic the expert prediction. The logging policy mo samples actions a ~ contextually to patient's body mass index (BMI) score ZBMı #BMI-A9M and can be analytically written with i.i.d noise e ~ (0, 1), moments of the therapeutic dose distribution pr, of such that a 14 00z + d 8E (0 0.5 in the setup of Kallus & Zhou (2018)). The logging probability density function thus is continuous density of a standard normal distribution over the quantity a-Hi to V0Zam: 01v1-8
daMr
34

---

Published in Transactions on Machine Learning Research (03/2025)
nolsymoons
noisycircles
anisotropic
Figure 18: Contexts (points in IF), and potentials represented by color map for the synthetic datasets.
Learned policies should vary with the context to adapt to the underlying potentials.
F.2 Reproducibility
We provide code for reproducibility and all experiments were run on a CPU cluster, each node consisting on 24 CPU cores (2 x Intel(R) Xeon(R) Gold 6146 CPU@ 3.20GHz), with 500GB of RAM.
Policy parametrization. In our experiments, we consider two forms of parametrizations:: (i) lognormal distribution with 0 = (0,, T(u.o) log. (mg,(m,, s) with /log (o²/ I); log(u) s2/2, -=g(2)) that Bawn [a] μ and Varawaaee) [a] = o²; (ii) normal distribution T(u.o) [N,,,, o). In both cases, the mean may depend on the context (see Section 4 while the standard deviation a is a learned constant. We add a positivity constraint for a and add an entropy regularization term to the objective in order to encourage exploratory policies and avoid degenerate solutions.
Models. For parametrized distributions, we experimented both with normal and lognormal distributions on all datasets, and different baseline parameterizations including constant, linear and quadratic feature maps. We also performed some of our experiments low-dimensional datasets with stratified pieco-wise contextual parameterization, which partitions the space by bucketizing each feature by taking K (for e.g K = 4) quantiles, and taking the cross product of these partitions for each feature. However this baseline is not scalable for higher dimensional datasets such as the Warfarin datasot.
Hyperparameters. In Table 3 we show the hyperparameters considered to run the experiments to reproduce all the results. Note that the grid of hyperparameters is larger for synthetic data. For our experiments involving anchor points, we validated the number of anchor points and kernel bandwidths similarly other hyperparameters.
Syathetic
Warfarin
CoCoA
Variance reg. A
(0., 0.001, 0.01, 100)
(0., 0.001, 0.1)
Clipping
11,17,2.8, 4.6, 7.7, 12.9,215, 35.9, 59.9, 100.0)
21, 4.5, 9.5, 20)
(1,2.1, 9.5, 10, 20, 1001
Prox. K
[0.001, 0.01, D.
{0.001, 0.01, 0.01, 0.1)
(0.001, 0.01, 0.1]
Hrg. param.
(0.00001) 0..0000,0.00001,00 0.01,60)
(0.00001, 00.00,,00 0.01,
(0.00000,, 0,0001, 0.000, 0.01,
Number of auchor points Solimax
3,, 7,
7, 10, 15,
[2,3,5)
10, 100)
[1,5, 10)
(0.1,
Table 5: Table of hyperparameters for the Synthetic and CoCoA datasets
G Additional Results and Additional Evaluation Metrics
In this section we provided additional results on both contextual modeling and optimization driven approaches of CRM.
G.1 Continuous vs Discrete strategies in Continuous-Actin Action Space
We provide in Figure 19 additional plots for the continuous vs discrete strategies for the synthetic setups described in Section 7.2.1
35

---

Published in Transactions on Machine Learning Research (03/2025)
PS Decade
Decreia PPa Condrncus
66
10
22 pric
Figure 19: Continuous vs discrete. Test rewards for CLP and (stochastic) direct method (DM) with Nyström parameterization, versus a discrete approach, with varying numbers of anchor points. We add a minimal amount of noise to the deterministic DM in order to pass the Belf validation criterion.
G.2 Soft clipping improves or competes with other importance weighting transformation strategies
Soft-clipping improves or competes with other importance weighting transformation strategies. We also experiment on the synthetic datasets comparing our soft clipping approach with other methods which focus is to improve upon the classic clipping strategy for the same optimization purposes. Notably, we consider the power-mean correction of importance sampling (PMCIS) (Metelli et al.) 2021 method which we adapt to our continuous modeling strategy to enable fair comparison. Moreover, we also added the SWITCH (Wang et al. 2017) as well as the CAB (Su et al., 2019) methods. However, both methods use a direct method term in their estimation, which is difficult to adapt for stochastic policies with continuous actions, as explained in our discussions on doubly robust estimators (see Appendix G.4). Therefore, we considered discretized strategies and compared them with soft clipped estimator applied to discretized policies. For the discretized strategics, we have used the same anchoring strategies as described before, namely using empirical quantiles of the logged actions (for 1D actions), and have optimized the number of anchor points along with the other hyperparameters using the offline evaluation protocol. We see overall in Table G that our soft-clipping strategy provides satisfactory performance or improves upon all weight transforming strategies on the synthetic datasets.
Noisymoons
Noisycircles
Anisotropic
Logging policy To
0.5301
0.5301
0.4533
SWITCH (discrete)
0.5786 + 0.0025
0.5520 + 0.0026
0.5741 + 0.0024
CAB (discrete)
0.5761 + 0.0024
0.5534 = 0.0025
0.5705 + 0.0021
scIPS (discreto)
0.5888 = 0.0022
0.5637 + 0.0024
0.5941 + 0.0019
PMCIS (CLP)
0.7244 + 0.0005
0.7189 + 0.0004
0.7739 + 0.0008
scIPS (CLP)
0.7674 = 0.0008
0.7805 + 0.0004
0.7703 + 0.0002
Table 6: Comparison of importance weight transformations on the synthetic datasets, for discretized straeggies and for continuous action policies.
G.3 Optimization Driven Approaches of CRM
In this part we provide additional results on optimization driven approaches of CRM for the Noisycircles, Anisotropic, Warfarin and CoCoAdatasets.
Both Noisycircles and Anisotropic datasets in Figure 20 show the improvements in test reward and in training objective of our optimization-driven strategies, namely the soft-clipping estimator and the use of the proximal point algorithm. Overall we see that for most configurations, the proximal point method better optimizes the objective function and provides better test performances, while the soft-clipping estimator performs better than its hard-clipping variant, which may be attributed to the better optimization properties. For semi-synthetic Warfarin and real-worid CoCoA datasets in Figure 20 we also show the improvements in test reward and in training objective of our optimizatioo-drrven strategies. More particularly we demonstrate the
36

---

Published in Transactions on Machine Learning Research (03/2025)
NossyCrccss
Fasswet [aoggeo
NoisyCirtee Dataset
8.7
NoleyCricles Dataset
Proximal Pork Algarthim
25
0.54
Emrmat On
23
23.3
naht
Solt
Better local minima 76%
244
106.0
13
Seahl
Modellig
0.4
Inerr
phcceas-contart
Irea"
phoondeecoralert
8:52
8:40
Classic ciipping
An.sotropcc Hettggram Crox Euounox
Anisotropic ic Detaset
Antsotropic D Dataset
0.3
68
3.0
0.33
Eatmaterr
25
0.55
0.
28
Selt
42.5
25.5
15
Setter local mirima 72%
0:55
1020
Hode Brng
Modding
0.43
-2.0
-15
-30
-05
8.0
35
0/6
0.50
3.35
3.3
515
ET0
015
0.00
6.4
93
8.5
37
03
Losses
opömizer
Classic dipsing
face Devest
Proema Paar. Argormm Dorset warlorn
Detaset warlare
3.4
-32
32
Eei mater SMPS
O
Cllsping N
-25
Mazel
Muze
ce
de
50
-38
LEK550
Tant octiviber
Hard cl pphhs
EaCoA Histogram [QSooRROV
CoCoA
CoCoA Dataset
Frodmaal Point Algertti
11:40
11.42
Cissio
ShIPS
13.95
DR:
21
4.5
20
11.14
Irea"
3326
11.33
L053e5
Base optimizer
Classic cloping
Optimization-driven approachee (NoisyCireles, Anisotropic, Warfarin and CoCoA datasets). Relative improvements in the training objective from using the proximal point method (left), comparison of test rewards for proximal point the simpler gradient-basod method (center), and for soft- vs hard-clipping (right).
Figure 20:
effectiveness of proximal point methods on the Warfarin dataset where most proximal configurations perform better than the base algorithm. Morcover, soft-clipping strategics perform better than its hard-clipping variant on real-world dataset with outliers and noises, which demonstrate the effectiveness of this smooth istimator for real-world setups.
37

---

Published in Transactions on Machine Learning Research (03/2025)
On further optimization perspectives and offline model selection As mentioned in Section 4 our use of the proximal point algorithu differs from approaches that enhance policies to stay close to the logging policies which modify the objective function as in (Schulman et al. 2017) in reinforcement learning. Another avenue for future work would be to investigate distributionnally robust methods that do such modifications of the objective function or add constraints on the distribution being optimized. The policy thereof obtained would thus be closer to the logging policy the CRM context. Morcover, as we showed in Section 6.2 with importance sampling estimates and diagnostics, the offline decision becomes less statistically significant the evaluated policy is far from the logging policy. Investigating how the distributionally robust optimization would yield better CRM solutions with regards to the offline evaluation protocol would make an interesting future direction of research.
G.4 Doubly Robust Estimators
In this section we detail the discussion on doubly robust. estimators and the difficulties that exist to oblain a suitable estimator. In policy based methods for discrete actions, the DR estimator takes the form
(asl (y 7x, a:)) To.i
LTn(T)
E (all aEA
Usually, the DR estimator should only improve on the vanilla IPS estimator thanks the lower variance induced by the outcome model ft. However, in contimtous-action setting with stochastic policies, the second term becomes Ez=Pxx.vxx(1e) (arr,))], which is intractable to optimize in closed form since it involves integrating over actions according to (1-lar). Thus, handling this term requires approximations (as described hereafter), which may overall lead to poorer performance compared to an IPW estimator that sidesteps the need for such a term.
The difficulty for stochastic policies with continuous actions is to derive an estimator of the term (x=-xx,->x(1)) (f(2,)) Unlike stochastic policies with discrete actions which allow use dis- crete summation over the action set, would need here to compute here estimator of the form 1 - Sae (() We note that in the case of deterministic policy learning this direct method term would casily boil down to Di= ((2,, x xx;)) ) and the DR estimator would be built with smoothing strategies for the IPW term in (Kallus & Zhou 2018
In our experiments for stochastic policies with one dimensional actions A C R, we tried to approximate the direct method term > Sae (alzz))(( a) with finite sum of CDFs differences over the m anchor points a1.... Om by computing:
m
ama;
We present a table below of some of the experiments we ran on the synthetic datasels we proposed, along with an evaluation the baselines that exist in the litterature for discrete actions. We see overall that our model improves indeed upon the logging policy, but does not compare to the performances of the scIPS and SNIPS estimators.
Demirer et al. (019)) provide doubly robust (DR) estimator continuous action using semi- parametric model of the policy value function. Their policy learning is performed in two stages (i) estimate doubly robust parameter (D in the semi-parametric model of the value function Ely a, = 2a, = (() and (ii) learn a policy in the empirical Monte Carlo estimate of the policy value by solving
(DR(T)== = " (D,, as rz), (((()))))
min Fell
38

---

Published in Transactions on Machine Learning Research (03/2025)
Noisymoons
Noisycircles
Anisotropic
Logging policy wo
0.5301
0.5301
0.4533
Doubly Robust (discrete)
0.5756 + 0.0022
0.5500
0.0024
0.5593
0.0026
SWITCH (discrete)
0.5786 + 0.0025
0.5520
0.0026
0.5741
0.0024
CAB-DR (discrate)
0.5683 + 0.0023
0.5326
0.0025
0.5361
0.0028
Doubly Robust (ours)
0.6115 + 0.0001
0.6113 + 0.0002
0.5977 + 0.0001
Table 7: Comparison of doubly robust estimators, discrctizod strategics and our model which approximates the direct method term.
The doubly robust estimation is performed with respect to the first parameter learned in (i) for the value function, while follow the CRM setting Swaminathan & Joachims (2015a) and directly derive estimators of the policy value (risk) itself, which would correspond to the phase (ii). Instead, to derive an estimate a DR estimator of such policy values, we tried extending the standard DR approach for discreto actions from
Dudik al. (2011) to continuous actions by using our anchors points, but these worked poorly in practice.
39


Reviews:
[{"review_id": "bIPU8WBDnp", "reviewer": "Reviewer_UPsJ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The manuscript is in a positive direction, and kind of includes all key elements that a work of this sort should have. Some points that I consider worth mentioning:\n\n- The presentation of the problem is decent, even if it could be a bit more clear and straightforward, I could get what the authors wanted to do.\n\n- The review and consideration of other SOTA methods is fine, at least, I do not feel something relevant is missing. However, I have some concerns about the similarities wrt Swaminathan and Joachims  (ICML 2015), as I perceive that it is the key reference work for the manuscript. To me, evaluating the degree of difference and contributions wrt that work is difficult, and at times, the lack of clarity and concise comments on the new contributions makes it a bit harder.\n\n- Technically, I think derivations in the main manuscript are well-developed, as I didn’t find any clear mistakes or errors. The decisions taken on the side of the kernel embedding and the Nystrom approximation make sense to me.\n\n(I will include my points of concern and thoughts on weaknesses in the following section [requested changes])", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper builds ideas on top of the problem of counterfactual learning of stochastic policies from logged bandit feedback. In particular, the manuscript considers the scenario where actions are continuous, which can be considered more or less harder. Two main technical contributions are added to the previous SOTA methods: i) the addition of a joint kernel embedding of context and actions, plus a Nystrom approximation to provide a finite-dimensional framework to the problem, and ii) a differentiable soft-clipping strategy such that optimization in CRM is stable and feasible.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I believe the paper in its current version has a severe problem with clarity. This is certainly my main concern, as I feel that in this way it should not be considered for acceptance, as it would be really difficult for readers to dig into it and its proposed ideas. \n\nFirst, the structure chosen for the thread of explanations is really difficult to follow and to understand what is going on. At times, one has to re-read sections several times to not get lost in the hyper-connected sentences about new equations, terms, parameters and threshold hyperparameters. In that sense, I could even feel that the text loses several times the thread and the goal of what it wants to communicate. What I want to say also, is that the way of dividing the manuscript into titled paragraphs is not helping at all. For example, I can clearly see that the “summary of the CLP policy” on pp.7 is added as a way of clarifying after one realizes that pp.5-6 are not a well-structured flow of continuously changing methods, and ideas, and approximations and techniques. These two pages do really need re-writing, otherwise it’s impossible to get what is going on.\n\nThis way of presenting scientific contributions makes also difficult to understand why some decisions are taken. A good example of parts where this is a bit better indeed is the part of modeling the loss predictor, with the kernel embedding and why the Nystrom approximation with its anchor points is needed. But even here, it should be clearly explained why the joint kernel embeddings make sense (I get it, but from my own thinking, not because it is explicitly mentioned in the paper — and I do think manuscripts like these should be self-contained and make basically things easier for the reader).\n\nLast but no least, I add some comments and details that make me be a bit concerned, and perhaps are good indicators of where new updates can be made:\n\n- After Eq. (4) — I understand that the empirical estimator might have a large variance, but why it is prone/subject to various overfitting phenomena? isn’t this counter-intuitive?\n\n- End of pp.5: what does it mean to avoid the optimization over actions? Could it be added something else to this point?\n\n- End of pp.6:  Anchor points, maybe a little larger discussion on discretization would be nice as it touched one of the main topics of the paper (continuous actions over discrete ones)\n\n- Summary in pp.7: I do believe it is redundant: and basically adding it is a result of the lack of clarity\n\n- Section 4.1 on soft-clipping IPS and why makes it non-differentiable. I think this part is a strength, but it is oddly written in my opinion. Can be improved quite a lot.\n\n- I don’t really know if Eq. (11) is a great contribution or just an added method to make it differentiable in the larger regime. Could authors help here to provide more insights?\n\n- I find it weird that generalization bounds are introduced just after the analysis on optimization in that way —> ok I see that it is pretending to justify its use as a good optimization objective for minimizing the expected risk, but it creates a super large jump in the text.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ClCwUeOj5q", "reviewer": "Reviewer_HdeN", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The modeling approach for continous actions is novel and inspiring for CRM problem.\n2. Concrete theoretical proof for the new proposed estimator.\n3. Well-organization for the paper presenation.\n\nWeaknesses:\n1. The idea of soft-clipping IPS, its proofs, and the evaluation protocol for logged data are kind of standard.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The submission discusses the problem of counterfactual learning of stochastic policies with continuous actions. The main contribution includes (1) a new modeling for counterfactual learning w.r.t. continuous actions, based on a joint kernel embedding of contexts and actions; (2) a novel dffirentiable estimator for counterfactual risk, based on soft-clipping technique on IPS, with generalization bound, and a corresponding optimizer based on proximal point methods; (3) a new released large-scale real-world dataset, for continous actions, and corresponding offline evaluation procedure for model selection in CRM problem.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I do not see much necessary changes and recommend acceptence as it is.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "5o09WOuwR1", "reviewer": "Reviewer_p3aZ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "__Strengths:__\n\nThe main strength of this paper is that the authors have found a practically useful approach for a difficult learning problem, and this is backed up by encouraging results on a wide variety of setups including real-world data that they have curated themselves. I am not at all familiar with the related literature, so I have low confidence in my evaluation of the validity of their experimental protocols, but overall the paper is quite well written with lucid exposition, and I felt that they did a good job of placing their proposed methods (sections 3 through 5) in the proper context given existing work. There is nothing particularly surprising about the learning theoretical results, nor do they capture optimization error in their analysis, but I think the results given definitely contribute to the overall exposition. Overall, their claims appear quite transparent, and look to be sufficiently well-rooted in the empirical evidence provided.\n\n\n__Weaknesses:__\n\nFor better or for worse, this is in my opinion applied machine learning research, and their main contributions are *empirical* in nature. This need not be a weakness to be sure, since they have found a combination of techniques which demonstrably work, which will assuredly be valuable to practitioners trying to tackle (continuous) counterfactual learning problems, but the *insights* obtained here are perhaps somewhat limited. This is just because the positive empirical performance was obtained as the result of aggregating a handful of different techniques (estimated loss-weighted average over \"anchor\" grid, soft-clipped importance weights, proximal point process), making it hard to pin down exactly what matters (in terms of modelling/algorithm design decisions) and when.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "I will start by briefly summarizing my understanding of the counterfactual risk minimization learning problem setting, since I am not well-acquainted with the related literature. Essentially, we have $n$ IID triplets $(x\\_{1},a\\_{1},y\\_{1}),\\\\ldots,(x\\_{n},a\\_{n},y\\_{n})$ where each $x_{i}$ (a feature vector of some kind) denotes some \"context\" for the decision maker to choose an action, which in the case of $x\\_{i}$, is $a\\_{i}$. Under this context and action, a loss $y\\_{i}$ is incurred. We (i.e., the learning algorithm) has access to these triplets, but does *not* know anything about the underlying \"policy\" upon which each action $a\\_{i}$ was selected (a potentially noisy mapping from $x\\_{i}$. To the best of my understanding, the interpretation of this $n$-sized dataset is information about \"what actually happened under some unknown policy,\" and what we want to know is \"*what would happen* if we used a certain policy $\\\\pi \\\\in \\\\Pi$?\" (without running the expensive/dangerous/questionable experiments that would be needed to actually generate new triplets based on a candidate $\\\\pi$). This \"what-if\" nuance aligns with the term \"counter-factual\" used in the paper. The ultimate goal is then to optimize the expected loss to be incurred under a given policy, but again with the critical constraint that we cannot re-run the (action-dependent) loss-generating process; this constraint is the critical difference between the concept of \"losses\" in standard empirical risk minimization, where losses can be re-computed at training time for any choice of policy.\n\nWith my above understanding of the basic problem of interest as context, my understanding of the authors' contribution in this submission is as follows. The main deficiency in the existing literature used as a springboard to motivate this work is a lack of effective methods when the action space (from which the policy samples) is *continuous*. In this direction, they propose the use of distributions (conditional on \"context\" $x$) which are characterized by mean and variance (see equation (6)). From my perspective, their key modelling decision is to fix the variance and set the mean as a function of $x$ by taking a weighted sum of \"anchor points\" (a finite subset of actions) whose weights are determined by loss predictions made by an ancillary \"loss predictor\" $\\\\eta\\_{\\\\beta}(x,a)$ which can be trained based on the available context-action-loss triplets. The authors also touch on this predictor in some detail, considering a kernel-based joint embedding of actions and contexts.\n\nSomewhat orthogonal to the above-mentioned model proposal is the authors' consideration of how to actually select an \"optimal\" policy from their newly designed model. There are two facets to this: the objective function to use, and the optimization scheme to use. Regarding the objective function, the starting point is the classic IPS method (see their equation (4)), but this can have serious volatility issues when the probability in the denominator is close to zero. A natural solution is clipping these importance weights (which leads to bounded losses), but clipping can reduce the efficacy of gradient-based optimizers since the loss is constant beyond the clipping threshold (i.e., zero slope). The authors thus consider a soft clipping approach; their penalty values become unbounded, but gradient information is more meaningful. They complement this proposal with learning theoretical analysis; they give one-sided tail bounds for their empirical objective (yielding upper bounds on the ideal objective; see Prop 4.1), as well as risk bounds for minimizers of their empirical objective function (modified to include an empirical variance term; see Prop 5.1). These results mostly follow in a mechanical way from well-established arguments (especially since they just assume their way out of unbounded losses), but I think makes a nice complement to the exposition of their approach.\n\nFinally, they also have put significant effort into empirical tests (sections 6 and 7). One side of this is a new dataset based on real-world bidding data, plus a protocol for evaluation when using such data. The other side is evaluating the efficacy of their approach using both controlled simulations and the aforementioned real data evaluation procedure. The authors conclude that their proposed combination of model/optimization methods tends to be superior to more naive traditional approaches.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Overall, I don't have any critical requests, just a handful of points I tripped up on. I hope that at least one of the other reviewers is a bit more knowledgeable than I am regarding standard experimental design in the counterfactual learning setting.\n\n- Page 2, \"findings\" list, point 3: the authors state that *\"Real-world counterfactual risk minimization requires a standard offline evaluation protocol,\"* but this is not really a finding, just a statement of opinion. The authors have a proposed protocol in section 6; can they say anything about its validity/transparency/utility relative to other plausible protocols? I think this kind of content is what readers would expect in a \"findings\" list.\n\n- Page 3: a \"logged\" dataset and a \"logging\" policy; are these standard terms? I had no idea what was meant until I finished reading the following page and inferred the overall problem setting.\n\n- Page 4: *\"The expected risk of a policy...\"* This should be expected loss or just risk, since later on the authors say *\"...with small risk.\"*\n\n- Page 4, after equation (2), *\"When using counterfactual estimators for $\\\\widehat{L}$...\"*. As a new reader to this domain, the term \"counterfactual estimator\" was totally impenetrable to me; even now, I'm not really sure how to digest this term. Estimators of the expected loss when we cannot obtain new losses as a function of action? (and only have access to the aforementioned \"logged\" triplets?)\n\n- After equation (4), the first sentence is quite poor. The statement *\"has large variance\"* comes with no qualification, and the statement *\"is subject to various overfitting phenomena\"* is too vague to have much meaning. The following sentence I also found troublesome; what do you mean by *\"negative\"* feedback values? Simply being below zero, or being *far* below zero?\n\n- Section 3.2: the dimension number $d\\_{x}$ for $x$; in my opinion it is bad practice to use one symbol $x$ for effectively two distinct purposes, one as a random variable, and one as a symbol to indicate what the dimension number is for. Simply $d$ would be better if it hasn't been used elsewhere.\n\n- I felt the anchor set used in (7) and the fixed variance in (6) are pretty critical, non-trivial design decisions, but they are done with minimal comment. A bit more discussion might be quite beneficial in my opinion.\n\n- Top of Page 8, *\"...we recall the empirical variance with scIPS that is...\": the meaning of \"scIPS\" is not defined anywhere in the preceding text. As far as I can tell, it just appears as a subscript on the left-hand side of (12).\n\n- Page 8, *\"...compared to a linear dependence for IPS\"*: please provide a clear citation to the relevant literature where such a result can be found.\n\n- Remark 4.1: the term \"costs\" is used, but I assume this refers to losses? It would be best to keep formal terms consistent.\n\n- Section 6: is it \"CoCoA dataset\" or \"CoCoAdataset\" (without a space)? The sub-section heading for 6.1 has a space, but immediately afterward the space is gone.\n\n- Page 11: *\"propensity over-fitting\"* --> *\"propensity __for__ over-fitting\"*.\n\n- Section 8: For future research, the authors mention that they would *\"like to discuss the doubly robust estimator\"*? Is discussing something really future research? The last sentence of the conclusions section feels like a throwaway sentence, and deserves to be a bit more well thought through.", "broader_impact_concerns": "No relevant concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a novel approach to counterfactual learning with continuous actions, addressing a critical challenge in offline policy learning. The key contributions include a joint kernel embedding framework for modeling counterfactual policies, a differentiable soft-clipping estimator to stabilize optimization, and a theoretical analysis establishing generalization bounds. Additionally, the paper introduces a new large-scale real-world dataset, which enhances the empirical validation of the proposed method and provides a valuable resource for future research. The combination of these elements results in a comprehensive approach to counterfactual risk minimization in the continuous action setting.

The motivation for accepting this work stems from its strong technical foundation, thorough empirical evaluation, and practical relevance. The proposed approach effectively tackles an important problem in the field, improving both theoretical understanding and practical applicability. Reviewers found the manuscript to be well-supported by rigorous analysis and experiments, demonstrating clear benefits over existing methods. While initial concerns regarding clarity and organization were raised, the revised version successfully addressed these issues, leading to unanimous agreement among the reviewers that the paper is now in an acceptable state for publication. Given the significance of the contributions and the improvements made in response to feedback, we are pleased to accept your paper for publication.
Decision: Accept (poster)