Leveraging semantic similarity for experimentation with AI-generated treatments
Authors: Lei Shi
David Arbour
Raghavendra Addanki
Ritwik Sinha
Avi Feller
Venue: NeurIPS 2025 poster ()
Keywords: Experimentation
Kernel methods
Causal inference
Embeddings
LLM-generated treatments
Primary area: probabilistic_methods

Abstract: Large Language Models (LLMs) enable a new form of digital experimentation where treatments combine human and model-generated content in increasingly sophisticated ways. The main methodological challenge in this setting is representing these high-dimensional treatments without losing their semantic meaning or rendering analysis intractable. Here we address this problem by focusing on learning low-dimensional representations that capture the underlying structure of such treatments. These representations enable downstream applications such as guiding generative models to produce meaningful treatment variants and facilitating adaptive assignment in online experiments. We propose double kernel representation learning, which models the causal effect through the inner product of kernel-based representations of treatments and user covariates. We develop an alternating-minimization algorithm that learns these representations efficiently from data and provide convergence guarantees under a low-rank factor model. As an application of this framework, we introduce an adaptive design strategy for online experimentation and demonstrate the method's effectiveness through numerical experiments.

Leveraging semantic similarity for experimentation with Al-generated treatments
Lei Shi
David Arbour Adobe Research San Jose, CA 95110 arbour@adobe.c com
Raghavendra Addanki Adobe Research San Jose, CA 95110 com
University of California, Berkeley Berkeley, CA 94720 leishi @berkeley edu
Ritwik Sinha Adobe Research San Jose, CA 95110 risinha@adobe.com com
Avi Feller
University of California, Berkeley Berkeley, CA 94720 afeller@berkeley edu
Abstract
Large Language Models (LLMs) enable a new form of digital experimentation where treatments combine human and model-generated content in increasingly sophisticated ways. The main methodological challenge in this setting repre- senting these high-dimensional treatments without losing their semantic meaning or rendering analysis intractable. Here we address this problem by focusing on learning low-dimensional representations that capture the underlying structure such treatments. These representations enable downstream applications such guiding generative models to produce meaningful treatment variants and fa- cilitating adaptive assignment in online experiments. We propose double kernel representation learning, which models the causal effect through the inner product dernel-based representations of treatments and user covariates. We develop minimization algorithm that learns these representations efficiently from data and provide convergence guarantees under a low-rank factor model. As application of this framework, introduce adaptive design strategy for online experimentation and demonstrate the method's 's effectiveness through numerical experiments.
Introduction
Randomized experiments are cornerstone of measuring the efficacy of content (c.g., messaging,
imagery) in modern industry (e.g.. Sun et al. 2021 Grimmer and Woolley 2014 Leung et al. 2017
and social science (e.g., Gerber et al. 2018 Green al. 2023 Salovey and Hillaam--Piehon 2004
contexts. Historically, generating the content itself has been the core challenge. The advent of Large
Language Models (LLMs)) and other generative models involving text, images, and videos (Gotąb
Andrzejak 2023 Back 2023] Kumar and Kapoor 2024] in recent years has dramatically decreased this design burden. By integrating human creativity with model-generated content, generative models are transforming digital experimentation and streamlining treatment generation.
The ability to generate many content variants for experiments, however, has come with new statistical challenges, especially since the corresponding experiment populations have largely stayed fixed. As result, practitioners often face the choice of either running underpowered experiments leaving many treatment variations unexamined. this paper, we explore an alternative: to improve statistical efficiency by explicitly incorporating semantic similarity from treatment embeddings (numeric representations of the treatments). This raises two key technical challenges: (i) Effectively leveraging
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

semantic information for better estimation and experimental designs, and (ii) Developing framework that appropriately captures the relationship between generated variants and user attributes. Dealing with these challenges can facilitate many downstream applications, such as guiding generative models to generate treatments and assisting adaptive treatment assignment in online experiments.
In this work, we propose kernel-based representation learning approach that addresses these challenges. Our framework integrates both treatment embeddings and user covariate information to estimate the causal effects of LLM-generated variants. Specifically, our core contributions are:
1. A double-kernel representation learning framework which the treatment effect the inner product of wwwdimensional representations of treatment embeddings and user covariates.
2. A computationally efficient alternating minimization-sty: algorithm to learn these unknown representations. This naturally extends to an adaptive algorithm that assigns treatments to users an online manner.
3. Theoretical guarantees, including convergence rates for treatment effect estimation and sublinear regret bounds of an algorithm for the adaptive experimentation setting.
2 Related Work and Background
Our work builds on several key threads.
Causal inference with structured treatments. There is extensive literature on causal inference with text; see Feder et al. (2022) for a comprehensive survey. Prior work has largely focused on the challenging problem of learning latent treatments from images text (Fong and Grimmer 2023 Egami et al. 2022). We sidestep several these issues by taking the generated texts as fixed treatments and only focusing using semantic information to improve efficiency. Many recent works have started exploring textual causal inference with LLMs, such as Imai and Nakamura (2024); Tierney al. (2025). However, they estimation and adaptive experimentation.
touching the problem of personalized treatment effect
High-dimensional and continuous treatments. Our work builds most directly recent results on heterogencous treatment effects with high-dimensional and continuous treatments. Kaddour et al. (2021) extended the R-learnr approach of Nie and Wager (2021) treatments. Liu et al. 2024) studied continuous treatment effect estimation by modeling dependences among treatment and responses using covariate embeddings from generative models. [Singh al.) (2024) proposed estimators based on kernel ridge regression for nonparametric causal functions such dose, heterogeneous, and incremental response curves.
setting with unstructured
More generally, there has been substantial recent work on experimentation with high-dimensional or continuous treatments, especially in the structured bandit problems, such as linear bandits (Rus mevichientong and Tsitsiklis 2010; Hao et al. 2020; Oh et al., 2021 Komiyama and Imaizumi
2024), generalized linear bandits (Filippi et al. 2010 Li ct al. 2017). low-rank bandits (Lu et al.
2021 lun et al., 2019), among others.
Matrix completion for causal inference. There is a substantial literature on decomposing function (e.g., treatment effect) based on matrix factorization, such as AAbernethy et al. (2006); Fithian and Mazumder (2013) 3); Mao et al. (2019); Zhou al. (2012); AAhhey et al. (2021). Most relevant for our work, Din et al. (2022) propose low-rank matrix completion when covariate information is available. Importantly, Jin et al. (2022) only have covariate information available for the treatment-user pair, which differs from our setting in which we have both treatment embeddings and user attributes available.
(ithiaaii
Kernel methods. We build on the expansive causal inference literature that leveraging kernel methods: kernel-based methods that incorporate treatment embeddings for experimental design and analysis offer several benefits. First, kernel-based methods allow for flexible response surface modeling, f Hg,, going beyond simpler parametric formulations such as linear models (Rusmevichientong and T'itssikii 2010 Lu et al., 2010). Second, kernel-based methods provide adaptive design frameworks through tools including kernel bandits (Srinivas al.] 22099] Valko et al. 2013) and Bayes optimization (Frazier] 2018 Martinez- Cantinn 2014, Ignatiadis and Wager 2019 Dimmery al. 2019). Third,
kernel-based methods have well-established statistical guarantees, including prediction/estimation

---

error analysis
Mendelson and Neeman 2010, Wainwright, 2019) and regret analysis (Srinivas et al.
2009
Frazier 2018
Chowdhury and Gopalan 2017] Valko et al. 2013) Vakili al. 2023)
Problem description
3.1 Setup
Notation. We first introduce notation. Throughout, for an integer k, [k] denotes the set (1...., k.. For a matrix r, ||] the Frobenius norm, ||T is the nuclear norm, and [T] the element-wise infinity norm. Pmin (T) and Pmax (T) denote the smallest and largest singular values, respectively.
Causal inference problem. Evaluating many content variants generated by LLMs creates chal- lenges for classical A/B testing. We formalize these challenges under a causal inference framework. For user with covariate x E X, we posit the following structural model for the potential outcome under the treatment E 2:
y(=)= - [ EE X
Here f' is an unknown smooth function, and is mean-zero random noise. 2 and X' can be either finite or continuous spaces, representing the treatment and user covariate space, respectively. The structural model | implicitly encodes SUTVA: (i) interference between units, meaning one units' treatments or outcomes do not affect another unit's potential outcomes; and (ii) no hidden version of treatments, meaning the realized outcome is the potential outcome at the assigned treatment.
For the estimation problem, we are interested in comparing a new treatment 2 with a control variant 20, for user with covariates The conditional average treatment effect is then defined as:
(2, f(z, x)) f(z, x).
When is binary, i.e., E 210. (22,2) is the standard conditional average treatment effect.
Learning 7(2, 2) efficiently from data is crucial for both estimation and experimentation. For traditional A/B testing, the treatment space Z involves a finite number of variants, which can be fully explored by assigning each element in 2 to a sufficient number of users. However, treating variants as separate discrete elements in LLM- are hugely underpowered since is difficult fully explore the treatment space Z due to the large number of variants; (ii) the experiments ignore the semantic information in the variants, which encodes similarity between arms and could deliver important information about the outcome function. The two questions motivate the idea of leveraging treatment embeddings to explore the similarity between treatment arms and pool information across variants to improve statistical efficiency, We will now describe each of these problems in turn.
experiments leads to two issues: (i) experiments
3.2 Warmup: learning with treatment embeddings
To illustrate the use of embeddings, we begin with a simple setup in which the outcome function is homogeneous among users, i.e., [(2, x)) f(z) is only function of the treatments, not the user covariates; we relax this below. In a typical A/B testing framework, 2 is a discrete set, which makes estimation difficult when there are large number of treatments. Existing approaches improve efficiency using shrinkage (Dimmery et al.) 2019 Ignatiadis and Wager] 2019) help alleviate this but still face fundamental limits because of the discrete treatment space. However, in our setting, each generated hypothesis (or treatment) is typically associated with an embedding that encodes important information. the generative setting, we can easily obtain such an embedding by, e.g., using the last layer of a deep neural network that generated the treatment. By representing the treatment space Z as an embedding space, with each treatment lying RP (possibly with large p), instead of a space of unrelated discrete elements, we avoid some of the difficulties associated with high cardinality treatments by implicitly pooling observations across treatments that are close in embedding space.
While conceptually straightforward, the continuous parameterization raises several estimation chal- lenges. We tackle those using kernel methods, which is easily adapted measuring similarity between treatments over the embedding space Z. In particular, we apply kernel ridge regression (based on a kernel XC) to obtain an estimator ff for the outcome response curve. Given sample (24, v.).
3

---

we fit the following penalized regression:
1
f= arg min
Σ (3s
f(2))) 2
AallfllEe
feBk
2n
We then predict the estimated outcome response given a specific treatment E 2 as f(z). See Appendix A for additional details.
3.3 Warmup: Incorporating covariates
Building on this simple case, we can then extend the idea of incorporating treatment embeddings into general setup that also includes user covariates. In an idealized setting, we would estimate the kernels with access to all n X n (expected) potential outcomes {f(2z, I;) [n)) For example, natural approach for this structure is the multi-task Gaussian process framework introduced by |Boillla et al.) (2007), which defines Kg and Kn, as kernel functions that depict the similarity between treatments and individual covariates. practice, however, the fundamental problem causal inference makes directly estimating (=) challenging because most values of this function are missing. In the next section, we propose kernel-based approach that makes progress despite these hurdles.
Learning with treatment embeddings and individual covariates
We present main approach, which we call Double Kernel Representation Learning for treatment effect estimation. In section 4.1 we introduce factorization that captures the estimation problem inner product of representations involving treatment embeddings and user attributes. We describe an alternating minimization style algorithm that recovers these representations. In Section[4.2) provide convergence guarantees of estimator constructed from these representations under fixed basis.
4.1
Double-kernel representation learning
4.1.1 Setup and problem formulation
For binary treatment, Nie and Wager 2021 considered the following reformulated outcome model:
f x) + 1( 21} ((x).
Define the outcome model m(x) E {y 3} and propensity score e(x) {1(z = 21 x}. Nie and Wager [202T) introduced the Robinson Decomposition (Robinson, 1988) and further transformed into the following partial linear model:
y = m(x) + =1 - ((x).
Kaddour et al. (002)) extended this approach to the setting where z is a high-dimensional or continuous treatment by directly assuming f(z, x) in (1) can be factorized as follows:
[(2, g(z) h()),
where g IRP + R' and h R2 + RR are mappings in the real vector space.
Here, we take slightly different perspective: instead of factorizing f(z, 2²), factorize the treatment effect 7(2, æ), similar to Nie and Wager (2021). We posit the following partial linear model for f:
f(2,z) - f(2. x) = g(2) h(x))
which yields:
y = f(20 g())) h2(x))x)+))))))))))))()))))
The above factorization states that the CATE function has a low- oow-dimesiiona representation despite the high-dimensional covariate and treatment inputs. Such a structure is common the literature (see Section 2 and has become foundational for estimation and optimization. Following Wainwright (2019): Rohde and Tsybakov (20TT), we can further relax this be approximately low-dimensional, allowing sparse set of large signals along with many small signals.

---

We can further condition on the user-level covariates by reparametrizing the model (3) and taking the conditional expectation with respect to z. Formally, assume the positivity assumption:
P (= m) > 0, for all 2 E 2, E X.
Let (at) = E (=())) x) Then,
m(x) = f(z,, x) + g(x) ((())){
(4)
Taking the difference between and
yields:
y - mx) (g(=) g(x)) )(x)) +
(5)
In the special case where randomization to treatment 2 is independent of z, g(r) is a constant function; we start with this case and still denote g(2) 9 as (z)). addition, we can directly use the residuals m(x) for estimation by plugging in an estimated outcome model m(x)) For simplicity, we therefore take y to be the residualized outcome here without loss of generality.
4.1.2 Estimation and main result
Building on the above, a natural idea is to solve for and h via the following double kernel regression:
g. h arg
min
Σ ((zz))auaa;)
+ An (lloe l2c + -1/velEe))
(6)
2m
We then prove the following representer theorem (see Appendix and D.I): Theorem 4.1 (Representer theorem). The optimal solutions g" and ht to the optimization problem o lie in IH and Hi respectively.
The representer theorem immediately suggests that the solutions 9 and h have the following form:
((2)
Σ (2,2) 1E|D)
h(z) Σ (x,,)) vE(N)
where the vectors U 1 V E R". For convenience, we capture them using the following matrices:
U = UU..... U E gaxr and V = VV..... V*] E R"xr, each row representing a feature representation for unit. Alternatively,
write U and V in terms of their columns: U [U U.] and V = [Vi.... V.J. With 77, we have a double RKHS model for f: (U",
f(x,t) = Σ
Kg Kz (x, Kg(2, zim) (I,
NE [m] JE(n]
where =U E Raxx is low-rank matrix. We therefore propose the following double RKHS regression:
(Ū, V) arg min
(yk (22, UV (1)) 32 An E KGUF Vi K.V)).
U.V
IEM
Computationally. have (Ü*
8 can be solved by alternatively minimizing over U and V. Concretely, suppose from the t-th iteration. In the (t + 1)-th round, we update the parameters as
follows:
(1+1)
min
Σ (ue Ky(=k, Zim 2in (T) Ka (1n)) T32 An [E(M)
n
UI JuiTK,t KGU,,
(9)
U
2n
k=1
(0+1)
Σ (Bk K.g Z ) (1 TT (ale,. 1
Σ Vi KG, V (10) te(r)
arg min
V
Both 9 and 10 have closed-form solutions when viewed as weighted ridge regression. Moreover, the alternating minimization process guarantees descending (thus convergent) loss function:
_(O ((0) > ((1)
v0))) I(ū"(t)) (((-1)
5

---

Algorithm Double Kernel Learning via Alternating Projection
Input: data (Ie, Res Mc), kemel Kgg, Kr. rank r, penalty level X., maximal iteration 7', accuracy tol Initialize U and ((() as random matrices with gaussian elements.
fort 0 to T do
Update U U(t++) by solving
Update V + ((++1) by solving 10
[(U)))) 11= / |F < tol and || - -(+ /+++////lVV < tol then
Break
end if
end for
Output: Ui0,, V
We summarize the procedure in Algorithm along with some discussions.
Complexity of Algorithm ! Algorithm I involves loop that alternates between solving 9 and (T) thus the computational complexity depends on the solver for (9) and (1O). In our implementation. we solve (9) and (10) by updating the columns of U and V recursively via ridge regression with a general quadratic form penalty. To run these ridge regressions,
compute the inverse of the kernel matrices before the loop. With these holdout inverse matrices, we only need ( to solve each round of the generalized ridge regression. Therefore, in total, the time complexity is
O
+
O
Compute inversion
matrices
loops
ridge regressions compute ridge updates
Scalability to large-scale datasets. To make the algorithm adaptive to large-scale datasets (especially with large N). we can leverage standard techniques to speed up kernel computation. For example, using the Nyström method (Williams and Seeger 2000) and its variations can reduce the inversion complexity to linear in N.
Convergence of the algorithm. The optimization performance of alternative minimization is well studied. Since we are more concerned with the statistical properties (see Theorem 4.2) and exper- imentation performance (see Theorem [5.T] of the proposed method, we refer interested readers to optimization- based discussions in Jain et al. (2013): Chi et al. (2019).
Comparison with a deep-learning bused framework. Deep-learning offers alternative framework for learning CATE from (3) (Kaddour et al. (02T). That approach focuses primarily on prediction, often resulting in cefficult-toiinerpptt representations for the learned treatment and covariates. The corresponding thcoretical results are also quite different: as we show in (.atioom[422bbleer we provide rigorous theoretical guarantees for the estimation accuracy of reconstructing feature representations. By contrast, theoretical results for deep learning barneng-based approaches mainly focus on proving bounds on excess risk as a metric to measure prediction accuracy, but do not provide results on how well the representations themselves are constructed. Finally, deep learning based approaches are computationally expensive relative the low computational cost of kernel-based methods.
Connection with classical approaches. The proposed double kernel learning method has a deep connection with two classical approaches: (i) kernelized probability matrix factorization, which was introduced by Zhou et al. (2012) and serves as the Gaussian process counterpart of our method; and (ii) low-rank matrix factorization, which explores low-rank signals without kernel structure and has been discussed extensively in the literature (Recht et al. 22010) Chi et al. 2019). We explore these connections in more depth in Section B of the Appendix.
4.2 Theoretical analysis for fixed bases
For our theoretical analysis, we follow Kaddour et al.] (202T) and consider a fixed basis setting. Suppose there are dj treatment bases Z = =[²}, which set of p-dimensional vectors, and de user bases X = {a=*), ). which is set of dimensional vectors. Z and X' summarize the information of the treatment to be tested and the pool of candidate users, respectively. Stack the bases into two matrices:
Z = [2 241] E Ipxxd
!..,...,, do E Re
'There are other methods in the literature that may be used for solving the program, such as (stochastic) gradient descent (Zhou et al. 2012) and Quasi-Newton methods (Abernethy al. 2006).

---

Let (ez,ez) Cx) be pair of independent uniform samples in the product set 2 X A' The observation i is associated with one covariate z; ee,, and one treatment 2; Zez Based on our discussion in Section 4.1 under the decomposition model (5). there exists a low rank matrix O such that the observed outcome is
= m(;;) +
24 + = m(x;) + te 2 O' + ct.
Define the new matrix
r TT Rdixds
(11)
whose rank is at most r if Theorem 4.2. Assume that the noise 64 's are i..d and expoeexonnttall, and the true signal matrix r has rank at most r and entries bounded by a". With probability greater than 1 - ci exp(- -dd log d) with d d1 + d2, the estimator T satisfies
We prove the following theorem:
[m(())) m(x))22, Vn
1
dlog
where An S max
V
d1d2
n
Theorem 4.2 suggests that I is consistent in large samples as long as A₂r - O. When the rank fixed, consistency then only requires a vanishing penalization level X, We can then consider each element of An in turn. The left term captures the eπrоr in the outcome regression function, m(xr). For this term to vanish, we need consistent estimator mxxx) of m(x) the sense that (1/vn)
m(xx) [|2 + 0. This is guaranteed by many statistical/machine learning methods m under the assumption that m(x) belongs to an appropriate function class. For example, if m((x) is fit
with LASSO, the loss vanishes at a rate of / ssloo p/ n where p is the dimension of the features and is the sparsity level (Bickel et al., 2008). If ((x) is fit via RKHS regression with a Gaussian kernel, the rate /log n/nn (Wainwright, 2019 Ma et 2023). The right term, v /dlog d/n,, captures the cost of low-rank matrix recovery. For this term to vanish, 72 > dr log d, where dr captures the intrinsic dimension of a low-rank matrix (Candes and Plan 201T). This rate has been verified to be minimax-opting Wainwright, 2012 Klopp 2014]. Finally, following Wainwright 2011); Rohde and Tsybakov (2011). we note that it is possible to relax this setup to be approximately low-rank, in the sense that the true signal can admit a of large signals along with many small signals. We relegate comprehensive analysis under the approximate low-rank regime to Appendix D
for matrix completion tasks (Rohde and Tsybakov [20 Negahban and
5 Online Experimentation: An Adaptive Strategy
In this section, we provide adaptive algorithm for assigning treatments to individuals they become available an online manner. We use Theorem 4.2 to justify the performance an explore- then-commit strategy. Consider bandit with T rounds, which we divide into two stages. In the first stage, we use T. rounds to explore the best arm for each user by drawing some samples and learning the outcome response matrix T. In the second stage, we keep pulling from the learned best arm for each selected user. Algorithm summarizes this approach.
We prove that the explore- poplore-then-commii algorithm will lead to sublinear regret. To see this, establish the following theorem:
Theorem 5.1. Assume the conditions in Theorem 4.2 Also, assume that m(z) lies in the RKHS generated by a Gaussian kernel. The explore-then-comme algorithm has the following sublinear regret bound with probability greater than 1 - ci exp(-c,d d): Regret < CT2/3 d dog /3/311/3 where d = d1 + d2.
We can see that the simple Explore-ther Chmm-C strategy suffices to guarantee regret with rate O(T2/3), which sublinear. We anticipate that we can use ideas from the literature on low-rank bandits, such as Jun et al. (2019) and Lu et al. (2021), to sharpen this regret bound in terms of the horizon T', though this is outside the scope of the current paper.
6 Experimental Evaluation
In this section, we conduct semi-synthetic experiments based on three open-source datasets. (i) The Upworthy Research Archive (Matias et al. 2021) is open dataset of thousands of A/B tests of
7

---

Algorithm 2 Explore-then-commit
Input:
Exploration rounds T. and exploitation rounds Tc, total rounds T, treatment bases 2, covariates 2 # Data collection for exploration:
for t= 1
Randomly sample cs.t with replacement and obtain a treatment 2; = c Randomly sample a c with replacement and obtain a target user with covariate 2; :;
Assign the user to treatment ž4 and observe outcome response yr:
end for
# Learn outcome responses:
Learn conditional mean (x)) with machine learning algorithms;
Learn outcome response matrix I by solving (
Commit to the best personalizen ized treatment:
for = + to 'I' do
Randomly sample ex.t with replacement from X:;
Assign the user to treatment 2 which maximizes the estimated outcome response vector Tez. Collect outcome response y::
end for
Output: outcome response matrix Π, data (2t; Tt, V-))
headlines conducted by Upworthy from January 2013 to April 2015, We use the Exploratory Dataset in The Upworthy Research Archive, which contains headlines and metrics (clicks, impressions, click-through rates, or CTR) from thousands of experiments. (ii) The MIND dataset (Wu et al. 2020) is benchmark dataset containing traffic from Microsoft for click rate prediction and news recommendation. (iii) The ASOS E-Commerce Datase is an open-source A/B testing dataset from ASOS in the fashion industry. Due space constraints, the Upworthy experiment and MIND dataset, and relegate the remaining results to Section B in the Appendix. The code for the algorithm and replication of the numerical experiments be found here: stttp:///githn
present subset of results regarding
6.1 The Upworthy experiment
Upworthy experimental setup. We choose | 50 headlines as candidate treatments a hypothetical experiment. We use the sentence transformer MiniLM (Wang et al., 2020) to encode the sampled headlines into sentence embeddings of dimension p = 384. Since user-level covariates are not available in the Upworthy Dataset, simulate n 500 Gaussian vectors of dimension = 200 as baseline covariates. The outcome of interest is the potential revenue that each user can contribute. When user with covariate views the headline 2, model the average (centered)
potential revenue generated by this particular user as f(2 = = O²x p"r + where O* E _pxx is matrix with varying ranks and is additive noise.
Evaluation the estimation error. We use Monte Carlo simulations to evaluate the train and test enror of the proposed method, double kernel representation learning (DKRL), and compare with two baseline methods: (i) Structured Intervention Network (SIN) by Kaddour et al. (202T), deep-learning based framework based on generalized Robinson decomposition; and (ii) RKHS regression with product kernel K. O Kc₁ (ProdKernel). We split the synthetic dataset into training set and test set, fit different methods on the training data, and evaluate the fitted results on the test dataset; we repeat this procedure across multiple ranks.
From Table we can see that in these cases, DKRL achieves the best training and testing emror. Neural network-based approaches are likely inferior here because they may not learn the low-rank representation; moreover, it is quite challenging to construct the relevant propensity functions for high-dimensional treatments (although this can be avoided in experimental settings). The low computational cost (measured on a Macbook Pro with M2 Max chip) of kernel-based methods also enables more rapid iteration for large-scale A/B testing.
2The data
available
www.ooooeeeetttteao eataset

---

Table 1: Upworthy Metrics by Rank and Method (Mean (Std), rounded to four decimals)
Rank
Method
Train Error
Test Error
Time (sec)
0.0105 (0.0006)
0.0104 (0.0017)
8.4686 (0.6584)
2
DKAL
0.0019 (0.0004)
0.0036 (0.0010)
0.1907 (0.0339)
ProdKernel
0.0043 (0.0002)
0.0080 (0.0013)
0.0014 (0.0002)
SIN
0.0113 (0.0006)
0.0113 (0,0016)
7.8103 (0.7073)
3
DKRL
0.0022 (0.0002)
0.0048 (0.0013)
0.5062 (0.1448)
ProdKernel
0.0048 (0.0002)
0.0090 (0.0013)
0.0014 (0.0002)
SIN
0.0143 (0.0006)
0.0144 (0.0018)
8.5115 (0.5318)
DKRL
0.0025 (0.0001)
0.0078 (0.0012)
1.6933 (0.3752)
ProdKermel
0.0062 (0.0002)
0.0115 (0.0014)
0.0015 (0.0003)
SIN
0.0171 (0.0007)
0.0172 (0.0021)
7.8928 (0.3724)
7
DKAL
0.0025 (0.0001)
0.0108 (0.0015)
3.6143 (0.5942)
ProdKernel
0.0072 (0.0002)
0.0136 (0.0017)
0.0015 (0.0005)
Dimension reduction and interpretation. Double kernel representation learning also enables low- dimensional summaries of high-dimensional features. Figure ([a) presents the kernel representation for the treatment learned from the data. It shows that the components have semantic interpretation (based on an evaluation by GPT 4o). More concretely,
first learned feature captures the scale of sharp contrasts with narrative surprises; the second captures the extent of a conffict-driver tone. Hence, we can view the values as semantic scores that can directly drive the average reward of the headlines and improve overall interpretability. In Table of the Appendix, show several example headlines with extreme semantic scores, showcasing how the features capture semantic meaning.
Peplane Ebarccsed truttentt
Conflict-drice driiee-divnnn Tome
0:35
Cumalatius 20
10.35
Sharp Contrasts With Narrative Surprises
-325 -338 c.05 305 1.20
Rounds
(a)
(b)
Figure 1: Results on the Upworthy experiment. Panel (a) shows kernel representation of treatments the semi-synthetic Upworthy Dataset. Panel (b) shows the cumulative regret the evaluation of the bandit algorithm. The shaded area is the 15%-85% quantile range of the regret over simulations.
Evaluation of the bandit algorithm. We also evaluate the Explore-ther Explore-then-commit commit algorithm for online experimentation (Algorithm (2). We add explore-thens then-commit with only treatment representa- tions as additional baseline online experimentation method for comparison. Figure ( presents cumulative regret of Algorithm [ As demonstrated by theoretical results in Theorem 5. Algorithm
2 produces sublinear regret in the long run; ignoring user-level covariates leads to failed exploration and thus linear regret. This highlights the importance of incorporating treatment-user interactions into both the outcome modeling and adaptive design strategy.

---

6.2 MIND dataset
Finally, we present results on the MIND dataset, an observational dataset with news recommended to users with unknown probabilities. We consider a semi-syythert setup: suppose our goal is to train this recommendation system from scratch by adaptively collecting recommendation-clie -cciat data to improve recommendation quality. We take æ to be user features such as news category preference and historical news click embeddings, and z to be the embeddings for a set of candidate news to be recommended to users. All embeddings are taken from the knowledge graph embeddings that were originally included in the dataset. We consider synthetic outcome model for the click thrkuhhh rate: y
+ where O* is a matrix that encodes the interaction mechanism between users and treatments; we can interpret this as a match between user preferences and news information. We consider θ UAV with A, = Diag (i-9), and vary q to ensure that the interaction model varies from low rank (high eigenvalue decay rates) to high rank (low eigenvalue decay rates) setting. We compare our method with four baselines: LASSO, XG-boost, feed-forward neural network, and kernel regression. We finally combine the methods with different estimation strategies; "Z" only incorporates treatment information, "X" only includes covariate information, and "ZX" includes a concatenated (Z, X) vector.
Table in the Appendix shows test RMSE. Overall, the performance of all methods is better with low-rank signals (larger q) than high-rank signals. However, DKRL exploits this low-rank structure most effectively, with the strongest performance. Even with a high-rank interaction matrix (e.g., a full-rank matrix), DKRL effectively exploits the similarity structure in the embeddings, improving accuracy and achieving comparable performance to XG Boost with combined features.
Conclusion
Summaary. As content creation at scale becomes more readily available, developing new causal inference methods becomes increasingly important. this work, we explore the design and analysis of (denAl-powerte digital experimentation. We propose to use GenAl for efficient and novel treatment generation and to learn the corresponding heterogeneous treatment effect function using double- kernel representation learning framework. We then propose an adaptive data collection algorithm for online experimentation. Finally, we provide theoretical guarantees and multiple numerical demonstrations.
Limitations and future directions. Several limitations remain. First, it important to consider more complex experimentation settings that are important in practice, such as delayed responses (Shi et al. 2023 2024b) or interference between participants (Li and Wager] (002).. Second, we established theoretical guarantees for the double-kernel representation learning framework in a fixed-basis setting. would be interesting to generalize this theory to nnfnite-dimensionall bases. Third, GenAl --ooweed experimentation raises key questions on issues like privacy and fairness, which are important to address independent of this current research. Fourth, it is natural to use other methods such as language model finetuning (Hu et al., (022)) contrast learning (Zha et al. 2023 to learn improved embeddings and to generally improve the pipeline.
Acknowledgments and Disclosure of Funding
We thank the AC and four reviewers for constructive comments and feedback.
References
Abernethy, J., Bach, F., Evgeniou, T., and Vert, J.PP. (2006). Low-rank matrix factorization with attributes. arXiv preprint cx/0611122
Athey, S., Bayati, M.. Doudchenko, N., Imbens, G., and Khosravi, K. (2021). Matrix completion methods for causal panel data models. Journal of the American Statistical Association, 1166536):1716-11336
Back, T. H. (2023). Digital advertising in the age of generative ai. Journal Current Issues Research in Advertising. 44(3):249-22
Bickel, P., Ritov, Y., and Tsybakov, A. (2008). Simultaneous analysis of lasso and dantzig selector. The Annals of Statistics, 37.
10

---

Bonilla, E. V., Chai, K., and Williams, C. (2007). Multi-task gaussian process prediction. Advances in neural information processing systems, 20.
Candes, E. J. and Plan, Y. (2011). Tight oracle inequalities for low-rank matrix recovery from minimal number of noisy random measurements. IEEE Transactions on Information Theory, 57(4):2342 2359.
Chemozhukov, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C., Newey, W., and Robins, (2018). Double/debiased machine learning for treatment and structural parameters.
Chi, Y., Lu, Y. M., and Chen, Y. (2019). Nonconvex optimization meets low-rank matrix factorization: An overview. IEEE Transactions on Signal Processing, 67(20):5239-5269.
Chowdhury, S. R. and Gopalan, A. (2017). On kernelized multi-armed bandits. In International Conference on Machine Learning, pages 844-853. PMLR.
Cui, X., Shi, L., Zhong, W., and Zou, C. (2023). Robust high-dimensional low-rank matrix estimation: Optimal rate and data-adaptive tuning. Journal of Machine Learning Research, 24(350):1-57.
Dimmery, D., Bakshy, E., and Sekhon, J. (2019). Shrinkage estimators in online experiments. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery data mining, pages 2914-2922.
Egami, N., Fong. C. J., Grimmer, Roberts, E., and Stewart, B. M. (2022). How to make causal inferences using texts. Science Advances, 8(42):eabg2652.
Esser, P. Fleissner, M., and Ghoshdastidar, D. (2024). Non-parametric representation learning with kernels. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 11910-11918.
Feder, A., Keith, K. A., Manzoor, E., Pryzant, R., Sridhar, D., Wood-Doughty. Z., Eisenstein, J., Grimmer, J., Reichart, R., Roberts, M. E., et al. (2022). Causal inference in natural language processing: Estimation, prediction, interpretation and heyond. Transactions the Association for Computational Linguistics, 10:1 133- 1158.
Filippi, S., Cappe, O.. Garivier, A., and Szepesvári, C. (2010). Parametric bandits: The generalized linear case.
Advances neural information processing systems, 23.
Fithian, W. and Mazumder, R. (2013). Flexible low-rank statistical modeling with side information. arXiv preprint arXiv: 108.44081111
Fong, C. and Grimmer, J. (2023), Causal inference with latent treatments. American Journal of Political Science, (7(2))374-389.
Frazier, P. I. (2018). A tutorial on bayesian optimization. arXiv preprint 107
Gerber, S., Huber, G. A., Fang, A. H., and Reardon, C. E. (2018). The comparative effectiveness on turnout of positively versus negatively framed descriptive norms in mobilization campaigns. American Politics Research. 4666):996-1011
Golab-Andrzejak, E. (2023). The impact generative ai and chatgpt on creating digital advertising campaigns. Cybernetics and Systems, pages 1-15.
Green, J., Druckman, J. N., Baum, M. A., Lazer, D., Ognyanova, K., Simonson, M. D., Lin, J., Santillana, M., and Perlis, R. H. (2023). Using general messages to persuade on politicized scientific issue. British Journal of Political Science, (3(2))9987706
Grimmer, M. and Woolley, M. (2014). Green marketing messages and consumers' purchase intentions: Promoting personal versus environmental benefits. Journal of Marketing Communications, 20(4):231-250.
Hao, B., Lattimore, T., and Wang, M. (2020). High-dimensional sparse linear bandits. Advances in Neural Information Processing Systems, 13:10753-10763.
Hu, J., Shen, Y., Wallis, P., Allen-Zhu, Z.,, Li, Y., Wang. S., Wang, L.,, Chen, W., et al. (2022). Lora: Low-rank adaptation of large language models. ICLR, 1(2):3.
Ignatiadis, N. and Wager, S. (2019). Covariate-powered empirical bayes estimation. Advances in Neural Information Processing Systems, 32.
Imai, K. and Nakamura, K. (2024). Causal representation learning with generative artificial intelligence: Application to texts treatments, arXiv preprint arXiv:2410.00903.
11

---

Jain, P., Netrapalli, P., and Sanghavi, S. (2013). Low-rank matrix completion using alternating minimization. In Proceedings of the forty-fifth annual ACM symposium on Theory computing, pages 663-674.
Jin, H., Ma, Y., and Jiang, F. (2022). Matrix completion with covariate information and informative missingness. Journal of Machine Learning Research, 23(180):1-63
Joachims, T. (1998). Text categorization with support vector machines: Learning with many relevant features. In European conference machine learning, pages 137-142. Springer.
Jun, K.-S., Willett, R., Wright, S., and Nowak, R. (2019). Bilinear bandits with low-rank structure. In International Conference on Machine Learning, pages 3163-3172. PMLR.
Kaddour, J., Zhu, Y.. Liu, Q. Kusner, M. J., and Silva, R. (2021). Causal effect inference for structured treatments. In Ranzato, M., Beygelzimer. A., Dauphin, Y., Liang. P. and Vaughan, J. W., editors, Advances in Neural Information Processing Systems, volume 34, pages
Curran Associates, Inc.
Klopp, O. (2014). Noisy low-rank matrix complction with general sampling distribution. Bermoulli, 30:222 303.
Komiyama, J. and Imaizumi, M. (2024). High-dimensional contextual bandit problem without sparsity. Advances in Neural Information Processing Systems, 36.
Kumar, M. and Kapoor, A. (2024). Generative ai and personalized video advertisements. Available at SSRN 4614118.
Leung, X. Y., Bai, B., and Erdem, M. (2017). Hotel social media marketing: a study on message strategy and its effectiveness. Journal of Hospitality and Tourism Technology, 8(2):239-25 255.
Li, L., Lu, Y., and Zhou, D. (2017). Provably optimal algorithms for generalized linear contextual bandits. In International Conference on Machine Learning, pages 2071-2080. PMLR.
Li, and Wager, S. (2022). Network interference in micro-randomized trials. arXiv preprint arXiv:2202.05356.
Liu, Q., Chen, Z.., and Wong, W. H. (2024). An encoding generative modeling approach dimension reduction covariate adjustment causal inference with observational studies. Proceedings of the National Academy of Sciences, 121(23)12222376121.
Lu, T., Pál, D., and Pál, M. (2010). Contextual multi-armed bandits. In Proceedings of the Thirteenth international conference on Artificial Intelligence and Statistics, pages 485-492. JMLR Workshop and Conference Proceedings.
Lu, X., Shi, I.., Liu, H., and Ding, P. (2025). Conditional cross-fitting for unbiased machine-leaming-asiise covariate adjustment in randomized experiments. arXiv preprint arXiv:2508. 15664.
Lu, Y., Meisami, A., and Tewari, A. (2021). Low-rank generalized linear bandit problems. In International Conference on Artificial Intelligence and Statistics, pages 460 468.. PMLR.
Ma, C., Pathak, R., and Wainwright, M. J. (2023). Optimally tackling covariate shift in rkhs-basch nonparametric regression. The Annals of Statistics, 51(2):3387761
Mao, X., Chen, S. X., and Wong, R. K. (2019). Matrix completion with covariate information. Journal of the American Statistical Association, 11 198225):19--21.
Martinez-Cantin, R. (2014). Bayesopt: bayesian optimization library for nonlinear optimization, experimental design and bandits. J. Mach. Learn. Res., 15(1)3733-3739.
Matias, J. N., Munger, K., Le Quere, M. A., and Ebersole, C. (2021). The upworthy research archive, time series of 32,487 experiments in us media. Scientific Data, 8(1):195.
Mendelson, S. and Neeman, J. (2010). Regularization in kernel learning. The Annals of Statistics, 38(1))526- 5665.
Negahban, S. and Wainwright, M. J. (2012). Restricted strong convexity and weighted matrix completion: Optimal bounds with noise. The Journal of Machine Learning Research, 13(1):1665-1697.
Nie, X. and Wager, S. (2021). Quasi-oracle estimation of heterogeneous treatment effects. 108(2):99--119
Biometrika.
Oh. M. Iyengar, G., and Zeevi, A. (2021). Machine Learning, pages 8271-82818 8280. PMLR.
lasso bandit. In International Conference on
12

---

Recht, B., Fazel, M., and Parrilo, P. A. (2010). Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization. SIAM review, 52(3)477-501.
the Econometric Robinson, P. M. (1988). Root-n-consiseett semiparametric regression. Econometrica: Journal the
Society, pages 931-954.
Rohde, A. and Tsybakov, B. (2011), Estimation of high-dimensional low-rank matrices. Annals of Statistics. 39(2):887 930.
Rusmevichientong, and Tsitsiklis, J. N. (2010). Linearly parameterized bandits. Mathematics of Operations Research, 35(2):395 411.
Salovey, P. and Silliams-Pichote P. (2004). Field experiments in social psychology: Message framing and the promotion of health protective behaviors, American Behavioral Scientist, 47(5):488-55) 505.
Schölkopf, B., Smola, A., and Mäller, K.-R. (1997). Kernel principal component analysis. In International conference on artificial neural networks, pages 583-588, Springer.
Shi, L.., Wang, G., and Zou, C. (2024a). Low-rank matrix estimation in the presence of change-points. Journal of Machine Learning Research, 25(220):1-71.
Shi, L.., Wang, J., and Wu, T. (2023). Statistical inference multi-armed bandits with delayed feedback. In International Conference on Machine Learning, pages 31328-31352. PMLR.
Shi, L., Wei, W., and Wang. J. (2024b). Using surrogates in covariate-adjusted iisponse-adaptive randomiza- tion experiments with delayed outcomes. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.
Shi, L. and Zou, C. (2020). Noisy low-rank matrix completion under general bases. Stat, 91 (1)xe333
Singh, R., Xu, I.., and Gretton, (2024). Kernel methods for causal functions: dose, heterogeneous and incremental response curves. Biometrika, 111(2):997-516.
Srinivas, N., Krause, A., Kakade, S. M., and Seeger, M. (2009). Gaussian process optimization in the bandit setting: No regret and experimental design. arXiv preprint arXiv:0912.3995.
Sun, T., Viswanathan, S., and Zheleva, E. (2021). Creating social contagion through firm-mediated message design: Evidence from randomized field experiment. Management Science, 67(2):808 8277
Tierney, G., Katta, S., Bail, C., Hillygus, S., Volfovsky, (2025). A design-based solution for causal inference with text: Can a language model be too large? arXiv preprint arXiv:2510.0875.
Vakili, S., Ahmed, D.,. Bernacchia, A., and Pike-Burke, C. (2023). Delayed feedback in kernel bandits. In International Conference on Machine Learning pages 34779-34792. PMLR.
Valko, M., Korda, N., Munos, R., Flaounas, I. and Cristianini, N. (2013). Finite-time analysis of kernelised contextual bandits. arXiv preprint arXiv: arXiv:1309 1309.6869.
Wainwright, M. J. (2019). High-dimensional statistics: A non-asymptotic viewpoint, volume 48. Cambridge university press.
Wang, W., Wei, F., Dong, L., Bao, H., Yang, N., and Zhou, (2020). Minilm: Deep aelf-attention distillation for
task agmostic compression of pre-trained transformers. Advances Neural Information Processing Systems, 33:5776-5788.
Williams, C. and Sceger, M. (2000). Using the nyström method to speed up kernel machines. Advances neural information processing systems, 13.
Williams, C. K. and Rasmussen, C. E. (2006). Gaussian processes for machine learning, volume 2. MIT press Cambridge. MA.
Wu, F., Qiao, Y., Chen, J.-H., Wu, C., Qi, T., Lian, J., Liu, D., Xie, X., Gao, J., Wu, W., et al. (2020). Mind: A sarge-scale dataset for news recommendation. In Proceedings of the 58th annual meeting of the association for computational linguistics, pages 3597-3606.
Zha, K., Cao, P., Son, J., Yang. Y., and Katabi, D. (2023). Rank-n monk-n-coonrast learning continuous representations for regression. Advances in Neural Information Processing Systems, 36: 77882-17900
Zhou, T., Shan, H., Banerjee, A., and Sapiro, G. (2012). Kemelizedd probabilistic matrix factorization: Exploiting graphs and side information. In Proceedings of the 2012 SIAM international Conference on Data mining, pages 403 414. SIAM.
13

---

A Kernel Methods
Kemel-based methods have been widely used in related practices such as representation learning (Esser et al. 2024), dimension reduction (Schölkopf al. 1997,, natural language processing Joachims ( (998), among others. We do brief overview of RKHS methods. Consider positíve definite kernel K 2 x - R supported on a compact d-dimensional set c RR. Hilbert space Hiy of functions on 2 equipped with an inner product (-, "H is called a reproducing kernel Hilbert space (RKHS) with reproducing kernel K the following properties are satisfied:
for all E 2, (C((,2)) IHK:
2. for all Z and f E Hx. (f K((,,))E f(z) (reproducing property).
A natural idea for estimation and policy learning is to incorporate embedding information with RKHS methods. Given sample (4, Wi). we can fit the following penalized nonlinear least squares:
-= arg min
- -vn (zz)) >n.1//lfRe
2n felly
Then we can predict the estimated outcome response given specific treatment E 2 as f(=). By the representer theorem, this leads program based on the data:
1 2n
= arg min
Σ EEe Σ [=1
(2, + Taa
which gives the interpolator
f(:) Σ (,) 1=1
Another viewpoint the RKHSS-asse method that we can put a distribution on the functions and fit Gaussian process model (Williams and Rasmussen 2006):
N(, f GP.
(12)
Then the interpolator equivalent posterior prediction from 12
In particular, if the kernel K. is linear inner product of finite dimensional features Φ(2), the above program is also equivalent ridge regression:
-(=)=((=]T3
where
B
Σ [m ((2))
T32 + 1m118
2n 1=1
Such connection has been pointed out by previous literature, egg. Valko et 2013
This is also equivalent to a Bayesian linear model:
(8, where 3~ ~~0,00,0f))
B Connection with classical approaches
For intuition now draw connections between the proposed method to two classical approaches: kernelized probability matrix factorization, which introduced by Zhou et al. (2012) and serves as the Gaussian process counterpart of our method, and low-rank matrix factorization algorithms.
14

---

Connection with kernelized probability matrix factorization. The proposed method is closely related kernelized probability matrix factorization (Zhou al. 2012). To see this, we revisit the program ( but instead of using the parametrization U and V consider the following transformation:
R K.U g"xr, S K g"xr.
Here, R and are the intrinsic representations of the treatment and covariate-level information, respectively. Then (8) can be written
R 5) arg min
Σ
Cy yk
R
An
R
K R + S K k
Kg
R.S 2n
We can instead follow the kemnliized probabilistic factorization in Zhou et al. (2012). following guttereeng process:
Under the
R ~ Kcg), ~ GP(O, [r],
and
yk ~ N S o²),
Zhou et al. (2012) showed that the oog-posterior over the latent features R and S is given by
1
log (R, y,o²)
(yke R
R Kg R + S kr $i
20
2
n log 0² (log([(g) + (K)) + C,
(13)
where C' is some universal constant that does not depend on R and S. Maximizing the log log-posterior equivalent to minimizing
13
L(R,
1
Σ
yk
R
5
Σ
R g
R S 1.'K''s (.)ss).
20
which is equivalent to
8
Connection with low-rank matrix learning with finite-dimensional features. given by the inner product of some feature mapping 6 E RP:
Consider the kernel
K,(2, 2/)
with feature matrix & [ó(2)). dzZ))l 1R Then the kernel matrix has the expression:
Kg ΦΦ
We prove the following reformulation results: Proposition B.1. Suppose the feature matrix has full (column or row) rank. Then
i V arg min U.V
2n tu 6(zx) (Ex)) "(xx)) + An. ||V**)))) (14)
Now following Proposition B.I penalization:
we have the following equivalent formulation based on nuclear-nom norm
Proposition B.2. The program 14 is equivalent to the following low-rank learning program:
θ arg min
Σ (un x(2x) T 2A, ||]|.
(15)
O
2n
This equivalence connects the matrix factorization approach with nuclear norm penalization for low-rank matrix estimation. The key insight is due to the variational expression of the nuclear norm (Recht et al. 2010):
||||-
min
(IU + [
U v.u
15

---

Additional results
C.1 Bounds on
The rate in Theorem 4.2 also implies rate for O when the number of tested variants and candidate users is large in the sense that d1 > and d2 > q. To see this, we can express O with I", as defined in
o = (ZZ")
x (XX)--
This motivates an estimator:
θ (ZZ zīx (Xx
(16)
Then we have the following bound:
} 0' Spmax ( (ZZ )- |maxx { X |/i |
Smmin ( (ZZ") /d1]} Pmin (XX /d2)] A2,r..
Therefore, as long the minimal eigenvalue of ZZ and X. T not degenerate, the estimator 16 also consistent.
Technical proofs
D.1 Proof of Theorem 4.1
Proof of Theorem 4.7 The RKHS space Hy and H2 has the following decomposition:
H1 H + Hy, H4, H e H...
where H and H are the orthogonal projection into the sample-generated function subspace. For any g and h. we have the decomposition
g = g=g² +g h h= h htt
Then for any g(,)) (z,) + (z(z) (z.), h(x;) h (x,,) ho t=,))) h -15 (x:). Meanwhile, due to the projection, we have reduction in the RKHS norm:
IR,, I l
|| MAll
Therefore, by replacing g with g and also h with h the objective function never increases. Hence, the optimal solution must lie in H and H
D.2 Proof of Proposition B.1
Proof of Proposition B.I Case 1. When n and the feature matrix & R"Xp has full column rank, we can conclude that the mapping + T is surjective mapping from Rexr to gx Therefore, (8) is equivalent solving the feature-based optimization program:
Σ (un ((=)) T (xx))² + Xn
min U. 2n
Case 2. When p and the feature matrix +b RRxp has full row rank, start from the program 14). For any U*, there exists U, such that
cΦ U 4U"
as one can take U = - Therefore, we can always reparametrize 14 as 8 J
16

---

D.3
Proof of Proposition B.2
Proof of Proposition B.2
The nuclear norm
|Ø||= of a matrix has the following variational represen-
tation:
||||.
min UV 2
(1U ||V [
Using this result, we can deduce the equivalence between the program and 15
1
min P
{yk
((z)) [x)x)))
2A, @
2n
k=1
(uk 6(2x) T 0xb()))) + 2An V...v.
min 1 (1| 11 + V 12:))
min
2n
k=1
= -min
min
Uk - UU [xxx)) U + |V
@
U U.VV.UV.TT 2n
k=1
min
1
Σ
n
(yk ((=))(=))))))) U. (.))) +
U V 2n
k=1
D.4 Statement and proof of a more general version of Theorem 4.2
In this section, prove
general version of Theorem 4.2 which extends the results to approximate low-rank setting. More concretely, let (T) be the k-th largest singular value of matrix
T. We define the following notion of by-ball in the matrix space:
Definition D.1 (ly ball). The Eg ball with radius Rg is defined the following set:
min(d1,d23
(Rg) {
Rdidds
Σ
PT S Rg for q E [0,
When g 0, this definition becomes the exact low-rank condition. When 0, the definition allows the existence of many small positive singular values, decaying at a proper rate (Negahban and Wainwright, 2012, Rohde and Tsybakov 2011 Shi et al. 2024a Shi and Zou| 2020 Cui et al. 2023).
Theorem D.2. Assume that the noise e, 's are i.i.d. and sub-exponential, and the true signal matrix m belongs to the lg ball B,G (Ra) and entry bounded by With probability greater than
1- oi d exp(-ccpp log d), the estimator I satisfies
1 Vn
m(x))l2 v dlog d
i 1 < CR where A, x max
[m(()))
d2
Proof of Theorem 4.2 Bd. dd2 + Raxl.
For the convenience of analysis, introduce the sampling operator G
G(T) = Te.. Texm))
C is linear operator. The adjoint operator, G* gax1 - [d xdx is defined
(E) Σ
T
i=1
The operator satisfies
(G(T), = = (T (T))
17

---

Step 1. Deriving a basic inequality. By the minimization program, for the estimated f, we have
- G Am AT.
2n m(x - + Am |
2n
This gives
[G(T r
2n
1
(i, G(Π -r*) Am | |n.
"
1
(m(x) - -m(x), - r'))
+ (c), f r-r + An | - Am ||.
1 [m(()) - (mm)))m(())]a]]]] - r')ll2 + - (s)lool f-
r- -
+ An| ||. - >m||f/l- (17)
"
Step 2. A cone inequality for the difference. Now we consider the projection of the difference - I onto subspace that will lead to a decomposition into low-rank part and remainder that is relatively small in scale.
UAV where U RR xdi and V E Rd2xd2 are orthonormal matrices and A is the singular value matrix with descending diagonals. For any matrix A Xd2, if we write
Let the singular value decomposition (SVD) of I" be r"
Un Q2
U UVV =
with Qn E Rexr and 022 E (d: ( r). We can define projection
Qu f211
12
Opxr IT (T) 0(d -r)xr
O xx (d2
ITA)
O ((d--r)((d--))))
Dzz
With this decomposition, we always have rank A)) < 2r. Now we choose an effective rank by trimming the matrix using threshold, T, and will discuss how pick T later.
Since I" E (B,g)) we have
r79 Pa(A) & Σ Pk(A)" S Rg: r RgT
(18)
N=
1
k=
Meanwhile, we also have
m (A)I- Σ
m
pk (A)
(a(A) ST Σ
R
(19)
k=
k=r+1
Using the triangle inequality, we have
| || f r+
|| π - r-) +T Tl
+ ) (I- -f ( (T-(()))
2
|| + | r')] -
r')]| (r")))))l),
which further gives
[T ||], [T[ r') - r (T
By choosing An that satisfies
2 16 )
X, 2 max
[m((x) - mz))ll2,
18

---

together with
we have
An 2
|| rr')
| - (f -r')l+-
2n
An
3
|| -
| r+ + 2\n |I-
(20)
For the term 2n the proof of their Corollary 1:
(c) Negahban and Wainwright 2012) established the following bound in
2n ||6 (c) 2 auydloog d/n] S 02 exep((=c3
Step 3. A final bound. We will prove the following result:
Consider two cases.
Case 1. Suppose we have
1 | -
r c P-r 1.
Now using
[
2r r r 1/
and
| r
S Aa +
|6 1
(Young's Incquality)
4n
with (2).. we have
r 1 1
9X3rr
r + 2X, (T*)).
Using
18 and 19). the above gives
26 RgT
P-r
+ 4hn R--2-
4
To minimize the bound above, the best to pick is r 26q -1)-1)n) > which leads
r 1 - CRu
Case 2. Suppose Case fails so that we have
r1)1
[ -- 1.
4n
20 gives
3
( < 2A, i -
+ |'
[[( - -r' + 2A.,|ITT' (T")l,
2
and from this, we can derive
S8
f- 1' +o-++ + (T"))l..
Case 2.1. Suppose
dlog
r ---
n
Note that An 2 C Cdllog d/n by definition. The equality might not directly hold due to the fitting emoor for m(x). It is of great interest to see whether such equality can be attained with methods such as the cross-fitting technique to avoid double-dipping and reduce bias (Chernozhukov al. 2018
et al. 2025).
19

---

Now Using that
Vd1d2l - 1c S 2a, || - r S 8v ] r
1| + (T))l
and the choice of T = An in Case 1, we obtain the inequality
[ r 3
- + CR
Applying Young's 's inequality again, we obtain
r
CR
Case 2.2. Suppose we have that
384d
[
r- || 2
then -f I 1536a/ /n and we obtain the desired rate directly.
Case 2.3. Suppose are under neither Case 2.1 and Case 2.2. Then the following inequality holds with probability at least e exp( -Gd log d) by our proof in Step
48d r - 10
6
l2
r
|F
-r
Vn.
16
Now using
20
we obtain that
3An
r
<2>n
r |
[[[
r' + 2An |T*
512
33v r
2Ap (T')
Combining the choice of r in Case
18 and 19 with Young's inequality, we have
S CRAAR
Step 4. Justify the RSC condition. It remains now to justify the RSC condition. We prove the following result holds with high probability:
48d A 00
2 8
Vn
for all A E R4 xd2 in the set
C= A AERR Rdi x
Ed
A 2 col
d log d
Define the following bad event:
B B- م2 EC|
3
- (6(()))
| F > A
48Ld A 00
Vn
Now we peel the set C with different radius:
C(D) = A C
|| d=
dd'. A| - D, |||
< D2 (cv dog d/n)}
and the peeled event
(3
1 |
48L
B(D) - 32 E C(D)
|6(T)
|A|F N D+
Vn
/n
By discretization argument, Lemma 4 and Lemma 5 in Negahban and Wainwright 2012) cnnD"), holds that
proved
the following bound: with probability at least 4 exp(
1 Vn
|| F S 3
48L
sup AEB(D)
|6
Vn
20

---

Now observing that for any A
C with
A 10 = ddd
we have that
dlog d
d dlog d
d d
| 2 col v d1d2) د
2 colla|1
2 co
FF
n
This then leads to
dlogd
| 2 201
Therefore, we only need to focus on C' where
C' A E A d A 2 co v log d/n).
By setting a = 7/6 and v = 00 / log d/n, we can peel C² into the following layers:
= A | so d= a S | F S v. | S
If the bad event B holds for some matrix A, then A must belong to some set Cý which satisfies
1
48L Vn
48L Vn"
||(
|||| F
- v+
-a
Thus, B(a'v) v) must hold. Applying a union bound, we must have
PP BBB S P (B(a'u))
sa
exp(-c2na?"
Se1
exp( 202n log(a)
1=1
C1 exp( - chd log d)
1 exp( c2 log d)
SC expl 1appp-cdloog d).
D.5 Proof of Theorem 5.1
Proof of Theorem 5.1 In the first stage, the regret is bounded by
T
Regret, = Σ
e I
I eas S 2T |
1=1
In the second stage, the regret is
Regret,
Σ
e
t=+
T
Σ
(e T(T' - -f)e, + (co, -Za)) Te
(T*
1=++11
T
Σ
(et, (2,1) T (T*- -f)e;
<
1=T+11
<2T.
|| - oo
STT.
| -fll
d1d2 log(d] dz)rd) 1/2
<2T,
Te
21

---

Therefore, the total regret across the two stages is
Regret =Regret + Regret,
(didz log(d1 + d2)rd
1/2
2TT | + +(T - -T)) I
Te
Now setting Te = T for some a E (0, 1). The above upper bound is minimized at logr (CT log(d1 d2)rd),
which leads final regret bound
Regret S CT2/3 d ///3/
E
Additional simulation results
E.1
Comparison of several kernel-based estimation strategies
Based on the setup of the Upworthy experiment, we also conduct a Monte Carlo simulation to evaluate the training and testing етror of the proposed method. For the first experiment, we evaluate the training and testing error the proposed method and compare it with several baseline methods. To do this, we split the synthetic dataset into a training set and testing set, fit the training data with different methods, and evaluate the fitted results on the testing dataset. The four baseline methods we evaluate are: (i) double kernel representation learning (DKRL); (ii) RKHS regression with only treatments (Treatment Only); (iii) RKHS regression with only covariates (Covariate Only); (iv) RKHS regression with both treatments and covariates using a product kernel K, @ KAn (Product Kernel). In our simulation, we vary across multiple levels of regularization parameters and compute the mean squared training and testing error. The results are summarized in Figure 2
(R2L
Coverizza
Product Band
03025
Testing Error
0.030
01815
032100
04530
20012
Training
Figure 2: Training and testing error for different methods
E.2 Interpretability of DKRL results
To further claborate on the interpretability of the learned feature, we show several example headlines with high low semantic scores in Table 2 which are good demonstrations on how the features capture the semantic meanings.
On the X row representing "sharp contrasts with narrative surprises", high-scored examples carry stronger "contrast + surprise" hooks. Each headline pairs familiar setup with jarring twist-Hank Green sparring with "corporate suit" over supposedly dull yet vital topic, a Black cop repeatedly stop-and-frisked. and cute animals foreshadowing a darker reveal-creating instant narrative surprise. The low-scored examples, the contrary, lean more curiosity emotional uplift (c.g., "A Lot Of
22

---

You Will Want To Know About It Anyway", "hese Pies Of A Baby Bird's Rescue Are So Moving") and less on jarring juxtapositions. They intrigue, but they don't 't rely heavily on opposites twists.
On the Y row representing "conffict-driven tone". high-scored examples signal confrontation, outrage, direct criticism, with words like "Deport THIS GUY", "attacks", "smackdown", "bad-ass", "tears into the awful folks". Besides, the examples pit one party against another (Hank Green vs. Corporate Suit, pink gang vs. predators, Fox Anchor vs. colleagues). The low-scored examples lean toward curiosity, wonder, and mild social critique. It uses softer hooks ("made me the happiest", "reason to think twice") and lacks the adversarial or combative language that characterizes the high-scored ones.
Table 2: Interpretability of the learned representations from DKRL in the Upworthy experiment. Feature X (X-axis Figure πa)) represents whether there are "sharp contrasts with narrative surprises", and feature Y (Y-axis in Figure I ()) represents whether the sentence is in a "conflict- driven tone".
Feature
Example headlines with high scores
Example headlines with low scores
"Anyone Able To Read This Won't
"Hank Green Goes Up Against A Corporate Suit To Debate An Unsexy Issue That Is Really A Big Deal";
Benefit From It, But A Lot Of You Will Want To Know About It Anyway";
"She Won A Golden Globe And Dedicated It To A Teen She Never Met.
"He's Black. He's a Cop. He's Also Been Stopped and Frisked 30 Times.";
X
Here's Why It's A Big Deal.";
"Floppy Ears, Soulful Eyes, Adorable Babies. And They Have Something Else
"These Pies Of A Baby Bird's Rescue
Are So Moving You Might Get Emotionally Involved Here"
People Can't 't Stop Taking."
"Heyy Have An Idea: Let's Deport THIS GUY, Instead";
"A Tiny Probe Millions Of Miles Away Just Made Me The Happiest I've Been In Weeks";
"This Pink Gang Attacks Sexual
Predators And Rapists That Harm Them. Tell Me, What Else Can They Do?":
"Here's A List Of Uncool Problems For Ladies That We Could Fix To Make Life Easier For Everyone":
"PLOT TWIST: Fox News Anchor Lays Epic Smackdown On Fox News Anchors For Obvious And Blatant Misogyny"
"A Reason To Think Twice Before Buying Your Kid All Those LEGOs"
E.3 Sensitivity to hyperparameter tuning
evaluated the sensitivity of our Algorithm T to two tuning parameters, penalty level and rank in Algorithm[1 I The results reported in Figure 3 In terms of selection of penalty level, an appropriate choice of penalty is important to balance the level of regularization and loss minimization. In practice, recommend using cross validation tune this parameter. In terms of rank selection, we can see that selecting a rank that is smaller than the truth could lead to bias, as this loses important features. In contrast, there is minimal bias from selecting slightly larger rank. In practice, recommend starting with a larger rank and trimming slightly.
E.4 The MIND dataset experiment
In this subsection, report the results for the synthetic simulation conducted for the MIND dataset in Table 3 As we have commented the main paper, while most methods benefit from the low-rank structure, DKRL exploits the low-rank structure more effectively. And even with high-rank interaction matrix, DKRL utilizes the similarity structure in the embeddings to enhance accuracy.
E.5
The ASOS experiment
In the second simulation based on the ASOS experiment, we generated a synthetic study following the similar DGP to the used in the main paper, with newly generated embeddings from the
23

---

Testing Penalty
Rank
Rank
0.0150
Naaan Testing Breer
0.0125
0.0100
0.0050
lng10(Penalty Levels)
Figure 3: Hyperparameter sensitivity against penalty level À and rank r.
Table 3: Testing MSE across methods (rows) and decay levels (columns)
Method
q
q = 2
=
DKRL
0.007 (0.003)
0.003 (0.001)
0.003 (0.001)
0.003 (0.001)
Lasso
0.034 (0.003)
0.009 (0.002)
0.008 (0.002)
0.008 (0.002)
XGB Z
0.036 (0.003)
0.007 (0.002)
0.007 (0.002)
0.007 (0.002)
FNN
0.034 (0.003)
0.011 (0.002)
0..11
0.011 (0.002)
Kernel Z
0.031 (0.003)
0,006 (0.002)
(0.000)
0.006 (0.002)
Lasso X
0.055 (0.005)
0.013 (0.004)
0.011(0.003))
0.011 (0.003)
XGB X
0.056 (0.006)
0.014 (0.004)
0.011 (0.003)
FNN X
0.055 (0.005)
(0.004)
0.011 (0.003)
0.011 (0.003)
Kerne X
0.055 (0.005)
0.013 (0.004)
(0.000)
0.011 (0.003)
Lasso ZX
0.032 (0.003)
0.009 (0.001)
0.008 (0.002)
0.008 (0.002)
XGB ZX
0.022 (0.001)
0,004 (0.001)
0.004 (0.001)
0.004 (0.001)
FNN ZX
0.016 (0.002)
0..10
0.009 (0.002)
0.010 (0.002)
Kernel ZX
0.035 (0.002)
0.008 (0.002)
0.007 (0.001)
0.007 (0.002)
shopping item information from the ASOS dataset. We then compared the KDRL method with two other methods: deep-learning based (SIN) method and product-kernel based (ProdKernel) method in terms of their training/testing error and computation time. The results are reported in Table 4 from which consolidate the finding that DKRL. performs fairly satisfactorily in terms of both training/testing errors and computation time.
24

---

Table 4: ASOS Experiment by Rank and Method (Mean (Std), rounded to four decimals)
Rank
Method
Train Error
Test Error
Time
SIN
0.0094 (0.0004)
0.0095 (0.0011)
7,9361 (0.4858)
DKRL
0.0022 (0.0002)
0.0035 (0.0009)
0.1739 (0.0305)
ProdKernel
0.0036 (0.0001)
0.0062 (0.0010)
0.0013 (0.0002)
SIN
0.0085 (0.0004)
0.0085 (0.0011)
8.0929 (0.3153)
3
DKRL
0.0025 (0.0002)
0.0047 (0.0010)
0,4702 (0. 1094)
ProdKernel
0.0044 (0.0002)
0.0075 (0.0012)
0.0014 (0.0003)
SIN
0.0119 (0.0014)
0.01 (0.0014)
7.9983 (0.4951)
5
DKRL
0.0027 (0.0000)
0.0076 (0.0012)
1.6868 (0.3907)
ProdKernel
0.0057 (0.0000)
0.0100 (0.0014)
0.0014 (0.0004)
SIN
0,0175 (0,0007)
0.0175 (0.0018)
8.1151 (0,4709)
10
DKRL
0.0026 (0.0001)
0.0128 (0.0017)
7.9518 (0.8747)
ProdKernel
0,0083 (0.0003)
0.0147 (0.0017)
0.0017 (0.0004)
NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: we outlined our contributions the abstract and Section | elaborates all the points we made.
Section 3 fully
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
we discussed the limitation of the work in Conclusion (Section 7
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
25

---

The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't It acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: presented the full sets of assumptions in the statement of the Theorems
(Theorem 4.1 4.2 5.1), and provide proof's to all the claims we made in Appendix D
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
we include the anonymized version of our code as supplementary materials and we are happy to release the code and data once accepted. Our main experimental results mainly involve an implementation of our algorithmss, which is detailed in [[] and 2 and running them on synthetic datasets which publicly available detailed in Section 6 of the paper.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same
26

---

dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We submit an anonymized version of our code as supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co
publicc/uuides/CoeeSubbuue
wow.eeddddbmmssssoooooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ praips.cc/uullcc/ee httppppulllguuudeeeeppppppp http://gggudee/ootSuuubimiibii www.bbmmssiiooooooooi.o for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
These are discussed in Section of the main paper.
Guidelines:
27

---

The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: we include standard deviations inter-quantile ranges for the experiment results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: we used personal computers (MBP with M2 Max chip) to run the experiments CPU and did not use additional computing resources. This information is included in Section
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http://///...eep...//////////aaa
28

---

Answer: [Yes]
Justification: we follow the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: we outlined the social impacts of the paper in advancing the techniques for digital experimentation in contribution. Yet there could negative impacts related to the use Large Language Models in experimentation such as privacy and fairness issues. We also outlined this in the Conclusion section.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation. generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: the paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
29

---

We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification: the paper does not existing assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. cow datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: the paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
30

---

According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country institution) were obtained?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: we discussed the use of LLMs this is a core part of our methodology. Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our L.M for what should or should not be described.
31


Reviews:
[{"review_id": "kPQKB829Ha", "reviewer": "Reviewer_pxjr", "summary": "This paper looks at estimating treatment effects in a randomized assignment setting where the interesting twist is that the set of possible treatments is high cardinality and it is possible that there is an interaction effect between user covariates and treatment. The basic idea is to model both the user covariates and treatments as embeddings in the same space. The key assumption is equation 3 at line 160, which (essentially) says y(x,z) = g(z)^T h(x) + noise, where g(z) is an embedding of treatment z and g(x) is embedding of unit covariates x. The paper’s main contribution is a kernel-based learning approach for this model.", "questions": "See strengths and weaknesses", "limitations": "yes", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "The paper is overall clear, the modeling assumption seems fine, and the theory seems fine (though I did not check it carefully). \n\nMy main concerns here are:\n1. (Why) is it important to view the treatments as LLM generated, and to consider this an experimentation setup? Like, what distinguishes the methodology here from any other problem using the form of (3) (e.g., most traditional approaches to recommender systems)\n2. Are the kernel methods themselves really important? It looks like equation 3 could be solved by parameterizing g and h as neural nets in the standard contrastive learning manner. Indeed, for the particular application in the experiments, taking g to the the original LLM that produced the embedding seems much more natural than manipulating that embedding with kernel learning. I’d at least like to see a baseline that just naively implements LoRA on the text-embedder.\n3. The only experiment included in the main paper has the user covariates entirely simulated! That seems like a substantial limitation of the analysis, above and beyond the missing LLM baseline noted above.", "was_revised": true, "final_justification": "I do not find my main concerns fully addressed by the author response, so I keep my score the same. But I am also not opposed to the paper being published as is.", "rebuttal": "We sincerely thank you for your careful review of our paper and insightful questions.\n\n**Re concern 1**: we agree with you that our setup shares many similarities with relevant problems, such as those in recommender systems. The reason we formulate the problem this way is that we are seeing an increasing practice in digital experimentation of incorporating LLM-generated treatments in A/B testing problems such as generating advertisements or email campaigns. The differences from traditional methods are two-fold: \n- Traditional recommendation methods typically choose among a fixed catalog of a few thousand items. Yet here, the \"treatment\" might be any of millions of possible sentences, with various styles or tones the LLM can produce. Designing an experiment to explore that space is qualitatively different from merely scoring existing items. Moreover, LLMs can be prompted to generate new variants on the fly (e.g. refine a headline after mid-experiment analysis), which turns the pipeline into an adaptive experiment rather than a static recommendation. \n- Here we are focusing on experimental design settings, where the treatments are \"Interventions\", not just features. This allows us to ask causal questions such as “What causal impact does wording X have on click-through, comprehension, or satisfaction?” rather than merely correlation questions like “What patterns correlate with clicks in historical logs?” This is guaranteed by unconfoundedness via randomization. By randomizing which LLM‐generated treatment each user sees, we break any link between unobserved user traits and the chosen treatment. That means we can estimate unbiased treatment effects (CATEs), whereas in a pure recommender system, we need to correct for selection bias and confounding issues on observational data.\n\n\n\n**Re your concern 2**: While one could indeed parameterize $g$ and $h$ as neural nets or fine-tune the original LLM (e.g. via LoRA or contrastive learning), our kernel-based formulation brings three key advantages:\n\n1. Nonparametric flexibility, avoiding the need to choose and tune a specific network architecture.\n\n2. Theoretical guarantees on convergence and generalization.\n\n3. Low-dimensional feature extraction, which enhances interpretability of the learned treatment effects.\n\nWe agree that a LoRA-based baseline would be a valuable comparison—this is a natural next step, and we intend to explore it in future work. In the present paper, we focus on CATE estimation with a fixed embedder and a downstream online experimentation strategy. We will leave treatment optimization and generation via fine-tuning as exciting directions for follow-up studies.\n\n**Regarding concern 3**: While we are eager to demonstrate our proposed framework on additional real-world problems, finding suitable, publicly available data that satisfies all the necessary desiderata (fully randomized, containing A/B tests on text, images, with user features) has proved challenging.  To address this, we therefore consider a range of different relaxations of the semi-synthetic simulation setup, which we believe is more representative of real-world settings. \n- First, we use a different real-world dataset to collect both text-based treatment and user-level covariates to avoid simulating user covariates. \n- Second, we relax the outcome generation process to allow more complex user-treatment interaction. \n- Third, we test out a broader set of baselines methods for a more comprehensive understanding. \n\nSpecifically, we analyze the MIND dataset, a benchmark dataset containing traffic from Microsoft for click rate prediction and news recommendation. It is an observational dataset where the probabilities of news being recommended to users are unknown. We consider a semi-synthetic scenario: suppose our goal is to train this recommendation system from scratch by adaptively collecting recommendation-clicking data and improve recommendation quality. We take $X$ to be user features such as news category preference and historical news click embeddings, and $Z$ as the embeddings for a set of candidate news to be recommended to users. All these embeddings are taken from the knowledge graph embeddings that were originally included in the dataset. We consider a synthetic outcome model for the click-through rate: $y = z^T\\Gamma x + epsilon$, where $\\Gamma$ is a matrix that encodes the interaction mechanism between users and treatments. It can be interpreted as a match between user preference and news information. We consider several setups of $\\Gamma = U \\Lambda V^\\top$, where $\\Lambda_i = i^{-q}$ ensures that the interaction model varies from low-rank (high eigenvalue decay rates $q$) to a high rank (low eigenvalue decay rates $q$) setting. We compare our method with four baselines: LASSO, XG boost, a Feed-forward neural network, and a kernel regression. The methods are combined with different estimation strategies: \"Z\" means only incorporating treatment information, \"X\" means only include covariate information, and \"ZX\" means include a concatenated (Z,X) vector.  The following two tables compare the training and testing RMSEs: \n\n**Training MSE**\n| Decay (q) | DKRL          | **Z**         |               |               |               | **X**         |               |               |               | **ZX**        |               |               |               |\n| --------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- |\n|           |               | Lasso\\_Z      | XGB\\_Z        | FNN\\_Z        | Kernel\\_Z     | Lasso\\_X      | XGB\\_X        | FNN\\_X        | Kernel\\_X     | Lasso\\_ZX     | XGB\\_ZX       | FNN\\_ZX       | Kernel\\_ZX    |\n| 0         | 0.006 (0.002) | 0.034 (0.002) | 0.027 (0.002) | 0.030 (0.002) | 0.031 (0.002) | 0.055 (0.005) | 0.053 (0.005) | 0.055 (0.005) | 0.055 (0.005) | 0.031 (0.002) | 0.002 (0.000) | 0.012 (0.002) | 0.035 (0.001) |\n| 2         | 0.003 (0.001) | 0.009 (0.002) | 0.005 (0.001) | 0.009 (0.002) | 0.006 (0.002) | 0.013 (0.004) | 0.013 (0.004) | 0.014 (0.004) | 0.013 (0.004) | 0.009 (0.001) | 0.001 (0.000) | 0.008 (0.002) | 0.008 (0.002) |\n| 4         | 0.003 (0.001) | 0.008 (0.002) | 0.005 (0.001) | 0.009 (0.002) | 0.006 (0.001) | 0.011 (0.003) | 0.010 (0.003) | 0.011 (0.003) | 0.011 (0.003) | 0.008 (0.002) | 0.001 (0.000) | 0.007 (0.002) | 0.006 (0.001) |\n| 6         | 0.002 (0.001) | 0.008 (0.002) | 0.005 (0.001) | 0.009 (0.002) | 0.006 (0.002) | 0.011 (0.003) | 0.011 (0.003) | 0.011 (0.003) | 0.011 (0.003) | 0.008 (0.002) | 0.001 (0.000) | 0.008 (0.002) | 0.007 (0.001) |\n\n**Testing MSE**\n| Decay (q) | DKRL          | **Z**         |               |               |               | **X**         |               |               |               | **ZX**        |               |               |               |\n| --------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- |\n|           |               | Lasso\\_Z      | XGB\\_Z        | FNN\\_Z        | Kernel\\_Z     | Lasso\\_X      | XGB\\_X        | FNN\\_X        | Kernel\\_X     | Lasso\\_ZX     | XGB\\_ZX       | FNN\\_ZX       | Kernel\\_ZX    |\n| 0         | **0.007 (0.003)** | 0.034 (0.003) | 0.036 (0.003) | 0.034 (0.003) | 0.031 (0.003) | 0.055 (0.005) | 0.056 (0.006) | 0.055 (0.005) | 0.055 (0.005) | 0.032 (0.003) | 0.022 (0.001) | 0.016 (0.002) | 0.035 (0.002) |\n| 2         | **0.003 (0.001)** | 0.009 (0.002) | 0.007 (0.002) | 0.011 (0.002) | 0.006 (0.002) | 0.013 (0.004) | 0.014 (0.004) | 0.014 (0.004) | 0.013 (0.004) | 0.009 (0.001) | **0.004 (0.001)** | 0.010 (0.002) | 0.008 (0.002) |\n| 4         | **0.003 (0.001)** | 0.008 (0.002) | 0.007 (0.002) | 0.011 (0.002) | 0.006 (0.001) | 0.011 (0.003) | 0.011 (0.003) | 0.011 (0.003) | 0.011 (0.003) | 0.008 (0.002) | **0.004 (0.001)** | 0.009 (0.002) | 0.007 (0.001) |\n| 6         | **0.003 (0.001)** | 0.008 (0.002) | 0.007 (0.002) | 0.011 (0.002) | 0.006 (0.002) | 0.011 (0.003) | 0.011 (0.003) | 0.011 (0.003) | 0.011 (0.003) | 0.008 (0.002) | **0.004 (0.001)** | 0.010 (0.002) | 0.007 (0.002) |\n\nFrom the table, we can tell that low-rank signals (larger $q$) do bring benefits to all methods—but DKRL exploits this structure most effectively, yielding the best generalization. Even when the signal is high-rank (for example, $q = 0$ which is full rank), DKRL can effectively exploit the similarity structure in embedding themselves and enhance accuracy. It actually achieves a comparable performance that is close to XG boost with combined features. \n\nThank you very much for the constructive points. We hope these explanations mitigate your concerns regarding our work."}, {"review_id": "BK408Xpnls", "reviewer": "Reviewer_1162", "summary": "This paper addresses the challenge of designing and analyzing experiments with a large number of treatments generated by Large Language Models (LLMs). The authors propose a novel method, Double Kernel Representation Learning (DKRL), to improve statistical efficiency by leveraging the semantic similarity between treatments, encoded in their embeddings. The core idea is to model the Conditional Average Treatment Effect (CATE) as a low-rank factorization of kernel-based representations of treatments and user covariates. The paper introduces an alternating minimization algorithm to learn these representations and provides theoretical guarantees, including convergence rates for estimation and a sublinear regret bound for an associated adaptive \"explore-then-commit\" online experimentation strategy. The method's effectiveness is demonstrated on semi-synthetic data derived from the Upworthy and ASOS datasets, where it shows improved performance over several baselines.", "questions": "I'd appreciate the authors' response to the weakness section.", "limitations": "Please see the weakness section.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The paper tackles a timely and significant problem. As LLMs make it easy to generate a vast number of content variations (e.g., ad creatives, email subject lines), the challenge of efficiently testing them becomes imminent. The proposed framework, which explicitly uses the semantic structure of these treatments via kernel methods, is a well-motivated approach to this problem.\n\n\n2. The proposed DKRL method is technically sound. The development of an alternating minimization algorithm for estimation is a standard but appropriate and computationally efficient solution. The theoretical contributions are a major strength, providing both estimation error bounds (Theorem 4.2) and a sublinear regret guarantee for the online algorithm (Theorem 5.1).  \n\nWeaknesses:\n\n1. The primary weakness is the reliance on semi-synthetic data for experimental validation. While the use of real treatment embeddings (from Upworthy headlines) is a good step, the user covariates are simulated Gaussian vectors, and the outcome is generated from a linear model with a low-rank ground truth. This setup is perfectly aligned with the model's assumptions, which almost guarantees that the proposed method will outperform others. The true challenge lies in real-world scenarios where the relationship between treatments, users, and outcomes is far more complex and does not necessarily adhere to a clean, low-rank structure. The lack of experiments on a fully real-world A/B testing dataset where user features are available makes it difficult to assess the method's practical utility and robustness.\n\n2. The paper's core assumption is that the CATE function has a low-rank structure.  While this is a common and powerful assumption in the matrix factorization literature, it may be too restrictive for many real-world applications. The paper briefly mentions that this can be relaxed to an approximately low-rank setting, but the theoretical analysis and experiments are based on the strict low-rank case. It is unclear how the method would perform if the true CATE function has a more complex, high-rank structure.\n\n3. The choice of baselines could be strengthened. For instance, the main competitor in the estimation task is SIN, a deep learning method. While DKRL outperforms it in the given setting, this is not surprising given that the data was generated from a model that favors DKRL.", "was_revised": true, "final_justification": "The authors have provided new empirical results and resolved my concern about their low-rank assumption. Although the empirical results are still preliminary, I find the motivation interesting and the theory rather solid.", "rebuttal": "We sincerely thank you for your careful review of our paper and insightful questions.\n\n**Re weakness 1**: Thank you very much for the critical comments. While we are eager to demonstrate our proposed framework on additional real-world problems, finding suitable, publicly available data that satisfies all the necessary desiderata (fully randomized, containing A/B tests on text, images, with user features) has proved challenging.  To address this, we therefore consider a range of different relaxations of the semi-synthetic simulation setup, which we believe is more representative of real-world settings. \n- First, we use a different real-world dataset to collect both text-based treatment and user-level covariates to avoid simulating user covariates. \n- Second, we relax the outcome generation process to allow more complex user-treatment interaction. \n- Third, we test out a broader set of baselines methods for a more comprehensive understanding. \n\nSpecifically, we analyze the MIND dataset, a benchmark dataset containing traffic from Microsoft for click rate prediction and news recommendation. It is an observational dataset where the probabilities of news being recommended to users are unknown. We consider a semi-synthetic scenario: suppose our goal is to train this recommendation system from scratch by adaptively collecting recommendation-clicking data and improve recommendation quality. We take $X$ to be user features such as news category preference and historical news click embeddings, and $Z$ as the embeddings for a set of candidate news to be recommended to users. All these embeddings are taken from the knowledge graph embeddings that were originally included in the dataset. We consider a synthetic outcome model for the click-through rate: $y = z^T\\Gamma x + epsilon$, where $\\Gamma$ is a matrix that encodes the interaction mechanism between users and treatments. It can be interpreted as a match between user preference and news information. We consider several setups of $\\Gamma = U \\Lambda V^\\top$, where $\\Lambda_i = i^{-q}$ ensures that the interaction model varies from low-rank (high eigenvalue decay rates $q$) to a high rank (low eigenvalue decay rates $q$) setting. We compare our method with four baselines: LASSO, XG boost, a Feed-forward neural network, and a kernel regression. The methods are combined with different estimation strategies: \"Z\" means only incorporating treatment information, \"X\" means only include covariate information, and \"ZX\" means include a concatenated (Z,X) vector.  The following two tables compare the training and testing RMSEs: \n\n**Training MSE**\n| Decay (q) | DKRL          | **Z**         |               |               |               | **X**         |               |               |               | **ZX**        |               |               |               |\n| --------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- |\n|           |               | Lasso\\_Z      | XGB\\_Z        | FNN\\_Z        | Kernel\\_Z     | Lasso\\_X      | XGB\\_X        | FNN\\_X        | Kernel\\_X     | Lasso\\_ZX     | XGB\\_ZX       | FNN\\_ZX       | Kernel\\_ZX    |\n| 0         | 0.006 (0.002) | 0.034 (0.002) | 0.027 (0.002) | 0.030 (0.002) | 0.031 (0.002) | 0.055 (0.005) | 0.053 (0.005) | 0.055 (0.005) | 0.055 (0.005) | 0.031 (0.002) | 0.002 (0.000) | 0.012 (0.002) | 0.035 (0.001) |\n| 2         | 0.003 (0.001) | 0.009 (0.002) | 0.005 (0.001) | 0.009 (0.002) | 0.006 (0.002) | 0.013 (0.004) | 0.013 (0.004) | 0.014 (0.004) | 0.013 (0.004) | 0.009 (0.001) | 0.001 (0.000) | 0.008 (0.002) | 0.008 (0.002) |\n| 4         | 0.003 (0.001) | 0.008 (0.002) | 0.005 (0.001) | 0.009 (0.002) | 0.006 (0.001) | 0.011 (0.003) | 0.010 (0.003) | 0.011 (0.003) | 0.011 (0.003) | 0.008 (0.002) | 0.001 (0.000) | 0.007 (0.002) | 0.006 (0.001) |\n| 6         | 0.002 (0.001) | 0.008 (0.002) | 0.005 (0.001) | 0.009 (0.002) | 0.006 (0.002) | 0.011 (0.003) | 0.011 (0.003) | 0.011 (0.003) | 0.011 (0.003) | 0.008 (0.002) | 0.001 (0.000) | 0.008 (0.002) | 0.007 (0.001) |\n\n**Testing MSE**\n| Decay (q) | DKRL          | **Z**         |               |               |               | **X**         |               |               |               | **ZX**        |               |               |               |\n| --------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- |\n|           |               | Lasso\\_Z      | XGB\\_Z        | FNN\\_Z        | Kernel\\_Z     | Lasso\\_X      | XGB\\_X        | FNN\\_X        | Kernel\\_X     | Lasso\\_ZX     | XGB\\_ZX       | FNN\\_ZX       | Kernel\\_ZX    |\n| 0         | **0.007 (0.003)** | 0.034 (0.003) | 0.036 (0.003) | 0.034 (0.003) | 0.031 (0.003) | 0.055 (0.005) | 0.056 (0.006) | 0.055 (0.005) | 0.055 (0.005) | 0.032 (0.003) | 0.022 (0.001) | 0.016 (0.002) | 0.035 (0.002) |\n| 2         | **0.003 (0.001)** | 0.009 (0.002) | 0.007 (0.002) | 0.011 (0.002) | 0.006 (0.002) | 0.013 (0.004) | 0.014 (0.004) | 0.014 (0.004) | 0.013 (0.004) | 0.009 (0.001) | **0.004 (0.001)** | 0.010 (0.002) | 0.008 (0.002) |\n| 4         | **0.003 (0.001)** | 0.008 (0.002) | 0.007 (0.002) | 0.011 (0.002) | 0.006 (0.001) | 0.011 (0.003) | 0.011 (0.003) | 0.011 (0.003) | 0.011 (0.003) | 0.008 (0.002) | **0.004 (0.001)** | 0.009 (0.002) | 0.007 (0.001) |\n| 6         | **0.003 (0.001)** | 0.008 (0.002) | 0.007 (0.002) | 0.011 (0.002) | 0.006 (0.002) | 0.011 (0.003) | 0.011 (0.003) | 0.011 (0.003) | 0.011 (0.003) | 0.008 (0.002) | **0.004 (0.001)** | 0.010 (0.002) | 0.007 (0.002) |\n\nFrom the table, we can tell that low-rank signals (larger $q$) do bring benefits to all methods—but DKRL exploits this structure most effectively, yielding the best generalization. Even when the interaction matrix is high-rank (for example, $q = 0$ which is full rank), DKRL can effectively exploit the similarity structure in embedding themselves and enhance accuracy. It achieves a comparable performance that is close to XG boost with combined features. \n\n\n**Re weakness 2**: We appreciate the reviewers questions regarding the low-rank assumption for user-treatment interactions. \n\nOur low rank assumption states:\n\n$\\text{rank}{(\\Gamma^\\star)} \\le r.$\n    \nFirst, we clarify that we can relax this assumption from a strict low rank to a so-called approximate low-rank structure by characterizing the spectral decay (i.e., the sorted eigenvalue distribution) of the matrix $\\Gamma^\\star$. This characterization is using the following inequality, where we bound the $l_q$ norm of the eigenvalues.\n\n$$\\sum \\lambda_i^q \\le R_q, ~\\text{for some } 0 \\le q < 1. $$\n\nThis is a much weaker version of strict low rank assumption, in the literature and has been well studied by references:\n- Negahban, Sahand, and Martin J. Wainwright. \"Restricted strong convexity and weighted matrix completion: Optimal bounds with noise.\" The Journal of Machine Learning Research 13.1 (2012): 1665-1697.\n- Rohde, Angelika, and Alexandre B. Tsybakov. \"ESTIMATION OF HIGH-DIMENSIONAL LOW-RANK MATRICES1.\" The Annals of Statistics 39.2 (2011): 887-930.\n\nUnder this assumption, we can establish the following guarantee in our setting:  \n\n$$ \\frac{1}{d_1d_2}||\\hat{\\Gamma} - \\Gamma^\\star||_F^2 \\le C R_q \\lambda_N^{2 - q} $$\n\n Strict low-rank assumption is a special case for this when q = 0; which then aligns with our theoretical results in the paper. More generally, it allows a continuously decaying signals, and there is a trade-off between the signal decaying speed and the estimation accuracy: the closer q is to 1, the harder it is to separate the signals and the wider the ERROR bound. Thus, our results continue to hold under this relaxed restriction. \n\n\nFrom a numerical perspective, we also added additional experiments to evaluate the performance of the method under the approximate low-rank setting with the MIND experiment in response to your point 1. As we can see from there, when there is low-rank struture, DKRL exploits the strict low-rank structure most effectively, yielding the best generalization. When the interation matrix is high-rank (for example, $q = 0$ which is full rank), DKRL can still effectively exploit the similarity structure in embedding themselves and enhance accuracy. It actually achieves a comparable performance that is close to XG boost with combined features. \n\n**Re weakness 3**: Thank you for this suggestion. In response to your point 1, we have performed a comparison with multiple baselines and estimation strategies, under different rank specifications. DKRL proves to have the advantage of adapting the low-dimensional information in both the embeddings and the interaction matrix. \n\nThank you very much for the constructive points. We truly hope that these explanations will mitigate your concerns regarding our work."}, {"review_id": "zo0mvR2D0h", "reviewer": "Reviewer_qjVW", "summary": "The paper proposes a kernel representation learning method for multiple continuous treatment variables and high-dimensional covariates, and applies it to adaptive treatment assignment in online experiments. The proposed method is theoretical guaranteed and validated through experiments on semi-synthetic datasets.", "questions": "1. What does d in line 235 mean? Is it $d_1+d_2$?\n\n2. What assumptions are required for the proposed method? For example, is the assumption of an additive-noise outcome model required? Are the SUTVA, overlap, and unconfoundedness assumptions required? Is there a need to assume a specific distribution for the noise term? (As Theorem 4.2 needs to assume that the noises are sub-exponential.)\n\n3. Is generating treatments by LLMs or other generative models only a downstream task? If I am mistaken, could the authors please clarify which step of the proposed method involves these related techniques?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n\n1. The theoretical results of the proposed method are extensive.\n\n2. The review of related work is thorough.\n\n**Weaknesses:**\n\n1. The paper is difficult to follow and gives a sense of exaggeration. The authors spend a lot of space in the title, abstract, introduction, and conclusion discussing the use of AI generative models, such as LLMs, to generate treatments. This gives the impression that the focus of the paper is on how to incorporate semantic information into generative models to guide the generation of treatments that better align with real-world scenarios. However, the subsequent sections of the paper hardly involve generative models, LLMs, or related topics. Instead, the focus is on using kernel representation learning to learn embeddings of complex treatments, with only a brief mention (mainly on page 7) of how this representation can guide treatment assignment in online experiments. Based on the content in the later sections, I believe the main motivation of the paper should be to learn embeddings of complex treatments, with guiding generative models to generate treatments and adaptive treatment assignment in online experiments being potential applications, rather than the primary focus of the work. I strongly recommend that the authors reconsider the positioning of the paper, clearly stating in the title, abstract, introduction, and conclusion that guiding generative models to generate treatments and adaptive treatment assignment in online experiments are potential applications of the proposed method, and that the real motivation behind the proposed method—learning embeddings for complex treatments—should be clearly highlighted.\n\n2. The paper does not provide a description of the assumptions required for the proposed method.\n\n3. It would be beneficial to include experiments in real online environments (rather than semi-synthetic data) and experiments where the proposed method guides LLMs to generate treatments, which can better validate the effectiveness of the proposed method in downstream applications.\n\n4. Many symbols are used before being defined, such as z and x in line 43, and d in line 235, etc.", "was_revised": true, "final_justification": "During the rebuttal process, the authors addressed most of my concerns and committed to revising the paper’s motivation, positioning, and core contributions. Additionally, they have provided more experimental results in richer scenarios. As a result, I am now inclined toward acceptance, provided these revisions are adequately incorporated in the final version.", "rebuttal": "We sincerely thank you for your careful review of our paper and insightful questions.\n\n**Re weakness 1**: Thank you for this suggestion. We have carefully considered the positioning of our paper, and will try to take your advice to reframe the paper. Since we cannot revise the manuscript during the rebuttal period, we hope to propose a new abstract following your suggestion:\n\n> The growing popularity of Large Language Models (LLMs) introduces exciting opportunities for digital experimentation, as marketers increasingly combine human-generated and model-generated content to design treatments. A central challenge in this setting lies in representing these complex, high-dimensional treatments in a way that preserves their semantic meaning and enables scalable analysis. In this paper, our primary focus is to learn low-dimensional representations of complex treatments that captures this semantic structure. These representations can then serve as a foundation for downstream applications such as guiding generative models to produce meaningful treatment variants and enabling adaptive treatment assignment in online experiments. To this end, we propose a method called double kernel representation learning, which models the causal effect through the inner product of kernel-based representations of treatments and user covariates. We develop an efficient alternating-minimization algorithm to learn these representations from data. Additionally, we propose an adaptive design strategy for online experimentation as an application. We provide convergence guarantees under a low-rank factor model and demonstrate the effectiveness of our approach through numerical experiments.\n\nWe will incorporate your suggestions in the next iteration of our paper. \n\n**Re weakness 2**: We apologize for not clearly stating the assumptions. We listed several assumptions for estimation in Theorem 4.2 and 5.1, including the noise distribution and the low-rank interaction structure. Moreover, as we are studying an experimental setup with no interference, we need SUTVA, unconfoundedness, and a weaker non overlapping assumptions ($e_z(x)\\in(0,1)$). We will clarify these in the updated draft.\n\n**Re weakness 3, part 1**: While we want to demonstrate our proposal on additional real-world problems, finding suitable, publicly available data that satisfies all the necessary desiderata (fully randomized, containing A/B tests on text, images, with user features) has proved challenging.  To address this, we consider a range of different relaxations of the semi-synthetic simulation setup, which we believe is more representative of real-world settings. \n- First, we use a different real-world dataset to collect both text-based treatment and user-level covariates to avoid simulating user covariates. \n- Second, we relax the outcome generation process to allow more complex user-treatment interaction. \n- Third, we test out a broader set of baselines methods for a more comprehensive understanding. \n\nSpecifically, we analyze the MIND dataset, a benchmark dataset containing traffic from Microsoft for click rate prediction and news recommendation. It is an observational dataset where the probabilities of item recommendation are unknown. We consider a semi-synthetic setup: suppose our goal is to train this recommendation system from scratch by adaptively collecting recommendation-clicking data and improve recommendation quality. We take $X$ to be user features such as news category preference and historical news click embeddings, and $Z$ as the embeddings for a set of candidate news to be recommended. All these embeddings are taken from the knowledge graph embeddings that are included in the dataset. We consider a synthetic outcome model for the click-through rate: $y = z^T\\Gamma x + epsilon$, where $\\Gamma$ is a matrix that encodes the interaction mechanism between users and treatments. It can be interpreted as a match between user preference and news information. We consider several setups of $\\Gamma = U \\Lambda V^\\top$, where $\\Lambda_i = i^{-q}$ ensures that the interaction model varies from low-rank (high eigenvalue decay rates $q$) to a high rank (low eigenvalue decay rates $q$) setting. We compare our method with four baselines: LASSO, XG boost, a Feed-forward neural network, and kernel regression. The methods are combined with different estimation strategies: \"Z\" means only incorporating treatment information, \"X\" means only include covariate information, and \"ZX\" means include a (Z,X) vector.  The following two tables compare the training and testing RMSEs: \n\n**Training MSE**\n| Decay (q) | DKRL          | **Z**         |               |               |               | **X**         |               |               |               | **ZX**        |               |               |               |\n| --------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- |\n|           |               | Lasso\\_Z      | XGB\\_Z        | FNN\\_Z        | Kernel\\_Z     | Lasso\\_X      | XGB\\_X        | FNN\\_X        | Kernel\\_X     | Lasso\\_ZX     | XGB\\_ZX       | FNN\\_ZX       | Kernel\\_ZX    |\n| 0         | 0.006 (0.002) | 0.034 (0.002) | 0.027 (0.002) | 0.030 (0.002) | 0.031 (0.002) | 0.055 (0.005) | 0.053 (0.005) | 0.055 (0.005) | 0.055 (0.005) | 0.031 (0.002) | 0.002 (0.000) | 0.012 (0.002) | 0.035 (0.001) |\n| 2         | 0.003 (0.001) | 0.009 (0.002) | 0.005 (0.001) | 0.009 (0.002) | 0.006 (0.002) | 0.013 (0.004) | 0.013 (0.004) | 0.014 (0.004) | 0.013 (0.004) | 0.009 (0.001) | 0.001 (0.000) | 0.008 (0.002) | 0.008 (0.002) |\n| 4         | 0.003 (0.001) | 0.008 (0.002) | 0.005 (0.001) | 0.009 (0.002) | 0.006 (0.001) | 0.011 (0.003) | 0.010 (0.003) | 0.011 (0.003) | 0.011 (0.003) | 0.008 (0.002) | 0.001 (0.000) | 0.007 (0.002) | 0.006 (0.001) |\n| 6         | 0.002 (0.001) | 0.008 (0.002) | 0.005 (0.001) | 0.009 (0.002) | 0.006 (0.002) | 0.011 (0.003) | 0.011 (0.003) | 0.011 (0.003) | 0.011 (0.003) | 0.008 (0.002) | 0.001 (0.000) | 0.008 (0.002) | 0.007 (0.001) |\n\n**Testing MSE**\n| Decay (q) | DKRL          | **Z**         |               |               |               | **X**         |               |               |               | **ZX**        |               |               |               |\n| --------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- |\n|           |               | Lasso\\_Z      | XGB\\_Z        | FNN\\_Z        | Kernel\\_Z     | Lasso\\_X      | XGB\\_X        | FNN\\_X        | Kernel\\_X     | Lasso\\_ZX     | XGB\\_ZX       | FNN\\_ZX       | Kernel\\_ZX    |\n| 0         | **0.007 (0.003)** | 0.034 (0.003) | 0.036 (0.003) | 0.034 (0.003) | 0.031 (0.003) | 0.055 (0.005) | 0.056 (0.006) | 0.055 (0.005) | 0.055 (0.005) | 0.032 (0.003) | 0.022 (0.001) | 0.016 (0.002) | 0.035 (0.002) |\n| 2         | **0.003 (0.001)** | 0.009 (0.002) | 0.007 (0.002) | 0.011 (0.002) | 0.006 (0.002) | 0.013 (0.004) | 0.014 (0.004) | 0.014 (0.004) | 0.013 (0.004) | 0.009 (0.001) | **0.004 (0.001)** | 0.010 (0.002) | 0.008 (0.002) |\n| 4         | **0.003 (0.001)** | 0.008 (0.002) | 0.007 (0.002) | 0.011 (0.002) | 0.006 (0.001) | 0.011 (0.003) | 0.011 (0.003) | 0.011 (0.003) | 0.011 (0.003) | 0.008 (0.002) | **0.004 (0.001)** | 0.009 (0.002) | 0.007 (0.001) |\n| 6         | **0.003 (0.001)** | 0.008 (0.002) | 0.007 (0.002) | 0.011 (0.002) | 0.006 (0.002) | 0.011 (0.003) | 0.011 (0.003) | 0.011 (0.003) | 0.011 (0.003) | 0.008 (0.002) | **0.004 (0.001)** | 0.010 (0.002) | 0.007 (0.002) |\n\nFrom the table, we can tell that low-rank signals (larger $q$) do bring benefits to all methods—but DKRL exploits this structure most effectively, yielding the best generalization. Even when the signal is high-rank (for example, $q = 0$ which is full rank), DKRL can effectively exploit the similarity structure in embedding themselves and enhance accuracy. It achieves a comparable performance that is close to XG boost with combined features. \n\n**Re weakness 2, part 2**: To your second point of using the proposal to guide LLMs to generate treatments, our online experimentation discussion in Section 5 can serve as an example. We use Explore-then-commit strategy to demonstrate how the proposed method can guide treatment optimization: in the exploration stage, our method can be used to learn a good user-system interaction model, and in exploitation, this model can be used to assign the best personalized treatment among a pool of LLM-generated treatments to maximize final rewards. Furthermore, we believe the downstream applications is extensive, for example, to finetune a language model to generate personalized treatment based on the learned representation model. These are saved as future work. \n\n**Re Q1**: yes, $d = d_1 + d_2$ is the sum of dimensions for user covariates and treatment embeddings. We apologize for the missing notation. \n\n** Re Q2**:  for theory, we need the additive noise model and low-rank assumption. Also, for identification, we need SUTVA, weak overlapping and unconfoundedness. We apologize that we did not formally state these, and will add these in the updated draft. \n\n**Re Q3**: generating treatments by LLMs is used in two aspects: (i) To learn how semantic information and user covariates interact and contribute to the outcome in A/B testings, we can build models in experiments with LLM generated treatments; (ii) it can guide optimizing personalized treatment generation with the learned model. So it's not just a downstream task; instead it is a full pipeline to borrow semantic information in LLM embeddings to understand and optimize treatment generation for maximizing rewards.\n\nThank you very much for the constructive points. We truly hope these explanations mitigate your concerns."}, {"review_id": "bk7fWbIXOz", "reviewer": "Reviewer_L7bj", "summary": "This work proposes a kernel-based representation learning method that models treatment effects as an inner product of unknown latent representations of treatment embeddings and user covariates. This lets practitioners test content (treatment) variations by changing their embeddings and quickly test hypotheses.", "questions": "See weaknesses above, I'm willing to raise my score if the authors can address them (which I think are reasonable).", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n- The problem of testing different treatment (content) variations is very relevant and well motivated.\n- As far as I could check, the theoretical statements are correct.\n- The preliminaries and warmup section are quite nice and important to understand the result.\n\n## Weaknesses \n- My main issue with this paper is connecting it with related works and more clearly seeing the contributions with respect to the setting they are presenting. Can the authors be more clear about what their estimator has that others works don't have? and how is that connected to using treatment embeddings? Maybe a table with existing literature, features of the estimators, including yours, could help visualize this. Right now the work seems disconnected, despite my understanding of the theoretical contribution and estimator.\n- The experimental setup seems very limited, can the authors justify not running the solution on more real-world settings?\n- A minor thing is to increase the resolution of the plots, consider using pdf figures.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your careful engagement with our paper and the great questions. \n\n**Re weakness 1**: \n- The novelty of the estimator: the key part of our estimator is that we introduce the kernel-based factorization approach to learn the CATE function. This estimator enjoys several advantages: (i) it allows us to learn the representations of treatment and user covariates contributing to the treatment effect. Treatment embeddings are taken in by the treatment kernel to learn a low-dimensional representation.  (ii) it is computationally efficient comparing with some deep learning baselines. This naturally extends to an efficient adaptive algorithm that assigns treatments to users in an online manner. (iii) backed with strong theoretical guarantees, including convergence rates for treatment effect estimation and a sublinear regret bounds of an algorithm for the adaptive experimentation setting. \n- Below is a table that compares several estimators :\n\n\n| Estimator          | Method              | Computation Efficiency | Incorporating both user and treatment info | Learning low-dimension representation |\n| ------------------ | ------------------- | ---------------------- | ------------------------------------ | ------------------------------- |\n| **DKRL**           | Kernel              | High                   | Yes                                  | Yes                             |\n| **SIN**            | Deep Neural Network | Low                    | Yes                                  | Yes but less interpretable      |\n| **Product Kernel** | Kernel              | High                   | Yes                                  | No                              |\n| **Treatment only** | General ML model    | High                   | No                                   | Not directly                    |\n| **Covariate only** | General ML model    | High                   | No                                   | Not directly                    |\n\nThe table demonstrates the advantages of our proposal. We will include the comparison in the updated draft. \n\n**Re weakness 2**: While we are eager to demonstrate our proposed framework on additional real-world problems, finding suitable, publicly available data that satisfies all the necessary desiderata (fully randomized, containing A/B tests on text, images, with user features) has proved challenging.  To address this, we therefore consider a range of different relaxations of the semi-synthetic simulation setup, which we believe is more representative of real-world settings. \n- First, we use a different real-world dataset to collect both text-based treatment and user-level covariates to avoid simulating user covariates. \n- Second, we relax the outcome generation process to allow more complex user-treatment interaction. \n- Third, we test out a broader set of baselines methods for a more comprehensive understanding. \n\nSpecifically, we analyze the MIND dataset, a benchmark dataset containing traffic from Microsoft for click rate prediction and news recommendation. It is an observational dataset where the probabilities of news being recommended to users are unknown. We consider a semi-synthetic scenario: suppose our goal is to train this recommendation system from scratch by adaptively collecting recommendation-clicking data and improve recommendation quality. We take $X$ to be user features such as news category preference and historical news click embeddings, and $Z$ as the embeddings for a set of candidate news to be recommended to users. All these embeddings are taken from the knowledge graph embeddings that were originally included in the dataset. We consider a synthetic outcome model for the click-through rate: $y = z^T\\Gamma x + epsilon$, where $\\Gamma$ is a matrix that encodes the interaction mechanism between users and treatments. It can be interpreted as a match between user preference and news information. We consider several setups of $\\Gamma = U \\Lambda V^\\top$, where $\\Lambda_i = i^{-q}$ ensures that the interaction model varies from low-rank (high eigenvalue decay rates $q$) to a high rank (low eigenvalue decay rates $q$) setting. We compare our method with four baselines: LASSO, XG boost, a Feed-forward neural network, and kernel regression. The methods are combined with different estimation strategies: \"Z\" means only incorporating treatment information, \"X\" means only include covariate information, and \"ZX\" means include a (Z,X) vector.  The following two tables compare the training and testing RMSEs: \n\n**Training MSE**\n| Decay (q) | DKRL          | **Z**         |               |               |               | **X**         |               |               |               | **ZX**        |               |               |               |\n| --------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- |\n|           |               | Lasso\\_Z      | XGB\\_Z        | FNN\\_Z        | Kernel\\_Z     | Lasso\\_X      | XGB\\_X        | FNN\\_X        | Kernel\\_X     | Lasso\\_ZX     | XGB\\_ZX       | FNN\\_ZX       | Kernel\\_ZX    |\n| 0         | 0.006 (0.002) | 0.034 (0.002) | 0.027 (0.002) | 0.030 (0.002) | 0.031 (0.002) | 0.055 (0.005) | 0.053 (0.005) | 0.055 (0.005) | 0.055 (0.005) | 0.031 (0.002) | 0.002 (0.000) | 0.012 (0.002) | 0.035 (0.001) |\n| 2         | 0.003 (0.001) | 0.009 (0.002) | 0.005 (0.001) | 0.009 (0.002) | 0.006 (0.002) | 0.013 (0.004) | 0.013 (0.004) | 0.014 (0.004) | 0.013 (0.004) | 0.009 (0.001) | 0.001 (0.000) | 0.008 (0.002) | 0.008 (0.002) |\n| 4         | 0.003 (0.001) | 0.008 (0.002) | 0.005 (0.001) | 0.009 (0.002) | 0.006 (0.001) | 0.011 (0.003) | 0.010 (0.003) | 0.011 (0.003) | 0.011 (0.003) | 0.008 (0.002) | 0.001 (0.000) | 0.007 (0.002) | 0.006 (0.001) |\n| 6         | 0.002 (0.001) | 0.008 (0.002) | 0.005 (0.001) | 0.009 (0.002) | 0.006 (0.002) | 0.011 (0.003) | 0.011 (0.003) | 0.011 (0.003) | 0.011 (0.003) | 0.008 (0.002) | 0.001 (0.000) | 0.008 (0.002) | 0.007 (0.001) |\n\n**Testing MSE**\n| Decay (q) | DKRL          | **Z**         |               |               |               | **X**         |               |               |               | **ZX**        |               |               |               |\n| --------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- |\n|           |               | Lasso\\_Z      | XGB\\_Z        | FNN\\_Z        | Kernel\\_Z     | Lasso\\_X      | XGB\\_X        | FNN\\_X        | Kernel\\_X     | Lasso\\_ZX     | XGB\\_ZX       | FNN\\_ZX       | Kernel\\_ZX    |\n| 0         | **0.007 (0.003)** | 0.034 (0.003) | 0.036 (0.003) | 0.034 (0.003) | 0.031 (0.003) | 0.055 (0.005) | 0.056 (0.006) | 0.055 (0.005) | 0.055 (0.005) | 0.032 (0.003) | 0.022 (0.001) | 0.016 (0.002) | 0.035 (0.002) |\n| 2         | **0.003 (0.001)** | 0.009 (0.002) | 0.007 (0.002) | 0.011 (0.002) | 0.006 (0.002) | 0.013 (0.004) | 0.014 (0.004) | 0.014 (0.004) | 0.013 (0.004) | 0.009 (0.001) | **0.004 (0.001)** | 0.010 (0.002) | 0.008 (0.002) |\n| 4         | **0.003 (0.001)** | 0.008 (0.002) | 0.007 (0.002) | 0.011 (0.002) | 0.006 (0.001) | 0.011 (0.003) | 0.011 (0.003) | 0.011 (0.003) | 0.011 (0.003) | 0.008 (0.002) | **0.004 (0.001)** | 0.009 (0.002) | 0.007 (0.001) |\n| 6         | **0.003 (0.001)** | 0.008 (0.002) | 0.007 (0.002) | 0.011 (0.002) | 0.006 (0.002) | 0.011 (0.003) | 0.011 (0.003) | 0.011 (0.003) | 0.011 (0.003) | 0.008 (0.002) | **0.004 (0.001)** | 0.010 (0.002) | 0.007 (0.002) |\n\nFrom the table, we can tell that low-rank signals (larger $q$) do bring benefits to all methods—but DKRL exploits this structure most effectively, yielding the best generalization. Even when the signal is high-rank (for example, $q = 0$ which is full rank), DKRL can effectively exploit the similarity structure in embedding themselves and enhance accuracy. It actually achieves a comparable performance that is close to XG boost with combined features. \n\n\n**Re weakness 3**: Thank you for the suggestion. We will take the advice and switch to pdf figures.\n\nThank you very much for the constructive points. We sincerely hope these explanations address your concerns!"}]

Author rebuttal:

Meta-review: This paper introduces a new method, double kernel representation learning, to enhance the statistical efficiency of designing and analysing LLM-generated treatments in large-scale experiments. The method improves statistical efficiency by explicitly incorporating semantic similarity from treatment embeddings. It works by factoring the causal effect through the inner product of low-dimensional, kernel-based representations of treatments and user covariates. The paper proposes an alternating-minimisation algorithm to efficiently learn these representations from data and also propose an adaptive design strategy for online experimentation.

After the author-reviewer discussion, the reviews for this paper are generally positive, and it's leaning toward borderline acceptance. The concerns raised by the expert reviewers are positioning of the paper, choice of methodology (i.e., kernel methods), and limited empirical evaluation. In the rebuttal, the authors provided additional empirical evaluation and promised to improve the positioning of the paper. The authors also justified their choice of kernel methods. In general, the reviewers found that the authors' responses addressed some of their original concerns.

My own assessment is that, the paper studies the timely topic and provides comprehensive theoretical and experimental results to support their claims. Given the authors keep their promises to improve the aforementioned aspects of the paper, the paper should be deemed acceptable to be published at NeurIPS.
Decision: Accept (poster)