Title: Leveraging semantic similarity for experimentation with AI-generated treatments
Venue: NeurIPS 2025
Accepted: yes
Keywords: Experimentation

Kernel methods

Causal inference

Embeddings

LLM-generated treatments
Abstract: Large Language Models (LLMs) enable a new form of digital experimentation where treatments combine human and model-generated content in increasingly sophisticated ways. The main methodological challenge in this setting is representing these high-dimensional treatments without losing their semantic meaning or rendering analysis intractable. Here we address this problem by focusing on learning low-dimensional representations that capture the underlying structure of such treatments. These representations enable downstream applications such as guiding generative models to produce meaningful treatment variants and facilitating adaptive assignment in online experiments. We propose double kernel representation learning, which models the causal effect through the inner product of kernel-based representations of treatments and user covariates. We develop an alternating-minimization algorithm that learns these representations efficiently from data and provide convergence guarantees under a low-rank factor model. As an application of this framework, we introduce an adaptive design strategy for online experimentation and demonstrate the method's effectiveness through numerical experiments.


Peer reviews:
comments: SUMMARY:
This work proposes a kernel-based representation learning method that models treatment effects as an inner product of unknown latent representations of treatment embeddings and user covariates. This lets practitioners test content (treatment) variations by changing their embeddings and quickly test hypotheses.

STRENGTHS AND WEAKNESSES:
## Strengths

- The problem of testing different treatment (content) variations is very relevant and well motivated.
- As far as I could check, the theoretical statements are correct.
- The preliminaries and warmup section are quite nice and important to understand the result.

## Weaknesses 
- My main issue with this paper is connecting it with related works and more clearly seeing the contributions with respect to the setting they are presenting. Can the authors be more clear about what their estimator has that others works don't have? and how is that connected to using treatment embeddings? Maybe a table with existing literature, features of the estimators, including yours, could help visualize this. Right now the work seems disconnected, despite my understanding of the theoretical contribution and estimator.
- The experimental setup seems very limited, can the authors justify not running the solution on more real-world settings?
- A minor thing is to increase the resolution of the plots, consider using pdf figures.

QUESTIONS:
See weaknesses above, I'm willing to raise my score if the authors can address them (which I think are reasonable).

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a kernel representation learning method for multiple continuous treatment variables and high-dimensional covariates, and applies it to adaptive treatment assignment in online experiments. The proposed method is theoretical guaranteed and validated through experiments on semi-synthetic datasets.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1. The theoretical results of the proposed method are extensive.\n\n2. The review of related work is thorough.\n\n**Weaknesses:**\n\n1. The paper is difficult to follow and gives a sense of exaggeration. The authors spend a lot of space in the title, abstract, introduction, and conclusion discussing the use of AI generative models, such as LLMs, to generate treatments. This gives the impression that the focus of the paper is on how to incorporate semantic information into generative models to guide the generation of treatments that better align with real-world scenarios. However, the subsequent sections of the paper hardly involve generative models, LLMs, or related topics. Instead, the focus is on using kernel representation learning to learn embeddings of complex treatments, with only a brief mention (mainly on page 7) of how this representation can guide treatment assignment in online experiments. Based on the content in the later sections, I believe the main motivation of the paper should be to learn embeddings of complex treatments, with guiding generative models to generate treatments and adaptive treatment assignment in online experiments being potential applications, rather than the primary focus of the work. I strongly recommend that the authors reconsider the positioning of the paper, clearly stating in the title, abstract, introduction, and conclusion that guiding generative models to generate treatments and adaptive treatment assignment in online experiments are potential applications of the proposed method, and that the real motivation behind the proposed method—learning embeddings for complex treatments—should be clearly highlighted.\n\n2. The paper does not provide a description of the assumptions required for the proposed method.\n\n3. It would be beneficial to include experiments in real online environments (rather than semi-synthetic data) and experiments where the proposed method guides LLMs to generate treatments, which can better validate the effectiveness of the proposed method in downstream applications.\n\n4. Many symbols are used before being defined, such as z and x in line 43, and d in line 235, etc.\n\nQUESTIONS:\n1. What does d in line 235 mean? Is it $d_1+d_2$?\n\n2. What assumptions are required for the proposed method? For example, is the assumption of an additive-noise outcome model required? Are the SUTVA, overlap, and unconfoundedness assumptions required? Is there a need to assume a specific distribution for the noise term? (As Theorem 4.2 needs to assume that the noises are sub-exponential.)\n\n3. Is generating treatments by LLMs or other generative models only a downstream task? If I am mistaken, could the authors please clarify which step of the proposed method involves these related techniques?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the challenge of designing and analyzing experiments with a large number of treatments generated by Large Language Models (LLMs). The authors propose a novel method, Double Kernel Representation Learning (DKRL), to improve statistical efficiency by leveraging the semantic similarity between treatments, encoded in their embeddings. The core idea is to model the Conditional Average Treatment Effect (CATE) as a low-rank factorization of kernel-based representations of treatments and user covariates. The paper introduces an alternating minimization algorithm to learn these representations and provides theoretical guarantees, including convergence rates for estimation and a sublinear regret bound for an associated adaptive \"explore-then-commit\" online experimentation strategy. The method's effectiveness is demonstrated on semi-synthetic data derived from the Upworthy and ASOS datasets, where it shows improved performance over several baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper tackles a timely and significant problem. As LLMs make it easy to generate a vast number of content variations (e.g., ad creatives, email subject lines), the challenge of efficiently testing them becomes imminent. The proposed framework, which explicitly uses the semantic structure of these treatments via kernel methods, is a well-motivated approach to this problem.\n\n\n2. The proposed DKRL method is technically sound. The development of an alternating minimization algorithm for estimation is a standard but appropriate and computationally efficient solution. The theoretical contributions are a major strength, providing both estimation error bounds (Theorem 4.2) and a sublinear regret guarantee for the online algorithm (Theorem 5.1).  \n\nWeaknesses:\n\n1. The primary weakness is the reliance on semi-synthetic data for experimental validation. While the use of real treatment embeddings (from Upworthy headlines) is a good step, the user covariates are simulated Gaussian vectors, and the outcome is generated from a linear model with a low-rank ground truth. This setup is perfectly aligned with the model's assumptions, which almost guarantees that the proposed method will outperform others. The true challenge lies in real-world scenarios where the relationship between treatments, users, and outcomes is far more complex and does not necessarily adhere to a clean, low-rank structure. The lack of experiments on a fully real-world A/B testing dataset where user features are available makes it difficult to assess the method's practical utility and robustness.\n\n2. The paper's core assumption is that the CATE function has a low-rank structure.  While this is a common and powerful assumption in the matrix factorization literature, it may be too restrictive for many real-world applications. The paper briefly mentions that this can be relaxed to an approximately low-rank setting, but the theoretical analysis and experiments are based on the strict low-rank case. It is unclear how the method would perform if the true CATE function has a more complex, high-rank structure.\n\n3. The choice of baselines could be strengthened. For instance, the main competitor in the estimation task is SIN, a deep learning method. While DKRL outperforms it in the given setting, this is not surprising given that the data was generated from a model that favors DKRL.\n\nQUESTIONS:\nI'd appreciate the authors' response to the weakness section.\n\nLIMITATIONS:\nPlease see the weakness section.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper looks at estimating treatment effects in a randomized assignment setting where the interesting twist is that the set of possible treatments is high cardinality and it is possible that there is an interaction effect between user covariates and treatment. The basic idea is to model both the user covariates and treatments as embeddings in the same space. The key assumption is equation 3 at line 160, which (essentially) says y(x,z) = g(z)^T h(x) + noise, where g(z) is an embedding of treatment z and g(x) is embedding of unit covariates x. The paper’s main contribution is a kernel-based learning approach for this model.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper is overall clear, the modeling assumption seems fine, and the theory seems fine (though I did not check it carefully). \n\nMy main concerns here are:\n1. (Why) is it important to view the treatments as LLM generated, and to consider this an experimentation setup? Like, what distinguishes the methodology here from any other problem using the form of (3) (e.g., most traditional approaches to recommender systems)\n2. Are the kernel methods themselves really important? It looks like equation 3 could be solved by parameterizing g and h as neural nets in the standard contrastive learning manner. Indeed, for the particular application in the experiments, taking g to the the original LLM that produced the embedding seems much more natural than manipulating that embedding with kernel learning. I’d at least like to see a baseline that just naively implements LoRA on the text-embedder.\n3. The only experiment included in the main paper has the user covariates entirely simulated! That seems like a substantial limitation of the analysis, above and beyond the missing LLM baseline noted above.\n\nQUESTIONS:\nSee strengths and weaknesses\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}