Title: Learning Personalized Ad Impact via Contextual Reinforcement Learning under Delayed Rewards
Venue: NeurIPS 2025
Accepted: yes
Keywords: online learning; Contextual Markov Decision Process; maximum likelihood estimation; delayed reward; reinforcement learning; auto-bidding
Abstract: Online advertising platforms use automated auctions to connect advertisers with potential customers, requiring effective bidding strategies to maximize profits. Accurate ad impact estimation requires considering three key factors: delayed and long-term effects, cumulative ad impacts such as reinforcement or fatigue, and customer heterogeneity. However, these effects are often not jointly addressed in previous studies. To capture these factors, we model ad bidding as a Contextual Markov Decision Process (CMDP) with delayed Poisson rewards. For efficient estimation, we propose a two-stage maximum likelihood estimator combined with data-splitting strategies, ensuring controlled estimation error based on the first-stage estimator's (in)accuracy. Building on this, we design a reinforcement learning algorithm to derive efficient personalized bidding strategies. This approach achieves a near-optimal regret bound of $\tilde{\mathcal{O}}(dH^2\sqrt{T})$, where $d$ is the contextual dimension, $H$ is the number of rounds, and $T$ is the number of customers. Our theoretical findings are validated by simulation experiments.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel approach to personalized ad bidding that simultaneously addresses delayed rewards, cumulative ad effects, and customer heterogeneity. The authors model this complex environment as a CMDP with delayed Poisson rewards, where context is used for personalization and states capture the history of ad exposure. A core contribution is the development of a two-stage maximum likelihood estimator (TS-MLE) combined with a data-splitting strategy, designed to efficiently and accurately estimate the delayed impacts of advertising. Based on this estimation framework, the paper proposes a reinforcement learning algorithm that learns personalized bidding strategies and is proven to achieve a near-optimal regret bound.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n1. The paper studied an important and interesting question in real-world ads bidding. All the three components have been extensively studied in the context of Ads CTR prediction task but not well studied under the auto bidding setting.\n2. The authors provide solid theoretical guarantees on the regret bound and proved that the proposed algorithm achieves near optimal regret.\n\nWeakness\n1. The proposed model is not realistic compared to real-world Ads applications. A fixed and simple linear structure is assumed to model cumulative effects while in CTR modeling sequence models are used. Moreover, the heterogeneity not only comes from user and context but also comes from different types of ads.\n2. The algorithm and the theoretical analysis relies on potentially strong assumptions that may be hard to generalize for example the state definition, highest competing bids follow a lognormal distribution, the specific reward function formula.\n3. The paper is entirely theoretical and lacks any experimental results to validate the proposed algorithms. Without experiments, even on synthetic data, it is difficult to assess the algorithm's practical performance, scalability, and robustness to assumption violations.\n\nQUESTIONS:\n1. The inner product often contains additional parenthesis and sometimes missing comma like in line 161\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nMotivated by the widely documented phenomenon of *recency bias*, the authors propose a reinforcement learning algorithm for efficient auto-bidding/ad placement which takes into consideration cumulative and delayed effects of prior advertising on user behaviour, rather than focusing solely on short-term engagement signals. Specifically, the auto-bidding decision process is modelled as a contextual MDP with delayed rewards represented through a Poisson distribution. The contextual factor allows for personalisation, accounting for preferences, seasonal factors etc., whilst the state captures the combined, long-term effect of multiple ads on the user over time. The authors note that prior work by Badanidiyuru et al. [2023] explores modeling the long-term effects of ads on users also via MDPs with Poisson rewards, however they assume homogenous treatment effects across the user-base, effectively ignoring personalisation as a factor in the outcome. This represents the first theoretical work on the design of bidding algorithms which addresses this factor alongside modeling delayed and cumulative ad impacts on users.\n\nA three-stage process is formulated for solving these MDPs. Firstly, an *exploration* stage, where ridge regression is used with variance estimators to estimate the transition dynamics. Secondly, an *exploitation* stage takes place which involves estimation of the advertisement effects. Finally, a two-stage likelihood estimator is developed by the authors in order to estimate the delayed advertisement effects. The latter approach is required due to two fundamental challenges, the non-concave nature of the log-likelihood, and the lack of closed form solutions to the scoring equations.\n\nThe contributions of the paper are theoretical, with the authors presenting proof of a near optimal regret bound for their algorithm. Evaluation on real-world data is left to future work, owing to the lack of publicly available online bidding datasets.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n* Overall, the theoretical analysis carried out by the authors is thorough, and results in a near-optimal regret bound for online contextual RL with delayed Poisson rewards, which is a high quality and novel contribution to the literature. This contribution lays the groundwork for future work to explore the practical implementation of such an algorithm in real-world settings with online bidding datasets, likely by organisations which have access to such data on a large scale, as well as for additional theoretical contributions on considerations relevant to large-scale deployment such as budget constraints.\n* The authors present and introduce the background and problem setting clearly. Early in the paper, the work is placed clearly in the context of prior work in the area, with an intuitive explanation given of where this work offers a novel contribution compared to this prior work.\n* Whilst it’s difficult to categorically assign a very high level of significance to an approach which has been analysed theoretically before it has been evaluated in a real-world deployment setting, this approach does seem to have the potential for considerable impact. As the authors note themselves, this problem of learning to bid is highly challenging for a number of reasons, such as the heterogeneity of preferences across user bases on online services, the delayed nature of any feedback on such services, and the cumulative nature by which advertising impacts the behaviour of end users. The development of an effective approach to bidding which simultaneously tackles all of these concerns is a considerable step forward. \n\n**Weaknesses**\n* Whilst the discussion of background and prior work in the Introduction is clearly outlined, it is somewhat brief, limited to effectively around a single paragraph. Whilst of course space is always a limitation, I feel it would be helpful to include (or at least point to in the appendix), a slightly extended discussion or table listing the other referenced approaches to learning to bid, including brief details about the core of their approach and any obvious advantages/disadvantages relative to the proposed method.\n* One question which arose whilst reading the section discussing the state formulation was that whilst I fully understand the motivation for using only very recent ad exposures due to recency bias, I wondered whether there is any support/discussion in the literature surrounding only focusing on the single most recent advertising event. This is presented as the default choice for the algorithm in the paper, so although we’re not able to empirically assess the algorithm to check, if possible it would be useful to have some further discussion of how much useful information is gained by for example, extending the state formulation to the previous two ad exposures versus just the most recent one.\n\nQUESTIONS:\n**Q1** - Regarding the point raised in *Weaknesses* above regarding the background literature, could the authors please extend the discussion of prior work in the area in either the main text or appendix through additional text or a table briefly detailing competing approaches and their properties?\n\n**Q2** - As also mentioned prior, could the authors expand slightly on the impact of the decision on how the state should be formulated, and if possible, how much benefit extending the state from one to two or more recent ad exposures may have on performance? I understand here that this cannot yet be assessed empirically, however if there is other prior work which may point to an answer to this question, that would be useful detail to include.\n\nThese minor points would certainly improve the paper and I encourage the authors to address these points, however I feel that the contribution is strong regardless, and therefore recommend acceptance.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the problem of optimizing online ad bidding strategies by creating a model that, unlike previous work, simultaneously accounts for three real-world complexities: 1) the delayed impact of ads on customer purchases, 2) the cumulative effects of repeated ad exposure, and 3) customer heterogeneity. To achieve this, the authors make the following key contributions: They model the ad bidding problem as a CMDP with delayed Poisson rewards. The \"contextual\" aspect allows for personalized strategies, while the MDP structure captures cumulative effects over time. They introduce a novel and efficient two-stage maximum likelihood estimator with a data-splitting strategy. This allows them to learn the parameters of their complex model online while carefully controlling for estimation errors caused by the delayed rewards. Based on this estimation method, they design a reinforcement learning algorithm that learns personalized bidding policies and prove that it achieves a near-optimal theoretical regret bound.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The paper tackles a highly significant and practical problem in computational advertising. Its primary contribution is the novel and unified CMDP framework that, for the first time, jointly models the crucial dynamics of delayed rewards, cumulative ad effects, and user heterogeneity.\n\n2. The technical quality of the paper is exceptionally high. The main theoretical contribution—the development of the two-stage MLE for online estimation under delayed Poisson rewards—is sophisticated and rigorously analyzed.\n\n3. The paper is well-written, clearly structured, and does an excellent job of motivating the problem by highlighting the limitations of existing approaches.\n\nWeakness\n\nThe most significant weakness of the paper is the complete absence of empirical results. The work is purely theoretical, with no simulations or experiments on real or synthetic data. While the theoretical guarantees are strong, this omission makes it impossible to assess the practical performance of the algorithm and the real-world magnitude of the constants hidden in the regret bound and the robustness of the method to violations of its underlying assumptions.\n\nQUESTIONS:\n1. The paper is entirely theoretical. Could you please justify this choice? While the theoretical results are strong, the absence of simulations makes it difficult to gauge the practical performance of the proposed algorithm. My evaluation would improve significantly if you could provide even a small-scale simulation study on synthetic data. \n\n2. The model relies on several specific parametric assumptions. How sensitive do you expect your theoretical results and your proposed algorithm to be if these assumptions are violated, as they likely would be in a real-world setting?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

comments: SUMMARY:
The paper addresses the challenge of modeling and optimizing online advertising bids by integrating delayed rewards, cumulative ad impacts, and customer heterogeneity. It formulates ad bidding as a Contextual Markov Decision Process (CMDP) with delayed Poisson rewards and proposes a two-stage maximum likelihood estimator combined with a reinforcement learning algorithm to derive personalized bidding strategies.

STRENGTHS AND WEAKNESSES:
Strengths:
1. It is interesting to modeling the online advertising problem through Contextual MDP.
2. Concrete theoretical proofs for the regret bound.

Weaknesses:
1. As the motivation of this work is the online advertising problem, the empirical experiments seem to be necessary. Basically, it is hard to determine whether the assumption about the conversion rate is reasonable.

QUESTIONS:
NA

LIMITATIONS:
NA
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-04