Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper first uses the MDP to formulate the sequential bidding process, and then studies the problem of optimizing the a specific sequence with unknown incrementality parameters of the bidding process. 

The incrementality rate function $d_h$ is defined to enjoy the Markovian property in Assumption 1 and plays the role of "reward" in the MDP. The state $\ell$ is formulated by the number of rounds between the last winning of the learner and the current round. Therefore, the transition function is defined by the distribution of the highest bid among other bidders $F_h(\cdot)$. A special problem is that the incrementality has to be calcuated with Poisson process estimation. 

The algorithm is designed based on the principle "Optimism on the face of uncertainty" and the concentration inequalities of the parameters of the MDP. For each episode, it selects the optimal policy associated with the parameters $(\beta,\lambda)$ within the confidence sets, and adjust the confidence interval accordingly. It achieves similar regret bound $O(\sqrt{T})$ without knowing the environments beforehand. 
strengths_and_weaknesses: I am willing to reconsider my scores if the authors could provide a more detailed argument or some references for the following concerns.

1. The idea of proposed algorithm is very similar to that of the UCRL2 or UCBVI and the concentration inequalities are the same, which makes the technical contribution of this work less obvious.  

2. It would be better to emphasize the procedure to estimate the incrementality. It is hard to understand the usage of $X$ and $Y$ in the PAMM algorithm and the necessarity. 

There are many notations in this paper and some of them are not clearly defined before using, such as $n_t$ and $k_t$ in lines 293,295. 
questions: By the work of Simchowitz and Jameison (2019) we know that it is possible to achieve $O(\log T)$ regret for unknown transition MDP problems. Is it able to achieve some problem-dependent / gap-dependent regret bounds ? 
limitations: None.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a new problem setting in reinforcement learning which is specialized for modeling incrementality bidding. Authors claim that this is a more realistic model compared with previous learning-to-bid models. The main difficulty in solving this RL problem is the mixed and delayed feedback, which is tackled by a novel pairwise moment-matching algorithm. The final regret bound is $\widetilde{O}(H^2\sqrt{T})$, which does not depends on the number of actions due to the special structure of bidding. 
strengths_and_weaknesses: First of all, I need to mention that I am not very familiar with (learning to) bidding literature so there might be some misunderstanding. 

Strengths:
- The MDP model is more realistic than previous learning-to-bid models because it considers multiple bidding for one user and the delayed and mixed feedback while the authors claim that previous models only consider single bid and thus use contextual bandits as the model. 
- A new parameter estimation algorithm for delayed and mixed feedback. 
- The paper is well written and easy to follow. 

Weakness:
- The model still makes some strong assumptions. First, it is assumed that auctions happen per unit time. Second, it is assumed that the dependence of incrementality function on previous actions is only on the elapsed time since the last bid was won, but I understand that this may be necessary for theoretical analysis and it does make sense to me. 
- The related work part is mainly on bidding literature and there should be more discussion on RL literature, e.g., a detailed comparison of previous works on delayed feedback in RL. 

questions: - Can authors provide some examples that satisfy auctions happening every unit time? 

limitations: I am not aware of any potential negative societal impact
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work considers an online ad-bidding problem where the reward can be mixed and delayed. It models the problem using an MDP,  and it proposes an UCB-based RL algorithm to solve this problem with an analysis of the regret, with a moment matching routine for parameter estimation that addresses the mixed and delayed reward issue. 
strengths_and_weaknesses: Strength: This is a very interesting work considering mixed and delayed rewards in an online setting, and it proposes a method to solve these challenges.

Weakness: certain assumptions of the model may be a bit strong (please see the next section).

questions: This is a very interesting work, some of the following may be suggestions for future consideration, which may or may not be necessary for the current version.

- In the paper, \beta_h(l) is assumed to be lower bounded by some positive number c_{\beta} to indicate that any previous ad presence can only increase the conversion rate. However, in practice, sometimes this can be negative as well, for example seeing too many ads that one is not interested in can have a negative impact on certain users. Would the current framework be able to accommodate the settings where  \beta_h(l) is negative as well?

- The MDP considered currently has states related to how long time ago one user has seen this ad before. Would some other related features be helpful for the modeling too? In that case, the states will be more complicated, would the current solutions still apply, or can be easily modified? 

- What is the high-level idea and intuition of this moment matching algorithm? It will be helpful to explain the algorithm and intuition more clearly in the main body of the paper.

- It would be great if some synthetic or even real-life experiments can be presented to further illustrate the power of this method.

limitations: Not applicable.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors study a practically-motivated learning-to-bid problem in the online setting, motivated by the importance of incrementality bidding. To capture the causal effects of incrementality, the authors pose and analyze an MDP formulation, while past work operated under simplifying assumptions that yielded bandit formulations. Combining the MDP formulation with a novel parameter estimation algorithm, the authors establish regret bounds on the overall procedure.
strengths_and_weaknesses: This work seems to fill in an important gap in the theoretical learning-to-bid literature. It relaxes assumptions that allowed previous work to study a simplified bandit formulation, and provides theoretical guarantees for an important practical setting introduced by Lewis and Wong. 
questions: As far as I can tell the work is theoretically sound and is a good contribution. I’m not intimately familiar nor up-to-date on the literature on bidding/autobidding/what is the state of the art with respect to these things in practice. Thus, when reading the paper, some of the important terms like “conversion”, “incrementality bidding”, and so on were unfamiliar to me and I had to do some digging to get some understanding. If somehow the bidding landscape could be set up in a more reader-friendly way, that would be good. But, I think for someone who is more familiar with this field the paper would be accessible and well written.

Are there experiments that the authors could include that would help the theory? I understand that this is a theory paper that complements Lewis and Wong, but perhaps an experimental comparison to the results of that paper would be interesting.
limitations: N/A
review_title: null

Meta-review: metareview: The paper studies sequential bidding problems by formulating them as MDPs (previous attempts made simplifying assumptions that yielded bandit formulations). The reviewers found the arguments about the causal effects of incrementality requiring richer modeling very convincing, and appreciated the difficulty of the problem due to delayed and mixed rewards. This necessitated a new combination of pairwise moment matching with optimism in the face of uncertainty. 

The authors clarified the reveiwers' technical questions about the PAMM algorithm during the feedback phase, and reviewers reached consensus that the paper's contributions are novel, interesting, and likely to lead to future research and real-world applications.

confidence: Certain
Decision: Accept