Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this paper, the authors provide a new online bidding algorithm using reinforcement learning and a safety function to accommodate the mismatch between online and offline data (IBOO challenge in the paper). The authors characterize the lipshitzness of the safety function and validate the effectiveness of the proposed algorithm through synthetic data and real advertising data.
strengths_and_weaknesses: Strengths: This paper investigates a very important problem in practice. Auto-bidding is getting more and more popular in real production system and this paper provides a very reasonable approach to design real time bidding algorithm. The paper is well-written and clearly states the contribution. In addition, the authors show their algorithm can achieve very good performance empirically. I have several question as follows.

Weaknesses: There is no theoretical guarantee for the proposed bidding algorithm, however, this doesn't really hurt the contribution of this paper since the authors show that their algorithm achieves good performance over real advertising.
questions: If I understand correctly, one of the main causes to IBOO problem is the mismatch of the data resource for RAS and VAS. Could the authors provide more details why we can only collect data in stage 2 in VAS?
limitations: N/A
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper tries to address the problem of inconsistency between the online and offline settings in the outbidding problem with safeguards. It proposes an online RL framework to improve the auto-bidding policies through direct interactions with the real environment. An online exploration policy is discussed trying to address safety and efficiency requirements. 

strengths_and_weaknesses: Strength: The paper tries to address the problem of inconsistency between the online and offline settings in the outbidding problem with safeguards, and the experiments look encouraging.

Weakness: In certain places, the presentation can be made clearer, and the algorithm details need more elaboration. 

questions: - It seems the discussion of the related literature is not very clear. For example, it is stated that if one trains the policy using data collected by a certain behavior policy directly from the RAS, then there will be certain extrapolation errors that degrade the policy performance. More discussion should be given here to avoid confusion. It is also helpful to be more specific and have some discussion on how this paper differs from the other safe online RL methods that try to explore the environment. It is also stated that there are methods using constraint functions, how does these methods related to this problem, and specifically why are they not good candidates?  

- In the algorithm section, the algorithm details need more elaboration, also the description of the warm boosting section can be made more clear.

limitations: Not applicable.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper tries to address the problem in auto-bidding that training can only be carried out offline due to safety concerns. The performance of the learned policy could deteriorate due to the difference between the offline simulation and online real environment. The authors propose a sustainable online reinforcement learning paradigm which uses a safe policy to explore the real enviroment.
strengths_and_weaknesses: Strengths:
(1) the problem to be addressed is important and novel.
(2) the proposed idea is interesting and technical depth is deep.
(3) experiments are conducted on a real world platform.

Weaknesses (The revised paper and the authors' response addressed these major concerns.):
(1) There are approximations/relaxations without proper justification or theoretical guarantee.
(2) There could be flaws in some proofs.
(3) The experiments fail to compare the proposed method to SOTA methods.

Detailed comments:

The major problems with this work are:

-There are approximations/relaxations without proper justification or theoretical guarantee. At the end of section 3, the authors said we can relax the constraint "the safty zone is effective only when the subsequent actions strictly follow the saft policy $\mu_s$". But there is neither jutification for the relaxation nor theoretical analysis on how this relaxation could affect the safe property of the online exploration. Another issue lies in section 4.3, where the authors modify the SER policy without justification.

-The proof for theorem 1 seems problematic. The upper bound in Eq.(12) could be violated by such a case: $\frac{p_{j,t}^2}{v_{j,t}^2} > a_2, \forall j \in \{1, \dots, N_t\}$, which means $a_2$ fails to win any impressions. In this case, subtracting the winning impression upperbound for $a_2$ seems to underestimate the bound.

-The authors fail to compare to the SOTA works in the auto-bidding field, e.g., ["Multi-Agent Cooperative Bidding Games for Multi-Objective Optimization in e-Commercial Sponsored Search." SIGKDD 2021] develops an evolutionary strategy to learn the optimal policy and achieves better performance than RL methods. Furthermore, as the authors argue in Figure 2(b), the bidding involving all advertisers naturally falls into the realm of multi-agent RL. However, the proposed method is still a single agent solution which cannot well handle the market price change issue. In comparison, the above work sets up a multi-agent bidding game for policy learning.


Minor issues:

-The presentation could be improved. The figures are not detailedly explained in the main text. Some notations are confusing. See the first question in the question part for an example. Another example is that the authors use both $\pi$ and $\mu$ to denote a policy.

-Writing could be improved. There are noticeable language errors, e.g., "derivation"-->"deviation", "are in four-fold", "is substitute by", etc.
questions: (1) What do you mean by $p_{j,t}$ and $v_{j,t}$ in proposition 2? Do you mean $p_{j,t}^2$ and $v_{j,t}^2$?
(2) What do you mean by $\mu_s$ in Figure 6?
(3) Does the $\pi_e$ policy mean the SER policy after 7 iterations?
(4) Why do you only show results after three iterations in Table 3?
limitations: No such issues are identified in the paper.
review_title: null

Meta-review: metareview: The authors address the issue of inconsistencies between modeled and training data for auto bidding RL policies and demonstrate the efficacy of their approach both analytically and experimentally. 
confidence: Certain
Decision: Accept