Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The budget constrained coupons allocation problem is an important task in business competition. This paper formulated budget constrained coupons allocation problem as a Lagrangian dual problem and proposed an offline reinforcement learning. The authors have shown solid experiments to validate the performances. 
main_review: This paper formulated the budget constrained coupons allocation problem as a LaGrange dual problem and proposed an offline reinforcement learning, BCORLE(λ) framework. The authors proposed R-BCQ and REME methods which can reduce the computation and improve the performance. Finally, the methods were evaluated on a simulation platform and a real mobile shopping app.

limitations_and_societal_impact: λ-generalization avoids re-learning the policy when the budget changes by extending the variable λ into the transition tuple. However, this method discretizes λ into L values, and enlarges the data set by L times. This may destroy the continuity value space of λ in real world, and the difficulty of training is closely related to L. Also, how to determine the candidate value set of λ in Appendix.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper studied the problem of coupons allocation within a fixed budget, which is usually formulated as a Lagrangian problem. In the existing work, it needs to re-learn the policy once the Lagrangian multiplier is updated, which may incur large computational cost. This work proposed \lambda-generalization method to lead the policy learning process that can be executed according to different lambda values adaptively. Furthermore, for policy learning, resemble batch-constrained Q-learning is proposed, which combines the advantages of two popular offline RL methods: batch-constrained Q-learning and random ensemble mixture. It also conducts numerical experiments to show the performance from four different perspectives.
main_review: Overall, I enjoy reading this paper. The main comments are listed as follows:

1. Why does the discounting factor in the constraint coincide with that in the objective function? It seems that the result will not change if two discounting factors are differentiated.

2. Introducing the Lagrangian multiplier in the objective function, the dimension is much enlarged. How to choose \lambda_1,\cdots, \lambda_L in line 169? What is the guarantee if lambda is discretized to densely distributed values or sparsely distributed values? In Appendix 3, it specifies the optimal coupons allocation policy for lambda when lambda is larger than a threshold but did not specify the choice of gaps.

3. What is \delta in line 149? Is it predetermined?

4. Please explain why the bisection method or gradient descent method can be applied to find the optimal value of lambda (in line 150-151).

5. Please define parameter \theta specifically in the loss function (6).

6. In Section 5.1.3, why choosing \lambda as 0, 0.5, and 1? Is it close to the optimal value?

7. Can you explain more on the conclusion in line 282-289: when lambda is 0, R-BCQ method and REM method generate similar policies with higher AvgCost than that of BCQ method, and their AvgLogins values are also higher. This indicates that R-BCQ method and REM method exhibit better performance because the strategies they learned are more in accordance with the expectation. Why is the conclusion made based on the choice of \lambda=0?



limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper presents an offline reinforcement learning and evaluation framework for coupons allocation in e-commerce. Coupon allocation policy increases user retention but requires a budget. The problem can be formulated into a Lagrangian problem, and existing solutions require updating the policy π when the multiplier variable λ is refined. The proposed algorithm BCORLE(λ) consists of three parts and the first component λ-generation enables policy learning according to different λ. The second component is R-BCO. It decides the optimal policy by confining the search space and optimize the policy learning network. The last component is REME evaluates the policy by employing a multi-head evaluation network. With these components, BCORLE(λ) determines the optimal coupon allocation policy which satisfies the budget constraints. Evaluation is done on offline and online environments in the Taobao Deals app. Results show 1) Efficiency of λ-generation, 2) Superiority of R-BCQ policy selection to currently adopted uplift models and offline RL algorithms, 3) Accurate error estimation by REME, 4) Performance change with λ.
main_review: The coupon allocation problem is formulated as a Constrained Markov Decision Process and then it is well converted into a Langrangian dual problem. In the appendix, the author provides proof of the existence of the solution to the Langrangian dual problem. The proposed BCORLE(λ) is a framework to solve the problem and it has three technical components; λ-generation, R-BCQ, and REME. Each module claims to be new and experiments have shown the efficacy of these modules. 

The coupon allocation framework BCORLE(λ) was first verified in a simulated environment, and after thorough evaluation on the large-scale offline Taobao dataset, it was even deployed in online A/B testings on Taobao Deals. The results show that AvgLogin and ROI increased and AvgCost decreased significantly with proposed components. 

Since I am not an expert in off-line RL, these questions could be simple for you to answer, but I wasn't sure about the answer to these questions even after reading the paper several times.

1)  λ-generation expands the training dataset with different several lambda values and calculates all the corresponding policies. Does this method provide the same optimal policy with λ-update based framework? What can be the potential weakness of the λ-generation method?

2) According to Figure 1, the policy learning and policy evaluation module seem to have similar architecture. Why R-BCQ and REME have to be separated? 

3) What's the difference between the existing and proposed models. How are BCQ and R-BCQ different? The same goes for REM and REME. 

4) It seems to be related to Figure 5. When using R-BCQ and other algorithms, it is not immediately clear how many discount coupons are given to what percentage of people.

Other than that, the manuscript is well written, and code to create and run simulated data was submitted as supplementary material. Therefore, I would like to give a positive evaluation of this paper.
limitations_and_societal_impact: ROI and AvgCost were used as an evaluation method from the viewpoint of maximizing company profits, and AvgLogin was used as an evaluation method in order to increase customer retention. Is there a way to find out if the customer came back because they were really satisfied, or if they came in again simply because they received a coupon?
review_title: null

Meta-review: The paper considers an interesting problem with timely applications. It provides techniques for Constrained Markov Decision Process and then converting to a Langrangian dual problem and provides a proof of the existence of the solution to the Langrangian dual problem (which is natural). The paper applies the results to coupon selection problem which is well motivated. All reviewers are inclined to accept the paper and the paper would be of interest to NeurIPS audience.

Feedback for authors:
There is a related work which is missing and needs to be cited and put in context:
Budget Optimization for Online Campaigns with Positive Carryover Effects, by
Nikolay Archak, V. Mirrokni, S. Muthukrishnan 2012
The above paper also models satisfying budget constraint in a Markov process environments and techniques are relevant.
Decision: Accept (Poster)