SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\boldsymbol {X} _ {*} ^ {t} = \underset {\boldsymbol {X} ^ {t} \in \mathcal {X} ^ {t}} {\arg \max } \sum_ {u \in \mathcal {N}} \sum_ {i \in \mathcal {I}} x _ {u i} ^ {t} v _ {u i}. \tag {7}
$$
We note that the integer program in (7) can be written as an instance of maximum weight bipartite matching. Then, using a variant of the Hungarian algorithm for unbalanced bipartite graphs (Ramshaw and Tarjan, 2012), this problem can be solved in space  $\mathcal{O}(NM)$  and time  $\mathcal{O}(NML)$  in the worst case. (See Appendix A for details.)
Since there is no randomness in user responses, every response from a user  $u$  about item  $i$  gives the provider complete information about a lower or upper bound on  $v_{ui}$ , depending on whether the user response was to accept or reject the offered price for the item  $i$ . For this reason, the algorithm operates by keeping track of intervals  $[a_{ui}, b_{ui}]$  that contain  $v_{ui}$  value for each  $u \in \mathcal{N}$ ,  $i \in \mathcal{I}$  at all rounds. At each time  $t$ , the algorithm chooses which items to offer to each user using these intervals and then determines a price for each offered item. The selection of the allocations is done according to the OFU principle (Dani et al., 2008; Abbasi-Yadkori, 2011) in order to ensure low regret. In particular, the offers are chosen by replacing each  $v_{ui}$  in problem (7) with  $b_{ui}^{t}$  to obtain
$$
\boldsymbol {X} ^ {t} = \underset {\boldsymbol {X} ^ {t} \in \mathcal {X} ^ {t}} {\arg \max } \sum_ {u \in \mathcal {N}} \sum_ {i \in \mathcal {I}} x _ {u i} ^ {t} b _ {u i} ^ {t}. \tag {8}
$$
Having decided on the offers, the next step is to decide on the price of the offered items. Similar to the challenge we encountered in selecting the offers, the selection of prices should also serve two different goals that are in tension with each other. On one hand, learning new information about users' valuations requires us to set prices to values from  $(a_{ui}, b_{ui})$ . On the other hand, to ensure the offers are accepted and generate revenue, we would need to select prices lower than or equal to  $a_{ui}$ . The crucial property that enables us to obtain  $O(NM \log \log (T))$  regret is that the function  $p \to v_{ui} - p \mathbb{1}\{p \leq v_{ui}\}$  is asymmetric and decreases more slowly on the left than on the right as discussed in Kleinberg and Leighton (2003). Based on this observation, we design a pricing algorithm
that operates by offering item  $i$  to user  $u$  at prices increasing with increments of  $\beta_{ui}$ . When the width of the interval  $[a_{ui}, b_{ui}]$  becomes smaller than the precision parameter  $\beta_{ui}$ , we set the new precision parameter as  $\beta_{ui}^2$  and continue exploration with this smaller step size. However, as we show in the proof of Theorem 1, continuing to explore indefinitely would result in a regret linear in  $T$ . To avoid this issue, the algorithm should stop exploration after achieving a certain level of precision and offer item  $i$  to user  $u$  at the maximum price that is certainly acceptable, namely  $a_{ui}$ .
We provide a summary of this algorithm in Algorithm 1 and provide an upper bound for its regret in Theorem 1. As the lower bound provided in Theorem 2 shows, the regret of this algorithm is order-optimal up to smaller terms.
Theorem 1 (Upper bound for Fixed valuations). Assuming fixed valuations in a market with  $N$  users,  $M$  items, and maximum load  $L$ ; the regret of Algorithm 1 in  $T$  rounds satisfies
$$
\mathcal {R} (T, \pi) \leq 2 N M \log \log (L T) + 1.
$$
Theorem 2 (Lower bound for Fixed valuations). If  $N$ ,  $M$ , and  $L \leq M$  are given parameters and  $\pi$  is any randomized policy; there exist randomly generated market instances of  $N$  users with fixed valuations for  $M$  items and maximum load  $L$  such that the expected regret of  $\pi$  in  $T \geq \frac{4NM}{L}$  rounds is  $\Omega(NM\log\log(LT/NM))$ .
Proof. See Appendix B.2.
4.2 Revenue maximization under random experiences
This section focuses on the case where the valuations of the users are given as the average of their past experience. We assume that each user obtains independent experiences about the items that they accept at every round. Then, the users form their valuation at each time as the average of their experiences so far. Formally, we make the following assumption.
Assumption 2. Whenever a user  $u \in \mathcal{N}$  accepts an item  $i \in \mathcal{I}$  at round  $t \in [T]$ , it obtains an independent random experience  $z_{ui}^{t}$  with unknown cdf  $F_{ui}$  over  $[0,1]$  with mean  $v_{ui}$ . Then, the valuation of any user  $u \in \mathcal{N}$  for any item  $i \in \mathcal{I}$  at round  $t \in [T]$  is given as
$$
v _ {u i} ^ {t} = \frac {1}{m _ {u i} ^ {t}} \sum_ {\tau \in \mathcal {T} _ {u i} ^ {t}} z _ {u i} ^ {t}, \tag {9}
$$
where  $\mathcal{T}_{ui}^t = \{\tau < t : i \in \mathcal{A}_u^t\}$  is the set of rounds before round  $t$  at which user  $u$  has accepted and experienced item  $i$ , and  $m_{ui}^t := |\mathcal{T}_{ui}^t|$  denotes the size of this set.
Under the random experiences model, we define the optimum strategy as the one that maximizes the revenue with complete information on possible user experiences  $z_{ui}^{t}$ , and hence user valuations  $v_{ui}^{t}$  at all times. Therefore, the optimum revenue that can be obtained in  $T$  rounds is given by
$$
\mathrm {O P T} = \max  _ {\substack {\boldsymbol {X} ^ {t} \in \mathcal {X} ^ {t}: t \in [ T ] \\ \boldsymbol {p} ^ {t} \in \mathbb {R} _ {+} ^ {M}: t \in [ T ]}} \sum_ {t = 1} ^ {T} \sum_ {u \in \mathcal {N}} \sum_ {i \in \mathcal {I}} x _ {u i} ^ {t} p _ {i} ^ {t} \mathbb {1} \left\{v _ {u i} ^ {t} \geq p _ {i} ^ {t} \right\}, \tag{10}
$$
which is a random variable due to the randomness in  $z_{ui}^{t}$  and hence the randomness in  $v_{ui}^{t}$ . Next, we note that for any fixed sequence of offers  $\{X^t | t \in [T]\}$ , the objective is maximized at prices that satisfy  $p_i^t = \sum_{u \in \mathcal{N}} x_{ui}^t v_{ui}$  at all  $t$ . Therefore, we can write the optimum value for the objective as
$$
\mathrm {O P T} = \max  _ {\boldsymbol {X} ^ {t} \in \mathcal {X} ^ {t}: t \in [ T ]} \sum_ {t = 1} ^ {T} \sum_ {u \in \mathcal {N}} \sum_ {i \in \mathcal {I}} x _ {u i} ^ {t} v _ {u i} ^ {t}. \tag {11}
$$
Note that in problem (11), we need to globally maximize over  $\mathbf{X}^t$  for all rounds  $t \in [T]$  because the values of  $v_{ui}^{t}$  depend on selections of  $\mathbf{X}^t$  at previous rounds. In order to deal with this dependency between the selections of  $\mathbf{X}^t$ , in Lemma 9, we show that OPT is not likely to be much larger than the sum of the mean valuations of the best offers at all rounds.
Algorithm 2 Offerings with scheduled learning
$a_{ui}\gets 0,b_{ui}\gets 1,n_{ui}\gets 0$  , and  $m_{ui}\gets 0$  for all  $u\in \mathcal{N},i\in \mathcal{I}$    
for  $t = 1,2,\ldots ,T$  do Calculate  $\pmb{X}^{t}$  by solving problem (8) using  $b_{ui}$  values for  $(u,i)\in \{(u,i):x_{ui}^{t} = 1\}$  do  $\triangleright$  offer item i to user u if  $\exists j\in \mathbb{N}$  s.t.  $n_{ui} = 2^j$  then  $p_i^t\gets (a_{ui} + b_{ui}) / 2$  else  $p_i^t\gets a_{ui}$  Offer item i to user u at price  $p_i^t$  and observe  $\mathbb{1}\{v_{ui}^{t}\geq p_{i}^{t}\}$ $\gamma \leftarrow \sqrt{8\log(NMT) / m_{ui}}$  if  $v_{ui}^{t}\geq p_{i}^{t}$  then  $a_{ui}\gets a_{ui}\lor (p_i^t -\gamma)$  else  $b_{ui}\gets b_{ui}^{t}\land (p_{i}^{t} + \gamma)$  if  $v_{ui}^{t}\geq p_{i}^{t}$  then  $m_{ui}\gets m_{ui} + 1$ $n_{ui}\gets n_{ui} + 1$  end for   
end for
Next, we describe our algorithm in Algorithm 2. As the users accept and experience the items, their valuations converge to the mean of their experience distribution. Therefore, before extracting the mean valuation information from the users, the provider must ensure that users accept and sufficiently experience the items. Then, to extract this information, the algorithm occasionally asks for higher prices while taking the risk of rejected offers. However, since there are at most  $\log (T)$  such learning rounds per user-item pair, it does not cause any significant loss in revenue.
Based on these observations, we establish an upper bound for our algorithm's regret in Theorem 3. Furthermore, as indicated by the regret lower bound presented in Theorem 4, our algorithm's regret is order-optimal up to smaller terms.
Theorem 3 (Upper bound for random experiences). In any market of  $N$  users satisfying the random experience model given in Assumption 2 for  $M$  items and maximum load  $L$ , with probability  $1 - 2\delta$ , the revenue regret of Algorithm 2 satisfies
$$
\mathcal {R} (T, \pi) = O \left(\sqrt {N M L T \log (N M / \delta)} + N M \log T\right).
$$
Proof. See Appendix C.2.
Theorem 4 (Lower bound for random experiences). If  $N$ ,  $M$ , and  $L \leq M$  are given parameters and  $\pi$  is any randomized policy; there exist randomly generated market instances of  $N$  users satisfying the random experience model given in Assumption 2 for  $M$  items and maximum load  $L$  such that the expected regret of  $\pi$  in  $T \geq N$  rounds is  $\Omega(\sqrt{NMLT})$ .
Proof. See Appendix C.3.
4.3 Revenue maximization under random valuations
In this section, we consider the case where the valuations of the users are given as independent random variables drawn from distributions specific to each user and item. Formally,
Assumption 3. The valuation of any user  $u \in \mathcal{N}$  for any item  $i \in \mathcal{I}$  at round  $t \in [T]$  is given as an independent random variable  $v_{ui}^{t}$  with unknown cdf  $F_{ui}$  over  $[0,1]$ .
Given foreknowledge of the distribution of valuations, but not of the individual valuations at different rounds, it is easy to see what the optimal pricing strategy would be. The expected revenue obtained from offering item  $i$  to user  $u$  at price  $p$  is given as  $\psi_{ui}(p) = p(1 - F_{ui}(p))$ , which we call to be the revenue function. Since valuations are independent over time and their distribution is known, the individual responses provide no useful information about future realizations of the valuations. Therefore, the best price for offering item  $i$  to user  $u$  is given by
$$
p _ {u i} ^ {*} = \underset {p \in \mathbb {R} _ {+}} {\arg \max } p (1 - F _ {u i} (p)). \tag {12}
$$
Thus, letting  $\psi_{ui}^{*} = \psi_{ui}(p_{ui}^{*})$  denote the maximum expected revenue that can be obtained by offering item  $i$  to user  $u$ , an optimum policy that knows the distribution of valuations can obtain revenue
$$
\mathrm {O P T} = \sum_ {t = 1} ^ {T} \left\{\max  _ {\boldsymbol {X} \in \mathcal {X} ^ {t}} \sum_ {u \in \mathcal {N}} \sum_ {i \in \mathcal {I}} x _ {u i} \psi_ {u i} ^ {*} \right\}. \tag {13}
$$
Algorithm 3 Offerings with quantized pricing
$K = (LT / (NM\log (LT)))^{1 / 4},n_{uik}\gets 0,$  and  $\psi_{uik}\gets 1$  for all  $u\in \mathcal{N},i\in \mathcal{I},k\in [K]$    
for  $t = 1,2,\ldots ,T$  do   
for  $(u,i)\in \mathcal{N}\times \mathcal{I}$  do  $b_{uik}\gets (\psi_{uik} + \sqrt{8\log(NMKT) / n_{uik}})\wedge 1,\forall k\in [K]$  compute UCB  $k_{ui}\gets \arg \max_{k}b_{uik}$  and  $b_{ui}\gets \max_{k}b_{uik}$  compute best price levels   
end for   
Calculate  $\pmb{X}^t$  by solving problem (8) using  $b_{ui}$  values.   
for  $(u,i)\in \{(u,i):x_{ui}^{t} = 1\}$  do make offers Offer item i to user u at price  $p_{ui} = k_{ui} / K$  and observe  $z_{ui} = 1\{v_{ui}^{t}\geq p_{ui}\}$ $\psi_{uik}\gets (n_{uik}\psi_{uik} + p_{ui}z_{ui}) / (n_{uik} + 1)$  for  $k = k_{ui}$ $n_{uik}\gets n_{uik} + 1$  for  $k = k_{ui}$    
end for   
end for
However, without the knowledge of the distributions, we are required to learn the expected revenue (i.e. rewards) at different prices via exploration. As previously done in the literature on pricing under random valuation models (Kleinberg and Leighton, 2003), we apply techniques from the literature on the multi-armed bandit problem to develop an algorithm with low regret guarantees. To do so, we quantize the set of possible prices by limiting the provider to strategies that only offer prices belonging to the set  $\{1 / K, 2 / K, \ldots\}$  for suitably chosen  $K$ . This brings us into a setting where each offer of item  $i$  to a user  $u$  at price  $k / K$  yields a revenue which is a random variable taking values in  $[0,1]$ , whose distribution depends on  $(u,i,k)$ , but the rewards for a given action are i.i.d. across the rounds. Therefore, offering item  $i$  to user  $u$  and  $k^{th}$  price level can be represented as pulling an arm that generates revenue with expectation  $\psi_{uik} = \psi_{ui}(k / K)$ . In total, the expected revenue of any offering and pricing is
$$
\sum_ {t = 1} ^ {T} \sum_ {u \in \mathcal {N}} \sum_ {i \in \mathcal {I}} \sum_ {k = 1} ^ {K} x _ {u i k} ^ {t} \psi_ {u i k}, \tag {14}
$$
where  $x_{uik}^{t}$  are binary variables that denote whether user  $u$  is offered item  $i$  at  $k^{th}$  price level at round  $t$ . Due to endowment and demand constraints, these variables at time  $t$  must satisfy conditions (1)  $\sum_{i\in \mathcal{I}}\sum_{k = 1}^{K}x_{uik}\leq d_u^t$ , (2)  $\sum_{u\in \mathcal{N}}\sum_{k = 1}^{K}x_{uik}\leq e_i^t$ , and (3)  $\sum_{k = 1}^{K}x_{uik}\leq 1$  for all  $u\in \mathcal{N}, i\in \mathcal{I}$ . In the literature on combinatorial multi-armed bandits, the standard regret bounds for UCB-based algorithms have an inverse dependency on the gap between the rewards of optimal and suboptimal arms (Kveton et al., 2015). To use similar techniques in proving regret bounds for our algorithm, we make the following hypothesis on the distributions of valuations, which translates directly into bounds on price sub-optimality gaps  $\max_k\psi_{uik} - \psi_{uik}$ .
Assumption 4. The revenue function  $\psi_{ui}(p)$  has a unique global maximum at  $p_{ui}^{*}\in (0,1)$ , and  $\psi_{ui}^{\prime \prime}(p_{ui}^{*})$  is defined and strictly negative.
Next, we show that our algorithm can attain the regret upper bound stated in Theorem 5. Moreover, as illustrated in Theorem 6, our algorithm's regret is order-optimal up to smaller terms.
Theorem 5 (Upper bound for random valuations). In any market of  $N$  users satisfying the random valuation model given in Assumptions 3 and 4 for  $M$  items and maximum load  $L$ , with probability  $1 - \delta$ , the revenue regret of Algorithm 3 satisfies
$$
\mathcal {R} (T, \boldsymbol {\pi}) = \mathcal {O} \left(\sqrt {N M L T \log (L T) \log (N M T / \delta)}\right).
$$
Proof. See Appendix D.2.
Theorem 6 (Lower bound for random valuations). If  $N, M$ , and  $L \leq M$  are given parameters and  $\pi$  is any randomized policy; there exist randomly generated market instances of  $N$  users satisfying the random valuation model given in Assumptions 3 and 4 for  $M$  items and maximum load  $L$  such that the expected regret of  $\pi$  is  $\Omega(\sqrt{NMLT})$  in  $T$  rounds.
Proof. See Appendix D.3.
In this section, we demonstrate the efficacy of our proposed algorithms through a numerical study. We provide our results in Figures 3 and 4. At each round  $t \in [T]$ , the provider is endowed with each item  $i \in \mathcal{I}$  (i.e.  $i \in \mathcal{E}^t$ ) independently with probability 0.5. On the other hand, each user  $u \in \mathcal{N}$  has a random demand  $d_u^t$  with uniform probability over  $\{0,1,2\}$ . For the case of fixed valuations, we choose each  $v_{ui}$  independently from  $\mathrm{Beta}(2,2)$ . For other two models, we set each  $F_{ui}$  as the cdf of  $\mathrm{Beta}(\alpha_{ui},\beta_{ui})$  where  $\alpha_{ui}$  and  $\beta_{ui}$  are uniformly and independently chosen over [1,5].
Figure 3: Instantaneous regret under different valuation models. The darker lines correspond to the mean across 20 experiments with  $N = 150$  users and  $M = 100$  items. The shaded areas indicate the region of error spanning two standard deviations. Results demonstrate the efficacy of our algorithms in achieving diminishing regret as our theoretical results predict.
Figure 4: Regret as a function of time horizon  $T$  under different valuation models. The darker lines correspond to the mean across 5 experiments with  $N = 150$  users and  $M = 100$  items. The shaded areas indicate the region of error spanning two standard deviations. Note that the horizontal axis is logarithmic in the plot for fixed valuations and linear in the plots for the other two cases. Results verify that our algorithms can achieve sub-logarithmic regret under the fixed valuations model, sub-linear regret under random experiences model, and sub-linear regret under random valuations model.
Our study presents a comprehensive solution to maximizing expected revenue in a repeated interaction setting, where a provider seeks to sell multiple items to multiple users. By focusing on different valuation models, we design online learning algorithms that can infer user valuations and offer items to those who value them most to ensure approximately optimal revenue regret. The results of this study have important implications for online marketplaces and can help providers optimize pricing strategies and maximize revenue in a dynamic and uncertain environment.
This work was supported in part by NSF grants CIF-2007669 and CIF-1937357.
Abbasi-Yadkori, Y. (2011). Improved algorithms for linear stochastic bandits. Proceedings of Twenty-Fifth Conference on Neural Information Processing Systems (NeurIPS), pages 2312-2320.
Araman, V. F. and Caldentey, R. (2009). Dynamic pricing for nonperishable products with demand learning. Operations Research, 57(5):1169-1188.
Audibert, J.-Y., Bubeck, S., and Lugosi, G. (2011). Minimax policies for combinatorial prediction games. In Proceedings of the Twenty-Fourth Annual Conference on Learning Theory (COLT), volume 19, pages 107-132.
Kleinberg, R. and Leighton, T. (2003). The value of knowing a demand curve: bounds on regret for online posted-price auctions. In Proceedings of the Forty-Fourth Annual IEEE Symposium on Foundations of Computer Science (FOCS), pages 594-605.
Kveton, B., Wen, Z., Ashkan, A., and Szepesvari, C. (2015). Tight Regret Bounds for Stochastic Combinatorial Semi-Bandits. In Proceedings of the Eighteenth International Conference on Artificial Intelligence and Statistics (AISTATS), volume 38, pages 535-543.
Lavi, R. and Nisan, N. (2000). Competitive analysis of incentive compatible on-line auctions. In Proceedings of the Second ACM Conference on Electronic Commerce.
Liu, L. T., Mania, H., and Jordan, M. I. (2020). Competing bandits in matching markets. In Proceedings of the Twenty-Third International Conference on Artificial Intelligence and Statistics (AISTATS), pages 1618–1628.
Morgenstern, J. and Roughgarden, T. (2016). Learning simple auctions. In Proceedings of the 29th Annual Conference on Learning Theory (COLT), pages 1298-1318.
Paes Leme, R. and Schneider, J. (2018). Contextual search via intrinsic volumes. In Proceedings of the Fifty-Ninth Annual IEEE Symposium on Foundations of Computer Science (FOCS), pages 268-282.
Ramshaw, L. and Tarjan, R. E. (2012). On minimum-cost assignments in unbalanced bipartite graphs. HP Labs, Palo Alto, CA, USA, Tech. Rep. HPL-2012-40R1, 20.
Rothschild, M. (1974). A two-armed bandit theory of market pricing. Journal of Economic Theory, 9(2):185-202.
Xu, J. and Wang, Y.-X. (2021). Logarithmic regret in feature-based dynamic pricing. In Proceedings of Thirty-Fifth Conference on Neural Information Processing Systems (NeurIPS), volume 34, pages 13898-13910.
A Algorithmic Details for Solving the Integer Program
The integer linear program in (7) can be written as an instance of maximum weight bipartite matching. In this graph, we represent each user  $u$  with  $d_u^t$  left nodes and we represent each endowed item  $i$  with a right node. Then, we construct the complete weighted bipartite graph where the weight of an edge between a node for user  $u$  and a node for item  $i$  is given as  $v_{ui}$ . In total, this graph has  $D_t$  left vertices,  $E_t$  right vertices, and  $D_tE_t$  weighted edges where  $D_t$  and  $E_t$  correspond to total demand and endowment at time  $t$  (as given in Definition 2).
Then, using a variant of the Hungarian algorithm for unbalanced bipartite graphs (Ramshaw and Tarjan, 2012), this problem can be solved in space  $\mathcal{O}(D_tE_t)$  and time  $\mathcal{O}(D_tE_t\min \{D_t,E_t\})$
Since we can upper bound  $D_{t} \leq N$ ,  $E_{t} \leq M$ , and  $\min \{D_t, E_t\} \leq L$ , we conclude that the algorithm runs in space  $\mathcal{O}(NM)$  and time  $\mathcal{O}(NML)$  in the worst case.
B Proofs for Revenue Maximization with Fixed Valuations
Letting  $\mathrm{OPT}_t$  denote the optimum revenue at each time  $t$  and  $R_{t}$  denote the regret at round  $t$ ,
$$
\begin{array}{l} \mathcal {R} _ {t} = \mathrm {O P T} _ {t} - \sum_ {u \in \mathcal {N}} \sum_ {i \in \mathcal {I}} x _ {u i} ^ {t} p _ {i} ^ {t} \mathbb {1} \left\{v _ {u i} \geq p _ {i} ^ {t} \right\} \\ = \max  _ {X \in \mathcal {X} ^ {t}} \left\{\sum_ {u \in \mathcal {N}} \sum_ {i \in \mathcal {I}} x _ {u i} v _ {u i} \right\} - \sum_ {u \in \mathcal {N}} \sum_ {i \in \mathcal {I}} x _ {u i} ^ {t} p _ {i} ^ {t} \mathbb {1} \left\{v _ {u i} \geq p _ {i} ^ {t} \right\} \\ \leq \max  _ {X \in \mathcal {X} ^ {t}} \left\{\sum_ {u \in \mathcal {N}} \sum_ {i \in \mathcal {I}} x _ {u i} b _ {u i} ^ {t} \right\} - \sum_ {u \in \mathcal {N}} \sum_ {i \in \mathcal {I}} x _ {u i} ^ {t} a _ {u i} ^ {t} \mathbb {1} \{v _ {u i} \geq p _ {i} ^ {t} \} \\ = \sum_ {u \in \mathcal {N}} \sum_ {i \in \mathcal {I}} x _ {u i} ^ {t} b _ {u i} ^ {t} - \sum_ {u \in \mathcal {N}} \sum_ {i \in \mathcal {I}} x _ {u i} ^ {t} a _ {u i} ^ {t} \mathbb {1} \left\{v _ {u i} \geq p _ {i} ^ {t} \right\} \\ = \sum_ {u \in \mathcal {N}} \sum_ {i \in \mathcal {I}} x _ {u i} ^ {t} \left(b _ {u i} ^ {t} - a _ {u i} ^ {t} \mathbb {1} \left\{v _ {u i} \geq p _ {i} ^ {t} \right\}\right). \\ \end{array}
$$
Then, we sum over all  $1 \leq t \leq T$  to have
$$
\begin{array}{l} \mathcal {R} (T, \boldsymbol {\pi}) = \sum_ {t = 1} ^ {T} \mathcal {R} _ {t} \\ \leq \sum_ {t = 1} ^ {T} \sum_ {u \in \mathcal {N}} \sum_ {i \in \mathcal {I}} x _ {u i} ^ {t} \left(b _ {u i} ^ {t} - a _ {u i} ^ {t} \mathbb {1} \left\{v _ {u i} \geq p _ {i} ^ {t} \right\}\right) \\ = \sum_ {u \in \mathcal {N}} \sum_ {i \in \mathcal {I}} \sum_ {t \in \mathcal {T} _ {u i}} \left(b _ {u i} ^ {t} - a _ {u i} ^ {t} \mathbb {1} \left\{v _ {u i} \geq p _ {i} ^ {t} \right\}\right), \\ \end{array}
$$
where  $\mathcal{T}_{ui} = \{t:1\leq t\leq T,x_{ui}^{t} = 1\}$  denotes the time indices where  $(u,i)$  is offered. Now, we let
$$
\bar {\mathcal {R}} _ {u i} = \sum_ {t \in \mathcal {T} _ {u i}} \left(b _ {u i} ^ {t} - a _ {u i} ^ {t} \mathbb {1} \left\{v _ {u i} \geq p _ {i} ^ {t} \right\}\right)
$$
be an upper bound for the total regret incurred from offering the pair  $(u,i)$  such that we have  $\mathcal{R}(T,\pi)\leq \sum_{u\in \mathcal{N}}\sum_{i\in \mathcal{I}}\overline{\mathcal{R}}_{ui}$ . Now, let  $k$ th learning epoch for pair  $(u,i)$  correspond to the time indices in which  $\beta_{ui}^{t} = (1 / 2)^{k}$  and the pair  $(u,i)$  is offered. That is,
$$
\mathcal {T} _ {u i} ^ {k} = \{t \in \mathcal {T} _ {u i}: \beta_ {u i} ^ {t} = (1 / 2) ^ {k} \}.
$$
Note that we have  $b_{ui}^{t} - a_{ui}^{t} \leq \beta_{ui}^{t} = (1/2)^{k}$  for each  $t \in \mathcal{T}_{ui}^{k}$ . During the learning phase, each epoch  $\mathcal{T}_{ui}^{k}$  ends either when the offer  $(u,i)$  is rejected or the offer  $(u,i)$  is accepted  $2^{k} - 1$  times in a row. Therefore, in epoch  $k$ , there are at most one rejection and  $2^{k} - 1$  acceptances. As a result, the regret incurred by offers  $(u,i)$  during each  $\mathcal{T}_{ui}^{k}$  is upper bounded as
$$
\sum_ {t \in \mathcal {T} _ {u i} ^ {k}} \left(b _ {u i} ^ {t} - a _ {u i} ^ {t} \mathbb {1} \left\{v _ {u i} \geq p _ {i} ^ {t} \right\}\right) \leq 1 + \left(2 ^ {k} - 1\right) \left(1 / 2\right) ^ {k} \leq 2. \tag {15}
$$