SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Note that a learning epoch  $\mathcal{T}_{ui}^k$  can last for anywhere between 1 to  $2^{k}$  rounds in total. Therefore, if we were to continue exploration indefinitely, the number of learning epochs could be as large as  $T$  in the worst case. Consequently, continuing to explore indefinitely would result in linear regret. To avoid this issue, the algorithm should stop exploration after achieving a certain level of precision and offer item  $i$  to user  $u$  at the maximum price that is certainly acceptable, namely  $a_{ui}$ . We let this precision level be  $\epsilon$  and make the algorithm choose price  $a_{ui}$  when  $b_{ui}^t - a_{ui}^t \leq \epsilon$ . Based on this stopping rule,  $K_{ui} = \lceil \log \log (1 / \epsilon) \rceil$  becomes the last epoch for  $(u,i)$  and therefore we run at most  $K_{ui} - 1$  learning epochs before the last epoch.
Since the offers are always accepted in this last epoch, we incur at most  $\epsilon$  regret at each round of it. Consequently, we have
$$
\begin{array}{l} \bar {\mathcal {R}} _ {u i} \leq 2 \log \log (1 / \epsilon) + \epsilon | \mathcal {T} _ {u i} ^ {K _ {u i}} | \\ \leq 2 \log \log (1 / \epsilon) + \epsilon | T _ {u i} |. \\ \end{array}
$$
Since  $X^t$  includes at most  $L$  offers, we also have  $\sum_{u\in \mathcal{N}}\sum_{i\in \mathcal{I}}|\mathcal{T}_{ui}|\leq LT$ . Thus,
$$
\mathcal {R} (T, \pi) \leq 2 N M \log \log (1 / \epsilon) + \epsilon L T. \tag {16}
$$
Letting  $\epsilon = 1 / (LT)$ , we obtain
$$
\mathcal {R} (T, \pi) \leq 2 N M \log \log (L T) + 1. \tag {17}
$$
Consider a market with users  $\mathcal{N} = [N]$  where the demands of users are given as
$$
d _ {u} ^ {t} = \left\{ \begin{array}{l l} L, & \text {i f} \quad t = (u - 1) \mod N \\ 0, & \text {o t h e r w i s e .} \end{array} \right.
$$
so that only one user has nonzero demand in each round. Next, let  $M'$  be the smallest prime number larger than or equal to  $M$ . If  $N = M'$ , let  $M'$  be the next smallest prime number. By the prime number theorem, we always have  $M' \leq 4M$ , and  $N$  and  $M'$  are always co-prime. Then, consider a set of items  $\mathcal{I} = [M']$  such that the valuation of all users for the first  $M$  items are uniformly and independently chosen over  $[0,1]$  while the last  $M' - M$  items are artificial items that have zero valuation for all users. Next, assume that the set of available items  $\mathcal{E}^t$  at each time  $t$  is given according to
$$
i \in \mathcal {E} ^ {t} \quad \Longleftrightarrow \quad i - t + 1 \in \{1, 2, \dots , L \} \mod M ^ {\prime}
$$
such that at most  $L$  of the items are available at each round, and hence the maximum load parameter in this market is equal to  $L$ . By construction, the optimum offering pattern at each time is to offer all available items to a single user that has non-zero demand. Hence, the problem of the provider reduces to only learning the price at which it should offer each item. There are  $NM$  actual user-item pairs and each item  $i \in [M]$  is offered to user  $u \in [N]$  for at least  $\lfloor LT / NM' \rfloor \geq \lfloor LT / 4NM \rfloor \geq 1$  rounds. In the literature on pricing optimization, each pricing problem is known to have  $\Omega (\log \log (T_o))$  regret in  $T_{o}$  rounds (Kleinberg and Leighton, 2003). Therefore, any policy must have  $\Omega (NM\log \log (LT / NM))$  regret in total.
C Proofs for Revenue Maximization with Random Experiences
C.1 Preliminary Lemmas
Lemma 7. With probability  $1 - \delta$
$$
\left| v _ {u i} ^ {t} - v _ {u i} \right| \leq \sqrt {\frac {8 \log (N M / \delta)}{m _ {u i} ^ {t}}}
$$
for all  $(u,i)\in \mathcal{N}\times \mathcal{I}$  and for all  $t\in \mathbb{N}$
Proof. Let  $\mathcal{H}_{t-1}$  be the  $\sigma$ -algebra generated by  $(H_t, X^t, p^t)$  and let  $\mathcal{H}_0 = \sigma(\emptyset, \Omega)$ . Fix some  $(u, i)$  and define  $\epsilon_{ui}^t := z_{ui}^t - v_{ui}$  for all  $t \in \mathbb{N}$ . By previous assumptions,  $\mathbb{E}[\epsilon_{ui}^t | \mathcal{H}_{t-1}] = 0$  and  $\mathbb{E}[\exp(\lambda \epsilon_{ui}^t) | \mathcal{H}_{t-1}] \leq \exp(\lambda^2 / 2)$  for all  $t \in \mathbb{N}$ .
Define  $\delta_{ui}^{t} := [(z_{ui}^{t} - v_{ui})^{2} - (z_{ui}^{t} - v_{ui}^{t})^{2}] \mathbb{1}\{x_{ui}^{t} = 1\}$ . Then, we have
$$
\delta_ {u i} ^ {t} = \left[ - \left(v _ {u i} ^ {t} - v _ {u i}\right) ^ {2} + 2 \epsilon_ {u i} ^ {t} \left(v _ {u i} ^ {t} - v _ {u i}\right) \right] \mathbb {1} \{x _ {u i} ^ {t} = 1 \}.
$$
Therefore, the conditional mean and conditional cumulant generating function satisfy
$$
\mu_ {u i} ^ {t} := \mathbb {E} [ \delta_ {u i} ^ {t} | \mathcal {H} _ {t - 1} ] = - (v _ {u i} ^ {t} - v _ {u i}) ^ {2} \mathbb {1} \{x _ {u i} ^ {t} = 1 \}
$$
$$
\begin{array}{l} \psi_ {u i} ^ {t} (\lambda) := \log \mathbb {E} \left[ \exp \left(\lambda \left[ \delta_ {u i} ^ {t} - \mu_ {u i} ^ {t} \right]\right) \mid \mathcal {H} _ {t - 1} \right] \\ = \log \mathbb {E} [ \exp (2 \lambda \epsilon_ {u i} ^ {t} (v _ {u i} ^ {t} - v _ {u i})) | \mathcal {H} _ {t - 1} ] \mathbb {1} \{x _ {u i} ^ {t} = 1 \} \\ \leq \frac {(2 \lambda (v _ {u i} ^ {t} - v _ {u i})) ^ {2}}{2} \mathbb {1} \left\{x _ {u i} ^ {t} = 1 \right\}. \\ \end{array}
$$
Applying Lemma 14, we have, for all  $x \geq 0$  and  $\lambda \geq 0$ ,
$$
\mathbb {P} \left(\sum_ {\tau = 1} ^ {t - 1} \delta_ {u i} ^ {\tau} \leq \frac {x}{\lambda} + \sum_ {\tau = 1} ^ {t - 1} (v _ {u i} ^ {t} - v _ {u i}) ^ {2} (2 \lambda - 1) \mathbb {1} \{x _ {u i} ^ {\tau} = 1 \}, \forall t \in \mathbb {N}\right) \geq 1 - e ^ {- x}.
$$
$$
\mathbb {P} \left(\sum_ {\tau \in \mathcal {T} _ {u i} ^ {t}} \left[ \left(z _ {u i} ^ {\tau} - v _ {u i}\right) ^ {2} - \left(z _ {u i} ^ {\tau} - v _ {u i} ^ {t}\right) ^ {2} \right] \leq \frac {x}{\lambda} + \sum_ {\tau \in \mathcal {T} _ {u i} ^ {t}} \left(v _ {u i} ^ {t} - v _ {u i}\right) ^ {2} (2 \lambda - 1), \forall t \in \mathbb {N}\right) \geq 1 - e ^ {- x}.
$$
Noting that  $v_{ui}^{t} = \frac{1}{m_{ui}^{t}}\sum_{\tau \in \mathcal{T}_{ui}^{t}}z_{ui}^{\tau}$ , we have  $\sum_{\tau \in \mathcal{T}_{ui}^{t}}[(z_{ui}^{\tau} - v_{ui})^{2} - (z_{ui}^{\tau} - v_{ui}^{t})^{2}] \geq 0$  for all  $t$ . Then, choosing  $\lambda = \frac{1}{4}$  and  $x = \log \frac{1}{\delta}$  gives
$$
\mathbb {P} \left(\sum_ {\tau \in T _ {u i} (t)} \left(v _ {u i} ^ {t} - v _ {u i}\right) ^ {2} \leq 8 \log (1 / \delta), \forall t \in \mathbb {N}\right) \geq 1 - \delta ,
$$
$$
\mathbb {P} \left(\left| v _ {u i} ^ {t} - v _ {u i} \right| \leq \sqrt {\frac {8 \log (1 / \delta)}{m _ {u i} ^ {t}}} \quad , \forall t \in \mathbb {N}\right) \geq 1 - \delta .
$$
By applying a union bound over all  $(u,i)$  pairs, we have
$$
\mathbb {P} \left(| v _ {u i} ^ {t} - v _ {u i} | \leq \sqrt {\frac {8 \log (N M / \delta)}{m _ {u i} ^ {t}}} \quad , \forall (u, i) \in \mathcal {N} \times \mathcal {I}, \forall t \in \mathbb {N}\right) \geq 1 - \delta .
$$
Lemma 8. With probability  $1 - \delta$
$$
v _ {u i} \in \left[ a _ {u i} ^ {t}, b _ {u i} ^ {t} \right]
$$
Proof. From Lemma 7, we have  $|v_{ui}^t - v_{ui}| \leq \sqrt{8\log(NM / \delta) / m_{ui}^t}$  for all  $(u,i) \in \mathcal{N} \times \mathcal{I}$  and for all  $t \in [T]$  with probability  $1 - \delta$ . Assume that this condition holds true.
We prove the statement of the lemma by induction. Assume  $v_{ui} \in [a_{ui}^t, b_{ui}^t]$  as the inductive hypothesis. Then, if user  $u$  accepts item  $i$  at price  $p_i^t$ , then we have  $v_{ui}^t \geq p_i^t$ . Therefore,  $p_i^t - v_{ui} \leq v_{ui}^t - v_{ui} \leq \sqrt{8\log(NM / \delta) / m_{ui}^t}$ . So,  $a_{ui}^{t + 1} = a_{ui}^t \vee (p_i^t - \sqrt{8\log(NM / \delta) / m_{ui}^t}) \leq v_{ui}$ . Similarly, we can also show  $b_{ui}^{t + 1} = b_{ui}^t \wedge (p_i^t + \sqrt{8\log(NM / \delta) / m_{ui}^t}) \geq v_{ui}$ . Therefore,  $v_{ui}^{t + 1} \in [a_{ui}^{t + 1}, b_{ui}^{t + 1}]$ .
Lemma 9. With probability  $1 - \delta$ , the value of OPT satisfies
$$
\operatorname {O P T} - \sum_ {t = 1} ^ {T} \left\{\max  _ {\boldsymbol {X} \in \mathcal {X} ^ {t}} \sum_ {u \in \mathcal {N}} \sum_ {i \in \mathcal {I}} x _ {u i} v _ {u i} \right\} \leq O \left(\sqrt {N M L T \log (N M / \delta)}\right). \tag {18}
$$
Proof. Let us define
$$
\widetilde {\mathrm {O P T}} = \sum_ {t = 1} ^ {T} \left\{\max  _ {\boldsymbol {X} \in \mathcal {X} ^ {t}} \sum_ {u \in \mathcal {N}} \sum_ {i \in \mathcal {I}} x _ {u i} v _ {u i} \right\}. \tag {19}
$$
Then, using the definition of OPT and  $\widetilde{\mathrm{OPT}}$  together with Lemma 7, we have
$$
\begin{array}{l} \mathrm {O P T} = \max  _ {\boldsymbol {X} ^ {t} \in \mathcal {X} ^ {t}: t \in [ T ]} \sum_ {t = 1} ^ {T} \sum_ {u \in \mathcal {N}} \sum_ {i \in \mathcal {I}} x _ {u i} ^ {t} v _ {u i} ^ {t} \\ = \max _ {\boldsymbol {X} ^ {t} \in \mathcal {X} ^ {t}: t \in [ T ]} \left\{\sum_ {t = 1} ^ {T} \sum_ {u \in \mathcal {N}} \sum_ {i \in \mathcal {I}} x _ {u i} ^ {t} v _ {u i} + \sum_ {t = 1} ^ {T} \sum_ {u \in \mathcal {N}} \sum_ {i \in \mathcal {I}} x _ {u i} ^ {t} (v _ {u i} ^ {t} - v _ {u i}) \right\} \\ \leq \max _ {\pmb {X} ^ {t} \in \mathcal {X} ^ {t}: t \in [ T ]} \left\{\sum_ {t = 1} ^ {T} \sum_ {u \in \mathcal {N}} \sum_ {i \in \mathcal {I}} x _ {u i} ^ {t} v _ {u i} \right\} + \max _ {\pmb {X} ^ {t} \in \mathcal {X} ^ {t}: t \in [ T ]} \left\{\sum_ {t = 1} ^ {T} \sum_ {u \in \mathcal {N}} \sum_ {i \in \mathcal {I}} x _ {u i} ^ {t} (v _ {u i} ^ {t} - v _ {u i}) \right\} \\ = \widetilde {\mathrm {O P T}} + \max _ {\boldsymbol {X} ^ {t} \in \mathcal {X} ^ {t}: t \in [ T ]} \left\{\sum_ {t = 1} ^ {T} \sum_ {u \in \mathcal {N}} \sum_ {i \in \mathcal {I}} x _ {u i} ^ {t} (v _ {u i} ^ {t} - v _ {u i}) \right\} \\ \leq \widetilde {\mathrm {O P T}} + \max  _ {\boldsymbol {X} ^ {t} \in \mathcal {X} ^ {t}: t \in [ T ]} \left\{\sum_ {t = 1} ^ {T} \sum_ {u \in \mathcal {N}} \sum_ {i \in \mathcal {I}} x _ {u i} ^ {t} \sqrt {\frac {8 \log (N M / \delta)}{m _ {u i} ^ {t}}} \right\} \\ \leq \widetilde {\mathrm {O P T}} + \sqrt {8 \log (N M / \delta)} \max  _ {\boldsymbol {X} ^ {t} \in \mathcal {X} ^ {t}: t \in [ T ]} \left\{\sum_ {t = 1} ^ {T} \sum_ {u \in \mathcal {N}} \sum_ {i \in \mathcal {I}} x _ {u i} ^ {t} \sqrt {1 / m _ {u i} ^ {t}} \right\}. \\ \end{array}
$$
Recall that  $n_{ui}^{t}$  counts the number of times user  $u$  is offered item  $i$  and  $m_{ui}^{t}$  counts the number of corresponding acceptances. Since an item  $i$  can be rejected only when  $n_{ui}^{t} = 2^{j}$  for some  $j \in \mathbb{N}$ , we can write
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \sum_ {u \in \mathcal {N}} \sum_ {i \in \mathcal {I}} x _ {u i} ^ {t} \sqrt {\frac {1}{m _ {u i} ^ {t}}} = \sum_ {u \in \mathcal {N}} \sum_ {i \in \mathcal {I}} \sum_ {t: x _ {u i} ^ {t} = 1} \sqrt {\frac {1}{m _ {u i} ^ {t}}} \\ \leq \sum_{u\in \mathcal{N}}\sum_{i\in \mathcal{I}}\left(\sum_{\substack{t:x_{ui}^{t} = 1\\ \not\nexists j\in \mathbb{N}:n_{ui}^{t} = 2^{j}}}\sqrt{\frac{1}{m_{ui}^{t}}} +\sum_{\substack{t:x_{ui}^{t} = 1\\ \exists j\in \mathbb{N}:n_{ui}^{t} = 2^{j}}}\sqrt{\frac{1}{m_{ui}^{t}}}\right) \\ \leq \sum_ {u \in \mathcal {N}} \sum_ {i \in \mathcal {I}} \left(\sum_ {k = 1} ^ {m _ {u i} ^ {T}} \sqrt {\frac {1}{k}} + \log T\right) \\ \leq \sum_ {u \in \mathcal {N}} \sum_ {i \in \mathcal {I}} \left(2 \sqrt {m _ {u i} ^ {T}} + \log T\right) \\ \end{array}
$$
where we upper bound the first term inside the parentheses using  $\sum_{k=1}^{n} \frac{1}{\sqrt{k}} \leq \int_{0}^{n} \frac{\mathrm{d}x}{\sqrt{x}} = 2\sqrt{n}$  and we upper bound the second term by noting that it can have at most  $\log T$  terms. Also note that the total number of offers over all time intervals is upper bounded by  $LT$ , and therefore  $\sum_{u \in \mathcal{N}} \sum_{i \in \mathcal{I}} m_{ui}^T \leq LT$ . Then, using the Cauchy-Schwartz inequality, we can write
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \sum_ {u \in \mathcal {N}} \sum_ {i \in \mathcal {I}} x _ {u i} ^ {t} \sqrt {\frac {1}{m _ {u i} ^ {t}}} \leq 2 \sqrt {N M \sum_ {u \in \mathcal {N}} \sum_ {i \in \mathcal {I}} m _ {u i} ^ {T}} + N M \log T \\ \leq O (\sqrt {N M L T} + N M \log T). \\ \end{array}
$$
Putting all together, we conclude the proof of the lemma.
By Lemma 8, the confidence bounds include the mean, i.e.,  $v_{ui} \in [a_{ui}^{t}, b_{ui}^{t}]$  with probability  $1 - \delta$ . Assume that this condition holds.
Then, the offered items are accepted whenever there does not exist  $j \in \mathbb{N}$  such that  $n_{ui}^{t} = 2^{j}$ . So, we have that  $m_{ui}^{t} \geq n_{ui}^{t} / 2$  because  $n_{ui}^{t}$  counts the number of times item  $i$  is offered to user  $u$ , and  $m_{ui}^{t}$  counts the number of times item  $i$  is accepted by user  $u$ .
Furthermore, we recall that intervals are only updated at time step  $t$  if  $x_{ui}^{t} = 1$  and  $n_{ui}^{t} = 2^{j}$  for some  $j \in \mathbb{N}$ . Therefore, we can show an upper bound for the width of the intervals  $[a_{ui}^{t}, b_{ui}^{t}]$ . Define the width  $g_{ui}^{t} = b_{ui}^{t} - a_{ui}^{t}$ . When the update happens, the interval's size  $g_{ui}^{t}$  becomes at most  $g_{ui}^{t-1}/2 + \sqrt{8\log(NM/\delta)/m_{ui}^{t}} \leq g_{ui}^{t-1}/2 + \sqrt{16\log(NM/\delta)/n_{ui}^{t}}$ . Therefore,
$$
g _ {u i} ^ {t} \leq \sum_ {j = 1} ^ {K} \left(\frac {1}{2}\right) ^ {K - j} \sqrt {\frac {1 6 \log (N M / \delta)}{2 ^ {j + 1}}},
$$
where  $K = \log_2 n_{ui}^t$  is the number of times the interval is updated before time  $t$ . Then,
$$
\begin{array}{l} g _ {u i} ^ {t} \leq \frac {\sqrt {1 6 \log (N M / \delta)}}{2 ^ {K}} \sum_ {j = 1} ^ {K} \left(\sqrt {2}\right) ^ {j - 1} \\ \leq \frac {\sqrt {1 6 \log (N M / \delta)}}{2 ^ {K}} (2 ^ {K / 2} - 1) (1 + \sqrt {2}) \\ \leq \sqrt {\frac {1 4 4 \log (N M / \delta)}{2 ^ {K}}} \\ \leq \sqrt {\frac {1 4 4 \log (N M / \delta)}{2 ^ {\log n _ {u i} ^ {t}} - 1}} \\ \leq \sqrt {\frac {2 8 8 \log (N M / \delta)}{n _ {u i} ^ {t}}}. \\ \end{array}
$$
Now, let  $L_{ui}^{t} = \{x_{ui}^{t} = 1, \exists j \in \mathbb{N}, n_{ui}^{t} = 2^{j}\}$  denote the learning event for pair  $(u,i)$  at round  $t$ . Also note that  $x_{ui}^{t} = 1$  at some  $t$ , then  $p_i^t = \mathbb{1}\{\neg L_{ui}^t\} a_{ui}^t + \mathbb{1}\{L_{ui}^t\} (a_{ui}^t + b_{ui}^t) / 2$ . Next, we recall the definition of  $\widehat{\mathrm{OPT}}$  in (19) and let
$$
\boldsymbol {Y} ^ {t} = \underset {\boldsymbol {X} \in \mathcal {X} ^ {t}} {\arg \max } \sum_ {u \in \mathcal {N}} \sum_ {i \in \mathcal {I}} x _ {u i} v _ {u i}, \tag {20}
$$
with entries  $y_{ui}^{t}$ . Then, the regret with respect to  $\widetilde{\mathrm{OPT}}$  is given as the difference
$$
\begin{array}{l} \widetilde {\mathcal {R}} (T, \boldsymbol {\pi}) := \widetilde {\mathrm {O P T}} - \sum_ {t = 1} ^ {T} \sum_ {u \in \mathcal {N}} \sum_ {i \in \mathcal {I}} x _ {u i} ^ {t} p _ {i} ^ {t} \mathbb {1} \left\{v _ {u i} ^ {t} \geq p _ {i} ^ {t} \right\} \\ = \sum_ {t = 1} ^ {T} \left\{\sum_ {u \in \mathcal {N}} \sum_ {i \in \mathcal {I}} y _ {u i} ^ {t} v _ {u i} - \sum_ {u \in \mathcal {N}} \sum_ {i \in \mathcal {I}} x _ {u i} ^ {t} p _ {i} ^ {t} \mathbb {1} \{v _ {u i} ^ {t} \geq p _ {i} ^ {t} \} \right\}. \\ \end{array}
$$
Recalling that the confidence bounds include the mean, i.e.,  $v_{ui} \in [a_{ui}^{t}, b_{ui}^{t}]$ ,
$$
\begin{array}{l} \widetilde {\mathcal {R}} (T, \boldsymbol {\pi}) \leq \sum_ {t = 1} ^ {T} \left\{\sum_ {u \in \mathcal {N}} \sum_ {i \in \mathcal {I}} y _ {u i} ^ {t} v _ {u i} - \sum_ {u \in \mathcal {N}} \sum_ {i \in \mathcal {I}} x _ {u i} ^ {t} a _ {u i} ^ {t} \mathbb {1} \left\{\neg L _ {u i} ^ {t} \right\} \right\} \\ \leq \sum_ {t = 1} ^ {T} \left\{\sum_ {u \in \mathcal {N}} \sum_ {i \in \mathcal {I}} y _ {u i} ^ {t} v _ {u i} - \sum_ {u \in \mathcal {N}} \sum_ {i \in \mathcal {I}} x _ {u i} ^ {t} a _ {u i} ^ {t} + \sum_ {u \in \mathcal {N}} \sum_ {i \in \mathcal {I}} x _ {u i} ^ {t} a _ {u i} ^ {t} \mathbb {1} \left\{L _ {u i} ^ {t} \right\} \right\} \\ \leq \sum_ {t = 1} ^ {T} \left\{\sum_ {u \in \mathcal {N}} \sum_ {i \in \mathcal {I}} y _ {u i} ^ {t} v _ {u i} - \sum_ {u \in \mathcal {N}} \sum_ {i \in \mathcal {I}} x _ {u i} ^ {t} a _ {u i} ^ {t} \right\} + \sum_ {t = 1} ^ {T} \sum_ {u \in \mathcal {N}} \sum_ {i \in \mathcal {I}} \mathbb {1} \{L _ {u i} ^ {t} \}. \\ \end{array}
$$
Since  $\pmb{X}^t$  is selected according to
$$
\boldsymbol {X} ^ {t} = \underset {\boldsymbol {X} \in \mathcal {X}} {\arg \max } \sum_ {u \in \mathcal {N}} \sum_ {i \in \mathcal {I}} x _ {u i} b _ {u i} ^ {t}, \tag {21}
$$
we can upper bound the first term as
$$
\begin{array}{l} \sum_ {u \in \mathcal {N}} \sum_ {i \in \mathcal {I}} \left(y _ {u i} ^ {t} v _ {u i} - x _ {u i} ^ {t} a _ {u i} ^ {t}\right) \leq \sum_ {u \in \mathcal {N}} \sum_ {i \in \mathcal {I}} \left(y _ {u i} ^ {t} b _ {u i} ^ {t} - x _ {u i} ^ {t} a _ {u i} ^ {t}\right) \\ \leq \sum_ {u \in \mathcal {N}} \sum_ {i \in \mathcal {I}} \left(x _ {u i} ^ {t} b _ {u i} ^ {t} - x _ {u i} ^ {t} a _ {u i} ^ {t}\right) \\ = \sum_ {u \in \mathcal {N}} \sum_ {i \in \mathcal {I}} x _ {u i} ^ {t} g _ {u i} ^ {t}. \\ \end{array}
$$
On the other hand, the second term is upper bounded as
$$
\sum_ {t = 1} ^ {T} \sum_ {u \in \mathcal {N}} \sum_ {i \in \mathcal {I}} \mathbb {1} \left\{L _ {u i} ^ {t} \right\} \leq N M \log T.
$$
Therefore, we can upper bound  $\widetilde{\mathcal{R}}(T, \pi)$  as
$$
\widetilde {\mathcal {R}} (T, \boldsymbol {\pi}) \leq \sum_ {t = 1} ^ {T} \sum_ {u \in \mathcal {N}} \sum_ {i \in \mathcal {I}} x _ {u i} ^ {t} g _ {u i} ^ {t} + N M \log T.
$$
Then, we can show
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \sum_ {u \in \mathcal {N}} \sum_ {i \in \mathcal {I}} x _ {u i} ^ {t} g _ {u i} ^ {t} \leq \sum_ {t = 1} ^ {T} \sum_ {u \in \mathcal {N}} \sum_ {i \in \mathcal {I}} x _ {u i} ^ {t} \sqrt {\frac {2 8 8 \log (N M / \delta)}{n _ {u i} ^ {t}}} \\ = \sqrt {2 8 8 \log (N M / \delta)} \sum_ {t = 1} ^ {T} \sum_ {u \in \mathcal {N}} \sum_ {i \in \mathcal {I}} x _ {u i} ^ {t} \sqrt {1 / n _ {u i} ^ {t}} \\ \leq O (\sqrt {N M L T \log (N M / \delta)}), \\ \end{array}
$$
where the proof of the last step is similar to the proof of Lemma 9. Therefore, with probability  $1 - \delta$
$$
\widetilde {\mathcal {R}} (T, \boldsymbol {\pi}) \leq O (\sqrt {N M L T \log (N M / \delta)} + N M \log T).
$$
Then, we use Lemma 9 which holds with probability  $1 - \delta$ . Therefore, by applying a union bound, we can show that
$$
\begin{array}{l} \mathcal {R} (T, \pi) = \mathrm {O P T} - \widetilde {\mathrm {O P T}} + \widetilde {\mathcal {R}} (T, \pi) \\ \leq O (\sqrt {N M L T \log (N M / \delta)} + N M \log T), \\ \end{array}
$$
Recall that our observations are limited to accept-reject signals given by indicator variables  $\mathbb{1}\{v_{ui}^t\geq p_i\}$  for all  $(u,i)$  such that  $x_{ui}^{t} = 1$  . Furthermore, the valuations are given by
$$
v _ {u i} ^ {t} = \frac {1}{n _ {u i} ^ {t}} \sum_ {\tau \in \mathcal {T} _ {u i} ^ {t}} z _ {u i} ^ {t},
$$
where  $z_{ui}^{t}$  are i.i.d. random variables with mean  $v_{ui}$  and  $\mathcal{T}_{ui}(t) = \{\tau < t : x_{ui}^{t} = 1\}$ . Hence, the signal  $\mathbb{1}\{v_{ui}^{t} \geq p_{i}\}$  is a function of  $z_{ui}^{t}$  in  $\mathcal{T}_{ui}(t)$ . As a result, the problem of learning revenue-maximizing offers by observing  $z_{ui}^{t}$  for all  $(u,i)$  such that  $x_{ui}^{t} = 1$  is no harder than that of learning revenue-maximizing offers by observing  $\mathbb{1}\{v_{ui}^{t} \geq p_{i}\}$  signals for all  $(u,i)$  such that  $x_{ui}^{t} = 1$ .
On top of observing  $z_{ui}^{t}$ , if we further assume that an oracle sets the prices  $p_i = v_{ui}^t$  whenever  $x_{ui}^{t} = 1$  (such that all the offers are accepted by achieving revenue  $v_{ui}^{t}$ ), the corresponding problem becomes an online linear optimization problem with semi-bandit feedback. In this problem, at each round  $t$ , the algorithm chooses an offering  $X^{t}$  from the set of feasible offers  $\mathcal{X}^t$ , observes  $z_{ui}^{t}$  for all  $(u,i)$  such that  $x_{ui}^{t} = 1$ , and aims to maximize the total revenue (note that the revenue at time  $t$  is equal to the sum of valuations for all  $(u,i)$  such that  $x_{ui}^{t} = 1$ ). Since this problem is also no harder than our original problem, any lower bound for this problem also applies to our setting.
Let  $\mathcal{N} = [N]$  and  $\mathcal{I} = [M]$  be the set of users and items respectively. Then, consider a market setting where the demands of users are given as
$$
d _ {u} ^ {t} = \left\{ \begin{array}{l l} L, & \text {i f} \quad t = (u - 1) \mod N \\ 0, & \text {o t h e r w i s e} \end{array} \right.
$$
so that only one user has nonzero demand in each round and each user is active for  $\Theta (T / N)$  rounds. Let  $\mathcal{E}_t = \mathcal{I}$  for all rounds and note that the maximum load in this market is  $L$  as needed. Consequently, we have  $N$  independent sub-problems where each sub-problem corresponds to offering the best  $L$  items in  $\mathcal{I}$  to a single user. In the literature on online linear optimization with semi-bandit feedback, this problem is known as the L-sets problem and it is known that for any algorithm there exists an instance such that expected regret  $\Omega (\sqrt{LMT_o})$  in  $T_{o}$  rounds Kleinberg and Leighton (2003). Therefore, for any algorithm, each of  $N$  sub-problems has  $\Omega (\sqrt{LMT / N})$  regret which corresponds to  $\Omega (\sqrt{NMLT})$  regret in total for the whole problem.
D Proofs for Revenue Maximization with Random Valuations
D.1 Preliminary Lemmas
Lemma 10. For any  $u \in \mathcal{N}$  and  $i \in \mathcal{I}$ , there exists constants  $C_1$  and  $C_2$  such that
$$
C _ {1} \left(p - p _ {u i} ^ {*}\right) ^ {2} <   \psi_ {u i} \left(p _ {u i} ^ {*}\right) - \psi_ {u i} (p) <   C _ {2} \left(p - p _ {u i} ^ {*}\right) ^ {2} \tag {22}
$$
for all  $p\in [0,1]$
Proof. Since  $\psi_{ui}''(p_{ui}^*)$  is strictly negative, there exists constants  $A_1, A_2, \epsilon > 0$  such that  $A_1(p - p_{ui}^*)^2 < \psi_{ui}(p_{ui}^*) - \psi_{ui}(p) < A_2(p - p_{ui}^*)^2$  for all  $p \in (p_{ui}^* - \epsilon, p_{ui}^* + \epsilon)$ . Since the set  $\mathcal{S} = \{p \in [0,1] : |p - p_{ui}^| \geq \epsilon\}$  is compact and  $\psi_{ui}(p_{ui}^*) - \psi_{ui}(p) > 0$  for all  $p \in \mathcal{S}$ , there exists constants  $B_1, B_2$  such that  $B_1(p - p_{ui}^*)^2 < \psi_{ui}(p_{ui}^*) - \psi_{ui}(p) < B_2(p - p_{ui}^*)^2$  for all  $p \in \mathcal{S}$ . Hence, if we set  $C_1 = A_1 \wedge B_1$  and  $C_2 = A_2 \vee B_2$ , we obtain the statement of the lemma.
Lemma 11. For any  $u \in \mathcal{N}$  and  $i \in \mathcal{I}$ , define  $\Delta_{uik} = \max_k \psi_{uik} - \psi_{uik}$ . If  $\widetilde{\Delta}_{ui0} \leq \widetilde{\Delta}_{ui1} \leq \dots \leq \widetilde{\Delta}_{ui(K-1)}$  are the elements of the set  $\{\Delta_{ui1}, \ldots, \Delta_{uiK}\}$  sorted in ascending order, then  $\widetilde{\Delta}_{uik} \geq c_1(k/2K)^2$ .
Proof. Applying Lemma 10 and using the definition of  $\Delta_{uik}$ , we can show that  $\Delta_{uik} \geq C_1(p_{ui}^* - k / K)^2$ . Then, lower bound on  $\widetilde{\Delta}_{uik}$  follows upon observing that at most  $j$  elements of the set  $\{1 / K, 2 / K, \ldots, 1\}$  lie within a distance  $j / 2K$  of  $p_{ui}^*$ .
Lemma 12. For any  $u \in \mathcal{N}$  and  $i \in \mathcal{I}$ , we have  $\psi_{ui}^{*} - \max_{k} \psi_{uik} \leq C_2 / K^2$ .
Proof. Applying Lemma 10 and noting that at least one of the prices  $\{1 / K,2 / K,\dots ,1\}$  lies within a distance  $1 / K$  of  $p_{ui}^{*}$ , we obtain the statement of the lemma.
Let  $\pi^{*}$  be the algorithm that chooses the optimum choice of offers and prices. Let  $\pi_{K}^{*}$  be the algorithm that chooses the optimum choice of offers, but chooses the prices as the best price from the set  $\{1 / K, 2 / K, \dots, 1\}$ . Let  $\pi$  denote our algorithm described in Algorithm 3. Furthermore, let  $\rho(\cdot)$  represent the expected revenue obtained by an algorithm. Then, the regret of policy  $\pi$  satisfies
$$
\mathcal {R} (T, \boldsymbol {\pi}) = \rho (\boldsymbol {\pi} ^ {*}) - \rho (\boldsymbol {\pi}) = \left(\rho (\boldsymbol {\pi} ^ {*}) - \rho \left(\boldsymbol {\pi} _ {K} ^ {*}\right)\right) + \left(\rho \left(\boldsymbol {\pi} _ {K} ^ {*}\right) - \rho (\boldsymbol {\pi})\right).
$$