SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We begin with upper bounding  $\rho (\pi_K^*) - \rho (\pi)$ . Recall that we can represent the expected revenue of any offering and pricing (that is constrained to set the set  $\{1 / K,2 / K,\dots ,1\}$ ) as
$$
\sum_ {u \in \mathcal {N}} \sum_ {i \in \mathcal {I}} \sum_ {k = 1} ^ {K} x _ {u i k} ^ {t} \psi_ {u i k},
$$
where  $x_{uik}^t$  are binary variables that denote whether user  $u$  is offered item  $i$  at  $k^{th}$  price level. Due to the endowment, demand constraints, and the constraint that requires the prices to be from the set  $\{1 / K, 2 / K, \dots, 1\}$ , the  $x_{uik}^t$  variables at each time  $t$  need to belong to the set
$$
\begin{array}{l} \mathcal {X} _ {e} ^ {t} = \left\{\boldsymbol {X} \in \{0, 1 \} ^ {N \times M \times K}: \sum_ {i \in \mathcal {I}} \sum_ {k = 1} ^ {K} x _ {u i k} \leq d _ {u} ^ {t}, \forall u \in \mathcal {N} \right. \\ \text {a n d} \sum_ {u \in \mathcal {N}} \sum_ {k = 1} ^ {K} x _ {u i k} \leq e _ {i} ^ {t}, \forall i \in \mathcal {I} \\ \left. \text {a n d} \sum_ {k = 1} ^ {K} x _ {u i k} \leq 1, \forall u \in \mathcal {N}, i \in \mathcal {I} \right\}. \\ \end{array}
$$
Similar to the proof of Lemma 7, we have  $|\widehat{\psi}_{uik}^t - \psi_{uik}| \leq \sqrt{8\log(NMK / \delta) / n_{uik}^t}$  for all  $u \in \mathcal{N}$ ,  $i \in \mathcal{I}$  and  $t \in [T]$ . For ease of notation, let us define  $B := \sqrt{8\log(NMK / \delta)}$  and  $w_{uik}^t = \sqrt{1 / n_{uik}^t}$ . Assuming that this condition holds, we can show that
$$
\begin{array}{l} \rho \left(\boldsymbol {\pi} _ {K} ^ {*}\right) - \rho (\boldsymbol {\pi}) \leq \sum_ {t = 1} ^ {T} \left[ \max  _ {\mathbf {X} \in \mathcal {X} _ {e} ^ {t}} \sum_ {u \in \mathcal {N}} \sum_ {i \in \mathcal {I}} \sum_ {k = 1} ^ {K} x _ {u i k} \psi_ {u i k} - \sum_ {u \in \mathcal {N}} \sum_ {i \in \mathcal {I}} \sum_ {k = 1} ^ {K} x _ {u i k} ^ {t} \psi_ {u i k} \right] \\ \leq \sum_ {t = 1} ^ {T} \left[ \max  _ {\mathbf {X} \in \mathcal {X} _ {e} ^ {t}} \sum_ {u \in \mathcal {N}} \sum_ {i \in \mathcal {I}} \sum_ {k = 1} ^ {K} x _ {u i k} \left(\widehat {\psi} _ {u i k} ^ {t} + \frac {B}{w _ {u i k} ^ {t}}\right) - \sum_ {u \in \mathcal {N}} \sum_ {i \in \mathcal {I}} \sum_ {k = 1} ^ {K} x _ {u i k} ^ {t} \psi_ {u i k} \right] \\ \leq \sum_ {t = 1} ^ {T} \left[ \sum_ {u \in \mathcal {N}} \sum_ {i \in \mathcal {I}} \sum_ {k = 1} ^ {K} x _ {u i k} ^ {t} \left(\widehat {\psi} _ {u i k} ^ {t} + \frac {B}{w _ {u i k} ^ {t}}\right) - \sum_ {u \in \mathcal {N}} \sum_ {i \in \mathcal {I}} \sum_ {k = 1} ^ {K} x _ {u i k} ^ {t} \psi_ {u i k} \right] \\ \leq \sum_ {t = 1} ^ {T} \sum_ {u \in \mathcal {N}} \sum_ {i \in \mathcal {I}} \sum_ {k = 1} ^ {K} x _ {u i k} ^ {t} \left(\widehat {\psi} _ {u i k} ^ {t} - \psi_ {u i k} + \frac {B}{w _ {u i k} ^ {t}}\right). \\ \end{array}
$$
Hence, with probability  $1 - \delta$ , we have
$$
\rho \left(\boldsymbol {\pi} _ {K} ^ {*}\right) - \rho (\boldsymbol {\pi}) \leq 2 B \sum_ {t = 1} ^ {T} \sum_ {u \in \mathcal {N}} \sum_ {i \in \mathcal {I}} \sum_ {k = 1} ^ {K} \frac {x _ {u i k} ^ {t}}{w _ {u i k} ^ {t}}. \tag {23}
$$
At each time step  $t$ , consider the list consisting of  $w_{uik}^{t}$  for all  $(u,i,k)\in \mathcal{O}_t\coloneqq \{(u,i,k):x_{uik}^t = 1\}$ . Let's now consider the overall list consisting of the concatenation of all of these lists over all rounds. Let's order this list in decreasing order to obtain a list  $\tilde{w} = (\tilde{w}_1,\tilde{w}_2,\dots ,\tilde{w}_J)$  where  $J = \sum_{t = 1}^{T}|\mathcal{O}_t|\leq LT$ . Using this notation, we have
$$
\rho \left(\boldsymbol {\pi} _ {K} ^ {*}\right) - \rho (\boldsymbol {\pi}) \leq 2 B \sum_ {j = 1} ^ {J} \widetilde {w} _ {j}.
$$
First, for any  $(u,i,k)$  pair, the list  $\tilde{w}$  can contain at most  $1 + 1 / c^2$  elements that are associated with  $(u,i,k)$  and larger than or equal to  $c$ . Secondly, note that  $x_{uik}^{t} = 1$  only if  $B / w_{uik}^{t} \geq \max_k \psi_{uik} - \psi_{uik} = \Delta_{uik}$ . As a result, size of  $\{t : x_{uik}^{t} = 1\}$  is at most  $B^2 / \Delta_{uik}^2$ . Therefore, the number of times a  $(u,i,k)$  pair can appear in the list  $\tilde{w}$  is also upper bounded by  $B^2 / \Delta_{uik}^2$ . Therefore, summing over all  $k \in [K]$ , we can upper bound the number of elements that are associated
with  $(u,i)$  and larger than or equal to  $c$  by
$$
\begin{array}{l} \sum_ {k = 1} ^ {K} \min  \left\{1 + \frac {1}{c ^ {2}}, \frac {B ^ {2}}{\Delta_ {u i k} ^ {2}} \right\} = \sum_ {k = 0} ^ {K - 1} \min  \left\{1 + \frac {1}{c ^ {2}}, \frac {B ^ {2}}{\widetilde {\Delta} _ {u i k} ^ {2}} \right\} \\ \leq 1 + \frac {1}{c ^ {2}} + \sum_ {k = 1} ^ {K - 1} \min  \left\{1 + \frac {1}{c ^ {2}}, \frac {B ^ {2}}{C _ {1} ^ {2}} \left(\frac {2 K}{k}\right) ^ {4} \right\} \\ \leq 1 + \frac {1}{c ^ {2}} + \frac {B}{C _ {1}} \sqrt {1 + \frac {1}{c ^ {2}}} \sum_ {k = 1} ^ {K - 1} \left(\frac {2 K}{k}\right) ^ {2} \\ = O \left(1 + \frac {1}{c ^ {2}} + B K ^ {2} \sqrt {1 + \frac {1}{c ^ {2}}}\right). \\ \end{array}
$$
where the second step uses Lemma 11. Thus, the total number of times that any confidence set can have size at least  $\widetilde{w}_j$  is upper bounded by  $O\left(NM\left(1 + \frac{1}{\widetilde{w}_j^2}\right) + NMBK^2\sqrt{1 + \frac{1}{\widetilde{w}_j^2}}\right)$ . Using this result, we can write  $\widetilde{w}_j = O\left(\min \left\{1,\frac{1}{\sqrt{(j / BNKM^2)^2 - 1}} +\frac{1}{\sqrt{j / NM - 1}}\right\}\right)$ . Hence,
$$
\begin{array}{l} \sum_ {j = 1} ^ {J} \widetilde {w} _ {j} = O \left(\sum_ {j = 1} ^ {J} \min  \left\{1, \frac {1}{\sqrt {(j / B N M K ^ {2}) ^ {2} - 1}} + \frac {1}{\sqrt {j / N M - 1}} \right\}\right) \\ = O \left(\sum_ {j = 1} ^ {L T} \min  \left\{1, \frac {1}{\sqrt {(j / B N M K ^ {2}) ^ {2} - 1}} + \frac {1}{\sqrt {j / N M - 1}} \right\}\right) \\ = O \left(B N M K ^ {2} \log (L T) + \sqrt {N M L T}\right). \\ \end{array}
$$
$$
\rho \left(\boldsymbol {\pi} _ {K} ^ {*}\right) - \rho (\boldsymbol {\pi}) = O \left(B ^ {2} N M K ^ {2} \log (L T) + \sqrt {N M L T}\right). \tag {24}
$$
Next, we bound  $\rho(\pi^{*}) - \rho(\pi_{K}^{*})$ . Note that for each  $(u,i)$ , at least one of the numbers  $\{1 / K, 2 / K, \ldots, 1\}$  lies within  $1 / K$  of  $p^{*}(u,i)$ . Then, by Lemma 12,  $\psi_{ui}^{*} - \max_{k} \psi_{uik} \leq C_{2} / K^{2}$  for some absolute constant  $C_{2}$ . Therefore, the gap  $\rho(\pi^{*}) - \rho(\pi_{K}^{*})$  is upper bounded as
$$
\rho \left(\boldsymbol {\pi} ^ {*}\right) - \rho \left(\boldsymbol {\pi} _ {K} ^ {*}\right) \leq \frac {C _ {2} L T}{K ^ {2}}. \tag {25}
$$
Combining equations (24) and (25), we have
$$
\mathcal {R} (T, \boldsymbol {\pi}) = O \left(B ^ {2} N M K ^ {2} \log (L T) + \sqrt {N M L T} + \frac {L T}{K ^ {2}}\right).
$$
Lastly, we choose  $K^4 = \Theta \left(\frac{LT}{NM\log(LT)}\right)$  to obtain
$$
\begin{array}{l} \mathcal {R} (T, \boldsymbol {\pi}) = O \left(B ^ {2} \sqrt {N M L T \log (L T)}\right) \\ = O \left(\sqrt {N M L T \log (L T) \log^ {2} (N M K / \delta)}\right) \\ = O \left(\sqrt {N M L T \log (L T) \log^ {2} (N M T / \delta)}\right). \\ \end{array}
$$
where the last step uses  $0 \leq K^4 \leq T$  and  $T \geq 1$ .
D.3 Proof of Theorem 6
Consider the same market setting (demands and endowments) in Appendix Section B.2, but assume that the valuations are i.i.d. random instead of being fixed. By construction, the optimum offering pattern at each time is to offer all available items to a single user that has non-zero demand. Hence, the problem of the provider reduces to only learning the price at which it should offer each item. There are  $NM$  actual user-item pairs and each item  $i \in [M]$  is offered to user  $u \in [N]$  for  $\Theta(LT/NM')$  rounds. In the literature on pricing optimization with i.i.d. valuations, each pricing problem is known to have  $\Omega(\sqrt{T_o})$  regret in  $T_o$  rounds (Kleinberg and Leighton, 2003). Therefore, any policy must have  $\Omega(\sqrt{NMLT})$  regret in total.
E Martingale Exponential Inequalities
Consider a sequence of random variables  $(Z_{n})_{n\in \mathbb{N}}$  adapted to the filtration  $(\mathcal{H}_n)_{n\in \mathbb{N}}$ . Assume  $\mathbb{E}[\exp (\lambda Z_i)]$  is finite for all  $\lambda$ . Define the conditional mean  $\mu_{i} = \mathbb{E}[Z_{i}|\mathcal{H}_{i - 1}]$ , and define the conditional cumulant generating function of the centered random variable  $[Z_i - \mu_i]$  by  $\psi_{i}(\lambda)\coloneqq \log \mathbb{E}[\exp (\lambda [Z_i - \mu_i])|\mathcal{H}_{i - 1}]$ . Then, for a process  $(M_n(\lambda))_{n\in \mathbb{N}}$  defined as
$$
M _ {n} (\lambda) = \exp \left\{\sum_ {i = 1} ^ {n} \lambda [ Z _ {i} - \mu_ {i} ] - \psi_ {i} (\lambda) \right\},
$$
we can prove the following properties.
Lemma 13.  $(M_{n}(\lambda))_{n\in \mathbb{N}}$  is a martingale with respect to the filtration  $(\mathcal{H}_n)_{n\in \mathbb{N}}$ , and  $\mathbb{E}[M_n(\lambda)] = 1$  for any  $\lambda$ .
$$
\mathbb {E} \left[ M _ {1} (\lambda) \mid \mathcal {H} _ {0} \right] = \mathbb {E} \left[ \exp \left\{\lambda \left[ Z _ {1} - \mu_ {1} \right] - \psi_ {1} (\lambda) \right\} \mid \mathcal {H} _ {0} \right] = 1.
$$
Then, for any  $n \geq 2$ ,
$$
\begin{array}{l} \mathbb {E} \left[ M _ {n} (\lambda) \mid \mathcal {H} _ {n - 1} \right] = \mathbb {E} \left[ M _ {n - 1} (\lambda) \exp \left\{\lambda \left[ Z _ {n} - \mu_ {n} \right] - \psi_ {n} (\lambda) \right\} \mid \mathcal {H} _ {n - 1} \right] \\ = M _ {n - 1} (\lambda) \mathbb {E} \left[ \exp \left\{\lambda \left[ Z _ {n} - \mu_ {n} \right] - \psi_ {n} (\lambda) \right\} \mid \mathcal {H} _ {n - 1} \right] \\ = M _ {n - 1} (\lambda), \\ \end{array}
$$
since  $M_{n - 1}(\lambda)$  is a measurable function of the filtration  $\mathcal{H}_{n - 1}$ .
Lemma 14. For all  $x \geq 0$  and  $\lambda \geq 0$ ,
$$
\mathbb {P} \left(\sum_ {i = 1} ^ {t} \lambda Z _ {i} \leq x + \sum_ {i = 1} ^ {t} [ \lambda \mu_ {i} + \psi_ {i} (\lambda) ] \quad , \forall t \in \mathbb {N}\right) \geq 1 - e ^ {- x}.
$$
Proof. For any  $\lambda$ ,  $(M_{n}(\lambda))_{n\in \mathbb{N}}$  is a martingale with respect to  $(\mathcal{H}_n)_{n\in \mathbb{N}}$  and  $\mathbb{E}[M_n(\lambda)] = 1$  by Lemma 13. For arbitrary  $x\geq 0$ , define  $\tau_{x} = \inf \{n\geq 0|M_{n}(\lambda)\geq x\}$  and note that  $\tau_{x}$  is a stopping time corresponding to the first time  $M_{n}$  crosses the boundary  $x$ . Since  $\tau$  is a stopping time with respect to  $(\mathcal{H}_n)_{n\in \mathbb{N}}$ , we have  $\mathbb{E}[M_{\tau_x\wedge n}(\lambda)] = 1$ . Then, by Markov's inequality
$$
x \mathbb {P} \left(M _ {\tau_ {x} \wedge n} (\lambda) \geq x\right) \leq \mathbb {E} \left[ M _ {\tau_ {x} \wedge n} (\lambda) \right] = 1.
$$
Noting that the event  $\{M_{\tau_x\wedge n}(\lambda)\geq x\} = \bigcup_{k = 1}^{n}\{M_k(\lambda)\geq x\}$ , we have
$$
\mathbb {P} \left(\bigcup_ {k = 1} ^ {n} \{M _ {k} (\lambda) \geq x \}\right) \leq \frac {1}{x}.
$$
Taking the limit as  $n\to \infty$  , and applying monotone convergence theorem shows that
$$
\mathbb {P} \left(\bigcup_ {k = 1} ^ {\infty} \{M _ {k} (\lambda) \geq x \}\right) \leq \frac {1}{x}.
$$
Consequently, we can write
$$
\mathbb {P} \left(\bigcup_ {k = 1} ^ {\infty} \{M _ {k} (\lambda) \geq e ^ {x} \}\right) \leq e ^ {- x}.
$$
Then, by definition of  $M_{k}(\lambda)$ , we conclude
$$
\mathbb {P} \left(\bigcup_ {k = 1} ^ {\infty} \left\{\sum_ {i = 1} ^ {n} \lambda \left[ Z _ {i} - \mu_ {i} \right] - \psi_ {i} (\lambda) \geq x \right\}\right) \leq e ^ {- x}
$$