SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \frac {1}{\gamma} \left(\frac {p _ {1} ^ {2}}{1 - p _ {1}} + \frac {(1 - p _ {1}) ^ {2}}{p _ {1}} + \frac {1 - 2 p _ {a ^ {\star}}}{W _ {a ^ {\star}}}\right) + \left\langle p - e _ {a ^ {\star}}, \widehat {f} \right\rangle \\ = \frac {1}{\gamma} \left(\frac {p _ {1} ^ {2}}{1 - p _ {1}} + \frac {(1 - p _ {1}) ^ {2}}{p _ {1}} + 1\right) + p _ {1} \widehat {f} _ {1} + p _ {2} \widehat {f} _ {2} - \widehat {f} _ {a \star} \\ \leq \frac {1}{\gamma} \left(\frac {p _ {1} ^ {2}}{1 - p _ {1}} + \frac {(1 - p _ {1}) ^ {2}}{p _ {1}} + 1\right) - p _ {1} \widehat {f} _ {2} \quad (\widehat {f} _ {a ^ {\star}} \geq \widehat {f} _ {2} \geq \widehat {f} _ {1} = 0) \\ \leq \frac {1}{\gamma} \left(\frac {1}{1 - p _ {1}} + 1 + 1\right) - p _ {1} \widehat {f} _ {2} \quad (p _ {1} \in [ \frac {1}{2}, 1 ], p _ {1} \geq p _ {2} \in [ 0, \frac {1}{2} ]) \\ = \frac {1}{\gamma} \left(4 + \gamma \widehat {f _ {2}}\right) - \left(1 - \frac {1}{2 + \gamma \widehat {f _ {2}}}\right) \widehat {f} _ {2} \\ \leq \frac {5}{\gamma}. \\ \end{array}
$$
If  $a^\star = a_2$ , we can obtain that
$$
\begin{array}{l} \frac {1}{\gamma} \left(\frac {p _ {1} ^ {2}}{1 - p _ {1}} + \frac {(1 - p _ {1}) ^ {2}}{p _ {1}} + \frac {1 - 2 p _ {a ^ {\star}}}{W _ {a ^ {\star}}}\right) + \left\langle p - e _ {a ^ {\star}}, \widehat {f} \right\rangle \\ = \frac {1}{\gamma} \left(\frac {p _ {1} ^ {2}}{1 - p _ {1}} + \frac {(1 - p _ {1}) ^ {2}}{p _ {1}} + \frac {1 - 2 p _ {2}}{p _ {1}}\right) + p _ {1} \widehat {f} _ {1} + p _ {2} \widehat {f} _ {2} - \widehat {f} _ {2} \\ \leq \frac {1}{\gamma} \left(\frac {p _ {1} ^ {2}}{1 - p _ {1}} + \frac {(1 - p _ {1}) ^ {2}}{p _ {1}} + \frac {1 - 2 (1 - p _ {1})}{p _ {1}}\right) - p _ {1} \widehat {f} _ {2} \quad (\widehat {f} _ {1} = 0) \\ \leq \frac {1}{\gamma} \left(\frac {1}{1 - p _ {1}} + 1 + 2 - \frac {1}{p _ {1}}\right) - p _ {1} \widehat {f} _ {2} \quad (p _ {1} \in [ \frac {1}{2}, 1 ], p _ {2} \in [ 0, \frac {1}{2} ]) \\ \leq \frac {1}{\gamma} \left(5 + \gamma \widehat {f _ {2}}\right) - \left(1 - \frac {1}{2 + \gamma \widehat {f _ {2}}}\right) \widehat {f _ {2}} \quad (p _ {1} = \frac {1}{2 + \gamma \widehat {f _ {2}}}) \\ \leq \frac {6}{\gamma}. \\ \end{array}
$$
If  $a^{\star} = a_{1}$  , we have
$$
\begin{array}{l} \frac {1}{\gamma} \left(\frac {p _ {1} ^ {2}}{1 - p _ {1}} + \frac {(1 - p _ {1}) ^ {2}}{p _ {1}} + \frac {1 - 2 p _ {a ^ {\star}}}{W _ {a ^ {\star}}}\right) + \left\langle p - e _ {a ^ {\star}}, \widehat {f} \right\rangle \\ \leq \frac {1}{\gamma} \left(\frac {p _ {1} ^ {2}}{1 - p _ {1}} + \frac {(1 - p _ {1}) ^ {2}}{p _ {1}} + \frac {1 - 2 p _ {1}}{1 - p _ {1}}\right) + (1 - p _ {1}) \widehat {f} _ {2} \\ \leq \frac {1}{\gamma} \left(1 - p _ {1} + \frac {(1 - p _ {1}) ^ {2}}{p _ {1}}\right) + (1 - p _ {1}) \widehat {f} _ {2} \\ \leq \frac {1}{\gamma} \left(1 + \frac {1}{2}\right) + \frac {\widehat {f} _ {2}}{2 + \gamma \widehat {f} _ {2}} \quad (p _ {1} \in [ \frac {1}{2}, 1 ]) \\ \leq \frac {3}{\gamma}. \\ \end{array}
$$
Putting everything together, we prove that  $\overline{\mathrm{dec}}_{\gamma}(p; \widehat{f}, x, G_{\mathrm{CR}}) \leq \frac{6}{\gamma} \leq \mathcal{O}\left(\frac{1}{\gamma}\right)$ .
B.2 Apple Tasting Graph
Proof. For the apple tasting graph and closed-form solution  $p$  in Eq. (7), Eq. (23) becomes:
$$
\overline {{\mathrm {d e c}}} _ {\gamma} (p; \widehat {f}, x, G) = \max  _ {a ^ {\star} \in [ K ]} \left\{\frac {1}{\gamma} \left(p _ {1} + \frac {(1 - p _ {1}) ^ {2}}{p _ {1}} + \frac {1 - 2 p _ {a ^ {\star}}}{W _ {a ^ {\star}}}\right) + \left\langle p - e _ {a ^ {\star}}, \widehat {f} \right\rangle \right\}.
$$
Suppose  $\widehat{f}_1 = 0$ , we know that  $p_1 = 1, p_2 = 0$  and
1. If  $a^\star = 1$ , we have
$$
\frac {1}{\gamma} \left(p _ {1} + \frac {(1 - p _ {1}) ^ {2}}{p _ {1}} + \frac {1 - 2 p _ {a ^ {\star}}}{W _ {a ^ {\star}}}\right) + \left\langle p - e _ {a ^ {\star}}, \widehat {f} \right\rangle = 0.
$$
2. If  $a^\star = 2$ , direct calculation shows that
$$
\frac {1}{\gamma} \left(p _ {1} + \frac {(1 - p _ {1}) ^ {2}}{p _ {1}} + \frac {1 - 2 p _ {a ^ {\star}}}{W _ {a ^ {\star}}}\right) + \left\langle p - e _ {a ^ {\star}}, \widehat {f} \right\rangle \leq \frac {2}{\gamma}.
$$
Suppose  $\widehat{f_2} = 0$ , we know that  $p_1 = \frac{2}{4 + \gamma \widehat{f_1}}$ ,  $p_2 = 1 - p_1$  and
1. If  $a^{\star} = 1$ , we have
$$
\begin{array}{l} \frac {1}{\gamma} \left(p _ {1} + \frac {\left(1 - p _ {1}\right) ^ {2}}{p _ {1}} + \frac {1 - 2 p _ {a ^ {\star}}}{W _ {a ^ {\star}}}\right) + \left\langle p - e _ {a ^ {\star}}, \widehat {f} \right\rangle \\ = \frac {1}{\gamma} \left(p _ {1} + \frac {\left(1 - p _ {1}\right) ^ {2}}{p _ {1}} + \frac {1 - 2 p _ {1}}{p _ {1}}\right) - (1 - p _ {1}) \widehat {f} _ {1} \\ = \frac {2 (1 - p _ {1}) ^ {2}}{\gamma p _ {1}} - (1 - p _ {1}) \widehat {f} _ {1} \\ = \frac {(2 + \gamma \widehat {f} _ {1}) ^ {2}}{\gamma (4 + \gamma \widehat {f} _ {1})} - (1 - p _ {1}) \widehat {f} _ {1} \\ \leq \frac {4 + \gamma \widehat {f} _ {1}}{\gamma} + \frac {2 \widehat {f} _ {1}}{4 + \gamma \widehat {f} _ {1}} - \widehat {f} _ {1} \leq \frac {6}{\gamma}. \\ \end{array}
$$
$$
\frac {1}{\gamma} \left(p _ {1} + \frac {(1 - p _ {1}) ^ {2}}{p _ {1}} + \frac {1 - 2 p _ {a ^ {\star}}}{W _ {a ^ {\star}}}\right) + \left\langle p - e _ {a ^ {\star}}, \widehat {f} \right\rangle = \frac {2 p _ {1}}{\gamma} + p _ {1} \widehat {f} _ {1} \leq \frac {1}{\gamma} + \frac {2 \widehat {f} _ {1}}{4 + \gamma \widehat {f} _ {1}} \leq \frac {3}{\gamma}.
$$
Putting everything together, we prove that  $\overline{\mathrm{dec}}_{\gamma}(p; \widehat{f}, x, G_{\mathrm{AT}}) \leq \frac{6}{\gamma} \leq \mathcal{O}\left(\frac{1}{\gamma}\right)$ .
B.3 Inventory Graph
Proof. Based on the distribution defined above, define  $A \subseteq [K]$  to be the set such that for all  $i \in A$ ,  $p_i > 0$  and denote  $N = |A|$ . We index each action in  $A$  by  $k_1 < k_2 < \dots < k_N = K$ . According to the definition of  $p_i$ , we know that  $p_i$  is strictly positive only when  $\widehat{f}_i < \widehat{f}_j$  for all  $j > i$  and specifically, when  $p_i > 0$ , we know that  $W_i = \sum_{j \geq i} p_j = \frac{1}{1 + \gamma \widehat{f}_i}$  (recall that  $\min_i f_i = 0$  since we shift  $\widehat{f}$ ). Therefore, define  $W_{k_{N+1}} = 0$  and we know that
$$
\begin{array}{l} \overline {{\operatorname {d e c}}} _ {\gamma} (p; \widehat {f}, x, G _ {\text {i n v}}) \\ = \sum_ {i = 1} ^ {N} p _ {k _ {i}} \widehat {f} _ {k _ {i}} + \frac {1}{\gamma} \sum_ {a = 1} ^ {K} \frac {p _ {a} ^ {2}}{W _ {a}} + \max  _ {a ^ {\star} \in [ K ]} \left\{\frac {1 - 2 p _ {a ^ {\star}}}{\gamma W _ {a ^ {\star}}} - \widehat {f} _ {a ^ {\star}} \right\} \\ \leq \sum_ {i = 1} ^ {N} \left(W _ {k _ {i}} - W _ {k _ {i + 1}}\right) \widehat {f} _ {k _ {i}} + \frac {1}{\gamma} + \max  _ {a ^ {\star} \in [ K ]} \left\{\frac {1 - 2 p _ {a ^ {\star}}}{\gamma W _ {a ^ {\star}}} - \widehat {f} _ {a ^ {\star}} \right\} \\ \leq \frac {2}{\gamma} + \sum_ {i = 1} ^ {N - 1} \left(\frac {1}{1 + \gamma \widehat {f} _ {k _ {i}}} - \frac {1}{1 + \gamma \widehat {f} _ {k _ {i + 1}}}\right) \widehat {f} _ {k _ {i}} + \max  _ {a ^ {\star} \in [ K ]} \left\{\frac {1 - 2 p _ {a ^ {\star}}}{\gamma W _ {a ^ {\star}}} - \widehat {f} _ {a ^ {\star}} \right\} \\ \leq \frac {3}{\gamma} + \sum_ {i = 2} ^ {N} \frac {\widehat {f} _ {k _ {i}} - \widehat {f} _ {k _ {i - 1}}}{1 + \gamma \widehat {f} _ {k _ {i}}} + \max _ {a ^ {\star} \in [ K ]} \left\{\frac {1 - 2 p _ {a ^ {\star}}}{\gamma W _ {a ^ {\star}}} - \widehat {f} _ {a ^ {\star}} \right\}. \\ \end{array}
$$
According to Lemma 9 of [Alon et al., 2013] (included as Lemma E.2 for completeness), we know that
$$
\sum_ {i = 2} ^ {N} \frac {\widehat {f} _ {k _ {i}} - \widehat {f} _ {k _ {i - 1}}}{1 + \gamma \widehat {f} _ {k _ {i}}} = \frac {1}{\gamma} \sum_ {i = 2} ^ {N} \frac {\widehat {f} _ {k _ {i}} - \widehat {f} _ {k _ {i - 1}}}{\frac {1}{\gamma} + \widehat {f} _ {k _ {i}}} \leq \frac {\operatorname {m a s} (G _ {A})}{\gamma} = \frac {1}{\gamma}, \tag {24}
$$
where  $G_{A}$  is the subgraph of  $G$  restricted to node set  $A$  and  $\mathrm{mas}(G)$  is the size of the maximum acyclic subgraphs of  $G$ . It is direct to see that any subgraph  $G$  of  $G_{\mathrm{inv}}$  has  $\mathrm{mas}(G) = 1$ .
Next, consider the value of  $a^{\star} \in [K]$  that maximizes  $\frac{1 - 2p_{a^{\star}}}{\gamma W_{a^{\star}}} - \widehat{f}_{a^{\star}}$ . If  $a^{\star} \leq k_{1}$ , then we know that  $W_{a^{\star}} = 1$  and  $\frac{1 - 2p_{a^{\star}}}{\gamma W_{a^{\star}}} - \widehat{f}_{a^{\star}} \leq \frac{1}{\gamma}$ . Otherwise, suppose that  $k_{i} < a^{\star} \leq k_{i + 1}$  for some  $i \in [N - 1]$ . According to the definition of  $p$ , if  $a^{\star} \neq k_{i + 1}$  we know that  $p_{a^{\star}} = 0$  and
$$
\frac {1}{1 + \gamma \widehat {f} _ {a ^ {\star}}} \leq \sum_ {j ^ {\prime} > a ^ {\star}} p _ {j ^ {\prime}} = W _ {k _ {i + 1}} = W _ {a ^ {\star}}.
$$
$$
\frac {1 - 2 p _ {a ^ {\star}}}{\gamma W _ {a ^ {\star}}} - \widehat {f} _ {a ^ {\star}} = \frac {1}{\gamma W _ {a ^ {\star}}} - \widehat {f} _ {a ^ {\star}} \leq \frac {1}{\gamma}.
$$
Otherwise,  $W_{a^{\star}} = W_{k_{i + 1}}$  and  $\frac{1 - 2p_{a^{\star}}}{\gamma W_{a^{\star}}} - \widehat{f}_{a^{\star}} \leq \frac{1}{\gamma W_{k_{i + 1}}} - \widehat{f}_{k_{i + 1}} = \frac{1}{\gamma}$ . Combining the two cases above and Eq. (24), we obtain that
$$
\overline {{\operatorname {d e c}}} _ {\gamma} (p; \widehat {f}, x, G _ {\text {i n v}}) \leq \frac {3}{\gamma} + \frac {1}{\gamma} + \frac {1}{\gamma} = \mathcal {O} \left(\frac {1}{\gamma}\right).
$$
B.4 Undirected and Self-Aware Graphs
Proof. We first introduce the closed-form of  $p$  and then show that  $\overline{\mathrm{dec}}_{\gamma}(p; \widehat{f}, x, G) \leq \mathcal{O}(\frac{\alpha}{\gamma})$ . Specifically, we first sort  $\widehat{f}_a$  in an increasing order and choose a maximal independent set by choosing the nodes in a greedy way. Specifically, we pick  $k_1 = \operatorname*{argmin}_{i \in [K]} \widehat{f}_i$ . Then, we ignore all the nodes that are connected to  $k_1$  and select the node  $a$  with the smallest  $\widehat{f}_a$  in the remaining node set. This forms a maximal independent set  $I \subseteq [K]$ , which has size no more than  $\alpha$  and is also a dominating set. Set  $p_a = \frac{1}{\alpha + \gamma \widehat{f}_a}$  for  $a \in I \setminus \{k_1\}$  and  $p_{k_1} = 1 - \sum_{a \neq k_1, a \in I} p_a$ . This is a valid distribution as we only choose at most  $\alpha$  nodes and  $p_a \leq 1 / \alpha$  for all  $a \in I \setminus \{k_1\}$ . Now we show that  $\overline{\mathrm{dec}}_{\gamma}(p; \widehat{f}, x, G) \leq \mathcal{O}\left(\frac{\alpha}{\gamma}\right)$ . Specifically, we only need to show that with this choice of  $p$ , for any  $a^\star \in [K]$ ,
$$
\sum_ {a = 1} ^ {K} p _ {a} \widehat {f} _ {a} - \widehat {f} _ {a ^ {\star}} + \frac {1}{\gamma} \sum_ {a = 1} ^ {K} \frac {p _ {a} ^ {2}}{W _ {a}} + \frac {1 - 2 p _ {a ^ {\star}}}{\gamma W _ {a ^ {\star}}} \leq \mathcal {O} \left(\frac {\alpha}{\gamma}\right).
$$
Plugging in the form of  $p$ , we know that
$$
\begin{array}{l} \sum_ {a = 1} ^ {K} p _ {a} \widehat {f} _ {a} - \widehat {f} _ {a ^ {\star}} + \frac {1}{\gamma} \sum_ {a = 1} ^ {K} \frac {p _ {a} ^ {2}}{W _ {a}} + \frac {1 - 2 p _ {a ^ {\star}}}{\gamma W _ {a ^ {\star}}} \\ \leq \sum_ {a \in I \backslash \{k _ {1} \}} \frac {\widehat {f} _ {a}}{\alpha + \gamma \widehat {f} _ {a}} - \widehat {f} _ {a ^ {\star}} + \frac {1 - 2 p _ {a ^ {\star}}}{\gamma W _ {a ^ {\star}}} + \frac {1}{\gamma} \quad (p _ {a} \leq W _ {a} \text {f o r a l l} a \in [ K ]) \\ \leq \frac {\alpha}{\gamma} - \widehat {f} _ {a ^ {\star}} + \frac {1 - 2 p _ {a ^ {\star}}}{\gamma W _ {a ^ {\star}}}. \quad (| I | \leq \alpha) \\ \end{array}
$$
If  $a^{\star} = k_{1}$ , then we can obtain that  $\frac{1 - 2p_{a^{\star}}}{\gamma W_{a^{\star}}} \leq \frac{1}{\gamma W_{k_{1}}} \leq \frac{\alpha}{\gamma}$  as  $p_{k_1} \geq \frac{1}{\alpha}$  according to the definition of  $p$ . Otherwise, note that according to the choice of the maximal independent set  $I$ ,  $W_{a^{\star}} \geq \frac{1}{\alpha + \gamma\widehat{f}_{a^{\prime}}}$  for some  $a^\prime \in I$  such that  $\widehat{f}_{a^{\prime}} \leq \widehat{f}_{a^{\star}}$ . Therefore,
$$
- \widehat {f} _ {a ^ {\star}} + \frac {1 - 2 p _ {a ^ {\star}}}{\gamma W _ {a ^ {\star}}} \leq - \widehat {f} _ {a ^ {\star}} + \frac {1}{\gamma W _ {a ^ {\star}}} \leq - \widehat {f} _ {a ^ {\star}} + \frac {\alpha + \gamma \widehat {f} _ {a ^ {\prime}}}{\gamma} \leq \frac {\alpha}{\gamma}.
$$
Combining the two inequalities above together proves the bound.
C Implementation Details in Experiments
C.1 Implementation Details in Section 5.1
We conduct experiments on RCV1 [Lewis et al., 2004], which is a multilabel text-categorization dataset. We use a subset of RCV1 containing 50000 samples and  $K = 50$  sub-classes. Therefore, the feedback graph in our experiment has  $K = 50$  nodes. We use the bag-of-words vector of each sample as the context with dimension  $d = 47236$  and treat the text categories as the arms. In each round  $t$ , the learner receives the bag-of-words vector  $x_{t}$  and makes a prediction  $a_{t} \in [K]$  as the text category. The loss is set to be  $\ell_{t,a_t} = 0$  if the sample belongs to the predicted category  $a_{t}$  and  $\ell_{t,a_t} = 1$  otherwise.
The function class we consider is the following linear function class:
$$
\mathcal {F} = \left\{f: f (x, a) = \operatorname {S i g m o i d} ((M x) _ {a}), M \in \mathbb {R} ^ {K \times d} \right\},
$$
where  $\operatorname{Sigmoid}(u) = \frac{1}{1 + e^{-u}}$  for any  $u \in \mathbb{R}$ . The oracle is implemented by applying online gradient descent with learning rate  $\eta$  searched over  $\{0.1, 0.2, 0.5, 1, 2, 4\}$ . As suggested by [Foster and
[Krishnamurthy, 2021], we use a time-varying exploration parameter  $\gamma_{t} = c\cdot \sqrt{\alpha t}$ , where  $t$  is the index of the iteration,  $c$  is searched over  $\{8,16,32,64,128\}$ , and  $\alpha$  is the independence number of the corresponding feedback graph. Our code is built on PyTorch framework [Paszke et al., 2019]. We run 5 independent experiments with different random seeds and plot the mean and standard deviation value of PV loss.
C.2 Implementation Details in Section 5.2
C.2.1 Details for Results on Random Directed Self-aware Graphs
We conduct experiments on a subset of RCV1 containing 10000 samples with  $K = 10$  sub-classes. Our code is built on Vowpal Wabbit [Langford and Zhang, 2007]. For SquareCB, the exploration parameter  $\gamma_{t}$  at round  $t$  is set to be  $\gamma_{t} = c \cdot \sqrt{Kt}$ , where  $t$  is the index of the round and  $c$  is the hyperparameter searched over set  $\{8, 16, 32, 64, 128\}$ . The remaining details are the same as described in Appendix C.1.
C.2.2 Details for Results on Synthetic Inventory Dataset
In this subsection, we introduce more details in the synthetic inventory data construction, loss function constructions, oracle implementation, and computation of the strategy at each round.
Dataset. In this experiment, we create a synthetic inventory dataset constructed as follows. The dataset includes  $T$  data points, the  $t$ -th of which is represented as  $(x_{t}, d_{t})$  where  $x_{t} \in \mathbb{R}^{m}$  is the context and  $d_{t}$  is the realized demand given context  $x_{t}$ . Specifically, in the experiment, we choose  $m = 100$  and  $x_{t}$ 's are drawn i.i.d from Gaussian distribution with mean 0 and standard deviation 0.1. The demand  $d_{t}$  is defined as
$$
d _ {t} = \frac {1}{\sqrt {m}} x _ {t} ^ {\top} \theta + \varepsilon_ {t},
$$
where  $\theta \in \mathbb{R}^m$  is an arbitrary vector and  $\varepsilon_{t}$  is a one-dimensional Gaussian random variable with mean 0.3 and standard deviation 0.1. After all the data points  $\{(x_t,d_t)\}_{t = 1}^T$  are constructed, we normalize  $d_{t}$  to [0, 1] by setting  $d_{t}\gets \frac{d_{t} - \min_{t^{\prime}\in[T]}d_{t^{\prime}}}{\max_{t^{\prime}\in[T]}d_{t^{\prime}} - \min_{t^{\prime}\in[T]}d_{t^{\prime}}}$ . In all our experiments, we set  $T = 10000$ .
Loss construction. Next, we define the loss at round  $t$  when picking the inventory level  $a_{t}$  with demand  $d_{t}$ , which is defined as follows:
$$
\ell_ {t, a _ {t}} = h \cdot \max  \left\{a _ {t} - d _ {t}, 0 \right\} + b \cdot \max  \left\{d _ {t} - a _ {t}, 0 \right\}, \tag {25}
$$
where  $h > 0$  is the holding cost per remaining items and  $b > 0$  is the backorder cost per remaining items. In the experiment, we set  $h = 0.25$  and  $b = 1$ .
Regression oracle. The function class we use in this experiment is as follows:
$$
\mathcal {F} = \{f: f (x, a) = h \cdot \max  \{a - (x ^ {\top} \theta + \beta), 0 \} + b \cdot \max  \{x ^ {\top} \theta + \beta - a, 0 \}, \theta \in \mathbb {R} ^ {m}, \beta \in \mathbb {R} \}.
$$
This ensures the realizability assumption according to the definition of our loss function shown in Eq. (25). The oracle uses online gradient descent with learning rate  $\eta$  searched over  $\{0.01, 0.05, 0.1, 0.5, 1\}$ .
Calculation of  $p_t$ . To make SquareCB.G more efficient, instead of solving the convex program defined in Eq. (5), we use the closed-form of  $p_t$  derived in Proposition 3, which only requires  $\mathcal{O}(K)$  computational cost and has the same theoretical guarantee (up to a constant factor) as the one enjoyed by the solution solved by Eq. (5). Similar to the case in Appendix C.1, at each round  $t$ , we pick  $\gamma_t = c \cdot \sqrt{t}$  with  $c$  searched over the set  $\{0.25, 0.5, 1, 2, 3, 4\}$ . Note again that the independence number for inventory graph is 1.
We run 8 independent experiments with different random seeds and plot the mean and standard deviation value of PV loss.
D Adaptive Tuning of  $\gamma$  without the Knowledge of Graph-Theoretic Numbers
In this section, we show how to adaptively tune the parameter  $\gamma$  in order to achieve  $\widetilde{\mathcal{O}}\left(\sqrt{\sum_{t=1}^{T}\alpha_{t}\mathbf{Reg}_{\mathbf{Sq}}}\right)$  regret in the strongly observable graphs case and  $\widetilde{\mathcal{O}}\left(\left(\sum_{t=1}^{T}\sqrt{d_{t}}\right)^{\frac{2}{3}}\mathbf{Reg}_{\mathbf{Sq}}^{\frac{1}{3}} + \sqrt{\sum_{t=1}^{T}\widetilde{\alpha}_{t}\mathbf{Reg}_{\mathbf{Sq}}}\right)$  in the weakly observable graphs case.
D.1 Strongly Observable Graphs
In order to achieve  $\widetilde{\mathcal{O}}\left(\sum_{t=1}^{T} \alpha_{t} \mathbf{Reg}_{\mathbf{Sq}}\right)$  regret guarantee without the knowledge of  $\alpha_{t}$ , we apply a doubling trick on  $\gamma$  based on the value of  $\min_{p \in \Delta(K)} \overline{\mathrm{dec}}_{\gamma}(p; \widehat{f_{t}}, x_{t}, G_{t})$ . Specifically, our algorithm goes in epochs with the parameter  $\gamma$  being  $\gamma_{s}$  in the  $s$ -th epoch. We initialize  $\gamma_{1} = \sqrt{\frac{T}{\mathbf{Reg}_{\mathbf{Sq}}}}$ . As proven in Theorem 3.2, we know that
$$
\gamma \cdot \min  _ {p \in \Delta (K)} \overline {{\operatorname* {d e c}}} _ {\gamma} (p; \widehat {f} _ {t}, x _ {t}, G _ {t}) \leq \widetilde {\mathcal {O}} (\alpha_ {t}).
$$
Therefore, within each epoch  $s$  (with start round  $b_{s}$ ), at round  $t$ , we calculate the value
$$
Q _ {t} = \sum_ {\tau = b _ {s}} ^ {t} \min  _ {p \in \Delta (K)} \overline {{\operatorname {d e c}}} _ {\gamma_ {s}} \left(p; \widehat {f} _ {t}, x _ {t}, G _ {t}\right), \tag {26}
$$
which is bounded by  $\widetilde{\mathcal{O}}\left(\frac{1}{\gamma_s}\sum_{\tau = b_s}^t\alpha_\tau\right)$  and is in fact obtainable by solving the convex program. Then, we check whether  $Q_{t}\leq \gamma_{s}\mathbf{Reg}_{\mathbf{Sq}}$  . If this holds, we continue our algorithm using  $\gamma_{s}$  ; otherwise, we set  $\gamma_{s + 1} = 2\gamma_{s}$  and restart the algorithm.
Now we analyze the performance of the above described algorithm. First, note that for any  $t$ , within epoch  $s$ ,
$$
\gamma_ {s} Q _ {t} \leq \widetilde {\mathcal {O}} \left(\sum_ {\tau = 1} ^ {T} \alpha_ {\tau}\right),
$$
meaning that the number of epoch  $S$  is bounded by  $S = \log_2 C_1 + \log_4 \frac{\sum_{t=1}^T \alpha_t}{T}$  for certain  $C_1 > 0$  which only contains constant and log terms.
Next, consider the regret in epoch  $s$  with  $I_{s} = [b_{s}, e_{s}]$ . According to Eq. (8), we know that the regret within epoch  $s$  is bounded as follows:
$$
\begin{array}{l} \mathbb {E} \left[ \sum_ {t \in I _ {s}} f ^ {\star} (x _ {t}, a _ {t}) - \sum_ {t \in I _ {s}} f ^ {\star} (x _ {t}, \pi^ {\star} (x _ {t})) \right] \\ \leq \mathbb {E} \left[ \sum_ {t \in I _ {s}} \overline {{\operatorname {d e c}}} _ {\gamma_ {s}} \left(p _ {t}; \widehat {f} _ {t}, x _ {t}, G _ {t}\right) \right] + \frac {\gamma_ {s}}{4} \mathbf {R e g} _ {\mathrm {S q}} \\ \leq \mathbb {E} \left[ \sum_ {t \in [ b _ {s}, e _ {s} - 1 ]} \overline {{\operatorname {d e c}}} _ {\gamma_ {s}} \left(p _ {t}; \widehat {f} _ {t}, x _ {t}, G _ {t}\right) \right] + \widetilde {\mathcal {O}} \left(\frac {\alpha_ {e _ {s}}}{\gamma_ {s}}\right) + \frac {\gamma_ {s}}{4} \mathbf {R e g} _ {\mathrm {S q}} \\ \leq \widetilde {\mathcal {O}} \left(\gamma_ {s} \operatorname {R e g} _ {\mathrm {S q}}\right), \tag {27} \\ \end{array}
$$
where the last inequality is because at round  $t = e_s - 1$ ,  $Q_t \leq \gamma_s \mathbf{Reg}_{\mathbb{S}^q}$  is satisfied. Taking summation over all  $S$  epochs, we know that the overall regret is bounded as
$$
\begin{array}{l} \mathbb {E} [ \mathbf {R e g} _ {\mathrm {C B}} ] \leq \sum_ {s = 1} ^ {S} \widetilde {\mathcal {O}} (\gamma_ {s} \mathbf {R e g} _ {\mathrm {S q}}) = \sum_ {s = 1} ^ {S} \widetilde {\mathcal {O}} \left(2 ^ {s - 1} \sqrt {T \mathbf {R e g} _ {\mathrm {S q}}}\right) \\ \leq \widetilde {\mathcal {O}} \left(2 ^ {S} \sqrt {T \mathbf {R e g} _ {\mathbf {S q}}}\right) = \widetilde {\mathcal {O}} \left(\sqrt {\sum_ {t = 1} ^ {T} \alpha_ {t} \mathbf {R e g} _ {\mathbf {S q}}}\right), \tag {28} \\ \end{array}
$$
D.2 Weakly Observable Graphs
For the weakly observable graphs case, to achieve the target regret without the knowledge of  $\widetilde{\alpha}_t$  and  $d_t$ , which are the independence number of the subgraph induced by nodes with self-loops in  $G_t$  and the weak domination number of  $G_t$ , we apply the same approach as the one applied in the strongly observable graph case. Note that according to Theorem 3.4, within epoch  $s$ , we have  $Q_t \leq C_2 \left( \frac{\sum_{\tau = b_s}^t \sqrt{d_\tau}}{\sqrt{\gamma_s}} + \frac{\sum_{\tau = b_s}^t \widetilde{\alpha}_\tau}{\gamma_s} \right)$  for certain  $C_2 > 1$  only containing constants and log factors. In the weakly observable graphs case, we know that the number of epoch is bounded by
$$
S = 2 + \log_ {2} C _ {2} + \max \left\{\log_ {4} \frac {\sum_ {t = 1} ^ {T} \widetilde {\alpha} _ {t}}{T}, \log_ {2} \left(\frac {\left(\sum_ {t = 1} ^ {T} \sqrt {d _ {t}}\right) ^ {\frac {2}{3}}}{\sqrt {T} \cdot \mathbf {R e g} _ {\mathrm {S q}} ^ {\frac {1}{6}}}\right) \right\} \text {s i n c e w h a v e}
$$
$$
\gamma_ {S} = 4 C _ {2} \cdot \sqrt {\frac {1}{\mathbf {R e g} _ {S q}}} \max \left\{\sqrt {\sum_ {t = 1} ^ {T} \widetilde {\alpha} _ {t}}, \frac {(\sum_ {t = 1} ^ {T} \sqrt {d _ {t}}) ^ {\frac {2}{3}}}{\mathbf {R e g} _ {S q} ^ {\frac {1}{6}}} \right\},
$$
and at round  $t$  in epoch  $S$ ,
$$
Q _ {t} \leq C _ {2} \left(\frac {\sum_ {\tau = 1} ^ {T} \widetilde {\alpha} _ {\tau}}{\gamma_ {S}} + \frac {\sum_ {\tau = 1} ^ {T} \sqrt {d _ {\tau}}}{\sqrt {\gamma_ {S}}}\right) \leq \gamma_ {S} \mathbf {R e g} _ {S \mathbf {q}},
$$
meaning that epoch  $S$  will never end. Therefore, following Eq. (27) and Eq. (28), we can obtain that
$$
\mathbb {E} \left[ \operatorname {R e g} _ {\mathrm {C B}} \right] \leq \widetilde {\mathcal {O}} \left(2 ^ {S} \sqrt {T \operatorname {R e g} _ {\mathrm {S q}}}\right) = \widetilde {\mathcal {O}} \left(\sqrt {\sum_ {t = 1} ^ {T} \widetilde {\alpha} _ {t} \operatorname {R e g} _ {\mathrm {S q}}} + \left(\sum_ {t = 1} ^ {T} \sqrt {d _ {t}}\right) ^ {\frac {2}{3}} \operatorname {R e g} _ {\mathrm {S q}} ^ {\frac {1}{3}}\right),
$$