SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Subsequently, the algorithm swaps the positions: the assortment  $A_{tk^*}'$  is moved to the top of  $S_t$ , becoming  $A_{t1}$ , and the initially top assortment  $A_{t1}'$  in  $S_t'$  is relocated to the  $k^*$ -th position of  $S_t$ , now  $A_{tk^*}$ . The positions of the other assortments remain the same, that is,  $A_{tk} = A_{tk}'$  for all  $k \in [K] \setminus \{1, k^*\}$ . This procedure is viable as the expected reward is unaffected by the display order of assortments in the cascade, as shown in Lemma 4.5
3.3 Regularized Maximum Likelihood Estimation
UCB-CCA computes a regularized maximum likelihood estimate of the unknown parameter  $\theta^{*}$ . The negative log-likelihood is given by  $\ell_t(\theta) = -\sum_{\tau=1}^{t-1} \sum_{k=1}^{O_\tau} \sum_{m=0}^M y_{\tau km} \log p_\tau(i_m | A_{\tau k}, w_\tau)$ , where
3While the swapping occurs between the first and the  $k^{*}$  -th positions for specificity, it is sufficient to place  $A_{tk^*}^\prime$  at the top position of  $S_{t}$ . The sequence of the remaining assortments is not critical.
$w_{t} = (w_{ti})_{i\in [N]}$  is a weight vector, and its element is  $w_{ti} = x_{ti}^{\top}\theta$  . For penalty parameter  $\lambda \geq 1$  , the  $\ell_2$  -regularized MLE is given by
$$
\hat {\theta} _ {t} = \underset {\theta} {\operatorname {a r g m i n}} \left[ \ell_ {t} (\theta) + \frac {\lambda}{2} | | \theta | | _ {2} ^ {2} \right] = \underset {\theta} {\operatorname {a r g m i n}} \left[ - \sum_ {\tau = 1} ^ {t - 1} \sum_ {k = 1} ^ {O _ {\tau}} \sum_ {m = 0} ^ {M} y _ {\tau k m} \log p _ {\tau} \left(i _ {m} \mid A _ {\tau k}, w _ {\tau}\right) + \frac {\lambda}{2} | | \theta | | _ {2} ^ {2} \right]. \tag {2}
$$
4 Regret Analysis of UCB-CCA
4.1 Regularity Condition
Assumption 4.1.  $||x||_{ti}\leq 1$  for all round  $t$  and items  $i\in [N]$ , and also  $||\theta^{*}||\leq 1$ .
Assumption 4.2. There exists  $\kappa >0$  such that for all  $t\in [T]$ , any assortment  $A\in \mathcal{A}$ , and any item  $i\in A,\inf_{\theta \in \mathbb{R}^d}p_t(i|A,w)p_t(i_0|A,w)\geq \kappa$ , where  $w = (w_{i})_{i\in [N]}$  and  $w_{i} = x_{i}^{\top}\theta$ .
Discussion of Assumptions. Assumption 4.1 makes the regret bound independent on the scale of the feature vector and parameter. This is the standard assumption used in the contextual bandit literature [1; 18; 22]. Assumption 4.2 is the standard regularity assumption in the contextual assortment bandit literature [8; 27; 22; 7; 23], adapted from the standard assumption for the link function in the generalized linear contextual bandit literature [16] to ensure that the Fisher information matrix is non-singular.
4.2 Regret Bound of UCB-CCA
Theorem 4.3 ( $\alpha$ -regret upper bound of UCB-CCA). Suppose Assumptions [4.1] and [4.2] hold, and we run UCB-CCA for total  $T$  rounds with  $\beta_{t} = \frac{1}{2\kappa}\sqrt{d\log\left(1 + \frac{tKM}{d\lambda}\right) + 4\log t + \frac{\sqrt{\lambda}}{\kappa}}$  and with  $\lambda \geq 1$ . Then, the  $\alpha$ -regret of UCB-CCA is upper-bounded by
$$
\mathcal {R} ^ {\alpha} (T) \leq \left(\frac {K}{K + 1}\right) ^ {K + 1} \left[ \frac {1}{2 \kappa} \sqrt {d \log \left(1 + \frac {T K M}{d \lambda}\right) + 4 \log T} + \frac {\sqrt {\lambda}}{\kappa} \right] \sqrt {2 d T \ln \left(1 + \frac {T K M}{\lambda d}\right)}.
$$
Discussion of Theorem 4.3. Theorem 4.3 establishes that UCB-CCA achieves a regret bound of  $\tilde{\mathcal{O}}\left(\frac{d}{\kappa}\sqrt{T}\right)$ . Notably, this regret bound removes dependence on  $p^*$  completely and removes polynomial dependence on  $K$ , achieving the best-known bound in contextual cascading bandits [18; 25; 20], a special case of our problem setting. Apart from the generalization we consider in this work, a key distinction between our work and the previous contextual cascading bandit models lies in our adoption of the MNL model, as opposed to the linear model assumed by the existing literature. This model choice introduces a dependence on the parameter  $\kappa$  within the regret bound of UCB-CCA, which is an aspect we address and refine in subsequent sections. It is essential to highlight that our work tackles a more general problem yet achieves improved bounds concerning the key problem parameters previously considered suboptimal. The factor comprised of the length of the cascade,  $(K / (K + 1))^{K + 1}$ , in Theorem 4.3 is also notable, which is bounded above by 1 regardless of the value of  $K$ . Consequently, the regret bound does not increase polynomially with  $K$ , ensuring scalability.
4.3 Proof Outline
In this subsection, we present the proof sketch of Theorem 4.3. One of the key components of the regret analysis that enables carving off the dependence on  $K$  is the following lemma.
Lemma 4.4 (Maximal Lipschitz continuity). Suppose  $u_{ti} \geq w_{ti}^{*}$  for all  $i \in [N]$ . Then
$$
f \left(S _ {t}, u _ {t}\right) - f \left(S _ {t}, w _ {t} ^ {*}\right) \leq \left(\frac {K}{K + 1}\right) ^ {K + 1} \max  _ {A _ {t k} \in S _ {t}} \max  _ {i \in A _ {t k}} \left(u _ {t i} - w _ {t i} ^ {*}\right). \tag {3}
$$
Lemma 4.4 demonstrates that the difference between the reward functions under the UCB parameter and the true parameter is upper bounded by the maximal difference between the UCB and true parameters. This implies that the regret bound remains unaffected by increases in the cascade length  $K$  or the assortment size  $M$ .
Eliminating the dependence on  $p^*$  is another key element of our analysis. To this end, we first show that the order of assortments in the cascade model with the disjunctive objective does not affect the expected reward. We formalize this property in the following lemma.
Lemma 4.5. Let  $p_k$  be the probability that the user clicks on any item in  $A_k$ . Given a collection of assortments  $\{A_1, \dots, A_K\}$  with probabilities  $\{p_1, \dots, p_K\}$ , their order of display does not matter. Further, for every permutation  $\rho : [K] \to [K]$ , we have
$$
\sum_ {k \in [ K ]} p _ {k} \prod_ {\dot {k} <   k} (1 - p _ {\dot {k}}) = 1 - \prod_ {k \in [ K ]} (1 - p _ {k}) = \sum_ {k \in [ K ]} p _ {\rho^ {- 1} (k)} \prod_ {\dot {k} <   k} \left(1 - p _ {\rho^ {- 1} (\dot {k})}\right).
$$
Consolidating these key results, we proceed to bound the cumulative regret. We begin by leveraging the monotonicity of the expected reward function and the definition of the  $\alpha$ -approximate optimization oracle to bound the cumulative regret.
$$
\begin{array}{l} \mathcal {R} ^ {\alpha} (T) \leq \mathbb {E} \left[ \sum_ {t = 1} ^ {T} f \left(S _ {t}, u _ {t}\right) - f \left(S _ {t}, w _ {t} ^ {*}\right) \right] \leq C _ {K} \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \max  _ {A _ {t k} \in S _ {t}} \max  _ {i \in A _ {t k}} \left(u _ {t i} - w _ {t i} ^ {*}\right) \right] \\ \leq 2 C _ {K} \mathbb {E} \left[ \beta_ {T} \sum_ {t = 1} ^ {T} \max  _ {A _ {t k} \in S _ {t}} \max  _ {i \in A _ {t k}} | | x _ {t i} | | _ {V _ {t - 1} ^ {- 1}} \right] = 2 C _ {K} \mathbb {E} \left[ \beta_ {T} \sum_ {t = 1} ^ {T} \max  _ {k \in [ O _ {t} ]} \max  _ {i \in A _ {t k}} | | x _ {t i} | | _ {V _ {t - 1} ^ {- 1}} \right]. \tag {4} \\ \end{array}
$$
The second inequality is from Lemma 4.4, letting  $C_K \coloneqq (K / (K + 1))^{K + 1}$ . The third inequality is given by the concentration of the UCB weights (see Lemma B.5). Note that the assortment including the item with the largest value of  $||x_{ti}||_{V_t^{-1}}$  is always examined by the user since it is included in the first assortment of  $S_t$  by the optimistic exposure swapping technique as described in Section 3.2. Note that a change in the order of assortments incurred by the optimistic exposure swapping does not affect the expected reward which is shown in Lemma 4.5. Hence, for every round  $t \in [T]$ , we obtain
$$
\max  _ {A _ {t k} \in S _ {t}} \max  _ {i \in A _ {t k}} \| x _ {t i} \| _ {V _ {t - 1} ^ {- 1}} = \max  _ {k \in [ O _ {t} ]} \max  _ {i \in A _ {t k}} \| x _ {t i} \| _ {V _ {t - 1} ^ {- 1}}. \tag {5}
$$
Therefore, the last equality in Eq. (4) is given by Eq. (5). Then, we can apply the maximal version of elliptical potential lemma (see Lemma B.8) to bound the cumulative regret.
5 Improved Dependence on  $\kappa$
While UCB-CCA achieves the regret bound of  $\tilde{\mathcal{O}}\left(\frac{1}{\kappa} d\sqrt{T}\right)$  improving dependence on  $K$ , the bound includes the problem-dependent constant  $\kappa$ . This implies a potential risk of the regret bound becoming large when  $\kappa$  becomes very small. In order to circumvent this challenge, we propose a new optimism in the face of uncertainty (OFU) algorithm, UCB-CCA+, which exhibits a regret bound that is independent of  $\kappa$  in the leading term. The pseudocode of UCB-CCA+ is detailed in Algorithm 2
5.1 Algorithm: UCB-CCA+
5.1.1 Confidence Set
UCB-CCA+ computes a regularized MLE  $\hat{\theta}_t$ , following the same procedure described in Section 3.3. Then, the algorithm constructs a new confidence set centered around  $\hat{\theta}_t$  utilizing a Bernstein-type tail inequality for self-normalized martingales [11; 3]. Nevertheless, a simple adaptation of the previous approaches may incur increased dependence on  $M$ . Hence, a more intricate analysis and refined algorithmic strategy are imperative to effectively address this challenge.
First, we define  $g_{t}(\theta) \coloneqq \sum_{\tau=1}^{t} \sum_{k \in [O_{\tau}]} \sum_{i \in A_{\tau k}} p_{\tau}(i|A_{\tau k}, w_{\tau}) x_{\tau i} + \lambda_{t} \theta$  where  $w_{t} = (w_{ti})_{i \in [N]}$  and  $w_{ti} = x_{ti}^{\top} \theta$ . We also denote the partial derivative of  $p_{t}(i|A_{tk}, w_{t})$  with respect to  $w_{ti}$  as  $\dot{p}_{t}(i|A_{\tau k}, w_{\tau}) \coloneqq p_{t}(i|A_{tk}, w_{t}) p_{t}(i_{0}|A_{tk}, w_{t})$ . Additionally, we define the new design matrix containing local information, denoted as  $H_{t}(\theta) \coloneqq \sum_{\tau=1}^{t} \sum_{k \in [O_{\tau}]} \sum_{i \in A_{\tau k}} \dot{p}_{\tau}(i|A_{\tau k}, w_{\tau}) x_{\tau i} x_{\tau i}^{\top} + \lambda_{t} I_{d}$ , and, for convenience,  $H_{t} \coloneqq \sum_{\tau=1}^{t} \sum_{k \in [O_{\tau}]} \sum_{i \in A_{\tau k}} \dot{p}_{\tau}(i|A_{\tau k}, w_{\tau}^{*}) x_{\tau i} x_{\tau i}^{\top} + \lambda_{t} I_{d}$ . Then, the algorithm constructs a confidence set as below:
$$
B _ {t} (\delta) := \left\{\theta \in \mathbb {R} ^ {d}: | | g _ {t} \left(\hat {\theta} _ {t}\right) - g _ {t} (\theta) | | _ {H _ {t} ^ {- 1} (\theta)} \leq \gamma_ {t} (\delta) \right\} \tag {6}
$$
Algorithm 2 UCB-CCA+
Input: confidence radius  $\gamma_{t}$  and ridge penalty parameter  $\lambda \geq 1$
1: for  $t = 1, \dots, T$  do
2: Observe  $x_{ti}$  for all  $i \in [N]$
3: Construct a confidence set  $B_{t-1}(\delta)$  as defined in Eq.(6).
4: Compute a candidate cascade  $(S_t' = (A_{tk}')_{k\in [K]},\theta_t) = \arg \max_{S\in S,\theta \in B_t(\delta)}f(S,w_t)$
5: Find optimistic exposure assortments  $L_{t,H}$  and  $L_{t,V}$  (and their positions  $h$  and  $v$ ) in  $S_t'$
6:  $S_{t}\gets (A_{tk})_{k\in [K]}$  where  $A_{tk} = \left\{ \begin{array}{ll}L_{t,H} & \mathrm{if} k = 1\\ L_{t,V} & \mathrm{if} k = 2\\ A_{t1}^{\prime} & \mathrm{if} k = h\\ A_{t2}^{\prime} & \mathrm{if} k = v\\ A_{tk}^{\prime} & \mathrm{otherwise} \end{array} \right.$
7: Offer  $S_{t}$  and observe  $O_{t},y_{t} = (y_{tk})_{k\in [O_{t}]}$
8: Update  $H_{t}\gets H_{t - 1} + \sum_{k = 1}^{O_{t}}\sum_{i\in A_{tk}}\dot{p}_{t}(i|A_{tk},w_{t}^{*})x_{ti}x_{ti}^{\top}$
9: Update  $V_{t} \gets V_{t-1} + \sum_{k=1}^{O_{t}} \sum_{i \in A_{tk}} x_{ti} x_{ti}^{\top}$
0: Compute the regularized MLE  $\hat{\theta}_t$  by solving  $\nabla_{\theta}\left[\ell_t(\theta) + \frac{\lambda}{2} ||\theta||_2^2\right] = \mathbf{0}$
1: end for
where the confidence radius  $\gamma_t(\delta)$  is suitably specified to ensure that the true parameter  $\theta^*$  lies in the confidence set  $B_t(\delta)$  with high probability. On the event of  $\theta^* \in B_t(\delta)$ , The following lemma bounds the weighted  $\ell_2$ -norm of the difference between  $\theta$  and  $\theta^*$ .
Lemma 5.1. Suppose  $\theta^{*}\in B_{t}(\delta)$ . Then, for any  $\theta \in B_t(\delta)$ , we have  $||\theta -\theta^{*}||_{H_t}\leq 6\gamma_t(\delta)$
5.1.2 Doubly Optimistic Exposure Swapping
UCB-CCA+ still faces the challenge outlined in Section 2.4.2. The complication is exacerbated for UCB-CCA+ as the algorithm concurrently updates two gram matrices,  $H_{t}$  and  $V_{t}$ , which only contain the information of the observed items. Building upon the technique of optimistic exposure swapping detailed in Section 3.2, in each round  $t$ , UCB-CCA+ assigns the assortment  $L_{t,H}$  — containing the item with the largest uncertainty with respect to  $H_{t-1}$  among the top  $KM$  items — to the first slot of cascade  $S_{t}$ . Similarly, the algorithm places  $L_{t,V}$  — with the item that has the largest uncertainty with respect to  $V_{t-1}$  — in the second slot of  $S_{t}$ .
5.2 Regret Analysis of UCB-CCA+
Theorem 5.2 (Regret upper bound of UCB-CCA+). Suppose Assumptions [4.1] and [4.2] hold, and we run UCB-CCA+ for total  $T$  rounds with  $\lambda \geq 1$  and  $\gamma_t(\delta) \coloneqq \frac{3\sqrt{\lambda_t}}{2} + \frac{2}{\sqrt{\lambda_t}} \log \left( \frac{(\lambda_t + KMt / d)^{d/2} \lambda_t^{-d/2}}{\delta} \right) + \frac{2d}{\sqrt{\lambda_t}} \log 2$  with  $\delta = \frac{1}{t^2}$ . Then, the regret of UCB-CCA+ is upper-bounded by
$$
\mathcal {R} ^ {\alpha} (T) \leq C _ {1} \gamma_ {T} (\delta) \sqrt {2 d T \log \left(1 + \frac {K M T}{d \lambda}\right)} + \frac {C _ {2}}{\kappa} \gamma_ {T} (\delta) ^ {2} d \log \left(1 + \frac {K M T}{d \lambda}\right)
$$
where  $C_1 = 36$  and  $C_2 = 216(1 + Me)$
Discussion of Theorem 5.2. Theorem 5.2 establishes the regret bound of  $\tilde{\mathcal{O}}(d\sqrt{T})$ . The leading term of the regret bound is independent of  $\kappa$ . Although the second term exhibits dependence on  $\kappa$ , the term only scales logarithmically in  $T$ , whose comparative effect diminishes as  $t$  increases compared to the leading term. Hence, the worst-case regret guarantee of UCB-CCA+ improves from that of UCB-CCA. The comprehensive proof of Theorem 5.2 is provided in the appendix.
6 Approximation Algorithm for Optimal Combinatorial Action
In this section, we show that computing the optimal cascade is, in general, a weakly NP-hard problem and give a (fast) polynomial time algorithm that the agent can use to compute a 0.5-approximation to
the optimal cascade for any weight  $w$ . We assume that the MNL weights are given and consider the problem of finding the cascade that maximizes the expected reward. This is an optimization problem on selecting a sequence of  $K$  assortments with size  $M$  each from a ground set of  $N$  items. The number of feasible cascades is  $O\left(\binom{N}{M}^{K}\right)$ . In fact, we establish the following hardness result.
Lemma 6.1. For general  $M$ , the optimization problem is weakly NP-hard even for  $K = 2$ .
The hardness of our problem follows from the hardness result of [19] for a related setting of unconstrained cascade optimization where the size of each assortment can be arbitrary. In light of this hardness, we turn our attention to finding fast approximation algorithms for the problem. While there has been a lot of recent work on the unconstrained cascade optimization problem (see [19; 12; 10] and the references therein), none of the previous algorithms apply to our constrained setting (where the size of each assortment is  $M$ ).
We consider the following greedy approach for the problem. Consider arbitrary weights at round  $t$ , i.e.  $w_{t} = (w_{ti})_{i\in [N]}$ , is given. We order the items in  $[N]$  in decreasing order of given weights and consider the following cascade of assortments.
$$
D _ {1} = \{1, 2, \dots , M \}, D _ {2} = \{M + 1, M + 2, \dots , 2 M \}, \dots , D _ {K} = \{(K - 1) M + 1, \dots , K M \}
$$
We call these assortments "decreasing order assortments". Let OPT denote the value of the optimal solution to the problem. We establish that showing these assortments in any possible sequence gives a good approximation to the problem.
Lemma 6.2. Let OPT denote the overall click probability in the optimal solution. The decreasing order assortments  $D_{1},\ldots ,D_{K}$ , shown in any order, have overall click probability at least 0.5 OPT.
We present the proof in Appendix E. We also show that our algorithm is optimal for  $M = 1$ , which captures the classic cascade optimization problem.
Figure 2:  $N = 10$ ,  $K = 2$ ,  $M = 2$ , and  $d = 5$ .
Figure 3:  $N = 15$ ,  $K = 2$ ,  $M = 2$ , and  $d = 10$ .
In this section, we evaluate the performances of our proposed algorithms UCB-CCA and UCB-CCA+ in numerical experiments and compare their performances with the existing combinatorial bandit algorithms CombCascade and  $C^3$ -UCB. For simulations, we generate a random sample of the unknown time-invariant parameter  $\theta^{*}$  from  $\mathcal{N}(0,1)$  at the beginning of the simulation. We sample  $N$  feature vectors from  $\mathcal{N}(0,1)$  in each round  $t$ . At each round  $t$ , the oracle computes a sequence of assortments in decreasing order, forming a cascade  $S_t$  based on given  $w_t$ . We assess the cumulative regret of UCB-CCA, UCB-CCA+, CombCascade, and  $C^3$ -UCB. Note that a user's choice for UCB-CCA and UCB-CCA+ is determined by the MNL logit choice model, whereas  $C^3$ -UCB utilizes a linear model and CombCascade is a non-contextual model. Figure 2 and Figure 3 indicate that both UCB-CCA and UCB-CCA+ significantly outperform  $C^3$ -UCB and CombCascade. We also observe that UCB-CCA+ shows a slight performance advantage over UCB-CCA, although the difference is not statistically significant. While UCB-CCA+ has a sharper worst-case regret guarantee, UCB-CCA can provide favorable practical performances, with simpler implementation and computational efficiency.
This work was supported by the National Research Foundation of Korea(NRF) grant funded by the Korea government(MSIT) (RS-2023-00222663, No. 2022R1C1C1006859, No. 2022R1A4A103057912, No. 2021M3E5D2A01024795) and by Creative-Pioneering Researchers Program through Seoul National University and by Naver.
[20] Xutong Liu, Jinhang Zuo, Siwei Wang, John CS Lui, Mohammad Hajiesmaili, Adam Wierman, and Wei Chen. Contextual combinatorial bandits with probabilistically triggered arms. In International Conference on Machine Learning, pages 22559-22593. PMLR, 2023.
[21] Daniel McFadden. Modeling the choice of residential location. Transportation Research Record, (673), 1978.
[22] Min-hwan Oh and Garud Iyengar. Thompson sampling for multinomial logit contextual bandits. Advances in Neural Information Processing Systems, 32, 2019.
[23] Min-hwan Oh and Garud Iyengar. Multinomial logit contextual bandits: Provable optimality and practicality. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 9205-9213, 2021.
[24] Noemie Perivier and Vineet Goyal. Dynamic pricing and assortment under a contextual mnl demand. Advances in Neural Information Processing Systems, 35:3461-3474, 2022.
[25] Daniel Vial, Sujay Sanghavi, Sanjay Shakkottai, and R Srikant. Minimax regret for cascading bandits. arXiv preprint arXiv:2203.12577, 2022.
[26] Kun Wang. Conservative contextual combinatorial cascading bandit. IEEE Access, 9:151434-151443, 2021.
[27] Xue Wang, Mike Mingcheng Wei, and Tao Yao. Online assortment optimization with high-dimensional data. Available at SSRN 3521843, 2019.
[28] Zixin Zhong, Wang Chi Chueng, and Vincent YF Tan. Thompson sampling algorithms for cascading bandits. The Journal of Machine Learning Research, 22(1):9915-9980, 2021.
[29] Shi Zong, Hao Ni, Kenny Sung, Nan Rosemary Ke, Zheng Wen, and Branislav Kveton. Cascading bandits for large-scale recommendation problems. In Proceedings of the 32nd Conference on Uncertainty in Artificial Intelligence, 2016.