SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Note that we can always learn an  $\varepsilon$ -optimal policy for any context-lumpable bandit within  $\widetilde{O}(SK / \varepsilon^2)$  samples by invoking any existing near-optimal algorithm for finite contextual bandits. As a result, by combining Theorem 2 with the  $\widetilde{O}(SK / \varepsilon^2)$  upper bound, we obtain that  $\widetilde{O}(\min\{r(S + K), SK\} / \varepsilon^2)$  samples suffice for learning any context-lumpable bandit, which according to the following theorem is minimax optimal up to a logarithmic factor.
Theorem 3. Learning an  $\varepsilon$ -optimal policy for a context-lumpable bandit with probability no smaller than  $1/2$  requires at least  $\Omega(\min\{r(S + K), SK\}/\varepsilon^2)$  samples in the worst case.
4 Regret Minimization in Context-lumpable Bandits
In this section, we extend the idea from the PAC setting to the online setting. To better introduce the key ideas, we first consider a special case when both context and block distributions are uniform (Section 4.1). Then we consider the most general case in Section 4.2.
4.1 Special Case: Uniform Context and Block Distribution
In this section, we assume that distributions  $\nu$  and  $\omega$  are uniform, and thus,  $g$  evenly splits the contexts into  $r$  blocks so that there are  $S / r$  contexts in each block and every context appears with the same probability at each timestep. We will relax these assumptions and consider the general case in the next subsection.
For this case, we introduce Algorithm 4, which uses phased elimination in combination with a clustering procedure. The algorithm runs in phases  $h = 1,2,\ldots$  that are specified by error tolerance parameter  $\varepsilon_{h} = 2^{-h / 2}$ . Like phased elimination algorithms for multi-armed bandits, we need to ensure at phase  $h$  actions the algorithms play are all  $\widetilde{O} (\varepsilon_h)$ -optimal. Thus, at the end of each phase  $h$ , we eliminate all actions that are not  $\widetilde{O} (\varepsilon_h)$ -optimal. However, the set of  $\widetilde{O} (\varepsilon_h)$ -optimal actions of each block is different. Therefore, we also perform clustering on contexts and perform elimination for each subset of the partition. Specifically, we maintain a partition of contexts  $\mathcal{P}_h$  at each phase  $h$  and initialize  $\mathcal{P}_1 = \{[S]\}$ . For each cluster  $\mathcal{C}\in \mathcal{P}_h$ , we maintain a set of good arms  $\mathrm{GOOD}_h(\mathcal{C})$  which we will prove is  $\widetilde{O} (\varepsilon_h)$ -optimal for contexts in the cluster with high probability.
We use Algorithm 2 to collect data similar to Algorithm 1 (Line 6). At phase  $h$ , we try to estimate mean reward up to error  $\widetilde{O}(\varepsilon_h)$  with probability  $1 - \delta_h$ . The difference is that we assume  $\omega$  is uniform, so we don't need different accuracy levels  $n$ , which will be required for the algorithm that handles the general case. Also, for every context  $i \in \mathcal{C}$ , we only explore arms good for now, that is, in  $\mathrm{GOOD}_h(\mathcal{C})$  instead of exploring all the arms  $[K]$ . This reflects that in the online setting, we need to minimize regret and cannot afford to explore bad arms too many times.
Based on the data we collect during the exploration stage, we then check if there is a large gap across contexts in the same subset for any arms (Line 7). A large gap suggests that (i) the subset contains at least two blocks (ii) the mean reward of the arm is significantly different in these blocks and we can use this arm to partition the contexts by running Algorithm 5 (clustering stage). We repeatedly do clustering (Line 9) and split heterogeneous subsets (Line 10) until we cannot find a large gap within the same subset. If no large gap is detected, then each arm has similar mean rewards (up to error  $\widetilde{O}(\varepsilon_h)$ ) across all blocks in the same subset. Then we eliminate arms that are significantly worse than the empirical best arm (elimination stage) from  $\mathrm{GOOD}_h(\mathcal{C})$  for every subset  $\mathcal{C}$  and start a new episode (Line 14).
Algorithm 5 plays  $j$  for each context  $i \in \mathcal{C}$  for  $\widetilde{O}(1 / \varepsilon^2)$  rounds and calculates empirical means of arm  $j$  for each context in  $\mathcal{C}$ . It then sorts the contexts by the empirical means and performs clustering (Line 4). Specifically, the algorithm enumerates contexts in descending order of empirical means and splits contexts until a large gap between consecutive values is detected (Line 7). As we call Algorithm 5 only if a large gap is detected, we prove that in the appendix it correctly separates the subset into at least two parts without putting any contexts in the same block into different parts by setting  $\varepsilon' = \varepsilon_h / r$ .
Algorithm 4 Algorithm for Uniform Block Distribution
1 Initialize  $\mathcal{P}_1\gets \{[S]\}$  ,  $\mathrm{GOOD}_1(\mathcal{C})\leftarrow [K]$  for  $\mathcal{C}\in \mathcal{P}_1$    
2 Let  $t$  denote the current time   
3 for phase  $h = 1,2,\ldots$  , do   
4 Let  $\varepsilon_h\gets 2^{-h / 2},\delta_h\gets \varepsilon_h^2 /(r^3 SK),\widetilde{\lg}_h\gets 64\log (rSK / \delta_h),\widetilde{\varepsilon}_h\gets \sqrt{\widetilde{\lg}_h}\cdot \varepsilon_h$  Step 1. Data collection   
5 Define  $L_{h}\gets \frac{r(S + K)\widetilde{\lg}_{h}}{\varepsilon_{h}^{2}},n_{h}\gets \log (\frac{1}{\varepsilon_{h}^{2}}),\mathcal{K}_{h}(i)\gets \mathrm{GOOD}_{h}(\mathcal{C}),\mathcal{C}\in \mathcal{P}_{h},\mathcal{C}\ni i,\forall i\in [S]$    
6 Execute Algorithm 2 with input  $L_{h},n_{h},\kappa_{h}$  , and receive  $\mathcal{D}_h$  and  $\widehat{A}_h$  Step 2. Test homogeneity and perform clustering on heterogeneous subsets   
7 while  $\exists \mathcal{C}\in \mathcal{P}_h,\bar{i},\underline{i}\in \mathcal{C},j\in [K]$  such that  $\widehat{A}_h(\bar{i},j) - \widehat{A}_h(\underline{i},j)\geq \widetilde{\varepsilon}_h$  do   
8 Define  $\varepsilon^{\prime}\gets \frac{\varepsilon_{h}}{4r},\delta^{\prime}\gets \frac{\delta_{h}}{r},\mathcal{K}(i)\gets \{j\}$  if  $i\in \mathcal{C}$  else  $\mathrm{GOOD}_h(\mathcal{C})$  for  $\mathcal{C}\in \mathcal{P}_h,\mathcal{C}\ni i$    
9 Execute Algorithm 5 with input  $\varepsilon^{\prime},\delta^{\prime},\mathcal{K},\mathcal{C}$  , and  $j$  , and get  $\mathcal{P}$  , a partition of C   
10 Initialize  $\mathrm{GOOD}_h(\mathcal{C}')\gets \mathrm{GOOD}_h(\mathcal{C}),\forall \mathcal{C}'\in \mathcal{P}$  and update  $\mathcal{P}_h\gets (\mathcal{P}\cup \mathcal{P}_h)\backslash \{\mathcal{C}\}$  Step 3. Eliminate suboptimal actions in each subset   
11 P+h+1 P+h+1 for C E P+h+1 do   
13 Calculate  $\overline{\mu}_h(\mathcal{C},j)\gets \max_{i:i\in \mathcal{C},(i,j)\in \mathcal{D}_h}\widehat{A}_h(i,j),\forall j\in \mathrm{GOOD}_h(\mathcal{C})$  Update  $\mathrm{GOOD}_{h + 1}(\mathcal{C})\gets \{j:j\in \mathrm{GOOD}_h(\mathcal{C}),\max_{j'}\overline{\mu}_h(\mathcal{C},j') - \overline{\mu}_h(\mathcal{C},j)\leq 2\widetilde{\varepsilon}_h\}$
Similar to the analysis of other phased elimination algorithms, we have to show that in a phase specified by error level  $\varepsilon_{h}$ , with high probability, (i) the optimal arm is not eliminated and (ii) all  $\widetilde{\omega} (\varepsilon_h)$ -suboptimal arms are eliminated, that is, all arms in  $\mathrm{GOOD}_h(\mathcal{C})$  for all  $\mathcal{C}$  are  $\widetilde{O} (\varepsilon_h)$ -optimal. We show the final regret here and defer the details to Appendix D
Theorem 4. Under the assumption that context distribution  $\nu$  and block distribution  $\omega$  are uniform, regret of Algorithm 6 is bounded as  $Reg_{T} = \widetilde{O} (\sqrt{r^{3}(S + K)T})$ .
Algorithm 5 Split Contexts Into Multiple Blocks
1 Input: error  $\varepsilon^{\prime}$ , confidence  $\delta^{\prime}$ , exploration sets  $\mathcal{K}$ , subset  $\mathcal{C}$ , separating arm  $j$
2 Initialize  $\widetilde{\lg} \gets 64\log(S/\delta')$ ,  $L' \gets S\widetilde{\lg}/\varepsilon'^2$ ,  $n' \gets \lceil 1/\varepsilon'^2 \rceil$
3 Execute Algorithm 2 with input  $L^{\prime}$ $n^\prime ,\mathcal{K},$  and reassign output to  $\mathcal{D}$  and  $\widehat{A}$
4 Sort contexts in  $\mathcal{C}$  and label them as  $i_1, \ldots, i_{|\mathcal{C}|}$  so that  $\widehat{A}(i_1, j) \geq \widehat{A}(i_2, j) \geq \dots \geq \widehat{A}(i_{|\mathcal{C}|}, j)$ .
5 Initialize  $\mathcal{P}_1\gets \{i_1\}$ $b\gets 1$
6 for  $k = 2,3,\ldots ,|\mathcal{C}|$  do
7 if  $\widehat{A} (i_{k - 1},j) - \widehat{A} (i_k,j)\geq \sqrt{\lg}\cdot \varepsilon '$  then   
8 Update  $b\gets b + 1$  and initialize  $\mathcal{P}_b\gets \{\}$    
9  $\mathcal{P}_b\gets \mathcal{P}_b\cup \{i_k\}$
10 Output  $\{\mathcal{P}_1,\dots ,\mathcal{P}_b\}$
Compared to our PAC result, we get an extra dependency on  $r$ . This is because we pay  $\widetilde{O}(S / \varepsilon'^2) = \widetilde{O}(r^2 S / \varepsilon_h^2)$  samples to do clustering instead of  $\widetilde{O}(1 / \varepsilon_h^2)$  in order to ensure a "perfect" partition, that is, we never separate contexts in the same block with high probability. This is crucial for our phase-elimination algorithm as we may call Algorithm 5 in different phases. We left getting better than  $\widetilde{O}(\sqrt{r^3(S + K)T})$  regret upper bounds or better than  $\Omega(\sqrt{r(S + K)T})$  regret lower bounds (even for this uniform special case) as an important future direction.
4.2 Non-uniform Context and Block Distribution
Similar to the PAC learning setting, we can use Algorithm 3 to reduce general context distributions to (nearly) uniform ones. We provide more details in Appendix F. As a result, without loss of generality, we may assume  $\nu$  is almost-uniform, and we focus on how to handle non-uniform block distribution  $\omega$ . In the extreme, there may only be one context for some blocks, which becomes challenging to estimate their mean rewards.
For this case, we introduce Algorithm 6. Intuitively, based on Algorithm 4, we can further employ different accuracy levels as Algorithm 1. However, as our goal becomes minimizing regret, it is difficult to control regret for smaller  $n$  and larger blocks. Specifically, for smaller  $n$ , we explore more actions (with fewer samples) for each context in a single phase. Since fewer samples are used, we may unavoidably play suboptimal actions and suffer large regret. This becomes worse for a large block as more contexts in the block suffer large regret. We note that this is not a problem in the PAC setting because we only need to control sample complexity.
We fix the issue by setting different lengths  $L$  for different accuracy levels. Recall in Algorithm 1, we use the same length  $L = \widetilde{O}(r(S + K) / \varepsilon^2)$  for every  $n$ . Intuitively, since we allow less accurate estimations for smaller  $n$ , we may use fewer data. It turns out indeed we can set  $L = \widetilde{O}(r(S + K)2^{(n + h) / 2})$  for level  $n$  at phase  $h$ , and with more refined analysis, it provides similar guarantees as before. More importantly, since smaller  $n$  uses (exponentially) fewer samples, the overall regret is well controlled.
In addition to setting the lengths sophisticatedly, we also need to carefully maintain sets of good actions  $\mathrm{GOOD}_{h,n}$  not only at each phase  $h$  but also at each accuracy level  $n$ . The partition of contexts  $\mathcal{P}_h$ , however, only evolves with phases and is shared between different levels at the same phase. Similar to Algorithm 4, we also do clustering (Step 2) and elimination (Step 3) but do the procedures in parallel for every accuracy level  $n$ . In the clustering stage, we use different thresholds to define a "large gap". This reflects that  $n$  represents different accuracy levels and thus has different widths of confidence intervals. For the elimination stage, we enforce the inclusion relation  $\mathrm{GOOD}_h(n,\mathcal{C}) \subseteq \mathrm{GOOD}_h(n',\mathcal{C})$  for  $n' \leq n$  (Line 17), which will be useful in the regret analysis. We now present the main theorem and put the complete proof in Appendix G.
Theorem 5. Regret of Algorithm 6 is bounded as  $\operatorname{Reg}_T = \widetilde{O}(\sqrt{r^3(S + K)T})$ .
We remark that Algorithm 6 is anytime, that is, it doesn't require the time horizon  $T$  as input. However, it does require the number of blocks  $r$  as prior knowledge. Removing the knowledge of  $r$  is an interesting future direction. One promising idea is to apply a doubling trick on  $r$ . Specifically, we have an initial guess  $r = \widetilde{O}(1)$  and run Algorithm 6; when  $|\mathcal{P}_h| > r$ , we double  $r$  and restart
Algorithm 6 Algorithm for Non-uniform Block Distribution  
1 Initialize  $\mathcal{P}_1\gets \{[S]\}$  ,  $\mathrm{GOOD}_{1,1}(\mathcal{C})\leftarrow [K],\mathcal{C}\in \mathcal{P}_1$    
2 Let  $t$  denote the current time   
3 for phase  $h = 1,2,\ldots$  , do   
4 Let  $\varepsilon_h\gets 2^{-h / 2},N_h\gets h,\delta_h\gets \varepsilon_h^2 /(r^3 SK),\widetilde{\lg}_h\gets 128\log (rSKN_h / \delta_h)$  Step 1. Data collection   
5 for accuracy level  $n = 1,\dots ,N_{h}$  do   
6 Define  $L_{h,n}\gets r(S + K)\widetilde{\lg}_h2^{(n + h) / 2},\mathcal{K}_{h,n}(i)\gets \mathrm{GOOD}_{h,n}(\mathcal{C}),\mathcal{C}\in \mathcal{P}_h,\mathcal{C}\ni i,\forall i\in [S]$    
7 Execute Algorithm 2 with input  $L_{h},n,\mathcal{K}_{h,n}$  , and receive  $\mathcal{D}_{h,n}$  and  $\widehat{A}_{h,n}$    
Step 2. Test homogeneity and perform clustering on heterogeneous subsets   
8 while  $\exists \mathcal{C}\in \mathcal{P}_h,\bar{i},\underline{i}\in \mathcal{C},j\in [K],n\in [N_h]$  such that  $\widehat{A}_{h,n}(\bar{i},j) - \widehat{A}_{h,n}(\underline{i},j)\geq \sqrt{\frac{\widetilde{\lg}_h}{2^n}}$  do   
9 Define  $\varepsilon^{\prime}\gets \frac{\varepsilon_{h}}{4r},\delta^{\prime}\gets \frac{\delta_{h}}{r},\mathcal{K}(i)\gets \{j\}$  if  $i\in \mathcal{C}$  else  $\mathrm{GOOD}_{h,n}(\mathcal{C})$  for  $\mathcal{C}\in \mathcal{P}_h,\mathcal{C}\ni i$    
10 Execute Algorithm 5 with input  $\varepsilon^{\prime},\delta^{\prime},\mathcal{K},\mathcal{C}$  , and  $j$  , and get  $\mathcal{P}$  , a partition of  $\mathcal{C}$    
11 Initialize  $\mathrm{GOOD}_{h,n}(\mathcal{C}')\gets \mathrm{GOOD}_{h,n}(\mathcal{C}),\forall \mathcal{C}'\in \mathcal{P}$  and update  $\mathcal{P}_h\gets (\mathcal{P}\cup \mathcal{P}_h)\backslash \{\mathcal{C}\}$    
Step 3. Eliminate suboptimal actions in each subset   
12  $\mathcal{P}_{h + 1}\gets \mathcal{P}_h$  ,  $\mathrm{GOOD}_{1,1}(\mathcal{C})\gets [K],\mathcal{C}\in \mathcal{P}_{h + 1}$    
for accuracy level  $n = 2,\ldots ,N_{h}$  do   
14 for  $\mathcal{C}\in \mathcal{P}_{h + 1}$  do   
15 Calculate  $\overline{\mu}_{h,n}(\mathcal{C},j)\gets \max_{i:i\in \mathcal{C},(i,j)\in \mathcal{D}_{h,n}}\widehat{A}_{h,n}(i,j),\forall j\in \mathrm{GOOD}_{h,n}(\mathcal{C})$    
16 Let  $\mathcal{G}_{h,n}(\mathcal{C})\gets \Bigl \{j:j\in \mathrm{GOOD}_{h,n}(\mathcal{C}),\max_{j'}\overline{\mu}_{h,n}(\mathcal{C},j') - \overline{\mu}_{h,n}(\mathcal{C},j)\leq 2\sqrt{\frac{\widetilde{\lg}_h}{2^n}}\Bigr \}$    
17 Update  $\mathrm{GOOD}_{h + 1,n}(\mathcal{C})\gets \mathrm{GOOD}_{h + 1,n - 1}(\mathcal{C})\cap \mathcal{G}_{h,n}(\mathcal{C})$
the algorithm. The analysis, though, may be much more complicated. Finally, we note that when knowing  $r$ , one can calculate in advance if  $\sqrt{SKT}$  is less than  $\sqrt{r^3(S + K)T}$  and switch to a standard algorithm achieving  $\widetilde{O}(\sqrt{SKT})$  in that case. This modification guarantees the regret is never worse than the standard bound.
5 From Context-lumpable Bandits to Contextual Low-rank Bandits
Finally, we consider the more general contextual low-rank bandit problem. Specifically, we allow  $A$  to have rank  $r$ , that is  $A = UV$  for some  $S \times r$  matrix  $U$  and  $r \times K$  matrix  $V$ . Lumpability is a special case in the sense that  $U$  is binary where each row has a single nonzero element.
To solve the problem, We show a reduction from contextual low-rank bandits to context-lumpable bandits. Consider an  $\alpha$ -covering of rows of  $U$ , and notice that the covering number,  $\mathcal{R}_{\alpha}$ , is  $1 / \alpha^r$ . The context-lumpable bandits can be seen as  $\alpha$ -approximate context-lumpable bandits with  $\mathcal{R}_{\alpha}$  blocks, where the reward of contexts on the same block differs at most  $\alpha$ . Ignoring this misspecification, we may run Algorithm 1 and Algorithm 6 for the PAC and online settings, respectively. Moreover, it turns out that our algorithms are robust to this misspecification when  $\alpha$  is sufficiently small ( $O(\varepsilon)$  in the PAC setting, for example). Therefore, the sample complexity and the regret bounds of these algorithms will be in terms of  $\mathcal{R}_{\alpha}$  despite having an exponential dependency on  $r$ . The resulting bounds can still be smaller than the naive  $SK / \varepsilon^2$  and  $\sqrt{SKT}$  bounds in some scenarios, for example, when  $S$  and  $K$  are super large. We put the details in Appendix H.
6 Conclusions and Future Directions
We consider a contextual bandit problem with  $S$  contexts and  $K$  actions. Under the assumption that the context-action reward matrix has  $r \leq \min \{S, K\}$  unique rows, we show an algorithm that outputs an  $\varepsilon$ -optimal policy and has the optimal sample complexity of  $\widetilde{O}(r(S + K) / \varepsilon^2)$  with high
probability. In the regret minimization setting, we show an algorithm whose cumulative regret up to time  $T$  is bounded as  $\widetilde{O}(\sqrt{r^3(S + K)T})$ .
An immediate next question is whether a regret bound of order  $\widetilde{O}(\sqrt{r(S + K)T})$  is achievable in the regret minimization setting. A second open question is concerned with obtaining a  $\widetilde{O}(\sqrt{\mathrm{poly}(r)(S + K)T})$  regret bound in contextual low-rank bandits. Our regret analysis heavily relies on the assumption that contexts arrive in an I.I.D fashion. Extending our results to the setting with adversarial contexts remains another important future direction.
Yasin Abbasi-Yadkori, David Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. Advances in neural information processing systems, 24, 2011.
Sanjeev Arora, Rong Ge, and Ankur Moitra. Learning topic models—going beyond svd. In 2012 IEEE 53rd annual symposium on foundations of computer science, pages 1-10. IEEE, 2012.
P. Auer. Using confidence bounds for exploitation-exploration trade-offs. Journal of Machine Learning Research, 3:397-422, 2002.
Peter Auer, Nicolo Cesa-Bianchi, Yoav Freund, and Robert E Schapire. The nonstochastic multiarmed bandit problem. SIAM journal on computing, 32(1):48-77, 2002.
Alina Beygelzimer, John Langford, Lihong Li, Lev Reyzin, and Robert Schapire. Contextual bandit algorithms with supervised learning guarantees. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pages 19-26. JMLR Workshop and Conference Proceedings, 2011.
Leonardo Cella, Alessandro Lazaric, and Massimiliano Pontil. Meta-learning with stochastic linear bandits. *Arxiv*, 2020.
Yuxin Chen, Yuejie Chi, Jianqing Fan, Cong Ma, and Yuling Yan. Noisy matrix completion: Understanding statistical guarantees for convex relaxation via nonconvex optimization. SIAM journal on optimization, 30(4):3098-3121, 2020.
Wei Chu, Lihong Li, Lev Reyzin, and Robert Schapire. Contextual bandits with linear payoff functions. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pages 208-214. JMLR Workshop and Conference Proceedings, 2011.
Christoph Dann, Nan Jiang, Akshay Krishnamurthy, Alekh Agarwal, John Langford, and Robert E Schapire. On oracle-efficient PAC RL with rich observations. Advances in neural information processing systems, 31, 2018.
Simon Du, Akshay Krishnamurthy, Nan Jiang, Alekh Agarwal, Miroslav Dudik, and John Langford. Provably efficient RL with rich observations via latent state decoding. In International Conference on Machine Learning, pages 1665-1674. PMLR, 2019.
Yaqi Duan, Tracy Ke, and Mengdi Wang. State aggregation learning from Markov transition data. Advances in Neural Information Processing Systems, 32, 2019.
Miroslav Dudik, Daniel Hsu, Satyen Kale, Nikos Karampatziakis, John Langford, Lev Reyzin, and Tong Zhang. Efficient optimal learning for contextual bandits. In Proceedings of the Twenty-Seventh Conference on Uncertainty in Artificial Intelligence, pages 169-178, 2011.
Fei Feng, Ruosong Wang, Wotao Yin, Simon S Du, and Lin Yang. Provably efficient exploration for reinforcement learning using unsupervised learning. Advances in Neural Information Processing Systems, 33:22492-22504, 2020.
Dylan Foster and Alexander Rakhlin. Beyond ucb: Optimal and efficient contextual bandits with regression oracles. In International Conference on Machine Learning, pages 3199-3210. PMLR, 2020.
Dipendra Misra, Mikael Henaff, Akshay Krishnamurthy, and John Langford. Kinematic state abstraction and provably efficient rich-observation reinforcement learning. In International conference on machine learning, pages 6961-6971. PMLR, 2020.
Aditya Modi, Jinglin Chen, Akshay Krishnamurthy, Nan Jiang, and Alekh Agarwal. Model-free representation learning and exploration in low-rank MDPs. arXiv preprint arXiv:2102.07035, 2021.
Chengzhuo Ni, Anru R Zhang, Yaqi Duan, and Mengdi Wang. Learning good state and action representations via tensor decomposition. In 2021 IEEE International Symposium on Information Theory (ISIT), pages 1682-1687. IEEE, 2021.
Matteo Papini, Andrea Tirinzoni, Aldo Pacchiano, Marcello Restelli, Alessandro Lazaric, and Matteo Pirotta. Reinforcement learning in linear MDPs: Constant regret and representation selection. Advances in Neural Information Processing Systems, 34:16371-16383, 2021.
Zhiyuan Ren and B.H. Krogh. State aggregation in Markov decision processes. In Proceedings of the 41st IEEE Conference on Decision and Control, volume 4, pages 3819-3824, 2002.
Rajat Sen, Karthikeyan Shanmugam, Murat Kocaoglu, Alexandros G. Dimakis, and Sanjay Shakkottai. Contextual bandits with latent confounders: An NMF approach, 2016.
David Simchi-Levi and Yunzong Xu. Bypassing the monster: A faster and simpler optimal algorithm for contextual bandits under realizability. Mathematics of Operations Research, 47(3):1904-1931, 2022.
Aleksandrs Slivkins. Introduction to multi-armed bandits. Foundations and Trends® in Machine Learning, 12(1-2):1-286, 2019.
A. Tewari and S. A. Murphy. From ads to interventions: Contextual bandits in mobile health. In Mobile Health, pages 495-517. Springer, 2017.
Cindy Trinh, Emilie Kaufmann, Claire Vernade, and Richard Combes. Solving Bernoulli rank-one bandits with unimodal Thompson sampling. In Algorithmic Learning Theory, pages 862-889. PMLR, 2020.
Masatoshi Uehara, Xuezhou Zhang, and Wen Sun. Representation learning for online and offline RL in low-rank MDPs. In International Conference on Learning Representations, 2022.
Amy Zhang, Shagun Sodhani, Khimya Khetarpal, and Joelle Pineau. Learning robust state abstractions for hidden-parameter block MDPs. In International Conference on Learning Representations, 2020.
Weitong Zhang, Jiafan He, Dongruo Zhou, Amy Zhang, and Quanquan Gu. Provably efficient representation learning in low-rank Markov decision processes. arXiv preprint arXiv:2106.11935, 2021.
Xuezhou Zhang, Yuda Song, Masatoshi Uehara, Mengdi Wang, Alekh Agarwal, and Wen Sun. Efficient reinforcement learning in block MDPs: A model-free representation learning approach. In International Conference on Machine Learning, pages 26517-26547. PMLR, 2022.
A.1 From bilinear to low-rank bandits