SlowGuess/NeurIPS_2023_OCR huggingface dataset:
and  $\mathbf{z} \in \mathcal{Z}$  for each  $m$ , and sample them following the  $E$ -optimal design in step 12 of Algorithm 2. We also show in Lemma 16 (Appendix A.6) that  $\sigma_{\mathrm{min}}\left(\sum_{\widetilde{\mathbf{w}} \in \widetilde{\mathcal{W}}} \mathbf{b}_{\widetilde{\mathbf{w}}} \widetilde{\mathbf{w}}^{\top}\right) > 0$  which enables us to sample following  $E$ -optimal design. Then use the estimator in (9). Then in Lemma 19 we show that  $\|\widehat{\mathbf{S}}_{m,\ell} - \mu^{*} \mathbf{S}_{m,*}\|_F^2 \leq C_1 k_1 k_2 r \log \left(\frac{2(k_1 + k_2)}{\delta_\ell}\right) / \tau_m^E$  holds with probability greater than  $(1 - \delta)$ . Also, note that in the second phase by setting  $\widetilde{\tau}_{m,\ell}^E = \sqrt{8 k_1 k_2 r \log(4\ell^2 |\mathcal{W}| / \delta_\ell)} / S_r$  and sampling each arm  $\overline{\mathbf{w}} \in \overline{\mathcal{W}}$  exactly  $\lceil \widehat{\mathbf{b}}_{\ell, \overline{\mathbf{w}}}^E \widetilde{\tau}_{m,\ell}^E \rceil$  times we are guaranteed that  $\|\theta_{k+1:p}^*\|_2 = O(k_1 k_2 r / \widetilde{\tau}_{m,\ell}^E)$  in the  $\ell$ -th phase. Summing up over  $\ell = 1$  to  $\lceil \log_2(4\Delta^{-1}) \rceil$  across each task  $M$  we get that the total samples complexity of the second stage is bounded by  $\widetilde{O}(M \sqrt{k_1 k_2 r} / S_r)$ .
Step 4 (Convert to  $k_{1}k_{2}$  bilinear bandits): Once GOBLIN recovers  $\widehat{\mathbf{S}}_m,\tau_\ell^E$  it rotates the arm set following (10) to build  $\underline{\mathcal{W}}_m$  to get the  $k_{1}k_{2}$  bilinear bandits. The rest of the steps follow the same way as in steps 2, 3 and 4 of proof of Theorem 1.
Step 5 (Total Samples): We show the total samples in the third phase are bounded by  $O\left(\frac{k}{\gamma_{\mathcal{Y}}^{2}} \log \left( \frac{k \log_{2}(\Delta^{-1}) |\mathcal{W}|}{\delta} \right) \lceil \log_{2}(\Delta^{-1}) \rceil \right)$  where the effective dimension  $k = (k_{1} + k_{2})r$ . The total samples of phase  $\ell$  is given by  $\tau_{\ell}^{E} + \sum_{m} (\widetilde{\tau}_{m,\ell}^{E} + \tau_{m,\ell}^{G})$ . Finally, we get the total sample complexity by summing over all phases from  $\ell = 1$  to  $\lceil \log_2(4\Delta^{-1}) \rceil$ . The claim of the theorem follows by noting  $\widetilde{O}(k / \gamma_{\mathcal{Y}}^2) \leq \widetilde{O}(k / \Delta^2)$ .
In this section, we conduct proof-of-concept experiments on both single and multi-task bilinear bandits. In the single-task experiment, we compare against the state-of-the-art RAGE algorithm (Fiez et al., 2019). We show in Figure 1 (left) that GOBLIN requires fewer samples than the RAGE with an increasing number of arms. In the multi-task experiment, we compare against the state-of-the-art DouExpDes algorithm (Du et al., 2023). We show in Figure 1 (right) that GOBLIN requires fewer samples than DouExpDes with an increasing number of tasks. As experiments are not a central contribution, we defer a fuller description of the experimental set-up to Appendix A.8.
Figure 1: (Left) Single-task experiment: results show the number of samples required to identify the optimal action pair for differing numbers of actions. (Right) Multi-task experiment: results show the number of samples required to identify the optimal action pair for varying numbers of tasks.
5 Conclusions and Future Directions
In this paper, we formulated the first pure exploration multi-task representation learning problem. We introduce an algorithm, GOBLIN that achieves a sample complexity bound of  $\widetilde{O}((d_1 + d_2)r / \Delta^2)$  which improves upon the  $\widetilde{O}((d_1d_2) / \Delta^2)$  sample complexity of RAGE (Fiez et al., 2019) in a single-task setting. We then extend GOBLIN for multi-task pure exploration bilinear bandit problems by learning latent features which enables sample complexity that scales as  $\widetilde{O}(M(k_1 + k_2)r / \Delta^2)$  which improves over the  $\widetilde{O}(M(k_1k_2) / \Delta^2)$  sample complexity of DouExpDes (Du et al., 2023). Our analysis opens an exciting opportunity to analyze representation learning in the kernel and neural bandits (Zhu et al., 2021; Mason et al., 2021). We can leverage the fact that this type of optimal design does not require the arm set to be an ellipsoid (Du et al., 2023) which enables us to extend our analysis to non-linear representations.
Mukherjee, S., Xie, Q., Hanna, J., and Nowak, R. (2023). Speed: Experimental design for policy evaluation in linear heteroscedastic bandits. arXiv preprint arXiv:2301.12357.
Pukelsheim, F. (2006). Optimal design of experiments. SIAM.
Reyes, L. J. P., Oviedo, N. B., Camacho, E. C., and Calderon, J. M. (2021). Adaptable recommendation system for outfit selection with deep learning approach. IFAC-PapersOnLine, 54(13):605-610.
Rockafellar, R. (2015). Convex analysis. *princeton landmarks in mathematics and physics*.
Shamir, O. (2011). A variant of azuma's inequality for martingales with subgaussian tails. arXiv preprint arXiv:1110.2392.
Shen, Q., Han, S., Han, Y., and Chen, X. (2023). User review analysis of dating apps based on text mining. Plos one, 18(4):e0283896.
Soare, M., Lazaric, A., and Munos, R. (2014). Best-arm identification in linear bandits. Advances in Neural Information Processing Systems, 27.
Stein, C., Diaconis, P., Holmes, S., and Reinert, G. (2004). Use of exchangeable pairs in the analysis of simulations. Lecture Notes-Monograph Series, pages 1-26.
Tripuraneni, N., Jin, C., and Jordan, M. (2021). Provable meta-learning of linear representations. In International Conference on Machine Learning, pages 10434-10443. PMLR.
Valko, M., Munos, R., Kveton, B., and Kocák, T. (2014). Spectral bandits for smooth graph functions. In International Conference on Machine Learning, pages 46-54. PMLR.
Yang, J., Hu, W., Lee, J. D., and Du, S. S. (2020). Impact of representation learning in linear bandits. arXiv preprint arXiv:2010.06531.
Yang, J., Lei, Q., Lee, J. D., and Du, S. S. (2022). Nearly minimax algorithms for linear bandits with shared representation. arXiv preprint arXiv:2203.15664.
Zhang, D., Shen, D., Initiative, A. D. N., et al. (2012). Multi-modal multi-task learning for joint prediction of multiple regression and classification variables in alzheimers disease. NeuroImage, 59(2):895-907.
Zhao, Y., Kosorok, M. R., and Zeng, D. (2009). Reinforcement learning design for cancer clinical trials. Statistics in medicine, 28(26):3294-3315.
Zhu, Y., Zhou, D., Jiang, R., Gu, Q., Willett, R., and Nowak, R. (2021). Pure exploration in kernel and neural bandits. Advances in neural information processing systems, 34:11618-11630.