SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Parameter Influence. From the regret bound (10), we can observe that a larger batch size  $B$  results in a larger first term (of order  $O(B)$ , called policy error) but a smaller second term (of order  $O(1 / B)$ , called sketching error), indicating that a suitable batch size  $B$  needs to be set. This conclusion was empirically verified in Figure 3(e), where  $B = 1,000$  ( $C_B = 25$ ) yields better empirical performance in terms of the average reward. Similar phenomenon can also be observed on Criteo dataset and commercial product. All of the results verified the theoretical results in Remark 4:  $B = C_B\sqrt{T / d} = C_B^2 N / d$  is a suitable choice while setting  $C_B \approx 25$ . From the results in Figure 3(f) we observe that, for our SPUIR and SPUIR-RS, the performances significantly increased when the sketch size  $c$  reached  $10\% B$  ( $\approx d\log d$ ), which demonstrates the conclusion in Remark 4 that only the sketch size of order  $c = \Omega(d\log d)$  is needed for satisfactory performance.
This paper presents a computationally efficient reward imputation approach for contextual batched bandits that addresses the challenge of partial-information feedback in real-world applications. The proposed approach mimics the reward generation mechanism of the environment, approximating full-information feedback. It reduces time complexity using sketching, achieves a relative-error bound for approximation, and exhibits regret with controllable bias and reduced variance. The theoretical formulation and algorithmic implementation may provide an efficient reward imputation scheme for online learning under limited feedback.
This work was funded by the National Key R&D Program of China (2022ZD0114802), the National Natural Science Foundation of China (62376275, U2001212, 62377044), Intelligent Social Governance Interdisciplinary Platform, Major Innovation & Planning Interdisciplinary Platform for the "Double-First Class" Initiative, Renmin University of China. Supported by fund for building world-class universities (disciplines) of Renmin University of China. Supported by Public Computing Cloud, Renmin University of China. Supported by the Fundamental Research Funds for the Central Universities, and the Research Funds of Renmin University of China (23XNKJ13).
Nelson, J. and Nguyen, H. L. (2013). OSNAP: Faster numerical linear algebra algorithms via sparser subspace embeddings. In Proceedings of the 54th Annual Symposium on Foundations of Computer Science, pages 117-126.
Neu, G. and Olkhovskaya, J. (2020). Efficient and robust algorithms for adversarial linear contextual bandits. In Proceedings of the 33rd Conference on Learning Theory, pages 3049-3068.
Perchet, V., Rigollet, P., Chassang, S., and Snowberg, E. (2016). Batched bandit problems. The Annals of Statistics, 44(2):660-681.
Rahimi, A. and Recht, B. (2008). Weighted sums of random kitchen sinks: Replacing minimization with randomization in learning. In Advances in Neural Information Processing Systems 21, pages 1313-1320.
Ren, Z. and Zhou, Z. (2020). Dynamic batch learning in high-dimensional sparse linear contextual bandits. arXiv preprint arXiv:2008.11918.
Shalev-Shwartz, S. (2011). Online learning and online convex optimization. Foundations and Trends in Machine Learning, 4(2):107-194.
Valko, M., Korda, N., Munos, R., Flaounas, I. N., and Cristianini, N. (2013). Finite-time analysis of kernelised contextual bandits. In Proceedings of the 29th Conference on Uncertainty in Artificial Intelligence.
Wang, C. and Cheng, G. (2020). Online batch decision-making with high-dimensional covariates. In Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics, pages 3848-3857.
Woodruff, D. P. (2014). Sketching as a tool for numerical linear algebra. Foundations and Trends in Theoretical Computer Science, 10(1-2):1-157.
Yang, J., Hu, W., Lee, J. D., and Du, S. S. (2021). Impact of representation learning in linear bandits. In Proceedings of the 9th International Conference on Learning Representations.
Yom-Tov, E., Feraru, G., Kozdoba, M., Mannor, S., and Hochberg, I. (2017). Encouraging physical activity in patients with diabetes: Intervention using a reinforcement learning system. Journal of Medical Internet Research, 19(10):e338.
Zhang, K. W., Janson, L., and Murphy, S. A. (2020). Inference for batched bandits. In Advances in Neural Information Processing Systems 33, pages 9818-9829.
Zhang, X., Dai, S., Xu, J., Dong, Z., Dai, Q., and Wen, J. (2022). Counteracting user attention bias in music streaming recommendation via reward modification. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 2504-2514.
Zhang, X., Jia, H., Su, H., Wang, W., Xu, J., and Wen, J. (2021). Counterfactual reward modification for streaming recommendation with delayed feedback. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 41-50.
Zhang, X. and Liao, S. (2019). Incremental randomized sketching for online kernel learning. In Proceedings of the 36th International Conference on Machine Learning, pages 7394-7403.
Zhou, Z. (2023). Stream efficient learning. CoRR, abs/2305.02217.