SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 1: The boxplot of the discounted return over 50 repeated experiments.
experiments, we set the hyper-parameters satisfying the condition  $\mathcal{O}\left(\frac{n^{1/4}}{d\log(V\sqrt{n})}\right)$ . Figure 1 shows that our algorithm almost consistently outperforms competing methods in different settings. This performance mainly benefits from the advantages exposed in our theoretical analysis, such as model extrapolation enhancement, relaxation of completeness-type assumptions on function approximation, etc. The only exception is the slightly poorer performance compared to COMBO in a high exploration setting, where COMBO may learn a good dynamic model with relatively sufficient exploration. We provide the experiment details in Appendix due to page limit.
Benchmark data. We evaluate our proposed approach on the D4RL benchmark of OpenAI Gym locomotion (walker2d, hopper, halfcheetah) and Maze2D tasks [15], which encompasses a variety
of dataset settings and domains and positions our algorithm within the existing baselines. We take the results of COMBO, OptiDICE and ATAC from their original papers for Gym locomotion, and run COMBO and ATAC using author-provided implementations for Maze2D. The results of BCQ, BEAR methods from the D4RL original paper. In addition, CQL, IQL and TD3+BC are re-run to ensure a fair evaluation process for all tasks. As shown in Table 1, the proposed algorithm achieves the best performance in 7 tasks and is comparable to the baselines in the remaining tasks. In addition to the evaluation of the policy performance, we also conduct sensitivity analyses on the hyperparameter-tuning and study the regret rate of convergence.
Table 1: The normalized score of the policy at the last iteration of training, averaged over 5 random seeds. The highest performing scores are highlighted. The med, med-rep, and med-exp is shorthand for medium, medium-replay, and medium-expert, respectively.
Tasks	Proposed	COMBO	BCQ	BEAR	OptiDICE	ATAC	CQL	IQL	TD3+BC
walker2d-med	80.8 ± 5.1	81.9 ± 2.8	53.1	59.1	21.8 ± 7.1	89.6	77.2 ± 4.2	78.3 ± 4.3	81.7 ± 2.3
hopper-med	94.9 ± 4.3	97.2 ± 2.2	54.5	52.1	94.1 ± 3.7	85.6	74.3 ± 5.8	66.3 ± 6.4	98.4 ± 1.6
halfcheetah-med	58.1 ± 1.4	54.2 ± 1.5	40.7	41.7	38.2 ± 0.1	53.3	37.2 ± 0.3	47.4 ± 1.1	27.8 ± 0.7
walker2d-med-rep	99.6 ± 2.9	56.0 ± 8.6	15.0	19.2	21.6 ± 2.1	92.5	20.8 ± 1.6	73.9 ± 2.8	34.4 ± 4.2
hopper-med-rep	113.0 ± 2.1	89.5 ± 1.8	33.1	33.7	36.4 ± 1.1	102.5	32.6 ± 1.9	94.7 ± 1.5	44.4 ± 3.7
halfcheetah-med-rep	49.3 ± 2.1	55.1 ± 1.0	38.2	38.6	39.8 ± 0.3	48.0	41.9 ± 1.1	44.2 ± 2.5	48.3 ± 0.7
walker2d-med-exp	108.2 ± 7.4	103.3 ± 5.6	57.5	40.1	74.8 ± 9.2	114.2	103.8 ± 6.9	109.6 ± 7.0	100.5 ± 8.9
hopper-med-exp	117.8 ± 1.9	111.1 ± 2.9	110.9	96.3	111.5 ± 0.6	119.2	111.4 ± 1.2	91.5 ± 2.2	112.4 ± 0.3
halfcheetah-med-exp	98.5 ± 3.8	90.0 ± 5.6	64.7	53.4	91.1 ± 3.7	94.8	66.7 ± 8.9	86.7 ± 3.6	95.9 ± 3.9
walker2d-random	11.2 ± 3.8	7.0 ± 3.6	4.9	7.3	9.9 ± 4.3	6.8	4.7 ± 1.5	5.8 ± 2.8	3.4 ± 1.7
hopper-random	18.7 ± 1.5	17.9 ± 1.4	10.6	11.4	11.2 ± 1.1	17.5	10.7 ± 0.1	10.8 ± 0.6	11.1 ± 0.2
halfcheetah-random	37.6 ± 2.4	38.8 ± 3.7	2.2	25.1	11.6 ± 1.2	3.9	26.7 ± 1.4	22.4 ± 1.8	26.1 ± 1.8
maze2d-umaze	96.5 ± 27.8	34.2 ± 8.6	12.8	3.4	111.0 ± 8.3	84.4 ± 24.8	50.5 ± 7.9	41.5 ± 4.7	13.8 ± 22.8
maze2d-med	137.5 ± 18.9	49.9 ± 13.9	8.3	29.0	145.2 ± 17.5	152.3 ± 34.6	28.6 ± 9.2	38.5 ± 4.2	59.1 ± 44.7
maze2d-large	187.8 ± 15.2	128.2 ± 17.3	6.2	4.6	155.7 ± 33.4	142.1 ± 33.8	46.2 ± 16.2	54.2 ± 18.1	87.6 ± 15.4
Real-world application. The Ohio Type 1 Diabetes (OhioT1DM) dataset [33] comprises a cohort of patients with Type-1 diabetes, where each patient exhibits different dynamics and 8 weeks of life-event data, including health status measurements and insulin injection dosages. Clinicians aim to adjust insulin injection dose levels [33, 4] based on a patient's health status in order to maintain glucose levels within a specific range for safe dose recommendations. The state variables consist of health status measurements, and the action space is a bounded insulin dose range. The glycemic index serves as a reward function to assess the quality of dose suggestions. Since the data-generating process is unknown, we follow [31, 29] to utilize the Monte Carlo approximation of the estimated value function on the initial state of each trajectory to evaluate the performance of each method. The mean and standard deviation of the improvements on the Monto Carlo discounted returns are presented in Table 2. As a result, our algorithm achieves the best performance for almost all patients, except for Patient 552. The main reason for the desired performance in real data is from the enhanced model extrapolation and relaxed function approximation requirements and outperforms the competing methods. This finding is consistent with the results in the synthetic and benchmark datasets, demonstrating the potential applicability of the proposed algorithm in real-world environments.
Table 2: The baseline policy improvements over 50 repeated experiments in the OhioT1DM dataset.
Patient ID	Proposed	COMBO	BCQ	BEAR	OptiDICE	ATAC	CQL	IQL	TD3+BC
596	6.5 ± 1.1	4.1 ± 0.8	3.8 ± 0.9	2.7 ± 1.1	4.7 ± 1.1	5.1 ± 2.0	4.6 ± 0.6	3.4 ± 0.7	4.8 ± 1.3
584	33.1 ± 1.8	27.0 ± 1.3	20.3 ± 1.2	22.9 ± 1.6	27.7 ± 1.9	26.9 ± 2.6	21.6 ± 1.2	22.7 ± 1.3	22.4 ± 1.7
567	36.9 ± 1.3	30.6 ± 2.0	24.3 ± 1.4	25.6 ± 1.4	28.8 ± 2.2	29.7 ± 2.8	26.5 ± 1.4	25.8 ± 1.4	27.8 ± 1.5
552	7.9 ± 0.9	6.8 ± 0.7	5.7 ± 0.5	5.0 ± 0.8	8.1 ± 0.9	7.2 ± 1.5	6.7 ± 0.4	6.1 ± 0.5	7.4 ± 0.8
544	13.2 ± 1.9	9.8 ± 1.5	7.5 ± 2.5	5.9 ± 0.8	10.3 ± 1.8	10.1 ± 2.1	8.7 ± 1.0	7.8 ± 0.9	9.7 ± 0.8
540	20.4 ± 0.5	17.5 ± 0.9	14.3 ± 0.6	12.7 ± 0.5	17.9 ± 0.9	18.2 ± 1.4	16.5 ± 0.5	14.0 ± 0.6	17.1 ± 0.8
We study offline RL with limited exploration in function approximation settings. We propose a bi-level policy optimization framework, which can be further solved by a computationally practical penalized adversarial estimation algorithm, offering strong theoretical and empirical guarantees. Regarding limitations and future work, while the penalized adversarial estimation is more computationally efficient than the previously constrained problem, it may still be more challenging to solve than single-stage optimization problems. Another future direction is to explore environments with unobservable confounders. It will be interesting to address these limitations in future works.
8 Acknowledgments
The author is grateful to the five anonymous reviewers and the area chair for their valuable comments and suggestions.
[1] Alekh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. Optimality and approximation with policy gradient methods in markov decision processes. In Conference on Learning Theory, pages 64-66. PMLR, 2020.
[2] András Antos, Csaba Szepesvári, and Rémi Munos. Learning near-optimal policies with bellman-residual minimization based fitted policy iteration and a single sample path. Machine Learning, 71(1):89-129, 2008.
[3] Leemon Baird. Residual algorithms: Reinforcement learning with function approximation. In Machine Learning Proceedings 1995, pages 30-37. Elsevier, 1995.
[4] Jiansong Bao, Heather R Gilbertson, Robyn Gray, Diane Munns, Gabrielle Howard, Peter Petocz, Stephen Colagiuri, and Jennie C Brand-Miller. Improving the estimation of mealtime insulin dose in adults with type 1 diabetes: the normal insulin demand for dose adjustment (nidda) study. Diabetes Care, 34(10):2146-2151, 2011.
[5] Amir Beck. First-order methods in optimization. SIAM, 2017.
[6] Stephen Boyd, Stephen P Boyd, and Lieven Vandenberghe. Convex Optimization. Cambridge University Press, 2004.
[7] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540, 2016.
[8] Jinglin Chen and Nan Jiang. Information-theoretic considerations in batch reinforcement learning. In International Conference on Machine Learning, pages 1042-1051. PMLR, 2019.
[9] Jinglin Chen and Nan Jiang. Offline reinforcement learning under value and density-ratio realizability: the power of gaps. In Uncertainty in Artificial Intelligence, pages 378-388. PMLR, 2022.
[10] Ching-An Cheng, Tengyang Xie, Nan Jiang, and Alekh Agarwal. Adversarily trained actor critic for offline reinforcement learning. In International Conference on Machine Learning, pages 3852-3878. PMLR, 2022.
[11] Euisun Choi and Chulhee Lee. Feature extraction based on bhattacharyya distance. Pattern Recognition, 36(8):1703-1709, 2003.
[12] Yaqi Duan, Zeyu Jia, and Mengdi Wang. Minimax-optimal off-policy evaluation with linear function approximation. In International Conference on Machine Learning, pages 2701-2709. PMLR, 2020.
[13] Damien Ernst, Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research, 6:503-556, 2005.
[14] Amir-massoud Farahmand, Mohammad Ghavamzadeh, Csaba Szepesvári, and Shie Mannor. Regularized policy iteration with nonparametric function spaces. The Journal of Machine Learning Research, 17(1):4809-4874, 2016.
[15] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
[16] Drew Fudenberg and Jean Tirole. Game theory. MIT press, 1991.
[17] Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132-20145, 2021.
[54] Tengyang Xie and Nan Jiang. Q* approximation schemes for batch reinforcement learning: A theoretical comparison. In Conference on Uncertainty in Artificial Intelligence, pages 550-559. PMLR, 2020.
[55] Lin Yang and Mengdi Wang. Reinforcement learning in feature space: Matrix bandit, kernels, and regret bound. In International Conference on Machine Learning, pages 10746-10756. PMLR, 2020.
[56] Ming Yin, Yu Bai, and Yu-Xiang Wang. Near-optimal offline reinforcement learning via double variance reduction. Advances in neural information processing systems, 34:7677-7688, 2021.
[57] Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimization. Advances in neural information processing systems, 34:28954-28967, 2021.
[58] Andrea Zanette, Martin J Wainwright, and Emma Brunskill. Provable benefits of actor-critic methods for offline reinforcement learning. Advances in neural information processing systems, 34:13626-13640, 2021.
[59] Wenhao Zhan, Baihe Huang, Audrey Huang, Nan Jiang, and Jason Lee. Offline reinforcement learning with realizability and single-policy concentrability. In Conference on Learning Theory, pages 2730–2775. PMLR, 2022.
[60] Wenzhuo Zhou, Yuhan Li, Ruoqing Zhu, and Annie Qu. Distributional shift-aware off-policy interval estimation: A unified error quantification framework. arXiv preprint arXiv:2309.13278, 2023.
[61] Wenzhuo Zhou, Ruoqing Zhu, and Annie Qu. Estimating optimal infinite horizon dynamic treatment regimes via pt-learning. Journal of the American Statistical Association, pages 1-14, 2022.