## 6 Experiment and Results
BLR with action centering specifically does not require the knowledge of the baseline features at decision time (See Appendix C.1.1). This means that baseline features only need to be available at update time and we can incorporate more features that were not available in real time at the decision time. \begin{table}
\begin{tabular}{l l l l l} \hline \hline \multicolumn{5}{c}{**RL Algorithm Candidates**} \\ \hline  & \multicolumn{4}{c}{Average Rewards} \\ \hline RL Algorithm & S\_Het & NS\_Het & S\_Pop & NS\_Pop \\ \hline ZIP \(k=1\) & 100.038 (0.597) & 102.566 (0.526) & 107.184 (0.626) & 109.379 (0.552) \\ ZIP \(k=4\) & 100.463 (0.586) & 103.035 (0.539) & 108.217 (0.609) & 110.242 (0.562) \\ ZIP \(k=N\) & 100.791 (0.596) & 103.391 (0.546) & 108.410 (0.617) & 110.542 (0.554) \\ BLR \(k=1\) & 97.196 (0.585) & 99.691 (0.527) & 103.692 (0.615) & 105.590 (0.546) \\ BLR \(k=4\) & 99.772 (0.590) & 102.310 (0.547) & 107.568 (0.619) & 109.454 (0.547) \\ BLR \(k=N\) & **101.267 (0.590)** & **104.024 (0.542)** & **108.974 (0.610)** & **111.201 (0.546)** \\ \hline  & \multicolumn{4}{c}{25 th Percentile Rewards} \\ \hline RL Algorithm & S\_Het & NS\_Het & S\_Pop & NS\_Pop \\ \hline ZIP \(k=1\) & 67.907 (1.150) & 73.830 (0.403) & 74.898 (1.016) & 78.651 (0.556) \\ ZIP \(k=4\) & 68.865 (1.067) & 73.836 (0.464) & 75.933 (1.114) & 80.413 (0.629) \\ ZIP \(k=N\) & 69.448 (1.201) & 74.580 (0.475) & 76.312 (1.122) & 80.424 (0.648) \\ BLR \(k=1\) & 65.600 (1.139) & 70.703 (0.457) & 70.915 (1.024) & 74.782 (0.596) \\ BLR \(k=4\) & 68.045 (1.122) & 73.322 (0.505) & 75.766 (1.097) & 79.809 (0.622) \\ BLR \(k=N\) & **69.757 (1.171)** & **75.393 (0.427)** & **77.272 (1.096)** & **81.675 (0.583)** \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Average and 25 th Percentile Rewards. Average and 25 th Percentile Rewards are defined in Section 3.1. The naming convention for environment variants is found in Table 1. “k” refers to the cluster size. Average rewards are averaged across time, users, and 100 trials. For the 25 th percentile rewards, we average rewards for each user across time, find the lower 25 th percentile across \(N=72\) users, and then average that across 100 trials. The value in the parenthesis is the standard error of the mean. The best performing algorithm candidate in each environment variant is bolded. BLR (\(k=N\)) performs better than other algorithm candidates across all simulated environments. Notice that the average rewards are lower than the 120-s dentist-recommended brushing duration. This is because of the zero-inflated nature of our setting (i.e., the user does not brush). 2. **Cluster Size:** RL algorithms with larger cluster sizes \(k\) perform better overall, especially for simulation environments with population-level treatment effects (rather than heterogeneous treatment effects). At first glance, one might think that algorithms with smaller cluster sizes may perform better because they can learn more personalized policies for users (especially in environments with heterogeneous treatment effects). Interestingly, though, the algorithms with larger cluster sizes performed better across all environments in terms of average reward (average across users and time) and the 25 th-percentile of the average reward (average over time) across users (Table 2); this means that the RL algorithm candidates with larger cluster sizes performed better for both the average user and for the worst off users. The better performance of algorithms with larger cluster sizes is likely due to their ability to reduce noise and learn  
Figure 3: Average User Rewards Over Time. Above, we show simulation results of the six candidate algorithms (BLR and ZIP respectively for different cluster sizes \(k\)) across the four simulation environments. The y-axis is the mean and \(\pm 1.96\cdot\) standard error of the average user rewards (\(R=\frac{1}{72}\sum_{i=1}^{72}\frac{1}{t_{b}}\sum_{s=1}^{t_{b}}R_{is}\)) for decision times \(t_{0}\in[20,40,60,80,100,120,140]\) across 100 Monte Carlo simulated trials.