## 6 Experiment and Results  
We evaluate the RL algorithm candidates in each of the environment variants (stability). Specifically, the RL algorithm candidates will be comprised of a posterior sampling algorithm with two different reward models: (i) a Bayesian linear regression model (BLR) and (ii) a zero-inflated Poisson regression model (ZIP); see Appendices B and C for more discussion of these algorithms. Additionally, for each of these two reward approximating functions, we will consider different cluster sizes (with \(k=1,4\), \(N\) users). Each cluster will have one RL algorithm instantiation per cluster (no data shared across clusters). We cluster users by their entry date into the study (e.g., the first \(k\) users are in the first cluster, the next \(k\) users are in the second cluster, and so on). Since for the real Oralytics study we will incrementally recruit users into the study at a rate of about four users per week, for our experiments, we also simulate four users entering the study every week. To simulate a study, we draw \(N=72\) users (approximately the expected sample size for the Oralytics study) with replacement and cluster them by their entry date into the simulated study. The algorithms for each cluster are updated weekly with the first update taking place after one week (at decision time \(t=14\) for each cluster). To evaluate personalization, we use the following metrics to compare algorithms: average rewards (average across users and time) and the 25 th percentile of average rewards (averaged over time) across users. The purpose of looking at the 25 th percentile of average rewards across users is to evaluate how well the algorithms perform on the worst off users (i.e., users who have a lower average reward than the average user). We ran 100 Monte Carlo trials for each environmental variant and algorithm candidate pairing. Table 2 shows the average and the 25 th percentile of users' average (across time) rewards. Figure 3 shows the average reward over time. We highlight the following takeaways from our experiments:  
1. **BLR vs. ZIP:** We prefer BLR to ZIP. BLR with cluster size \(k=N\) results in higher user rewards than all other RL algorithm candidates in all environments in terms of average reward and 25 th percentile reward (Table 2) and for average reward across all user decision times (Figure 3). It is interesting to note that BLR with cluster size \(k=4\) performs comparably to ZIP for all cluster sizes \(k\) (Table 2, Figure 3). Originally, we hypothesized that ZIP would perform better than BLR because the ZIP-based algorithms can better model the zero-inflated nature of the rewards. We believe that \begin{table}
\begin{tabular}{p{113.8 pt} p{113.8 pt}} \hline \hline \multicolumn{1}{p{113.8 pt}}{**S\_Pop:** Stationary Base Model, Population Effect Size} & \multicolumn{1}{p{113.8 pt}}{**NS\_Pop:** Nonstationary Base Model, Population Effect Sizes} \\ \hline \multicolumn{1}{p{113.8 pt}}{**S\_Het:** Stationary Base Model, Heterogeneous Effect Size} & \multicolumn{1}{p{113.8 pt}}{**NS\_Het:** Nonstationary Base Model, Heterogeneous Effect Sizes} \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Four Environment Variants. We consider two environment base models (stationary and nonstationary) and two effect sizes (population effect size, heterogeneous effect size). the ZIP-based algorithms suffered in performance because they require fitting more parameters and thus require more data to learn effectively. On the other hand, the BLR model trades off bias and variance more effectively in our data-sparse settings. Beyond considerations of their ability to personalize, we also prefer the BLR-based RL algorithms because they have an easy-to-compute closed-form posterior update (computability and stability). The ZIP-based algorithms involve using approximate posterior sampling, which is more computationally intensive and numerically unstable. In addition, BLR with action centering is robust, namely, it is guaranteed to be unbiased even when the baseline reward model is incorrect [1].