## 5 Evaluation
Figure 5: The comparison of models for each dataset. (a) PUCK; (b) ARAS (house A); (c) ARAS (house B); (d) ADL. \begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline \multirow{2}{*}{**Policy**} & \multirow{2}{*}{**Note**} & \multicolumn{8}{c}{**Hyperparameters**} \\ \cline{3-8}  & & **Beta\_PRIOR** & **Alpha Smoothing** & **Decay Refit\_BUFFER** & **Active\_CHOIC** & **Deoxy\_TYPE** \\ \hline LinUCB [30] & LinUCB policy stores a square matrix, which has dimension equal to total numbers of features for the fitted model. & None & 0.1 & - & - & - & - & - \\ \hline AdaptiveGreedy [40] & It focuses on taking the action that has the highest reward. & None & - & (1,2) & 0.9997 & - & - & percentile \\ \hline AdaptiveGreedy(Active) & It is the same for AdaptiveGreedy but with different hyperparameters & None & 0.9997 & - & weighted & percentile \\ \hline SoftmaxExplorer [40] & It depends on softmax function to select the action & None & - & (1,2) & - & 50 & - & - \\ \hline ActiveExplorer [40] & It depends on an active learning heuristic for taking & ((3./nchocises, 4), 2) & - & None & - & 50 & - & - \\ the action & & & & & & & & \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Tuning hyperparameters for the OSL model policies. Figure 6: The Reward Delay Periods \(T_{r}\) = 5 s. (a) PUCK; (b) ARAS (house A); (c) ARAS (house B); (d) ADL. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline \multirow{2}{*}{**Dataset**} & \multicolumn{5}{c}{**Policies**} \\ \cline{2-6}  & **LinUCB** & **Adaptive Active Greedy** & **Adaptive Greedy** & **Softmax Explorer** & **Active Explorer** \\  & **(OSL)** & **(OLS)** & **(OSL)** & **(SGD)** & **(SGD)** \\ \hline PUCK & 0.68 & 0.64 & 0.63 & **0.79** & 0.65 \\ \hline ARAS & & & & & \\ House (A) & 0.80 & 0.81 & 0.77 & **0.85** & 0.69 \\ \hline ARAS & & & & & \\ House (B) & **0.92** & 0.91 & 0.91 & 0.90 & 0.75 \\ \hline ADL & **0.99** & **0.99** & **0.99** & 0.97 & 0.83 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: The cumulative mean reward of our system among three datasets. \begin{table}
\begin{tabular}{c c c c c c c} \multicolumn{6}{c}{**Policies**} \\ \multirow{2}{*}{**Dataset**} & \multirow{2}{*}{**The Reward Delay Period**} & \multirow{2}{*}{\begin{tabular}{c} **LinUCB** \\ **(OSL)** \\ \end{tabular} } & \multirow{2}{*}{\begin{tabular}{c} **Adaptive Active Greedy** \\ **(OLS)** \\ \end{tabular} } & \multirow{2}{*}{\begin{tabular}{c} **Adaptive Active Greedy** \\ **(OSL)** \\ \end{tabular} } & \multirow{2}{*}{\begin{tabular}{c} **Adaptive Greedy** \\ **(OSL)** \\ \end{tabular} } & \multirow{2}{*}{\begin{tabular}{c} **Softmax Explorer** \\ **(SGD)** \\ \end{tabular} } & \multirow{2}{*}{\
\begin{tabular}{c} **Active Explorer** \\ **(SGD)** \\ \end{tabular} } \\ \hline \multirow{4}{*}{PUCK} & 5 s & 0.68 & 0.64 & 0.63 & 0.79 & 0.65 \\ \cline{2-6}  & 10 s & 0.74 & 0.55 & 0.65 & 0.75 & 0.60 \\ \cline{2-6}  & 15 s & 0.62 & 0.52 & 0.48 & 0.70 & 0.58 \\ \cline{2-6}  & 5 s & 0.80 & 0.81 & 0.77 & **0.55** & 0.69 \\ \cline{2-6}  & 10 s & 0.68 & 0.65 & 0.58 & **0.72** & 0.56 \\ \cline{2-6}  & 15 s & 0.68 & **0.72** & 0.68 & 0.60 & 0.49 \\ \hline \multirow{4}{*}{
\begin{tabular}{c} **AR s** \\ **(B)** \\ \end{tabular} } & 5 s & **0.92** & 0.91 & 0.91 & 0.90 & 0.75 \\ \cline{2-6}  & 10 s & 0.76 & 0.76 & 0.78 & **0.79** & 0.66 \\ \cline{2-6}  & 15 s & 0.71 & 0.65 & 0.76 & **0.74** & 0.61 \\ \cline{2-6}  & 5 s & **0.99** & **0.99** & **0.99** & 0.97 & 0.83 \\ \cline{2-6}  & 10 s & **0.99** & **0.99** & **0.99** & 0.98 & 0.83 \\ \cline{2-6}  & 15 s & **0.95** & **0.95** & **0.95** & 0.90 & 0.78 \\ \hline \end{tabular}
\end{table} 
Table 4: The cumulative mean reward of our system among three datasets using different reward delay periods. Figure 7: The Reward Delay Periods \(T_{r}=10\) s. **(a)** PUCK; **(b)** ARAS(house A); **(c)** ARAS (house B); **(d)** ADL.