# A Time-Series Model for Varying Worker Ability in Heterogeneous Distributed Computing Systems
## 4 Performance Evaluation
\begin{table}
\begin{tabular}{c c c c c c} \hline \multirow{2}{*}{\(N_{w}\)} & \multicolumn{2}{c}{**COR**} & \multicolumn{2}{c}{**MSE**} & \multicolumn{2}{c}{**MSE**} \\ \cline{2-7}  & **Mean** & **St. Dev.** & **Mean** & **St. Dev.** & **Mean** & **St. Dev.** \\ \hline
10 & 0.9967 & 0.0020 & 0.0065 & 0.0034 & 0.0615 & 0.0125 \\
30 & 0.9990 & 0.0006 & 0.0020 & 0.0007 & 0.0357 & 0.0069 \\
50 & 0.9993 & 0.0004 & 0.0016 & 0.0005 & 0.0319 & 0.0054 \\ \hline \end{tabular}
\end{table} 
Table 2: COR, MSE, and MAE results for the datasets with the cases \(N_{w}=10,30,50\). We present the mean and standard deviations of 30 repetitions for each case. \begin{table}
\begin{tabular}{c c c c c c c} \hline \multirow{2}{*}{\(N_{w}\)} & \multicolumn{2}{c}{**\(m\)**} & \multicolumn{2}{c}{**\(\sigma\)**} \\ \cline{2-7}  & **Mean** & **St. Dev.** & **Mean** & **St. Dev.** & **Mean** & **St. Dev.** \\ \hline
10 & 1.0149 & 0.3102 & 0.9720 & 0.2512 \\
30 & 1.0133 & 0.2538 & 0.9568 & 0.2086 \\
50 & 1.0382 & 0.2607 & 1.0395 & 0.1977 \\ \hline \end{tabular}
\end{table} 
Table 1: Parameter estimates for the datasets with the cases \(N_{w}=10,30,50\). The true parameter is set as \(m=1\), \(\sigma=1\). We present the mean and standard deviation of 30 repetitions for each case. where \(\hat{x}_{t}=\frac{1}{C}\sum_{i\in[G]}\hat{x}_{i,t}\) and \(\hat{x}_{t}=\frac{1}{C}\sum_{i\in[G]}x_{i,t}\) are the means of the estimated and true abilities at each time, respectively. The other two measures are represented by  
\[MSE(t) =\frac{1}{G}\sum_{i\in[G]}(\hat{x}_{i,t}-x_{i,t})^{2},\] \[MAE(t) =\frac{1}{G}\sum_{i\in[G]}|\hat{x}_{i,t}-x_{i,t}|,\]  
at each time \(t\in[T]_{0}\). We can see the errors of estimates tend to be uniform over time, implying that the ability estimates can be reliable at any time point. Figure 3 presents the true and estimated abilities for each group \(i\in[G]\) for the first dataset of the case \(N_{w}=50\). The true abilities tend to lie within the \(\pm 2\) standard deviation of the \(L=200\) particle sequences, implying that the presented algorithm estimates the true abilities very well. The results for groups 1 and 6 indicate that the estimates can capture increasing or decreasing patterns as well. Figure 3 on the right-hand side shows the analogous plot for the working rate given in (1), suggesting that the ability estimates are useful for estimating the working rates. Note that the estimated working rates are quite accurate for a broad range of working rates. In conclusion, the simulation result demonstrates that the estimation/inference works well for both model parameters and workers' abilities. In Figure 4, we compare the expected execution time of the uncoded scheme, the uniform load allocation with code rate \(\frac{1}{2}\), the load allocation in (8) with the true working rates, and the load allocation in (8) with estimated working rates. For performance comparisons, the following simulations are conducted with the fixed parameters \(m=1\) and \(\sigma=1\), \(\sigma_{c}=0.01\). Moreover, we set \(G=2\), \((N_{1},N_{2})=(20,30)\), \(T=20\), and the true working rate \((\mu_{1},\mu_{2})=(1,5)\) at time \(t=0\). At time \(T\), the true working rate varies to \((\mu_{1,T},\mu_{2,T})=(1.0109,4.9999)\). We obtain the estimated working rates of the groups \((\hat{\mu}_{1,T},\hat{\mu}_{2,T})=(0.9892,4.8364)\) using the proposed algorithms. This simulation shows that the load allocation in (8) with the estimated working rates can achieve the same performance as the optimal load allocation in (8) with the true working rates. It is also observed that the load allocation in (8) with the estimated working rates shows 54% and 14% reductions in the expected execution times compared to the uncoded scheme and the uniform  
Figure 2: \(\mathit{COR}(t)\), \(MSE(t)\), and \(MAE(t)\) results for the datasets with the cases \(N_{w}=10,30,50\) for each time \(t\in[T]_{0}\). load allocation with code rate \(\frac{1}{2}\), respectively.