# A Time-Series Model for Varying Worker Ability in Heterogeneous Distributed Computing Systems
## 4 Performance Evaluation  
We perform a simulation study in order to validate the suggested inference methods. We fix the true parameters \(m=1\) and \(\sigma=1\) so that the initial ability of groups has an average of \(1\) and a standard deviation of \(1\). We allow a minor degree of ability changes over time by setting \(\sigma_{c}=0.01\). We consider \(T=20\) timepoints after the initial time and \(G=10\) groups of workers. The number of particles is set to \(L=200\). For simplicity, we assume that all workers are assigned unit loads over time so that \(l_{i,t}=1\) for \(i\in[G]\), \(j\in[N_{i}]\), and \(t\in[T]_{0}\), and each group has the same number of workers \(N_{w}\) so that \(N_{1}=\cdots=N_{G}=N_{w}\). We vary the number of workers, \(N_{w}=10,30,50\), to check that the algorithm works for various situations and to see how the number of workers affects the performance of the proposed algorithms. We generate the ability \(x_{1:G,0:T}\) of workers using (2) and (3) and use them to generate the observation data \(\eta_{1:G,1:N_{w},0:T}\) that follows the distribution given in (4). This process is repeated \(30\) times for each case \(N_{w}=10,30,50\). Then we apply the inference algorithm to estimate the model parameters and the ability of workers. Table 1 presents the parameter estimates of \(30\) repeated datasets for the cases \(N_{w}=10,30,50\). The algorithm estimates the parameters close to the true parameters \(m=1\) and \(\sigma=1\) in all of the cases of \(N_{w}\), suggesting that the algorithm works well in estimating the true parameter. Moreover, we can see that the standard deviations of all the estimates over \(30\) repetitions tend to decrease as \(N_{w}\) increases, implying the larger number of workers in each group tends to yield more accurate estimates. We now study the accuracy of the ability estimate \(\hat{x}_{1:G,0:T}\) using the smoothed estimates given in (14). We use three criteria, the Pearson correlation coefficient (COR), the mean squared error (MSE), and the mean absolute error (MAE) to evaluate the performance. The measure COR is the correlation coefficient between the estimated and the true worker abilities over all the groups \(i\in[G]\) and timepoints \(t\in[T]_{0}\), given by  
\[COR=\frac{\sum_{i\in[G]}\sum_{t\in[T]_{0}}(\hat{x}_{i,t}-\bar{x})(x_{i,t}-\bar{ x})}{\sqrt{\sum_{i\in[G]}\sum_{t\in[T]_{0}}(\hat{x}_{i,t}-\bar{x})^{2}}\sqrt{ \sum_{i\in[G]}\sum_{t\in[T]_{0}}(x_{i,t}-\bar{x})^{2}}},\]  
where \(\bar{x}=\frac{1}{G(T+1)}\sum_{i\in[G]}\sum_{t\in[T]_{0}}\hat{x}_{i,t}\) and \(\bar{x}=\frac{1}{G(T+1)}\sum_{i\in[G]}\sum_{t\in[T]_{0}}x_{i,t}\) are the means of the estimated and true abilities, respectively. On the other hand, the other measures of MSE and MAE are based on the differences between the true and estimated item qualities over all the groups \(i\in[G]\) and timepoints \(t\in[T]_{0}\), given by  
\[MSE=\frac{1}{G(T+1)}\sum_{i\in[G]}\sum_{t\in[T]_{0}}(\hat{x}_{i,t}-x_{i,t})^{ 2},\]  
\[MAE=\frac{1}{G(T+1)}\sum_{i\in[G]}\sum_{t\in[T]_{0}}|\hat{x}_{i,t}-x_{i,t}|.\]  
Table 2 shows the COR, MSE, and MAE results across 30 datasets. The average COR values are almost equal to 1, which suggests that the proposed algorithm properly estimates the true ability of worker groups. The average MSE and MAE values turn out to be close to 0, which suggests that the difference between the true and estimated abilities is almost the same. It is reasonable to have fewer discrepancies between the true and estimated values in the case of large \(N_{w}\) since we have more observations. We also depict the three measures in Figure 2, where the three measures are computed at each time \(t\in[T]_{0}\). The \(COR(t)\) at each time \(t\in[T]_{0}\) is given by  
\[COR(t)=\frac{\sum_{i\in[G]}(\hat{x}_{i,t}-\bar{x}_{t})(x_{i,t}-\bar{x}_{t})}{ \sqrt{\sum_{i\in[G]}(\hat{x}_{i,t}-\bar{x}_{t})^{2}}\sqrt{\sum_{i\in[G]}(x_{ i,t}-\bar{x}_{t})^{2}}},\]