dataesr/scientific-paragraphs-categorization huggingface dataset:
VI. PERFORMANCE EVALUATION A. Model Implementation and Frameworks 1) Evaluation setup: Each dataset is zero-mean normalized and standardized. Under the time series prediction settings, we forecast the four following QoS metrics as: (i) Throughput; (ii) PDR; (iii) PLR and (iv) Latency. Additionally, the prediction is performed in two time series settings as: (i) Univariate and (ii) Multivariate. The window size for both settings is set to be 30. The total data generation lasted seven days. All of the five datasets are divided into three parts as follows: i) training dataset, which contains the first five days of data; ii) validation dataset, which contains the sixth day data and iii) testing dataset which contains the seventh day data. All of the models were trained and tested on two compute clusters offered by Compute Canada namely, Cedar and Beluga.
4) Hyper-parameter Tuning: For the hyper-parameter search and tuning, we performed a random search of the search space using the keras tuner. In particular, for all methods and all datasets, the input length of the input time series sequence is set as 30. In other words, the rolling window sample is set to be 30, which we believe is a sufficient value for long-term prediction. The hyperparameters that were searched for the baseline models consist of the number of neurons, dropout rate, learning rate and number of layers. For the stacked LSTM, the number of neurons were selected from the range 8 to 128 with a step of 8. For the MLP and bidirectional LSTM, the number of neurons were selected between 8 and 512, with the same step. For the dropout rate, the value is taken from the {0, 0. 1, 0.
For the number of heads, an optimal value was found within the range of 4 to 32 with a step of 2. The dropout rate was selected between 0 and 0. 5 with a step size of 0. 1 and the number of transformer blocks was chosen from the range {4, 8, 12, 16}. For the linear layer, which is included as part of the transformer architecture, the number of neurons was selected between 4 and 128 with a step of 8 and their dropout rate was chosen between 0 and 0. 5 with a step of 0. 1. Regarding the neural network optimizer, the Adam optimizer was used for all baseline methods and for our trans- Finally, keras tuner selected the best trial that gave the best set of hyperparameters for all of the application datasets. Table  III  summarizes the hyperparameters and the best selected value from keras tuner for all five application datasets.
B. Explanatory Data Analysis In this part, we provide the explanatory analysis of the applications' datasets along with their properties. The statistical properties of each dataset are presented in Table  IV. In Fig. 3 , the density plots for each of the QoS metrics within each dataset are also presented. The density plots are used to observe the distribution of the datasets with a continuous interval. For the emergency application, we have a positively skewed distribution for all four QoS metrics and this is because the mean in the datasets of throughput, PDR, PLR and latency are greater than their median values. For the HVAC application, the throughput, PLR and latency also exhibit a skewed distribution and more specifically a right skewness however, PDR presents a multi-modal distribution as it has three different peaks.
For the lighting application, the throughput and latency both datasets are rightly skewed, but PDR and PLR are both multi-modal datasets. For the surveillance application, the throughput is multi-modal with more than 12 modes, PDR exhibits a normal distribution, PLR and latency both are rightly skewed. Lastly, for the VoIP application, we have a normal distribution for all of the three QoS metrics i. e. , throughput, PDR and PLR, however, the latency dataset is slightly skewed towards right as the mean in latency data i. e. , 0. 004661 is slightly higher than the median value i. e. , 0. 004562.
C. Results 1) Univariate time series forecasting: For the univariate TSF, we included a representative range of the 5 IoT datasets to ensure the diversity and applicability of our transformer model with respect to the dimensionality and length of the time series samples, as well as the number of samples. Table V shows the MAE, MSE and RMSE achieved by the baseline methods and transformer model. As it can be seen, the transformer model worked well for the throughput prediction as compared to the other models across all datasets. We have also plotted the MSE and MAE values of all methods in Figs. 4  and 5  to better illustrate the results. It should be noted that the y axis of both figures goes from large values towards small values and we also include the data points for the transformer model to better position its efficiency.
Our first observation, is that all applied models give the least values for all error metrics for the emergency application followed by the lighting application. In contrast, for the surveillance application, the models achieve higher error values followed by the VoIP and HVAC applications. The main reason for having less accurate results for surveillance, VoIP and HVAC applications is that the datasets of these applications contain several extreme values also known as outliers. Hence, as deep learning models do not learn easily such extreme values, such behavior can cause performance degradation. We can also detect the outliers from the statistical properties of the datasets as shown in Table  IV. For instance, for the surveillance application, the throughput dataset has a standard deviation value of 0.
167844 and a mean value of 0. 337204. This is because the more extreme outliers exist in the dataset, the more the standard deviation is affected with respect to the mean value. Similarly, for the VoIP and HVAC applications, the standard deviations are also highly affected as they appear to be 0. 108589 and 0. 204743 respectively, while their corresponding mean values are 0. 323644 and 0. 253958. In contrast, for the lighting and emergency applications, such kind of extreme values appear more frequent and cannot be considered outliers, as the outliers by their nature are rare events that happen in a dataset. Therefore, the deep learning models adapt better to those frequent extreme events to some extend and produce better performance for the lighting and emergency datasets as compared to the other application datasets.
To better understand which model is able to capture this behavior more accurately, we shift our focus on Figs. 4  and 5. It becomes apparent that the temporal transformer provides the least error in the prediction of throughput values as compared to all other algorithms and for all datasets. This happens for the following two reasons: (i) For a longer input window size, also called input sequence length, i. e. , 30 in this work, the prediction ability of the deep learning models decreases, which leads to a rise in the error metrics. This also reveals a real problem faced by the time series forecasting. However, our transformer model is well suited for solving such long       Following, the results for the PDR prediction are presented in Table  VI. As it can be seen, once more the transformer model performed better for almost all of the applications.
Nonetheless, there are two applications for which other models also provide promising results and these are: (i) for the HVAC dataset the bidirectional LSTM provides the least MSE and RMSE values as 2. 54e-5 and 5. 04e-3. The reason that the transformer could not match these values are probably because our model tried to learn the outliers and this had an impact on the relation between the features as provided by the attention module of the transformer, which can lead to higher errors than the bidirectional LSTM model. At the same time, MSE and RMSE are more sensitive to the outliers as the squaring of high errors will lead to lower performance; (ii) for the lighting application, MLP provides the least MAE value i. e. , 2. 47e-4, however, its MSE and RMSE are also affected by the outliers.
Nonetheless, the impact of the outliers for the particular application was less on the transformer model which led to the least attained MSE and RMSE values. Next for illustration purposes, in Fig. 6  we also plot the predicted values (orange curves) and the collected true values (blue curves) for the PDR dataset of the surveillance application. In order to not further increase the length of the paper, we have just selected the surveillance application as it has more fluctuations and presents a more interesting behavior for the QoS metrics prediction. From the figure, we notice that the PDR data is usually noisy which means that we have peaks and troughs (i. e. , fall of data points in downward direction). This means that the PDR of the surveillance application is sometimes higher and sometimes very lower than the normal pattern.
This is because of the exponential distribution pattern of the application and the high network contention, since the rest of the IoT devices belonging to other applications may transmit at the same time. From this, we can deduct that the peaks and troughs are not normal patterns of the dataset and therefore, it is not necessary that all peaks and troughs appear the one after another by following a specified and periodic behavior. Given this type of fluctuating dataset, we see from Following, we provide the univariate PLR results for the five IoT applications in Table  VII. As it can be seen, the transformer model provides the least error values for almost all datasets for this particular QoS metric as well. However, two particular cases are drawn from these results: 1) for the lighting application, the MLP model provides the least MAE, yet, MSE and RMSE are higher than the transformer model and the reason for such behavior is the same as the one explained for the PDR case; 2) for the emergency application, all algorithms provide the best accuracy performance with respect to the other four applications.
The reason for this is that the particular dataset is not affected by outliers as the standard deviation value i. e. , 0. 127102 does not deviate a lot from the mean value i. e. , 0. 134196. Nonetheless, the transformer model provides the best performance and for these types of applications. Once more, we plot the actual vs. predicted values for the PLR data of the surveillance application only in Fig. 7 , as it has more fluctuating patterns compared to the other applications. In general, we can see that the transformer model can capture very well the general behavior of the PLR dataset. There is only just a small difference between the actual and predicted values when there are small PLR spikes as noticed at the 50ms, 150ms, 470ms, 550ms and 790ms time instances. These spikes can by attributed to high network contention time instances which can lead to an increased packet loss.
Nonetheless, the transformer was able to closely follow the unusual fluctuations between the time period from 450 ms to 900 ms. This is due to the fact that the particular model can capture the time series features with long-term time dependency easily. Following, we provide the univariate latency results for the five IoT applications in Table  VIII. As it can be seen, the temporal transformer model performs better for all of the datasets and in terms of all error metrics as compared to the baseline methods. From Table  VIII , we have the following observations:  (1)  The latency datasets of all applications are positive (right) skewed. The distribution is right skewed because of the lower bound in the dataset. So if the lower bound of the dataset is extremely low relative to the rest of the data, then this will cause the data to be skewed right.
The lower bound for an application reveal that lower latency is experienced during the transmission of the packets. Furthermore, the emergency application followed by lighting and HVAC have more extreme smaller values for latency as their standard deviations is less distant from their mean value than the surveillance and VoIP applications. However, this does not affect the performance of the proposed temporal transformer model and it always outer-performs the baseline methods for all skewed datasets in term of all error metrics. (2) The second observation is that the second best model is the bidirectional LSTM as it performed well for 3 out of 5 applications after the transformer model. The reason is that the particular model is able to learn the input sequence in both forward and backward direction.
2) Multivariate time series forecasting: In this part of the section, we present the obtained results under the multivariate setting. Regarding the multivariate throughput prediction, the prediction results are provided in Table  IX. To better illustrate these results w. r. t. MAE, we plot them as well in Fig. 8. Similar to the univariate setting, the scale for MAE is logarithmic and goes from high i. e. , 1. 00E+00 to small values i. e. , 1. 00E-03. From this plot, it is shown that the LSTNet method provides the worst performance i. e. , the highest MAE for all of the applications and this is because the particular method is unable to deal with the dynamic periodic patterns or the nonperiodic patterns of our datasets. However, the bidirectional LSTM presents a good performance, similar to the one of our proposed temporal transformer model.
Specifically, the transformer model provides 1% improvement for HVAC and VoIP application, 2% improvement for lighting and emergency applications and a noticeably 32% improvement for the surveillance application as compared to the best performing baseline method. The reason for the major improvement in the surveillance application dataset is that the surveillance application has the long-term fluctuating patterns and our transformer model is the most suitable approach for capturing and predicting this long-term behavior. Similarly, Table  X  shows that the temporal transformer achieves the least MAE values for all applications in terms of PDR. However, there are two cases for which bidirectional LSTM achieves the least performance in terms of MSE and RMSE values and these are for the lighting and HVAC applications.
There are several reasons for this. Firstly, such application datasets contain extreme values for specific timestamps. Secondly, the PDR data of these two applications are smaller compared to the other applications and the transformer requires a larger number of training samples compared to the other baseline methods. Thirdly, the good performance of the bidirectional LSTM can be attributed to the fact that it runs the given input sequence in two ways from past to future and future to past. Thus, it is able to better learn even for datasets that have smaller number of training samples. However, the transformers can closely follow the performance of the bidirectional LSTM even in these situations. This can be corroborated by Fig. 9 , which presents the MAE metric for all applications and it can be concluded that the transformer performed consistently well, followed by the stacked LSTM for the emergency and surveillance applications and by the bidirectional LSTM for the HVAC, VoIP and lighting applications.
It can be seen that the proposed model outer-performs all the baselines for all applications and in terms of all metrics. The second best performing baseline method is bidirectional as it gives reasonable results for 4 out of 5 applications. Once more, the LSTNet method shows poor performance compared to the rest of the methods and this is because it is unable to capture all the dependencies among input sequences and other QoS features in the datasets. To conclude, regarding MAE, there is 1% to 92% improvements provided by our transformer model. Furthermore, for latency, there is 2% to 37% improvement in term of MAE, 6% to 66. 25% in term of MSE and 3% to 42% in term of RMSE provided by our proposed temporal transformer model compared with the second best performing baseline method. Finally, our proposed transformer model achieves the best performance on 20 out of 20 settings for the MAE case, and on 16 out of 20 settings for the MSE and RMSE respectively, for the multivariate forecasting task.
Regarding the impact of the problem setting as either univariate or multivariate on the prediction of the QoS metrics, we observed that our proposed model performed better in the univariate setting than the multivariate. This is because there are only 4 univariate cases and 8 multivariate cases in which our proposed transformer model performed worse than the other models. It is to be noted that multivariate models are good to model interesting inter-dependencies however, in the expense of an additional complexity. One of the reason for this behavior is that some IoT application's QoS dataset may include outliers which can more adversely affect the multivariate than the univariate forecasts. Moreover, it is easier to spot and control outliers in the univariate context. Also, the QoS datasets showed a nonlinear behavior w.
VII. CONCLUSION In this work, we investigated the QoS prediction problem by formulating it as a univariate and multivariate time series forecasting problem. A new framework was introduced that promotes an efficient QoS prediction for a number of coexisting and heterogeneous IoT applications that stress the IoT access network creating several levels of QoS uncertainty. We firstly generated five different real time datasets for HVAC, lighting, VoIP, surveillance and emergency response applications. Following, we presented a novel transformerbased architecture, which learns temporal representations and their complex dependencies in a long-term fashion, for the prediction of four important QOS metrics, namely, throughput, PDR, PLR and latency. The transformer architecture leverages the attention mechanism, which is effective at modelling time series.
Algorithm 2 Temporal Transformer Algorithm INPUT EMBEDDING(D train , pos, dim) // D train is the training dataset instances, pos is the input sequence length and dim is the dimension representation. 1. set input ← Input Layer(D train ) 2. set pos ← P ositional encoding(pos, dim) 3. set emb res ← Add(input, pos) Output: emb res ▷ Module 1 ENCODER MODULE(emb res , h s , num h ,d rate , f il, k s , act) // emb res is the output of Module 1; h s is the size of the head; num h is the number of heads used; d rate is the dropout rate; f il is the number of filters; k s is the kernel size and act is the activation function. 4. set x ← layer norm(emb res ) 5. set x ← M HA(h s , num h , d.
FORECASTING RESULTS FOR PLR, BEST RESULTS ARE HIGHLIGHTED IN BOLD -02 3. 88e-03 6. 23e-02 3. 88e-02 2. 26e-03 4. 75e-02 Surveillance 2. 41e-02 2. 68e-01 5. 05e-01 2. 53e-04 1. 16e-07 3. 40e-04 2. 49e-04 9. 66e-08 3. 11e-04 1. 56e-04 3. 26e-08 1. 81e-04 Fig. 9. MAE of multivariate PDR prediction across all datasets.
Algorithm 2 Temporal Transformer Algorithm INPUT EMBEDDING(D train , pos, dim) // D train is the training dataset instances, pos is the input sequence length and dim is the dimension representation. 1. set input ← Input Layer(D train ) 2. set pos ← P ositional encoding(pos, dim) 3. set emb res ← Add(input, pos) Output: emb res ▷ Module 1 ENCODER MODULE(emb res , h s , num h ,d rate , f il, k s , act) // emb res is the output of Module 1; h s is the size of the head; num h is the number of heads used; d rate is the dropout rate; f il is the number of filters; k s is the kernel size and act is the activation function. 4. set x ← layer norm(emb res ) 5. set x ← M HA(h s , num h , d rate , x) 6. set x ← dropout(x) 7. set res ← x+input ▷ MHA 8. set x ← layer N orm(res) 9. set x ← layer Conv1D(f il, k s, act, x) 10.
Acknowledgements Zoi Terzopoulou  is a fellow of the  Paris Region Fellowship Programme , supported by the  Paris Region. This project has received funding from the  European Union 's  Horizon 2020 research and innovation programme  under the  Marie Sklodowska-Curie  grant agreement No  945298-ParisRegionFP. Simon Rey  is supported by  NWO   Vici grant   639. 023. 811 (" Collective Information "). Sirin Botan  is supported by the  ARC Laureate  Project  FL200100204  (" Trustworthy AI ").
Strength and limitations The large sample size and the inclusion of multiple LMICs are clear strengths of the present study. However, findings must be interpreted in light of the study's limitations. First, data were self-reported potentially introducing social desirability and recall bias into the findings. Second, we lacked information on some characteristics of caregiving (e. g. , intensity, duration), financial loss associated with caregiving, or the characteristic of the care recipient. Thus, our estimate for anxiety may be conservative as individuals with very little involvement in caregiving may have been considered to be caregivers. Third, anxiety symptoms were assessed by a single non-validated question in our study. However, the use of extreme categories (i. e. , severe and extreme) is likely to have improved specificity.
The goal is to obtain a model that is able to capture at the same time (i) the level of coordination observed in human groups performing oscillatory tasks, and (ii) the reduction in the frequency of oscillation when compared to a solo performance. After developing three alternative models built around different biologically-relevant explanations of the slowing down phenomenon, we test them against experiments on a group synchronization task, a multi-agent extension of the socalled mirror game  (Noy et al. , 2011). Specifically, using the three collected datasets, we experimentally tuned, validated, and compared the three models, in terms of their ability to match the synchronization levels and oscillation frequency reduction observed in the experiments.
Data Collection We considered a dataset coming from experiments on group motor coordination performed at the University of Naples Federico II. The experiments were run via the computer-based architecture Chronos  (Alderisio et al. , 2017b) , which allows remote motor coordination between players in the absence of social (visual and/or acoustic) interaction. The Chronos    (Guna et al. , 2014). The positions of each agent are then broadcast to the others, through a Wi-Fi network, and appear on the monitor of each individual personal computer (see Figure  1 ). Task description. The dataset referred to experiments with groups of 5 participants that were asked to move their index finger on a Leap Motion controller so as to move a ball on the screen representing their own avatar, oscillating from left to right and vice versa.
Each experiment started with a 30 s solo session to capture the natural frequency of each individual, identified as its average frequency across 7 solo trials, see Table  1. The second session of the experiments (6 trials, 30 s each) was devoted to investigating group synchronization. Participants were connected through the software platform Chronos and were asked to oscillate their fingers in synchrony with the others. Namely, they were instructed to "Synchronize the movement of your finger from left to right with the movement of the others, as naturally as possible, as if you could do it for 30 min" (see   3  for the optimal parameter values for each model.
Parameterizing the Models Comparing Tables 1, 2, we observe that participants reduce the frequency of their oscillations when coordinating with their partners. Independent t-tests run between solo and group frequencies in each group showed significant differences [t (11) = 17. 07, p < 0. 001]. We observed that the reduction in group frequency was beneficial for coordination, since participants reach a synchronization level significantly different from that obtained when the phases are randomly extracted from a uniform distribution in [0, 2π], that is, r = 0. 40 ± 0. 20. The standard Kuramoto model (Equation  7 ) is capable of reproducing the emergence of coordination when agents interact but it fails to capture the observed slowing down in their motions' frequencies. In what follows, we calibrate Models 1-3 and the standard Kuramoto model (Equation  7 ) on the collected dataset.