dataesr/scientific-paragraphs-categorization huggingface dataset:
Conclusions Our attempts to replicate Robbeets et al. 's results show significant discrepancies on all three fronts. Moreover, Robbeet's et al. claim to use "triangulation" to aggregate the results of the three studies, but they neither define nor describe the method of "triangulation". It is thus impossible to follow their method and combine the three corrected analyses. Taken together, our re-evaluation of all three datasets completely undermines Robbeets et al. 's narrative. The Transeurasian hypothesis remains unsubstantiated. We respectfully suggest the paper be withdrawn.
Internet of Things (IoT) devices generate a tremendous amount of time series data that is extremely dynamic, heterogeneous and time dependent. Such types of data introduce significant challenges for the real-time prediction of QoS metrics of IoT applications with different traffic characteristics. To this end, in this paper, we propose a temporal transformer model and a unified system to predict several QoS metrics of heterogeneous IoT applications when they communicate with the Edge of the network. The transformer model also leverages an attention module to provide a solution for both short-term and long-term sequence prediction of QoS metrics that allows to better extract any time dependencies. In particular, in our framework, we firstly generate a set of datasets containing real-time traffic information of five different IoT applications such as Heating, Ventilation, and Air Conditioning (HVAC), lighting, Voice over Internet Protocol (VoIP), surveillance and emergency response using the 802.
15. 4 access technology and the RPL routing protocol. Following, we perform the data cleaning, downsampling and pre-processing of the datasets and we construct the QoS datasets, which include four QoS metrics, namely throughput, packet delivery ratio, packet loss ratio and latency. Finally, we evaluate the transformer model through extensive experimentation using both short-term and long-term dependencies and we show that our model can guarantee a robust performance and accurate QoS prediction.
I. INTRODUCTION T HE number of Internet of Things (IoT) applications have considerably increased, while generating a tremendous amount of data. According to Cisco, the number of connected devices will reach up to 14. 7 billion by 2023  [1]. The devices are expected to continuously generate large volumes of data requiring extensive analysis to capture valuable information that can help in the intelligent decision making. However, the device's CPU, memory, and disk capacity restricts the data processing on the device itself. Thus, the data and the analysis processing have to be offloaded to more resource powerful platforms, such as the newly introduced Edge Computing  [2]. Edge computing can facilitate the data processing very close to the source of the data, reducing thus the overall latency perceived.
In this way also, the processing burden is shifted/offloaded to the Edge of the network through a process that is called task offloading  [3]. However, the amount of A. Hameed, J. Violos and A. Leivadeas are with the Department of Software and Information Technology Engineering, École de Technologie Supérieure, University of Quebec, Montreal, Canada. N. Santi and N. Mitton are with INRIA Lille-Nord, France. R. Grünblatt is with SAMOVAR, Télécom SudParis, Institut Polytechnique de Paris, Évry, France E-mails: {aroosa. hammed, ioannis. violos}. 1@ens. etsmtl. ca, aris. leivadeas@etsmtl. ca, {nina. santi, nathalie. mitton}. inria. fr, remy@grunblatt. org Edge resources needed for each IoT application depends on the volume of the data generated from the IoT devices. This creates an important challenge related to the accurate workload (e.
g. throughput) profiling of an IoT application. At the same time, IoT applications consist of heterogeneous devices that send data of different contexts, with different reporting frequencies usually over a random access channel generating thus, high interference levels  [4]. All these add several levels of complexity when it comes to the prediction of typical Key Performance Indicators (KPIs) in IoT. Regarding the reporting frequency, IoT devices follow very dynamic models ranging from periodic to event-based transmissions. Hence, the feature of time dependence make such data different and more challenging than traditional data. Therefore, each IoT application, when generating/offloading data, will have different instantaneous Quality of Service (QoS) behavior, which will be time dependent.
Hence, it is necessary to propose an efficient model that will analyze and predict the QoS metrics using IoT time series data. A time series data is a series of data points that are ordered by their chronological order. Time dependency is a very important feature of the IoT time series data, since data are becoming widespread in an IoT context  [5]. Accordingly, the time feature is affecting the way prediction and analysis of IoT data is done. One way to predict the data at a next time step is to use the data from previous time steps in the short or long past  [6]. Therefore, there is huge interest in analyzing the IoT traffic profiles by applying various machine learning techniques  [7]. For example, several studies applied traditional time series algorithms or deep learning models to predict the IoT traffic behavior  [8] -  [23].
Additionally, some works, such as  [18] -  [23] , applied deep learning specifically for the time series fore-casting task. These works proposed various learning networks such as Temporal Convolutional Network (TCN), DeepAR, LSTNet and an improved versions of LSTM as stacked LSTM and bidirectional LSTMs for time series forecasting problems. However, this set of works lacks the ability to handle both the short and long-term dependencies at the same time, while training over long sequences of data degrades the accuracy of the prediction. To overcome the above described research gaps of the current studies, we deploy five different IoT applications to investigate four different QoS metrics. Moreover, this research also investigates the multivariate prediction of QoS metrics for each application.
Last but not least, a novel model that makes an efficient use of the time features of the IoT applications and accurately predicts their QoS behavior, in a dynamic network environment, is proposed. The model also investigates the short and long input sequence dependencies without any performance degradation. To this end, the main contributions of this paper can be summarized as follows: • We consider 5 different IoT smart building applications that present different requirements in terms of number of devices, packet length, context of message, and message frequency transmission. We deploy the applications in a real testbed  [24]  comprised of approximately 300 IoT devices and generate data over an IEEE 802. 15. 4 access network. • We provide the predictions of four major QoS metrics such as Throughput, Packet Delivery Ratio (PDR), Packet Loss Ratio (PLR) and Latency.
As multivariate time series forecasting poses a challenge of how to capture and leverage the dependencies among multiple variables, we provide both univariate and multivariate multi-step prediction for all four QoS metrics of the five IoT applications under consideration. • We design and implement a QoS prediction mechanism based on Temporal Transformers that models temporal dependencies within input sequences consisting of IoT data and that is able to handle the long input sequences with the attention module to make prediction. The model accurately provides the multi-step QoS prediction and its temporal relation with its preceding QoS values from past observations. The rest of the paper is organized as follows: Section II presents the related work and current limitations. Section III gives a detailed information on the challenges of IoT time series data.
A. Deep Learning for QoS Prediction The authors in  [8] , predicted the delay using a nonlinear autoregressive exogenous (NARX) RNN following both a single-step and a multi-step ahead prediction. The prediction accuracy is measured using MSE, RMSE and MAPE metrics. However, they used a simulated dataset of an IoT environment. Furthermore, the delay metric is also predicted in  [9]  using a simple Deep Neural Network (DNN) consisting of forward with backward passes and also providing the analysis of hyperparameters, which presented good results such as size of training data, number of layers, number of neurons in each layer and epochs. The features utilized by this work were extracted from the application layer, MAC layer and physical layer of the network. The authors in  [10]  proposed a deep learning model that predicts the throughput, delay, and packet loss of an IoT communication system.
This is why and in order to mitigate this centralized distribution they resorted to a vectorization technique. However, the dataset was generated from a simulated factory scenario. Fan et al. [14]  proposed a deep learning based Recurrent Neural Network (RNN) model using an attention mechanism for the IoT data processing at the Edge. All input time series were fed into the RNN and attention network to calculate the extrinsic correlations and to provide the final prediction. The proposed model, called UrbanEdge, used four different datasets such as traffic volume, building occupancy, electricity and Air Quality Index (AQI) consisting of time series based sensor readings. The results proved that the proposed UrbanEdge model outperforms several baseline methods such as Autoregressive Integrated Moving Average (ARIMA), Vector Autoregression (VAR), LSTM and Sequence-to-Sequence (Seq2Seq).
However, there is the vanishing gradient problem for the training of the RNN and the model also requires a high bandwidth for the transfer of the monitoring metrics. The authors in  [15] , proposed EdgeLSTM, which is an Edge-based deep learning system that utilizes grid LSTM along with Support Vector Machine (SVM). The pipeline of this framework followed a data processing, a hyperparameter selection, and a construction of multi-class SVM models to be trained using four different datasets. The output was to get the results for four different tasks such as data prediction, network maintenance, anomaly detection and mobility management. Abdellah et al. [16]  performed the prediction of throughput of IoT traffic in a 5G communication network using an LSTM network. The dataset is generated using an IoT traffic generator.
The features of the dataset includes the timestamp, bytes count and packets count. Finally, the authors in  [17]  proposed the forecasting of IoT traffic by using a stochastic gradient descent algorithm and a neural network architecture called gaNET. The dataset used in the paper consists of features such as obfuscated mobile identification and timestamp of records. There are also few recent studies that applied regression based approaches  [12] ,  [13] , to predict throughput and packet delivery ratio (PDR), since regression based techniques tend to be a light weight alternative for the prediction of QoS metrics. However, most of the IoT data used for the QoS prediction consist of time series sequences which are better predicted using deep learning approaches, such as Recurrent Neural Networks (RNN) or Long Short-Term Memory (LSTM) networks, that are specifically designed for handling time series data.
The first model is used for learning the input sequence and the second learns the reverse of that sequence. Furthermore, a Temporal Convolutional Network (TCN) which combines the dilations and residual connections with the causal convolutions needed for autoregressive prediction, was proposed in  [21]. The authors showed that TCN performed better than RNN models for time series forecasting tasks. Salinas et al. [22]  proposed a model called DeepAR for probabilistic forecasting using autoregressive recurrent networks that learns from historical data of all time series in the dataset and provides the forecasting results. Another deep learning model for multivariate time series forecasting, was proposed in  [23]  called Long-and Short-term Time-series Network (LSTNet). This work combined the convolutional layer along with recurrent layer to learn both local patterns and long-term dependencies among multi-dimensional input variables.
III. PROBLEM FORMULATION OF QOS PREDICTION In this section, we describe and formulate the QoS prediction problem, when we have multiple QoS metrics such as throughput, PDR, PLR and latency to be predicted and when IoT devices belonging to different IoT applications communicate with an Edge infrastructure. In particular, the IoT applications are represented by the set A = {a 1 , a 2 , a 3 , a 4 , a 5 } where a 1 represents the first IoT application, a 2 represents the second IoT application and so on. Similarly, the set D = {d 1 a1 , d 2 a2 ,. , d m ai } represents the data generated by each IoT application where d 1 a1 represents the first dataset in the set D and it is generated by the IoT application a 1. The d m ai denotes the m th dataset generated and it is for the i th IoT application where m <= 5 and i <= 5 as data is generated for five different IoT applications.
Furthermore, each network dataset generated for an i th IoT application is constituted by a sending and receiving information which is denoted as D = {(u 1 a1 , s 1 a1 ), (u 2 a2 , s 2 a2 ), (u 3 a3 , s 3 a3 ), (u 4 a4 , s 4 a4 ), (u 5 a5 , s 5 a5 )} where (u 1 a1 , s 1 a1 ) represents the pair of sending and receiving information for IoT application a 1. More specifically, U = {u 1 a1 , u 2 a1 ,. , u j a1 } denotes the set of the transmitting information by the IoT devices of the IoT application a 1. Similarly, S = {s 1 a1 , s 2 a1 ,. , s j a1 } represents the set of the receiving information at the Edge server side, where s j a1 is the j th receiving information of the i th IoT application. Regarding the features used, the set U F denotes the features related to the transmitting data in the network by the IoT devices as: U F = {u f1 , u f2 , u f3 , u f4 , u f5 , u f6 } where u f1 denotes the timestamp at which the packet is sent; u f2 is the sensor node ID that is sending the packet; u f3 represents the size of the UDP payload in bytes; u f4 is the IPv6 destination address (we use an 802.
15. 4 access network with 6LoWPAN); u f5 is the destination port; u f6 is the actual payload in a hexadecimal format. In a similar way, the set SF represents the features related to the receiving information at the Edge server side and is further expressed as SF = {s f1 , s f2 , s f3 , s f4 }, where s f1 represents the timestamp at which the packet is received; s f2 is the IPv6 address from which the packet originates; s f3 denotes the receiver port on which the packet has been received; and s f4 is the hexadecimal payload of the packet. Given the sets U F and SF , we computed the QoS datasets for each IoT application. The throughput is represented as Q = {q 1 1 , q 2 2 ,. , q t i } where q t i is the i th throughput value at timestamp t, such that 0 < t < T , where T represents the total timestamps for which data are generated.
A. Applications and Edge Computing Infrastructure Five different IoT applications and their respective datasets are considered in this work. These applications are: 1) Emergency Response: The emergency system is used to monitor the critical areas of the building such as gas pipes or fire alarms. If a situation occurs where the pipelines reach high pressure, which may cause an explosion, then the IoT devices at a specific location will detect this and send an alert with relevant contextual information to a control system to remedy the situation. 2) Heating, Ventilation and Air Conditioning (HVAC): The HVAC system provides various handling systems inside the building by controlling factors such as temperature, humidity etc. , in order to provide the necessary comfort and indoor air quality to the occupants.
B. Feature Engineering The dataset generated for the five different IoT applications provide the receiving and transmitting information of the packets within the network. Each application has its own database with UDP and server tables. The UDP table contains information about packets as they are transmitted by the sensors and the Server table contains information about packets as they are received by the server. The raw features are highlighted in Table  II. In order to extract the most useful features from the given raw data, we engineered several features as described below: 1) Timestamp: It is the time that is associated with each packet in the network. Initially, data were collected and added to the raw dataset at a nanosecond granularity. However, we changed the granularity of the dataset from 1 nanosecond to 5 milliseconds, to better capture the QoS metrics fluctuations.
For example, it was not always possible to calculate the QoS metrics for each nanosecond as in most of the nanosecond timestamps we did not have any sending or receiving packets in the network that was causing the generation of many null values for the QoS datasets. Thus, each of the below described features are computed for a time interval t of 5 milliseconds without however losing significant information. 2) time f irst pack : It is the time at which the first packet is transmitted in a specific time interval of 5 ms. 3) time last pack : It is the time at which the last packet is transmitted to the server in a specific time interval of 5 ms. 4) total trans pack : It is the total number of packets transmitted by a node during a specific time interval of 5 ms. 5) total rec pack : It is the total number of packets received by the server during a specific time interval of 5 ms.
C. Data Preprocessing Each application dataset is stored in a SQLite3 database and compressed with the zstd compression algorithm. We firstly decompress the dataset and read the sql table in the. csv format. Then we engineer the QoS related features and create a second QoS dataset for each of the IoT applications. However, before the QoS datasets are fed to our proposed transformer models for training or validation purposes, several preprocessing operations are applied to refine their quality and thereby the QoS forecasting performance. In particular, we remove any outliers that are caused by some unseemly situations in the datasets. There are also some missing values in the QoS dataset because it may occur that no packets are transmitted and received for some time intervals. For instance, the HVAC and lighting applications are generating packets with very low frequencies, as can be seen in Table  I.
For the particular applications, the missing values are filled by average values of their respective features. Finally, the features of each application dataset is normalized in a particular range using the min-max normalization given as: X normalized = x -x min x max -x min ( 5 ) where x is the original QoS value of the metric/feature under consideration (e. g. Throughput, PDR, PLR and Latency), x min represents the minimum value of that feature and x max denotes its maximum value. Thus, the normalized data lie in the range from 0 to 1.
A. Overview of Proposed Framework Given the ability of temporal transformer models to get the time dependencies of a dataset, we proposed a framework which adopts the benefits of the particular model to process and estimate the QoS metrics for IoT applications in an edge environment. In the proposed framework as shown in Fig. 1 , we first generate the real IoT data for five different applications as discussed already in the section IV. A. Then, our second step is to take all of these raw datasets and engineer the new useful features as discussed in section IV. B. Then we process these data by performing data cleaning, data down-sampling and data normalization. Then the new pre-processed QoS datasets for the five IoT applications are divided into training sets, validation sets, and testing sets.
C. Algorithm Description Our proposed QoS prediction algorithm (Algorithm 1) consists of either univariate or multivariate inputs that can be a QoS dataset in form of training data, validation data and testing data. The first step is to build the transformer model using the build model() function, which takes the training and validation data as input. Following, the random search is performed with the keras tuner to search the number of models, using the RandomSearch() function, which takes the transformer model as an object, the search objective, the max trials allowed and the number of trials per search as an input. Then, the BestM odel() function takes the tuner object and the total number of search models by the tuner as input and it returns the best model which has the highest validation accuracy across all models given by the RandomSearch() function.
Lastly, the best selected model is trained for a specific number of epochs using the f it() function and the final prediction of the QoS values are provided as Ỹ using the predict() function. Algorithm 2 depicts the temporal transformer model and it consists of the three main modules described above: 1) INPUT EMBEDDING, which takes as an input the training dataset, the sequence length of the input and the dimension used to represent the input sequence vector. This module is used to take the input into a specific tensor shape for the transformer along with providing the positional encoding of the time series input as well. In this module, firstly the input layer is applied, which instantiates a tensor for the temporal input sequence of the training dataset so that the input sequence is passed to the transformer model.
Next, is the FFN which takes as input the residual connection values res and it passes them to the normalization layer. The results of the normalization layer along with the filters, kernel dimensions and activation function are passed to the Conv1D layer and the final dropout is performed. 3) OUTPUT MODULE is used to provide the final prediction of the dataset. It takes the previous layer output i. e. , x along with the residual connection value i. e. , res as an input. Firstly, the x is passed to the GlobalAvgP ooling1D() layer, which is used specifically for the temporal data and it takes the average among all time steps. Then, the output is passed to the Dense() layer, the Add() layer, and the layer norm() functions, in order to get the predicted values of QoS as an output.
D. Complexity Analysis Proposition 1: The computational complexity of Algorithm 1 is O(n 2 d). Proof : Line 1 of Algorithm 1 uses the build model() function, which is the temporal transformer model and its time complexity is O(n 2 d) as it is represented and proved by the proposition 3. Following, line 2 takes O(n) as RandomSearch() searched all n number of models for the worst scenario and line 3 takes a constant amount of time i. e. , O  (1). Next, the model. f it() function in line 4 takes O(t) time in the worst case, where t represents the length of the training dataset which is always more than the validation dataset. Lastly, line 5 predicts the QoS for a given testing dataset in O(n) times. Hence, the overall complexity of Algorithm 1 is: O(n 2 d) + O(n) + O(1) + O(t) + O(n) = O(n 2 d).
To add up the three types of costs and calculate the total implementation cost, we begin with the cost of model infrastructure that comes down to a commodity computer which is approximately $1. 000 1. In the implementation cost we should also add the electricity cost which is approximately $160. 16 per year. 2 and the maintenance cost which ranges from $40 to $90 per hour for the work of a technician 3. The data support cost is significantly higher due to the work of the data engineer. We estimate a senior data engineer can implement the proposed model, the data preprocessing and the extracttransform-load process in one man-month which results in a cost close to $9. 649  4. The deployment cost is reduced to the manual work of a network engineer that will integrate and run the python scripts in the edge infrastructure.