dataesr/scientific-paragraphs-categorization huggingface dataset:
Feature Extraction. In supervised classification strategies, a set of examples is required for training the classifier model. is set is commonly defined as the signature database. Each instance of the database has a set of characteristics or variables associated with a label or a class. In this work, the goal is to identify characteristics in network traffic that are able to distinguish the normal network behavior from DoS attacks. e study is focused on the analysis of the header variables of the network and transport layer packets of the TCP/IP architecture because it allows saving computational resources and simplifies the deployment in the ISP networks. In IPv4-compliant networks, the network and transport layer protocols are IP, TCP, and UDP, which are specified in RFC 791  [40] , RFC 793  [41] , and RFC 768  [42] , respectively.
In the calculation context of the database variables, the references to the mean, median, variance (var), and ip proto � N proto K , ( 1 ) where N proto is the code of the protocol and K is an normalization constant set to the value 1,000. For instance, N proto � 6 and N proto � 17 in TCP and UDP protocols, respectively. With the main four variables mostly used in flow monitoring, it is possible to calculate the following associated statistic measurements: (i) Entropy: the entropy of the variable is calculated by Entropy(X) � -􏽘 i p X i 􏼁log 2 p X i 􏼁, ( 2 ) where X is the variable of interest, e. g. , the source port. (ii) Coefficient of variation: the coefficient of variation is calculated by cv(X) � std(X) mean(X) , ( 3 ) where std(X) is the estimated standard deviation and mean(X) is the estimated average of the variable.
(iii) Quantile coefficient: this parameter is defined herein by cvq(X) � 􏽢 Q X (1 -p) -􏽢 Q X (p) 􏽢 Q X (1 -p) + 􏽢 Q X (p) , ( 4 ) where  Q X (p) is the sample p-quantile, p ∈ (0, 0. 5), expressed by  [44] 􏽢 Q X (p) � X (k) + X (k+1) -X (k) 􏼐 􏼑 * f, (5) Start (X) � U X S X , (6) (v) where U X is the amount of unique values and S X is the overall number of X values. e data traffic with normal activity behavior was extracted from the ISCXIDS2012 dataset  [45]. e data traffic with DoS behavior was obtained in a laboratory controlled environment using tools such as hping3  [46] , hulk  [47] , Goldeneye  [48] , and slowhttptest  [49]. Processes such as extracting, transforming, and labeling the database instances are summarized in Figure  5. e raw network traffic was extracted from the capture files, as the packets were then grouped into sessions.
Feature and MLA Selection. Feature selection is an important step in the pattern recognition process and consists of defining the smallest possible set of variables capable of efficiently describing a set of classes  [50]. Several techniques for variable selection are available in the literature and implemented in software libraries as scikit-learn  [51]. In this work, the selection of variables was performed in two stages. First, Recursive Feature Elimination with Cross-Validation (RFECV) was used with some machine learning algorithms widely used in the scientific literature, i. e. , random Forest (RF), logistic regression (LR), AdaBoost, stochastic gradient descent (SGD), decision tree (DTree), and perceptron. RF obtained higher precision using 28 variables, while AdaBoost selected seven variables, but obtained lower accuracy, as shown in Table  4.
Results e network traffic was classified by the detection system in a controlled network environment using different sampling rates. In the experiments, raw network traffic of the CIC-DoS  [16] , CICIDS2017  [25] , and CSE-CIC-IDS2018  [25]  datasets and the raw network traffic captured in the customized testbed experiments were employed. e Smart Detection system has reached high accuracy and low false-positive rate. Experiments were conducted using two Virtual Linux boxes,  Define all the descriptor database variables as the current variables;  (5)  while True do  (6)  Split dataset in training and test partitions;  (7)  Create and train the model using training data partition;  (8)  Select the most important variables from the trained model;  (9)  Calculate the cumulative importance of variables from the trained model;  (10)  if max (cumulative importance of variables) < Variable importance threshold then  (11)  Exit loop;  (12)  end  (13)  Train the model using only the most important variables;  (14)  Test the trained model and calculate the accuracy;  (15)  if Calculated accuracy < Accuracy threshold then  (16)  Exit loop;  (17)  end  (18)  Add current model to optimized model set;  (19)  Define the most important variables from the trained model as the current variables;  (20)  end (  21 ) end  (22)  Group the models by number of variables;  (23)  Remove outliers from the grouped model set;  (24)  Select the group of models with the highest frequency and their number of variables "N";  (25)  Rank the variables by the mean of the importance calculated in step 7;  (26)  Return the "N" most important variables; each one of them using 8 virtual CPUs (vCPUs) with 8GB RAM.
Description of the Benchmark Datasets. Many different datasets such as DARPA (Lincoln Laboratory 1998-99), KDD′99 (University of California, Irvine 1998-99), and LBNL (Lawrence Berkeley National Laboratory and ICSI 2004-2005) have been used by the researchers to evaluate the performance of their proposed intrusion detection and prevention approaches. However, many such datasets are out of date and unreliable to use  [25]. In this study, the datasets CIC-DoS, CICIDS2017, and CSE-CIC-IDS2018 and customized dataset were used since they include modern threats and DoS techniques.
e ISCXIDS2012 Dataset. e Information Security Center of Excellence (ISCX) IDS 2012 dataset (ISC-XIDS2012) was built in at the University of New Brunswick to provide a contemporary benchmark. e dataset traced real packets for seven days of network activity, including HTTP, SMTP, SSH, IMAP, POP3, and FTP protocols covering various scenarios of normal and malicious activities. ISCXIDS2012 consists of labeled network traces, including full packet payloads in pcap format and is publicly available (https://www. unb. ca/cic/datasets/ids. html)  [45]. is work focuses on the normal activities of the ISC-XIDS2012 pcap file for extraction of signatures, more specifically data file of Friday, 11/6/2010.
e CIC-DoS Dataset. e CIC-DoS dataset focuses on application layer DoS attacks mixed with the ISC-XIDS2012 dataset attack-free traces. Four kinds of attacks were produced with different tools, yielding 8 different DoS attack strokes from the application layer  [16]. e resulting set contains 24 hours of network traffic with a total size of 4. 6 GB and is publicly available (https://www. unb. ca/cic/ datasets/dos-dataset. html). A summary of the attack events and tools used in the CIC-DoS is presented in Table  6. In the execution of the low-volume attacks using tool slowhttptest  [49] , the default value of 50 connections per attack was adopted, thus making the attacks more sneaky according to  [16].
4. 1. 3. e CICIDS2017 Dataset. CICIDS2017 dataset was recently developed by ISCX and contains benign traffic and the most up-to-date common attacks. is new IDS dataset includes seven common updated family of attacks that met the real-world criteria and is publicly available (http://www. unb. ca/cic/datasets/IDS2017. html)  [25]. is work focuses on the malicious DoS activities of the Wednesday, July 5, 2017, capture file, which consist of five DoS/DDoS attacks and a wide variety of normal network traffic. e resulting set contains 8h of network traffic with a total size of 13G. e attack tools used include slowloris, Slowhttptest, Hulk, GoldenEye, and Heartbleed.
e CSE-CIC-IDS2018 Dataset. is dataset is a collaborative project between the Communications Security Establishment (CSE) and the Canadian Institute for Cybersecurity (CIC). e final dataset includes seven different attack scenarios: brute force, Heartbleed, Botnet, DoS, DDoS, web attacks, and infiltration of the network from inside. e attacking infrastructure includes 50 machines, and the victim organization has 5 departments and includes 420 machines and 30 servers. A research document outlining the dataset analysis details and similar related principles has been published by  [25]   e customized dataset was developed in a controlled network environment, as shown in Figure  8. VLANs 10, 20, 30, and 40 are used as victim hosts. VLAN 165 is dedicated to the users of an academic unit. VLAN 60 is used as an attacking host, while monitoring occurs in VLAN 1.
Online Experiments. Online experiments were performed in a controlled laboratory environment according to the following validation methodology: (1) Raw data of the network traffic are obtained for analysis in pcap file format. (2) e attack plan indicating the origin, destination, attack type, and respective duration for the traffic indicated in step 1 is drawn. (3) e environment for reprocessing and classifying the traffic is configured. (4) e traffic is processed and classified. (5) e system performance is properly evaluated by comparing the output from step 4 with the attack plan described in step 2. Following this validation methodology, the sources of traffic capture were used: CIC-DoS, CICIDS2017, CSE-CIC-IDS2018, and customized dataset, thus fulfilling steps 1 and 2. e environment for traffic reprocessing and classification as described in step 3 was configured using two Virtual Linux boxes running Open Virtual Switch (OVS)  [28] , TcpReplay software  [53] , and the Smart Detection system, as shown in Figure  9.
In the reprocessing, classification, and evaluation of the traffic during steps 4 and 5, the raw data traffic was replayed by TcpReplay software in a specific OVS port and sampled by the sFlow agent for OVS. e sampled traffic was sent to the Smart Detection system and the classification result was compared with the attack plan. Figure  9  summarizes the procedures carried out by the proposed validation methodology. e raw network traffic file is reprocessed on VM-01, and the sFlow agent collects traffic samples and sends them to Smart Detection on VM-02.
Results and Discussion. e proposed approach has been evaluated using the aforementioned datasets, system setup, and metrics. Table  8  summarizes the system performance for each dataset. As can be observed, the best performance was obtained in the CSE-CIC-IDS2018 dataset, with a DR of 100%, an FAR of 0. 000%, and a PREC of 100%. During the analysis, there was a low occurrence of normal network traffic and well-defined bursts of malicious traffic. is type of behavior facilitates detection by the system and justifies the high hit rates achieved. However, a slightly lower performance was obtained in the customized dataset and CIC-DoS dataset, with a DR of 96. 5% and 93. 6%, an FAR of 0. 2% and 0. 04%, and a PREC of 99. 5% and 99. 9%, respectively. In those datasets, there is a higher volume of normal traffic and various types of attacks, including stealth application layer attacks.
In this more realistic scenario, the proposed system presented some detection failures, but still obtained a competitive performance. On the other hand, the worst result was obtained with the CICIDS2017 dataset with 80%    DR, 2% FAR, and 99. 2% PREC. is dataset expresses a more realistic network scenario, which includes normal traffic mixed with high-volume and low-volume malicious traffic with sneaky behavior, such as slow application layer attacks. Even so, the proposed system detected 4 out of 5 attacks with PREC greater than 90% and FAR less than 1%, showing that the method is feasible. To discuss online detection and consumption of computing resources during experimentation, the CICIDS2017 dataset was chosen because it is quite realistic, recent, and summarizes the major vectors of DoS attacks.
Even in the most adverse scenario, the experiment was completed normally, as shown in Figures  10  and 11. Overall network traffic is demonstrated in Figure  10 (a), while Figure  10(b)  highlights the sampled traffic sent to the detection system. As can be seen, for network traffic of 81. 3 Mbps, the detection system receives only 1. 74 Mbps, making this approach scalable. Overall traffic rating is shown in Figure  11 (a), while Figure  11 (b) exclusively highlights malicious traffic rating. It can be said that the system was efficient in distinguishing the normal traffic from the DoS attacks because of all the attacks performed, only the Heartbleed attack was not detected, highlighted in Figure  10 (b) between 15 h and 16 h. is kind of attack is primarily intended to collect data by exploiting OpenSSL software vulnerabilities as described in CVE-2014-0160, although it can also assume the behavior of a DDoS attack, as in any application.
Additional Comparison. Compared with some recent similar works available in the literature, the approach introduced in this work is quite competitive in terms of the evaluated performance metrics, as shown in Table  9. e comparison is not completely fair because the experimental scenarios and data were slightly different, but it is sufficient to allow an evaluation of the obtained results. For instance, in the offline experiments performed with the CIC-DoS dataset in  [16] , DR was 76. 92% using an SR of 20%, while the proposed system obtained an online DR of 90% for the attacks with a FAR of 1. 8% using the same sampling technique. In  [37] , a PREC of 82. 1% was obtained using the CICIDS2017 dataset in the offline and unsampled analysis. In this work, the proposed method obtained a PREC of 99.
Conclusion is article has presented the Smart Detection system, an online approach to DoS/DDoS attack detection. e software uses the Random Forest Tree algorithm to classify network traffic based on samples taken by the sFlow protocol directly from network devices. Several experiments were performed to calibrate and evaluate system performance. Results showed that the proposed method is feasible and presents improved performance when compared with some recent and relevant approaches available in the literature. e proposed system was evaluated based on three intrusion detection benchmark datasets, namely, CIC-DoS, CICIDS2017, and CSE-CIC-IDS2018, and was able to classify various types of DoS/DDoS attacks, such as TCP flood, UDP flood, HTTP flood, and HTTP slow. Furthermore, the performance of the proposed method was compared against recent and related approaches.
Based on the experimental results, the Smart Detection approach delivers improved DR, FAR, and PREC. For example, in the CIC-DoS and CSE-CIC-IDS2018 datasets, the proposed system acquired DR and PREC higher than 93% with FAR less than 1%. Although the system has achieved significant results in its scope, it needs some improvements, such as a better hit rate among attack classes and an automatic parameter calibration mechanism that maximizes the detection rate of attacks. CIC-DoS 0. 7690 N/A N/A  [37]  CICIDS2017 N/A N/A 0. 8210 e proposed approach CIC-DoS 0. 9360 0. 0004 0. 9990 e proposed approach CICIDS2017 0. 8000 0. 0020 0. 9920 Future works include analysis of DDoS attacks based on the vulnerabilities of services such as Heartbleed and web brute force attack, enhancement in the multiple-class classification, self-configuration of the system, developing methods for correlating triggered alarms, and formulating protective measures.
Data Availability We produced a customized dataset and a variable selection algorithm and used four additional datasets to support the findings of this study. e customized dataset used to support the findings of this study has been deposited in the IEEE Data Port repository (https://doi. org/10. 24433/CO. 0280398. v2). e feature selection algorithm used to support the findings of this study has been deposited in the Code Ocean repository (https:// doi. org/10. 24433/CO. 0280398. v2). e benchmark datasets used to support the findings of this study have been deposited in the Canadian Institute for Cybersecurity repository publicly available as follows: (1) the ISCXIDS2012 dataset (https://www. unb. ca/cic/datasets/ids. html), (2) the CIC-DoS dataset (https:// www. unb. ca/cic/datasets/dos-dataset.
All data are publicly available (https://www. unb. ca/cic/datasets/ids-2018. html). is work focuses on the malicious DoS/DDoS activities of Friday, February 16, 2018, and Tuesday, February 20, Variables tcp_flags_rte ip_len_std ip_len_cv ip_len_mean ip_len_var tcp_flags_mean tcp_flags_median tcp_flags_std tcp_flags_var ip_proto ip_len_entropy ip_len_rte tcp_flags_cv sport_entropy tcp_flags_entropy sport_rte sport_cv sport_cvq sport_std sport_mean 0. 0 0. 0 0. 0 0. 0 0. 016 0. 013 0. 01 0. 009 0. 009 0. 009 0. 008 0. 005 0. 002 0. 002 0. 002 0. 025 0. 024 0. 024 0. 048 0. 069 0. 00 0. 01 0. 02 0. 03 0. 04 0. 05 0. 06 0. 07 Importance.
Magnetic coils were voltage-controlled through a data acquisition card (S626 Sensoray, 5kHz sampling frequency) and amplified through Maxon Motor LSC 30/2 50 W servomotor controller (Neyron, France) 50W power | V |< 10V and I < 2A. Transmission lighting was achieved using a fiber optic visible light illuminator. When it is connected to an Optronis CL600x2 CMOS camera, it can deliver a bandwidth 655GB/s i. e. 5000 fr /s with a 512 × 268px 2 resolution. Control was achieved on our custom-built software platform including a parallel control of visual feedback and magnetic actuation and a graphical user interface (GUI). If nothing was specified, analyses were achieved in 20% isopropyl alcohol diluted in milliQ water medium, allowing for the maintenance of a sterile and chemically clean chamber and the facilitation of bubble evacuation.
All trajectories were achieved at constant pressure, in a renewing chamber in case of bubble contamination. For translational motion analysis, every trajectory started from the opposite side of the chip, with fixed orientation of the robot. For rotary motion analysis, the robot was maintained close from the chamber center. A C# program was developed treating simultaneously signal emissions and high-throughput data acquisition. Image processing was achieved using ImageJ, an open source imaging software (ImageJ  [35] ). From the postprocessed images, trajectories, kinetics and dynamics could be deduced using an open source visual servoing platform library (ViSP) blob tracker  [36]  for shape detection and moving-edge tracker for edge detection. To determine the cut-off frequency, the < Ω > of the robot was measured on 10 magnetic cycles to detect the breakdown phenomenon in a reproducible manner.
The frequency response scan defined the temporal resolution. The cut-off was defined as the highest frequency with an average rotation rate < Ω > equals to field pulsation ω c. A first scan was performed with 10 Hz step to detect the rough cut-off frequency, and then a more precise cut-off frequency was obtained with 1 Hz step zoom centered on the value. Another strategy for quicker convergence could be done using such a dichotomy. It was essential to calibrate the electromagnetic coil setup prior to the rotary or translational dynamics measurements. All data analysis and plots were achieved with Matlab 8. 1. Supporting information provide more details of control and imaging processing in SI1 section.
Acknowledgements The material is based on work supported by the  French National Research Agency  under grant  ANR-11-JS03-0002. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the  French National Research Agency. For her support and fruitful discussions on the manuscript, the authors gratefully acknowledge Celine Agnes. The authors also acknowledge LPN cleanroom staffs in particular  S. Guilet ,  L. Ferlazzo , and  C. Roblin  for microfabrication advices. H. S. thanks  C. Gosse  and  S. Barbay  for fruitful discussions.
2. We considered two different settings: (i) the true analytic gradient is used in the estimation, and (ii) the covariates are discretized on a 8 × 8 regular grid, and the gradient is obtained through the interpolation of the covariates. This second setting corresponds to the more realistic case where covariates are only observed on a discrete grid, and the gradient needs to be approximated. The gradient approximations were performed for the covariates c 1 and c 2 using the R package nloptr  (Ypma et al. , 2014). The gradient of the Euclidean distance c 3 is computed exactly in both cases, as it would be in a real analysis. Boxplots of the parameter estimates in the 600 replications are shown in Figure  3. All parameters were correctly estimated in this benchmark scenario, even when the covariates were discretized to a coarse grid.
2. We evaluated the gradients of the covariates at each simulated location with a bilinear interpolation method. This procedure was done using the function interp. surface from the R package fields (Douglas  Nychka et al. , 2017). We then obtained the approximated gradient of the interpolated covariates with the R package nloptr. Point estimates and 95% confidence intervals of the habitat selection parameters β and the speed parameter γ 2 are displayed in Figure  5. The estimates of the speed parameter γ 2 were very close to the true value, for all simulation experiments. However, there was a lot of variability in the accuracy and precision of habitat selection parameter estimates. The uncertainty on the estimates of the habitat selection parameters decreased as the time interval increased.
(2018)  for more detail about the data set. To correct for the measurement error, and to follow the preprocessing performed by  Wilson et al. (2018) , we first fitted a continuous-time correlated random walk (CTCRW) to the tracks, using the R package crawl  (Johnson et al. , 2008; Johnson and London, 2018). The CTCRW is a continuous-time state-space model, that can be used on irregular and noisy telemetry data. The package crawl implements the Kalman filter for this  state-space model. We used the code provided by  Wilson et al. (2018)  to fit the CTCRW to each track, and obtained predicted locations for the times of the observations. We then fitted the Langevin movement model to the filtered tracks, using the inference method of Section 3. Most of the computation time is needed to evaluate the gradient of each covariate at all observed locations, which took about 1.
In tracking and tracing, the relatedness of bacterial sequences from outbreaks as well as the food production chain is assessed to determine if they could be part of the same transmission chain. However, as discussed in section 3. 3 WGS data must be backed up by epidemiological evidence to prove and characterise a transmission chain. Currently there are two main approaches to analysing genomic data to determine the relatedness between strains, namely SNP-based and the gene by gene-based approaches. Analysis of WGS data by either approach is a complex process in which multiple steps are combined to produce final results, such as SNP or allele matrices and phylogenetic trees  (Timme et al. , 2017). The large amount of data generated in WGS brings challenges for its analysis  (Deurenberg et al.
3. 2. 2. Gene by gene approach-Gene by gene analysis consists of assessing the variation in the coding regions i. e. the genes (or 'loci') of a bacterial genome  (Maiden et al. , 2013). In an extension to traditional 7-loci multi-locus sequence typing (MLST), the genes in either a defined core genome (cgMLST) or the whole genome (wgMLST), which includes the more variable accessory genes, are compared against a reference database of all known gene variants (alleles) for a particular species. Each gene or allele sequence is reduced to a number and genomes are compared based on the number of allele differences there are, comparable to the way the number of SNP differences are used. Since the reference is a database of loci and alleles from numerous strains, the analysis does not depend on the selection of a closely related reference strain for the precise assessment of the relatedness of genetically similar isolates.