esa-sceva/satcom-chunk-collection huggingface dataset:
# Representing and Inferring Massive Network Traffic Condition: A Case Study in Nashville, Tennessee
## 2 Materials and Methods
### CUR Integrated with Clustering  
The primary objective of this study is to explore the effects of incorporating clustering method into CUR sampling process. The reason for considering clustering method is to investigate whether homogenous subnetworks can enhance the overall performance of the network representation. The idea of integrating clustering analysis into CUR method is inspired by the methodology of devising a composite index for stock market such as Nasdaq-100.  
Stocks from different sectors will go through a rigorous screening process and then be selected to represent the whole market using a weighted average index. Similarly, to select most representative traffic detectors in a massive road network, the first step is to filter out columns with missing rate greater than 5% and to impute the remaining columns using data from adjacent time intervals (linear interpolation). Then, we can classify the detectors into different clusters based on their speed during the training period, followed by the normal CUR decomposition process within each individual cluster given a fixed compression rate (CR). Finally, the selected columns and corresponding relationship matrix will be merged as the final CUR outputs. The entire column selection process is displayed in Figure 4.  
[PERSON] et al. has taken into account the clustering method to compress large-scale traffic data [10]; however, they simply clustered the network according to road category instead of using a real clustering method based on the speed condition on the road. It is obvious that roads in the same category do not necessarily have similar traffic conditions.  
In this study, k-means clustering was adopted due to its simplicity, computational efficiency, and interpretability. A detailed description of k-means clustering is provided as follows.  
Figure 4: Column selection and decomposition process.  
In k-means, clusters are defined so that the total intra-cluster variation (known as total within-cluster variation) is minimized. Within-cluster variation is defined as: \(\sum\left|{{\left|{x-\mu_{i}}\right|}\right|^{2}}\), where we only consider x of a given cluster \(\mathbf{S}_{i}\), and \(\mu_{i}\) is the mean (centroid) of points in cluster \(\mathbf{S}_{i}\). Then, total within-cluster variation is to perform the previous calculation for each cluster and take the sum of all clusters.  
There are five overall steps in naive k-means method:  
* Step 1: specify k (number of clusters).
* Step 2: randomly select k instances from the data as the initial cluster centroids.
* Step 3: assign each instance to its closest centroid based on Euclidean distance.
* Step 4: for each of the k clusters, recompute the centroid by calculating the new mean of all the instances in the cluster.
* Step 5: repeat step 3 and 4 until centroids converge or the max number of iterations is reached.
## 2 Materials and Methods
### Experimental Design  
After infection, lambs (8 lambs per diet) were given _ad libitum_ access to control (CON) and WRF diets (Table 1) which was formulated according to the nutrient requirements of sheep [16]. The only difference of WRF diet from CON diet was untreated corn straw replaced with WRF-treated corn straw. The entire experiment consisted of two periods: 7 d of adaption and 70 d of formal feeding and sample collection period. All lambs were fed individual in the signal pen (2.0 m long and 2.0 m width), except for metabolic experiment and feed were provided twice daily at 06:00 and 18:00. During metabolic experiment, each lamb was confined in a single metabolic cage (140 cm long, 100 cm wide and 124 cm high) with ad libitum access to fresh water and feed. At each meal, the concentrate was offered first and then sufficient untreated corn straw or WRF-pretreated corn straw was provided to ensure that each lamb would have no more than \(\sim\)10% refusal. In the next morning before feeding, all orts including concentrate and straw were weighed and recorded. Chemical analyses of diets were conducted using the methods of the Association of Official Analytical Chemists [17] for dry matter (DM), ash, ether extract and crude protein (CP). Neutral detergent fiber (NDF) and acid detergent fiber (ADF) contents were determined according to [PERSON] et al. [18].  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Feed Ingredients (\% DM Basis)** & **CON Diet** & **WRF Diet** \\ \hline Ingredients (\%) & Corn straw & 40.0 & 0.0 \\ Fungal-pretreated corn straw & 0 & 40.0 \\ Corn meal & 21.0 & 21.0 \\ Soybean meal & 10.2 & 10.2 \\ Corn germ meal & 9.0 & 9.0 \\ Sunflower seed meal & 5.0 & 5.0 \\ Wheat bran & 10.5 & 10.5 \\ Limestone & 1.0 & 1.0 \\ CaHPO\({}_{4}\) & 0.8 & 0.8 \\ NaCl & 0.5 & 0.5 \\ Minerals and vitamins salt & 2.0 & 2.0 \\ Chemical composition (n = 8) & & \\ Dry matter (\% of the feed) & 97.8 & 97.3 \\ Crude protein (\% of dry matter) & 9.3 & 12.8 \\ Ether extract (\% of dry matter) & 5.4 & 5.2 \\ Neutral detergent fiber (\% of dry matter) & 51.7 & 48.7 \\ Acid detergent fiber (\% of dry matter) & 24.1 & 23.9 \\ Starch (\% of dry matter) & 22.6 & 20.8 \\ Metabolizable energy (M/J/kg of dry matter) & 13.5 & 14.8 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Ingredients and chemical composition of the experiment’s diets.
# A Machine Learning Method for Modeling Wind Farm Fatigue Load
## 2 Main Framework and Theoretical Knowledge
### Fatigue Load Modeling  
The raw load data cannot visually show the equivalent damage value because it is presented in time series. Therefore the rainflow counting (RFC) method and the Palmgren-Miner (PM) rule were proposed to calculate damage equivalent load (DEL). The calculation process of DEL is shown in Figure 2. The RFC converts the load signals to a series of cycles (\(n_{i}\)) in terms of stress amplitude(\(s_{i}\)) [21]. The PM rule combined with the S-N curve (material properties) transforms RFC results to equivalent damage load [22].  
Suppose that the cycles \(n_{i}\) and the corresponding stress amplitude \(s_{i}\) are obtained by the rainflow counting algorithm. \(i\) is calculated automatically and different \(i\) represent different loop amplitudes. The ratio of fatigue cycles to total life failure cycles can be defined as damage factor \(d_{i}\)[23]:  
\[d_{i}=\frac{n_{i}}{N_{i}} \tag{1}\]  
where \(n_{i}\) is cycles calculated from RFC; \(N_{i}\) is number of cycles to failure in terms of the stress amplitude \(S_{i}\).  
As shown in Figure 3, the PM rule is used to express the relationship between stress and the number of failure cycles, which is linear in the log-log coordinate system, where \(m\) is the Wohler exponent.  
The relationship can be expressed as  
\[N_{i}=\left(\frac{S_{0}}{S_{i}}\right)^{m} \tag{2}\]  
Then, Equation (1) can be expressed as  
\[d_{i}=\frac{n_{i}}{N_{i}}=\frac{n_{i}S_{i}^{n}}{S_{0}^{m}} \tag{3}\]  
Figure 1: Fatigue load prediction flowchart.  
Figure 2: Damage equivalent load estimation procedure.  
The total damage \(DEL\)(Damage Equivalent Load) can be summed for cycles of difference amplitudes:  
\[DEL=\frac{1}{S_{0}^{m}}\sum{n_{i}S_{i}^{m}} \tag{4}\]  
For a given number of cycles \(n_{eq}\) and the corresponding amplitude \(S_{eq}\), the DEL can be expressed as  
\[\frac{n_{eq}S_{eq}^{m}}{S_{0}^{m}}=DEL=\frac{1}{S_{0}^{m}}\sum{n_{i}S_{i}^{m}} \tag{5}\]  
Finally, when \(n_{eq}\) equals signal length in seconds, the DEL of the signal is equal to the damage of a 1 Hz sinusoidal signal of the same length with amplitude \(S_{eq}\). This method avoids the use of extreme loads.  
\[S_{eq}=\left(\frac{n_{i}S_{i}^{m}}{n_{eq}}\right)^{\frac{1}{m}} \tag{6}\]
## 5 Evaluation Metrics  
Metrics commonly used for evaluation include training accuracy, validation accuracy, recall, precision, F1-score, and the confusion matrix. These metrics are calculated using the following equations [36; 37]:  
1. Training/validation accuracy: This metric measures the percentage of true detections through total traffic trace. It is computed as follows: \[\text{Trained/validate Accuracy}=\frac{TP+TN}{TP+TN+FP+FN}\] (4) where _TP_ is the (true positive), which is the number of anomaly records that is correctly classified. _TN_ is the (true negative), which represents the number of normal records that is correctly classified. _FP_ is the (false positive), which is the number of normal records that is incorrectly classified. _FN_ is the (true negative), which represents the number of anomaly records that is incorrectly classified. 2. Recall: This metric is used to show the percentage of predicted intrusions against all intrusions presented. The aim is to achieve higher recall values. It is computed using the following equation: \[\text{Recall}=\frac{TP}{TP+FN}\] (5)
3. Precision: This metric is used to show the many intrusions predicted by the intrusion detection system (NIDS), which are actual intrusions. The aim is to achieve higher precisions than the lower false alarms. It is computed using the following equation: \[\text{Precision}=\frac{TP}{TP+FP}\] (6)
4. F1-score: This metric attempts to better measure the accuracy of an intrusion detection system (NIDS) by considering both the precision and recall. The aim is to achieve higher F1-scores. It is computed as follows: \[\text{F1-Score}=\frac{2\times\text{Precision}\times\text{Recall}}{\text{ Precision}+\text{Recall}}\] (7)  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Layer (Type)** & **Output Shape** \\ \hline Conv 2D & (None, 48, 48, 50) \\ Max Pooling & (None, 24, 24, 50) \\ Conv 2D & (None, 22, 22, 64) \\ Max Pooling & (None, 11, 11, 64) \\ Conv 2D & (None, 9, 64) \\ flatten (Flatten) & (None, 5184) \\ Dense & (None, 64) \\ Dense & (None, 1) \\ \hline \hline \end{tabular}
\end{table}
Table 1: CNN structure.  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Layer (Type)** & **Output Shape** \\ \hline Dense & (None, 18, 64) \\ Bidirectional & (None, 128) \\ Dense & (None, 32) \\ Dense & (None, 16) \\ Dense & (None, 12) \\ Dense & (None, 1) \\ \hline \hline \end{tabular}
\end{table}
Table 2: BDLSTM Structure.  
Using the confusion matrix (CM), all of the above metrics can be obtained, as well as receiver operating characteristics (ROC).
## 1 Introduction  
Sediment gravity flow or mass transport process [1] is the movement of sediment that occurs due to the pull of gravity from the top to the bottom of a slope. It is also defined as a sediment flow or sediment fluid mixture influenced by sediment grains moving due to the influence of gravity [2]. The gravity flow and sediment distribution can be affected by the basinal geometries, sediment supply, and also tectonic activity which becomes part of the triggering mechanisms [3, 4]. There are two types of gravity-flow sediment which are (i) sediment flow, and (ii) mass transport flow [5, 6, 7, 8, 9]. The end product is classified into (i) complex mass transport, (ii) debris flow, (iii) turbidite and debris co-genetics, and (iv) turbidite current. Turbidite is a dominant sequence of sediments deposited where this process can transport sediments from the shallower to the deep-water basin [10] and can travel further distances and be deposited in large areas if related to a submarine landslide [11]. Turbidite sediments have various characteristics [4], are usually classified based on several dominant components, have different facies characteristics, and can be identified through field observations and also borehole recordings. The three main components are (i) sand and sediment quantity [12], (ii) flow efficiency [13], and (iii) characteristics of sediment contributors and sediment supply [14], as summarized in Table 1.  
Understanding gravity-flow sediments is very important in understanding their natural phenomena [15; 16; 17] and in the application of science, especially in petroleum exploration and production, determining potential hazards during petroleum production and also determining the risk of geohazards in the future [18]. In northwest Borneo, specifically the Crocker Formation (Figure 1), a large expanse of Paleogene clastic sedimentary rocks of mainly deep-water origin has been outlined by [19; 20; 21; 22; 23; 24; 25; 26; 27; 28; 29; 30] and many more. The turbidite succession represents deposition in a tectonically active margin during the subduction of the Proto-South China Sea with the Sunda Shelf opening from Eocene to Holocene [31; 32; 33]. In this paper, the sedimentological characteristics of the northern part of the formation are described, based on field observation in the new cutting outcrops of the Pan Borneo Highway, which extends from Kota Kinabalu to the Telipok areas, and includes seven (7) sedimentary rock outcrop localities. The seven localities are as follows: (1) Hap Seng Quarry, Telipok, (2) KFC, UMS, (3) Prime Kondo University, (4) Lorong Suria Inanam, (5) Kg. Divatto, (6) Taman Okk Lojungah, and (7) Pekan Kinarut (Figure 1). Detailed sedimentological work in conjunction with the occurrence of sediment deposits with associated turbidites gives further insight into the sedimentary processes, the depositional setting, and the type of turbidite depositional systems in this Paleogene basin.
## 3 Methods
### Super Learner (SL)  
SL is a prediction algorithm designed to find the optimal combination of base learners [32]. The base learners should come from contextual knowledge and a large set of default algorithms, ranging from a simple linear regression model to a multi-step algorithm [33]. To combine the prediction results of various algorithms and eliminate the inherent bias of the algorithm, we use six base learners: SVR, KNN, DT, RF, GBM, and XGB [34].  
Base learners that predict the same validation set are strongly correlated. To explore the coefficients of the base learners, [PERSON] (1996) proposed non-negative constraints that ensure the coefficients are not sensitive to small changes in data. The predicted values of the SL are also constrained to be between the minimum and maximum predicted values of base learners. The coefficients selected by cross-validation are combined with the predictions of base learners on the original dataset, not the cross-validated one. In this study, we calculate the coefficients using 5-fold cross-validation to generate SL. Algorithm 1, presented below, describes the SL algorithm that utilizes these six base learners.  
```
Input : The dataset \(\mathcal{D}=\{(X_{i},Y_{i}):i=1,\cdots,n\}\) Output : A super learner Step 1: Train a base learner \(h_{k}\) with the dataset \(\mathcal{D}\) for \(k=1,\cdots,6\). Step 2: Split the dataset into 5 equal-size subsets \(for\ v=1,\cdots,5\) : Set the \(\sigma\)-th fold as a validation set \(V(v)\) and the rest folds as a training set \(T(v)\), \(T(v)=D\setminus V(v)\). \(end\ for\) Step 3: For each base learner \(h_{k}\), create the prediction set \(h_{k}(\mathcal{D}^{*})\) by aggregating predictions on \(V(v)\), i.e., \(h_{k}(V(v))\) in order \(v=1,\cdots,5\). \(for\ v=1,\cdots,5\) : Train a base learner \(h_{k}\) on \(T(v)\) and save the prediction set \(h_{k}(V(v))\). \(end\ for\) Step 4: Let \(a_{k}\ (k=1,\cdots,6)\)be the coefficient giving weight to \(h_{k}(\mathcal{D}^{*})\) under the constraints. \(\sum\limits_{k=1}^{6}a_{k}h_{k}(\mathcal{D}^{*})\), \(\quad\quad s.t.\ \ a_{k}\geq 0\ \forall k,\ \ \ \sum\limits_{k=1}^{6}a_{k}=1\) Step 5: Determine the weight vector \(a\) that minimizes the cross-validation error using the following optimization. \(\hat{a}=\underset{a}{argmin}\sum\limits_{i=1}^{n}\ (Y_{i}-\sum\limits_{k=1}^{6}a_{k}h_{k}( \mathcal{D}_{i}^{*}))^{2}\) Step 6: To create Super Learner, combine \(\hat{a}\) with the prediction of base learners that are generated in line \(1\).
```  
**Algorithm 1**Super Learner
# Joint QoS and Congestion Control Based on Traffic Prediction in SDN
## 7 Simulation Results  
In this section, the proposed scheme is evaluated via different measurement metrics.
## 2 Materials and Methods
### Architecture Design of the Cloud-Edge Hybrid Intelligent System (CEHIS)
#### 2.1.2 Modeling  
In the simulation experiment, we model a computational node within the CEHIS as a one-dimensional vector \([node\_id,node\_type,cpu,memory,storage]\). The _node_id_ is the unique identifier for the node. The _node_type_ denotes the type of node, which is either an edge-based node (edge) or a cloud-based node (cloud). The _cpu_ represents the computational resources allocated to the node, while _memory_ indicates the memory resources provided to the node, and _storage_ signifies the storage resources available to the node. If a task is assigned to an edge node, it can be executed immediately; if it is allocated to a cloud-based node, it requires a certain amount of time before it can be executed, accounting for delay introduced by the transmission time.
###### Abstract
It has seen a substantial rise in its application within the realm of communication, providing a new avenue for enhancing the performance of physical layer communication systems [12; 13]. Deep learning has proven effective in OFDM communication scenarios where the channel is either unknown or of a high complexity [14; 15; 16]. In [17], a neural network is trained to fit the transmission process of the channel, and the receiver of OFDM mainly learns how to recover the transmitted symbols from the received signal, rather than estimating the time-frequency response of the channel. In [18], an OFDM communication receiver system based on a five-layer fully connected deep neural network was developed for channel estimation and equalization in UWA communication. The model performed advantages over traditional algorithms in situations with limited pilot subcarriers. In [19], a computer vision tool is applied in channel estimation work for the first time, the authors estimate the channel matrix with the help of a denoising convolutional neural network (DnCNN). In [20], the channel response is regarded as an image, meaning that image super-resolution (SR) and image restoration (IR) techniques are introduced. The specific method is cascaded super-resolution convolutional neural network (SRCNN) and denoising neural network (DnCNN) completes channel estimation, and its performance is approximately equal to the MMSE estimator at low SNR. In [21], the up-sampling function in the network is implemented as transposed convolution layer, which can scale up image height and width by different factors, which is suitable for any pilot pattern. Considering that the use of a deep learning-based channel estimation method can achieve a certain performance improvement compared to traditional channel estimation methods, it can achieve higher interpolation accuracy and noise reduction. Very deep super-resolution (VDSR) is one of the most representative networks in the field of deep convolutional neural networks for super-resolution, which can be utilized as the foundation for designing channel estimation neural network. This paper proposes an OCDM channel estimation method based on SR technology which improves the performance and capability of multicarrier underwater acoustics communication systems in shallow water acoustic channels. The remainder of this paper is organized as follows: in Section 2, a brief survey of the OCDM model as well as the relationship between channel estimation and super-resolution problem are formulated. Section 3 presents the structure of the super resolution-based channel estimation method. The simulation results are presented in Section 4, and the conclusions are summarized in Section 5.
###### Abstract  
Many components in industry are subjected to high loads during operation and therefore often do not reach their intended service life. Conventional steels frequently do not provide sufficient protection against wear and corrosion. One solution is to coat these components using methods like thermal spraying to apply cermet coatings such as Cr\({}_{3}\)C\({}_{2}\)-NiCr or WC-Co-Cr. In light of increasingly strict environmental regulations, more eco-friendly alternatives are needed, especially ones that use little or no Cr, Ni, Co, or W. Another alternative is the recycling of powder materials, which is the focus of this research project. This study investigated whether filter dust from an HVOF system could be used to develop a new coating suitable for use in applications requiring resistance to wear and corrosion. This is challenging as the filter dusts have heterogeneous compositions and irregular particle sizes. Nevertheless, this recycled material, referred to as \"Green Cermets\" (GCs), offers previously untapped potential that may also be of ecological interest. An established WC-Co-Cr coating served as a reference. In addition to friction wear and corrosion resistance, the study also examined particle size distribution, hardness, microstructure, and susceptibility to crack formation at the interface and inside the coating. Even though the results revealed a diminished performance of the GC coatings relative to the conventional WC-CoCr, they may still be applicable in various industrial applications.  
recycling; HVOF; WC-Co-Cr; thermal spraying; filter dust; corrosion; wear Article
## 7 Proposed Solution
### ANN-Based Detection System
#### 7.1.2 Data Preprocessing  
Data preprocessing is a crucial step in any machine learning or data analysis project. It involves cleaning, transforming, and reformatting raw data to make it suitable for analysis. The following are some of the common data preprocessing steps:  
* Handling Missing Values: Missing data is a common problem in datasets, and it can create bias and affect the accuracy of the model. Therefore, it is important to handle missing values before training the model. The most common techniques for handling missing values include deletion, imputation, and prediction. Deletion techniques include listwise deletion, pairwise deletion, and mean substitution. Imputation techniques include mean imputation, median imputation, and regression imputation. Prediction techniques include k-nearest neighbors (KNN), expectation-maximization (EM), and regression imputation. The choice of technique depends on the nature and distribution of the missing data in the dataset. In the frame of this work, the missed data are removed since they have small proportionally number regarding the hole dataset traffic.
* Feature Scaling: Feature scaling is a technique used to normalize the data and bring all the features to a common scale. This is important because the features may have different units, ranges, or distributions, and this can affect the performance of the model. The most common techniques for feature scaling include min-max scaling, standard scaling, and robust scaling. Min-max scaling scales the data to a range between 0 and 1. Standard scaling scales the data to have a mean of 0 and a standard deviation of 1. Robust scaling scales the data based on the median and interquartile range, which is more robust to outliers. This implementation considers the standard scaling.
* One-Hot Encoding: One-hot encoding is a technique used to transform categorical data into numerical data. This is important because most machine learning algorithms only work with numerical data. One-hot encoding creates a binary vector for each category, where the vector is all zeros except for the index corresponding to the category, which is set to 1. For example, if a dataset has a categorical feature for color with three categories (red, green, blue), one-hot encoding will create three binary features (red = 001, green = 010, blue = 100) to represent the color feature. The categorical data in the dataset was eliminated while selecting the best feature; therefore, this encoding was not used.