esa-sceva/satcom-chunk-collection huggingface dataset:
# Deep Neural Networks Training by Stochastic Quasi-Newton Trust-Region Methods
### Contribution and Outline  
The BFGS update is the most widely used type of quasi-Newton method for general optimization and the most widely considered quasi-Newton method for general machine learning and deep learning. Almost all the previously cited articles considered BFGS, with only a few exceptions using the symmetric rank one (SR1) update instead [29]. However, a clear disadvantage of BFGS occurs if one tries to enforce the positive-definiteness of the approximated Hessian matrices in a non-convex setting. In this case, BFGS has the difficult task of approximating an indefinite matrix (the true Hessian) with a positive-definite matrix which can result in the generation of nearly singular Hessian approximations.  
In this paper, we analyze the behavior of both updates on real modern deep neural network architectures and try to determine whether more efficient training can be obtained when using the BFGS update or the cheaper SR1 formula that allows for indefinite Hessian approximations and, thus, can potentially help to better navigate the pathological saddle points present in the non-convex loss functions found in deep learning. We would like to determine whether better training results could be achieved by using SR1 updates, as these allow for indefinite Hessian approximations. We study the performance of both quasi-Newton methods in the trust region framework for solving (1) in realistic large-size DNNs. We introduce stochastic variants of the two quasi-Newton updates, based on an overlapping sampling strategy which is well-suited to trust-region methods. We have implemented and applied these algorithms to train different convolutional and residual neural networks, ranging from a shallow LeNet-like network to a self-built network with and without batch normalization layers and the modern ResNet-20, for image classification problems. We have compared the performance of both stochastic quasi-Newton trust-region methods with another stochastic quasi-Newton algorithm based on a progressive batching strategy and with the first-order Adam optimizer running with the optimal values of its hyperparameters, obtained by grid searching.  
The rest of the paper is organized as follows: Section 2 provides a general overview of trust-region quasi-Newton methods for solving problem (1) and introduces the stochastic algorithms sL-BFGS-TR and sL-SR1-TR, together with a suitable minibatch sampling strategy. The results of an extensive empirical study on the performance of the considered algorithms in the training of deep neural networks are presented and discussed in Section 3. Finally, some concluding remarks are given in Section 4.
# Methods for Corrosion Detection in Pipes Using Thermography: A Case Study on Synthetic Datasets
## 1 Introduction
### Subsurface Defects  
Subsurface defects in materials significantly impact the strength, durability, and overall performance of these objects. Early detection is crucial for ensuring the safe and effective functioning of materials and structures. Non-destructive testing (NDT) methods evaluate the quality and condition of materials and structures without causing damage, allowing for regular and ongoing monitoring. This ensures that any potential issues are identified and addressed promptly [3].
# Automatic Decentralized Vibration Control System Based on Collaborative Stand-Alone Smart Dampers
## 5 Results
### Experimental Results
#### 5.3.1 Non-Dimensional Damping  
The decay method proposed in Section 2.1 has been tested for all the three actuators. As an example, only the results related ot Actuator 1 are reported, which are shown in Table 4. Looking at the results, the decay method provides results consistent with Hilbert transformation method (Figure 6).  
\begin{table}
\begin{tabular}{c c c c c c} \hline \multirow{2}{*}{**Mode**} & \multirow{2}{*}{**Decay Method**} & **Hilbert tr.** & **Half-Power** & **Error w.r.t.** & **Error w.r.t.** \\  & & **Method** & **Method** & **Hilbert tr. m. [\%]** & **Half-Power m. [\%]** \\ \hline
**2** & 0.0125 & 0.0146 & 0.0321 & 14.3 & 61.1 \\
**3** & 0.0035 & 0.0036 & 0.0042 & 2.2 & 17.3 \\
**4** & 0.0057 & 0.0063 & 0.0102 & 9.7 & 44.3 \\ \hline \end{tabular}
\end{table}
Table 4: Estimation of the damping ratio \(\xi_{i,j}\) for the Actuator 1 compared with Hilbert transform and Half-power methods.  
Figure 5: Comparison between the vibration of the uncontrolled (blue line) and controlled (red line) system on the three sensors locations in numerical simulation.
# Research on Coal and Gangue Recognition Model Based on CAM-Hardswish with EfficientNetV2
## 4 Experimental Results and Analysis
### Comparative Analysis of Model Performance  
To verify the effectiveness of the improved EfficientNetV2 network, the performance was experimentally compared before and after modifications to the attention mechanism module and activation function based on the EfficientNetV2-S basic feature network backbone. EfficientNetV2-SE with the SE attention mechanism and ReLu activation function, EfficientNetV2-CAMReLu with the CAM attention mechanism and ReLu activation function, EfficientNetV2-CBAM with the CBAM attention mechanism and ReLu activation function, and the EfficientNetV2-CAMHardswish network were compared. Figure 9 shows the test accuracy graph, where EV2 stands for EfficientNetV2-S.  
According to Figure 9, the accuracy curve of the improved EfficientNetV2 network is higher and converges faster than that of the original network. The graph shows that after 50 epochs, the EfficientNetV2-S network based on CAM-Hardswish converged more quickly and steadily and achieved an accuracy improvement of 3.98% compared with the original model, reaching 98.24%. As shown in Table 3, the introduced CAM module and the usage of the Hardswish activation function resulted in a significant increase in the  
Figure 9: Test accuracy of the improved network model compared with the original network and its variants.  
accuracy of the newly formed network model as compared with the original network and its variants.
## 3 Implementation of Proposed Method
### Single Feature Extraction Network
#### 3.2.3 Structure of Single Feature Extraction Network  
The feature extraction network designed in this paper mainly uses the improved depthwise separable convolution described in Section 3.2.2, and has the following considerations: (i) The dataset of sonar images is small, so fewer network layers are used to reduce the complexity of the network; (ii) The size of the convolution kernel is closely related to the amount of model parameters and computation. Larger convolution kernel will lead to excessive model parameters and loss of image details, so we tend to use a small convolution kernel in this paper; and (iii) Pooling layer will lose several original features while controlling for overfitting, so it is necessary to reduce the use of pooling layer in feature extraction.  
All the above considerations are to make the network more suitable for underwater object classification and achieve better results. The structure of the single feature extraction network designed in this paper is shown in Figure 4, which has two traditional convolutions and seven depthwise separable convolution blocks (Block). Among the structure, each Block is composed of several improved depthwise separable modules. The network layer structure and parameters of the improved depthwise separable module are shown in Table 1, which is composed of three different convolutions. The structure and parameters of the whole feature extraction network are shown in Table 2, where \(t\) is the extension factor of the convolution layer in the depthwise separable module, \(c\) and \(n\) are the numbers of output channels of the network layer and the repetition times of the depthwise separable module, and \(s\) is the convolution stride.
# Toward Comprehensive Chronic Kidney Disease Prediction Based on Ensemble Deep Learning Models
## 1 Introduction  
### Overview  
The kidney is the main organ that controls the human body's blood balance and blood pressure; it also plays a role in producing important hormones. Kidney Disease Improving Global Outcomes (KDIGO) defined chronic kidney disease (CKD) as a structural or functional abnormality of the kidney, which persists for more than three months [1].  
Several conditions and disorders may lead to kidney diseases, including kidney stones, nephrolithiasis, kidney cyst formation, a rare blood disorder, muscle tissue breakdown, hemolytic uremic syndrome, blood clots, and glomerulitis, etc. [2, 3].  
Many cases of CKD do not present symptoms until the late stages of the disease; this makes estimating the actual prevalence of the disease complicated [4]. However, according to medical statistics, in 2005, CKD caused the death of approximately 38 million out of 57 million cases of CKD. During the COVID-19 period, the mortality rate among COVID-19 patients with CKD was 44.5%, whereas it was 4.5% among non-COVID-19 CKB patients. According to World Health Organization (WHO), by 2050, more than 150 million will have type 2 diabetes, which is considered the leading cause of several kidney disorders [5]. Common kidney abnormalities, such as hydronephrosis, cysts, and stones, are easily avoided and treated in early stages [6]. However, such disorders may lead to CKD and kidney tumors (i.e., cardiorenal syndrome, uremia). CKD is associated with primary adverse outcomes, but cardiovascular disease (CVD) is considered the leading cause of death in this population [7]. Consequently, screening and identification of chronic kidney disease patients during earlier stages can provide interventions that may modify the natural history and reduce the risk of progression to end-stage kidney disease and major cardiovascular events [8].
## 3 Results  
The prevalence of 4 categories of SOS in control vs. 3 different phenotype EETG in number was 0 = 1323 \(>\) 23 \(>\) Control \(>\) 13, 1 = 13 \(>\) 23 \(>\) Control \(>\) 1323, 2 = 13 \(>\) 1323 \(>\) Control \(>\) 23 and 3 = 1323 \(>\) 23 \(>\) Control \(>\) 13, respectively (Figure 2). SOS in control vs. 2 different phenotype EETG in position are more prevalent in, 0, 1, 2 and 3 = buccal (Figure 3). Overall SOS prevalence was 0 = 12%, 1 = 7%, 2 = 9% and 3 = 72.22%, respectively (Figure 4).  
Table 2 shows the details of descriptive and comparative disparities among control and 3 different phenotypes of EETG in position (13, 23 and 1323). Overall STM was presented, showing significant disparities in 6 out of 7 measured variables. In all 7 measured variables of STM, the common trends of shorter measurements were observed in  
\begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline
**Variable** & \multicolumn{1}{c}{**Mean (mm)**} & \multicolumn{1}{c}{**SD (mm)**} & \multicolumn{1}{c}{**MD (mm)**} & \multicolumn{1}{c}{**95\%Confidence Interval**} & \multicolumn{1}{c}{\(p\) Value**} \\  & & & & **Lower** & **Upper** & \\  & & & & **Bound** & **Bound** & \\ \hline a. TS-Pclin & 13 & 8.726 & 1.327 & 13 vs. 23 & \(-0.885\) & \(-1.732\) & \(-0.038\) & 0.035 \\  & 23 & 9.611 & 1.983 & 13 vs. 1323 & 0.451 & \(-0.346\) & 1.248 & 0.801 \\  & 1323 & 8.274 & 1.774 & 13 vs. 4.00 & \(-0.400\) & \(-1.263\) & 0.463 & 1.000 \\  & 4.00 & 9.126 & 1.870 & 23 vs. 1323 & 1.336 & 0.528 & 2.144 & \(<\)0.001 \\  & & & & 23 vs. 4.00 & 0.485 & \(-0.388\) & 1.358 & 0.843 \\  & & & 1323 vs. 4.00 & \(-0.851\) & \(-1.676\) & \(-0.027\) & 0.039 \\ b. SA-SP & 13 & 8.380 & 1.537 & 13 vs. 23 & \(-0.388\) & \(-1.159\) & 0.382 & 1.000 \\  & 23 & 8.769 & 1.602 & 13 vs. 1323 & 0.597 & \(-0.128\) & 1.323 & 0.177 \\  & 1323 & 7.783 & 1.672 & 13 vs. 4.00 & \(-0.165\) & \(-0.950\) & 0.620 & 1.000 \\  & 4.00 & 8.546 & 1.535 & 23 vs. 1323 & 0.986 & 0.250 & 1.721 & 0.003 \\  & & & & 23 vs. 4.00 & 0.223 & \(-0.571\) & 1.018 & 1.000 \\ c.TS-DS & 13 & 9.798 & 2.429 & 13 vs. 23 & \(-0.766\) & \(-1.915\) & 0.384 & 0.466 \\  & 23 & 10.564 & 2.767 & 13 vs. 1323 & 0.322 & \(-0.760\) & 1.403 & 1.000 \\  & 1323 & 9.476 & 2.463 & 13 vs. 4.00 & \(-0.751\) & \(-1.921\) & 0.420 & 0.536 \\  & 4.00 & 10.549 & 1.617 & 23 vs. 1323 & 1.087 & \(-0.009\) & 2.184 & 0.053 \\  & & & & 23 vs. 4.00 & 0.015 & \(-1.169\) & 1.199 & 1.000 \\  & & & & 1323 vs. 4.00 & \(-1.072\) & \(-2.191\) & 0.046 & 0.068 \\ \hline \hline \end{tabular}
\end{table}
Table 2: STM details and comparison in relation to EETG in number with matched control.  
Figure 4: SOS prevalence in relation to EETG position.  
Figure 3: SOS prevalence in relation to EETG (13 and 23 = unilateral EET, 1323 = bilateral EETG, and 4 = control).  
Table 3 shows the details of descriptive and comparative disparities among occlusal and 2 different phenotypes of EETG (buccal and palatal). Overall STM was presented, showing insignificant disparities in all 7 measured variables. In 6 out of 7 measured variables of STM, the common trends of shorter measurements were observed in the palatal EETG group.
## 5 Illustrative Example
### Discussion of Efficiency and Accuracy  
Among all the approaches tested above, BC-based and RE-based approaches are of the highest efficiency, but the performance of their results is low, especially for cases with different bridge vulnerabilities. On the contrary, MI-based and JMI-based approaches presented in this paper have a significant improvement in mitigation performance but are less efficient. However, as verified in this example, the proposed method could provide a great approximate to the real optimal solution, but the computational cost is only 1/100 of the enumeration method. Given that this is a sample-based method, the number will be much smaller when dealing with multidimensional problems, which is impossible for heuristic algorithms.  
In addition to the seismic mitigation performance discussed above, a more detailed comparison is how accurate the JMI-based retrofit strategy is as an alternative to the optimal solution. Table 5 lists the optimal strategy and the JMI-based retrofit strategy in each case, where the different selected bridge ID is shaded. As indicated, the accuracy of the JMI-based retrofit strategy is overall very high, and the slight difference mainly occurs on the bridges with close MI values. Note that the difference comes from two aspects. One is that the proposed method, which greatly increases the decision efficiency, is essentially an approximation approach based on feature selection, the other is that the randomness in MCS could introduce errors in the estimation of MI and JMI values.  
The sample size in this example is arbitrarily set as 10,000 to verify the applicability of the proposed method. However, in engineering practice, the users need to decide on a reasonable number of samples to balance the computational efficiency and accuracy, and other refined stochastic sampling algorithms are also recommended, which do not affect the use of the proposed methodology.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline \multirow{2}{*}{**No. of Case**} & \multicolumn{4}{c}{**Sum of \(P_{F}\) Gap from \(m\)=1 to \(m\)=9**} \\ \cline{2-5}  & **BC-Based** & **RE-Based** & **MI-Based** & **JMI-Based** \\ \hline Case 1 & 14.58\% & 8.33\% & 1.20\% & 0.66\% \\ \hline Case 2 & 5.42\% & 7.07\% & 0.52\% & 0.11\% \\ \hline Case 3 & 58.53\% & 33.74\% & 1.40\% & 0.76\% \\ \hline Case 4 & 14.61\% & 33.47\% & 1.41\% & 0.21\% \\ \hline \hline \end{tabular}
\end{table}
Table 4: Sum of failure probability gaps between each approach and optimal solution.
## 3 Architecture  
This section describes the architecture and components of the MIRD system (see Figure 2). As previously stated, the primary objective of MIRD is to forecast selected PQ indexes for nearly 800 PQM in the CFE-DCO power distribution network. This results in a big-data environment that places the following primary requirements on MIRD's architecture:  
1. MIRD relies on an efficient and resilient computing framework mounted on the available infrastructure. A set of widely tested data processing and analytics libraries integrate this computing framework.
2. Although the first version of MIRD considered execution on a single server, the system architecture must provide the foundations for parallelization, scaling, and fault-tolerance in a future cluster-based operation.
3. MIRD must be developed, to the maximum possible extent, using widely used, open-source software.  
The preceding requirements match the foundation of most big-data architectures implemented nowadays. In this context, the architecture pattern selected for MIRD was the generic data pipeline: a succession of decoupled software components where the output of each component constitutes the input for the downstream component, with layers for artifact persistence in the middle. This architecture pattern renders simplicity and autonomy to development operations, allows the system to run processes (whenever possible, in parallel), and enables the components, if required, to scale independently. Moreover, the artifact-persistence layers in the middle not only store information but also serve a couple of essential purposes: (a) to perform faster queries on data over the different processing stages, and (b) to feed data for processing or analytics variants at different pipeline stages. MIRD components were coded in Python to ease the integration of the two programming resources that were considered essential to the architecture requirements: the Apache Spark [16] unified engine for large-scale data analytics and the SciKit Learn [17] library for scientific computing. The following subsections provide a detailed description of the components that integrate MIRD's architecture.  
Figure 2: The MIRD pipeline. The left column shows the data persistence layers. The right column shows the pipeline components and the technologies they use.
# Effect of Soil Washing with Ferric Chloride on Cadmium Removal and Soil Structure
## 3 Results and Discussion
### Changes in Total Organic Carbon during Extraction  
Compared with the untreated soil, there were no significant losses of TOC in the control and initial pH 4.0 FeCl\({}_{3}\) treatments (Table 1). However, in the initial pH 2.0 HCl and in the pH 2.0 and 3.0 FeCl\({}_{3}\)-treated soils, the content of TOC decreased significantly by 15.5%, 27.1%, and 17.5%, respectively.  
The organic carbon in the treated soils was partly released into the solutions. In contrast to the TOC in the soil, the amount of DOC in the initial pH 2.0 FeCl\({}_{3}\) solution was significantly higher than that in the other treatments, followed by the HCl-treated solution (Table 1). The release of DOC into the solutions may be attributed to the dissolution of iron oxides in soils, which are important sorbents for DOC. [PERSON] et al. reported that microbial iron reduction on ferrihydrite-adsorbed natural DOC led to the coupled release of DOC and Fe(II) [30]. Similarly, in our study, the DOC in the solutions positively and significantly correlated with the concentration of Fe(II) (correlation coefficient = 0.987; \(p\) < 0.01). However, it is not clear whether reducing microorganisms played a role under the conditions of initial pH 2.0 in HCl- and FeCl\({}_{3}\)-treated soils. In addition, the amount of DOC in solution increased with the decrease in equilibrium pH. Thus, a change in soil pH may also contribute to the DOC loss [31; 32].
## 2 Background
### Bit Error of MLC NAND Flash  
Bit errors in MLC NAND flash memory are mainly caused by program disturbances and leakage of current (also called retention errors). When programming one selected physical page, a high voltage such as 20 Volts must be applied to the word line (WL) of the target page. This high voltage will then cause disturbances on other unselected pages. An illustration of this process is given in Figure 1a. These disturbances will cause some electrons to be injected into the floating gate (FG) from the base. As a result, the reference voltage of the current will change and the information stored in this cell could be altered.  
On the other hand, the conductivity of the oxide may be higher as the P/E cycle increases or the temperature rises. The electrons stored in the FG will leak into the base [36], as shown in Figure 1b. This will also change the reference voltage of the FG and damage the stored information. The leaked current can be calculated using Equations (1) and (2) [20], where A, B and \(V_{hi}\) are constants; \(E\) is the electric field intensity of the Control Gate (CG); \(E_{\alpha x}\) is the electric field intensity of the oxide; \(C_{\alpha\alpha}\) and \(C_{\alpha\alpha}\) represent the capacitance of ONO (\(S_{i}O_{2}\)-\(S_{i3}N_{4}\)-\(S_{i}O_{2}\)) and the oxide, respectively; \(V_{hi}\) is the reference voltage of the cell and \(T_{\alpha\alpha}\) is the thickness of the oxide. Through this model we can observe that the lower the reference voltage of the flash memory cell, the less current will leak. The BER of this cell also would be lower:  
\[I_{\textit{total}}=A\cdot E^{2}\cdot\exp(-B\,/\,E_{\alpha x}) \tag{1}\]  
\[E_{\alpha x}=\{C_{\textit{ano}}\,/\,(C_{\alpha x}+C_{\textit{ano}})\}\cdot(V_{ \textit{hi}}-V_{\textit{hi}})\,/\,T_{\alpha x} \tag{2}\]
## 5 Performance Evaluation
### Implementation Technical Details
#### 5.2.4 Validation Technique  
For the time series data, using k-fold cross validation does not work perfectly. This is because it does not consider the order of data [19].  
In this work, we use a validation technique which is called walk-forward (i.e., a technique preserving the order of data).  
As Figure 1 shows, the walk-forward technique divides dataset into several units (e.g., in our work, one week as a unit). After that, units are chronologically ordered, and in each run, all data available before the unit to predict is considered as the training set, and the unit to predict is used as test-set. Finally, the model accuracy is considered as the average among runs. The number of runs is equal to one less than the number of data units [19]. We considered around 84% of the dataset (i.e., five continues last weeks/units out of 6 weeks) for training and testing our models.