esa-sceva/satcom-chunk-collection huggingface dataset:
# An Efficient Document Retrieval for Korean Open-Domain Question Answering Based on ColBERT
## 4 Korean Open-Domain Question Answering Dataset Construction  
### Training Dataset  
To construct a dataset for Korean open-domain question answering, we used four distinct domains datasets provided by AI-Hub [28]: _general knowledge_[29], _machine reading comprehension_[30], _book material machine reading comprehension_[31], and _large-scale web-based Korean corpus_[32]. _General knowledge_ is the used MRC dataset containing knowledge from the Korean Wikipedia, with a query count of 67,775, a query length in characters ranging from 1 to 92, and a document count of 43,499. The document length in characters ranges from 46 to 1782. _Machine reading comprehension_ is a news-based MRC dataset and contains news article data in nine fields (politics, economy, society, etc.). It consists of a query count of 440,322, a query length of 0-254, a document count of 89,164, and a document length of 44-26,747. _Book material machine reading comprehension_ is an MRC dataset utilizing book material on various topics. It consists of a query count of 949,332 with a query length of 6-231, a document count of 181,198, and a document length of 343-678. _Large-scale web-based Korean corpus_ for document augmentation is based on website-based large-scale text data. It contains more than 1 billion words of text data and includes news from 17 different fields (IT/tech, culture/fashion/beauty, international, etc.). The overall construction process of the dataset is depicted in Figure 3, and the code to generate the data is available at [[https://github.com/movie112/ColBERTforKorean](https://github.com/movie112/ColBERTforKorean)]([https://github.com/movie112/ColBERTforKorean](https://github.com/movie112/ColBERTforKorean)) (accessed on 20 November 2023).  
The dataset is structured to reflect the format of the MS MARCO Ranking [33] dataset used in ColBERT. First, we limit the data to documents with lengths of 1000 or less, based on the maximum document length in [33], using three datasets: general knowledge, machine reading comprehension, and book material machine reading comprehension. The dataset is constructed in the form of triples <q, d+, d-> for training, following the format of [33]. The query and relevant document pairs <q, d+> are derived from the general knowledge, machine reading comprehension, and book material machine reading comprehension datasets. To obtain low-relevance documents d-, we extracted the top 20 documents using the BM25 algorithm from the entire document collection, which is augmented with AI-Hub's large-scale web-based Korean corpus dataset.  
To assess performance by query length, we partitioned datasets based on maximum query lengths. Queries start from a length of 5, set to reduce noise, and datasets were partitioned with max query lengths of 20, 50, and the overall longest query.The statistical information of the final dataset used for training is shown in Table 1.
# Iterative Receiver Design for the Estimation of Gaussian Samples in Impulsive Noise +
## 2 System Model  
A sequence of Gaussian samples \(\{s_{k}\}_{k=0}^{K-1}\) is transmitted over a channel impaired by impulsive noise. The received samples are thus expressed by:  
\[y_{k}=s_{k}+n_{k}\qquad\ \ (k=0,1,\cdots,K-1) \tag{1}\]  
where \(\{n_{k}\}_{k=0}^{K-1}\) is a sequence of (zero-mean) additive Gaussian noise samples whose variance depends on the time index \(k\), as detailed in the following.  
The transmitted samples are assumed to be correlated according to an autoregressive model of order one. The AR(1) sequence is thus obtained as the output of a single-pole infinite impulse response (IIR) digital filter, fed by independent and identically distributed (i.i.d.) Gaussian samples \(\{\omega_{k}\}_{k=0}^{K-1}\), where \(\omega_{k}\sim\mathcal{N}(0,\sigma_{\omega}^{2})\):  
\[s_{k}=a_{1}s_{k-1}+\omega_{k}\quad, \tag{2}\]  
where \(a_{1}\) is the pole of the filter. The variance \(\sigma_{s}^{2}\) of \(s_{k}\) is taken as a reference; hence, we set \(\sigma_{\omega}^{2}=(1-a_{1}^{2})\sigma_{s}^{2}\) in (2).  
The noise samples \(n_{k}\) in (1) follow the statistical description of either the Markov-Gaussian model or of the Markov-Middleton class \(A\) noise model. Both models, as detailed in the following subsections, account for a correlation among noise samples, which in turn reflects the physical property of burstiness. As is, in fact, well known, impulsive noise events occur in bursts, i.e., through a sequence of noise samples whose average power (variance) becomes suddenly larger.
## 5 Methods
### Application of CNN-LSTM for Artifact Removal
#### 5.1.4 Proposed CNN-LSTM Structure and Training
A dataset of 80,000 generated examples was prepared for training the neural network. The data were split into three subsets: training (70%), validation (15%), and testing (15%). Table 3 provides details about how the data were divided. Training was conducted using batches of size 32. This choice balanced training speed with the stability of the optimization process. The model was trained for a maximum of 50 epochs. An early stopping mechanism was implemented, monitoring the loss on the validation set, with a patience of 3 epochs. The algorithm stopped at the 7 th epoch, where \begin{table}
\begin{tabular}{c c c} \hline \hline
**Data Subset** & **Number of Examples** & **Description** \\ \hline \multirow{3}{*}{Training} & \multirow{3}{*}{56,000} & A total of 70\% of the data to \\  & & optimize network weights \\  & & during training. \\ \hline \multirow{3}{*}{Validation} & \multirow{3}{*}{12,000} & A total of 15\% of data used to \\  & & monitor model performance. \\ \hline \multirow{3}{*}{Test} & \multirow{3}{*}{12,000} & A total of 15\% of data for model \\  & & quality assessment after training. \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Splitting the dataset into training, validation, and test sets. the MSE was 0.092 for the training data and 0.211 for the validation data. To improve generalization and robustness, data augmentation was applied by injecting diverse and intensified synthetic artifacts--stronger and more varied than real EEG disturbances. This helped the model learn to generalize across a wider spectrum of potential noise. Training results were stable across different random weight initializations. After training, the MSE on the test data was 0.189, indicating a properly executed training process and acceptable generalization performance, especially considering the complexity and variability of EEG signals. Importantly, visual and spectral analysis confirmed that the model did not introduce distortions in clean signal segments, which is critical for reliable artifact removal.
## 2 Materials and Methods
In previous studies, the RTD demonstrated good fundamental characteristics related to energy dependence, dose linearity, dose rate dependence, and angular dependence [62; 63; 64]. The entrance surface doses to the lenses, thyroid lobes, parotid glands, and sublingual glands were measured because these organs are radiosensitive in the head and neck region [65]. First, the sensors were placed at positions corresponding to the right and left lenses and thyroid lobes on the surface of the phantom (ch.1: right lens; ch.2: left lens; ch.3: right lobe of thyroid; and ch.4: left lobe of thyroid). Next, the sensors were placed at positions corresponding to the right and left parotid and sublingual glands on the surface of the phantom, as shown in Figure 4 (ch.1: right parotid gland; ch.2: left parotid gland; ch.3: right sublingual gland; and ch.4: left sublingual gland). Sensors of the dosimeter were attached to the surface of the phantom, because we did not have a child phantom in which dosimeters could be placed inside. Since the actual organs are distributed at a small distance from the body surface, for more accurate dosimetry, the sensors of the dosimeter must be embedded within the phantom. Dose calibration of RTD was performed against an ionization chamber dosimeter (9015, Radcal Corporation, Monrovia, CA, USA) with a 6 cm\({}^{3}\) thimble ionization chamber (10X5\(-\)6, Radcal Corporation) traceable to a national standard. The chamber dosimeter and RTD were placed freely in air adjacent to each other in the irradiated field at the same distance from the X-ray focus. The correction factor was determined using this calibration procedure. Figure 4: RTDs were placed on lens, thyroid, parotid gland, and sublingual gland surfaces of the phantom surfaces of the pediatric whole-body phantom. Position of the dosimeter for measuring (**A**) the right lens, right lobe of thyroid, (**B**) left lens, left lobe of thyroid, (**C**) right parotid, right sublingual, and (**D**) left parotid, and left sublingual. Furthermore, the entrance surface dose was converted to the dose by each organ by multiplying the mass energy absorption coefficient ratio for each organ to air from Publication 44 of the International Commission on Radiation Units and Measurements (ICRU) (lens: 0.94; thyroid: 1.15). For other organs (i.e., the parotid and sublingual gland), no specific coefficients are available in ICRU Publication 44 [66]. Considering that the correction factors for most soft tissue organs were close to 1, no correction was applied to these two organs. However, in this study, the dosimeters were attached to the surface of the phantom, so the organ-absorbed dose could not be calculated accurately. Therefore, the entrance surface dose was approximated to the organ-absorbed dose by multiplying the entrance surface dose by the coefficient ratio.
## 7 Irrigation Suitability of Groundwater
of Samples (\%)**} \\ \cline{2-4} \multicolumn{1}{c}{} & **Jan-20** & **Jan-21** \\ \hline \multirow{4}{*}{EC [57]} & \(<\)250 (Excellent) & 0 (0\%) & 0 (0\%) \\  & 250–750 (Good) & 14 (35\%) & 0 (0\%) \\  & 750–2000 (Permissible) & 15 (37.5\%) & 17 (42.5\%) \\  & 2000–3000 (Doubtful) & 1 (2.5\%) & 9 (22.5\%) \\  & \(>\)3000 (Unsuitable) & 10 (25\%) & 14 (35\%) \\  & \(<\)1000 (Excellent) & 24 (60\%) & 10 (25\%) \\  & 1000–3000 (Suitable) & 11 (27.5\%) & 26 (65\%) \\  & \(>\)3000 (Unsuitable) & 5 (12.5\%) & 4 (10\%) \\  & \(<\)75 (Soft) & 3 (7.5\%) & 0 (0\%) \\  & 75–150 (Moderate) & 12 (30\%) & 0 (0\%) \\  & 150–300 (Hard) & 10 (25\%) & 3 (7.5\%) \\  & \(>\)300 (Very hard) & 15 (37.5\%) & 37 (92.5\%) \\  & \(<\)20 (Excellent) & 0 (0\%) & 13 (32.5\%) \\ Na\% & 20–40 (Good) & 3 (7.5\%) & 10 (25\%) \\  & 40–60 (Permissible) & 24 (60\%) & 13 (32.5\%) \\  & 60–80 (Doubtful) & 13 (32.5\%) & 4 (10\%) \\  & \(>\)80 (Unsuitable) & 0 (0\%) & 0 (0\%) \\  & \(<\)10 (Excellent) & 37 (92.5\%) & 39 (97.5\%) \\  & 10–18 (Good) & 3 (7.5\%) & 1 (2.5\%) \\  & 18–26 (Doubtful) & 0 (0\%) & 0 (0\%) \\  & \(>\)26 (Unsuitable) & 0 (0\%) & 0 (0\%) \\ PI & \(=\frac{\left(\text{Na}^{+}+\text{\text{\text{\text{\text{\text{\text{\text{\text{ \text{\text{\text{\  
2020, 67.5% were suitable, and 32.5% were unsuitable in January 2021. The carbonate ion ratio with the sum of calcium-magnesium ions indicates the residual sodium carbonate for irrigation water quality [63]. The RSC values are reported as safe \(<\) 1.25, moderate 1.25-2.5, and unsuitable \(>\)2.5. The groundwater was considered safe for irrigation practices based on the RSC values in both time periods. [PERSON] proposed a method for evaluating the quality of irrigation water in 1963 [64]. According to his method, the ratio of Na\({}^{+}\) to all other major cations (Ca\({}^{2+}\) + Mg\({}^{2+}\)) below 1 is recommended, and above 1 is not recommended for irrigation. In this study, the Kelly ratio in January 2021 was calculated as 30% suitable and 70% unsuitable, with 62.5% suitable for irrigation and 37.5% unsuitable. It is suggested that, in the regions where groundwater is affected by salinity, suitable crops that can withstand the high salinity of water should be grown.
## 3 Results and Discussion
### Mechanisms of the Enhanced Performance of the Wolframite-Cathode MFC  
The primary mechanisms for the enhanced performance of wolframite photocathode MFC may be depicted as follows (Figure 7). At the anode, with the help of microbial metabolic activity for electricigens, an electrochemically active biofilm formed on the carbon felt, which oxidized glucose and produced electrons/protons. Subsequently, the protons permeated to the cathode chamber after passing the PEM. The bioelectrons were transferred from the anode chamber and were delivered to the cathode via the external circuit. Notably, wolframite is a p-type semiconducting mineral, and its semiconducting properties can be activated under light irradiation. Moreover, catalysts lower the overpotential of the cathode. On the surface of photocathode, the presence of abundant holes attracts electrons. These photoholes featured more positive potentials and were more easily combined with bioelectrons from the bioanode. This was consistent with the observation from cathode potential curves in Figure 6. More positive potentials forced the microorganisms to \"work harder\" and rapidly  
Figure 6: Anode potential curves (**a**) and cathode potential curves (**b**) with natural wolframite-coated graphite cathodes and bare graphite cathodes under dark/light conditions.  
neutralize electrons that were transferred from the anode. Therefore, enhanced performance of the MFC was realized with the natural wolframite-coated photocathode.  
This novel wolframite photocathode MFC provided a cost-effective, environment-friendly, and energy clean cathode with long-term stability, and has broad prospects in various processes, including wastewater treatment and bioelectricity generation.
# Investigation on Ship Hydroelastic Vibrational Responses in Waves
## 4 Analysis of Numerical and Experimental Results
### Modal Analysis
#### 4.3.2. Motions in Regular Waves
In comparison, the high frequency slamming \(M_{BV}\)s are caused by transient local wave impact, meaning that during the very short duration of whipping, the change of the wetted hull surface is negligible, and therefore, the above-mentioned effects play little role in the asymmetric characteristics of hogging and sagging \(M_{BV}\)s of the high frequency \(M_{BV}\)s. The omission of some strong nonlinear effects in extreme waves from the present numerical method, such as wave breaking, large eddies, large ship motions, etc., results in the evident discrepancy between the numerical and experimental results in large to extreme waves. Wave breaking and large eddies generated by water viscosity in experiments will dissipate the energy of the ship model system, leading to mitigated ship model responses. The tank side wall effect may also slightly contribute to the difference between the numerical predictions and the experimental results, since this effect is not considered in the present numerical model. On the other hand, large ship motions in extreme waves indicate that not only the [\(\mathbf{F}_{I}\)] and [\(\mathbf{F}_{H}\)], but [\(\mathbf{F}_{D}\)] and [\(\mathbf{F}_{R}\)] should be calculated based on the 3D instantaneous wetted ship hull, which is currently still a challenge in potential flow approach. The present numerical methodology can reasonably predict the hydroelastic responses in large to extreme waves, even though some discrepancies can be found. Compared with the low frequency \(M_{BV}\)s, the hogging and sagging values of the total \(M_{BV}\)s and high frequency slamming \(M_{BV}\)s change more rapidly as the wave height increases. This is because the transient high frequency impact \(M_{BV}\)s are more sensitive to the local change of the wave impact pressure resulted from the increase of incident wave height. Since the high frequency \(M_{BV}\)s constitute an important part of the total \(M_{BV}\)s, the total \(M_{BV}\)s accordingly change rapidly with the increase of wave heights.
## 5 Experimental Results
### Correlation Discovery: Dynamic Time Warping
#### 5.5.1 Forecasting Methods and Models  
Time series forecasting performance is usually evaluated upon training some model over a given period of time and then asking the model to forecast the future values for some given horizon. Provided that someone already knows the real values of the time series for the given horizon, it is straightforward to check the accuracy of the prediction by comparing them with the forecasting values.  
Denoting a time series of interest as \(y_{t}\) with \(N\) points and a forecast of it as \(f_{t}\), the resulting forecast error is given as \(et=yt-ft\), \(for\ t=1,\ldots,N\). Using this notation, the most common set of forecast evaluation statistics considered can be presented as below (Table 2).  
Intuitively, RMSE and MAE focus on the forecasting accuracy; RMSE assigns a greater penalty on large forecast errors than the MAE, while the \(U2\) statistic focuses on the quality, which will take the value of one under the naive forecasting method. Values less than one indicate greater forecasting accuracy than the naive forecasting method, and values greater than one indicate the opposite.  
According to the literature, the most frequently used methods for time series forecasting include Autoregressive Integrated Moving Average (ARIMA) [20; 21], Linear Regression (LR) [22], the Generalized Linear Model (GLM) [22], Support Vector Machines (SVM) [23] and Artificial Neural Networks (ANN) [24].
## 5 Case Studies
### Case One  
5.1.1. Calculation of Coal Mine Goaf Site Stability  
The catastrophe series method was used to calculate the stability of coal mine goat site, organizing the coal mine goat parameters and normalizing them, and the results are shown in Table 10.  
Through the normalization formula of the corresponding catastrophe model, the catastrophe progression was calculated, and the values of \(x_{\text{B1}}\), \(x_{\text{B2}}\), \(x_{\text{B3}}\) and \(x_{\text{B4}}\) in the coal mine goat were 0.856, 0.778, 0 and 0.925, which met the threshold requirements. Therefore, the catastrophe progression \(A\) is selected as the \"average principle\", so the value of \(A\) is 0.640, which belongs to the \"unstable\" coal mine goat site.
# Distributed Fuzzy Cognitive Maps for Feature Selection in Big Data Classification
###### Abstract  
The features of a dataset play an important role in the construction of a machine learning model. Because big datasets often have a large number of features, they may contain features that are less relevant to the machine learning task, which makes the process more time-consuming and complex. In order to facilitate learning, it is always recommended to remove the less significant features. The process of eliminating the irrelevant features and finding an optimal feature set involves comprehensively searching the dataset and considering every subset in the data. In this research, we present a distributed fuzzy cognitive map based learning-based wrapper method for feature selection that is able to extract those features from a dataset that play the most significant role in decision making. Fuzzy cognitive maps (FCMs) represent a hybrid computing technique combining elements of both fuzzy logic and cognitive maps. Using Spark's resilient distributed datasets (RDDs), the proposed model can work effectively in a distributed manner for quick, in-memory processing along with effective iterative computations. According to the experimental results, when the proposed model is applied to a classification task, the features selected by the model help to expedite the classification process. The selection of relevant features using the proposed algorithm is on par with existing feature selection algorithms. In conjunction with a random forest classifier, the proposed model produced an average accuracy above 90%, as opposed to 85.6% accuracy when no feature selection strategy was adopted.  
big data; feature selection; distributed fuzzy cognitive maps; genetic algorithms Article
## 4 Results and Discussion
### Technical Evaluation
Other authors have reported techniques to reduce the impact of ambient light, signal saturation, and movement artefacts in pulsatile signals [29; 30], which could be added to the current ZenTBI system to reduce common noise sources. For instance, an ambient light cancellation circuit would avoid signal saturation due to high levels of ambient light or small gaps between the sensor and the measuring surface. Moreover, the separation of the AC and DC components using analogue filters could allow higher amplification of the pulsatile signals to improve the quality of these signals, as well as the implementation of an offset cancelling circuit to reduce the movement artefacts in the pulsatile signals. Figure 9: Light intensity DC from all three photodetectors at 810 nm.
## 1 Introduction
DSSD [17] (Deconvolutional Single Shot Detector) improves the detection of multi-scale objects by introducing deconvolution layers, which enhance detection performance on low-resolution targets. However, DSSD still faces performance instability in adverse weather conditions, especially in sandstorm environments, where image degradation is significant. Its ability to describe blurred features remains limited, leading to suboptimal detection results. Currently, there are many reliable results on license plate detection in harsh environments, such as the techniques studied by [PERSON], [PERSON] et al. [18], which were tested based on descriptors and classifiers with different textures and are highly adaptable to machine learning-based algorithms, and [PERSON], [PERSON] et al. [19], who worked on a new ALPD method that can efficiently detect license plate regions from complex environments. Therefore, in this study, we will focus on the work of license plate number recognition and classification after the completion of license plate detection segmentation, and the optimization of license plate detection techniques under sandstorm conditions will be studied in detail in future work. At the same time, taking into account the different advantages and disadvantages of the networks described above, in this research, we adopt the AlexNetBN network to address the performance bottlenecks of existing methods in sandstorm weather. AlexNetBN, an enhanced version of the classic AlexNet [20], introduces Batch Normalization (BN) layers to improve model training stability and generalization [21], particularly when handling degraded images. Compared to other methods, AlexNetBN excels at feature extraction in challenging conditions, such as blurred and low-contrast images. By optimizing the network structure to focus on key feature regions that are less affected by blur and contrast reduction, our approach improves target recognition accuracy and robustness in sandstorm environments. The main contributions of this paper are as follows: (1) We propose a robust target recognition algorithm tailored for sandstorm conditions, optimizing detection accuracy and resilience through the AlexNetBN network. (2) We develop and release a custom dataset of license plates affected by sandstorms to validate the effectiveness of our method, while also comparing its performance with existing advanced algorithms. In the following sections, we will first provide a detailed discussion of the AlexNetBN architecture, explaining why it is suitable for target recognition in sandstorm weather. Next, we will describe the construction of our custom dataset and the pre-processing techniques used to simulate the real-world conditions of sandstorms. Finally, we will present our experimental results, comparing the performance of our method with alternative approaches in terms of accuracy, efficiency, and robustness. This work not only offers a new solution for target recognition under adverse weather conditions but also provides valuable insights into the application of CNN-based algorithms in scenarios where visibility is significantly compromised.