EAPCR: A Universal Feature Extractor for Scientific Data Without Explicit Feature Relation Patterns

## Abstract

Abstract Conventional methods, including Decision Tree (DT)-based methods, have been effective in scientific tasks, such as non-image medical diagnostics, system anomaly detection, and inorganic catalysis efficiency prediction. However, most deep-learning techniques have struggled to surpass or even match this level of success as traditional machine learning methods.
The primary reason is that these applications involve multi-source, heterogeneous data, where features lack explicit relationships. This contrasts with image data, where pixels exhibit spatial relationships; textual data, where words have sequential dependencies; and graph data, where nodes are connected through established associations. The absence of explicit F eature R elation P atterns (FRPs) presents a significant challenge for deep learning techniques in scientific applications that are not image, text, and graph-based.
In this paper, we introduce EAPCR , a universal feature extractor designed for data without explicit FRPs. Tested across various scientific tasks, EAPCR consistently outperforms traditional methods and bridges the gap where deep learning models fall short. To further demonstrate its robustness, we synthesize a dataset without explicit FRPs. While Kolmogorov–Arnold Network (KAN) and feature extractors like Convolutional Neural Networks (CNNs), Graph Convolutional Networks (GCNs), and Transformers struggle, EAPCR excels, demonstrating its robustness and superior performance in scientific tasks without FRPs.

## 1 Introduction

In various scientific applications, such as non-image medical diagnostics, system anomaly detection, inorganic catalysis efficiency prediction, and etc., traditional machine learning techniques, such as \acfdt (Ali et al., ) and \acdt-based method (e.g., \acrf and \acxgboost), have been reported as highly effective (Coşkun & Kuncan, ; Mutlu et al., ; Alizargar et al., ; Schossler et al., ; Mallioris et al., ). In contrast, few studies have reported deep learning models as the best-performing methods, indicating that more complex deep learning techniques, such as \accnns (LeCun et al., ), \acgcns (Kipf & Welling, ; Bronstein et al., ), and Transformers  (Vaswani, ), have not shown the same level of success in those scientific applications.

The primary reason deep learning techniques often underperform in scientific applications is that the data in these fields differ significantly from traditional tasks like images, text, and graphs. For example, in non-image medical diagnostics, patient data come from diverse sources, such as physical measurements (e.g., weight, blood pressure) and chemical tests (e.g., glucose levels) (Fig. 1-a-1). Unlike pixels in images or words in text, which have spatial or sequential relationships (LeCun et al., ; Bronstein et al., ) (Fig. 1-b-1), or nodes in graphs with known connections (Kipf & Welling, ; Bronstein et al., ) (Fig. 1-b-2), features in scientific data lack such explicit relationships. This absence of explicit feature relationships is common across various scientific tasks, such as system anomaly detection (Wang et al., ; Tian, ) (Fig. 1-a-2) and inorganic catalysis efficiency prediction (Sun et al., ) (Fig. 1-a-3), where features are collected from heterogeneous sources, such as electrical signals and temperature, and have different units, like pH levels and illumination time.

Figure: Figure 1: Motivation for designing feature extractors for data without FRPs.
Refer to caption: extracted/5995640/figure1.png

The feature relationship patterns, including spatial relationships and connections, are referred to as \acfFRPs in this work. \acFRPs are critical for deep learning-based feature extractors as they contain essential information about feature associations or correlations. For instance, in image and text data, spatial relationships between pixels and words reveal correlations where nearby features have stronger associations and distant ones weaker. This inherent relational information enables feature extractors to quickly identify important combinations of strongly interacting features. For example, \accnns in image data leverage spatial relationships to focus on local feature patterns while ignoring irrelevant non-local ones (Bronstein et al., ; Yun et al., ) (Fig. 1-b-1). Similarly, \acgcns use adjacency matrices in graph data to capture meaningful node connections (Bronstein et al., ; Schlichtkrull et al., ) (Fig. 1-b-2). However, when explicit \acFRPs, like spatial relationships or known connections, are absent, deep learning methods often struggle (Fig. 1-b-3) as assumed \acFRPs may not match the actual implicit relationships. This raises a key question:

*How can we design universal feature extraction modules for data that lack explicit \acFRPs?*

In this paper, we propose a feature extraction module, *EAPCR*, designed as a universal feature extractor for data without explicit FRPs. Traditional feature extraction modules rely on known FRPs to distinguish between important and unimportant feature combinations. However, without the guidance of FRPs, EAPCR adopts a different approach. First, it exposes all possible \acFRPs. Second, it accelerates the sampling of these combinations to ensure a wide range of feature interactions are evaluated, allowing it to effectively identify important combinations of strongly interacting features.

To evaluate EAPCR’s effectiveness, we apply it to various scientific domains, including non-image medical diagnostics (Anderies et al., ), system anomaly detection (Tian, ), and inorganic catalysis efficiency prediction (Liu et al., ). EAPCR consistently outperforms traditional methods in such tasks lacking explicit FRPs. To further assess its robustness, we synthesize a dataset without explicit FRPs, where models like \accnns (LeCun et al., ), \acgcns (Kipf & Welling, ; Bronstein et al., ), Transformers (Vaswani, ), and KAN (Liu et al., , ) struggle, while EAPCR excels in capturing meaningful features.

In summary:

- •
EAPCR is designed as a universal feature extractor for tasks lacking explicit FRPs, addressing a critical gap in deep learning for scientific applications. It consistently outperforms other models across various real-world scientific applications.
- •
We synthesize a representative dataset to investigate the challenges of modeling without FRPs, revealing the limitations of traditional methods and validating the robustness and effectiveness of EAPCR.

#### Related Works

*Feature engineering at the early stage:* Feature engineering (Bengio et al., ) plays a crucial role in improving the accuracy of classification models.
Early approaches primarily focus on addressing feature redundancies and nonlinear relationships. For instance, principal component analysis (PCA) (Abdi & Williams, ) reduces linear correlations, while nonlinear methods like nonlinear PCA (Linting et al., ) and autoencoders (Bank et al., ) handle redundancies through nonlinear transformations. Although classifiers such as \acsvms (Hearst et al., ), \acmlps (Rumelhart et al., ), and more recent models like \ackan (Liu et al., , ) can manage complex nonlinear feature relationships, their performance heavily depends on how input data is represented. For example, using large pre-trained models to encode images improves classification and retrieval accuracy by emphasizing critical features like edges and shapes (Liu et al., ; Holliday & Dudek, ; Zhou et al., ). Therefore, more advanced feature extraction techniques are required to go beyond capturing nonlinear relationships, further refining feature representations for better performance.

*Feature engineering for data with FRPs:*
Accurately capturing implicit correlations between features is essential for effective classification. For example, determining obesity cannot solely rely on weight; height must also be considered to provide a more accurate assessment. In more complex scenarios, classification depends on interactions between features, where their joint contribution exceeds the sum of their individual effects (Koh & Liang, ; Ali et al., ; Beraha et al., ; Deng et al., ). This is why traditional classifiers, like \acmlps (Rumelhart et al., ), rely on advanced feature extractors to improve performance by identifying complex feature interactions. In this vein, recent advancements, such as ConvNeXt (Woo et al., ), \acbert (Devlin, ), \acgpt (Radford et al., ), \acvit (Dosovitskiy, ), and \actft (Lim et al., ), efficiently capture interaction patterns of features in structured or Euclidean data like images and texts. For non-Euclidean data, techniques like manifold learning (McInnes et al., ; Tenenbaum et al., ) and \acgnns, including \acGraphSAGE (Hamilton et al., ) and \acDGCNN (Wang et al., ), address unique challenges. Regardless of the data type, these methods rely on explicit \acFRPs (e.g., spatial, sequential, or relational connections) of data, which contain the implicit feature correlations essential for effective feature extraction.

*Feature engineering for data without FRPs:* As discussed earlier, many scientific tasks lack explicit \acFRPs. Machine learning techniques like \acfdt and \acdt-based methods perform well in these tasks by handling various data types (numerical and categorical) and automatically capturing interaction effects between features, as each decision split evaluates the relationship between a feature and the target variable (Gregorutti et al., ). In contrast, deep learning models, such as \accnns, \acgcns, and Transformers, struggle due to their reliance on predefined \acFRPs. For example, in heart failure and maternal health risk prediction, the best-performing models are \acrf (Coşkun & Kuncan, ) and \acdt (Mutlu et al., ). Similarly, in hepatitis C (Alizargar et al., ), TiO2 photocatalytic degradation (Schossler et al., ), and centrifugal pump health status prediction (Mallioris et al., ), \acxgboost, another \acdt-based method, consistently outperforms deep learning methods. Despite advancements in multimodal techniques (Lahat et al., ) and methods for non-Euclidean data (Bronstein et al., ), feature heterogeneity does not always align with distinct modalities. Even features from the same source may lack explicit \acFRPs, rendering multimodal approaches ineffective. Moreover, inconsistent feature dimensions complicate the definition of feature distances in Euclidean space. For instance, determining how a 1 kg increase in weight correlates with a change in height for a particular disease is non-trivial. Data without explicit \acFRPs differ from traditional Euclidean and non-Euclidean data (Bronstein et al., ), making deep learning techniques less effective in these applications.

## 2 EAPCR: a feature extractor without the need of explicit feature relation patterns

In many applications, identifying feature combinations with strong interactions is not difficult, as feature extraction modules like CNNs and GCNs use predefined \acFRPs to narrow the range of possible combinations. For example, CNNs leverage spatial relationships between pixels to efficiently sample local regions and filter out weak feature interactions, quickly identifying critical patterns like textures in image recognition. However, when explicit \acFRPs are absent, searching for important feature interactions becomes more random and inefficient. Traditional methods often fail in these cases because the \acFRPs chosen by feature extraction modules may not align with the implicit patterns in the data. Additionally, the sample complexity increases exponentially with the number of features, making exhaustive search impractical. Unlike these traditional approaches, EAPCR exposes all feature combinations to ensure no fundamental interaction patterns of features are missed (Fig. 2-b), then optimizes the efficiency of combination sampling to address this challenge (Fig. 2-c).

Further discussion about why FRPs is important gives in App. F, where the relationship between feature interaction, feature correlation, and FRPs is discussed.

### 2.1 Expose possible feature relations: Embedding and bilinear Attention

Unlike existing works that use one-hot encoding for categorical feature representation (Seger, ) and bilinear attention to focus on interactions between two input modalities (Fukui et al., ), we leverage Embedding (Mikolov, ) and bilinear Attention (Kim et al., ) to construct a correlation matrix.

For an input with $N$ features, we first convert each feature into a categorical (string-based) one. Categorical features, like gender or catalyst substrate, remain unchanged, while numerical features are discretized into categories (e.g., “high”, “medium”, “low”) based on context-specific thresholds. These thresholds are chosen to balance between overly fine granularity, which leads to sparse categories, and overly coarse granularity, which reduces feature separation. For example, temperature might be categorized into “very high”, “high”, “medium”, “low”, and “very low”. This process generates an input matrix of shape $[N,1]$, where each element is an integer index, assigned via a dictionary mapping categorical values to indices.

Figure: Figure 2: The illustration of the method. (a) an overview of EAPCR. (b) an illustration of how Embedding and bilinear Attention can expose all possible \acFRPs. (c) an illustration of how the permuted \accnns considers combinations of originally close matrix elements as well as combinations of originally distant elements.
Refer to caption: extracted/5995640/figure2.png

The embedding operation substitutes each component by a corresponding dense vector, giving $E$ with shape $[N,E_{s}]$, where $E_{s}$ is the embedding size. Then, we consider the bilinear attention defined as:

$$ $A=\text{Tanh}(EE^{\top}),$ (1) $$

where the matrix $A$ with shape $[N,N]$ is the constructed correlation matrix (Fig. 2-a-1). $\text{Tanh}(x)$ is Hyperbolic Tangent function. The matrix $A$ is important, because each element in the matrix represents the relationship between two features, with any combination of these elements exposing corresponding potential FRPs. Thus, by using matrix $A$, all possible relation patterns between features are being exposed in a matrix form, as shown in Fig. 2-b. The target combinations that consist of features with strong interactions are encapsulated within the matrix $A$.

Additionally, the embedding process not only captures feature relationships but also helps the model understand nuances across multi-source heterogeneous features.

### 2.2 Efficiently sample feature relations: permuted \ac cnn

To identify the target combinations that consist of features with strong interactions within the matrix $A$, we leverage \accnn to sample such combinations of elements in matrix $A$ defined in (1). Because \accnns can efficiently focus on local elements within a matrix
(Bronstein et al., ).
Instead of expanding the receptive field of \accnns by increasing kernel size or layers, we propose a Permuted CNN that efficiently samples diverse element combinations from the matrix $A$.

The designed permutation matrix $M$. The designed permutation $M$ rearranges the matrix elements of $A$, bringing originally distant elements closer and pushing originally close elements further apart, as shown in Fig. 2-c. The details of constructing the designed permutation matrix $M$ can be found in App. A. We apply a designed permutation matrix $M$ on $A$ giving a new matrix $P$, defined as:

$$ $P\triangleq MAM^{\top}.$ (2) $$

The permuted CNN. The permuted \accnn architecture is designed to capture both local and non-local relationships within the matrix elements by applying the \accnn to two different representations of the matrix $A$, as illustrated in Fig. 2-a-2. Specifically, the \accnn is applied to the raw matrix $A$ as well as to a permuted version of $A$, that is $P$ in (2). The \accnn varies on different tasks but is only equipped with a lightweight structure (e.g., a two-layered architecture with kernel sizes of $3\times 3$ and channel numbers of $8$ and $16$). Afterward, the outputs from the \accnn operations on both the raw matrix $A$ and the permuted matrix $P$ are transformed into vectors. These vectors are then concatenated, resulting in a single feature vector of size $[C,1]$ by dense connection, where $C$ is the number of classes.

Moreover, by incorporating existing feature extraction modules, such as \acmlp, as a Residual connection, we combine the strengths of different feature extractors to further enhance feature extraction efficiency. As shown in Fig. 2-a-3, the average pooling is applied on $E$, resulting in a vector $z$ with shape $[N,1]$. Then, an \acmlp transforms $z$ into a vector with shape $[C,1]$. Therefore, the residual networks also help the model better train the embedding vectors.

## 3 Main experiment results

In Sec. 3.1, we present experimental results on real-world applications, including *non-image medical diagnostics*, *inorganic catalysis efficiency prediction*, and *system anomaly detection*.
The EAPCR can also be used in a wider range of scientific applications beyond experiments mentioned in this section; see Tab. 11, Tab. 12, Tab. 13, Tab. 14, and Tab. 15 in App. E for more discussion.
In summary, the experiment results show that EAPCR as a deep learning method outperforms traditional deep/non-deep methods, including the SOTA \acdt-based methods,
demonstrating the effectiveness of the proposed EAPCR method for tasks involving data without explicit FRPs.

In Sec. 3.2, to investigate the challenges of modeling without FRPs, we construct an illustrative dataset lacking explicit FRPs. The results reveal the limitations of traditional deep learning methods, such as CNN (LeCun et al., ), GCN (Kipf & Welling, ; Bronstein et al., ), Transformer (Vaswani, ), MLP (Rumelhart et al., ), and KAN (Liu et al., ), which are often underutilized in scientific tasks. We then validate the robustness and effectiveness of EAPCR. This dataset highlights the shortcomings of traditional models in tasks that lack explicit FRPs while demonstrating EAPCR’s superior performance in such scenarios.

### 3.1 Benchmarking comparison of EAPCR on Scientific tasks

#### Non-image medical diagnostics:

In this study, we validated the proposed EAPCR method using the UCI Cleveland heart disease dataset (Janosi et al., ).
The dataset consists of 13 feature attributes (including Sex, Age, and CP [Chest Pain] Type) and one categorical attribute, comprising eight categorical (string-based) and five numerical features (details in App. E). Following the data processing approach outlined in Sec. 2.1, all features were converted to categorical form, and a dictionary was created to map each categorical feature to an integer index. These indices were then mapped to 128 dimensions using an embedding mechanism and fed into the model for training. We benchmarked the EAPCR method against various machine learning algorithms, including \acsvm, \acNB, logistic regression, \acdt, and \acknn, as presented in Tab. 1, based on the study by Anderies et al. (Anderies et al., ). The results show that EAPCR outperforms conventional machine learning techniques.

**Table 1: Comparison of our method with others in the diagnosis of non-image medical data.**
| Method | Accuracy | Precision | Recall | F1 Score |
| --- | --- | --- | --- | --- |
| \acdt | 70% | 86% | 63% | 72% |
| \acknn | 78% | 90% | 74% | 81% |
| Logistic Regression | 83% | 94% | 79% | 86% |
| \acNB | 83% | 96% | 76% | 85% |
| \acsvm | 85% | 97% | 79% | 87% |
| EAPCR | 93% | 97% | 92% | 94% |

#### Inorganic catalysis efficiency prediction:

In this study, we used the TiO2 photocatalysts dataset from Liu et al. () to evaluate the performance of our EAPCR model in inorganic catalysis applications. The dataset contains nine attributes, including dopant, molar ratio, calcination temperature, and pollutant, with a total of 760 samples (details in App. E). Using a 256-dimensional embedding, we benchmarked EAPCR against LightGBM, the best-performing model in Liu et al. (). As shown in Tab. 2, EAPCR outperforms LightGBM in terms of the $R^{2}$ metric.

**Table 2: Comparison of our method with others in the Inorganic catalysis data set.**
| Model | MAE | MSE | RMSE | $R^{2}$ |
| --- | --- | --- | --- | --- |
| Linear Regression | $0.513\pm 0.104$ | $0.601\pm 0.237$ | $0.762\pm 0.401$ | $0.048\pm 0.014$ |
| \acrf | $0.235\pm 0.062$ | $0.180\pm 0.126$ | $0.417\pm 0.148$ | $0.805\pm 0.035$ |
| \acxgboost | $0.145\pm 0.103$ | $0.086\pm 0.034$ | $0.293\pm 0.136$ | $0.884\pm 0.024$ |
| LightGBM (Liu et al., ) | / | / | / | 0.928 |
| EAPCR | $\textbf{0.128}\pm\textbf{0.003}$ | $\textbf{0.041}\pm\textbf{0.001}$ | $\textbf{0.203}\pm\textbf{0.004}$ | $\textbf{0.937}\pm\textbf{0.003}$ |

#### System anomaly detection:

In our study on system anomaly detection, we used sensor data from the Kaggle dataset (https://www.kaggle.com/datasets/umerrtx/machine-failure-prediction-using-sensor-data), which is designed for predicting machine failures in advance.
The dataset consists of 944 samples with nine feature attributes, including sensor readings like footfall and tempMode, and a binary target attribute (1 for failure, 0 for no failure) (details in App. E). Using a 64-dimensional embedding, we benchmarked our model against \acrf, Logistic Regression, \acsvm, and Gradient Boosting. As shown in Tab. 3, our model outperformed the others across all metrics.

**Table 3: Comparison of our method with others in the Sensor data-based system anomaly detection tasks**
| Algorithm | Accuracy | Precision | Recall | F1-score |
| --- | --- | --- | --- | --- |
| \acrf | 87.83% | / | / | / |
| Logistic Regression | 87.83% | / | / | / |
| \acsvm | 87.83% | / | / | / |
| Gradient Boosting | 88.89% | 87.50% | 88.51% | 88.00% |
| EAPCR | 89.42% | 87.64% | 89.66% | 88.64% |

### 3.2 Benchmarking comparison of EAPCR on synthesized data

#### Synthesize data without explicit FRPs.

In image recognition, the correlation between pixels and their spatial positions is typically consistent, with nearby pixels having higher correlations and distant ones having lower correlations (see App. B). Previous research (Yun et al., ) shows that random transformations can disrupt this spatial relationship, such as by shuffling pixel positions. However, our approach differs by using a carefully designed permutation, as outlined in Sec. 2.2, which strategically moves adjacent matrix elements further apart and brings distant elements closer (Fig. 3-a). As an example, we used the handwritten digit recognition MNIST dataset (LeCun et al., ) to generate data without explicit FRPs, see Figs. 3-b and  3-c.

This designed permutation more effectively disrupts spatial relationships while visually maintaining them, thereby concealing underlying \acFRPs more thoroughly compared to random permutations (Yun et al., ). The proof can be found in Fig. 3-b, where CNN performance degrades progressively, demonstrating that the designed permutation more effectively breaks spatial correlations. This is why we applied the designed permutation when synthesizing the dataset.

Although both the permuted CNN in EAPCR and the synthesized data use a permutation matrix that disrupts the original spatial relationships, they are not identical. For instance, the synthesized data matrix is $[28,28]$, while the model’s matrix is $[784,784]$, ensuring the model has no prior knowledge of hidden \acFRPs and preventing leakage. Further evidence comes in Fig. 3-b, where EAPCR performs consistently across raw data, randomly permuted data, and data with designed permutation.

We conducted extensive experiments on both the original data and the synthesized data, using existing machine learning models, mainstream deep learning models, and our proposed EAPCR model. For convenience, we refer to these two datasets as the “raw image (data with predefined FRPs)” and “synthesized data (data without explicit FRPs),” since the format of the synthesized data is no longer important.

#### Performance of different models on data with FRPs and synthesized data without explicit FRPs.

We summarize the results of this section in Fig. 3-c, where the horizontal axis represents the parameters of different models and the vertical axis shows the accuracy of these models on Data With/Without explicit FRPs. Fig. 3-c is essential to understanding the limitations of traditional methods and the robustness and effectiveness of EAPCR.

*\acmlp as the baseline*.
The \acmlp is a basic neural network architecture that uses fully connected layers to process input data. As shown in Fig. 3-c-1, \acmlp performs similarly on tasks with and without explicit FRPs. Thus, we use \acmlp’s performance as a baseline to assess the feature extraction effectiveness of other models. Models that exceed \acmlp’s performance demonstrate effective feature extraction capabilities, while those with similar or lower performance indicate insufficient feature extraction.

*EAPCR demonstrated superior performance*. Interestingly, on raw data, both the CNN (CNN+MLP) and the ensemble model CNN+KAN exhibited strong performance, as shown in Fig. 3-c-3 and Fig. 3-c-8, respectively. However, when evaluated on synthesized data, only EAPCR and EACR (the ablation model of EAPCR where permuted CNN is eliminated) obviously outperforms the MLP, with results depicted in Fig. 3-c-7 for EAPCR and Fig. 3-c-6 for EACR. Notably, EAPCR achieved $94.5\%$ accuracy with 37,355 parameters, outperforming its ablation model EACR, which achieved $94.3\%$ accuracy with 59,361 parameters. Other models, such as the \acrf, \accnn, and \acgcn (uses the data’s correlation matrix as its adjacency matrix), only slightly surpassed the MLP. Their performances are shown in Fig. 3-c-2 for CNN, Fig. 3-c-4 for GCN, and Fig. 3-c-5 for RF. It is important to note that the GCN’s performance declined significantly when provided with a randomly generated adjacency matrix instead of the correlation matrix (Fig. 3-c-4). Transformer (Vaswani, ) (Fig. 3-c-11) and KAN (Liu et al., ) (Figs. 3-c-10 and  3-c-9) do not exhibit obvious impact on synthesized data without FRPs.
More details of the models’ parameter settings are given in the Tabs. 5 and 6 and in App. D.

*EAPCR recovers hidden FRPs.*
The reason EAPCR demonstrates supreme performance compared to other methods is that it successfully reconstructs the implicit FRPs in the synthesized dataset. We verify this by comparing the correlation matrix recovered by EAPCR with the original correlation matrix. The details about the result are shown in App. G.

Figure: Figure 3: Results on synthesized data without explicit \acFRPs. (a) Illustration of synthesized data: In the raw image, pixel correlations align with spatial positions (a-1), but in the synthesized data, the spatial correlations breaks (a-2). (b) Experimental results confirm that EAPCR’s strong performance is not due to the use of a similarly designed permutation when synthesizing the data. (c) Comparison of various methods on data with \acFRPs and synthesized data without explicit \acFRPs.
Refer to caption: extracted/5995640/figure3.png

### 3.3 Robustness and ablation study

To further validate the robustness of EAPCR, we synthesized additional datasets beyond the MNIST dataset to Flower (Nilsback & Zisserman, ), ImageNet (Deng et al., ), and CIFAR-10 (Krizhevsky & Hinton, ) datasets. We also conducted experiments comparing EAPCR to EACR, an ablation model without permutation but with larger convolutional kernels and more layers, to demonstrate that EAPCR’s permuted CNN is more effective than simply increasing the capacity of standard CNNs. In addition, experiments in App. C show that the permutation we designed in the permuted CNN outperforms a random one. The results (Fig. 4) show: 1) traditional \accnn-based algorithms, such as ConvNeXt-V2 (Woo et al., ), perform well on data with \acFRPs but fail on data without explicit \acFRPs; 2) EAPCR consistently performs well across all three synthesized datasets without explicit \acFRPs; and 3) EAPCR outperforms EACR, even though EACR uses larger kernels and more layers.

Figure: Figure 4: The comparison and ablation experiments conducted on the more synthesized data. Asterisks indicate that only subsets of gray data were used for experimental efficiency. This involves employing limited random sampling to create combined training and testing datasets. For example, from ImageNet, 10 categories, each comprising 400 randomly selected images, were selected for training and 50 for testing, optimizing both time and computational resources.
Refer to caption: extracted/5995640/figure4.png

## 4 Conclusion

The absence of explicit Feature Relation Patterns (FRPs) presents a significant challenge in many scientific tasks that are often overlooked by the ML community. This limitation contributes to the underperformance of deep learning methods compared to traditional methods, such as Decision Tree-based (DT-based) machine learning approaches, in scientific applications.
To address this issue, we introduce an innovative method, EAPCR, specifically designed for data lacking FRPs. We evaluate the effectiveness and efficiency of EAPCR across a variety of real-world scientific tasks and demonstrate that it consistently outperforms established methods on several scientific datasets.
Additionally, we synthesize a dataset that deliberately excludes explicit FRPs to further assess the performance of EAPCR. The results demonstrate that EAPCR outperforms CNN, GCN, MLP, RF, Transformer, and KAN on the dataset without explicit FRPs. Our findings underscore the potential of EAPCR as a robust solution for scientific tasks lacking explicit FRPs, bridging the gap where
deep learning models fall short and paving the way for enhanced data analysis in this domain.

## Appendix A Generating Permutation Matrix M

Constructing the designed permutation matrix in Sec. 2.2 is a key aspect of this paper. Although the method is relatively simple, we describe the process in detail in the appendix. The core idea is to create a reversible operation that rearranges the positions of $N$ elements ($1,2,3,\cdots,N$), ensuring that originally adjacent elements are no longer adjacent while non-adjacent elements become adjacent. This generates the reversible permutation matrix $M$.

The process works as follows: First, arrange the $N$ elements in order into an $R\times L$ matrix, where $N=R\times L$ with $R$ and $L$ being roughly equal in size. Transpose of this matrix and then reshape it into an $N\times 1$ vector. This new sequence represents the transformed positions of the original data. Next, create an all-zero matrix $M$ of size $N\times N$. Using the transformed positions, place $1$s in the corresponding row and column positions of $M$, resulting in the reversible permutation matrix.

For example, with $N=9$, arrange the numbers 1 to 9 into a $3\times 3$ matrix $[[1,2,3],[4,5,6],[7,8,9]]$. Transposing the matrix and reshaping it gives the new sequence $[1,4,7,2,5,8,3,6,9]$. Using this sequence, place $1$s in the appropriate positions in a $9\times 9$ matrix $M$. The result is a permutation matrix where the distance between two adjacent elements in the new sequence is at least 3, meaning originally adjacent elements are no longer adjacent, and previously distant elements are now adjacent. Further details can be found in the code.

## Appendix B Analysis of Pixel Distance vs. Pixel relations

To investigate the relationship between pixel distance and correlation in images, we randomly selected 200 images from 10 subfolders of the MNIST handwriting dataset. We calculated the Pearson correlation coefficient and mutual information for different pixel pairs based on their distance. Using the reference point $(5,5)$, we compared its relationship with all other pixels in each image. The results were sorted by distance, and for each unique distance, we retained only the highest correlation coefficient. We then plotted the variation of the Pearson correlation coefficient and mutual information with pixel distance at distances of $1$, $\sqrt{2}$, $2$, etc., as shown in Fig. 5.

Figure: Figure 5: Analysis of inter-pixel distance and statistical correlations. (a) Relationship between the Pearson correlation coefficient and inter-pixel distance. (b) Relationship between mutual information and inter-pixel distance.
Refer to caption: extracted/5995640/distance.png

## Appendix C The permuted CNN with designed permutation outperforms that with random permutation

To demonstrate that the designed permutation used in our permuted CNN (see Sec. 2.2) outperforms random permutation, we conducted comparative experiments on synthesized data based on the MNIST dataset.
The experimental results are presented in Tab. 4. The results show that the designed permutation performs better, as it maximally separates nearby matrix elements while bringing distant ones closer, enabling the CNN to effectively sample both nearby and distant matrix elements.

**Table 4: Experiment of permuted CNN with designed permutation versus that with random permutation.**
| Method | Parameters | Accuracies |
| --- | --- | --- |
| EAPCR with designed permuted CNN | 37355 | 94.5% |
| EAPCR with random permuted CNN | 37355 | 93.2% |

## Appendix D Experiment details: synthesized dataset without explicit FRPs

An illustration of the EAPCR for the synthesized data of the handwritten digital MNIST dataset is given in Fig. 6.
Tab. 5 shows the details of the structure and parameters of our EACPR and ablation model EACR.
Tab. 6 shows the details of the structures and parameters of other models used in the experiment of synthesized data.

Figure: Figure 6: An illustration of the EAPCR, showing the detailed structure and parameter setting for the synthesized data without FRPs.
Refer to caption: extracted/5995640/mnist_eapcr.png

**Table 5: The details of EAPCR and its ablation model EACR on synthesized data without FRPs.**
| Model | Our ablation model EACR | Our model EAPCR |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Parameters | 29151 | 59361 | 79236 | 37355 | 67565 | 87440 |
| Train | 30000 | 30000 | 30000 | 30000 | 30000 | 30000 |
| Test | 5000 | 5000 | 5000 | 5000 | 5000 | 5000 |
| Batch size | 64 | 64 | 64 | 64 | 64 | 64 |
| Epoch | 100 | 100 | 100 | 100 | 100 | 100 |
| Learning rate | 0.003 | 0.003 | 0.003 | 0.003 | 0.003 | 0.003 |
| Dropout | 0.5 | 0.5 | 0.5 | 0.5 | 0.5 | 0.5 |
| Embedding size | 128 | 128 | 128 | 128 | 128 | 128 |
| \accnn Layer1<br>kernel size<br>Channels<br>Max pooling kernel size | Conv1(4x4, 4, 2x2 )<br>Conv2(4x4, 8, 2x2 )<br>Conv3(4x4, 16, 2x2 )<br>Conv4(4x4, 16, 2x2 ) | Conv1(4x4, 4, 2x2 )<br>Conv2(4x4, 8, 2x2 )<br>Conv3(4x4, 16, 2x2 )<br>Conv4(4x4, 16, 2x2 ) | Conv1(4x4, 4, 2x2 )<br>Conv2(4x4, 8, 2x2 )<br>Conv3(4x4, 16, 2x2 )<br>Conv4(4x4, 16, 2x2 ) | Conv1(4x4, 4, 2x2 )<br>Conv2(4x4, 8, 2x2 )<br>Conv3(4x4, 16, 2x2 )<br>Conv4(4x4, 16, 2x2 ) | Conv1(4x4, 4, 2x2 )<br>Conv2(4x4, 8, 2x2 )<br>Conv3(4x4, 16, 2x2 )<br>Conv4(4x4, 16, 2x2 ) | Conv1(4x4, 4, 2x2 )<br>Conv2(4x4, 8, 2x2 )<br>Conv3(4x4, 16, 2x2 )<br>Conv4(4x4, 16, 2x2 ) |
| \accnn Layer2<br>kernel size<br>Channels<br>Max pooling kernel size | / | / | / | Conv1(4x4, 4, 2x2 )<br>Conv2(4x4, 8, 2x2 )<br>Conv3(4x4, 16, 2x2 )<br>Conv4(4x4, 16, 2x2 ) | Conv1( 4x4, 4, 2x2 )<br>Conv2(4x4, 8, 2x2 )<br>Conv3(4x4, 16, 2x2 )<br>Conv4(4x4, 16, 2x2 ) | Conv1(4x4, 4, 2x2 )<br>Conv2(4x4, 8, 2x2 )<br>Conv3(4x4, 16, 2x2 )<br>Conv4(4x4, 16, 2x2 ) |
| Residual | 784to26,26to10 | 784to64,64to10 | 784to89,89to10 | 784to26,26to10 | 784to64,64to10 | 784to89,89to10 |
| \acmlp | 144to10 | 144to10 | 144to10 | 288to10 | 288to10 | 288to10 |
| Permutation matrix size | / | / | / | 784×784 | 784×784 | 784×784 |

**Table 6: The details of other models on synthesized data without FRPs.**
| Model | \acmlp | \accnn | \acgcn | Transformer |
| --- | --- | --- | --- | --- |
| Train | 30000 | 30000 | 30000 | 30000 |
| Test | 5000 | 5000 | 5000 | 5000 |
| Batch size | 16 | 16 | 128 | 64 |
| Epoch | 100 | 100 | 1000 | 100 |
| Learning rate | 0.001 | 0.001 | 0.003 | 0.0003 |
| Dropout | 0.5 | 0.5 | 0.5 | 0.5 |
| Embedding | / | / | / | Embedding (2,128) |
| Feature extraction | / | Conv1(5x5,8,2x2)<br>Conv2(5x5,16,2x2) | GCN1(1,64)<br>GCN2(64,128) | Transformer (128,4,1) |
| Residual | / | / | Linear1(784,128)<br>Linear2(128, 10) | Linear1(784, 64)<br>Linear2(64, 10) |
| Classification layer | Linear1(784 ,64)<br>Linear2(64,10) | Linear1(784, 64)<br>Linear2(64, 10) | Linear1(128,10) | Linear1(784,128)<br>Linear2(128,10) |

*\acgcns* primarily extract features through the relationships between nodes within a graph structure. Unlike \acmlp’s fully connected layers and \accnn’s convolutional layers, \acgcn use adjacency matrices and node feature matrices for feature extraction, leveraging the graph’s local structure to capture relationships between nodes. Here we test \acgcn only on data without \acFRPs. When \acgcn with adjacency matrices (AM) given by the correlation matrix recovered from simple statistics on the data, its performance slightly improved, achieving an accuracy of 91.2%, as shown in Fig. 3-c-4. However, the classification accuracy of \acgcn with random AM drops to only 88.2%.

*\acrf* is an algorithm in machine learning that can consider the information gain from combinations of features. It enhances prediction accuracy and stability by constructing multiple \acdt and aggregating their predictions. Unlike \acmlp, RF can naturally account for the combinations and interactions between features when processing various types of data, thereby demonstrating certain effectiveness in feature extraction. Particularly in tasks that require capturing complex data structures and relationships, RF can utilize the structure of its \acdt to assess and exploit the information gain among features. In this experiment, as shown in Fig. 3-c-5, RF generally outperforms \acmlp. However, the advantage of RF over \acmlp is not obvious, showing only a slight improvement in performance.

*Our ablation model, EACR*, which differs from EAPCR with no permuted \accnn, also achieved outstanding performance with fewer parameters than \acmlp, highlighting its advanced feature extraction capabilities. For example, experimental results demonstrate that despite using fewer parameters (for example, 29,151 parameters achieving 93.3% accuracy, 59,361 parameters achieving 94.3% accuracy, and 79,236 parameters achieving 95.2% accuracy, as shown in Fig. 3-c-6), EACR still exhibits exceptional performance in handling complex tasks, surpassing traditional models like \acmlp.

*\ackan* is a neural network based on the Kolmogorov-Arnold representation theorem (Liu et al., ). Compared with \acmlp, \ackan does not only rely on fully connected layers to process data but builds a network formed by a combination of nested functions. This structure can more effectively capture and represent the complex features of the input data. However, our experimental results show that when the number of parameters is large, the effect of \ackan does not reach the level of \acmlp. As shown in Fig. 3-c-10, when the number of parameters is about 100,000, the accuracy of \ackan is less than 90%. However, when \accnn is combined with \ackan, the feature extraction capability of \accnn can effectively extract local features and edge information from the image, and these features are used as input to \ackan. The results show that this combination significantly improves the performance of the model. As shown in Fig. 3-c-8, when the number of parameters is 93,624, the accuracy is 97.5%. However, when processing data without \acFRPs, it is difficult for \accnn to capture effective features for \ackan to use, resulting in a decline in overall model performance. As shown in Fig. 3-c-9, under the same parameters, the accuracy rate is only 89.7%.

*Transformers* (Vaswani, ), through their attention mechanisms, are capable of capturing global dependencies among all elements in the input data. Transformers calculate the mutual influences of all pairs of elements within the input sequence using self-attention layers, providing high flexibility and strong capability for information integration. However, this mechanism also leads to a significant increase in the number of model parameters. Despite the large number of parameters, Transformers did not significantly outperform \acmlp in our experiments on new types of image processing tasks. Specifically, even though a Transformer was configured with 1,252,949 parameters, its accuracy was the same as that of an \acmlp configured with 101,770 parameters, both achieving 94.4% (Fig. 3-c-11).

The specific parameters and accuracy of \acmlp, \accnn, \acgcn, \acrf, \acdt, Transformers,\aclasso, \acEla,our ablation method EACR and our method EAPCR results are shown in the Tab. 7.

**Table 7: Experimental results of \acmlp, \accnn,\acgcn,\acrf,\acdt, Transformers, our ablation method EACR, and our method EAPCR.**
| Raw Image (data with \acFRPs) | Synthesized data (data without \acFRPs) |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
| Method | Parameter quantity | Accuracy | Method | Parameter quantity | Accuracy |
| \acmlp-1 | 20680 | 90.5% | \acmlp-1 | 20680 | 90.6% |
| \acmlp-2 | 31810 | 91.4% | \acmlp-2 | 31810 | 91.5% |
| \acmlp-3 | 41350 | 92.3% | \acmlp-3 | 41350 | 92.0% |
| \acmlp-4 | 50890 | 92.7% | \acmlp-4 | 50890 | 92.8% |
| \acmlp-5 | 63610 | 93.5% | \acmlp-5 | 63610 | 93.5% |
| \acmlp-6 | 71560 | 93.7% | \acmlp-6 | 71560 | 93.6% |
| \acmlp-7 | 83485 | 94.2% | \acmlp-7 | 83485 | 93.9% |
| \acmlp-8 | 91435 | 94.2% | \acmlp-8 | 91435 | 94.3% |
| \acmlp-9 | 101770 | 94.3% | \acmlp-9 | 101770 | 94.4% |
| \acrf-1 | 28729 | 92.0% | \acrf-1 | 29241 | 91.7% |
| \acrf-2 | 57831 | 93.7% | \acrf-2 | 58390 | 93.6% |
| \acrf-3 | 87038 | 94.2% | \acrf-3 | 87377 | 94.2% |
| \acrf-4 | 116294 | 94.7% | \acrf-4 | 116168 | 94.2% |
| \acrf-5 | 145773 | 94.7% | \acrf-5 | 145613 | 94.8% |
| \acrf-6 | 291533 | 94.9% | \acrf-6 | 292517 | 94.9% |
| \acrf-7 | 584091 | 95.2% | \acrf-7 | 583980 | 95.2% |
| \acrf-8 | 876758 | 95.2% | \acrf-8 | 875610 | 95.1% |
| \acrf-9 | 1169481 | 95.3% | \acrf-9 | 1168306 | 95.1% |
| \acrf-10 | 1461923 | 95.4% | \acrf-10 | 1461973 | 95.1% |
| \acdt | 4815 | 83.3% | \acdt | 4815 | 83.1% |
| \aclasso | 7850 | 88.7% | \aclasso | 7850 | 88.8% |
| \acEla | 7850 | 88.9% | \acEla | 7850 | 88.9% |
| \ackan | 91390 | 87.4% | \ackan | 91390 | 88.0% |
| \ackan | 118386 | 86.6% | \ackan | 118386 | 87.5% |
| \accnn+\ackan | 93624 | 97.5% | \accnn+\ackan | 93624 | 89.7% |
| \accnn+\acmlp | 21928 | 97.7% | \accnn+\acmlp | 21928 | 90.8% |
| / | / | / | \acgcn with Random AM | 22451 | 88.2% |
| / | / | / | \acgcn with AM given by correlation matrix | 22451 | 91.2% |
| / | / | / | Transformer | 1252949 | 94.4% |
| / | / | / | Ablation EACR-1 | 29151 | 93.3% |
| / | / | / | Ablation EACR-2 | 59361 | 94.3% |
| / | / | / | Ablation EACR-3 | 79236 | 95.2% |
| / | / | / | Our EAPCR-1 | 37355 | 94.5% |
| / | / | / | Our EAPCR-2 | 67565 | 95.0% |
| / | / | / | Our EAPCR-3 | 87440 | 95.5% |

## Appendix E Experiment details: scientific tasks

Tab. 8 provides details of the heart disease dataset from the UCI Machine Learning Repository, consisting of 13 feature attributes and 1 target attribute. The attributes are as follows:
age (age in years),
sex (1 = male, 0 = female),
cp (chest pain type, where 1 = typical angina, 2 = atypical angina, 3 = non-anginal pain, 4 = asymptomatic),
trestbps (resting blood pressure in mm Hg on admission to the hospital),
chol (serum cholesterol in mg/dl),
fbs (fasting blood sugar $>120$ mg/dl, 1 = true, 0 = false),
restecg (resting electrocardiographic results, where 0 = normal, 1 = having ST-T wave abnormality such as T wave inversions or ST elevation/depression $>0.05$ mV, 2 = showing probable or definite left ventricular hypertrophy by Estes’ criteria),
thalach (maximum heart rate achieved),
exang (exercise-induced angina, 1 = yes, 0 = no),
oldpeak (ST depression induced by exercise relative to rest),
slope (slope of the peak exercise ST segment, where 1 = upsloping, 2 = flat, 3 = downsloping),
ca (number of major vessels colored by fluoroscopy, ranging from 0 to 3),
thal (where 3 = normal, 6 = fixed defect, 7 = reversible defect).
The target attribute indicates the presence of heart disease, where 0 represents no disease and 1 represents disease.

Tab. 9 provides details of the inorganic catalysis dataset. The value ranges for the characteristic variables are as follows: Dopant (Ag, Bi, C, Ce, Cd, F, Fe, Ga, I, Mo, N, Ni, S), Dopant/Ti mole ratio (0–93:5), Calcination temperature (400–900°C), Pollutant (Methylene blue [MB], phenol, rhodamine B [RhB], methyl orange [MO], methyl red [MR], acid orange [AO]), Catalyst/pollutant mass ratio (5:1–1000:1), pH (2–13), Experimental temperature (16–32°C), Light wavelength (254–600 nm), and Illumination time (5–480 minutes).

Tab. 10 provides details of the sensor dataset. The dataset includes the following variables: footfall (the number of people or objects passing by the machine), tempMode (the temperature mode or setting of the machine), AQ (air quality index near the machine), USS (ultrasonic sensor data, indicating proximity measurements), CS (current sensor readings, indicating the electrical current usage of the machine), VOC (volatile organic compounds level detected near the machine), RP (rotational position or RPM of the machine parts), IP (input pressure to the machine), Temperature (the operating temperature of the machine), and fail (binary indicator of machine failure, where 1 indicates failure and 0 indicates no failure).

**Table 8: UCI heart disease dataset**
| age | sex | cp | trestbps | chol | fbs | restecg | thalach | exang | oldpeak | slope | ca | thal | target |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 63 | 1 | 1 | 145 | 233 | 1 | 2 | 150 | 0 | 2.3 | 3 | 0 | 6 | 0 |
| 67 | 1 | 4 | 160 | 286 | 0 | 2 | 108 | 1 | 1.5 | 2 | 3 | 3 | 2 |
| 67 | 1 | 4 | 120 | 229 | 0 | 2 | 129 | 1 | 2.6 | 2 | 2 | 7 | 1 |
| … | … | … | … | … | … | … | … | … | … | … | … | … | … |
| … | … | … | … | … | … | … | … | … | … | … | … | … | … |
| … | … | … | … | … | … | … | … | … | … | … | … | … | … |
| 37 | 1 | 3 | 130 | 250 | 0 | 0 | 187 | 0 | 3.5 | 3 | 0 | 3 | 0 |
| 41 | 0 | 2 | 130 | 204 | 0 | 2 | 172 | 0 | 1.4 | 1 | 0 | 3 | 0 |
| 56 | 1 | 2 | 120 | 236 | 0 | 0 | 178 | 0 | 0.8 | 1 | 0 | 3 | 0 |
| 62 | 0 | 4 | 140 | 268 | 0 | 2 | 160 | 0 | 3.6 | 3 | 2 | 3 | 3 |

**Table 9: Inorganic catalysis dataset**
| Dopant | Dopant/Ti<br>mole ratio | Calcination<br>temperature | Pollutant | Catalyst/Pollutant<br>mass ratio | pH | Experimental<br>temperature | Light<br>wavelength | Illumination<br>time | Degradation<br>rate |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| C | 16 | 400 | MB | 100 | 7 | 25 | 425 | 20 | 30 |
| C | 16 | 400 | MB | 100 | 7 | 25 | 425 | 40 | 43 |
| C | 16 | 400 | MB | 100 | 7 | 25 | 425 | 60 | 48 |
| … | … | … | … | … | … | … | … | … | … |
| … | … | … | … | … | … | … | … | … | … |
| … | … | … | … | … | … | … | … | … | … |
| Fe | 0 | 500 | MB | 100 | 7 | 0 | 545 | 60 | 28.1 |
| Fe | 0 | 500 | MB | 100 | 7 | 0 | 545 | 120 | 44 |
| Fe | 0 | 500 | MB | 100 | 7 | 0 | 545 | 180 | 52.8 |
| Fe | 0 | 500 | MB | 100 | 7 | 0 | 545 | 300 | 69 |

**Table 10: Sensor measurements dataset**
| footfall | tempMode | AQ | USS | CS | VOC | RP | IP | Temperature | fail |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 0 | 7 | 7 | 1 | 6 | 6 | 36 | 3 | 1 | 1 |
| 190 | 1 | 3 | 3 | 5 | 1 | 20 | 4 | 1 | 0 |
| 5 | 5 | 3 | 3 | 6 | 1 | 24 | 6 | 1 | 0 |
| … | … | … | … | … | … | … | … | … | … |
| … | … | … | … | … | … | … | … | … | … |
| … | … | … | … | … | … | … | … | … | … |
| 74 | 7 | 4 | 4 | 7 | 2 | 88 | 2 | 2 | 0 |
| 190 | 0 | 2 | 4 | 6 | 2 | 20 | 4 | 2 | 0 |
| 12 | 3 | 4 | 6 | 3 | 2 | 27 | 3 | 2 | 0 |
| 0 | 7 | 6 | 1 | 6 | 6 | 44 | 4 | 2 | 1 |

We also applied our EAPCR method to additional non-image medical data, inorganic catalysis data, and system anomaly detection data, with the specific results shown below.

#### More non-image medical diagnosis:

Data1: Lung cancer dataset (Mamun et al., ), which contains 309 samples, 15 feature attributes, and 1 classification attribute. The feature attributes include: Gender (M = male, F = female), Smoking (YES = 2, NO = 1), Yellow fingers (YES = 2, NO = 1), Anxiety (YES = 2, NO = 1), Peer pressure (YES = 2, NO = 1), Chronic disease (YES = 2, NO = 1), Fatigue (YES = 2, NO = 1), Allergy (YES = 2, NO = 1), Wheezing (YES = 2, NO = 1), Alcohol consumption (YES = 2, NO = 1), Coughing (YES = 2, NO = 1), Shortness of breath (YES = 2, NO = 1), Swallowing difficulty (YES = 2, NO = 1), and Chest pain (YES = 2, NO = 1). The classification attribute is Lung Cancer (YES, NO). The specific results are shown in Tab. 11.

Data2: Breast cancer dataset (https://www.kaggle.com/datasets/abdelrahman16/breast-cancer-prediction), which contains 213 samples and 9 feature attributes.
The attributes include: Year (the year when the data was recorded), Age (age of the patient), Menopause (menopausal status of the patient, 1 for postmenopausal, 0 for premenopausal), Tumor Size (size of the tumor in centimeters), Inv-Nodes (presence of invasive lymph nodes), Breast (breast affected: Left or Right), Metastasis (presence of metastasis, 0 for no, 1 for yes), Breast Quadrant (quadrant of the breast where the tumor is located, e.g., Upper inner, Upper outer), History (patient’s history of breast cancer, 0 for no, 1 for yes), and Diagnosis Result (Benign or Malignant).
The specific experimental results are shown in Tab. 12.

**Table 11: Comparison of our method with others in the diagnosis of Lung cancer dataset**
| Method | Accuracy | Precision | Recall | F1 Score | AUC |
| --- | --- | --- | --- | --- | --- |
| Bagging | 89.76% | 91.88% | 89.35% | 90.00% | 95.30% |
| AdaBoost | 90.70% | 90.70% | 90.70% | 90.70% | 97.62% |
| LightGBM | 92.56% | 93.93% | 92.10% | 92.71% | 92.71% |
| XGBoost | 94.42% | 95.66% | 94.46% | 94.74% | 98.14% |
| EAPCR | 96.30% | 96.49% | 96.49% | 96.49% | 98.61% |

**Table 12: Comparison of our method with others in the diagnosis of Breast cancer dataset**
| Method | Accuracy | Precision | Recall | F1 Score |
| --- | --- | --- | --- | --- |
| XGBoost | 83.72% | 81.25% | 76.47% | 78.79% |
| \acdt | 83.72% | 81.25% | 76.47% | 78.79% |
| \acknn | 86.05% | 82.35% | 82.35% | 82.35% |
| Logistic Regression | 88.37% | 87.50% | 82.35% | 84.85% |
| Extra Trees Classifier | 88.37% | 87.50% | 82.35% | 84.85% |
| EAPCR | 93.02% | 100% | 82.35% | 90.32% |

#### More inorganic catalysis efficiency prediction:

Data 1: Thermocatalytic dataset (Schossler et al., ), which includes three metal elements (M1, M2, M3), support ID, M1M2M3 ratio, temperature, volume flow rate, methane flow rate, time, and methane/O2 ratio. The specific experimental results are shown in Tab. 13.

Data 2: Each sample in Dataset 2 (Puliyanda, ) contains 8 experimental variables, including organic pollutants (OC), UV light intensity (I, mW/cm^2), wavelength (W, nm), dosage (D, mg/cm^2), humidity (H, %), experimental temperature (T, ^∘C), reactor volume (R, L), and initial concentration of pollutants (InitialC, ppmv). The light intensity ranges from 0.36 to 75 mW/cm^2, the illumination wavelength ranges from 253.7 to 370 nm, the titanium dioxide dosage ranges from 0.012 to 5.427 mg/cm^2, the humidity ranges from 0 to 1600%, the experimental temperature ranges from 22 to 350^∘C, the reactor volume ranges from 0.04 to 216 L, and the initial concentration of pollutants ranges from 0.001 to 5944 ppmv. The photodegradation rate (k, min^-1/cm^2) is set as the response variable. The specific experimental results are shown in Tab. 14.

**Table 13: Comparison of our method with others in the Inorganic catalysis data set.**
| Model | MAE | MSE | RMSE | $R^{2}$ |
| --- | --- | --- | --- | --- |
| ANN with BO (Puliyanda, ) | / | 0.559 | / | 0.438 |
| Catboost + adaboost (Puliyanda, ) | / | 0.064 | / | 0.922 |
| GBM with BO (Puliyanda, ) | / | 0.117 | / | 0.882 |
| XGB with HYPEROPT (Puliyanda, ) | / | 0.073 | / | 0.927 |
| EAPCR | 0.131 ± 0.004 | 0.054 ± 0.001 | 0.233 ± 0.003 | 0.940 ± 0.002 |

**Table 14: Comparison of our method with others in the Inorganic catalysis data set.**
| Model | MAE | MSE | RMSE | $R^{2}$ |
| --- | --- | --- | --- | --- |
| \acrf | / | / | 3.40 | 0.89 |
| EAPCR | 1.27 ± 0.02 | 2.88 ± 0.09 | 1.69 ± 0.02 | 0.97 ± 0.00 |

#### More system anomaly detection:

Centrifugal pumps dataset (Mallioris et al., ). The dataset contains 5,118 rows of measurements from two centrifugal pumps from the same manufacturer. These measurements include key features: value_ISO, value_DEM, value_ACC, value_P2P, value_TEMP, minute, second, year, month, day, hour, and Machine_ID (1 for healthy, 2 for maintenance status). The specific results are shown in Tab. 15.

**Table 15: Comparison of our method with others in the diagnosis of Centrifugal pumps dataset.**
| Method | Accuracy | Precision | Recall | F1 Score |
| --- | --- | --- | --- | --- |
| \acsvm | 96.51% | 97.77% | 95.50% | 96.66% |
| \acNB | 96.51% | 97.77% | 95.50% | 96.66% |
| \acrf | 98.25% | 96.73% | 100% | 98.33% |
| XGBoost | 98.83% | 97.82% | 100% | 98.89% |
| EAPCR | 100% | 100% | 100% | 100% |

## Appendix F Why \ac FRPs matters?

In this study, we show that \acFRPs are an important type of prior knowledge for the application of deep learning methods in various tasks. Another well-recognized prior knowledge is the annotated dataset. Despite significant research on overcoming the scarcity of annotated data in various scientific applications (Zhou et al., ; Gao et al., ; Liu et al., ), fewer efforts have focused on designing feature extractors without prior knowledge of \acFRPs.

The key to designing an effective feature extractor is to effectively sample the feature combinations consisting of strongly interactive features where the combined effect exceeds the sum of their individual contributions  (Koh & Liang, ; Ali et al., ; Beraha et al., ; Deng et al., ). In this section, we demonstrate that these interactive feature combinations are selected from the combinations of correlated features. Typically, FRPs contain the underlying correlation information between features. As a result, combinations of related features often lead to combinations of correlated features, highlighting the importance of FRPs.

Here, we prove that these interactive features are correlated.

###### Proposition F.1 .

If features $A$ and $B$ are independent, then:

$$ $IG(Y,A)+IG(Y,B)=IG(Y,A,B)$ $$

where:

$$ $IG(Y,A)\triangleq H(Y)-H(Y|A)$ $$

$$ $IG(Y,B)\triangleq H(Y)-H(Y|B)$ $$

$$ $IG(Y,A,B)\triangleq H(Y)-H(Y|A,B)$ $$

with $H(Y)$ the entropy of $Y$, $IG(Y,A)$ the information gain of $Y$ given $A$, $H(Y|A)$ the conditional entropy of $Y$ given $A$, and $IG(Y,A,B)$ the information gain of $Y$ given $A$ and $B$

###### Proof.

Since $A$ and $B$ are independent, their information contribution to $Y$ is completely independent; therefore, their joint effect equals the simple sum of their individual effects minus the entropy of $Y$, i.e.:

$$ $H(Y|A,B)=H(Y|A)+H(Y|B)-H(Y)$ $$

Substituting the definition of information gain into the assumption of independent conditional entropy:

$$ $IG(Y,A,B)=H(Y)-H(Y|A,B)=H(Y)-(H(Y|A)+H(Y|B)-H(Y))$ $$

$$ $IG(Y,A,B)=2H(Y)-H(Y|A)-H(Y|B)$ $$

The sum of information gains:

$$ $IG(Y,A)+IG(Y,B)=(H(Y)-H(Y|A))+(H(Y)-H(Y|B))=2H(Y)-H(Y|A)-H(Y|B)$ $$

Thus, it is proved that $IG(Y,A)+IG(Y,B)=IG(Y,A,B)$.
∎

###### Proposition F.2 .

If:

$$ $IG(Y,A,B)>IG(Y,A)+IG(Y,B),$ (3) $$

then, features $A$ and $B$ are correlated.

###### Proof.

Rearranging (3) gives us

$$ $IG(Y,A,B)>IG(Y,A)+IG(Y,B)$ $$

Following the definition of information gain, we have:

$$ $H(Y)-H(Y|A,B)>(H(Y)-H(Y|A))+(H(Y)-H(Y|B))$ $$

$$ $H(Y)-H(Y|A,B)>2H(Y)-H(Y|A)-H(Y|B)$ $$

$$ $H(Y|A)+H(Y|B)-H(Y)>H(Y|A,B)$ $$

The above inequalities imply that $A$ and $B$ jointly provide more information than the sum of the information provided individually.
This is typically because there is some interaction or dependency between $A$ and $B$ that causes their combined information gain to exceed the individual gains, hence $A$ and $B$ are not independent.
∎

###### Proposition F.3 .

If feature $A$ and $B$ have an interaction, i.e.:

$$ $H(Y|A,B)<H(Y|A)+H(Y|B)-H(Y),$ (4) $$

then:

$$ $IG(Y,A,B)>IG(Y,A)+IG(Y,B)$ $$

###### Proof.

Rewriting information gain:

$$ $IG(Y,A)+IG(Y,B)=(H(Y)-H(Y|A))+(H(Y)-H(Y|B))=2H(Y)-H(Y|A)-H(Y|B)$ $$

Substituting (4) into the calculation of information gain gives:

$$ $IG(Y,A,B)=H(Y)-H(Y|A,B)$ $$

$$ $IG(Y,A,B)>H(Y)-(H(Y|A)+H(Y|B)-H(Y))$ $$

$$ $IG(Y,A,B)>2H(Y)-H(Y|A)-H(Y|B)$ $$

This implies that the joint information gain $IG(Y,A,B)$ exceeds the sum of the individual gains. This indicates a positive interaction between features $A$ and $B$ in influencing $Y$, where their combined impact reduces the uncertainty of $Y$ more than their individual effects summed simply.
∎

###### Proposition F.1 .

###### Proof.

###### Proposition F.2 .

###### Proof.

###### Proposition F.3 .

###### Proof.

#### Design a feature extractor with \ac FRPs.

If the feature correlation patterns are embedded in the \acFRPs, it is only necessary to sample the combinations of features that are known to be correlated, Because those features that are not correlated do not have interaction effects, there is no need to consider their combinations, as proved in  F.2. The sampling scope will be largely limited.

#### The challenge of designing a feature extractor without FRPs.

When the \acFRPs is unknown correlation patterns among features are unknown, for $N$ features, combinations of different features need to be considered. For example, when sampling one feature, the number of samplings is $C_{N}^{1}$, when sampling two features, it is $C_{N}^{2}$, and when sampling all $N$ features, it is $C_{N}^{N}$. Therefore, for a sample composed of $N$ features, the total number of samplings required is: $C_{N}^{1}$ + $C_{N}^{2}$ + $C_{N}^{3}$ + $\cdots$ + $C_{N}^{N}$ = $2^{N}-1$.

## Appendix G EAPCR recovers the hidden \ac FRPs

The effectiveness of our method lies in its ability to accurately reconstruct the correlation patterns within the data, even when these patterns become less apparent or lost after transformations. As shown in Fig. 7, whether the images are in their original state or transformed, the matrices reconstructed by our EAPCR model consistently reflect the true pixel correlation patterns with a recall rate of $84.6\%$. This highlights the model’s precision and reliability in restoring these patterns, even when pixel positions are altered.

In a more detailed analysis, we observed a $55.0\%$ recall rate when comparing the correlation matrix of transformed images to the original pixel patterns. Conversely, comparing the correlation matrix from the original images to the transformed image patterns yielded a recall rate of $66.3\%$. This indicates that for data containing different feature correlation patterns, the patterns restored by our model vary significantly. This demonstrates that our EAPCR is capable of adaptively restoring the unique hidden relationships between features. EAPCR shows a strong ability to recover underlying data relationships, further emphasizing its robustness and effectiveness in scenarios where explicit correlation patterns are not directly accessible.

Figure: Figure 7: The alignment between the pixel correlation matrices reconstructed using our technique and the actual pixel correlation patterns in both original and transformed images is demonstrated. The correlation matrices were binarized.
Refer to caption: x1.png

## Appendix H Data and Code

The public datasets can be found in the corresponding references. Other data and code will be released.

The experiments were conducted on devices with the following specifications. GPU: NVIDIA GeForce RTX 3090 with 24GB, RTX4080 with 16GB, and RTX4090 with 24GB of VRAM, CPU: 13th Gen Intel(R) Core(TM) i9-13900K. The source code is freely available at the GitHub repository after the peer review process.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: See weakness. The author needs to justify the reason for experimental design and why they can support the augments made.
soundness: 2
strengths: - The paper addresses an important machine learning challenge within the domain of structured scientific data, and the motivation for tackling this problem is clearly articulated.

- The proposed method and presentation that makes both the approach and the experiments easy to follow.
summary: The paper proposes a method to learn the representation of structured scientific data with no explicit connection among features. The method employs bilinear attention and permuted CNN to understand the implicit structure. The method achieves better performance on three real-world datasets and one synthetic dataset.
weaknesses: - The paper defines the term "feature relationship patterns" but fails to explain the concept well. It is unclear what relationship the proposed method learns among features. Also, the paper only experiments with the proposed method on three datasets, which is not sufficient to call a method "universal".

- The experiments are not comprehensive. In Table 1, there's no comparison with RF and XGBosst. In Table 3, there is no XGBoost. Also, no NN-based methods are shown in Tables 1, 2, 3 (e.g. MLP, Transformer). How the conclusion comes "While Kolmogorov–Arnold Network (KAN) and feature extractors like Convolutional Neural Networks (CNNs), Graph Convolutional Networks (GCNs), and Transformers struggle". 

- The synthetic experiment does not sound reasonable. Permuting patches in an image does corrupt the spatial relationship within an image, but it is irrelevant to the scientific data. The permutation corruption may highly overfit to the design of permuted CNN in the proposed method. 

- The proposed method is quite similar to a Transformer, but the paper fails to explain why it performs better than the Transformer. The only comparison is in Figure 3, but it is unclear which positional encoder, masks, number of layers and heads are used here. A Transformer can always be designed to learn different relationships. It is unclear that the benchmark in Figure 3 is the best a Transformer can do.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: See the weakness above
soundness: 2
strengths: Innovative approach to feature extraction: EAPCR’s ability to handle data without explicit Feature Relation Patterns (FRPs) is a significant advantage, enabling deep learning to be applied effectively in scientific domains traditionally dominated by simpler machine learning models.

Improved generalization and adaptability: By exposing and sampling all potential feature interactions, EAPCR offers a universal solution applicable to a wide range of scientific tasks, making it valuable for fields like healthcare and anomaly detection where deep learning previously struggled.
summary: The paper introduces EAPCR, a universal feature extractor designed for scientific data that lacks explicit Feature Relation Patterns (FRPs), which are spatial, sequential, or relational correlations typically leveraged by deep learning models. Traditional machine learning methods, like Decision Trees and their variants, have been more successful than deep learning models in such scientific tasks due to this absence of clear feature relationships.
weaknesses: 1. What is the difference between EAPCR and creating a all connected graphs with each feature as a node? There are lot of work that creating graph with unknown edge relations, for example, use correlation and etc. Given this, I think this work is limited on innovation.

2. If you do want to claim that EAPCR is a better model compared to many other algorithms, it is more fair to compare EAPCR with graph algorithms like different GNNs for experiments in Table1 and 2.

3. Some of the experiments are using Kaggle datasets. I know that lot of kaggle datasets can improve performance by simply ensemble different simple ML algorithms. So this should also be included in experiments as a baseline.

4. Will this cost too much for high dimensional features? It will be better to include some complexity analysis here.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: none
soundness: 2
strengths: ●	The paper targets the problem of feature extraction in scientific data where explicit relational structures are absent, a common issue in many applications. 
●	EAPCR is extensively tested across various datasets, and the results consistently show its advantages over several traditional baselines.
summary: The paper introduces EAPCR, a feature extraction method aimed at handling scientific datasets that lack explicit Feature Relation Patterns. EAPCR employs a permutation-based approach to transform feature relationships, followed by CNNs to capture both local and non-local correlations. The model is evaluated across multiple tasks and synthetic datasets, where it reportedly outperforms traditional ML and DL methods.
weaknesses: ●	The paper lacks comparisons with simple DL models on the UCI Cleveland heart disease and photocatalyst datasets, such as a small MLP without inductive biases. The ML baselines are based on rigid assumptions, while EAPCR leverages a shallow MLP to generate the embeddings. The current comparisons make it difficult to determine the true source of the performance gains. 
●	The need for permutation before applying the CNN is not well justified. A multi-layer CNN should, in theory, learn both local and global correlations without this step. There is no strong theoretical argument provided for why permutation should improve the ability to capture feature interactions over standard CNN techniques. 
●	More importantly, if a single permutation is helpful, why not use multiple permutations concatenated before feeding them into CNNs? 
●	Another concern is that shuffling or permuting data to generate synthetic datasets disrupts important structural information, which does not reflect real-world challenges where meaningful relationships are usually present. 
●	The details about how the permutation matrix is constructed should be moved to the main body of the paper, given its central role in the proposed method. 
●	It is uncertain whether the superiority of EAPCR would persist as the dataset size and model complexity increase.

---

confidence: 2
contribution: 2
initial_rating: 3
presentation: 2
questions: In addition to the questions above, I have a few more questions : 

-  How do you handle missing values and outliers?
- I am still confused by why you need to convert all features into categorical features? For ex. how do you handle the iris dataset? 
- Related to the question above : how do you handle a noisy pinwheel dataset with different colors and the objective is to predict the color based on the coordinates? 
- What kind of relations does our method learn in various tabular datasets? 
- Without error bars, I am not sure if results in Table 3 are statistically significant. Also Table 3 is mostly empty. 
- I am quite confused by Fig 3. An MLP on MNIST gets around 98%. The numbers reported are quite low. 
- I may be missing something but what is the utility of graph datasets where the adjacency matrices are random and image datasets where the pixels are randomly shuffled.  
- How is your method related to SSL? For example, treat $\mathbf{A}$ and permuted $\mathbf{A}$ as positive pairs.  
- Related to the works on vision datasets, how does your method relate to MLP mixer [1]?

[1] MLP-Mixer: An all-MLP Architecture for Vision. Tolstikhin et al. NeurIPS 2021
soundness: 2
strengths: - The problem definition is very important and is relevant to the ML community and practitioners. 
- The method seems general and can be used in various settings. 
- Empirical results look strong.
summary: In this work, the authors introduce EAPCR, a universal feature extractor designed for data lacking explicit relationships between features. This absence of explicit relationships often poses challenges for deep learning models on tabular datasets, where traditional ML models like decision trees tend to excel. The authors demonstrate the effectiveness and robustness of EAPCR on UCI datasets as well as on synthetic datasets.
weaknesses: - Since the permutation $\mathbf{M}$ plays such a key role, please provide some details or intuition in the main paper on how to create these permutations for various datasets and modalities. 
- Regarding the comment on GCN performance deteriorating on random adjacency matrices (line 428), this is not surprising. However to show that EAPCR is robust to these changes, the authors should use benchmark graph datasets and validate their findings. 
- For tabular datasets, LightGBM or XGB would be better baselines (table 1). 
- This method seems extremely computationally expensive. For a $N \times N$ image, it creates a permutation matrix of shape $N^2 \times N^2$. 
- For tabular datasets, it is not clear how the binning is performed as creating a very fine/coarse binning may affect the values of the design matrix.