SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Encoder  $\phi$ : The Encoder aims to map any given postfix expression to an embedding (a.k.a., hidden state)  $\mathbf{E}$ . We adopt a single layer long short-term memory [10] (LSTM) as Encoder and acquire the continuous representation of  $\Upsilon$ , denoted by  $\mathbf{E} = \phi(\Upsilon) \in \mathbf{R}^{M \times d}$ , where  $M$  is the total length of input sequence  $\Upsilon$  and  $d$  is the hidden size of the embedding.
Decoder  $\psi$ : The Decoder aims to reconstruct the postfix expression of the feature transformation operation sequence  $\Upsilon$  from the hidden state  $\mathbf{E}$ . In MOAT, we set the backbone of the Decoder as a single-layer LSTM. For the first step,  $\psi$  will take an initial state (denoted as  $h_0$ ) as input. Specifically, in step- $i$ , we can obtain the decoder hidden state  $h_i^d$  from the LSTM. We use the dot product attention to aggregate the encoder hidden state and obtain the combined encoder hidden state  $h_i^e$ . Then, the distribution in step- $j$  can be defined as:  $P_{\psi}(\gamma_i|\mathbf{E},\Upsilon_{<i}) = \frac{\exp(W_{\gamma_i}(h_i^d\oplus h_i^e))}{\sum_{c\in\mathcal{C}}\exp(W_c(h_i^d\oplus h_i^e))}$ , where  $\gamma_i\in \Upsilon$  is the  $i$ -th token in sequence  $\Upsilon$ , and  $\mathcal{C}$  is the token set.  $W$  stand for the parameter of the feedforward network.  $\Upsilon_{<i}$  represents the prediction of the previous or initial step. By multiplying the probability in each step, we can form the distribution of each token in  $\Upsilon$ , given as:  $P_{\psi}(\Upsilon |\mathbf{E}) = \prod_{i = 1}^{M}P_{\psi}(\gamma_i|\mathbf{E},\Upsilon_{<i})$ . To make the generated sequence similar to the real one, we minimize the negative log-likelihood of the distribution, defined as:  $\mathcal{L}_{rec} = -\log P_{\psi}(\Upsilon |\mathbf{E})$ .
**Evaluator  $\omega$ : The Evaluator is designed to estimate the quality of continuous embeddings. Specifically, we will first conduct mean pooling on  $\mathbf{E}$  by column to aggregate the information and obtain the embedding  $\bar{\mathbf{E}} \in \mathbf{R}^d$ . Then  $\bar{\mathbf{E}}$  is input into a feedforward network to estimate the corresponding model performance, given as:  $\hat{v} = \omega(\mathbf{E})$ . To minimize the gap between estimated accuracy and real-world gold accuracy, we leverage the Mean Squared Error (MSE) given by:  $\mathcal{L}_{est} = MSE(v, \omega(\mathbf{E}))$ .
Joint Training Loss  $\mathcal{L}$ : We jointly optimize the encoder, decoder, and evaluator. The joint training loss can be formulated as:  $\mathcal{L} = \alpha \mathcal{L}_{rec} + (1 - \alpha)\mathcal{L}_{est}$ , where  $\alpha$  is the trade-off hyperparameter that controls the contribution of sequence reconstruction and accuracy estimation loss.
3.5 Gradient-Ascent Optimal Embedding Search
To conduct the optimal embedding search, we first select top-  $T$  transformation sequences ranked by the downstream predictive accuracy. The well-trained encoder is then used to embed these postfix expressions into continuous embeddings, which later will be used as seeds (starting points) of gradient ascent. Assuming that one search seed embedding is  $\mathbf{E}$ , we search, starting from  $\mathbf{E}$ , toward the gradient direction induced by the evaluator  $\omega$ :  $\tilde{\mathbf{E}} = \mathbf{E} + \eta \frac{\partial \omega}{\partial \mathbf{E}}$ , where  $\tilde{\mathbf{E}}$  denotes the refined embedding,  $\eta$  is the size of each searching step. The model performance of  $\tilde{\mathbf{E}}$  is supposed to be better than  $\mathbf{E}$  due to  $\omega(\tilde{\mathbf{E}}) \geq \omega(\mathbf{E})$ . For  $T$  seeds, we can obtain the enhanced embeddings  $[\tilde{\mathbf{E}}_1, \tilde{\mathbf{E}}_2, \dots, \tilde{\mathbf{E}}_T]$ .
3.6 Transformation Operation Sequence Reconstruction and Evaluation
We reconstruct the transformation sequences by the well-trained decoder  $\psi$  using the collected candidate (i.e., acceptable optimal) embeddings  $[\tilde{\mathbf{E}}_1, \tilde{\mathbf{E}}_2, \dots, \tilde{\mathbf{E}}_T]$ . This process can be denoted by:  $[\tilde{\mathbf{E}}_1, \tilde{\mathbf{E}}_2, \dots, \tilde{\mathbf{E}}_T] \xrightarrow{\psi} \{\tilde{\Upsilon}_i\}_{i=1}^T$ . To identify the best transformation sequence, we adopt the
beam search strategy [11, 12, 13] to generate feature transformation operation sequence candidates. Specifically, given a refined embedding  $\tilde{\mathbf{E}}$ , at step-t, we maintain the historical predictions with beam size  $b$ , denoted as  $\{\Upsilon_{<t}^i\}_{i=1}^b$ . For the  $i$ -th beam, the probability distribution of the token identified by the well-trained decoder  $\psi$  at the  $t$ -th step is  $\gamma$ , which can be calculated as follows:  $P_t^i(\gamma) = P_\psi(\gamma|\tilde{\mathbf{E}}, \tilde{\Upsilon}_{<t}^i) * P_\psi(\tilde{\Upsilon}_{<t}^i|\tilde{\mathbf{E}})$ , where the probability distribution  $P_t^i(\gamma)$  is the continued multiplication between the probability distribution of the previous decoding sequence and that of the current decoding step. We can collect the conditional probability distribution of all tokens for each beam. After that, we append tokens with top- $b$  probability values to the historical prediction of each beam to get a new historical set  $\{\tilde{\Upsilon}_{<t+1}^i\}_{i=1}^b$ . We can iteratively conduct this decoding process until confronted with the <EOS> token. We select the transformation sequence with the highest probability value as output. Hence,  $T$  enhanced embeddings may produce  $T$  transformation sequences  $\{\tilde{\Upsilon}_i\}_{i=1}^T$ . We divide each of them into different parts according to the <SEP> token and check the validity of each part and remove invalid ones. Here, the validity measures whether the mathematical compositions represented by the postfix part can be successfully calculated to produce a new feature. These valid postfix parts reconstruct a feature transformation operation sequence  $\{\tilde{\Gamma}_i\}_{i=1}^T$ , which are used to generate refined feature space  $\{\tilde{X}_i\}_{i=1}^T$ . Finally, we select the feature set with the highest downstream ML performance as the optimal feature space  $\mathbf{X}^*$ .
This section reports the results of both quantitative and qualitative experiments that were performed to assess MOAT with other baseline models. All experiments were conducted on AMD EPYC 7742 CPU, and 8 NVIDIA A100 GPUs. For more platform information, please refer to Appendix B.2.
We used 23 publicly available datasets from UCI [14], LibSVM [15], Kaggle [16], and OpenML [17] to conduct experiments. The 23 datasets involve 14 classification tasks and 9 regression tasks. Table 1 shows the statistics of these datasets. We used F1-score, Precision, Recall, and ROC/AUC to evaluate classification tasks. We used 1-Relative Absolute Error (1-RAE) [4], 1-Mean Average Error (1-MAE), 1-Mean Square Error (1-MSE), and 1-Root Mean Square Error (1-RMSE) to evaluate regression tasks. We used the Valid Rate to evaluate the transformation sequence generation. A valid sequence means it can successfully conduct mathematical compositions without any ambiguity and errors. The valid rate is the average of all correct sequence numbers divided by the total number of generated sequences. The greater the valid rate is, the superior the model performance is. Because it indicates that the model can capture the complex patterns of mathematical compositions and search for more effective feature transformation sequences.
We compared our method with eight widely-used feature generation methods: (1) RDG generates feature-operation-feature transformation records at random for generating new feature space; (2) ERG first applies operation on each feature to expand the feature space, then selects the crucial features as new features. (3) LDA [18] is a matrix factorization-based method to obtain the factorized hidden state as the generated feature space. (4) AFAT [2] is an enhanced version of ERG that repeatedly generate new features and use multi-step feature selection to select informative ones. (5) NFS [7] models the transformation sequence of each feature and uses RL to optimize the entire feature generation process. (6) TTG [5] formulates the transformation process as a graph, then implements an RL-based search method to find the best feature set. (7) GRFG [4] uses three collaborated reinforced agents to conduct feature generation and proposes a feature grouping strategy to accelerate agent learning. (8) DIFER [8] embeds randomly generated feature transformation records with a seq2seq model, then employs gradient search to find the best feature set. MOAT and DIFER belong to the same setting. We demonstrate the differences between them in Appendix ??. Besides, we developed two variants of MOAT in order to validate the impact of each technical component: (i)  $\mathbf{MOAT}^{-d}$  replaces the RL-based data collection component with collecting feature transformation-accuracy pairs at random. (ii)  $\mathbf{MOAT}^{-a}$  removes the data augmentation component. We randomly split each dataset into two independent sets. The prior  $80\%$  is used to build the continuous embedding space and the remaining  $20\%$  is employed to test transformation performance. This experimental setting avoids any test data leakage and ensures a fairer transformation performance comparison. We adopted Random Forest as the downstream machine learning model. Because it is a robust, stable, well-tested method, thus,
Table 1: Overall performance comparison. 'C' for binary classification, and 'R' for regression. The best results are highlighted in bold. The second-best results are highlighted in underline. (Higher values indicate better performance.)
Dataset	Source	C/R	Samples	Features	RDG	ERG	LDA	AFAT	NFS	TTG	GRFG	DIFER	MOAT
Higgs Boson	UCIrvine	C	50000	28	0.695	0.702	0.513	0.697	0.691	0.699	0.707	0.669	0.712
Amazon Employee	Kaggle	C	32769	9	0.932	0.934	0.916	0.930	0.932	0.933	0.932	0.929	0.936
PimaIndian	UCIrvine	C	768	8	0.760	0.761	0.638	0.765	0.749	0.745	0.754	0.760	0.807
SpectF	UCIrvine	C	267	44	0.760	0.757	0.665	0.760	0.792	0.760	0.818	0.766	0.912
SVMGuide3	LibSVM	C	1243	21	0.787	0.826	0.652	0.795	0.792	0.798	0.812	0.773	0.849
German Credit	UCIrvine	C	1001	24	0.680	0.740	0.639	0.683	0.687	0.645	0.683	0.656	0.730
Credit Default	UCIrvine	C	30000	25	0.805	0.803	0.743	0.804	0.801	0.798	0.806	0.796	0.810
Messidor_features	UCIrvine	C	1150	19	0.624	0.669	0.475	0.665	0.638	0.655	0.692	0.660	0.749
Wine Quality Red	UCIrvine	C	999	12	0.466	0.461	0.433	0.480	0.462	0.467	0.470	0.476	0.559
Wine Quality White	UCIrvine	C	4900	12	0.524	0.510	0.449	0.516	0.525	0.531	0.534	0.507	0.536
SpamBase	UCIrvine	C	4601	57	0.906	0.917	0.889	0.912	0.925	0.919	0.922	0.912	0.932
AP-omentum-ovary	OpenML	C	275	10936	0.832	0.814	0.658	0.830	0.832	0.758	0.849	0.833	0.885
Lymphography	UCIrvine	C	148	18	0.108	0.144	0.167	0.150	0.152	0.148	0.182	0.150	0.267
Ionosphere	UCIrvine	C	351	34	0.912	0.921	0.654	0.928	0.913	0.902	0.933	0.905	0.985
Housing Boston	UCIrvine	R	506	13	0.404	0.409	0.020	0.416	0.425	0.396	0.404	0.381	0.467
Airfoil	UCIrvine	R	1503	5	0.519	0.519	0.220	0.521	0.519	0.500	0.521	0.558	0.629
Openml_618	OpenML	R	1000	50	0.472	0.561	0.052	0.472	0.473	0.467	0.562	0.408	0.692
Openml_589	OpenML	R	1000	25	0.509	0.610	0.011	0.508	0.505	0.503	0.627	0.463	0.656
Openml_616	OpenML	R	500	50	0.070	0.193	0.024	0.149	0.167	0.156	0.372	0.076	0.526
Openml_607	OpenML	R	1000	50	0.521	0.555	0.107	0.516	0.519	0.522	0.621	0.476	0.673
Openml_620	OpenML	R	1000	25	0.511	0.546	0.029	0.527	0.513	0.512	0.619	0.442	0.642
Openml_637	OpenML	R	500	50	0.136	0.152	0.043	0.176	0.152	0.144	0.307	0.072	0.465
Openml_586	OpenML	R	1000	25	0.568	0.624	0.110	0.543	0.544	0.544	0.646	0.482	0.700
* We reported F1-Score for classification tasks, and 1-RAE for regression tasks.
(a) Spectf
(b) SpamBase
Figure 4: The influence of data collection  $(\mathrm{MOAT}^{-d})$  and data augmentation  $(\mathrm{MOAT}^{-a})$  in MOAT. we can reduce performance variation caused by the model, and make it easy to study the impact of feature space. We provided more experimental and hyperparameter settings in Appendix B.1.
(c) Openml_616
(d) Openml_618
4.3 Performance Evaluation
Overall Performance This experiment aims to answer: Can MOAT effectively generate transformation sequence for discovering optimal feature space with excellent performance? Table 1 shows the overall comparison between MOAT and other models in terms of F1-score and 1-RAE. We noticed that MOAT beats others on all datasets. The underlying driver is that MOAT builds an effective embedding space to preserve the knowledge of feature transformation, making sure the gradient-ascent search module can identify the best-transformed feature space following the gradient direction. Another interesting observation is that MOAT significantly outperforms DIFER and has a more stable performance. There are two possible reasons: 1) The high-quality transformation records produced by the RL-based data collector provide a robust and powerful foundation for building a discriminative embedding space; 2) Postfix notation-based transformation sequence greatly decreases the search space, making MOAT easily capture feature transformation knowledge. Thus, this experiment validates the effectiveness of MOAT.
Data collection and augmentation. This experiment aims to answer: Is it essential to conduct data collection and augmentation to maintain the performance of MOAT? To achieve this goal, we developed two model variants of MOAT: 1)  $\mathrm{MOAT}^{-d}$ , which randomly collects feature transformation records for continuous space construction; 2)  $\mathrm{MOAT}^{-a}$ , which removes the data augmentation step in MOAT. We select two classification tasks (i.e., Spectf and SpamBase) and two regression tasks (i.e., Openml_616 and Openml_618) to show the comparison results. The results are reported in the Figure 4. Firstly, we found that the performance of MOAT is much better than  $\mathrm{MOAT}^{-d}$ . The underlying driver is that high-quality transformation records collected by the RL-based collector build a robust foundation for embedding space learning. It enables gradient-ascent search to effectively identify the optimal feature space. Moreover, we observed that the performance of  $\mathrm{MOAT}^{-a}$  is inferior to MOAT. This observation reflects that limited data volume and diversity cause the construction
Table 3: Time complexity comparison between MOAT and DIFER
Dataset	Model Name	Data Collection 512 instances	Model Training	Solution Searching	Performance
Wine Red	MOAT	921.6	5779.2	101.2	0.559
Wine White	MOAT	2764.8	7079.6	103.4	0.536
Openml_618	MOAT	4556.8	30786.1	111.3	0.692
Openml_589	MOAT	4044.8	8942.3	105.2	0.656
Wine Red	DIFER	323.2	11	180	0.476
Wine White	DIFER	1315.8	33	624	0.507
Openml_618	DIFER	942.3	33	534	0.408
Openml_589	DIFER	732.5	157	535	0.463
of embedding space to be unstable and noisy. Thus, this experiment shows the necessity of the data collection and augmentation components in MOAT.
Beam search. This experiment aims to answer What is the influence of beam search for improving the valid rate of generated transformation sequences? To observe the impacts of beam search, we set the beam size as 5 and 1 respectively, and add DIFER as another comparison object. We compare the generation performance in terms of valid rate. Figure 5 shows the comparison results in terms of valid rate. We noticed that the 5-beams search outperforms the 1-beam search. The underlying driver is that the
Figure 5: The influence of beam size in MOAT.
increased beam size can identify more legal and reasonable transformation sequences. Another interesting observation is that DIFER is significantly worse than MOAT and its error bar is longer. The underlying driver is that DIFER collects transformation records at random and it generates transformation sequences using greedy search. The collection and generation ways involve more random noises, distorting the learned embedding space.
Robustness check. This experiment aims to answer: Is MOAT robust to different downstream machine learning models? We replaced the downstream ML models with Random Forest (RF), XGBoost (XGB), Support Vector Machine (SVM), K-Nearest Neighborhood (KNN), Ridge, LASSO, and Decision Tree (DT) to observe the variance of model performance respectively. Table 2 shows the comparison results on Spectf in terms of the F1-score. We observed that MOAT keeps the best performance regard
Table 2: Robustness check of MOAT with distinct ML models on Spectf dataset in terms of F1-score.
RF	XGB	SVM	KNN	Ridge	LASSO	DT
RDG	0.760	0.818	0.750	0.792	0.718	0.749	0.864
ERG	0.757	0.813	0.753	0.766	0.778	0.750	0.790
LDA	0.665	0.715	0.760	0.749	0.759	0.760	0.665
AFAT	0.760	0.808	0.722	0.759	0.723	0.770	0.844
NFS	0.792	0.799	0.732	0.792	0.744	0.749	0.864
TTG	0.760	0.819	0.765	0.750	0.716	0.749	0.842
GRFG	0.818	0.842	0.580	0.760	0.729	0.744	0.786
DIFER	0.766	0.794	0.727	0.777	0.647	0.744	0.809
MOAT	0.912	0.897	0.876	0.916	0.780	0.844	0.929
less of downstream ML models. A possible reason is that the RL-based data collector can customize the transformation records based on the downstream ML model. Then, the learned embedding space may comprehend the preference and properties of the ML model, thereby resulting in a globally optimal feature space. Thus, this experiment shows the robustness of MOAT.
Time complexity analysis. This experiment aims to answer What is the time complexity of each technical component of MOAT? To accomplish this, we selected the SOTA model DIFER as a comparison baseline. Figure 3 shows the time costs comparison of data collection, model training, and solution searching in terms of seconds. We let both MOAT and DIFER collect 512 instances in the data collection phase. We found that the increased time costs for MOAT occur in the data collection and model training phases. However, once the model converges, MOAT's inference time is significantly reduced. The underlying driver is that the RL-based collector spends more time gathering high-quality data, and the sequence formulation for the entire feature space increases the learning time cost for the sequential model. But, during inference, MOAT outputs the entire feature transformation at once, whereas DIFER requires multiple reconstruction iterations based on the generated feature space's dimension. The less inference time makes MOAT more practical and suitable for real-world scenarios.
To thoroughly analyze the multiple characteristics of MOAT, we also analyzed the space complexity (See Appendix C.1), model scalability (See Appendix C.2), and parameter sensitivity (See Appendix C.3). Meanwhile, we provided two qualitative analyses: learned embedding visualization (See Appendix C.4) and traceability case study (See Appendix C.5).
Automated Feature Transformation (AFT) can enhance the feature space by automatically transforming the original features via mathematical operations [19, 20]. Existing works can be divided into three categories: 1) expansion-reduction based approaches [1, 3, 21, 2, 22]. Those methods first expand the original feature space by explicitly [23] or greedily [24] decided mathematical transformation, then reduce the space by selecting useful features. However, they are hard to produce or evaluate both complicated and effective mathematical compositions, leading to inferior performance. 2) evolution-evaluation approaches [4, 5, 6, 25, 9, 26]. These methods integrate feature generation and selection into a closed-loop learning system. They iteratively generate effective features and keep the significant ones until they achieve the maximum iteration number. The entire process is optimized by evolutionary algorithms or reinforcement learning models. However, they still focus on how to simulate the discrete decision-making process in feature engineering. Thus, they are still time-consuming and unstable. 3) Auto ML-based approaches [7, 8]. Auto ML aims to find the most suitable model architecture automatically [27, 28, 29, 30]. The success of auto ML in many area [31, 32, 33, 34, 35, 36] and the similarity between auto ML and AFT inspire researchers to formulate AFT as an auto ML task to resolve. However, they are limited by: 1) incapable of producing high-order feature transformation; and 2) unstable transformation performance. Recent studies tried to convert the feature transformation problem into a continuous space optimization task to search for the optimal feature space efficiently. DIFER [8] is a cutting-edge method that is comparable to our work in problem formulation. However, the following constraints limit DIFER's practicality: 1) DIFER collects transformation-accuracy data at random, resulting in many invalid training data with inconsistent transformation performances; 2) DIFER embeds and reconstructs each transformed feature separately and, thus, ignores feature-feature interactions; 3) DIFER needs to manually decide the number of generated features, making the reconstruction process more complicated. 4) the greedy search for transformation reconstruction in DIFER leads to suboptimal transformation results. To fill these gaps, we first implement an RL-based data collector to automate high-quality transformation record collection. We then leverage the postfix expression idea to represent the entire transformation operation sequence to model feature interactions and automatically identify the number of reconstructed features. Moreover, we employ beam search to advance the robustness, quality, and validity of transformation operation sequence reconstruction.
In this paper, we propose an automated feature transformation framework, namely MOAT. In detail, we first develop an RL-based data collector to gather high-quality transformation-accuracy pairs. Then, we offer an efficient postfix-based sequence expression way to represent the transformation sequence in each pair. Moreover, we map them into a continuous embedding space using an encoder-decoder-evaluator model structure. Finally, we employ a gradient-ascent search to identify better embeddings and then use beam search to reconstruct the transformation sequence and identify the optimal one. Extensive experiments show that the continuous optimization setting can efficiently search for the optimal feature space. The RL-based data collector is essential to keep an excellent and stable transformation performance. The postfix expression sequence enables MOAT to automatically determine the transformation depth and length, resulting in more flexible transformation ways. The beam search technique can increase the validity of feature transformation. The most noteworthy research finding is that the success of MOAT indicates that the knowledge of feature transformation can be embedded into a continuous embedding space to search for better feature space. Thus, it inspires us to regard the learning paradigm as the backbone to develop a large feature transformation model and quickly fine-tune it for different sub-tasks, which is also our future research direction.
This research was partially supported by the National Science Foundation (NSF) via grant numbers: 2040950, 2006889, and 2045567.