SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[1] James Max Kanter and Kalyan Veeramachaneni. Deep feature synthesis: Towards automating data science endeavors. In 2015 IEEE international conference on data science and advanced analytics (DSAA), pages 1-10. IEEE, 2015.
[2] Franziska Horn, Robert Pack, and Michael Rieger. The autofeat python library for automated feature engineering and selection. arXiv preprint arXiv:1901.07329, 2019.
[3] Udayan Khurana, Deepak Turaga, Horst Samulowitz, and Srinivasan Parthasarathy. Cognito: Automated feature engineering for supervised learning. In 2016 IEEE 16th International Conference on Data Mining Workshops (ICDMW), pages 1304-1307. IEEE, 2016.
[4] Dongjie Wang, Yanjie Fu, Kunpeng Liu, Xiaolin Li, and Yan Solihin. Group-wise reinforcement feature generation for optimal and explainable representation space reconstruction. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '22, page 1826-1834, New York, NY, USA, 2022. Association for Computing Machinery.
[5] Udayan Khurana, Horst Samulowitz, and Deepak Turaga. Feature engineering for predictive modeling using reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[6] Binh Tran, Bing Xue, and Mengjie Zhang. Genetic programming for feature construction and selection in classification on high-dimensional data. Memetic Computing, 8(1):3-15, 2016.
[7] Xiangning Chen, Qingwei Lin, Chuan Luo, Xudong Li, Hongyu Zhang, Yong Xu, Yingnong Dang, Kaixin Sui, Xu Zhang, Bo Qiao, et al. Neural feature search: A neural architecture for automated feature engineering. In 2019 IEEE International Conference on Data Mining (ICDM), pages 71-80. IEEE, 2019.
[8] Guanghui Zhu, Zhuoer Xu, Chunfeng Yuan, and Yihua Huang. Difer: differentiable automated feature engineering. In International Conference on Automated Machine Learning, pages 17-1. PMLR, 2022.
[9] Meng Xiao, Dongjie Wang, Min Wu, Ziyue Qiao, Pengfei Wang, Kunpeng Liu, Yuanchun Zhou, and Yanjie Fu. Traceable automatic feature transformation via cascading actor-critic agents. Proceedings of the 2023 SIAM International Conference on Data Mining (SDM), pages 775-783.
[10] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):1735-1780, 1997.
[11] Markus Freitag and Yaser Al-Onaizan. Beam search strategies for neural machine translation. arXiv preprint arXiv:1702.01806, 2017.
[12] Thibault Viglino, Petr Motlicek, and Milos Cernak. End-to-end accented speech recognition. In Interspeech, pages 2140-2144, 2019.
[13] Chunhui Bao and Qianru Sun. Generating music with emotions. IEEE Transactions on Multimedia, 2022.
[14] Public. Uci dataset download. [EB/OL], 2022. https://archive.ics.uci.edu/.
[15] Lin Chih-Jen. Libsvm dataset download. [EB/OL], 2022. https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/.
[16] Jeremy Howard. Kaggle dataset download. [EB/OL], 2022. https://www.kaggle.com/datasets.
[17] Public. Openml dataset download. [EB/OL], 2022. https://www.openml.org.
[18] David M Blei, Andrew Y Ng, and Michael I Jordan. Latent dirichlet allocation. the Journal of machine Learning research, 3:993-1022, 2003.
A Postfix Transformation Conversion
Algorithm 1 is the pseudo-code for the convert the original feature transformation sequence  $\Gamma$  to the postfix notation based sequence  $\Upsilon$ . In detail, we first initialize a list  $\Upsilon$  and two stacks  $S_{1}$  and  $S_{2}$ , respectively. For each element  $\tau$  in  $\Gamma$ , we scan it from left to right. When getting a feature ID token, we push it to  $S_{2}$ . When receiving a left parenthesis, we push it to  $S_{1}$ . When obtaining any operations, we pop each element in  $S_{1}$  and push them into  $S_{2}$  until the last component of  $S_{1}$  is the left bracket. Then we push this operation into  $S_{1}$  when getting a right parenthesis, we pop every element from  $S_{1}$  and then push into  $S_{2}$  until we confront a left bracket. Then we remove this left bracket from the top of  $S_{1}$ . When the end of the input  $\tau$  encounters, we append every token from  $S_{2}$  into  $\Upsilon$ . If this  $\tau$  is not the last element in  $\Gamma$ , we will append a <SEP> token to indicate the end of this  $\tau$ . After we process every element in  $\Gamma$ , we add <SOS> and <EOS> tokens to the beginning and end of  $\Upsilon$  to form the postfix notation-based transformation sequence  $\Upsilon = [\gamma_{1}, \dots, \gamma_{M}]$ . Each element in  $\Upsilon$  is a feature ID token, operation token, or three other special tokens. We convert each transformation sequence through this algorithm and construct the training set with them.
Algorithm 1: Postfix transformation sequence conversion  
input :Feature transformation sequence Γ  
output:Postfix notation based transformation sequence Y  
1 Y←∅;  
2 for τ← Γ do  
3 S1, S2←∅,∅;  
4 for γ← τ do  
5 if γ is left bracket then  
6 |S1.push(γ);  
7 else if γ is right bracket then  
8 while t← S1.pop() is not left bracket do  
9 |S2.push(t);  
10 else if γ is operation then  
11 while S1peek() is not left bracket do  
12 |S2.push(S1.pop());  
13 S1.push(γ);  
14 else  
15 |S2.push(γ);  
16 while S2not = ∅ do  
17 |Y.append(S2.pop(0));  
18 if τ is not the last element then  
19 |Y.append(SEP>);  
20 Add <SOS> and <EOS> to the head and tail of Y respectively;  
21 return Y
B Experimental Settings and Reproducibility
B.1 Hyperparameter Settings and Reproducibility
The operation set consists of square root, square, cosine, sine, tangent, exp, cube, log, reciprocal, quantile transformer, min-max scale, sigmoid, plus, subtract, multiply, divide. For the data collection part, we ran the RL-based data collector for 512 epochs to collect a large amount of feature transformation-accuracy pairs. For the data augmentation part, we randomly shuffled each transformation sequence 12 times to increase data diversity and volume. We adopted a single-layer LSTM
Table 4: Space complexity comparison on MOAT with different dataset
Dataset	Sample Number	Column Number	Parameter Size
Airfoil	1503	5	139969
Amazon employee	32769	9	138232
ap_omentum OVARY	275	10936	155795
german_credit	1001	24	141127
higgs	50000	28	141899
housing boston	506	13	139004
ionosphere	351	34	143057
lymphography	148	18	139969
messidor_features	1150	19	140162
openml_620	1000	25	141320
pima_indian	768	8	138039
spambase	4601	57	147496
spectf	267	44	144987
svmguide3	1243	21	140548
uci_credit_card	30000	25	141127
wine_red	999	12	138618
wine_white	4900	12	138618
openml_586	1000	25	141320
openml_589	1000	25	141320
openml_607	1000	50	146145
openml_616	500	50	146145
openml_618	1000	50	146145
openml_637	500	50	146145
as the encoder and decoder backbones and utilized 3-layer feed-forward networks to implement the predictor. The hidden state sizes of the encoder, decoder and predictor are 64, 64, and 200, respectively. The embedding size of each feature ID token and operation token was set to 32. To train MOAT, we set the batch size as 1024, the learning rate as 0.001, and  $\lambda$  as 0.95 respectively. For inferring new transformation sequences, we used top-20 records as the seeds with beam size 5.
B.2 Experimental Platform Information
All experiments were conducted on the Ubuntu 18.04.6 LTS operating system, AMD EPYC 7742 CPU, and 8 NVIDIA A100 GPUs, with the framework of Python 3.9.10 and PyTorch 1.8.1.
C Experimental Results
C.1 Space Complexity Analysis.
To analyze the space complexity of MOAT, we illustrate the parameter size of MOAT when confronted with different datasets. Table 4 shows the comparison results. We can find that the model size of MOAT keeps relatively stable without significant fluctuations. The underlying driver is that the encoder-evaluator-decoder learning paradigm can embed the knowledge of discrete sequences with variant lengths into a fixed-length embedding vector. Thus, such an embedding process can make the parameter size to be stable instead of increasing as the data size grows. Thus, this experiment indicates that MOAT has good scalability when confronted with different scaled datasets.
C.2 Scalability Check
We visualized the changing trend of the time cost of searching for better feature spaces over sample size and feature dimensions of different datasets. Figure 6 shows the comparison results. We found that the time cost of MOAT keeps stable with the increase of sample size of the feature set. A possible reason is that MOAT only focuses on the decision-making benefits of feature ID and operation tokens instead of the information of the entire feature set, making the searching process sample size irrelevant. Another interesting observation is that the search time is still stable although the feature dimension of the feature set varies significantly. A possible explanation is that we map transformation records of varying lengths into a continuous space with a constant length. The searching time in this space is input dimensionality-agnostic. Thus, this experiment shows the MOAT has excellent scalability.
(a) Sample Size
(b) Feature Number
Figure 6: Scalability check of MOAT in search time based on sample size and feature number.
(a) Search Step Size  $\eta$  (under  $\alpha = 0.05$ )
(b) Training Trade-off  $\alpha$  (under  $\eta = 1$ )
Figure 7: Parameter sensitivity on search step size  $\eta$  and trade-off parameter  $\alpha$  on Spectf dataset.
(a) Openml_616
Figure 8: The visualization of learned transformation sequence embedding (from MOAT).
(b) Airfoil
C.3 Parameter sensitivity analysis
To validate the parameter sensitivity of the search step size  $\eta$  (See section 3.5) and the trade-off parameter  $\alpha$  in the training loss (See section 3.4), we set the value of  $\eta$  from 1 to 10, and set the value of  $\alpha$  from 0.05 to 0.50 to observe the difference. Figure 7 shows the comparison results in terms of precision, recall, and F1-score. When the search step size grows, the downstream ML performance initially improves, then declines marginally. A possible reason for this observation is that a too-large step size may make the gradient-ascent search algorithm greatly vibrate in the continuous space, leading to missing the optimal embedding point and transformed feature space. Another interesting observation is that the standard deviation of the model performance is lower than 0.01 under different parameter settings. This observation indicates that MOAT is not sensitive to distinct parameter settings. Thus, the learning and searching process of MOAT is robust and stable.
C.4 Learned embedding Analysis.
We selected Airfoil and Openml_616 as examples to visualize their learned continuous embedding space. In detail, we first collected the latent embeddings generated by the transformation records. Then, we use T-SNE to map them into a 2-dimensional space for visualization. Figure 8 shows
(a) Original Feature Space
(b) MOAT Generated Feature Space
Figure 9: Comparison of traceability on the original feature space and the MOATgenerated one. the visualization results, in which each point represents a unique feature transformation sequence. The size of each point means its downstream performance. The bigger point size indicates that the downstream performance is superior. We colored the top 20 embedding points according to the performance in red. We found that the distribution locations of the top 20 embedding points are different. A potential reason is that the corresponding transformation sequences of the top 20 embedding points are different lengths. The sequence reconstruction loss distributes them to different areas of the embedding space. Moreover, we observed that the top 20 embedding points are close in the space even though the positions are different. The underlying driver is that the estimation loss makes these points with good performance clustered. Thus, this case study reflects that the reconstruction loss and estimation loss make the continuous space associate the transformation sequence and the corresponding model performance.
C.5 Traceability case study
We selected the top 10 essential features for prediction in the original, and MOAT transformed feature space of the Wine Quality Red dataset for comparison. Figure 9 shows the comparison results. The texts associated with each pie chart are the corresponding feature name. The larger the pie area is, the more critical the feature is. We found that almost  $70\%$  critical features in the new feature space are generated by MOAT and they improve the downstream ML performance by  $22.6\%$ . This observation indicates that MOAT really comprehends the properties of the feature set and ML models in order to produce a more effective feature space. Another interesting finding is that '[alcohol]' is the essential feature in the original feature set. But MOAT generates more mathematically composited features using '[alcohol]'. This observation reflects that MOAT not only can capture the significant features but also produce more effective knowledge for enhancing the model performance. Such composited features can make domain experts trace their ancestor resources and summarize new analysis rules for evaluating the quality of red wine.