SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Dongjie Wang*
Department of CS  
University of Central Florida  
dongjie.wang@ucf.edu
Meng Xiao*
CNIC, CAS University of CAS shaow@cnic.cn
Institute for Infocomm Research A*STAR wumin@i2r.a-star.edu.sg
Pengfei Wang
CNIC, CAS University of CAS wpf@cnic.cn
Yuanchun Zhou
CNIC, CAS University of CAS zyc@cnic.cn
Yanjie Fu†
School of Computing and AI  
Arizona State University  
yanjiefu@asu.edu
Feature transformation aims to generate new pattern-discriminative feature space from original features to improve downstream machine learning (ML) task performances. However, the discrete search space for the optimal feature explosively grows on the basis of combinations of features and operations from low-order forms to high-order forms. Existing methods, such as exhaustive search, expansion reduction, evolutionary algorithms, reinforcement learning, and iterative greedy, suffer from large search space. Overly emphasizing efficiency in algorithm design usually sacrifices stability or robustness. To fundamentally fill this gap, we reformulate discrete feature transformation as a continuous space optimization task and develop an embedding-optimization-reconstruction framework. This framework includes four steps: 1) reinforcement-enhanced data preparation, aiming to prepare high-quality transformation-accuracy training data; 2) feature transformation operation sequence embedding, intending to encapsulate the knowledge of prepared training data within a continuous space; 3) gradient-steered optimal embedding search, dedicating to uncover potentially superior embeddings within the learned space; 4) transformation operation sequence reconstruction, striving to reproduce the feature transformation solution to pinpoint the optimal feature space. Finally, extensive experiments and case studies are performed to demonstrate the effectiveness and robustness of the proposed method. The code and data are publicly accessible https://www.dropbox.com/sh/imh8ckui7va3k5u/AACulQegVx0MuywYyoCqSdVPa?dl=0.
Feature transformation aims to derive a new feature space by mathematically transforming the original features to enhance the downstream ML task performances. However, feature transformation is usually manual, time-consuming, labor-intensive, and requires domain knowledge. These limitations motivate us to accomplish Automated Feature Transformation (AFT). AFT is a fundamental task because AFT can 1) reconstruct distance measures, 2) form a feature space with discriminative patterns, 3) ease machine learning, and 4) overcome complex and imperfect data representation.
There are two main challenges in solving AFT: 1) efficient feature transformation in a massive discrete search space; 2) robust feature transformation in an open learning environment. Firstly, it
*These authors have contributed equally to this work. †Corresponding Author
is computationally costly to reconstruct the optimal feature space from a given feature set. Such reconstruction necessitates a transformation sequence that contains multiple combinations of features and operations. Each combination indicates a newly generated feature. Given the extensive array of features and operations, the quantity of possible feature-operation combinations exponentially grows, further compounded by the vast number of potential transformation operation sequences. The efficiency challenge seeks to answer: how can we efficiently identify the best feature transformation operation sequence? Secondly, identifying the best transformation operation sequence is unstable and sensitive to many factors in an open environment. For example, if we formulate a transformation operation sequence as a searching problem, it is sensitive to starting points or the greedy strategy during iterative searching. If we identify the same task as a generation problem, it is sensitive to training data quality and the complexity of generation forms. The robustness challenge aims to answer: how can we robustify the generation of feature transformation sequences?
Prior literature partially addresses the two challenges. Existing AFT algorithms can be grouped into three categories: 1) expansion-reduction approaches [1, 2, 3], in which all mathematical operations are randomly applied to all features at once to generate candidate transformed features, followed by feature selection to choose valuable features. However, such methods are based on random generation, unstable, and not optimization-directed. 2) iterative-feedback approaches [4, 5, 6], in which feature generation and selection are integrated, and learning strategies for each are updated based on feedback in each iteration. Two example methods are Evolutionary Algorithms (EA) or Reinforcement Learning (RL) with downstream ML task accuracy as feedback. However, such methods are developed based on searching in a massive discrete space and are difficult to converge in comparison to solving a continuous optimization problem. 3) Neural Architecture Search (NAS)-based approaches [7, 8]. NAS was originally to identify neural network architectures using a discrete search space containing neural architecture parameters. Inspired by NAS, some studies formulated AFT as a NAS problem. However, NAS-based formulations are slow and limited in modeling all transformation forms. Existing studies show the inability to jointly address efficiency and robustness in feature transformation. Thus, we need a novel perspective to derive a novel formulation for AFT.
Our Contribution: A Postfix Expression Embedding and Generation Perspective. To fundamentally fill these gaps, we formulate the discrete AFT problem as a continuous optimization task and propose a reinforcement-enhanced autoregressive feAture Transformation framework, namely MOAT. To advance efficiency and robustness, this framework implements four steps: 1) reinforcement-enhanced training data preparation; 2) feature transformation operation sequence embedding; 3) gradient-steered optimal embedding search; 4) beam search-based transformation operation sequence reconstruction. Step 1 is to collect high-quality transformation operation sequence-accuracy pairs as training data. Specifically, we develop a cascading reinforcement learning structure to automatically explore transformation operation sequences and test generated feature spaces on a downstream predictor (e.g., decision tree). The self-optimizing policies enable agents to collect high-quality transformation operation sequences. The key insight is that when training data is difficult or cost expensive to collect, reinforcement intelligence can be used as an automated training data collector. Step 2 is to learn a continuous embedding space from transformation-accuracy training data. Specifically, we describe transformation operation sequences as postfix expressions, each of which is mapped into an embedding vector by jointly optimizing the transformation operation sequence reconstruction loss and accuracy estimation loss. Viewing feature transformation through the lens of postfix expressions offers the chance to mitigate the challenges associated with an exponentially expanding discrete search problem. By recasting the feature transformation operation sequence in postfix form, the search space becomes smaller, as each generation step involves selecting a single alphabetical letter pertaining to a feature or mathematical operation, rather than considering high-order expansions. Moreover, this postfix form captures feature-feature interaction information and empowers the generation model with the capacity to autonomously determine the optimal number and segmentation way of generated features. Step 3 is to leverage the gradient calculated from the improvement of the accuracy evaluator to guide the search for the optimal transformation operation sequence embedding. Step 4 is to develop a beam search-based generative module to reconstruct feature transformation operation sequences from embedding vectors. Each sequence is used to obtain a transformed feature space, and subsequently, the transformed space that yields the highest performance in the downstream ML task is identified as the optimal solution. Finally, we present extensive experiments and case studies to show the effectiveness and superiority of our framework.
2 Definitions and Problem Statement
2.1 Important Definitions
Operation Set. To refine the feature space for improving the downstream ML models, we need to apply mathematical operations to existing features to generate new informative features. All operations are collected in an operation set, denoted by  $\mathcal{O}$ . Based on the computation property, these operations can be classified as unary operations and binary operations. The unary operations such as "square", "exp", "log", etc. The binary operations such as "plus", "multiply", "minus", etc.
Cascading Agent Structure. We create a cascading agent structure comprised of head feature agent, operation agent, and tail feature agent to efficiently collect quantities of high-quality feature transformation records. The selection process of the three agents will share the state information and sequentially select candidate features and operations for refining the feature space.
Feature Transformation Operation Sequence. Assuming that  $D = \{X, y\}$  is a dataset, which includes the original feature set  $X = [f_1, \dots, f_N]$  and predictive targets  $y$ .
As shown in Figure 1, we transform the existing ones using mathematical compositions  $\tau$  consisting of feature ID tokens and operations to generate new and informative features.  $K$  transformation compositions are adopted to refine  $X$  to a better feature space  $\tilde{X} = [\tilde{f}_1,\dots ,\tilde{f}_K]$ . The collection of the  $K$  compositions refers to the feature transformation sequence, which is denoted by  $\Gamma = [\tau_{1},\dots ,\tau_{K}]$
$$
\tau_ {1}: \left(f _ {1} + \left(\frac {\left(f _ {1} + f _ {2}\right)}{f _ {3}}\right) - f _ {2}\right), \tau_ {2}: \left(f _ {1} - \left(\sqrt {f _ {3}}\right)\right), \dots , \tau_ {K}: \left(\left(f _ {2}\right) ^ {2}\right)
$$
Figure 1: An example of feature transformation sequence:  $\tau_{(.)}$  indicates the generated feature that is the combination of original features and mathematical operations.
We aim to develop an effective and robust deep differentiable automated feature transformation framework. Formally, given a dataset  $D = \{X,y\}$  and an operation set  $\mathcal{O}$ , we first build a cascading RL-agent structure to collect  $n$  feature transformation accuracy pairs as training data, denoted by  $R = \{(\Gamma_i,v_i)\}_{i=1}^n$ , where  $\Gamma_i$  is the transformation sequence and  $v_i$  is the associated downstream predictive performance. We pursue two objectives thereafter: 1) building an optimal continuous embedding space for feature transformation sequences. We learn a mapping function  $\phi$ , a reconstructing function  $\psi$ , and an evaluation function  $\omega$  to convert  $R$  into a continuous embedding space  $\mathcal{E}$  via joint optimization. In  $\mathcal{E}$ , each embedding point is associated with a feature transformation sequence and corresponding predictive performance. 2) identifying the optimal feature space. We adopt a gradient-based search to find the optimal feature transformation sequence  $\Gamma^*$ , given by:
$$
\Gamma^ {*} = \psi (\mathbf {E} ^ {*}) = \operatorname {a r g m a x} _ {\mathbf {E} \in \varepsilon} \mathcal {A} (\mathcal {M} (\psi (\mathbf {E}) (X)), y), \tag {1}
$$
where  $\psi$  can reconstruct a feature transformation sequence from any embedding point of  $\mathcal{E}$ ;  $\mathbf{E}$  is an embedding vector in  $\mathcal{E}$  and  $\mathbf{E}^*$  is the optimal one;  $\mathcal{M}$  is the downstream ML model and  $\mathcal{A}$  is the performance indicator. Finally, we apply  $\Gamma^*$  to transform  $X$  to the optimal feature space  $X^*$  maximizing the value of  $\mathcal{A}$ .
Figure 2 shows the framework of MOAT including four steps: 1) reinforcement-enhanced transformation-accuracy data preparation; 2) deep postfix feature transformation operation sequence embedding; 3) gradient-ascent optimal embedding search; 4) transformation operation sequence reconstruction. In Step 1, a cascading agent structure consisting of two feature agents and one operation agent is developed to select candidate features and operators for feature crossing. The transformed feature sets are applied to a downstream ML task to collect the corresponding accuracy. The data collection process is automated and self-optimized by policies and feedback in reinforcement learning. We then convert these feature transformation operation sequences into postfix expressions. In Step 2, we develop an encoder-evaluator-decoder model to embed transformation operation sequence-accuracy pairs into a continuous embedding space by jointly optimizing the sequence reconstruction loss and performance evaluation loss. In detail, the encoder maps these transformation operation
sequences into continuous embedding vectors; the evaluator assesses these embeddings by predicting their corresponding model performance; the decoder reconstructs the transformation sequence using these embeddings. In Step 3, we first learn the embeddings of top-ranking transformation operation sequences by the well-trained encoder. With these embeddings as starting points, we search along the gradient induced by the evaluator to find the acceptable optimal embeddings with better model performances. In Step 4, the well-trained decoder then decodes these optimal embeddings to generate candidate feature transformation operation sequences through the beam search. We apply the feature transformation operation sequences to original features to reconstruct refined feature spaces and evaluate the corresponding performances of the downstream predictive ML task. Finally, the feature space with the highest performance is chosen as the optimal one.
Figure 2: An overview of our framework. MOAT consists of four main components: 1) transformation-accuracy data preparation; 2) deep feature transformation embedding; 3) gradient-ascent optimal embedding search; 4) transformation sequence reconstruction and evaluation.
3.2 Reinforcement Training Data Preparation
Why Using Reinforcement as Training Data Collector. Our extensive experimental analysis shows that the quality of embedding space directly determines the success of feature transformation operation sequence construction. The quality of the embedding space is sensitive to the quality and scale of transformation sequence-accuracy training data: training data is large enough to represent the entire distribution; training data include high-performance feature transformation cases, along with certain random exploratory samples. Intuitively, we can use random sample features and operations to generate feature transformation sequences. This strategy is inefficient because it produces many invalid and low-quality samples. Or, we can use existing feature transformation methods (e.g., AutoFeat [2]) to generate corresponding records. However, these methods are not fully automated and produce a limited number of high-quality transformation records without exploration ability. We propose to view reinforcement learning as a training data collector to overcome these limitations.
Reinforcement Transformation-Accuracy Training Data Collection. Inspired by [4, 9], we formulate feature transformation as three interdependent Markov decision processes (MDPs). We develop a cascading agent structure to implement the three MDPs. The cascading agent structure consists of a head feature agent, an operation agent, and a tail feature agent. In each iteration, the three agents collaborate to select two candidate features and one operation to generate a new feature. Feedback-based policy learning is used to optimize the exploratory data collection to find diversified yet quality feature transformation samples. To simplify the description, we adopt the  $i$ -th iteration as an example to illustrate the reinforcement data collector. Given the former feature space as  $X_{i}$ , we generate new features  $X_{i+1}$  using the head feature  $f_{h}$ , operation  $o$ , and tail feature  $f_{t}$  selected by the cascading agent structure.
1) Head feature agent. This learning system includes: State: is the vectorized representation of  $X_{i}$ . Let  $Rep(\cdot)$  be a state representation method, and the state can be denoted by  $Rep(X_{i})$ . Action: is the head feature  $f_{h}$  selected from  $X_{i}$  by the reinforced agent.
2) Operation agent. This learning system includes: State: includes the representation of  $X_{i}$  and the head feature, denoted by  $\operatorname{Rep}(X_i) \oplus \operatorname{Rep}(f_h)$ , where  $\oplus$  indicates concatenation. Action: is the operation  $o$  selected from the operation set  $\mathcal{O}$ .
3) Tail feature agent. This learning system includes: State: includes the representation of  $X_{i}$ , selected head feature, and operation, denoted by  $\operatorname{Rep}(X_i) \oplus \operatorname{Rep}(f_h) \oplus \operatorname{Rep}(o)$ . Action: is the tail feature  $f_{t}$  selected from  $X_{i}$  by this agent.
4) State representation method,  $Rep(\cdot)$ . For the representation of the feature set, we employ a descriptive statistical technique to obtain the state with a fixed length. In detail, we first compute the descriptive statistics (i.e. count, standard deviation, minimum, maximum, first, second, and third quantile) of the feature set column-wise. Then, we calculate the same descriptive statistics on the output of the previous step. After that, we can obtain the descriptive matrix with shape  $\mathbb{R}^{7 \times 7}$  and flatten it as the state representation with shape  $\mathbb{R}^{1 \times 49}$ . For the representation of the operation, we adopt its one-hot encoding as  $Rep(o)$ .
5) Reward function. To improve the quality of the feature space, we use the improvement of a downstream ML task performance as the reward. Thus, it can be defined as:  $\mathcal{R}(X_i,X_{i + 1}) = \mathcal{A}(\mathcal{M}(X_{i + 1}),y) - \mathcal{A}(\mathcal{M}(X_i),y)$ .
6) Learning to Collect Training Data. To optimize the entire procedure, we minimize the mean squared error of the Bellman Equation to get a better feature space. During the exploration process, we can collect amounts of high-quality records  $(\Gamma, v)$  for constructing an effective continuous embedding space, where  $\Gamma$  is the transformation sequence, and  $v$  is the downstream model performance.
3.3 Postfix Expressions of Feature Transformation Operation Sequences
Why Transformation Operation Sequences as Postfix Expressions. After training data collection, a question arises: how can we organize and represent these transformation operation sequences in a computationally-tangible and machine-learnable format?
Figure 3 (a) shows an example of a feature transformation operation sequence with two generated features. To convert this sequence into a machine-readable expression, a naive idea is to enclose each calculation in a pair of brackets to indicate its priority (Figure 3(b)). But, the representation of Figure 3(b) has four limitations: (1) Redundancy. Many priority-related brackets are included to ensure the unambiguous property and correctness of mathematical calculations. (2) Semantic Sparsity. The quantity of bracket tokens can dilute the semantic information of the sequence, making model convergence diffi
$$
\begin{array}{c} \tau_ {1} \\ (f _ {1} + (f _ {1} / f _ {3}) - f _ {2}), \overline {{(\sqrt {f _ {1}})}} \end{array}
$$
(a) Feature transformation Sequence
$$
\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left. \right.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\left.\right.\right.\right.\right.\right.\right.\right.\right.\right.\right.\right.\right.\right.\right.\right.\right.\right.\right.\right.\right.\right.\right.\right.\right.\right.\right.\right\rangle_ {0} ^ {\infty} f _ {2} + [ ] [ ] [ f _ {1} ] [ f _ {3} ] - f _ {2} ] [ ] S E F [ ] f _ {1} ] e s
$$
(b) Original infix notation based sequence
$$
\begin{array}{c c c c c c c c} \text {S O S} & f _ {2} & f _ {1} & f _ {3} & 7 & f _ {3} & - + & \text {S E R} & f _ {2} & \sqrt {7} & \text {E O S} \end{array}
$$
(c) Postfix notation based sequence
Figure 3: Compared to infix-based expressions, postfix-based expressions optimize token redundancy, enhance semantics, prevent illegal transformations, and minimize the search space.
cult. (3) Illegal Transformation. If the decoder makes one mistake on bracket generation, the entire generated sequence will be wrong. (4) Large Search Space. The number of combinations of features and operations from low-order to high-order interactions is large, making the search space too vast.
Using Postfix Expression to Construct Robust, Concise, and Unambiguous Sequences. To address the aforementioned limitations, we convert the transformation operation sequence  $\Gamma$  into a postfix-based sequence expression. Specifically, we scan each mathematical composition  $\tau$  in  $\Gamma$  from left to right and convert it from the infix-based format to the postfix-based one. We then concatenate each postfix expression by the <SEP> token and add the <SOS> and <EOS> tokens to the beginning and end of the entire sequence. Figure 3(c) shows an example of such a postfix sequence. We denote it as  $\Upsilon = [\gamma_{1},\dots ,\gamma_{M}]$ , where each element is a feature ID token, operation token, or three other unique tokens. The detailed pseudo code of this conversion process is provided in Appendix A.
The postfix sequences don't require numerous brackets to ensure the calculation priority. We only need to scan each element in the sequence from left to right to reconstruct corresponding transformed features. Such a concise and short expression can reduce sequential modeling difficulties and computational costs. Besides, a postfix sequence indicates a unique transformation process, thus, reducing the ambiguity of the feature transformation sequence. Moreover, the most crucial aspect is the reduction of the search space from exponentially growing discrete combinations to a limited token set  $\mathcal{C}$  that consists of the original feature ID tokens, operation tokens, and other three unique tokens. The length of the token set is  $|\mathcal{O}| + |X| + 3$ , where  $|\mathcal{O}|$  is the number of the operation set,  $|X|$  is the dimension of the original feature set, and 3 refers to the unique tokens  $\langle \mathrm{SOS}\rangle$ ,  $\langle \mathrm{SEP}\rangle$ ,  $\langle \mathrm{EOS}\rangle$ .
Data Augmentation for Postfix Transformation Sequences. Big and diversified transformation sequence records can benefit the learning of a pattern discriminative embedding space. Be sure to notice that, a feature transformation sequence consists of many independent segmentations, each of which is the composition of feature ID and operation tokens and can be used to generate a new feature. These independent segmentations are order-agnostic. Our idea is to leverage this property to conduct data augmentation to increase the data volume and diversity. For example, given a transformation operation sequence and corresponding accuracy  $\{\Upsilon, v\}$ , we first divide the postfix expression into different segmentations by  $<\mathrm{SEP}>$ . We then randomly shuffle these segmentations and use  $<\mathrm{SEP}>$  to concatenate them together to generate new postfix transformation sequences. After that, we pair the new sequences with the corresponding model accuracy performance to improve data diversity and data volume for better model training and to create the continuous embedding space.
3.4 Deep Feature Transformation Embedding
After collecting and converting large-scale feature transformation training data to a set of postfix expression-accuracy pairs  $\{(\Upsilon_i,v_i)\}_{i = 1}^n$ , we develop an encoder-evaluator-decoder structure to map the sequential information of these records into an embedding space. Each embedding vector is associated with a transformation operation sequence and its corresponding model accuracy.