Title: Reinforcement-Enhanced Autoregressive Feature Transformation: Gradient-steered Search in Continuous Space for Postfix Expressions
Venue: NeurIPS 2023
Accepted: yes
Keywords: Feature Transformation

Autoregressive Generation

Continuous Space Optimization
Abstract: Feature transformation aims to generate new pattern-discriminative feature space from original features to improve downstream machine learning  (ML) task performances. However, the discrete search space for the optimal feature explosively grows on the basis of combinations of features and operations from low-order forms to high-order forms. Existing methods, such as exhaustive search, expansion reduction, evolutionary algorithms, reinforcement learning, and iterative greedy, suffer from large search space. Overly emphasizing efficiency in algorithm design usually sacrifice stability or robustness. To fundamentally fill this gap, we reformulate discrete feature transformation as a continuous space optimization task and develop an embedding-optimization-reconstruction framework. This framework includes four steps: 1) reinforcement-enhanced data preparation, aiming to prepare high-quality transformation-accuracy training data; 2) feature transformation operation sequence embedding, intending to encapsulate the knowledge of prepared training data within a continuous space; 3) gradient-steered optimal embedding search, dedicating to uncover potentially superior embeddings within the learned space; 4) transformation operation sequence reconstruction, striving to reproduce the feature transformation solution to pinpoint the optimal feature space. Finally, extensive experiments and case studies are performed to demonstrate the effectiveness and robustness of the proposed method. The code and data are publicly accessible https://www.dropbox.com/sh/imh8ckui7va3k5u/AACulQegVx0MuywYyoCqSdVPa?dl=0.

Reinforcement-Enhanced Autoregressive Feature Transformation: Gradient-steered Search in Continuous Space for Postfix Expressions

Abstract

Feature transformation aims to generate new pattern-discriminative feature space from original features to improve downstream machine learning (ML) task performances. However, the discrete search space for the optimal feature explosively grows on the basis of combinations of features and operations from low-order forms to high-order forms. Existing methods, such as exhaustive search, expansion reduction, evolutionary algorithms, reinforcement learning, and iterative greedy, suffer from large search space. Overly emphasizing efficiency in algorithm design usually sacrifices stability or robustness. To fundamentally fill this gap, we reformulate discrete feature transformation as a continuous space optimization task and develop an embedding-optimization-reconstruction framework. This framework includes four steps: 1) reinforcement-enhanced data preparation, aiming to prepare high-quality transformation-accuracy training data; 2) feature transformation operation sequence embedding, intending to encapsulate the knowledge of prepared training data within a continuous space; 3) gradient-steered optimal embedding search, dedicating to uncover potentially superior embeddings within the learned space; 4) transformation operation sequence reconstruction, striving to reproduce the feature transformation solution to pinpoint the optimal feature space. Finally, extensive experiments and case studies are performed to demonstrate the effectiveness and robustness of the proposed method. The code and data are publicly accessible https://www.dropbox.com/sh/imh8ckui7va3k5u/AACulQegVx0MuywYyoCqSdVPa?dl=0 .

1 Introduction

Feature transformation aims to derive a new feature space by mathematically transforming the original features to enhance the downstream ML task performances. However, feature transformation is usually manual, time-consuming, labor-intensive, and requires domain knowledge. These limitations motivate us to accomplish A utomated F eature T ransformation ( AFT ). AFT is a fundamental task because AFT can 1) reconstruct distance measures, 2) form a feature space with discriminative patterns, 3) ease machine learning, and 4) overcome complex and imperfect data representation.

There are two main challenges in solving AFT: 1) efficient feature transformation in a massive discrete search space; 2) robust feature transformation in an open learning environment. Firstly, it

Min Wu

Yanjie Fu †

is computationally costly to reconstruct the optimal feature space from a given feature set. Such reconstruction necessitates a transformation sequence that contains multiple combinations of features and operations. Each combination indicates a newly generated feature. Given the extensive array of features and operations, the quantity of possible feature-operation combinations exponentially grows, further compounded by the vast number of potential transformation operation sequences. The efficiency challenge seeks to answer: how can we efficiently identify the best feature transformation operation sequence? Secondly, identifying the best transformation operation sequence is unstable and sensitive to many factors in an open environment. For example, if we formulate a transformation operation sequence as a searching problem, it is sensitive to starting points or the greedy strategy during iterative searching. If we identify the same task as a generation problem, it is sensitive to training data quality and the complexity of generation forms. The robustness challenge aims to answer: how can we robustify the generation of feature transformation sequences?

Prior literature partially addresses the two challenges. Existing AFT algorithms can be grouped into three categories: 1) expansion-reduction approaches [1, 2, 3], in which all mathematical operations are randomly applied to all features at once to generate candidate transformed features, followed by feature selection to choose valuable features. However, such methods are based on random generation, unstable, and not optimization-directed. 2) iterative-feedback approaches [4, 5, 6], in which feature generation and selection are integrated, and learning strategies for each are updated based on feedback in each iteration. Two example methods are Evolutionary Algorithms (EA) or Reinforcement Learning (RL) with downstream ML task accuracy as feedback. However, such methods are developed based on searching in a massive discrete space and are difficult to converge in comparison to solving a continuous optimization problem. 3) Neural Architecture Search (NAS)based approaches [7, 8]. NAS was originally to identify neural network architectures using a discrete search space containing neural architecture parameters. Inspired by NAS, some studies formulated AFT as a NAS problem. However, NAS-based formulations are slow and limited in modeling all transformation forms. Existing studies show the inability to jointly address efficiency and robustness in feature transformation. Thus, we need a novel perspective to derive a novel formulation for AFT.

Our Contribution: A Postfix Expression Embedding and Generation Perspective. To fundamentally fill these gaps, we formulate the discrete AFT problem as a continuous optimization task and propose a reinforce M ent-enhanced aut O regressive fe A ture T ransformation framework, namely MOAT . To advance efficiency and robustness, this framework implements four steps: 1) reinforcement-enhanced training data preparation; 2) feature transformation operation sequence embedding; 3) gradient-steered optimal embedding search; 4) beam search-based transformation operation sequence reconstruction. Step 1 is to collect high-quality transformation operation sequenceaccuracy pairs as training data. Specifically, we develop a cascading reinforcement learning structure to automatically explore transformation operation sequences and test generated feature spaces on a downstream predictor (e.g., decision tree). The self-optimizing policies enable agents to collect high-quality transformation operation sequences. The key insight is that when training data is difficult or cost expensive to collect, reinforcement intelligence can be used as an automated training data collector. Step 2 is to learn a continuous embedding space from transformation-accuracy training data. Specifically, we describe transformation operation sequences as postfix expressions, each of which is mapped into an embedding vector by jointly optimizing the transformation operation sequence reconstruction loss and accuracy estimation loss. Viewing feature transformation through the lens of postfix expressions offers the chance to mitigate the challenges associated with an exponentially expanding discrete search problem. By recasting the feature transformation operation sequence in postfix form, the search space becomes smaller, as each generation step involves selecting a single alphabetical letter pertaining to a feature or mathematical operation, rather than considering high-order expansions. Moreover, this postfix form captures feature-feature interaction information and empowers the generation model with the capacity to autonomously determine the optimal number and segmentation way of generated features. Step 3 is to leverage the gradient calculated from the improvement of the accuracy evaluator to guide the search for the optimal transformation operation sequence embedding. Step 4 is to develop a beam search-based generative module to reconstruct feature transformation operation sequences from embedding vectors. Each sequence is used to obtain a transformed feature space, and subsequently, the transformed space that yields the highest performance in the downstream ML task is identified as the optimal solution. Finally, we present extensive experiments and case studies to show the effectiveness and superiority of our framework.

2 Definitions and Problem Statement

2.1 Important Definitions

Operation Set. To refine the feature space for improving the downstream ML models, we need to apply mathematical operations to existing features to generate new informative features. All operations are collected in an operation set, denoted by O . Based on the computation property, these operations can be classified as unary operations and binary operations. The unary operations such as "square", "exp", "log", etc. The binary operations such as "plus", "multiply", "minus", etc.

Cascading Agent Structure. We create a cascading agent structure comprised of head feature agent , operation agent , and tail feature agent to efficiently collect quantities of high-quality feature transformation records. The selection process of the three agents will share the state information and sequentially select candidate features and operations for refining the feature space.

Feature

Transformation

Operation Sequence.

which includes the original feature set X As shown in Figure 1, we transform the existing ones using mathematical compositions τ consisting of feature ID tokens and operations to generate new and informative features. K transformation compositions are adopted to refine X to a better feature space ˜ X = [ ˜ f 1 , · · · , ˜ f K ] . The collection of the K compositions refers to the feature transformation sequence, which is denoted by Γ = [ τ 1 , · · · , τ K ] .

2.2 Problem Statement

We aim to develop an effective and robust deep differentiable automated feature transformation framework. Formally, given a dataset D = { X,y } and an operation set O , we first build a cascading RL-agent structure to collect n feature transformation accuracy pairs as training data, denoted by R = { (Γ i , v i ) } n i =1 , where Γ i is the transformation sequence and v i is the associated downstream predictive performance. We pursue two objectives thereafter: 1) building an optimal continuous embedding space for feature transformation sequences. We learn a mapping function ϕ , a reconstructing function ψ , and an evaluation function ω to convert R into a continuous embedding space E via joint optimization. In E , each embedding point is associated with a feature transformation sequence and corresponding predictive performance. 2) identifying the optimal feature space. We adopt a gradient-based search to find the optimal feature transformation sequence Γ ∗ , given by:

where ψ can reconstruct a feature transformation sequence from any embedding point of E ; E is an embedding vector in E and E ∗ is the optimal one; M is the downstream ML model and A is the performance indicator. Finally, we apply Γ ∗ to transform X to the optimal feature space X ∗ maximizing the value of A .

3 Methodology

3.1 Framework Overview

Figure 2 shows the framework of MOAT including four steps: 1) reinforcement-enhanced transformation-accuracy data preparation; 2) deep postfix feature transformation operation sequence embedding; 3) gradient-ascent optimal embedding search; 4) transformation operation sequence reconstruction. In Step 1, a cascading agent structure consisting of two feature agents and one operation agent is developed to select candidate features and operators for feature crossing. The transformed feature sets are applied to a downstream ML task to collect the corresponding accuracy. The data collection process is automated and self-optimized by policies and feedback in reinforcement learning. We then convert these feature transformation operation sequences into postfix expressions. In Step 2, we develop an encoder-evaluator-decoder model to embed transformation operation sequenceaccuracy pairs into a continuous embedding space by jointly optimizing the sequence reconstruction loss and performance evaluation loss. In detail, the encoder maps these transformation operation

Assuming that D = { X,y } is a dataset, = [ f 1 , · · · , f N ] and predictive targets y .

<latexi sh

1_b

64="m nvPCZ0

5L+dDSr23G

JE

>A

B7

c

V

gN

Oy fUY9

o

K

zwF

X

M

Q

p

q

j

H

8

/

R

k

W

T

u

I

<latexi sh

1_b

64="f7R

V

B/roz

N2

yI

KE

p

>A

C

H

c

DLSgM

F

3U

v

Zdu

k

W

9

GTJm

5

wn+

X

O

q

8

Y

0

P

<latexi sh

1_b

64="q

Y

0Hu3NprdD

8J

KTSfO5wWz

>A

B

c

V

L

g

E

y

U

9

oP

Q

+

GvX

M

C

F2

mZ

/

R

7

I

k

n

j

<latexi sh

1_b

64="w

U

jX

RL

+

D

E

Y

387

r

>A

B

n

c

V

SgN

Oz

G

M

H

d

C09

m

k

5

Z

P

I

v

qK

u

J

o

y

W

F

p

T

2

Q

/

<latexi sh

1_b

64="

2

Vm3fv ygKB

XqFUL

C0j

>A

/

c

DJS

N

EO

W

z

u

8

Y

R

I9

p

7

Q

H

wd

Po

k

r

+

G5

Z

M

T

n

<latexi sh

1_b

64="A

qBk3vX

PZI

8V

CMz fO

>

7

c

DLSgN

E

y

r

UY9

o

Rd wT

W

G

H

mn pK

J

/+j

0

u

2

5

Q

F

<latexi sh

1_b

64="Wn qpN

O

k

+/A

E

>

B7

c

VD

z

G

H

Y

3

P

Z

m

9

y

CUf

I

5

0

d

R

w

J

X

u

QK

o

v

<latexi sh

1_b

64="S9Nd kIqpB

j

mn7AZcgv

QY

>

8H

VDLT

J

EOz

f

CPX

ry

R

0

M5GF

3

/

u

W

+

2

w

U

o

K

Figure 1: An example of feature transformation sequence: τ ( . ) indicates the generated feature that is the combination of original features and mathematical operations.

sequences into continuous embedding vectors; the evaluator assesses these embeddings by predicting their corresponding model performance; the decoder reconstructs the transformation sequence using these embeddings. In Step 3, we first learn the embeddings of top-ranking transformation operation sequences by the well-trained encoder. With these embeddings as starting points, we search along the gradient induced by the evaluator to find the acceptable optimal embeddings with better model performances. In Step 4, the well-trained decoder then decodes these optimal embeddings to generate candidate feature transformation operation sequences through the beam search. We apply the feature transformation operation sequences to original features to reconstruct refined feature spaces and evaluate the corresponding performances of the downstream predictive ML task. Finally, the feature space with the highest performance is chosen as the optimal one.

…

Feature

Feature

Feature

Feature

Feature

Feature

Original

Feature Set

…

Feature

Transformation-accuracy Data Preparation

Deep Feature  Transformation Embedding

Gradient-ascent Optimal Embedding Search

Evaluator

Decoder

Evaluator

Optimal Feature Space

L

rec

L

est

Embeddings

Generated

Embeddings

Encoder

Encoder

…

Top-

T

Records

As Seeds

Transformation Sequence

Reconstruction and Evaluation

Operation Set

Unary:

…

E

…

+

,

×

,

-

, etc.

Binary:

log ()

, etc.

Decoder with Beam Search

Step

f

1

f

2

f

3

f

4

f

5

f

6

f

N

…

Feature

Transformation

Records

√

()

,

exp()

,

,

Reinforcement Learning

Based Data Collector

Head Feature Agent

Tail Feature Agent

Operation Agent

State:

Action:

Policy:

State:

Action:

Policy:

State:

Action:

Policy:

Explore

Exploit

Explore

Exploit

Explore

Exploit

f

h

o

f

t

Record:

Rep

(

X

i

)

Rep

(

X

i

)

⊕

Rep

(

f

h

)

Rep

(

X

i

)

⊕

Rep

(

f

h

)

⊕

Rep

(

o

)

Γ

, v

(

f

1

+

f

2

)

,

√

f

4

,

· · ·

(

f

4

)

, sin

(

f

7

)

,

· · ·

(

f

2

×

f

5

)

,

· · ·

Seq:

Seq:

Seq:

Seq:

Eval: 0.76

Eval: 0.74

Eval: 0.69

Eval: 0.72

exp

(

f

3

)

,

(

· · ·

)

,

· · ·

Postfix

Notation Based

Records

Seq:

Seq:

Seq:

Seq:

Eval: 0.76

Eval: 0.74

Eval: 0.69

Eval: 0.72

…

…

…

…

…

…

…

0.76

Gradient

Directions

Embeddings

Remove

Invalid

Form

Transformation

Sequence

[

∅

]

SEP

EOS

<SEP>

…

<EOS>

…

…

…

…

…

1

2

k

…

˜

f

1

,

˜

f

2

,

˜

f

3

,

˜

f

4

, . . .

˜

f

2

,

˜

f

4

,

˜

f

6

,

˜

f

10

, . . .

˜

f

1

,

˜

f

2

,

˜

f

4

,

˜

f

5

, . . .

˜

X

k

˜

X

1

˜

X

2

Transformed Features

˜ Γ

1

˜ Γ

2

˜ Γ

k

…

Transform Feature Space

X

Refined Search Space

Step Size:

η

f

∗

1

, f

∗

2

, f

∗

3

, f

∗

4

, . . .

X

∗

f

1

f

N

+

…

SEP

EOS

…

f

1

f

N

+

…

SEP

EOS

…

[

∅

]

[

f

1

]

[

f

3

]

f

1

f

N

+

…

SEP

EOS

…

[

f

1

, f

1

]

[

f

1

, f

3

]

f

1

f

N

+

…

SEP

EOS

…

[

f

1

, f

3

,

-

]

[

f

1

, f

1

,

+]

[

τ

1

]

[

· · ·

]

Beam 1

Beam 2

★

-----

3.2 Reinforcement Training Data Preparation

Why Using Reinforcement as Training Data Collector. Our extensive experimental analysis shows that the quality of embedding space directly determines the success of feature transformation operation sequence construction. The quality of the embedding space is sensitive to the quality and scale of transformation sequence-accuracy training data: training data is large enough to represent the entire distribution; training data include high-performance feature transformation cases, along with certain random exploratory samples. Intuitively, we can use random sample features and operations to generate feature transformation sequences. This strategy is inefficient because it produces many invalid and low-quality samples. Or, we can use existing feature transformation methods (e.g., AutoFeat [2]) to generate corresponding records. However, these methods are not fully automated and produce a limited number of high-quality transformation records without exploration ability. We propose to view reinforcement learning as a training data collector to overcome these limitations.

Reinforcement Transformation-Accuracy Training Data Collection. Inspired by [4, 9], we formulate feature transformation as three interdependent Markov decision processes (MDPs). We develop a cascading agent structure to implement the three MDPs. The cascading agent structure consists of a head feature agent, an operation agent, and a tail feature agent. In each iteration, the three agents collaborate to select two candidate features and one operation to generate a new feature. Feedback-based policy learning is used to optimize the exploratory data collection to find diversified yet quality feature transformation samples. To simplify the description, we adopt the i -th iteration as an example to illustrate the reinforcement data collector. Given the former feature space as X i , we generate new features X i +1 using the head feature f h , operation o , and tail feature f t selected by the cascading agent structure.

Head feature agent. This learning system includes: State: is the vectorized representation of X i . Let Rep ( · ) be a state representation method, and the state can be denoted by Rep ( X i ) . Action: is the head feature f h selected from X i by the reinforced agent.

Operation agent. This learning system includes: State: includes the representation of X i and the head feature, denoted by Rep ( X i ) ⊕ Rep ( f h ) , where ⊕ indicates concatenation. Action: is the operation o selected from the operation set O .

<latexi sh

1_b

64="5wfEkg3

cW

U

PuTFvH/

j

q

>A

CBX

ZDNS

IV

K

LoJ

9

0M

z

r

d

R

O7n

p

2

8

y

+

m

Q

G

Y

<latexi sh

1_b

64="G

d

+HpU

TR

zW

AF

j

E

>

CBX

c

ZDNSgM

I

/

q

LoJ

K

Q

m

On

3

25

u9

Y

V

k

y

0

7

f

r

w

8

<latexi sh

1_b

64="G

d

+HpU

TR

zW

AF

j

E

>

CBX

c

ZDNSgM

I

/

q

LoJ

K

Q

m

On

3

25

u9

Y

V

k

y

0

7

f

r

w

8

<latexi sh

1_b

64="

gTnoZ

y

z

f/W9

S

Ov

8CuM

>A

BX

c

DL

N

FI

P

q3

UKG

0

25

Vr

E

k

p

j

Y+

w

d

m

Q

J

R

H

7

<latexi sh

1_b

64="

UrXqC29

0

N

+cR

o

>A

B

H

<latexi sh

1_b

64="

UrXqC29

0

N

+cR

o

>A

B

H

<latexi sh

1_b

64="

UrXqC29

0

N

+cR

o

>A

B

H

<latexi sh

1_b

64="

UrXqC29

0

N

+cR

o

>A

B

H

V

SwM

EJ

7

T

Q

d

j

D

k

m

3W

F

K

P

f

LZ

/

p

y

5

zg8

I

n

O

u

G

V

SwM

EJ

7

T

Q

d

j

D

k

m

3W

F

K

P

f

LZ

/

p

y

5

zg8

I

n

O

u

G

V

SwM

EJ

7

T

Q

d

j

D

k

m

3W

F

K

P

f

LZ

/

p

y

5

zg8

I

n

O

u

G

V

SwM

EJ

7

T

Q

d

j

D

k

m

3W

F

K

P

f

LZ

/

p

y

5

zg8

I

n

O

u

G

<latexi sh

1_b

64="Sq

EMzCf

LP

F

G

TY

7+A

>

B

H

c

V

N

8

3Ur o/

vX

Z

Igj

k

u

d

m

Q

J

w

K

Wn

0

p

D

yR

O

2

9

5

<latexi sh

1_b

64="Sq

EMzCf

LP

F

G

TY

7+A

>

B

H

c

V

N

8

3Ur o/

vX

Z

Igj

k

u

d

m

Q

J

w

K

Wn

0

p

D

yR

O

2

9

5

<latexi sh

1_b

64="Sq

EMzCf

LP

F

G

TY

7+A

>

B

H

c

V

N

8

3Ur o/

vX

Z

Igj

k

u

d

m

Q

J

w

K

Wn

0

p

D

yR

O

2

9

5

<latexi sh

1_b

64="

P+

Z

q

Q

vEKMXyw

>A

B

c

DNSg

FI

8

V

uCoz

n

p

H

O5R

k

m

0

9

J

W

/

j

7

T

Y

G

<latexi sh

1_b

64="kf

p

D

Mm

Qo+vGd

>A

B7X

c

Z

L

w

FI

Ky

2

R

O

0

q

9

5

8

U

E

C

T

/

W

V

N

J

3

n

j

z

H

Y

<latexi sh

1_b

64="kf

p

D

Mm

Qo+vGd

>A

B7X

c

Z

L

w

FI

Ky

2

R

O

0

q

9

5

8

U

E

C

T

/

W

V

N

J

3

n

j

z

H

Y

<latexi sh

1_b

64="uU0HqrF

MzW

K

T

8G2

7n

ZQ

>A

c

V

NS

E

3

o/

vX

J

g

w

L

P

Y

d

C

pmO

j

y

9

R

I

k

D

f

5

<latexi sh

1_b

64="Sq

EMzCf

LP

F

G

TY

7+A

>

B

H

c

V

N

8

3Ur o/

vX

Z

Igj

k

u

d

m

Q

J

w

K

Wn

0

p

D

yR

O

2

9

5

<latexi sh

1_b

64="Sq

EMzCf

LP

F

G

TY

7+A

>

B

H

c

V

N

8

3Ur o/

vX

Z

Igj

k

u

d

m

Q

J

w

K

Wn

0

p

D

yR

O

2

9

5

<latexi sh

1_b

64="Sq

EMzCf

LP

F

G

TY

7+A

>

B

H

c

V

N

8

3Ur o/

vX

Z

Igj

k

u

d

m

Q

J

w

K

Wn

0

p

D

yR

O

2

9

5

<latexi sh

1_b

64="Sq

EMzCf

LP

F

G

TY

7+A

>

B

H

c

V

N

8

3Ur o/

vX

Z

Igj

k

u

d

m

Q

J

w

K

Wn

0

p

D

yR

O

2

9

5

<latexi sh

1_b

64="Sq

EMzCf

LP

F

G

TY

7+A

>

B

H

c

V

N

8

3Ur o/

vX

Z

Igj

k

u

d

m

Q

J

w

K

Wn

0

p

D

yR

O

2

9

5

<latexi sh

1_b

64="Sq

EMzCf

LP

F

G

TY

7+A

>

B

H

c

V

N

8

3Ur o/

vX

Z

Igj

k

u

d

m

Q

J

w

K

Wn

0

p

D

yR

O

2

9

5

<latexi sh

1_b

64="

uMmjd

NW9L

UK

ERSqf kY

>A

B

n

c

V

8

J3

r

/

0

o

CF

T

Q

+

z

O

D

7

I

G

Z

w

y

v

X

g

H

p

P

2

5

<latexi sh

1_b

64="

uMmjd

NW9L

UK

ERSqf kY

>A

B

n

c

V

8

J3

r

/

0

o

CF

T

Q

+

z

O

D

7

I

G

Z

w

y

v

X

g

H

p

P

2

5

<latexi sh

1_b

64="

uMmjd

NW9L

UK

ERSqf kY

>A

B

n

c

V

8

J3

r

/

0

o

CF

T

Q

+

z

O

D

7

I

G

Z

w

y

v

X

g

H

p

P

2

5

<latexi sh

1_b

64="

uMmjd

NW9L

UK

ERSqf kY

>A

B

n

c

V

8

J3

r

/

0

o

CF

T

Q

+

z

O

D

7

I

G

Z

w

y

v

X

g

H

p

P

2

5

<latexi sh

1_b

64="KwCOfVz

3

U

uT

m

j

E

>A

B

n

c

NS8

J

r

q/

9L

0

GP

g

D

7

QvH

y5

+

M

X

W

d

F

oR

k

Z

2

p

Y

I

<latexi sh

1_b

64="KwCOfVz

3

U

uT

m

j

E

>A

B

n

c

NS8

J

r

q/

9L

0

GP

g

D

7

QvH

y5

+

M

X

W

d

F

oR

k

Z

2

p

Y

I

<latexi sh

1_b

64="KwCOfVz

3

U

uT

m

j

E

>A

B

n

c

NS8

J

r

q/

9L

0

GP

g

D

7

QvH

y5

+

M

X

W

d

F

oR

k

Z

2

p

Y

I

<latexi sh

1_b

64="KwCOfVz

3

U

uT

m

j

E

>A

B

n

c

NS8

J

r

q/

9L

0

GP

g

D

7

QvH

y5

+

M

X

W

d

F

oR

k

Z

2

p

Y

I

<latexi sh

1_b

64="

cZLP38

9AY

ou

p

V

U

/S

>

B

n

N

EJ

r

q

0mk

M

CF

T

Q

+

z

O

D

7

I

G

j

w

y

f

d

v

X

R

g

H

K

W

2

5

<latexi sh

1_b

64="

cZLP38

9AY

ou

p

V

U

/S

>

B

n

N

EJ

r

q

0mk

M

CF

T

Q

+

z

O

D

7

I

G

j

w

y

f

d

v

X

R

g

H

K

W

2

5

<latexi sh

1_b

64="

cZLP38

9AY

ou

p

V

U

/S

>

B

n

N

EJ

r

q

0mk

M

CF

T

Q

+

z

O

D

7

I

G

j

w

y

f

d

v

X

R

g

H

K

W

2

5

<latexi sh

1_b

64="

cZLP38

9AY

ou

p

V

U

/S

>

B

n

N

EJ

r

q

0mk

M

CF

T

Q

+

z

O

D

7

I

G

j

w

y

f

d

v

X

R

g

H

K

W

2

5

<latexi sh

1_b

64="qW

Az

V

yI

5M

JZg

BNFE8YO

Q

>

n

c

S

3Ur

/

9L

0

P

K

D

wm7G

vH

T

+

u

X

d

j

oR

k

C

2

p

f

<latexi sh

1_b

64="qW

Az

V

yI

5M

JZg

BNFE8YO

Q

>

n

c

S

3Ur

/

9L

0

P

K

D

wm7G

vH

T

+

u

X

d

j

oR

k

C

2

p

f

<latexi sh

1_b

64="qW

Az

V

yI

5M

JZg

BNFE8YO

Q

>

n

c

S

3Ur

/

9L

0

P

K

D

wm7G

vH

T

+

u

X

d

j

oR

k

C

2

p

f

<latexi sh

1_b

64="qW

Az

V

yI

5M

JZg

BNFE8YO

Q

>

n

c

S

3Ur

/

9L

0

P

K

D

wm7G

vH

T

+

u

X

d

j

oR

k

C

2

p

f

<latexi sh

1_b

64="NK

o/3QZ

SC2I

7GJ

v

D

0

>A

B

n

c

V

8

E

Ur

q

9L

P

g

wm

H

T

y5

+

u

M

X

W

d

O

z

F

j

R

k

p

Y

f

<latexi sh

1_b

64="NK

o/3QZ

SC2I

7GJ

v

D

0

>A

B

n

c

V

8

E

Ur

q

9L

P

g

wm

H

T

y5

+

u

M

X

W

d

O

z

F

j

R

k

p

Y

f

<latexi sh

1_b

64="NK

o/3QZ

SC2I

7GJ

v

D

0

>A

B

n

c

V

8

E

Ur

q

9L

P

g

wm

H

T

y5

+

u

M

X

W

d

O

z

F

j

R

k

p

Y

f

<latexi sh

1_b

64="NK

o/3QZ

SC2I

7GJ

v

D

0

>A

B

n

c

V

8

E

Ur

q

9L

P

g

wm

H

T

y5

+

u

M

X

W

d

O

z

F

j

R

k

p

Y

f

<latexi sh

1_b

64="

z

0w5n

QE

ujv

N

J

>A

B

c

V

S8

3Ur q/o

9L

GP

y

p

W

R

FK

k

2

O

H

T

m

7

ZY

f

I

X

M

d

C

<latexi sh

1_b

64="

z

0w5n

QE

ujv

N

J

>A

B

c

V

S8

3Ur q/o

9L

GP

y

p

W

R

FK

k

2

O

H

T

m

7

ZY

f

I

X

M

d

C

<latexi sh

1_b

64="

z

0w5n

QE

ujv

N

J

>A

B

c

V

S8

3Ur q/o

9L

GP

y

p

W

R

FK

k

2

O

H

T

m

7

ZY

f

I

X

M

d

C

<latexi sh

1_b

64="

z

0w5n

QE

ujv

N

J

>A

B

c

V

S8

3Ur q/o

9L

GP

y

p

W

R

FK

k

2

O

H

T

m

7

ZY

f

I

X

M

d

C

<latexi sh

1_b

64="Pw

XE

dn2

L

0

/

F+7Yc

>A

B

V

S

N

J

R

o

g

fZ

D

y

3

k

C

GIr

j

8M

W

p

K

q

H

T

Q

U

O

u

5

v

<latexi sh

1_b

64="Pw

XE

dn2

L

0

/

F+7Yc

>A

B

V

S

N

J

R

o

g

fZ

D

y

3

k

C

GIr

j

8M

W

p

K

q

H

T

Q

U

O

u

5

v

<latexi sh

1_b

64="Pw

XE

dn2

L

0

/

F+7Yc

>A

B

V

S

N

J

R

o

g

fZ

D

y

3

k

C

GIr

j

8M

W

p

K

q

H

T

Q

U

O

u

5

v

<latexi sh

1_b

64="Pw

XE

dn2

L

0

/

F+7Yc

>A

B

V

S

N

J

R

o

g

fZ

D

y

3

k

C

GIr

j

8M

W

p

K

q

H

T

Q

U

O

u

5

v

<latexi sh

1_b

64="fG

pL2I

Wz

9j gc

EqN

AP

>

CD3

<latexi sh

1_b

64="fG

pL2I

Wz

9j gc

EqN

AP

>

CD3

<latexi sh

1_b

64="fG

pL2I

Wz

9j gc

EqN

AP

>

CD3

<latexi sh

1_b

64="fG

pL2I

Wz

9j gc

EqN

AP

>

CD3

V

S

M

F

U

v

Z

B

Q

XR

O

y

d

Y

+

8

u

r

/o w7

n

J

K

0

k

m

H

T

5

V

S

M

F

U

v

Z

B

Q

XR

O

y

d

Y

+

8

u

r

/o w7

n

J

K

0

k

m

H

T

5

V

S

M

F

U

v

Z

B

Q

XR

O

y

d

Y

+

8

u

r

/o w7

n

J

K

0

k

m

H

T

5

V

S

M

F

U

v

Z

B

Q

XR

O

y

d

Y

+

8

u

r

/o w7

n

J

K

0

k

m

H

T

5

<latexi sh

1_b

64="

zTXSEOZ7

PwMYJL

29F

D+Bc

>A

C

V

g

3

r

U

k

y

u

I

f

d

N

Q

q

/

0

R

8G

W

5

K

p

j

n

m

o

H

v

<latexi sh

1_b

64="

zTXSEOZ7

PwMYJL

29F

D+Bc

>A

C

V

g

3

r

U

k

y

u

I

f

d

N

Q

q

/

0

R

8G

W

5

K

p

j

n

m

o

H

v

<latexi sh

1_b

64="

zTXSEOZ7

PwMYJL

29F

D+Bc

>A

C

V

g

3

r

U

k

y

u

I

f

d

N

Q

q

/

0

R

8G

W

5

K

p

j

n

m

o

H

v

<latexi sh

1_b

64="

zTXSEOZ7

PwMYJL

29F

D+Bc

>A

C

V

g

3

r

U

k

y

u

I

f

d

N

Q

q

/

0

R

8G

W

5

K

p

j

n

m

o

H

v

<latexi sh

1_b

64="A

TK

OECFu cfp

2/

Q

vr

>

B8X

V

NS

J3

W

M

o

LyUR

Y

9

+

z

D

q

Gm

Z

j

w

I

d

g

0

n

k

P

H

5

7

<latexi sh

1_b

64="A

TK

OECFu cfp

2/

Q

vr

>

B8X

V

NS

J3

W

M

o

LyUR

Y

9

+

z

D

q

Gm

Z

j

w

I

d

g

0

n

k

P

H

5

7

<latexi sh

1_b

64="A

TK

OECFu cfp

2/

Q

vr

>

B8X

V

NS

J3

W

M

o

LyUR

Y

9

+

z

D

q

Gm

Z

j

w

I

d

g

0

n

k

P

H

5

7

<latexi sh

1_b

64="A

TK

OECFu cfp

2/

Q

vr

>

B8X

V

NS

J3

W

M

o

LyUR

Y

9

+

z

D

q

Gm

Z

j

w

I

d

g

0

n

k

P

H

5

7

<latexi sh

1_b

64="UWCIo

F

nr2w

O

NR

d

M

B8

>A

c

V

S

EJ3

q/

9L

0

GP

K

g

D

T

m7

QvH

y5

+

u

f

z

Z

k

j

X

p

Y

<latexi sh

1_b

64="UWCIo

F

nr2w

O

NR

d

M

B8

>A

c

V

S

EJ3

q/

9L

0

GP

K

g

D

T

m7

QvH

y5

+

u

f

z

Z

k

j

X

p

Y

<latexi sh

1_b

64="UWCIo

F

nr2w

O

NR

d

M

B8

>A

c

V

S

EJ3

q/

9L

0

GP

K

g

D

T

m7

QvH

y5

+

u

f

z

Z

k

j

X

p

Y

<latexi sh

1_b

64="UWCIo

F

nr2w

O

NR

d

M

B8

>A

c

V

S

EJ3

q/

9L

0

GP

K

g

D

T

m7

QvH

y5

+

u

f

z

Z

k

j

X

p

Y

<latexi

<latexi

<latexi

<latexi sh

1_b

64="E

7

Y50d

LgjP

H3mG

n+A

U

>

B

c

V

NS8

J

W

X

O

KIoM

C

z

/Q

Z

q

u

y

D

v

w

9

2

r

R

p

F

T

k

f

sh

1_b

64="E

7

Y50d

LgjP

H3mG

n+A

U

>

B

c

V

NS8

J

W

X

O

KIoM

C

z

/Q

Z

q

u

y

D

v

w

9

2

r

R

p

F

T

k

f

sh

1_b

64="E

7

Y50d

LgjP

H3mG

n+A

U

>

B

c

V

NS8

J

W

X

O

KIoM

C

z

/Q

Z

q

u

y

D

v

w

9

2

r

R

p

F

T

k

f

sh

1_b

64="E

7

Y50d

LgjP

H3mG

n+A

U

>

B

c

V

NS8

J

W

X

O

KIoM

C

z

/Q

Z

q

u

y

D

v

w

9

2

r

R

p

F

T

k

f

<latexi sh

1_b

64="nvq

Z9

kJ

EyoNA

m

>

B

c

V

Sw

2LXzF+R

g

5

0D

Y

3

f

PC

GIr

8

W

u

O

d

U

T

7

K

Q

H

<latexi sh

1_b

64="nvq

Z9

kJ

EyoNA

m

>

B

c

V

Sw

2LXzF+R

g

5

0D

Y

3

f

PC

GIr

8

W

u

O

d

U

T

7

K

Q

H

<latexi sh

1_b

64="nvq

Z9

kJ

EyoNA

m

>

B

c

V

Sw

2LXzF+R

g

5

0D

Y

3

f

PC

GIr

8

W

u

O

d

U

T

7

K

Q

H

<latexi sh

1_b

64="nvq

Z9

kJ

EyoNA

m

>

B

c

V

Sw

2LXzF+R

g

5

0D

Y

3

f

PC

GIr

8

W

u

O

d

U

T

7

K

Q

H

<latexi sh

1_b

64="70

NwOGnjM

T5

Zy3+

u8

>A

B

H

c

VDLSg

E

z

r

Y

Q

9I

U

C

Pk

o

f

v

R

qK

m

X

W

/

d2

F

J

p

<latexi sh

1_b

64="70

NwOGnjM

T5

Zy3+

u8

>A

B

H

c

VDLSg

E

z

r

Y

Q

9I

U

C

Pk

o

f

v

R

qK

m

X

W

/

d2

F

J

p

<latexi sh

1_b

64="70

NwOGnjM

T5

Zy3+

u8

>A

B

H

c

VDLSg

E

z

r

Y

Q

9I

U

C

Pk

o

f

v

R

qK

m

X

W

/

d2

F

J

p

<latexi sh

1_b

64="70

NwOGnjM

T5

Zy3+

u8

>A

B

H

c

VDLSg

E

z

r

Y

Q

9I

U

C

Pk

o

f

v

R

qK

m

X

W

/

d2

F

J

p

<latexi sh

1_b

64="H

+O0

7U

dDJ

RB

qw

g

c

>A

C

V

LS

N

F

3

2v

M

uQo

Z

k

y

f

T

I

n

K

G

/

PX

9

m

z

5

p

Y

E

8

r

W

j

<latexi sh

1_b

64="

ONp

Fky

+CWD

cJQX

V

S

w

>A

L

gM

3U

2v

T

m9

Kj

Y

r

HIZ

o

5

n

d7

u

G

f

0

z

R

E

q

B

P

/

8

<latexi sh

1_b

64="S

wKcYnQ

TVO

H5

C

N

k

>A

DL

FJ3U

2vq

X

0EIp

r

u

GZ

7g yW

M

m

/o

9

P

+

d

z

8

f

R

j

B

<latexi sh

1_b

64="09

TBAdvjP/

ESUZn

NO

G8

W

>

C

H

c

VDL

FJ3

2

q

X

kK

y

+o

Mpm

Q

r

z

g

w

f

Y

7

I

u

5

R

<latexi sh

1_b

64="NwTV

U

pM

oGcj0+k

D

f

>A

B

3

S

r

q/

9

Ivg

7H

W

F

KJ

2

m

y5

n

d

Z

Y

P

CX

O

R

L

z

<latexi sh

1_b

64="S

Dk+m r7G

MgJzTH

pj

I

>A

B

n

c

V

L

N

F

3U

2v du

o

q

Q

W

C

Y

0

O

K

P

X

w

f

9

yZ

E

R

5

/

8

<latexi sh

1_b

64="CGn

2K

o

ju

Hkq

+zNrF0

>A

B

c

VDLS

p

f

7d

U

E

XR

Q

w

m

IJM

O

P

3

T

5

W

9Y

yZ

v/

g

8

<latexi sh

1_b

64="

WD2JGy

B

FZ

+

/mE

qp

U

>A

8n

c

V

LSgM

TX7

d

Iv

Y

rQ

k

w3

R

9

P

O

z

N

HC

j

f

0

5

K

u

o

<latexi sh

1_b

64="P

Aq+k9TjD

oMJZ

n

g

>

B8

c

V

NS

3

r

/

L

0

2

R

K

w

7

yG

H

5

Q

u

f

m

dW

I

C

O

F

X

Y

<latexi sh

1_b

64="D

pGTu

ZIAk8

mg

E

w

>

B+3

c

V

LS

N

F

PU

9

Y

R

Q

X

K

Mn

2

Hj

/o

7

W

O

d

0

y

f

5

v

q

C

<latexi sh

1_b

64="8XQV

EPT

Gc

>

+3

DL

F

2

W

d

q

g

m

0

C

M

I

R

o

5

p

n

r

9

z

/

H

f

wY

k

K

y

Z

O

j

<latexi sh

1_b

64="r

Iu

A5

VQHO

3RyzE

m8n9

>

c

DLS

N

FJ

PU

Y

X

w

TKZ

M

2

p

j

G

/og7

W

d

0

f

v

q

k

C

<latexi sh

1_b

64="P

Aq+k9TjD

oMJZ

n

g

>

B8

c

V

NS

3

r

/

L

0

2

R

K

w

7

yG

H

5

Q

u

f

m

dW

I

C

O

F

X

Y

<latexi sh

1_b

64="Rod9ISn zmu

V

Mqwk

F

C

Q

>A

B

c

DL

g

TX7W+

v

y

r2

J

0

O

j

G

E

/

3

58

U

N

pf

K

H

P

Z

Y

<latexi sh

1_b

64="KJ5mjcgD

C

U8n

r

VHPSLYI

>A

B7

N

E

3

q/

9

F

0

G

y

p

O2

X

Z

d

M

v

w

W

u

o

k

T

R

Q

f

z

+

<latexi sh

1_b

64="KJ5mjcgD

C

U8n

r

VHPSLYI

>A

B7

N

E

3

q/

9

F

0

G

y

p

O2

X

Z

d

M

v

w

W

u

o

k

T

R

Q

f

z

+

<latexi sh

1_b

64="KJ5mjcgD

C

U8n

r

VHPSLYI

>A

B7

N

E

3

q/

9

F

0

G

y

p

O2

X

Z

d

M

v

w

W

u

o

k

T

R

Q

f

z

+

<latexi sh

1_b

64="KJ5mjcgD

C

U8n

r

VHPSLYI

>A

B7

N

E

3

q/

9

F

0

G

y

p

O2

X

Z

d

M

v

w

W

u

o

k

T

R

Q

f

z

+

<latexi sh

1_b

64="Rod9ISn zmu

V

Mqwk

F

C

Q

>A

B

c

DL

g

TX7W+

v

y

r2

J

0

O

j

G

E

/

3

58

U

N

pf

K

H

P

Z

Y

<latexi sh

1_b

64="fHX

8A

r/

2+kmY0IQd

L

J

E

>

B

n

c

VD

SgN

Oy

U

9

oP

K

G

F

M

z

u

7

CW

jw

v

q

5Z

3

p

T

R

<latexi sh

1_b

64="Ag

7FzHBvUqYGPE

f

8

0

X

k

>

c

VDLS

N

Oy

r

9

oM

C

2

Q

3

u

W

I

n

d

R

/K

5

J

Z

p

T

+

<latexi sh

1_b

64="/nRpSI

8

oP9

U

2WF3M

>A

B7H

c

V

N

EJ

r

q

L

0mk

C

T

Q

+

z

u

D

G

Z

j

Y

w

y

f

d

v

O

X

g

K

5

<latexi sh

1_b

64="/nRpSI

8

oP9

U

2WF3M

>A

B7H

c

V

N

EJ

r

q

L

0mk

C

T

Q

+

z

u

D

G

Z

j

Y

w

y

f

d

v

O

X

g

K

5

<latexi sh

1_b

64="/nRpSI

8

oP9

U

2WF3M

>A

B7H

c

V

N

EJ

r

q

L

0mk

C

T

Q

+

z

u

D

G

Z

j

Y

w

y

f

d

v

O

X

g

K

5

<latexi sh

1_b

64="/nRpSI

8

oP9

U

2WF3M

>A

B7H

c

V

N

EJ

r

q

L

0mk

C

T

Q

+

z

u

D

G

Z

j

Y

w

y

f

d

v

O

X

g

K

5

<latexi sh

1_b

64="voF9I

XQ

L

kf

S

g/uGpY

>A

B7H

c

V

N

8

EJ3Ur

q

0n

2PR

K

D

T

wm y5

+

M

dW

Z

j

z

C

O

<latexi sh

1_b

64="7DrWSm

Z5E

jU0

F

c

g

>A

B

H

V

N

8

J3

q/

9L

n

P

K

2kI

y

du

v

G

+oO

z

M

w

f

C

Q

Y

<latexi sh

1_b

64="fHX

8A

r/

2+kmY0IQd

L

J

E

>

B

n

c

VD

SgN

Oy

U

9

oP

K

G

F

M

z

u

7

CW

jw

v

q

5Z

3

p

T

R

<latexi sh

1_b

64="KJ5mjcgD

C

U8n

r

VHPSLYI

>A

B7

N

E

3

q/

9

F

0

G

y

p

O2

X

Z

d

M

v

w

W

u

o

k

T

R

Q

f

z

+

<latexi sh

1_b

64="KJ5mjcgD

C

U8n

r

VHPSLYI

>A

B7

N

E

3

q/

9

F

0

G

y

p

O2

X

Z

d

M

v

w

W

u

o

k

T

R

Q

f

z

+

<latexi sh

1_b

64="KJ5mjcgD

C

U8n

r

VHPSLYI

>A

B7

N

E

3

q/

9

F

0

G

y

p

O2

X

Z

d

M

v

w

W

u

o

k

T

R

Q

f

z

+

<latexi sh

1_b

64="KJ5mjcgD

C

U8n

r

VHPSLYI

>A

B7

N

E

3

q/

9

F

0

G

y

p

O2

X

Z

d

M

v

w

W

u

o

k

T

R

Q

f

z

+

<latexi sh

1_b

64="Rod9ISn zmu

V

Mqwk

F

C

Q

>A

B

c

DL

g

TX7W+

v

y

r2

J

0

O

j

G

E

/

3

58

U

N

pf

K

H

P

Z

Y

<latexi sh

1_b

64="fHX

8A

r/

2+kmY0IQd

L

J

E

>

B

n

c

VD

SgN

Oy

U

9

oP

K

G

F

M

z

u

7

CW

jw

v

q

5Z

3

p

T

R

<latexi sh

1_b

64="Ag

7FzHBvUqYGPE

f

8

0

X

k

>

c

VDLS

N

Oy

r

9

oM

C

2

Q

3

u

W

I

n

d

R

/K

5

J

Z

p

T

+

<latexi sh

1_b

64="/nRpSI

8

oP9

U

2WF3M

>A

B7H

c

V

N

EJ

r

q

L

0mk

C

T

Q

+

z

u

D

G

Z

j

Y

w

y

f

d

v

O

X

g

K

5

<latexi sh

1_b

64="/nRpSI

8

oP9

U

2WF3M

>A

B7H

c

V

N

EJ

r

q

L

0mk

C

T

Q

+

z

u

D

G

Z

j

Y

w

y

f

d

v

O

X

g

K

5

<latexi sh

1_b

64="/nRpSI

8

oP9

U

2WF3M

>A

B7H

c

V

N

EJ

r

q

L

0mk

C

T

Q

+

z

u

D

G

Z

j

Y

w

y

f

d

v

O

X

g

K

5

<latexi sh

1_b

64="/nRpSI

8

oP9

U

2WF3M

>A

B7H

c

V

N

EJ

r

q

L

0mk

C

T

Q

+

z

u

D

G

Z

j

Y

w

y

f

d

v

O

X

g

K

5

<latexi sh

1_b

64="O

JCX

7

G

cSM

v

I

>A

B

n

VD

gN

K2

z

j

F

8R

Q

PUmT

p

u0Y

d

3

r

q

k

/

5

f

y

Z

<latexi sh

1_b

64="+AK7o

TZRz

H

pI

fFCQdk8

>

B

X

c

VDLSgN

EO

G

Mr

Y

J

P

vy

w

3

9

W

2

0

5

n

qU

m

uj

/

<latexi sh

1_b

64="kn fA3jMw

YW

qN

>

B8X

c

VDLS

F

2pr

Z

u

vgQ

7K

m

I

T

G

/o0

H

+

9

5

Rz

y

E

P

<latexi sh

1_b

64="m zXHw0F+j5WyENgQ

V

A

>

CBn

c

DLS

M

3U

2v

Z

K

I

fu

OG

kd

o

7/8

P

R

Y

J

9

r

p

q

T

<latexi sh

1_b

64="KJ5mjcgD

C

U8n

r

VHPSLYI

>A

B7

N

E

3

q/

9

F

0

G

y

p

O2

X

Z

d

M

v

w

W

u

o

k

T

R

Q

f

z

+

<latexi sh

1_b

64="KJ5mjcgD

C

U8n

r

VHPSLYI

>A

B7

N

E

3

q/

9

F

0

G

y

p

O2

X

Z

d

M

v

w

W

u

o

k

T

R

Q

f

z

+

<latexi sh

1_b

64="KJ5mjcgD

C

U8n

r

VHPSLYI

>A

B7

N

E

3

q/

9

F

0

G

y

p

O2

X

Z

d

M

v

w

W

u

o

k

T

R

Q

f

z

+

<latexi sh

1_b

64="KJ5mjcgD

C

U8n

r

VHPSLYI

>A

B7

N

E

3

q/

9

F

0

G

y

p

O2

X

Z

d

M

v

w

W

u

o

k

T

R

Q

f

z

+

<latexi sh

1_b

64="Rod9ISn zmu

V

Mqwk

F

C

Q

>A

B

c

DL

g

TX7W+

v

y

r2

J

0

O

j

G

E

/

3

58

U

N

pf

K

H

P

Z

Y

<latexi sh

1_b

64="fHX

8A

r/

2+kmY0IQd

L

J

E

>

B

n

c

VD

SgN

Oy

U

9

oP

K

G

F

M

z

u

7

CW

jw

v

q

5Z

3

p

T

R

<latexi sh

1_b

64="Ag

7FzHBvUqYGPE

f

8

0

X

k

>

c

VDLS

N

Oy

r

9

oM

C

2

Q

3

u

W

I

n

d

R

/K

5

J

Z

p

T

+

<latexi sh

1_b

64="/nRpSI

8

oP9

U

2WF3M

>A

B7H

c

V

N

EJ

r

q

L

0mk

C

T

Q

+

z

u

D

G

Z

j

Y

w

y

f

d

v

O

X

g

K

5

<latexi sh

1_b

64="/nRpSI

8

oP9

U

2WF3M

>A

B7H

c

V

N

EJ

r

q

L

0mk

C

T

Q

+

z

u

D

G

Z

j

Y

w

y

f

d

v

O

X

g

K

5

<latexi sh

1_b

64="/nRpSI

8

oP9

U

2WF3M

>A

B7H

c

V

N

EJ

r

q

L

0mk

C

T

Q

+

z

u

D

G

Z

j

Y

w

y

f

d

v

O

X

g

K

5

<latexi sh

1_b

64="/nRpSI

8

oP9

U

2WF3M

>A

B7H

c

V

N

EJ

r

q

L

0mk

C

T

Q

+

z

u

D

G

Z

j

Y

w

y

f

d

v

O

X

g

K

5

<latexi sh

1_b

64="

9pr g+wOVXSZ

vDKy

Q

k

>A

B

c

L

N

FJ3

PU

d

m

qu

H

CEM

2

z

5

T

j

G

/o

7

8

W

0n

f

R

Y

I

<latexi sh

1_b

64="pfYg

F08/P

uoR

VB9

W

+

EcI

>A

X

DLS

N

J3U

2vq

w

Kkn

y

G

m

M

z

5

d

7

r

H

Z

C

Q

j

O

T

<latexi sh

1_b

64="u8

nI

c

DX

Z

FR/O9

kvA

>

V

LS

N

3

W

r

Y

U

Q

d

Pq

J

TK

E

j

z

H

G

o

7

pm

2

w

0

f

y

C

g

M

5

<latexi sh

1_b

64="JM

rPTU

vDSg

VNW

Am

>

B9H

c

L

F

2p

f

Zdu

q

KC

Y

y

o

3

n

+

X

j

5

8

O

w

G

7

E

I

R

0

/

<latexi sh

1_b

64="u

8Pg

V

KHC

NB

XESnG

q

Mc

>A

9

DL

F

3

W

r

Y

J

5

O2

T

Zy od7

f+

0zU

w

v

p

m

j

I

k

R

Q

/

<latexi sh

1_b

64="5RKOpwYL0Tg

Ur

SQGVm8u+

>A

B

H

c

D

N

E

z

M

3

d

Z

Cy

J

n

X

k

/

I

o

q

v

2

j

7

W

P

F

9

f

<latexi sh

1_b

64="wyW

vPCzf

JO

d

HIX

K+0

c

>A

B8n

V

NS

E

3Ur q/

9LF

2

D

m7

G

g

T

5

Q

u

M

o

Z

j

Y

R

k

p

<latexi sh

1_b

64="cPGM0RQdE

u/ZqT

N8A

vVSB

>

n

3Ur

9

K

g

7H

W

F

w

p

J

2

5

+

D

f

m

o

X

ILO

k

y

Y

z

C

j

<latexi sh

1_b

64="M

9

8KOpPkcSg

Hwf

2

T

F

>A

B

n

VDL

N

J3U

vq

Z

7

C

ymQ

Yd

j

0X

+

u/

R

W

r

E

z

o

5

G

I

<latexi sh

1_b

64="0

/F+Sgz

R

MHu fK

E

>A

C

X

VD

NB

O2

W

w

k

T

n

pyZ

rPQ

Y

7

3

5mI

9q

d

L

U

j

v

<latexi sh

1_b

64="0

/F+Sgz

R

MHu fK

E

>A

C

X

VD

NB

O2

W

w

k

T

n

pyZ

rPQ

Y

7

3

5mI

9q

d

L

U

j

v

<latexi sh

1_b

64="0

/F+Sgz

R

MHu fK

E

>A

C

X

VD

NB

O2

W

w

k

T

n

pyZ

rPQ

Y

7

3

5mI

9q

d

L

U

j

v

<latexi sh

1_b

64="0

/F+Sgz

R

MHu fK

E

>A

C

X

VD

NB

O2

W

w

k

T

n

pyZ

rPQ

Y

7

3

5mI

9q

d

L

U

j

v

<latexi sh

1_b

64="JvW

9

D

y2

dgZ0m

YM

qjCA

>

Ln

c

V

S

NBEO

K

Xw

k

T

3o

G

p

RF

PUz7

H

Q8

r

u5

I

+

f

/

<latexi sh

1_b

64="JvW

9

D

y2

dgZ0m

YM

qjCA

>

Ln

c

V

S

NBEO

K

Xw

k

T

3o

G

p

RF

PUz7

H

Q8

r

u5

I

+

f

/

<latexi sh

1_b

64="JvW

9

D

y2

dgZ0m

YM

qjCA

>

Ln

c

V

S

NBEO

K

Xw

k

T

3o

G

p

RF

PUz7

H

Q8

r

u5

I

+

f

/

<latexi sh

1_b

64="JvW

9

D

y2

dgZ0m

YM

qjCA

>

Ln

c

V

S

NBEO

K

Xw

k

T

3o

G

p

RF

PUz7

H

Q8

r

u5

I

+

f

/

<latexi sh

1_b

64="K

jn

E+M5Hd vPR

QC

u

>A

c

VDJSgNB

O2

W

z

q

wIG

m

o

9

T

k

F7

L3

X

U8

p

r

y

f

Z

0

Y

/

<latexi sh

1_b

64="KJ5mjcgD

C

U8n

r

VHPSLYI

>A

B7

N

E

3

q/

9

F

0

G

y

p

O2

X

Z

d

M

v

w

W

u

o

k

T

R

Q

f

z

+

<latexi sh

1_b

64="KJ5mjcgD

C

U8n

r

VHPSLYI

>A

B7

N

E

3

q/

9

F

0

G

y

p

O2

X

Z

d

M

v

w

W

u

o

k

T

R

Q

f

z

+

<latexi sh

1_b

64="KJ5mjcgD

C

U8n

r

VHPSLYI

>A

B7

N

E

3

q/

9

F

0

G

y

p

O2

X

Z

d

M

v

w

W

u

o

k

T

R

Q

f

z

+

<latexi sh

1_b

64="KJ5mjcgD

C

U8n

r

VHPSLYI

>A

B7

N

E

3

q/

9

F

0

G

y

p

O2

X

Z

d

M

v

w

W

u

o

k

T

R

Q

f

z

+

<latexi sh

1_b

64="Rod9ISn zmu

V

Mqwk

F

C

Q

>A

B

c

DL

g

TX7W+

v

y

r2

J

0

O

j

G

E

/

3

58

U

N

pf

K

H

P

Z

Y

<latexi sh

1_b

64="fHX

8A

r/

2+kmY0IQd

L

J

E

>

B

n

c

VD

SgN

Oy

U

9

oP

K

G

F

M

z

u

7

CW

jw

v

q

5Z

3

p

T

R

<latexi sh

1_b

64="Ag

7FzHBvUqYGPE

f

8

0

X

k

>

c

VDLS

N

Oy

r

9

oM

C

2

Q

3

u

W

I

n

d

R

/K

5

J

Z

p

T

+

<latexi sh

1_b

64="/nRpSI

8

oP9

U

2WF3M

>A

B7H

c

V

N

EJ

r

q

L

0mk

C

T

Q

+

z

u

D

G

Z

j

Y

w

y

f

d

v

O

X

g

K

5

<latexi sh

1_b

64="/nRpSI

8

oP9

U

2WF3M

>A

B7H

c

V

N

EJ

r

q

L

0mk

C

T

Q

+

z

u

D

G

Z

j

Y

w

y

f

d

v

O

X

g

K

5

<latexi sh

1_b

64="/nRpSI

8

oP9

U

2WF3M

>A

B7H

c

V

N

EJ

r

q

L

0mk

C

T

Q

+

z

u

D

G

Z

j

Y

w

y

f

d

v

O

X

g

K

5

<latexi sh

1_b

64="/nRpSI

8

oP9

U

2WF3M

>A

B7H

c

V

N

EJ

r

q

L

0mk

C

T

Q

+

z

u

D

G

Z

j

Y

w

y

f

d

v

O

X

g

K

5

<latexi sh

1_b

64="ZRkcG

y

/

D70

C

YX

z

FUw

>A

B

3

V

LS

N

2pr

f

du qvg

T

K

mQ

O

M

P+H

J

o

9

j

W

n

I5

8

E

<latexi sh

1_b

64="wZES2U

VF

DL

d

0

B

X

P

j

>A

73

c

gN

Oy

r

f

Y9

q

K

8

I5

HJ

m

n

Q

p

C

v

o

k

G

T

W

u

R

M

/

+

z

3) Tail feature agent. This learning system includes: State: includes the representation of X i , selected head feature, and operation, denoted by Rep ( X i ) ⊕ Rep ( f h ) ⊕ Rep ( o ) . Action: is the tail feature f t selected from X i by this agent.

State representation method, Rep ( · ) . For the representation of the feature set, we employ a descriptive statistical technique to obtain the state with a fixed length. In detail, we first compute the descriptive statistics ( i.e. count, standard deviation, minimum, maximum, first, second, and third quantile) of the feature set column-wise. Then, we calculate the same descriptive statistics on the output of the previous step. After that, we can obtain the descriptive matrix with shape R 7 × 7 and flatten it as the state representation with shape R 1 × 49 . For the representation of the operation, we adopt its one-hot encoding as Rep ( o ) .

Reward function. To improve the quality of the feature space, we use the improvement of a downstream ML task performance as the reward. Thus, it can be defined as: R ( X i , X i +1 ) = A ( M ( X i +1 ) , y ) -A ( M ( X i ) , y ) .

Learning to Collect Training Data. To optimize the entire procedure, we minimize the mean squared error of the Bellman Equation to get a better feature space. During the exploration process, we can collect amounts of high-quality records (Γ , v ) for constructing an effective continuous embedding space, where Γ is the transformation sequence, and v is the downstream model performance.

3.3 Postfix Expressions of Feature Transformation Operation Sequences

Why Transformation Operation Sequences as Postfix Expressions. After training data collection, a question arises: how can we organize and represent these transformation operation sequences in a computationally-tangible and machine-learnable format?

Figure 3 (a) shows an example of a feature transformation operation sequence with two generated features. To convert this sequence into a machine-readable expression, a naive idea is to enclose each calculation in a pair of brackets to indicate its priority (Figure 3(b)). But, the representation of Figure 3(b) has four limitations: (1) Redundancy. Many priority-related brackets are included to ensure the unambiguous property and correctness of mathematical calculations. (2) Semantic Sparsity. The quantity of bracket tokens can dilute the semantic information of the sequence, making model convergence diffi-

τ

1

τ

2

)

(b) Original infix notation based sequence

-

f

2

+

(

(

f

1

/

f

3

)

f

2

√ ·

f

1

(a) Feature transformation Sequence ( f 1 +( f 1 /f 3 ) - f 2 ) , ( √ f 1 )

f

2

f

1

f

3

/

f

3

-

+

SEP

f

2

√ ·

…

SOS

SOS

SEP

EOS

EOS

Figure 3: Compared to infix-based expressions, postfix-based expressions optimize token redundancy, enhance semantics, prevent illegal transformations, and minimize the search space.

cult. (3) Illegal Transformation. If the decoder makes one mistake on bracket generation, the entire generated sequence will be wrong. (4) Large Search Space. The number of combinations of features and operations from low-order to high-order interactions is large, making the search space too vast.

Using Postfix Expression to Construct Robust, Concise, and Unambiguous Sequences. To address the aforementioned limitations, we convert the transformation operation sequence Γ into a postfix-based sequence expression. Specifically, we scan each mathematical composition τ in Γ from left to right and convert it from the infix-based format to the postfix-based one. We then concatenate each postfix expression by the <SEP> token and add the <SOS> and <EOS> tokens to the beginning and end of the entire sequence. Figure 3(c) shows an example of such a postfix sequence. We denote it as Υ = [ γ 1 , · · · , γ M ] , where each element is a feature ID token, operation token, or three other unique tokens. The detailed pseudo code of this conversion process is provided in Appendix A.

The postfix sequences don't require numerous brackets to ensure the calculation priority. We only need to scan each element in the sequence from left to right to reconstruct corresponding transformed features. Such a concise and short expression can reduce sequential modeling difficulties and computational costs. Besides, a postfix sequence indicates a unique transformation process, thus, reducing the ambiguity of the feature transformation sequence. Moreover, the most crucial aspect is the reduction of the search space from exponentially growing discrete combinations to a limited token set C that consists of the original feature ID tokens, operation tokens, and other three unique tokens. The length of the token set is |O| + | X | +3 , where |O| is the number of the operation set, | X | is the dimension of the original feature set, and 3 refers to the unique tokens <SOS>, <SEP>, <EOS>.

<latexi sh

1_b

64="SUvw

H

QpMdXfABL5

CO/

y

>

n

c

VD

gN

E

r

Y9

oP

K

2

z

j

I

7

R

Zm

k

8

0

q

+

u

G

F

J

3

T

W

<latexi sh

1_b

64="SUvw

H

QpMdXfABL5

CO/

y

>

n

c

VD

gN

E

r

Y9

oP

K

2

z

j

I

7

R

Zm

k

8

0

q

+

u

G

F

J

3

T

W

<latexi sh

1_b

64="Z

f+0

c

p

P

WMSG

RDH

NX

u8

>A

CE

V

7T

wF

Kq5

gJ

ok nm

2O

d

Q

ILv

j

z

9

U

y

B

/

Y

3

r

<latexi sh

1_b

64="GK

Tu3

7

BnIf

5/Vrm

Hv

QM

>A

c

DLSgN

EOy

UY9

oP

w

X8

F

Rz

J

p

d

+

W

2

j

Z

C

0

q

k

<latexi sh

1_b

64="GK

Tu3

7

BnIf

5/Vrm

Hv

QM

>A

c

DLSgN

EOy

UY9

oP

w

X8

F

Rz

J

p

d

+

W

2

j

Z

C

0

q

k

<latexi sh

1_b

64="Ag

7FzHBvUqYGPE

f

8

0

X

k

>

c

VDLS

N

Oy

r

9

oM

C

2

Q

3

u

W

I

n

d

R

/K

5

J

Z

p

T

+

<latexi sh

1_b

64="PjEzg

N

cFVRJC7

p

5L

>A

B

n

D

S

Oy

r

fUY

o

K

2

8

I

H

m

d

w

v

3

k

G

q

T

W

Z

0

u

M

/

+

<latexi sh

1_b

64="

V

g

LZ

HQ8qFkJU

Er

j

Mv

d

>A

B7X

c

D

S

N

Oy

f

Y9

oP

K

2

I5

m

n

p

/

w

3

G

T

W

0

u

C

+

R

z

<latexi sh

1_b

64="3S

TNV

JZz

v

M/

PG

OQ

Fw

>A

B

n

c

L

g

E

y

r

fUY9

o

K

I

Hp

X

m

W

5

8

q

k

u

R

2

7

C

d

0

+

<latexi sh

1_b

64="2g0

Pn9V/uyW58

d

7C

FQ

>

B

H

DLT

J

EOz

+I

j

R

Nr

w

v

Zm

U

k

p

3

q

X

Y

f

G

S

o

M

K

<latexi sh

1_b

64="2g0

Pn9V/uyW58

d

7C

FQ

>

B

H

DLT

J

EOz

+I

j

R

Nr

w

v

Zm

U

k

p

3

q

X

Y

f

G

S

o

M

K

<latexi sh

1_b

64="

8qB3

R/wTV

k5

cLXZ

y2nv+I

>A

H

D

gJ

EOzF

9

j

G

o

SM

Q

m

d

0

N

U

p7

r

W

K

C

u

f

Y

P

<latexi sh

1_b

64="PjEzg

N

cFVRJC7

p

5L

>A

B

n

D

S

Oy

r

fUY

o

K

2

8

I

H

m

d

w

v

3

k

G

q

T

W

Z

0

u

M

/

+

<latexi sh

1_b

64="3S

TNV

JZz

v

M/

PG

OQ

Fw

>A

B

n

c

L

g

E

y

r

fUY9

o

K

I

Hp

X

m

W

5

8

q

k

u

R

2

7

C

d

0

+

<latexi sh

1_b

64="

PX

B

FNjJ

wDAIgcu

M

>

H

V

LS

EOy

r

fUY9

o

v

5

T

Qp

d

CZ

m

k

8

n

R70

qK

+

G

2

/

z

3

W

<latexi sh

1_b

64="

8qB3

R/wTV

k5

cLXZ

y2nv+I

>A

H

D

gJ

EOzF

9

j

G

o

SM

Q

m

d

0

N

U

p7

r

W

K

C

u

f

Y

P

<latexi sh

1_b

64="J+jD

T

qkZcO0yP

L

YVf

IU7E

>A

B

X

SgN

z

G

Mr

H

8

d3

C

n

m

R

/w oK

u

W

2

F

v

p

9

Q

5

<latexi sh

1_b

64="Ag

7FzHBvUqYGPE

f

8

0

X

k

>

c

VDLS

N

Oy

r

9

oM

C

2

Q

3

u

W

I

n

d

R

/K

5

J

Z

p

T

+

<latexi sh

1_b

64="3S

TNV

JZz

v

M/

PG

OQ

Fw

>A

B

n

c

L

g

E

y

r

fUY9

o

K

I

Hp

X

m

W

5

8

q

k

u

R

2

7

C

d

0

+

<latexi sh

1_b

64="Rod9ISn zmu

V

Mqwk

F

C

Q

>A

B

c

DL

g

TX7W+

v

y

r2

J

0

O

j

G

E

/

3

58

U

N

pf

K

H

P

Z

Y

<latexi sh

1_b

64="EkO0PY

J

m

VQ

2N

M

BC

j

>A

H

c

DLSg

y

r

fU

9

o

7

v

5

T

I

p

d

Z

X

8

n

R

qK

+

u

G

/

w

z

F

3

W

<latexi sh

1_b

64="GK

Tu3

7

BnIf

5/Vrm

Hv

QM

>A

c

DLSgN

EOy

UY9

oP

w

X8

F

Rz

J

p

d

+

W

2

j

Z

C

0

q

k

<latexi sh

1_b

64="

PX

B

FNjJ

wDAIgcu

M

>

H

V

LS

EOy

r

fUY9

o

v

5

T

Qp

d

CZ

m

k

8

n

R70

qK

+

G

2

/

z

3

W

<latexi sh

1_b

64="n

AY

Z

9J5S73M

yL

f

8

>

B

H

c

VD

N

F

U

2vqk g0

w

R

P

TK

E

mX

p

C

O

+j u/

W

r

d

o

Q

I

z

G

<latexi sh

1_b

64="GK

Tu3

7

BnIf

5/Vrm

Hv

QM

>A

c

DLSgN

EOy

UY9

oP

w

X8

F

Rz

J

p

d

+

W

2

j

Z

C

0

q

k

<latexi sh

1_b

64="fHX

8A

r/

2+kmY0IQd

L

J

E

>

B

n

c

VD

SgN

Oy

U

9

oP

K

G

F

M

z

u

7

CW

jw

v

q

5Z

3

p

T

R

<latexi sh

1_b

64="EkO0PY

J

m

VQ

2N

M

BC

j

>A

H

c

DLSg

y

r

fU

9

o

7

v

5

T

I

p

d

Z

X

8

n

R

qK

+

u

G

/

w

z

F

3

W

<latexi sh

1_b

64="Rod9ISn zmu

V

Mqwk

F

C

Q

>A

B

c

DL

g

TX7W+

v

y

r2

J

0

O

j

G

E

/

3

58

U

N

pf

K

H

P

Z

Y

<latexi sh

1_b

64="n

AY

Z

9J5S73M

yL

f

8

>

B

H

c

VD

N

F

U

2vqk g0

w

R

P

TK

E

mX

p

C

O

+j u/

W

r

d

o

Q

I

z

G

<latexi sh

1_b

64="PjEzg

N

cFVRJC7

p

5L

>A

B

n

D

S

Oy

r

fUY

o

K

2

8

I

H

m

d

w

v

3

k

G

q

T

W

Z

0

u

M

/

+

<latexi sh

1_b

64="fHX

8A

r/

2+kmY0IQd

L

J

E

>

B

n

c

VD

SgN

Oy

U

9

oP

K

G

F

M

z

u

7

CW

jw

v

q

5Z

3

p

T

R

Data Augmentation for Postfix Transformation Sequences. Big and diversified transformation sequence records can benefit the learning of a pattern discriminative embedding space. Be sure to notice that, a feature transformation sequence consists of many independent segmentations, each of which is the composition of feature ID and operation tokens and can be used to generate a new feature. These independent segmentations are order-agnostic. Our idea is to leverage this property to conduct data augmentation to increase the data volume and diversity. For example, given a transformation operation sequence and corresponding accuracy { Υ , v } , we first divide the postfix expression into different segmentations by <SEP>. We then randomly shuffle these segmentations and use <SEP> to concatenate them together to generate new postfix transformation sequences. After that, we pair the new sequences with the corresponding model accuracy performance to improve data diversity and data volume for better model training and to create the continuous embedding space.

3.4 Deep Feature Transformation Embedding

After collecting and converting large-scale feature transformation training data to a set of postfix expression-accuracy pairs { (Υ i , v i ) } n i =1 , we develop an encoder-evaluator-decoder structure to map the sequential information of these records into an embedding space. Each embedding vector is associated with a transformation operation sequence and its corresponding model accuracy.

Encoder ϕ : The Encoder aims to map any given postfix expression to an embedding (a.k.a., hidden state) E . We adopt a single layer long short-term memory [10] (LSTM) as Encoder and acquire the continuous representation of Υ , denoted by E = ϕ (Υ) ∈ R M × d , where M is the total length of input sequence Υ and d is the hidden size of the embedding.

Decoder ψ : The Decoder aims to reconstruct the postfix expression of the feature transformation operation sequence Υ from the hidden state E . In MOAT, we set the backbone of the Decoder as a single-layer LSTM. For the first step, ψ will take an initial state (denoted as h 0 ) as input. Specifically, in stepi , we can obtain the decoder hidden state h d i from the LSTM. We use the dot product attention to aggregate the encoder hidden state and obtain the combined encoder hidden state h e i . Then, the

Evaluator ω : The Evaluator is designed to estimate the quality of continuous embeddings. Specifically, we will first conduct mean pooling on E by column to aggregate the information and obtain the embedding ¯ E ∈ R d . Then ¯ E is input into a feedforward network to estimate the corresponding model performance, given as: ˆ v = ω ( E ) . To minimize the gap between estimated accuracy and real-world gold accuracy, we leverage the Mean Squared Error (MSE) given by: L est = MSE ( v, ω ( E )) .

distribution in stepj can be defined as: P ψ ( γ i | E , Υ <i ) = exp( W γ i ( h d i ⊕ h e i )) ∑ c ∈C exp( W c ( h d i ⊕ h e i )) , where γ i ∈ Υ is the i -th token in sequence Υ , and C is the token set. W stand for the parameter of the feedforward network. Υ <i represents the prediction of the previous or initial step. By multiplying the probability in each step, we can form the distribution of each token in Υ , given as: P ψ (Υ | E ) = ∏ M i =1 P ψ ( γ i | E , Υ <i ) . To make the generated sequence similar to the real one, we minimize the negative log-likelihood of the distribution, defined as: L rec = -log P ψ (Υ | E ) .

Joint Training Loss L : We jointly optimize the encoder, decoder, and evaluator. The joint training loss can be formulated as: L = α L rec +(1 -α ) L est , where α is the trade-off hyperparameter that controls the contribution of sequence reconstruction and accuracy estimation loss.

3.5 Gradient-Ascent Optimal Embedding Search

To conduct the optimal embedding search, we first select topT transformation sequences ranked by the downstream predictive accuracy. The well-trained encoder is then used to embed these postfix expressions into continuous embeddings, which later will be used as seeds (starting points) of gradient ascent. Assuming that one search seed embedding is E , we search, starting from E , toward the gradient direction induced by the evaluator ω : ˜ E = E + η ∂ω ∂ E , where ˜ E denotes the refined embedding, η is the size of each searching step. The model performance of ˜ E is supposed to be better than E due to ω ( ˜ E ) ≥ ω ( E ) . For T seeds, we can obtain the enhanced embeddings [ ˜ E 1 , ˜ E 2 , · · · , ˜ E T ] .

3.6 Transformation Operation Sequence Reconstruction and Evaluation

We reconstruct the transformation sequences by the well-trained decoder ψ using the collected candidate (i.e., acceptable optimal) embeddings [ ˜ E 1 , ˜ E 2 , · · · , ˜ E T ] . This process can be denoted by: [ ˜ E 1 , ˜ E 2 , · · · , ˜ E T ] ψ - → { ˜ Υ i } T i =1 . To identify the best transformation sequence, we adopt the

beam search strategy [11, 12, 13] to generate feature transformation operation sequence candidates. Specifically, given a refined embedding ˜ E , at step-t, we maintain the historical predictions with beam size b , denoted as { Υ i <t } b i =1 . For the i -th beam, the probability distribution of the token identified by the well-trained decoder ψ at the t -th step is is γ , which can be calculated as follows: P i t ( γ ) = P ψ ( γ | ˜ E , ˜ Υ i <t ) ∗ P ψ ( ˜ Υ i <t | ˜ E ) , where the probability distribution P i t ( γ ) is the continued multiplication between the probability distribution of the previous decoding sequence and that of the current decoding step. We can collect the conditional probability distribution of all tokens for each beam. After that, we append tokens with topb probability values to the historical prediction of each beam to get a new historical set { ˜ Υ i <t +1 } b i =1 . We can iteratively conduct this decoding process until confronted with the <EOS> token. We select the transformation sequence with the highest probability value as output. Hence, T enhanced embeddings may produce T transformation sequences { ˜ Υ i } T i =1 . We divide each of them into different parts according to the <SEP> token and check the validity of each part and remove invalid ones. Here, the validity measures whether the mathematical compositions represented by the postfix part can be successfully calculated to produce a new feature. These valid postfix parts reconstruct a feature transformation operation sequence { ˜ Γ i } T i =1 , which are used to generate refined feature space { ˜ X i } T i =1 . Finally, we select the feature set with the highest downstream ML performance as the optimal feature space X ∗ .

4 Experiments

This section reports the results of both quantitative and qualitative experiments that were performed to assess MOAT with other baseline models. All experiments were conducted on AMD EPYC 7742 CPU, and 8 NVIDIA A100 GPUs. For more platform information, please refer to Appendix B.2.

4.1 Datasets and Evaluation Metrics

We used 23 publicly available datasets from UCI [14], LibSVM [15], Kaggle [16], and OpenML [17] to conduct experiments. The 23 datasets involve 14 classification tasks and 9 regression tasks. Table 1 shows the statistics of these datasets. We used F1-score, Precision, Recall, and ROC/AUC to evaluate classification tasks. We used 1-Relative Absolute Error (1-RAE) [4], 1-Mean Average Error (1-MAE), 1-Mean Square Error (1-MSE), and 1-Root Mean Square Error (1-RMSE) to evaluate regression tasks. We used the Valid Rate to evaluate the transformation sequence generation. A valid sequence means it can successfully conduct mathematical compositions without any ambiguity and errors. The valid rate is the average of all correct sequence numbers divided by the total number of generated sequences. The greater the valid rate is, the superior the model performance is. Because it indicates that the model can capture the complex patterns of mathematical compositions and search for more effective feature transformation sequences.

4.2 Baseline Models

We compared our method with eight widely-used feature generation methods: (1) RDG generates feature-operation-feature transformation records at random for generating new feature space; (2) ERG first applies operation on each feature to expand the feature space, then selects the crucial features as new features. (3) LDA [18] is a matrix factorization-based method to obtain the factorized hidden state as the generated feature space. (4) AFAT [2] is an enhanced version of ERG that repeatedly generate new features and use multi-step feature selection to select informative ones. (5) NFS [7] models the transformation sequence of each feature and uses RL to optimize the entire feature generation process. (6) TTG [5] formulates the transformation process as a graph, then implements an RL-based search method to find the best feature set. (7) GRFG [4] uses three collaborated reinforced agents to conduct feature generation and proposes a feature grouping strategy to accelerate agent learning. (8) DIFER [8] embeds randomly generated feature transformation records with a seq2seq model, then employs gradient search to find the best feature set. MOAT and DIFER belong to the same setting. We demonstrate the differences between them in Appendix ?? . Besides, we developed two variants of MOAT in order to validate the impact of each technical component: (i) MOAT -d replaces the RL-based data collection component with collecting feature transformation-accuracy pairs at random. (ii) MOAT -a removes the data augmentation component. We randomly split each dataset into two independent sets. The prior 80% is used to build the continuous embedding space and the remaining 20% is employed to test transformation performance. This experimental setting avoids any test data leakage and ensures a fairer transformation performance comparison. We adopted Random Forest as the downstream machine learning model. Because it is a robust, stable, well-tested method, thus,

* We reported F1-Score for classification tasks, and 1-RAE for regression tasks.

MOAT

0

0.2

0.4

0.6

0.8

1.0

Precision

Recall

F1-Score Valid Rate

MOAT

0

0.2

0.4

0.6

0.8

1.0

Precision

Recall

F1-Score Valid Rate

(b) SpamBase

MOAT

0

0.2

0.4

0.6

0.8

1.0

1-MAE

1-MSE

1-RAE

(c) Openml_616

MOAT

0

0.2

0.4

0.6

0.8

1.0

1-MAE

1-MSE

1-RAE

(d) Openml_618

Figure 4: The influence of data collection (MOAT

-

d

) and data augmentation (MOAT

-

a

) in MOAT.

we can reduce performance variation caused by the model, and make it easy to study the impact of feature space. We provided more experimental and hyperparameter settings in Appendix B.1.

4.3 Performance Evaluation

Overall Performance This experiment aims to answer: Can MOAT effectively generate transformation sequence for discovering optimal feature space with excellent performance? Table 1 shows the overall comparison between MOAT and other models in terms of F1-score and 1-RAE. We noticed that MOAT beats others on all datasets. The underlying driver is that MOAT builds an effective embedding space to preserve the knowledge of feature transformation, making sure the gradient-ascent search module can identify the best-transformed feature space following the gradient direction. Another interesting observation is that MOAT significantly outperforms DIFER and has a more stable performance. There are two possible reasons: 1) The high-quality transformation records produced by the RL-based data collector provide a robust and powerful foundation for building a discriminative embedding space; 2) Postfix notation-based transformation sequence greatly decreases the search space, making MOAT easily capture feature transformation knowledge. Thus, this experiment validates the effectiveness of MOAT.

Data collection and augmentation. This experiment aims to answer: Is it essential to conduct data collection and augmentation to maintain the performance of MOAT ? To achieve this goal, we developed two model variants of MOAT: 1) MOAT -d , which randomly collects feature transformation records for continuous space construction; 2) MOAT -a , which removes the data augmentation step in MOAT. We select two classification tasks (i.e., Spectf and SpamBase) and two regression tasks (i.e., Openml_616 and Openml_618) to show the comparison results. The results are reported in the Figure 4. Firstly, we found that the performance of MOAT is much better than MOAT -d . The underlying driver is that high-quality transformation records collected by the RL-based collector build a robust foundation for embedding space learning. It enables gradient-ascent search to effectively identify the optimal feature space. Moreover, we observed that the performance of MOAT -a is inferior to MOAT. This observation reflects that limited data volume and diversity cause the construction

of embedding space to be unstable and noisy. Thus, this experiment shows the necessity of the data collection and augmentation components in MOAT.

Beam search. This experiment aims to answer What is the influence of beam search for improving the valid rate of generated transformation sequences? To observe the impacts of beam search, we set the beam size as 5 and 1 respectively, and add DIFER as another comparison object. We compare the generation performance in terms of valid rate. Figure 5 shows the comparison results in terms of valid rate. We noticed that the 5-beams search outperforms the 1-beam search. The underlying driver is that the

MOAT (5-Beams)

DIFER

0

20

40

60

80

100

Amazon Employee

AP-Omentum-Ovary

Ionosphere

Lymphography

SpamBase

Credit Default

Wine Quality White

Openml_607

increased beam size can identify more legal and reasonable transformation sequences. Another interesting observation is that DIFER is significantly worse than MOAT and its error bar is longer. The underlying driver is that DIFER collects transformation records at random and it generates transformation sequences using greedy search. The collection and generation ways involve more random noises, distorting the learned embedding space.

Robustness check. This experiment aims to answer: Is MOAT robust to different downstream machine learning models? We replaced the downstream ML models with Random Forest (RF), XGBoost (XGB), Support Vector Machine (SVM), K-Nearest Neighborhood (KNN), Ridge, LASSO, and Decision Tree (DT) to observe the variance of model performance respectively. Table 2 shows the comparison results on Spectf in terms of the F1-score. We observed that MOAT keeps the best performance regard-

RF

XGB

SVM

KNN

LASSO

DT

RDG

0.760

0.818

0.750

0.792

0.718

0.749

0.864

ERG

0.757

0.813

0.753

0.766

0.778

0.750

0.790

LDA

0.665

0.715

0.760

0.749

0.759

0.760

0.665

AFAT

0.760

0.808

0.722

0.759

0.723

0.770

0.844

NFS

0.792

0.799

0.732

0.792

0.744

0.749

0.864

TTG

0.760

0.819

0.765

0.750

0.716

0.749

0.842

GRFG

0.818

0.842

0.580

0.760

0.729

0.744

0.786

DIFER

0.766

0.794

0.727

0.777

0.647

0.744

0.809

MOAT

0.912

0.897

0.876

0.916

0.780

0.844

0.929

less of downstream ML models. A possible reason is that the RL-based data collector can customize the transformation records based on the downstream ML model. Then, the learned embedding space may comprehend the preference and properties of the ML model, thereby resulting in a globally optimal feature space. Thus, this experiment shows the robustness of MOAT.

Time complexity analysis. This experiment aims to answer What is the time complexity of each technical component of MOAT ? To accomplish this, we selected the SOTA model DIFER as a comparison baseline. Figure 3 shows the time costs comparison of data collection, model training, and solution searching in terms of seconds. We let both MOAT and DIFER collect 512 instances in the data collection phase. We found that the increased time costs for MOAT occur in the data collection and model training phases. However, once the model converges, MOAT 's inference time is significantly reduced. The underlying driver is that the RL-based collector spends more time gathering high-quality data, and the sequence formulation for the entire feature space increases the learning time cost for the sequential model. But, during inference, MOAT outputs the entire feature transformation at once, whereas DIFER requires multiple reconstruction iterations based on the generated feature space's dimension. The less inference time makes MOAT more practical and suitable for real-world scenarios.

To thoroughly analyze the multiple characteristics of MOAT, we also analyzed the space complexity (See Appendix C.1), model scalability (See Appendix C.2), and parameter sensitivity (See Appendix C.3). Meanwhile, we provided two qualitative analyses: learned embedding visualization (See Appendix C.4) and traceability case study (See Appendix C.5).

5 Related Works

Automated Feature Transformation (AFT) can enhance the feature space by automatically transforming the original features via mathematical operations [19, 20]. Existing works can be divided into three categories: 1) expansion-reduction based approaches [1, 3, 21, 2, 22]. Those methods first expand the original feature space by explicitly [23] or greedily [24] decided mathematical transformation, then reduce the space by selecting useful features. However, they are hard to produce or evaluate both complicated and effective mathematical compositions, leading to inferior performance. 2) evolution-evaluation approaches [4, 5, 6, 25, 9, 26]. These methods integrate feature generation and selection into a closed-loop learning system. They iteratively generate effective features and keep the significant ones until they achieve the maximum iteration number. The entire process is optimized by evolutionary algorithms or reinforcement learning models. However, they still focus on how to simulate the discrete decision-making process in feature engineering. Thus, they are still time-consuming and unstable. 3) Auto ML-based approaches [7, 8]. Auto ML aims to find the most suitable model architecture automatically [27, 28, 29, 30]. The success of auto ML in many area [31, 32, 33, 34, 35, 36] and the similarity between auto ML and AFT inspire researchers to formulate AFT as an auto ML task to resolve. However, they are limited by: 1) incapable of producing high-order feature transformation; and 2) unstable transformation performance. Recent studies tried to convert the feature transformation problem into a continuous space optimization task to search for the optimal feature space efficiently. DIFER [8] is a cutting-edge method that is comparable to our work in problem formulation. However, the following constraints limit DIFER's practicality: 1) DIFER collects transformation-accuracy data at random, resulting in many invalid training data with inconsistent transformation performances; 2) DIFER embeds and reconstructs each transformed feature separately and, thus, ignores feature-feature interactions; 3) DIFER needs to manually decide the number of generated features, making the reconstruction process more complicated. 4) the greedy search for transformation reconstruction in DIFER leads to suboptimal transformation results. To fill these gaps, we first implement an RL-based data collector to automate high-quality transformation record collection. We then leverage the postfix expression idea to represent the entire transformation operation sequence to model feature interactions and automatically identify the number of reconstructed features. Moreover, we employ beam search to advance the robustness, quality, and validity of transformation operation sequence reconstruction.

6 Conclusion Remarks

In this paper, we propose an automated feature transformation framework, namely MOAT. In detail, we first develop an RL-based data collector to gather high-quality transformation-accuracy pairs. Then, we offer an efficient postfix-based sequence expression way to represent the transformation sequence in each pair. Moreover, we map them into a continuous embedding space using an encoderdecoder-evaluator model structure. Finally, we employ a gradient-ascent search to identify better embeddings and then use beam search to reconstruct the transformation sequence and identify the optimal one. Extensive experiments show that the continuous optimization setting can efficiently search for the optimal feature space. The RL-based data collector is essential to keep an excellent and stable transformation performance. The postfix expression sequence enables MOAT to automatically determine the transformation depth and length, resulting in more flexible transformation ways. The beam search technique can increase the validity of feature transformation. The most noteworthy research finding is that the success of MOAT indicates that the knowledge of feature transformation can be embedded into a continuous embedding space to search for better feature space. Thus, it inspires us to regard the learning paradigm as the backbone to develop a large feature transformation model and quickly fine-tune it for different sub-tasks, which is also our future research direction.

7 Acknowledgments

This research was partially supported by the National Science Foundation (NSF) via grant numbers: 2040950, 2006889, and 2045567.

References

James Max Kanter and Kalyan Veeramachaneni. Deep feature synthesis: Towards automating data science endeavors. In 2015 IEEE international conference on data science and advanced analytics (DSAA) , pages 1-10. IEEE, 2015.

Franziska Horn, Robert Pack, and Michael Rieger. The autofeat python library for automated feature engineering and selection. arXiv preprint arXiv:1901.07329 , 2019.

Udayan Khurana, Deepak Turaga, Horst Samulowitz, and Srinivasan Parthasrathy. Cognito: Automated feature engineering for supervised learning. In 2016 IEEE 16th International Conference on Data Mining Workshops (ICDMW) , pages 1304-1307. IEEE, 2016.

Dongjie Wang, Yanjie Fu, Kunpeng Liu, Xiaolin Li, and Yan Solihin. Group-wise reinforcement feature generation for optimal and explainable representation space reconstruction. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , KDD '22, page 1826-1834, New York, NY, USA, 2022. Association for Computing Machinery.

Udayan Khurana, Horst Samulowitz, and Deepak Turaga. Feature engineering for predictive modeling using reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 32, 2018.

Binh Tran, Bing Xue, and Mengjie Zhang. Genetic programming for feature construction and selection in classification on high-dimensional data. Memetic Computing , 8(1):3-15, 2016.

Xiangning Chen, Qingwei Lin, Chuan Luo, Xudong Li, Hongyu Zhang, Yong Xu, Yingnong Dang, Kaixin Sui, Xu Zhang, Bo Qiao, et al. Neural feature search: A neural architecture for automated feature engineering. In 2019 IEEE International Conference on Data Mining (ICDM) , pages 71-80. IEEE, 2019.

Guanghui Zhu, Zhuoer Xu, Chunfeng Yuan, and Yihua Huang. Difer: differentiable automated feature engineering. In International Conference on Automated Machine Learning , pages 17-1. PMLR, 2022.

Meng Xiao, Dongjie Wang, Min Wu, Ziyue Qiao, Pengfei Wang, Kunpeng Liu, Yuanchun Zhou, and Yanjie Fu. Traceable automatic feature transformation via cascading actor-critic agents. Proceedings of the 2023 SIAM International Conference on Data Mining (SDM) , pages 775-783.

Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation , 9(8):1735-1780, 1997.

Markus Freitag and Yaser Al-Onaizan. Beam search strategies for neural machine translation. arXiv preprint arXiv:1702.01806 , 2017.

Thibault Viglino, Petr Motlicek, and Milos Cernak. End-to-end accented speech recognition. In Interspeech , pages 2140-2144, 2019.

Chunhui Bao and Qianru Sun. Generating music with emotions. IEEE Transactions on Multimedia , 2022.

Public. Uci dataset download. [EB/OL], 2022. https://archive.ics.uci.edu/ .

Lin Chih-Jen. Libsvm dataset download. [EB/OL], 2022. https://www.csie.ntu.edu.tw/ ~cjlin/libsvmtools/datasets/ .

Jeremy Howard. Kaggle dataset download. [EB/OL], 2022. https://www.kaggle.com/ datasets .

Public. Openml dataset download. [EB/OL], 2022. https://www.openml.org .

David M Blei, Andrew Y Ng, and Michael I Jordan. Latent dirichlet allocation. the Journal of machine Learning research , 3:993-1022, 2003.

Yi-Wei Chen, Qingquan Song, and Xia Hu. Techniques for automated machine learning. ACM SIGKDD Explorations Newsletter , 22(2):35-50, 2021.

Andrew Kusiak. Feature transformation methods in data mining. IEEE Transactions on Electronics packaging manufacturing , 24(3):214-221, 2001.

Hoang Thanh Lam, Johann-Michael Thiebaut, Mathieu Sinn, Bei Chen, Tiep Mai, and Oznur Alkan. One button machine for automating feature engineering in relational databases. arXiv preprint arXiv:1706.00327 , 2017.

Udayan Khurana, Fatemeh Nargesian, Horst Samulowitz, Elias Khalil, and Deepak Turaga. Automating feature engineering. Transformation , 10(10):10, 2016.

Gilad Katz, Eui Chul Richard Shin, and Dawn Song. Explorekit: Automatic feature generation and selection. In 2016 IEEE 16th International Conference on Data Mining (ICDM) , pages 979-984. IEEE, 2016.

Ofer Dor and Yoram Reich. Strengthening learning algorithms by feature discovery. Information Sciences , 189:176-190, 2012.

Guanghui Zhu, Shen Jiang, Xu Guo, Chunfeng Yuan, and Yihua Huang. Evolutionary automated feature engineering. In PRICAI 2022: Trends in Artificial Intelligence: 19th Pacific Rim International Conference on Artificial Intelligence, PRICAI 2022, Shanghai, China, November 10-13, 2022, Proceedings, Part I , pages 574-586. Springer, 2022.

Meng Xiao, Dongjie Wang, Min Wu, Kunpeng Liu, Hui Xiong, Yuanchun Zhou, and Yanjie Fu. Traceable group-wise self-optimizing feature transformation learning: A dual optimization perspective. arXiv preprint arXiv:2306.16893 , 2023.

Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. The Journal of Machine Learning Research , 20(1):1997-2017, 2019.

Yaliang Li, Zhen Wang, Yuexiang Xie, Bolin Ding, Kai Zeng, and Ce Zhang. Automl: From methodology to application. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management , pages 4853-4856, 2021.

Xin He, Kaiyong Zhao, and Xiaowen Chu. Automl: A survey of the state-of-the-art. KnowledgeBased Systems , 212:106622, 2021.

Shubhra Kanti Karmaker, Md Mahadi Hassan, Micah J Smith, Lei Xu, Chengxiang Zhai, and Kalyan Veeramachaneni. Automl to date and beyond: Challenges and opportunities. ACM Computing Surveys (CSUR) , 54(8):1-36, 2021.

Ziwei Zhang, Xin Wang, and Wenwu Zhu. Automated machine learning on graphs: A survey. arXiv preprint arXiv:2103.00742 , 2021.

Marcel Wever, Alexander Tornede, Felix Mohr, and Eyke Hüllermeier. Automl for multi-label classification: Overview and empirical evaluation. IEEE transactions on pattern analysis and machine intelligence , 43(9):3037-3054, 2021.

Maroua Bahri, Flavia Salutari, Andrian Putina, and Mauro Sozio. Automl: state of the art with a focus on anomaly detection, challenges, and research directions. International Journal of Data Science and Analytics , 14(2):113-126, 2022.

Dakuo Wang, Josh Andres, Justin D Weisz, Erick Oduor, and Casey Dugan. Autods: Towards human-centered automation of data science. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems , pages 1-12, 2021.

Meng Xiao, Dongjie Wang, Min Wu, Pengfei Wang, Yuanchun Zhou, and Yanjie Fu. Beyond discrete selection: Continuous embedding space optimization for generative feature selection. arXiv preprint arXiv:2302.13221 , 2023.

Wangyang Ying, Dongjie Wang, Kunpeng Liu, Leilei Sun, and Yanjie Fu. Self-optimizing feature generation via categorical hashing representation and hierarchical reinforcement crossing. arXiv preprint arXiv:2309.04612 , 2023.

Appendix

A Postfix Transformation Conversion

Algorithm 1 is the pseudo-code for the convert the original feature transformation sequence Γ to the postfix notation based sequence Υ . In detail, we first initialize a list Υ and two stacks S 1 and S 2 , respectively. For each element τ in Γ , we scan it from left to right. When getting a feature ID token, we push it to S 2 . When receiving a left parenthesis, we push it to S 1 . When obtaining any operations, we pop each element in S 1 and push them into S 2 until the last component of S 1 is the left bracket. Then we push this operation into S 1 When getting a right parenthesis, we pop every element from S 1 and then push into S 2 until we confront a left bracket. Then we remove this left bracket from the top of S 1 . When the end of the input τ encounters, we append every token from S 2 into Υ . If this τ is not the last element in Γ , we will append a <SEP> token to indicate the end of this τ . After we process every element in Γ , we add <SOS> and <EOS> tokens to the beginning and end of Υ to form the postfix notation-based transformation sequence Υ = [ γ 1 , · · · , γ M ] . Each element in Υ is a feature ID token, operation token, or three other special tokens. We convert each transformation sequence through this algorithm and construct the training set with them.

B Experimental Settings and Reproducibility

B.1 Hyperparameter Settings and Reproducibility

The operation set consists of square root, square, cosine, sine, tangent, exp, cube, log, reciprocal, quantile transformer, min-max scale, sigmoid, plus, subtract, multiply, divide . For the data collection part, we ran the RL-based data collector for 512 epochs to collect a large amount of feature transformation-accuracy pairs. For the data augmentation part, we randomly shuffled each transformation sequence 12 times to increase data diversity and volume. We adopted a single-layer LSTM

as the encoder and decoder backbones and utilized 3-layer feed-forward networks to implement the predictor. The hidden state sizes of the encoder, decoder and predictor are 64, 64, and 200, respectively. The embedding size of each feature ID token and operation token was set to 32. To train MOAT, we set the batch size as 1024, the learning rate as 0.001, and λ as 0.95 respectively. For inferring new transformation sequences, we used top-20 records as the seeds with beam size 5.

B.2 Experimental Platform Information

All experiments were conducted on the Ubuntu 18.04.6 LTS operating system, AMD EPYC 7742 CPU, and 8 NVIDIA A100 GPUs, with the framework of Python 3.9.10 and PyTorch 1.8.1.

C Experimental Results

C.1 Space Complexity Analysis.

To analyze the space complexity of MOAT, we illustrate the parameter size of MOAT when confronted with different datasets. Table 4 shows the comparison results. We can find that the model size of MOAT keeps relatively stable without significant fluctuations. The underlying driver is that the encoder-evaluator-decoder learning paradigm can embed the knowledge of discrete sequences with variant lengths into a fixed-length embedding vector. Thus, such an embedding process can make the parameter size to be stable instead of increasing as the data size grows. Thus, this experiment indicates that MOAT has good scalability when confronted with different scaled datasets.

C.2 Scalability Check

We visualized the changing trend of the time cost of searching for better feature spaces over sample size and feature dimensions of different datasets. Figure 6 shows the comparison results. We found that the time cost of MOAT keeps stable with the increase of sample size of the feature set. A possible reason is that MOAT only focuses on the decision-making benefits of feature ID and operation tokens instead of the information of the entire feature set, making the searching process sample size irrelevant. Another interesting observation is that the search time is still stable although the feature dimension of the feature set varies significantly. A possible explanation is that we map transformation records of varying lengths into a continuous space with a constant length. The searching time in this space is input dimensionality-agnostic. Thus, this experiment shows the MOAT has excellent scalability.

Search Time

Sample Size

Time Cost (s)

0

50

100

Number

0

20,000

40,000

Credit Default

SpamBase

PimaIndian

Ionosphere

Lymphography

(a) Sample Size

Search Time

Feature Num.

Time Cost (s)

0

50

100

Number

20

40

60

SpamBase

Openml_616

Ionosphere

Lymphograpy

PimaIndian

(b) Feature Number

η

=1

η

=2

η

=3

η

=4

η

=5

η

=6

η

=7

η

=8

η

=9

η

=10

0.73

0.73

0.73

0.73

0.73

0.73

0.73

0.73

0.73

0.73

0.8

0.8

0.8

0.8

0.8

0.8

0.8

0.8

0.8

0.8

0.86

0.86

0.86

0.86

0.86

0.86

0.86

0.86

0.86

0.86

0.92

0.92

0.92

0.92

0.92

0.92

0.92

0.92

0.92

0.92

0.98

0.98

0.98

0.98

0.98

0.98

0.98

0.98

0.98

0.98

Precision

Recall

F1-Score

(a) Search Step Size

η

(under

α

= 0

.

05

)

α

=0.05

α

=0.10

α

=0.15

α

=0.20

α

=0.25

α

=0.30

α

=0.35

α

=0.40

α

=0.45

α

=0.50

0.87

0.87

0.87

0.87

0.87

0.87

0.87

0.87

0.87

0.87

0.89

0.89

0.89

0.89

0.89

0.89

0.89

0.89

0.89

0.89

0.9

0.9

0.9

0.9

0.9

0.9

0.9

0.9

0.9

0.9

0.92

0.92

0.92

0.92

0.92

0.92

0.92

0.92

0.92

0.92

0.94

0.94

0.94

0.94

0.94

0.94

0.94

0.94

0.94

0.94

Precision

Recall

F1-Score

(b) Training Trade-off

α

(under

η

= 1

)

-

40

-

30

-

20

-

10

0

10

20

30

40

-

40

-

30

-

20

-

10

0

10

20

30

(a) Openml_616

-

50

-

40

-

30

-

20

-

10

0

10

20

30

40

-

30

-

20

-

10

0

10

20

30

(b) Airfoil

C.3 Parameter sensitivity analysis

To validate the parameter sensitivity of the search step size η (See section 3.5) and the trade-off parameter α in the training loss (See section 3.4), we set the value of η from 1 to 10, and set the value of α from 0.05 to 0.50 to observe the difference. Figure 7 shows the comparison results in terms of precision, recall, and F1-score. When the search step size grows, the downstream ML performance initially improves, then declines marginally. A possible reason for this observation is that a too-large step size may make the gradient-ascent search algorithm greatly vibrate in the continuous space, leading to missing the optimal embedding point and transformed feature space. Another interesting observation is that the standard deviation of the model performance is lower than 0 . 01 under different parameter settings. This observation indicates that MOAT is not sensitive to distinct parameter settings. Thus, the learning and searching process of MOAT is robust and stable.

C.4 Learned embedding Analysis.

We selected Airfoil and Openml_616 as examples to visualize their learned continuous embedding space. In detail, we first collected the latent embeddings generated by the transformation records. Then, we use T-SNE to map them into a 2-dimensional space for visualization. Figure 8 shows

F1-Score:0.456

[pH]

[fixed acidity]

[citric acid]

[residual sugar]

[density]

[sulphates]

[volatile acidity]

[chlorides]

[total sulfur dioxide]

[alcohol]

Original Feature Set

F1-Score:0.559

[residual sugar]

qTrans(...) - [...] - qTrans(...)

[density]

([alcohol]

×

(...) - (...)

×

(...)

(...) - qTrans(sin([alcohol] - [...])

(((minmax([...]))

×

[...]) - (...)

[chlorides]

[alcohol]

×

(...)-

(...)

×

(...)

[alcohol]

[volatile acidity] - [alcohol]

the visualization results, in which each point represents a unique feature transformation sequence. The size of each point means its downstream performance. The bigger point size indicates that the downstream performance is superior. We colored the top 20 embedding points according to the performance in red. We found that the distribution locations of the top 20 embedding points are different. A potential reason is that the corresponding transformation sequences of the top 20 embedding points are different lengths. The sequence reconstruction loss distributes them to different areas of the embedding space. Moreover, we observed that the top 20 embedding points are close in the space even though the positions are different. The underlying driver is that the estimation loss makes these points with good performance clustered. Thus, this case study reflects that the reconstruction loss and estimation loss make the continuous space associate the transformation sequence and the corresponding model performance.

C.5 Traceability case study

We selected the top 10 essential features for prediction in the original, and MOAT transformed feature space of the Wine Quality Red dataset for comparison. Figure 9 shows the comparison results. The texts associated with each pie chart are the corresponding feature name. The larger the pie area is, the more critical the feature is. We found that almost 70% critical features in the new feature space are generated by MOAT and they improve the downstream ML performance by 22.6%. This observation indicates that MOAT really comprehends the properties of the feature set and ML models in order to produce a more effective feature space. Another interesting finding is that '[alcohol]' is the essential feature in the original feature set. But MOAT generates more mathematically composited features using '[alcohol]'. This observation reflects that MOAT not only can capture the significant features but also produce more effective knowledge for enhancing the model performance. Such composited features can make domain experts trace their ancestor resources and summarize new analysis rules for evaluating the quality of red wine.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nFeature transformation is an effective way to improve downstream task. This paper aims to improve the search efficiency of the optimal feature space while ensuring the stability and robustness of the transformation. They formulate the discrete Automated Feature Transformation (AFT) problem as a continuous optimization task and propose a reinforcement-enhanced autoregressive feature Transformation framework (MOAT). MOAT implements four steps to advance efficiency and robustness. Extensive experiments and case studies are performed to demonstrate the effectiveness and robustness of the proposed method.\n\nSTRENGTHS:\nThis paper provides a clear and detailed illustration of the proposed framework, especially the modules in the framework. They conduct extensive experiments to demonstrate the superiority of the framework, including ablation studies for each module, to demonstrate the efficiency and robustness of the framework.\n\nWEAKNESSES:\nMissing reference to experimental results on page 8, Data collection and augmentation section.\nSeveral methods have used reinforcement learning to improve search efficiency at present (e.g., in data augmentation, NAS), and the framework proposed in this paper lacks novelty to some extent.\n\nQUESTIONS:\n(1) MOAT uses multi-stage process to obtain a convincing performance. Is it possible to learn an optimal feature space using reinforcement learning by end-to-end?\n(2) Does the optimal feature space searching problem can be directly solved by representation learning in continuous space, rather than combining multiple operations in discrete space？\n\nLIMITATIONS:\nThis paper pays more attention to framework design and lacks some theoretical perspectives. In addition, how to design the framework proposed in this paper into an end-to-end training process is desirable.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a succinct yet effective framework for automatic feature transformation. The authors mapped decision sequences collected from reinforcement learning into a high-dimensional vector space. Optimization is carried out through the gradient direction provided by an evaluator, and a sequence reconstruction component is employed to rebuild the decision sequence of this feature transformation. Overall, the paper's experiments are comprehensive, and the discussions provide excellent validation of various features of the framework. The comparisons with existing models also aptly place this work among the latest contributions in the field.\n\nSTRENGTHS:\n1. The design of the entire framework is clear and comprehensible, and based on the authors' descriptions and experimental discussions, this framework also shows potential for reuse in other fields.\n\n2. The experimental design is detailed and particularly addresses my concerns about the time-consuming nature of using reinforcement learning for data collection.\n\n3. Similar to the second strength described above, a significant issue with using reinforcement learning in feature transformation tasks is that it might require an excessive number of search steps to achieve optimal. However, this framework can use a fixed number of search steps to construct the space for searching and achieve respectable downstream task performance, significantly mitigating the problem of uncertain duration in feature transformation tasks based on reinforcement learning.\n\nWEAKNESSES:\n1. There are a few typos in this paper, e.g., the legend of the figure 4, the model name is wrong.\n\n2. The authors have proposed an encoder-decoder model based on LSTM for sequence modeling. However, there is no discussion on why this specific method was chosen. It would be beneficial if the authors could explain the rationale behind this choice. Could other sequence modeling methods, such as Transformers, have been considered or used instead?\n\n3. While the design of the model and ample experiments mutually validate the feasibility of this framework, if there could be more in-depth insights provided, it might help establish this work as foundational for broader fields.\n\nQUESTIONS:\n1. The framework's description of the data collection part for the optimization objective lacks clarity. The authors should consider adding more detail in the appendix to better elucidate this aspect.\n\n2. Are there any way to make the framework more generalized, e.g., extend this method to graph-like data, picuture-like data.\n\n3. Would it be feasible to design a more generalized operation set that views these mathematical operations as analogous to sub-networks?\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a feature transformation method that reformulates the problem as a continuous space optimization task and utilizes a reinforcement-enhanced autoregressive framework for gradient-steered search. The method involves four steps: (1) reinforcement-enhanced data preparation, (2) feature transformation operation sequence embedding, (3) gradient-steered optimal embedding search, and (4) transformation operation sequence reconstruction. The proposed method is evaluated through extensive experiments and case studies, demonstrating its effectiveness and robustness.\n\nSTRENGTHS:\n1. This paper proposes a novel automated feature transformation framework that has not been explored before, and it achieves significant improvement in performance compared to previous methods.\n\n2. This paper provides clear and detailed explanations of the proposed method's four-step process, making it easy for readers to follow and replicate the method.\n\n3. This paper conducts extensive experiments and case studies to illustrate the effectiveness of the framework from different perspectives.\n\n4. The authors released the related code and data, which can help other researchers reproduce the experiments.\n\n5. This paper provides clear and detailed explanations of the proposed method's four-step process, making it easy for readers to follow and replicate the method.\n\nWEAKNESSES:\n1. The authors use LSTM as the backbone of their framework. Any reason for choosing LSTM? How about other alternatives, such as Transformer?\n\n2. There are some typos in this paper. For instance, in Figure 4 (c) and Figure 4 (d), the name of the model variant should be MOAT^-a instead of GBFG^-a. The authors should fix these typos for keeping consistency.\n\n3. Analyzing the experimental results, the RL-based data collector is important. But, the description for this part is limited. Can the authors provide more explanation on this?\n\nQUESTIONS:\n1. It is unclear why the performance of DIFER is not good for openml 616, openml 637. Can you provide some explanation on this?\n\n2. After reviewing the appendix section, I found that MOAT requires more training time but less inference time than DIFER. Can the authors explain the reason for it?\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nDistinct from existing work, this study collects feature transformation operation sequences, which have a well-researched background based on reinforcement learning. It then obtains hidden representations of these limited numbers of sequences in a self-supervised manner and finally optimizes this continuous vector representation to generate superior sequences guided by gradients. Specifically, to model the sequences, the authors propose an LSTM-based encoder-decoder-evaluator architecture with just a few collected samples. Detailed runtime analysis was conducted in this work to demonstrate its advantages over reinforcement learning-based (GRFG) and random generation-based (DIFER) approaches. The experiments, coupled with the authors' analytical discussions, effectively substantiate the viewpoints declared in the paper.\n\nSTRENGTHS:\n1) This paper is well-written and easy to understand, with an appropriate level of detail in describing the methods. It considers the primary issue, time consumption, associated with this framework in terms of experimental design, effectively addressing the readers' concerns.\n\n2) The paper's model architecture is thoughtfully designed, and the authors articulate their motivations for each component with great clarity. The streamlined nature of the entire framework demonstrates the depth of the authors' understanding and thoughtful consideration of this work.\n\n3) The authors provide thorough experiments and analysis for this work, covering aspects of interest in this field such as performance, robustness, runtime, scalability, memory usage, traceability, and efficacy tests for each component. The visualization of the hidden space well illustrates the reasons for the model's effectiveness in conducting gradient search.\n\nWEAKNESSES:\n1) In the sections 2.1 Important Definitions and Section 3.2 Reinforcement Training Data Preparation, the authors have inconsistencies in the naming of the agents within the cascading agent structure for the data collection part. For instance, it's named as 'feature agent1' in the definition section, while it's referred to as 'head feature agent' in the methodology part. The authors should maintain consistency in these definition names to reduce confusion for the readers.\n\n2) The description of the optimization objective for reinforcement learning in the data collection section is not detailed enough. The authors should provide a more comprehensive description of this critical component.\n\nQUESTIONS:\n1) Why does DIFER require more time in the inference phase? The authors should provide a detailed explanation for this.\n\n2) In the time complexity analysis, the authors have chosen four datasets - Wine Red, Wine White, Openml_618, and OpenML_589. However, they have not adequately described the rationale behind selecting these particular datasets.\n\n3) There are some avoidable writing errors, such as on page 13 of the appendix where the authors incorrectly referred to Table 3 as Figure 3. The authors should conduct a thorough check of the entire paper to prevent inconsistencies in naming.\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}