Sculpting Features from Noise: Reward-Guided Hierarchical Diffusion for Task-Optimal Feature Transformation
Authors: Nanxu Gong
Zijun Li
Sixun Dong
Haoyue Bai
Wangyang Ying
Xinyuan Wang
Yanjie Fu
Venue: NeurIPS 2025 poster ()
Keywords: Feature Transformation
Diffusion Model
Reward-Guided Generation
Primary area: applications

Abstract: Feature Transformation (FT) crafts new features from original ones via mathematical operations to enhance dataset expressiveness for downstream models. However, existing FT methods exhibit critical limitations: discrete search struggles with enormous combinatorial spaces, impeding practical use; and continuous search, being highly sensitive to initialization and step sizes, often becomes trapped in local optima, restricting global exploration. To overcome these limitations, DIFFT redefines FT as a reward-guided generative task. It first learns a compact and expressive latent space for feature sets using a Variational Auto-Encoder (VAE). A Latent Diffusion Model (LDM) then navigates this space to generate high-quality feature embeddings, its trajectory guided by a performance evaluator towards task-specific optima. This synthesis of global distribution learning (from LDM) and targeted optimization (reward guidance) produces potent embeddings, which a novel semi-autoregressive decoder efficiently converts into structured, discrete features, preserving intra-feature dependencies while allowing parallel inter-feature generation. Extensive experiments on 14 benchmark datasets show DIFFT consistently outperforms state-of-the-art baselines in predictive accuracy and robustness, with significantly lower training and inference times.

Sculpting Features from Noise: Reward-Guided Hierarchical Diffusion for Task-Optimal Feature
Transformation
Nanxu Gong', Li:; Sixun Dong', Haoyue Bai', Wangyang Ying',
Xinyuan Wang'. Yanjie Fu
'Arizona State University National University of Singapore, nanxugong@outtooo. com, com, zijunli@u. wws. edu, (sixundong, haoyuebal, wying4, xwang735, yanjie fu) @asu. edu,
Abstract
Feature Transformation (FT) crafts new features from original ones via mathemati- cal operations enhance dataset expressiveness for downstream models. However, existing FT methods exhibit critical limitations: discrete search struggles with enormous combinatorial spaces, impeding practical use; and continuous search, being highly sensitive to initialization and step sizes, often becomes trapped in local optima, restricting global exploration. To overcome these limitations, DIFFT redefines FT expressive latent space for feature sets using a Variational Latent Diffusion Model (LDM) then navigates this space to generate high-quality feature embeddings, its trajectory guided by a performance evaluator towards task-specific optima. This synthesis of global distribution learning (from LDM) and targeted optimization (reward guidance) produces potent embeddings, which novel semi-autoregressivve decoder efficiently converts into structured, discrete features, preserving intra-fcature dependencies while allowing parallel inter-feature generation. Extensive experiments 14 benchmark datasets show DIFFT consis- tently outperforms -tate-of-the-ss baselines in predictive accuracy and robustness, with significantly lower training and inference times. Our code and data are publicly available at https: //gi gighub..cc. com/ NanxuGong/DIFFT
eeward-guided generative task. It first learns compact and
(VAE). A
Introduction
In real-world practices, raw data features often contain imperfections and complex interdependencies that hinder model performance. To address the challenges, Feature Transformation (FT) has been widely used, because it can reconstruct distance measures, reshape discriminative patterns, and enhance data AI readiness (structural, predictive, interaction, and expression levels). Given a raw feature set F [fi, f2...., . fx] and operator set O (c.g., log, +, *), feature transformation aims to apply mathematical operations to original features and construct a better feature set, for instance, [fa f2, log fa. fa fa]. In this paper, study the FT problem that aims to apply mathematical operations to cross original features and create a better feature set.
In prior literature, manual transformation heavily relies on domain knowledge with limited general- ization. Discrete search methods explore feature-operator combinations using reinforcement learning, evolutionary algorithms, and genetic programming 6 29 26 7]. However, these approaches suffer
"Equal contribution Corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Feature Transformation Methods
Original Sequence
foree
Agest
Agrer
Optimal Sequence
Feature sequence
(1)
(ii)
Feature sequance
fz
Fu
Discrete Search
Confinuous Search Local search X
Reward- Guided Generation
Large search space X
Global sampling of
Figure 1: Motivation example. Discrete search methods explore various feature combinations directly, but often challenged by the sheer scale of the resulting combinatorial space. Continuous search methods, on the other hand, iteratively refine solutions from initial or current points, yet frequently converge local optima. In contrast, our proposed reward-guided generation paradigm, by leveraging the global sampling ability of diffusion models, aims to discover solutions closer to the global optimum.
from the large combinatorial discrete search space. With the rise of GenAl and the concept of any modality as language tokens, researchers see feature sets token sequences. For instance, transformed feature set [/i f2, log f3, f4 + fs] can be represented by postfix expression: "f1 f2 *, f3 log, f4 fs -" to reduce sequence length and ambiguity. FT therefore is reduced into a GenAl task: learning to generate a task-optimal feature token sequence given tasking data. Existing solutions follow an ambedding-searchh -eaurcgggnertiion paradigm [27 31 28 i) learning feature token sequence embeddings; ii) gradient ascent of optimal feature set embeddings: iii) autoregressive decoding of identified embeddings into transformed tables. The key idea is gradient search of optimal feature set embedding in continuous embedding space, instead of search in the large discrete space.
Although the continuous search paradigm of generative FT introduces computational benefits, there are two major challenges. First, the continuous gradient search can get stuck in a local optima and is sensitive initialization and search boundaries, leading to low robustness. A question that triggers our curiosity is: how can we better learn a generalized representation space describing all high or low-performance feature transformations on tasking data? Second, autoregressive decoding assumes that two consecutive features in a sequence are dependent, and, thus, considers redundant contexts and long attentions and time-costly. For instance, while the tokens in one transformed feature fi * fa are interdependent, fi * fa is not dependent on another transformed feature (og(fs). Another interesting question is: how can we enhance generative efficiency without compromising feature transformation embedding decoding quality? Thus, we need a new learning paradigm that goes static embedding space.
beyond the continuous search paradigm
The Opportunity. As one of the generative frameworks, diffusion models learn the data distribution by training on forward noising process that incrementally adds noise to input data and reverse denoising process that reconstructs the data step-by-step. in order to capture the probability landscape of the data. Once trained, the reverse diffusion process can be guided guided-either conditionally or through optimization cues to sample the most probable or desirable data points from the learned distribution in generative fashion. This offers a compelling alternative to continuous gradient search.
Our Insights: an integrated reward-guided diffusion and hierarchical decoding paradigm. We formulate generative FT as diffusion generation task, instead of gradient search task. To overcome the limitations of gradient search, our first insight is optimization as reward- reward-guided diffusion generation. The idea is leverage forward diffusion and reverse diffusion learn the global and generalizable distribution of feature set embedding space, then leverage reward (gradient from performance evaluator) to guide the feature set embedding generation towards performance maximization. The reward guidance is interpreted as aligning diffusion trajectories with reward optimization paths to combine global distribution with local refinement. To overcome the limitations of autoregressive decoding, we propose a hierarchical semi-autoregressive decoding method. We divide a feature token sequence into multiple chunks (c.g.. "f1 f2 *, f' log, fa fs +" has 3 chunks), each of which is one transformed feature (e.g., "f1 f2 *" is one feature). We adopt hierarchical strategy to firstly predict how many feature chunks to generate and then autoregress the token sequence of each chunk. This can enable parallel decoding to improve efficiency of decoding a feature token sequence for transformation. Collectively, these insights allow us to frame FT not as a search problem within a discrete or continuous space, but as guided generative task. Figure [] visually encapsulates this paradigm shift. contrasts our diffusion-basee generation approach with prior discrete and continuous search methodologies, highlighting how leveraging the global sampling

---

A. Variational Auto-Encoder and Evaluator Construction
B. Diffusion Model Construction
One-stup deredi
Enteeding Spoce
Evalvator
Toble
Festure sequence
0.96
Performace
&
Fechure
DIT
pesoder
Reward-Guided Generation
Decoter
Enalvstir
Feoture
Goussion
Condition
signal
Optinal feature sequerce
Geni-seferagreedee generation
DET
Denoke
Decoder
Figure 2: Framework overview. The framework consists of three key components: 1) VAE that encodes feature sequences into latent embeddings via semi-autoregressive decoder; 2) LDM trained to model the distribution of effective embeddings conditioned on tabular semantics; and 3) reward-guided sampling process that leverages gradients from a performance evaluator to steer the generation of high-quality feature embeddings, which are then decoded into final feature sets.
capabilities of diffusion and reward guidance can overcome the inherent limitations of large search spaces and local optima that challenged previous techniques.
Summary of Proposed Approach. Inspired by these insights, we propose Reward-Guide Diffu- sion Framework for Semi- mumiiAutoregressvve Generative FT (DIFFT), Our method includes a Variational Auto-Encoder (VAE) that learns compact embeddings of feature token sequences, evaluator net- work that guides the sampling process during inference in reverse diffusion, and Latent Diffusion Model (LDM) that generates task-optimal feature embeddings within the learned latent space. In particular, given tasking data, we explore and collect the performance of different transformed feature sets (represented by feature token sequences) as training data. The VAE encoder is trained to encode these feature token sequences into embedding vectors, while the VAE decoder predicts the number of features in the output feature token sequence, then autoregresses each transformed feature in step-by step-by-step by-eptp manner. After training the VAE. the feature token sequence embedding vectors of training data are exploited learn the LDM as a denoising generative model in the embedding space. After training the LDM, we extract gradient information from the evaluator that estimates the performance of feature set sequence embeddings guide the reverse denoising diffusion generate (a.k.a, sample) task-optima feature token sequence embeddings. Finally, we use the VAE decoder to decode the opstmal embedding into transformed feature set to augment input data.
Our Contributions. 1) We tackle FT as GenAl task and develop an integrated reward-guide diffusion and hierarchical decoding paradigm. 2) We convert task-optimal FT search evaluator- steered reverse diffusion, in which diffusion models learn feature set embedding distribution, then conduct generative sampling of the desirable embedding. 3) We design semi- autoregressvve decoding strategy (i.e.. predicting feature number then generating token sequence of each feature) to parallelize decoding and advance efficiency.
Diffusion Feature Transformation
2.1 Framework Overview
Figure shows that our method has three core components: (1) a Variational Auto-Encoder for feature knowledge learning: (2) Latent Diffusion Model for embedding generation; and (3) reward-guided sampling mechanism for performance-aware feature transformation. In step 1, we construct VAE with an encoder that maps discrete feature sequences into a compact latent space, and a semi-autee autmiiautoeeressivvee decoder that reconstructs each feature sequence by first predicting the number of features and then generating each feature autoregressively. Additionally, we train performance evaluator to predict the utility of latent embeddings for downstream tasks. In step 2, we use the VAE encoder to obtain latent representations of training feature sets and train an LDM as a denoising generative model to capture the distribution of effective embeddings. In step 3, during inference, we incorporate the evaluator's gradients into the denoising process, guiding the diffusion model to generate embeddings that are optimized for downstream performance. The generated embeddings
3

---

then decoded via the semi semi-autoregressive decoder to produce high-quality feature sets. We provide more information on important concepts in Appendix A
2.2 Embedding: Learning the Embedding Space of Feature Sets on Tasking Data
We construct a representation learning model (i.e., a variational probabilistic autoencoder structure) to encode discrete feature token sequences into embedding vectors in compact and informative embedding space, and a feature utility evaluator to predict the performance of a feature set given the corresponding feature token sequence embedding. Figure 2 A) shows our model includes an encoder, an evaluator, and a decoder.
Step 1: Transformer-basse
Seq2Vec Encoder. Given discrete feature token sequence F f2,.f2.... fa...., . }. the encoder (-) encodes the feature sequence into a latent vector z, which is sampled from a latent distribution 9s (z]F) [p,, a²). To reshape the distribution of the latent space into Gaussian, we impose KL divergence to measure and minimize the distance between the embedding space posterior distribution and standard Gaussian prior:
LKL (g.(z)))))lpe
Step 2: Hierarchical Decoder with Position Embedding, The decoder is to generate feature sequence from a given latent vector z. When the decoder's input is the decoder's output embedding. the task is reduced to reconstruct the original feature set. Unlike natural languages, feature set, although tokenized as feature token sequence, has no current-past feature dependencies. For instance, in the feature sequence fi + f2. f2/ fa. the second feature token segmentation f2/ fa is not dependent on the first feature token segmentation fi + fz. Only the token sequence within a single feature segmentation (e.g.. fi + f2) exhibits sequential dependencies. We name tokenized feature set as a compositional sequence.
To model the compositional sequence structure, we propose a hierarchical decoder that adopts two-level (inter-feature and intra-feature) generation strategy. Specifically, the inter-feature level, the decoder predicts how many features we should generate a categorical distribution using the embedding of the feature set. Formally, let T be the number of features, Pe denote the decoder's ' conditional probability distribution parameterized by 6, the feature size prediction is trained by minimizing a standard cross-entropy loss:
feature set, which is modeled as
L = - logpo((T (T)
After predicting the size of the feature set to decode, the decoder independently generates each feature in parallel. At the intra-fcature level, the decoder autoregressively generates each feature token segmentation starting with a special token of "BOS". Formally, let f(() be the k-th token of the I-th feature, Le the total token number of ft. The feature token sequence reconstruction is learned by optimizing a token-level cross-entropy loss over all the predicted features:
T L,
log logpo(st ( 1 ({((K) 2).
Lree
1=1
Step 3: Incorporating The Evaluator. To incorporate performance guidance, construct lightweight regression evaluator to predict the downstream task performance of feature set, based on the feature set's 's's embedding. Formally, Rs (-) is regression function, y is the real downstream task performance of a feature set. The evaluator is trained by minimizing a mean squared error loss:
Leva R. (z) vl
Solving the Optimization Problem. The by jointly optimizing the losses of Gaussian regularization, feature set size prediction, feature set reconstruction, and feature set performance evaluation:
structure is trained end-to-end
L = Lree + acott + BLea + 7LKL,
(5)
where o, B, and y the weights of corresponding loss terms.
Finally, after the encoder, decoder, and evaluator are trained, the encoder can create continuous embedding space to map various transformed feature sets specific dataset into embedding vectors,

---

thus laying foundation for the diffusion model to learn and generalize the distributions from these embedding vectors; the evaluator can provide performance-optima directions to enable diffusion models to sample (a.k.a., generate) the best transformed feature set embedding from the distribution via maximized likelihood estimation; the decoder, if well trained, can decode the best transformed feature set embedding into a feature token sequence.
2.3 Distribution: Diffusion Model of Probabilistic Feature Embedding Space
After completing the embedding step, utilize the learned encoder to project various feature token sequences (i.e.. potential feature transformations) into embedding vectors. The embedding vectors of potential feature transformations can be seen as samples that jointly form distribution about how feature performances are prohabilistically spread across various feature set embeddings. This provides new probabilistic angle: if we can describe the distribution of feature set emheddings over performance, we can identify the best feature transformation plan for a dataset by generating the mean feature set embedding with the highest probability from the distribution. The Latent Diffusion Model (LDM) exhibits the ability to learn distributions from noisy and limited data in reverse diffusion and generate performant samples from the distribution. This ability provides great potential for us to derive the distribution of feature set embeddings and identify the best feature set transformation. We next introduce the feature set embedding diffusion model.
Leveraging Feature Sets as Attributed Graphs and Graph Embedding for Diffusion Condition- external representation ing. To steer diffusion generation, condition the diffusion process Specifically, we convert that captures distributional semantics of the original raw dataset to transform. a feature token sequence into a tabular modality, by applying the feature token sequence to the raw data matrix, resulting in transformed table. Based on the transformed table, we construct feature- feature attributed graph, where nodes are features (columns) and edge weights are feature-feature similarities to represent tabular feature space topology, interaction, and semantics. We then exploit a Graph Convolutional Network (GCN) to learn an embedding of this graph. In this way, feature token sequence, corresponding to its transformed table and ceature-feature interaction graph, is associated with graph embedding vector. After that, we leverage this graph embedding as a conditioning signal during both training and sampling. Such conditioning strategy allows the diffusion model to generate embedding vectors aligned with the structure, topology, and semantics of the targeted tasking dataset to transform, enabling coherent and meaningful feature generation. Appendix B provides details about the diffusion condition acquisition.
Learning Feature Transformation Diffusion Model. Because we regard a transformed feature set as feature token sequence, not an image, we choose Transformer-based denoiser [33] instead of the conventional UNet architecture. Our diffusion model includes a Transformer-encodde of L blocks that apply sequentially the current latent state z₁. For the b-th block, we compute
h (f.1) = (h
(6)
h (f.2) CA(AdaLN,(h c),
(7)
h h6 FFF((((2)))
(8)
where SA is self-attention over latent tokens, CA cross-attends to the condition embedding c, and FFN is a two-layer feed-forward network. The timestep enters through the AdaLayerNome modulation (2I), and the condition
MLP cond(g) is obtained by passing the tabular embedding through slight MLPs. The network predicts the noise eØ [z, 1, c) and is trained with the standard
latent diffusion loss CLOM = [[I - ( t, c)]] Appendix D.4 details the training of the diffusion model.
2.4 Optimization as Generative Sampling: Reward-guided Optimal Feature Set Embedding Generation
Given a target reward a that indicates a high utility of features to generate, we follow the gradient- steered sampler of RCGDM [32 to consider the latent z as Gaussian and develop our reverse process.
At each denoising step, we first evaluate V. log (y = a z) x 3 Vz (R.. - a) a) where
Rg is the pretrained evaluator and x² is a hyper-parameter The DDIM update [2 then becomes
2 = Vor-1 2t
[co(ze, t, c) (Vx((((R((z)) - a)9)1) + v Ct-1 7 (9)
(()x))))))11))))))
vI Cz.
5

---

with A= 1/0² controlling the strength of reward guidance and n, (O, I) the optional stochastic- ity. Note that we do not apply classifier-free guidance [TO] in our model, since the reward guidance we employ shares the same fundamental principle as classifier- guidance, which is approxi- mating the conditional score Vlogpi (x+ y) [5]. Upon completion, the final latent 2g decoded by the semi-autoregressive decoder to yield feature set that is syntactically valid and empirically high-reward.
2.5 Hierarchical Decoding: Semi-autoregressive Feature Set Reconstruction from Optimal Embedding
After the reverse process samples or generates the optimal high-reward feature set embedding from the embedding distribution, we leverage the learned decoder to decode the optimal embedding into feature token sequence, in order to obtain a new transformation. The hierarchical decoding includes two steps: 1) predict feature size; 2) autoregress the token sequence of each feature chunk. Formally, the decoder first predicts the number of features T:
T (o %0).
For each feature chunk fi the decoder generates its token sequence in an autoregressive manner:
1 pa (s(
20), k I.... Lr.
This semi-autoregressive decoding strategy enables parallel generation across feature chunks while maintaining autoregressive modeling within each feature, balancing efficiency and generation quality.
Experiments
We conduct extensive experiments on public datasets evaluate the effectiveness, efficiency, and robustness of our method. The experiments are organized to answer the following research questions: RQ1: Does the proposed method perform better than the baselines? RQ2: Do reward-guided diffusion and semi-autoregressive generation contribute to the overall effectiveness and efficiency of the framework? RQ3: the proposed method efficient compared with baselines? RQ4: Is the proposed method robust when collaborating with different downstream ML models? RQ5: How does guidance strength in generation affect performance?
3.1 Experimental Setup
We collect 14 public datasets from LibSVM, UCIrvine, and OpenML to conduct experiments. The datasets are classified into two predictive tasks: classification and regression. We employ random forest as the default downstream model and use F-1 score and 1- relative absolute error (RAE) to measure the performance of classification task and regression task respectively. We compare the proposed method with 10 widely-used algorithms: RDG, PCA [18] LDA 3 ERG, AFAT [11] NFS TTG 15), GRFG (26,. MOAT 27 ELLM 6.. More details about datasets and baselines can be
found in Appendix h
To comprehensively evaluate the effectiveness each component, introduce four ablation variants of DIFFT: 1) AR employs a fully autoregressive generation strategy in the VAE decoder, replacing our autreretoreeeessvve design; 2) NAR adopts fully non-autoregressive decoder assess the importance of intra- intra-feature dependency modeling; 3) NoR removes reward guidance during diffusion to examine the role of performance aware optimization: 4) CS eliminates the diffusion model and instead applies a conventional continuous search method in the latent space to validate the effectiveness of the diffusion model.
3.2 Overall Comparison (RQ1)
Table [ reports the performance across 14 benchmark datasets. DIFFT consistently achieves the best results on all datasets, demonstrating strong effectiveness and robustness across wide range of tasks. For example, Messidor Features, DIFFT achieves an accuracy of 0.757, substantially higher than the second-best method EL.LM with 0.699. Openml_ _618, DIFFT reaches 0.632, outperforming MOAT with 0.477 and other baselines by significant margin. This performance

---

Table I: Overall comparison. We conduct experiments 14 datasets explore the performance of the proposed method. The best results for each dataset are highlighted in bold, and the second-best results are underlined.
Dataset
RDG
PCA
LDA
ERG
AFAT
NFS
TTG
GRFG
MOAT
ELLM
DIFFT
SpectF
0,760
0,709
0.665
0.792
0.792
0,760
0.776
0.796
0.848
0.837
0.877
SVMGuide3
0,789
0,676
0.635
0.764
0.795
0,785
0,766
0.804
0.821
0.826
0.866
German Credit
0.657
0.679
0.597
0.711
0.639
0.677
0.680
0.722
0.738
0.741
0.747
Messidor Features
0.686
0.672
0.463
0.602
0.634
0.649
0.626
0.694
0.644
0.699
0.757
SpamBase
0,921
0.816
0.885
0.923
0.920
0,925
0,921
0.928
0.930
0.928
0.931
Ap-omentum-ovary
0.849
0.736
0.696
0.830
0.845
0.845
0.830
0.830
0.849
0.849
0.885
lonosphere
0.898
0.928
0.743
0.914
0.899
0.899
0.928
0.928
0,942
0.971
Openmi_ 556
0.532
0.205
0.061
0.542
0.549
0.553
0.551
0.550
0.607
0.611
0.647
Openmi_ 559
0.222
0.503
0.590
0.510
0.595
0.615
Openmi 607
0.521
0.106
-0.040
0.518
0.513
0.518
0.518
0.524
0.521
0.550
0.561
Openml 616
0,120
-0.083
-0.174
0.193
0.163
0.166
0.161
0.166
0.162
0.329
0.335
Openml 618
0,425
0.141
0.030
0.470
0.473
0.473
0,472
0.472
0.477
0.476
0.632
Openml 0,502
0.138
-0.045
0.501
0.520
0.507
0.515
0.528
0.510
0.550
0.592
Openml 637
0,136
-0.052
-0.141
0.152
0.162
0.146
0.145
0.153
0.161
0.254
0.291
primarily driven by two key components: the latent diffusion model, which enables global exploration of the embedding space to discover high-quality feature representations, and the semi- autoregressive decoder, which efficiently reconstructs complex feature structures with high fidelity. Compared to MOAT, the representative continuous search-based method, DIFFT offers significantly better stability and generalization. Although MOAT occasionally produces competitive results, its performance fluctuates considerably across datasets and drops sharply more challenging tasks such as Openml 616, where it reaches only 0.162 compared to 0.335 achieved by DIFFT. In contrast, DIFFT performs consistently well, benefiting from its generative formulation and reward- reward-guided guided sampling strategy. By avoiding local search and leveraging global, performance-awan generation, DIFFT delivers more reliable and superior feature transformations across diverse tasks.
3.3 Ablation Study (RQ2)
To investigate the contribution of each compo- nent in DIFFT, we design variant models: 1) NAR and AR, which examine the impact of the autreervoree decoder by replacing it with fully non-autoregressive and fully autoregres- sive alternatives, respectively; 2) NoR, which removes reward guidance during the diffusion process to evaluate its effect on performance op- timization; and 3) CS. which replaces the diffu- sion model with a traditional continuous search approach assess the necessity of diffusion based generation. We conduct ablation study on two datasets to validate the effectiveness of the components. More details can be found in Appendix D.5
Table 2: Performance variant models and DIFFT
on 2 datasets.
Variants
Datasct
DIFFT
NAR
AR
NoR
CS
SVMGuide3
0.840
0.856
0.831
0.830
0.866
Openml_586 0.641
0.638
0.623
0.610
0.647
Reward-Guided Diffusion. As shown in Table 2 both NoR and CS suffer from significant per- formance drops compared to the full model. For example, on the SVMGuide3 dataset, removing reward signals (NoR) reduces the accuracy from 0,866 to 0.831. while replacing diffusion with continuous search (CS) causes an even greater decline 0.830. This highlights the critical role of reward guidance and the diffusion mechanism in driving feature generation toward tigh-perfomiig regions of the latent space. Interestingly, even without reward guidance, NoR still outperforms the continuous search-based based variant CS on the two datasets. This suggests that in the absence of a carefully constructed latent space and an informative initialization, continuous search methods often fail to explore effectively and are prone to suboptimal solutions. In contrast, by modeling global structure and semanties, our diffusion model provides a more robust foundation for feature embedding generation.
7

---

Semii-uttreerressive Generation. The results in Table 2 also demonstrate the effectiveness of the proposed seto-auutoregressivv (SAR) decoder. Compared with both AR and NAR variants, SAR achieves superior performance across datasets. For example, on SVMGuide3, the SAR decoder achieves an accuracy of 0.866, surpassing both the fully autoregressive model with 0.856 and the non-autoregressive variant with 0.840. This improvement demonstrates that SAR effectively balances dependency modeling and generation efficiency. By generating each feature independently while preserving Intra-feature autoregression, SAR captures fine-grained dependencies within individual features, while avoiding redundant coupling across features that are semantically unrelated. This selective dependency modeling enables the decoder to allocate attention more efficiently, resulting in more stable and interpretable feature generation.
To further analyze efficiency, Figure explores the decoding time of SAR decoders under dif- ferent configun sonftiutttinns to generate 2000 tokens. We vary the number of features and the length of each feature such that the total token count remains constant (i.e., number x length). The results show that SAR consistently achieves sig- nificantly lower decoding time than AR, regard- less of the configuration. Although decoding time slightly increases with longer features, our method maintains high efficiency by paralleliz- ing feature generation.
2.7
Performanc
63
0.2
0.027
0.051
0.063
bug
100"20
80"25
50h40
40"50
25h50
20"100
Contigguration (Number Longth)
Figure 3: Time analysis of generating 2000 tokens using different methods.
3.4 Time Complexity Analysis (RQ3)
Table 3: Comparison of per-epoch training time and per-inference pass time (in seconds) between
DIFFT and MOAT across different datasets. DIFFT (D) and DIFFT (V) denote the per-epoch training time of the diffusion model and VAE, respectively, while inference time refers to the duration of one complete inference pass.
Training Time (s)
Inference Time (s)
Dataset
Feature
DIFFT (D)
DIFFT (V)
DIFFT
MOAT
DIFFT
MOAT
Messidor Features
19
2.65
4.36
7.01
83.9
0.16
0.53
Openml_50 559
25
2.85
5.02
7.87
110.9
0.21
0.68
lonosphere
3.33
5.35
8.68
170.1
0.23
1.03
Openml_611 Ap. omentum ovary
50
4.74
7.45
12.19
200.9
0.25
1.18
10936
7.78
11.86
19.64
524.4
0.43
2.33
We compare the training and inference efficiency of DIFFT and MOAT evaluate the computational characteristics of the two generative feature transformation methods. Table 3 the per-epoch training time and per-instance inference time for both methods. For DIFFT. we separately record the training time of the VAE (denoted as DIFFT (V)) and the diffusion model (denoted as DIFFT (D)). DIFFT requires significantly less time to train per epoch compared to MOAT. For example, on the large-scale mp_omentum_ovee -vaay dataset with over 10,000 features, DIFFT completes one training epoch in 19.64 seconds, while MOAT takes more than 500 seconds. This efficiency gain largely attributed to architectural differences: DIFFT adopts a Transformer- based backbone, which supports efficient parallel computation and faster convergence, whereas MOAT relies on an L.STM-based model with sequential processing and limited scalability. Inference speed further highlights the efficiency of our approach. DIFFT generates a feature set in no more than 0.43 seconds on all datasets, whereas MOAT takes up 2.33 seconds in some cases. This leads speedup of more than times, mainly due to the auttorrrrreeeeivvee decoding strategy that allows parallel feature- generation while maintaining high generation quality. These results demonstrate that DIFFT offers not only strong performance but also substantial advantages in training and inference efficiency.
3.5 Robustness Check (RQ4)

---

To evaluate the robustness of DIFFT across dif- ferent learning scenarios, we apply five repre- sentative downstream machine learning models on the SpectF dataset, including Decision Tree (DT), K-Nearest Neighbors (KNN), Logistic Re- gression (LR), Support Vector Classifier (SVC), and Random Forest (RF). Table 4 reports the performance of DIFFT compared with the top three baselines. We observe that DIFFT con- sistently achieves the best performance across all downstream models, demonstrating strong robustness to variations in model architectures and learning biases. In contrast, ELLM performs reasonably well on some models but fails to achieve competitive results on RF, indicating limited generalization. Similarly, MOAT performs poorly when paired with SVC, suggesting sensitivity to the characteristics of the downstream learner. These findings highlight the robustness of DIFFT in optimizing feature transformation across diverse tasks. DIFFT generates task-specific features that align with the objective of the target model, enabling consistent improvements in predictive performance across a wide range of learning scenarios.
Table 4: Robustness check. We employ different downstream ML model to investiagte the robust- ness of our method.
Method
DT
KNN
LR
SVC
RF
GRFG
0.752
0.764
0.816
0.813
0.796
MOAT
0.873
0.843
0.835
0.836
0.848
ELLM
0.865
0.842
0.846
0.841
0.837
DIFFT
0.894
0.861
0.864
0.877
0.877
3.6 Guidance Strength Analysis (RQ5)
Table 5: Performance under different guidance strength À settings.
Dataset
10
25
50
100
200
400
SVMGuide3
0.831
0.857
0.867
0.874
0.871
0.860
0.860
Openml 0.237
0.262
0.232
0.278
0.324
0.302
0.170
To investigate the effect of the reward-guided strength À on the generation performance, we conduct experiments on two datasets, with À ranging from 0 to 400. The results are summarized in Table [] For SVMGuide3, we observe a clear improvement as À increases from 0 to 50, where the performance peaks at 0.874. This demonstrates that moderate guidance helps the diffusion process align more effectively with the reward objective. However, when À becomes too large (e.g.. 2 100), the performance slightly decreases, suggesting that excessive guidance may overconstrain the generation, leading to reduced sample diversity and potential overfitting to the reward signal. For Openml_616 the trend less monotonic but shows a similar pattern overall. The performance improves gradually and achieves its highest value (0.324) at À = 100, followed by decline when increases further. The larger variability across À values implies that this dataset may be more sensitive to the balance between reward alignment and generative flexibility. Overall, these results indicate that moderate guidance strength (around A = 50-100) yields the best trade-off between reward optimization and sample diversity. Too weak guidance fails to leverage the reward signal effectively, while too strong guidance can hinder the generative capacity of the diffusion model.
Related Work
4.1 Feature Transformation
Feature transformation aims to identify a new feature space by transforming the original features using mathematical operations. The existing methods are mainly two-fold: 1) discrete space search methods. These methods aim search for the optimal feature set the discrete feature combination space. DFS [13] transforms all original features and selects useful ones. Genetic programming applied for feature transformation search 24.. GRFG (26) develops three agents to generate features and leverages reinforcement learning to improve the search strategy. ELLM [6] employs few-shot prompting LLM to generate new feature sets. 2) continuous space search methods. MOAT 27 embeds feature sets into continuous space and utilizes the gradient-ascent method to explore embedding space. NEAT 300 extends the continuous space feature transformation search to unsupervised conditions.

---

4.2 Diffusion Model
Diffusion models have emerged as a dominant paradigm in generative modeling, offering superior sample quality and training stability compared to traditional approaches such GANs and VAEs. Their fundamental design involves a forward noising process and a learned reverse process that enables high- tigh-fidelity generation across modalities. Conditional variants of diffusion models, which incorporate task-specific signals such as class labels, text prompts, or reward functions, have further extended their applicability to controlled generation tasks in vision, language, and reinforcement learning. Recent applications span a diverse range of domains, including text-to-image synthesis (2, trajectory planning in reinforcement learning 12 1. and protein molecule design in life sciences [8]. Beyond applications, growing line of theoretical research has addressed key questions around score function learning, sample complexity, and optimization guarantees, especially in high- dimensional and structured settings [1). Recent works also reinterpret conditional diffusion models as flexible samplers for solving black-box optimization problems, where solution quality is guided by reward-driven conditioning [17]. Despite the growing empirical success, theoretical analysis remains an open frontier, particularly regarding guidance strength and generalization under distribution shifts.
4.3 Semi-Autoregressive Model
To achieve a better trade-off between decoding speed and output quality, semi-auttoegrressvve gen- eration (SAR) has been proposed as a hybrid decoding paradigm that blends the strengths both autoregressive (AR) and non-autooeeressivee (NAR) models. Early work by Wang et al. introduced the semi-au aumi-autoreeressivee Transformer, which generates tokens in chunks producing multiple tokens in parallel within each group, while maintaining autoregressive dependency across groups 25) This design allows for partial parallelization while preserving sufficient contextual dependency to mitigate issues such as repetition omission in NAR outputs. Building upon this, the Insertion Transformer supports flexible generation orders by allowing insertions at arbitrary positions. offering a more dynamic decoding process [23]. Other efforts, such as the DisCo Transformer [14), further relax strict left-to-right generation by enabling token predictions conditioned on arbitrary subsets of other tokens. Additionally, RecoverS 20) introduces segment-wise decoding strategy where each segment is generated non-autoregressivelyy while the tokens within segments follow an autoregressive pattern. These semi-autoregressivv frameworks consistently demonstrate superior balance between latency and accuracy compared to fully NAR models, making them attractive choices in time-sensitive applications.
Conclusion Remarks
In this paper, we propose DIFFT, a novel framework that formulates feature transformation as a reward-guided generative process. Departing from traditional pearch-based paradigms, DIFFT integrates a latent diffusion model with cemi-autoregressive decoder to generate high-quality. task- specific feature sets a performance-ane efficient and expressive feature reconstruction, while the reward- eeward-guided guided sampling process aligns the generative trajectory with downstream objectives. Extensive experiments on diverse collection of tabular datasets demonstrate that DIFFT consistently outperforms existing feature transformation and selection methods in terms of predictive performance, robustness across downstream models, and computational efficiency. Our ablation studies further validate the contributions of cach module, including the autrerrvtorer decoding and reward-driven diffusion. These results highlight the effectiveness and adaptability of DIFFT as scalable solution for feature engineering in modern machine learning pipelines.
manner. Our hierarchical decoding strategy enables
Acknowledgments
Dr. Yanjie Fu is supported by the National Science Foundation (NSF) via the grant numbers: 2426340, 2416727, 2421865, and 2421803.
10

---

References
Ajay, A., Du, Y., Gupta, Tenenbaum, J., Jaakkola, & Agrawal, P. (2022) conditional generative modeling you need for decision- makiio?? arXiv preprint arXiv:2211.1665
[2] Black, K., Janner M., Du Y., Kostrikov I., & Levine, (2023) Training diffusion models with reinforcement learning, arXiv preprint arXiv:2305.1330)
[3] Blei, D. M. Ng. A. Y., & Jordan, M. I. (2003) Latent dirichlet allocation. Journal of machine Learning research 301a))993-1022.
[4] Chen X., Lin Q.. Luo C., Li, X., Zhang H. Xu. Y.. Dang Sui K., Zhang X., Qiao B., others (2019) Neural feature search: neural architecture for automated feature engineering. In 2019 IEEE International Conference on Data Mining (ICDM) pages 71-80. IEEE.
[5] Dhariwal, P. & Nichol, A. (2021) Diffusion models beat gans on image synthesis. Advances in neural information processing systems 34:8780-8794.
[6] Gong. N., Reddy C. K., Ying, W., Chen, H., Fu (2025. Evolutionary large language model for automated feature transformation. Proceedings of the AAAI Conference on Artificial Intelligence 39, pp- 16844-16852.
[7] Gong, N., Wang X., Ying, W., Bai, H., Dong. S., Chen, H., & Fu, Y. (2025.) ) Unsupervised feature transformation via in-context generation, generator-critic Ilm agents, and duet-play teaming. arXiv preprint axXiv2504.21304
[8] Gruver N., Stanton, S., Frey N., Rudner, T. G., Hotzel, L., Lafrance-Vannnn J., Rajpal A., Cho K., & Wilson A. G. (2023) Protein design with guided discrete diffusion. Advances in neural information processing systems 36: 2489-12517.
Hang. T., Gu, S., Li C., Bao J., Chen D., Hu H., Geng X., & Guo. B. (2023) Efficient diffusion training via min-sur weighting strategy. Proceedings of the IEEE/CVF international conference on computer vision pages 7441-7451
[10] Ho, J. & Salimans, T. (2022) Classificr-free diffusion guidance. arXiv preprint arXiv:2207. 2259.
[11] Horn, F., Pack, R., & Rieger, M. (2020) The autofeat python library for automated feature engineering and selection. In Machine Learning and Knowledge Discovery Databases: International Workshops of ECML PKDD 2019, Würzburg, Germany, September 16-20, 2019, Proceedings, Part I pages 11-120.. Springer.
[12] Janner, M., Du, Y., Tenenbaum J. B., & Levine, S. (2022) Planning with diffuusion for flexible behavior synthesis. arXiv preprint arXiv:2205.09991
[13] Kanter, J. M. & Veeramachaneni K. (2015) Deep feature synthesis: Towards automating data science endeavors. In 2015 IEEE international conference data science and advanced analytics (DSAA) pages 1-10. IEEE.
[14] Kasai, J., Cross J., Ghazvininejad M., & Gu J. (2020) -on-autoreeressivv machine translation with disentangled context transformer. In International conference on machine learning pages 5144-5155. PMLR.
[15] Khurana. U.. Samulowitz. H., & Turaga D. (2018) Feature engineering for predictive modeling using reinforcement learning. In Proceedings of the AAAI Conference Artificial Intelligence 32..
[16] Kipf, T. N. & Welling M. (2016) Supi-supe classification with graph convolutional networks. arxiv preprint 1609v
[17] Krishnamoorthy. S., Mashkaria S. M., & Grover, A. (2023) Diffusion models for black-box optimization. In International Conference on Machine Learning pages 17842-17857 PMLR.
[18] Mačkiewicz, A. & Ratajezak, W. (1993) Principal components analysis (pca). Computers & Geosciences 193):303-34..
[19] Oko. K., Akiyama S., & Suzuki, T. (2023) Diffusion models are minimax optimal distribution estimators. In International Conference on Machine Learning pages 26517-26582. PMLR.
[20] Ran Q. Lin Y., Li P. & Zhou J. (2020) Learning to recover from non-autoreeressivv neural machine translation. arXiv preprint 05X51:2006.05165
emrors for

---

[21] Rombach, R., Blattmann A.. Lorenz, D.. Esser P., & Ommer B. (2022) High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition pages 10684-10695.
[22] Song J. Meng C., & Ermon S. (2020) Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502
[23] Stem, M., Chan W., Kiros J., & Uszkoreit, J. (2019) Insertion transformer: Flexible sequence generation via insertion operations. International Conference on Machine Learning pages 5976 5985. PMLR.
[24] Tran, B., Xue, B., & Zhang M. (2016) Genetic programming for feature construction and selection in classification on high-dimensional data, Memetic Computing 8:3-15.
[25] Wang. C., Zhang. J., Chen, H. (2018) Semi-autoregressivve neural machine translation. arXiv preprint arXiv:1888.068 08588
[26] Wang. D.. Fu, Y., Liu, X., & Solihin, Y. (2022) Group-wise reinforcement feature generation for optimal and explainable representation space reconstruction. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining pages 1826-1834.
[27] Wang D., Xiao M., Wu, M., Zhou Y., feature transformation: Information Processing Systems 01443363-43572
others (2023) cewtorrremetteehannneeeeee autoregressive search in continuous space for postfix expressions. Advances in Neural
[28] Wang.. D., Huang, Y. Ying, W., Bai, H., Gong N., Wang X., Dong S., Zhe, T., Liu, K., Xiao M., & others (2025) Towards data centric ai: A comprehensive survey of traditional, reinforcement, and generative approaches for tabular data transformation. arXiv preprint arXiv:2501.1053
[29] Xiao M. Wang D., Wu M., Qiao Z., Wang P. Liu K., Zhou, Y., & Fu, Y. (2023) Traceable automatic feature transformation via cascading actor-critic agents. In Proceedings the 2023 SIAM International Conference on Data Mining (SDM) pages 775-783, 783. SIAM.
[30] Ying W., Wang D., Hu, X., Zhou, Y., Aggarwal, C. C., & Fu, Y. (2024) Unsupervised generative feature transformation via graph contrastive pre-training and multi-objective fine-tuning, Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining pages 3966-3976.
[31] Ying. W., Wei, C., Gong N.. Wang X., Bai H., Malarkkan V., Dong. S., Wang D., Zhang D., Fu, Y. (2025) A survey on data-centric ai: Tabular learning from reinforcement learning and generative ai perspective. arXiv preprint arXiv:2502.08828
[32] Yuan , H., Huang, K., Ni, C., Chen M., Wang, M. (2023) Reward direetdd conditional diffusion: Provable distribution estimation and reward improvement. Advances in Neural Information Processing Systems 36:60599 60635.
[33] Zhang. B., Tang J., Niessner, M., & Wonka, P. (2023) 3dshape2vecset: A 3d shape representation for neural fields and generative diffusion models. ACM Transactions On Graphics (TOG) 42(4): 1-16.
12

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We clearly conclude contributions and scope in the abstract and introduc-
tion.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We illustrate the limitations of DIFFT in the appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
13

---

Justification:
We analyze the experimental results and elucidate the underlying reasons for the observed outcomes.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide the source code, datasets, and detailed experimental settings to ensure the full reproducibility of our results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
14

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
Both the code and datasets used this study are publicly accessible.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ccipps.c/puulli//guu /otee///uuddde/eooSuuubbiiibii ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: These details can be found in our code.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
We report results from a single run of each experiment.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
15

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The experimental environment is described in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
Our research is with the code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
16

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
There are no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: They are cited in our paper.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
17

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We organize our code and data well in the link.
Guidelines:
The answer NA means that the paper does not release new assets. Researchers should communicate the details of the dataset/cooe/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We don't 't have research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We don't have research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
18

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development does not involve LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
19

---

A Preliminaries
(1) Continuous Search Method
(2) Reward-Guided Generation Method
Eshmator
Optimal Emiedding
Ophimal Emibadding
Conditton
Reward signal
Encode
Decode
Gouasion Noie
Decode
A
Oriinall Sequence
Ophimol Sequence
Optimal Sequince
Gradient- Sased Search
Rewerd-Gulded Denolse
Figure 4: Comparison of continuous search method and optimized generation method.
Feature Transformation. Given a raw feature set Fraw = [/i, , Jn] and an operator set O (e.g., log, +. +), feature transformation aims to apply mathematical operations to original features and construct better feature set, for instance, [fi * f2, log fs. fa fs]. These transformed features reveal informative interactions and nonlinear relationships that improve downstream task performance.
Feature Sets as Language Token Sequences. feature can be described in a language token modality. For example, given a transformed feature set [fi f2, log fs. fa + fsl, we use token sequence "f1 f2. log fs. fa + fs" to represent in compact and generative-friende friendly form. To reduce sequence length and ambiguity, we adopt the postfix notion: "f1 f2 fa log, fa fs +".
Generative Feature Transformation and the Continuous Search Paradigm. Classic feature engineering relies on manual selection or discrete search over massive possible combinations. With the success of generative AI and everything tokens, researchers regard feature sequence, and formulates feature transformation as sequential generation task: given X and generate new feature token sequence (i.e., new transformed feature set) that reconfiguring the data that the model sees to align model behavior with prediction targets. The major methodological paradigm under the generative formulation is continuous search that consists of three steps: 1) embed feature token sequence into continuous embedding space; 2) apply gradient ascent to discover better embedding of feature transformations; decode identified embedding to a new table. Building on this idea, propose novel reward-guided generation framework that goes beyond searching within a static embedding space: instead, it directly generates and refines feature representations through performance-awee aware generative process. This enables dynamic, goal-oriented exploration of the transformation space, as summarized in Figure 4
token
Latent Diffusion Model. Diffusion models (22 21) synthesize data by learning to reverse a gradual noising process that turns clean sample into Gaussian noise. Let zg be latent drawn from the data distribution. The forward (noising) process adds variance variance-preserviie preservvig Gaussian noise for T steps
{xx 1) N(as v1 3, = B.I):
t= 1....,,
with a predefined schedule {,8,)T- A neural network co is trained to approximate the reverse (denoising) transitions by predicting the added noise:
Lom = Et..o.. [|c ((zz),))2
Vor vI Cr ā
(.))).
In latent diffusion, data are first compressed with an auto-encoder (Ene, Dec) into perceptually aligned latent space, and the diffusion process operates entirely on these low-dimensional latents. Compared with pixel-/token-sacce diffuusonn, LDMs (i) drastically reduce memory and computation, (ii) facilitate conditioning on vector-space signals, and (iii) retain high-fidelity details because decoding starts from semantically rich latents instead of raw noise. During inference, one samples 2T ~ (0, I) and applics the learned reverse transitions to obtain a clean latent 20, which mapped back to the data domain by Dec(2o). Importantly, each denoising step is differentiable, external gradients-ca e.g. from reward or performance predictor- be injected to steer the trajectory toward regions that maximize downstream objectives.
B Condition Acquisition
To provide compact and informative conditioning signal for the diffusion model, we extract table-level embedding that summarizes the structure of the transformed tabular data. The feature
20

---

sequence is interpreted as a table, where each column corresponds to a transformed feature. This table structure captures both individual feature characteristics and inter-feature statistical dependencies.
We adopt a GCN-based encoder & to generate the table embedding. Specifically, each feature is regarded as node in fully connected feature correlation graph with self-loops. The adjacency matrix A encodes the pairwise correlations, and the node input features form the matrix H()), We apply a simplified Graph Convolutional Network [16] using symmetric normalization:
(( A)=
= (D 1 AD HOW(
(10)
where A is the adjacency matrix with self-loops, D is the degree matrix, W is the trainable weight matrix at layer and is a non-linear activation function. After GCN, we obtain contextualized embeddings for all features. These are aggregated via average pooling to obtain table-level embedding:
h4
(11)
where h; denotes the final embedding of the i-th feature. The resulting es is then used as the condition input to the diffusion model during both training and generation, enabling it to guide the feature generation process based on the global structure the table.
C Training Data Collection
Generative feature transformation requires diverse and high-quality training dataset to construct meaningful embedding space. However, traditional manual data collection is often inefficient and lacks scalability. To address this challenge, we design a reinforcement learning (RL)-based data collector that automatically explores and generates effective feature transformations, inspired by recent advances in automated feature engineering 26).
The proposed RL-based data collector simulates feature transformation trajectories and consists of the following components:
1) Agents: A multi-agent system is employed, comprising:
Head Feature Agent ou; selects the first
feature.
Operation Agent az: selects an operator from the operator set O.
Tail Feature Agent og; selects the second feature.
2) Actions:
At each time step I, the three agents collaboratively generate new feature as:
fz = az (t) O (t(() @ ax(t)
(12)
where @ denotes the operator applied the selected feature pair.
3) State Representation:
The state S; = Rep( X,) is defined as 49-dimensional vector summarizing the current feature space. It includes seven descriptive statistics (count, standard deviation, min, max, and first/sccond/third quartiles) computed column-wise and then aggregated row-wise, capturing both global and local characteristics.
4) Reward Function: The reward is defined as the performance improvement of the downstream model after adding fı:
R(t) = V/ - Yt-1
(13)
where ye and yt-1 denote the performance metrics before and after the transformation. respectively.
The agents are trained to maximize the cumulative reward by minimizing the mean squared error in the Bellman equation, thereby learning transformation policies that consistently benefit downstream performance.
21

---

Algorithm RL-based Data Collection Procedure
1: Input: Initial feature set Xo. operator set O, downstream model
to N do 3: Initialize feature space Xn and state So do
2: for each episode
4:
for each step
5:
Select features: fA < 04 (S;-1) f fo + as(S)-1)
6:
Select operator: op 0.
7:
Generate new feature: ft = fh Oop ft Update feature set: X; X [fe}
8:
9:
Evaluate performance: Y/ +- downstream model performance on X,
Compute reward: R(t V/- -t-1
11:
Update state S. + Rep X+)
Update agents' policies using R(I) end for
13:
14:
end for
15: Output: Collected high-quality feature set
D Experiments
D.1 Datasets
To comprehensively evaluate the effectiveness and generalizability of method, conduct experiments on 14 benchmark datasets encompassing both classification and regression tasks. These datasets sourced from widely used repositories, including UCIrvine, LibSVM, and OpenMI. As summarized in Table 6 the datasets vary significantly in terms of sample size (ranging from 267 to 4601), feature dimensionality (from 19 to 10,936), and task type. This diversity allows us assess model robustness across heterogeneous data distributions and application scenarios.
Dataset
Source
Task
Samples
Features
SpectF
UCIrvine
267
SVMGuide3
LibSVM
C
1243
21
German Credit
UCIrvine
1000
24
Messidor Features
UCIrvine
C
1151
19
SpamBase
UCIrvine
4601
57
Ap-omentum-ood ovary
OpenmlML
C
275
10936
Ionosphere
UCIrvine
351
34
Openml_5p _586
OpenmlMI.
1000
Openml 589
2_amml__999
OpenmIMI.
R
1000
25
Openml 607
OpenmIML
1000
Openml 616
OpenmIML
R
500
50
Openml 618
OpenmIML
1000
Openml 620
OpenmIML
R
1000
25
Openml 637
OpenmIML
500
Table 6: Dataset statistics.
D.2 Baselines
We compare our method with 10 widely-used algorithms: 1) RDG, which randomly generates new feature transformations; 2) PCA [18). which generates new features based on linear correlations among original features; 3) LDA 3]. which constructs a new feature space through matrix factor- ization; 4) ERG. which expands the feature space by applying selected operators to each feature, followed by feature selection; 5) AFAT [T], which repeatedly generates new features and applies multi-step feature selection; 6) NFS 4 which models the feature transformation process decision-making task and leverages reinforcement learning for optimization; 7) TTG [55), which formulates feature transformation graph search problem and employs reinforcement learning
22

---

to explore it; 8) GRFG 26), which builds cascading agent architecture and proposes feature group crossing strategy to enhance RL-based transformation search; 9) MOAT [27], which adopts reinforcement learning-basee space. 10) ELLM [ which integrates evolutionary algorithm and large language model to iteratively generate high-quality feature sets.
data collector and explores the optimal feature set in the embedding
D.3 Experimental Environment
All experiments are conducted on the Ubuntu 22.04.3 LTS operating system, Intel(R) Core(TM) 19-13900KF CPU @ 3GHz, and 1 RTX 6000 Ada GPU with 48GB of RAM, using the framework of Python 3.11.4 and PyTorch 2.0.1.
D.4 Details of Diffusion Model
Our latent diffusion model comprises 8 identical Transformer blocks. To accelerate training and improve convergence, we adopt the Min-SNR weighting strategy proposed in [9) which emphasizes learning from informative timesteps during the early training phase. The model is trained for 800 epochs using the standard latent diffusion loss as mentioned [32]. During inference, we apply reward guidance with a reward scale set to 100 to enhance the quality of conditional generation. More design details and pyperaarameters can be found in our publicly available code.
D.5 Experimental Results
In this section, we present a more detailed ablation study. Each experiment is conducted five times, and the average performance reported in Table 7
Table 7: Performance of variant models and DIFFT on 8 datasets.
Variants
Dataset
DIFFT
NAR
AR
NoR
CS
SVMGuide3
0.840
0.856
0.831
0.830
0.871
Messidor Features
0.691
0.704
0.722
0.730
0.737
SpectF
0.792
0.823
0.851
0.876
0.881
lonosphere
0.958
0.956
0.955
0.943
0.963
Openml 586
0.621
0.628 0.623
0.610
0.633
Openml 589
0,510
0.606 0.603
0.561
0.619
Openml _66
0,230
0,302
0.237
0.289
0.324
Openml 618
0.487
0.617
0.623
0.620
0.630
E
Limitations
Although DIFFT delivers strong accuracy and robustness, two practical limitations remain. First, its task-optimal design requires regenerating a tailored feature set for every downstream model; when hundreds of learners or rapidly changing tasks are involved, the cumulative sampling and decoding cost can become non-trivial. Second, the current implementation has been validated on tables with up to roughly
K raw features: scaling to ultra-wide domains such as genome-scale assays or large-scale click logs would place increasing memory and time pressure on the VAE and the diffusion model. Future work will focus on extending DIFFT to a task-agnostic foundation model for feature generation.
23


Reviews:
[{"review_id": "7qb8mabYzc", "reviewer": "Reviewer_sxKk", "summary": "The paper presents a diffusion model for feature transformation. The idea is to learn a distribution of feature transformations (i.e., sequences of elementary operators) in a latent space, from which a transformation tailored to a particular dataset and task can be sampled at test time. The model consists of a VAE, a denoiser conditioned on a graph representation of a dataset, and a guidance model that takes downstream task performance as the reward. On a suite of benchmarks, the method demonstrates superior performance compared to the baselines.", "questions": "N/A", "limitations": "The paper has discussed the limitations of the method.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strengths\n\n- The method leverages diffusion models to model the distribution of feature transformations. At inference time, it applies additional guidance to steer the generation of suitable transformations for specific datasets and tasks. Although the overall model design is directly borrowed from other domains (e.g., vision), the paper successfully demonstrates its application in learning feature transformations.\n\n- The paper introduces several modifications to the diffusion model for modeling feature transformations. These include architectural changes in the VAE, conditioning on a graph representation of the dataset, and employing classifier guidance with a custom reward model.\n\n- The results suggest that the method outperforms the baselines at multiple fronts (accuracy, training time, inference time).\n\n## Weaknesses\n\nI want to first highlight that I am not an expert on this niche topic and don't have strong opinions about the paper.\n\n- The paper does not describe the set of operators used for constructing feature transformations. It is not intuitively clear how the operators are chained together. I imagine these are elementary operators like addition and multiplication, but this information should be included in the paper. Additionally, there must be constraints on how the operators can be combined, yet this information is also not available.\n\n- Conceptually, the task is to learn a \"meta-network\" that predicts a transformation given a dataset and a downstream task. There are numerous meta-learning frameworks that can solve the same task. Notable examples include MAML (optimization based) and Prototypical Networks (parsimony based). I am wondering if any of these can serve as baselines.\n\n- The experiments are conducted on small-scale datasets (a few hundred samples). I am not sure the GCN and reward model can be successfully trained at this scale. More training details would help clarify this.", "was_revised": true, "final_justification": "The paper presents a diffusion-based approach to feature transformation. The method is sound and demonstrates strong results on several benchmarks. Overall, it seems to be a valuable contribution to the community interested in this niche topic.", "rebuttal": "We thank you for acknowledging the **novelty, technical contributions, and excellent performance** of our paper.\n\n### **Q1-list of operators in experiments:**\n\nDuring paper preparation, due to page limits, we focus on presenting technical insights. In experiments, we use the following operators: {+, -, *, /, sqrt, square, sin, cos, tanh, sigmoid, log, reciprocal, standard_scaler, minmax_scaler, quantile_transform}, following the setting of MOAT [1].\n \n### **Q1-operator combination constraints in experiments:**\n\nThe operators are either binary or unary. If a generated transformation sequence is valid,  each operator must have the required number of operands, and can be parsed into a valid postfix expression tree. Otherwise, the transformation sequence is invalid and will be discarded.  \n\nWe will add such experimental setup details in the appendix.\n\n### **Q2-whether our framework is a meta-network:**\n\nOur goal is to learn a task/domain-optimal model that generates a feature transformation action sequence given a specific dataset-predictive task pair. However, meta-network is a generic network of generalizable knowledge learned over diverse predictive tasks/datasets. The two formulations are different. Besides, to the best of our knowledge, there is no related work adopting meta-network in feature transformation. During the different application scenarios, MAML and Propotypical Networks are not suitable to be baselines. We appreciate your suggestion and we will explore the opportunity to apply meta-learning in our future work.\n\n### **Q3-raw data and our method’s training data are different concepts:**\n\nIn terms of data details: raw data is the data of a predictive task; the training data fed to our model consists of different (feature transformation sequences and corresponding score) pairs, which are explored and collected by our data collection pipeline (Appendix C). Because the scale/space of feature crossing combinations is big, even raw data is not large-scale, our method’s training data on feature crossing combinations can be large. Therefore, the small scale of datasets will have limited influence on our training.\n\nIn terms of implementation details: we used a simplified GCN with symmetric normalization [2] (Appendix B). It can be seen as a fixed encoder. There is no need for additional training. We commit to add more details to the appendix in the revised version. More details can also be found in our publicly available code.\n\n\n[1] Wang, Dongjie, et al. \"Reinforcement-enhanced autoregressive feature transformation: Gradient-steered search in continuous space for postfix expressions.\" Advances in Neural Information Processing Systems 36 (2023): 43563-43578.\n\n[2] Kipf, T. N. \"Semi-Supervised Classification with Graph Convolutional Networks.\" arXiv preprint arXiv:1609.02907 (2016)."}, {"review_id": "QKoQjOZ75k", "reviewer": "Reviewer_Z55B", "summary": "The paper proposes DIFFT, a generative framework for FT that avoids the pitfalls of discrete and continuous search strategies by framing FT as a reward-guided diffusion process. DIFFT uses a VAE to encode discrete feature sequences into embeddings, LDM to sample high-quality embeddings guided by task performance, and a semi-autoregressive decoder to reconstruct structured feature sequences efficiently.", "questions": "See weakness", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- Novelly reframes FT as a generative task rather than a search problem, leveraging the power of diffusion models for global exploration and reward optimization.\n- Outperforms 10 baseline methods on classification and regression tasks.\n- Demonstrates robustness across different downstream models.\n- Extensive ablation clearly shows the importance of diffusion, reward guidance, and decoding strategy.\n\n**Weakness**\n- All results appear to be from a single run, limiting interpretability of performance claims.\n- DIFFT must regenerate features for each downstream task/model, which can become expensive in multi-task scenarios.\n- The method involves training multiple components, which increases implementation and training burden.\n- How does DIFFT scale in memory and time when the feature dimensionality exceeds 50K or more?\n- Is it possible to amortize feature generation across tasks or learn a task-agnostic representation space?\n- Are the ablation results consistent across all datasets, or are there cases where simpler decoding or unguided diffusion performs better?", "was_revised": true, "final_justification": "Many concerns were resolved and I stay my score.", "rebuttal": "We thank you for acknowledging the **novelty, excellent performance, robustness, and clear experiments** of our paper.\n\n### **Q1-beyond single run in experiments:**\nWe agree and fully follow your suggestion to run more experiments. We repeated each experiment five times and report the mean ± standard deviation: \n| **Dataset**       | **Mean ± Std (F-1 / 1-RAE)** |\n| ----------------- | ---------------------------- |\n| Spectf            | 0.881 ± 0.008                |\n| SVMGuide3         | 0.871 ± 0.004                |\n| German Credit     | 0.738 ± 0.008                |\n| Messidor Features | 0.737 ± 0.020                |\n| SpamBase          | 0.930 ± 0.000                |\n| Ap-omentum-ovary  | 0.877 ± 0.006                |\n| Ionosphere        | 0.963 ± 0.011                |\n| Openml\\_586       | 0.633 ± 0.010                |\n| Openml\\_589       | 0.619 ± 0.005                |\n| Openml\\_607       | 0.567 ± 0.006                |\n| Openml\\_616       | 0.324 ± 0.013                |\n| Openml\\_618       | 0.630 ± 0.002                |\n| Openml\\_620       | 0.554 ± 0.031                |\n| Openml\\_637       | 0.292 ± 0.007                |\n\nRepeated five-run averages reveal consistently low standard deviations across nearly all datasets (typically ≤ 0.02), indicating that our method’s performance is stable and not due to lucky single runs.\n\n### **Q2-single-task and multi-task scenario are two different problem settings:**\n\n1) The current problem setting is focused on addressing a widely-used single-task scenario. The multi-task scenario is another problem setting that needs another solution, because multiple predictive targets lead to deviated optimal feature transformations. \nIn the limitations and future work, we discussed the multi-task scenario can be solved by a task-agnostic foundation model. Our DIFFT exactly paves a way toward training a foundation generative feature transformation model. \n\n2) In our experiments, we observed that our DIFFT not just identify task-specific optimal feature transformations to boost predictive accuracy (Table 1), but also significantly reduce time costs. For example, DIFFT completes a training epoch on the large-scale Ap-omentum-ovary dataset more than 25 times faster than MOAT [1] (19.64s vs. 524.4s). \n\n### **Q3-multiple components but actual training costs are low:**\n\n1) Although we have multiple components (look sophisticated), indeed there is a widely-adopted latent diffusion model paradigm [2] (e.g., Stable Diffusion) which strictly requires two-stage training (VAE + diffusion). \n\n2) We proposed the idea of semi-autoregressive decoding to reduce training costs. \n\n3) Our experiments truly show that the total per-epoch training time for DIFFT is significantly reduced. For instance, DIFFT’s time cost is lower than for the end-to-end LSTM-based model in MOAT (Table 3).   \n\n### **Q4-what if feature dimensionality exceeds 50K (ultra-high dimensionality scenarios):**\n1) In our experiments, we tested DIFFT on datasets with up to 10,936 features (Ap-omentum-ovary) and demonstrated strong performance and efficiency. \n\n2) Conventionally, feature transformation methods aim to enhance the expressive power of low-dimensional datasets by expanding the feature dimension. Consequently, these techniques are typically applied to datasets with a limited number of features (e.g., fewer than one hundred). In ultra-high-dimensional scenarios, however, applying feature transformation is generally not a suitable approach. Instead, feature selection is the more conventional strategy, aimed at reducing redundant features to simplify the problem.\n\n3) We acknowledge that over 50K features will place increasing pressure on both memory and time for the VAE and diffusion model. However, DIFFT's architecture is built on a Transformer backbone that can be parallelized. So it is algorithmically more scalable than LSTM-based models in prior literature (e.g., MOAT). \n\n### **Q5-learning task-agnostic representation space of feature transformation:**\nIn this paper, we mainly focus on addressing task-optimal feature transformation problems, and the baselines we compared also follow the same problem.\n\nIf we want a task-agnostic representation space, using self-supervised loss and more feature transformation data can do the job. If we want the entire pipeline to be task-agnostic, we can use the strategy of foundation model. Our reward-guided diffusion framework can be extended to do that. \n\n### **Q6-Same ablation experiments on more datasets:**\n\nWe include results on six additional datasets to investigate the consistency of our ablation study as follows:\n\n| **Dataset** | **NAR** | **AR** | **NoR** |   **10R** | **CS** | **DIFFT (100R)** |\n| ----------- | ------: | -----: | ------: | --------: | -----: | ---------------: |\n| svmguide3   |   0.840 |  0.856 |   0.831 |     0.857 |  0.830 |        **0.871** |\n| messi       |   0.691 |  0.704 |   0.722 |     0.735 |  0.730 |        **0.737** |\n| spectf      |   0.792 |  0.823 |   0.851 |     0.861 |  0.876 |        **0.881** |\n| ionosphere  |   0.958 |  0.956 |   0.955 |     0.958 |  0.943 |        **0.963** |\n| openml\\_586 |   0.621 |  0.628 |   0.623 | **0.633** |  0.610 |        **0.633** |\n| openml\\_589 |   0.510 |  0.606 |   0.603 |     0.611 |  0.561 |        **0.619** |\n| openml\\_616 |   0.230 |  0.302 |   0.237 |     0.262 |  0.289 |        **0.324** |\n| openml\\_618 |   0.487 |  0.617 |   0.623 |     0.624 |  0.620 |        **0.630** |\n\nNotably, we add a new setting which sets the strength coefficient $\\lambda$ to 10 for further validating the effectiveness of the reward mechanism. More detailed description of the variant models (NAR, AR, NoR, and CS) can be found in Section 3.1.\n\nAs shown above, DIFFT(with $\\lambda$ = 100) achieves the best performance across all datasets. These results highlight the strength of our semi-autoregressive VAE design and reward-guided diffusion model.\n\n\n[1] Wang, Dongjie, et al. \"Reinforcement-enhanced autoregressive feature transformation: Gradient-steered search in continuous space for postfix expressions.\" Advances in Neural Information Processing Systems 36 (2023): 43563-43578.\n\n[2] Rombach, Robin, et al. \"High-resolution image synthesis with latent diffusion models.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022."}, {"review_id": "fm9ghzo4EG", "reviewer": "Reviewer_mgFP", "summary": "This paper introduces DIFFT, a framework that extract features from a diffusion model guided by a reward function. This reward is $R_\\psi$, which is a pretrained evaluator. DIFFT integrates VAE, Diffusion Model, and reward-based trajectory sampler and evaluator, which are so diverse component, so there is no theoretical claim on its convergence after feature extraction process. Some empirical results are given, and some ablation studies by checking the necessity of components.", "questions": "1. **Unified Optimization Objective**  \n   Can you formulate a single, unified learning objective that integrates the VAE, latent diffusion model, evaluator network, GCN conditioner, and semi-autoregressive decoder? What theoretical analysis can you provide to show that optimizing this joint objective converges or behaves predictably?\n\n2. **Latent Space & Reward Guidance**  \n   How does the reward signal reshape the latent space over training? Can you provide theoretical reasoning or empirical visualizations (e.g., t-SNE/UMAP plots of embeddings over epochs) to demonstrate how reward guidance tightens the data manifold and influences convergence?\n\n3. **Qualitative Feature Analysis**  \n   What qualitative analyses—such as embedding visualizations, feature attribution maps, or case studies—can you include to illustrate the properties of the extracted features and their relevance to downstream tasks, particularly under distribution shifts or unseen examples?\n\n4. **Dependence on Pretrained Evaluator & Diffusion**  \n   Table 2’s ablation shows that removing the pretrained evaluator (NoR) or diffusion (CS) heavily degrades performance. How can you isolate and quantify each component’s unique contribution? Can you justify the computational cost by demonstrating whether similar gains can be achieved with lighter-weight alternatives or retraining from scratch?", "limitations": "there is no limitation discussions on the paper. no societal impact for sure. however, there are limitations in the method. See the questions to discuss such problems.", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 1, "strengths_and_weaknesses": "Weakness \n\n1. Lack of a Unified Optimization Objective\nThe proposed framework combines multiple components—VAE, LDM, evaluator network, GCN-based conditioning module, and a semi-autoregressive decoder—each trained or updated using separate objectives (e.g., KL divergence, reconstruction loss, MSE loss from evaluator, diffusion denoising loss, etc.). These separate trainings will incur different objectives in its learning and results. There should be problems in these mismatched objective functions. Possibly, not enough gain given the increased model complexity, or catastrophic failure in edge cases between models, etc. Therefore, we need a theoretical analysis on unified objectives because that is the only way to anticipate the behavior of the given framework in the convergence sense. Specifically:\n- There is no formal reasoning about the structure of the latent space or how reward guidance modifies it over time.\n- The convergence behavior of the diffusion process under the influence of the reward signal is not analyzed.\n\n2. Absence of Qualitative Analysis on Extracted Features\nGiven that the paper provides no theoretical analysis explaining why the model works well, then the minimum would be the observation of qualitative results. i.e. visualization or characterization of the learned embedding space, extracted features and its relevance to the downstream task, etc.  Without such empirical analysis, it is difficult to assess when and why the proposed method will generalize or fail, especially under distribution shifts or unseen tasks.\n\n3. The heavy dependence on pretrained evaluator or diffusion\nTable 2 provides an ablation study. NoR and CS are the heaviest damage, so the diffusion and the pretrained evaluator are the key contributor. However, it seems that the pretrained evaluator is providing additional information that have not been used in the previous fair settings. Also, Diffusion models can be very heavy to utilize for this type of downstream task listed in Section 3.5.", "was_revised": true, "final_justification": "I discussed the loss consistency with the author, and there was no clear answer from the authors. I have concerns in many pipelined approach without principled objective function design.", "rebuttal": "We thank you for taking the time to review our paper.\n\n### **Q1-unified optimization objective:**\n\n1) Generally speaking, we don't need a unified optimization objective. According to multiple previous latent diffusion models, e.g. Stable Diffusion [1], DreamFusion [2] and PixArt$\\alpha$ [3], the VAE’s goal is to learn a lossless, geometry-faithful latent space via standard reconstruction, while the diffusion component focuses on generative consistency. \n\n2) When a single objective attempts to balance both, gradients from reconstruction and generative terms often conflict, leading to either blurry latents or unstable sampling, and even collapse. Empirically, separate optimization produces faster convergence, clearer division of labour, and simpler hyper-parameter tuning—hence we follow the established practice of first training the VAE to completion and then training the diffusion model in the fixed latent space.\n\n### **Q2-reward guidance in training:**\n\nWe clarify that 1) The reward signal does **NOT** reshape the latent embedding space, which is indeed learned by the VAE. 2) The reward signal is to guide optimal diffusion sampling. \n\nWe assume you are asking: how reward signal influences diffusion sampling. Below is our theoretical analysis, following RCGDM [4]: \n\nReward acts only at sampling time, tightening samples toward high-reward regions by adding $\\nabla_z R_\\psi(z)$ to the base score.\n\nLet the learned conditional latent distribution be $p_0(z\\mid c)$ with score  \n$s_0(z,t,c)=\\nabla_z \\log p_0(z_t\\mid c)$. Introducing guidance strength $\\lambda$, we effectively sample from the exponentially tilted distribution\n\n$$\np_{\\lambda}(z\\mid c)\\propto p_{0}(z\\mid c)\\exp(\\lambda\\,R_{\\psi}(z)),\n$$\n\nwhich yields the score decomposition\n\n$$\n\\nabla_{z}\\log p_{\\lambda}(z\\mid c)\n  = \\nabla_{z}\\log p_{0}(z\\mid c) + \\lambda\\,\\nabla_{z}R_{\\psi}(z),\n$$\n\ni.e., $\\tilde{s} = s_{0} + \\lambda\\,\\nabla_{z}R_{\\psi}$ in the DDIM/ODE reverse updates.\n\nDefine the energy\n\n$$\n\\mathcal{E}(z) = -\\log p_{0}(z\\mid c) - \\lambda\\,R_{\\psi}(z).\n$$\n\nDiscrete reverse-diffusion steps are gradient descent on $-\\nabla_{z}\\mathcal{E}(z)$. If $\\mathcal{E}$ is smooth and bounded below, the step size is small, and $\\lambda$ is finite, then\n\n$$\n\\mathcal{E}(z_{k+1}) \\le \\mathcal{E}(z_{k}) - \\eta\\,\\|\\nabla_{z}\\mathcal{E}(z_{k})\\|^{2}\n$$\n\nfor some $\\eta > 0$, so energy decreases monotonically and the trajectory converges to a stationary point—practically, high-reward/high-density regions of the manifold.\n\nFollowing Reward-Directed Conditional Diffusion, the reward gap satisfies\n\n$$\nR^{\\star} - \\mathbb{E}[R \\mid p_{\\lambda}]\n  \\le \\varepsilon_{\\text{eval}} + \\varepsilon_{\\text{diff}} + \\text{DistroShift}(\\lambda),\n$$\n\nwhere $\\varepsilon_{\\text{eval}}$ is evaluator error, $\\varepsilon_{\\text{diff}}$ is diffusion approximation error, and $\\text{DistroShift}(\\lambda)$ quantifies the tilt-induced shift. When these are controlled, $\\mathbb{E}[R \\mid p_{\\lambda}]$ exceeds unguided sampling. Under a linear-manifold assumption, sampling remains on the true data subspace, so reward redistributes probability within the manifold rather than distorting it. Thus, reward guidance is a posterior reweighting plus an energy-gradient flow mechanism at inference; training is unchanged and the procedure is predictable and convergent.\n\n### **Q3-case study and quality analysis:**\n\nTo provide the quality analysis of generated features, we conduct a case study on the svmguide3 dataset to\n(1) analyze the importance on downstream model of the generated features, \n(2) evaluate their robustness under a simulated distribution shift. \n\n**Analysis of Feature Importance:**\nOn the original table (21 features) the RandomForest reaches **0.778** F1 Score; After applying DIFFT we obtain **0.866** (+ 0.088).  7/10 most important positions are taken by DIFFT features. This observation shows that DIFFT creates significant features for the downstream model.\n \n**Evaluation under Distribution Shift:**\nThe following table summarizes the performance changes of models using original versus transformed dataset under scaling shifts of different severity. It analyzes the proportion of generated features within the Top-10 most important features for the model using the transformed dataset. \n| Shift Severity | F1 Score (Original Features) | F1 Score (Generated Features) | Generated Features in Top-10 Importance |\n|:--------------:|:---------------------------:|:-----------------------------:|:---------------------------------------:|\n| **0.1**        | 0.7793                      | 0.8411                       | **10 / 10 (100%)**                      |\n| **0.3**        | 0.7584                      | 0.8180                       | **10 / 10 (100%)**                      |\n| **0.5**        | 0.7128                      | 0.8180                       | **10 / 10 (100%)**                      |\n\n\n### **Q4-misunderstanding of the components:**\n\n1) Our diffusion and evaluator are **not pretrained** and **have no prior knowledge** transferred from other datasets. Instead, they are trained from scratch for task-optimal learning. \n\n2) How to perform ablation studies: we remove/replace each of these individual components from the entire framework (Section 3.3).\nWe blow present more ablation studies on six more datasets to show the importance of each individual component: \n\n| **Dataset** | **NAR** | **AR** | **NoR** |   **10R** | **CS** | **DIFFT (100R)** |\n| ----------- | ------: | -----: | ------: | --------: | -----: | ---------------: |\n| svmguide3   |   0.840 |  0.856 |   0.831 |     0.857 |  0.830 |        **0.871** |\n| messi       |   0.691 |  0.704 |   0.722 |     0.735 |  0.730 |        **0.737** |\n| spectf      |   0.792 |  0.823 |   0.851 |     0.861 |  0.876 |        **0.881** |\n| ionosphere  |   0.958 |  0.956 |   0.955 |     0.958 |  0.943 |        **0.963** |\n| openml\\_586 |   0.621 |  0.628 |   0.623 | **0.633** |  0.610 |        **0.633** |\n| openml\\_589 |   0.510 |  0.606 |   0.603 |     0.611 |  0.561 |        **0.619** |\n| openml\\_616 |   0.230 |  0.302 |   0.237 |     0.262 |  0.289 |        **0.324** |\n| openml\\_618 |   0.487 |  0.617 |   0.623 |     0.624 |  0.620 |        **0.630** |\n\nNotably, we add a new setting which sets the strength coefficient $\\lambda$ to 10 for further validating the effectiveness of the reward mechanism. More detailed description of the variant models (NAR, AR, NoR, and CS) can be found in Section 3.1.\n\n[1] Rombach, Robin, et al. \"High-resolution image synthesis with latent diffusion models.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.\n\n[2] Poole, Ben, et al. \"Dreamfusion: Text-to-3d using 2d diffusion.\" arXiv preprint arXiv:2209.14988 (2022).\n\n[3] Chen, Junsong, et al. \"Pixart-$\\alpha $: Fast training of diffusion transformer for photorealistic text-to-image synthesis.\" arXiv preprint arXiv:2310.00426 (2023).\n\n[4] Yuan, Hui, et al. \"Reward-directed conditional diffusion: Provable distribution estimation and reward improvement.\" Advances in Neural Information Processing Systems 36 (2023): 60599-60635."}, {"review_id": "ZK5xh1ExmE", "reviewer": "Reviewer_FZ7e", "summary": "This paper proposes DIFFT, a reward-guided generative framework for feature transformation. The method aims to address the limitations of both discrete and continuous search paradigms by leveraging a latent diffusion model guided by a performance evaluator. It first encodes feature token sequences using a VAE, then generates high-quality embeddings via reward-guided diffusion, and finally decodes them into discrete feature sets using a semi-autoregressive decoder. Extensive experiments and case studies are performed to demonstrate the effectiveness and robustness of the proposed method.", "questions": "1.\tThe paper evaluates performance under increasing feature dimensionality. Have the authors considered how varying sample sizes may affect training and inference time efficiency?\n2.\tIn Table 1, RDG performs surprisingly well on the ap-omentum-ovary dataset, achieving the same performance as MOAT and ELLM. Could the authors explain why a random baseline is competitive here?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1.\tThe paper presents a novel and well-motivated formulation of the feature transformation problem, framing it as a reward-guided generative process. This perspective is timely and aligns with recent trends in generative modeling.\n2.\tThe proposed method achieves strong performance across diverse datasets, consistently outperforming competitive baselines in both predictive accuracy and computational efficiency.\n3.\tThe experimental evaluation is thorough, including ablation studies and robustness checks, which help validate the effectiveness of the proposed components and support the overall claims.\n4.\tThe authors released the related code and data, which can help other researchers reproduce the experiments.\n\nWeaknesses\n1.\tThe paper introduces a GCN-based graph embedding for conditioning the diffusion model, but does not explain why this design is necessary or superior. A brief justification would help clarify its role and benefit.\n2.\tWhile time efficiency is reported, the paper lacks discussion on memory or space consumption. Given the added components like VAE and diffusion models, this could affect scalability in larger settings.", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for acknowledging the **novel and well-motivated formulation, strong performance, and thorough experimental evaluation** of our paper.\n\n### **Q1-more experiments in terms of varying sample sizes:**\n\nSince the training data consists of feature transformation sequence and corresponding score, only the feature dimensionality has an influence on the training efficiency (sequence length).\n\nTo verify this, we run an additional study in which we fix the feature dimensionality and vary the sample size only. \n\n| Dataset      | Sample # | Feature # | DIFFT (D) [Train (s)] | DIFFT (V) [Train (s)] | DIFFT [Train (s)] | DIFFT [Infer(s)] |\n|--------------|---------:|----------:|-----------------------:|-----------------------:|-------------------:|-------------------:|\n| Openml_616   | 500      | 50        | 5.21  | 7.57 | 12.77 | 0.254 |\n| Openml_637   | 500      | 50        | 4.65  | 8.13 | 12.78 | 0.255 |\n| Openml_607   | 1 000    | 50        | 4.15  | 8.45 | 12.61 | 0.253 |\n| Openml_618   | 1 000    | 50        | 4.74  | 7.45 | 12.19 | 0.258 |\n\nAccording to the results, DIFFT shows nearly identical training times across datasets with different sample sizes. \n\n### **Q2-why the random method (RDG)’s performance is not bad in a specific dataset:**\n\nThe random method is not our focus, but this is an interesting side observation. The random method didn’t perform well on many datasets. The specific ap-omentum-ovary dataset is high-dimensional (>10000 features), making the search space extremely large. While RDG randomly obtain a great performance, the two baselines MOAT and ELLM fall into a local optima in a large space.\n\n### **Q3-explain the GCN-embedding design:**\n\nFeature transformation sequences are “order-free bags of features”; their semantics are governed by the pairwise (and higher-order) relations among columns rather than by token order. We therefore convert each transformed table into a *feature–feature graph* and let a simplified GCN produce a permutation-invariant conditioning vector of diffusion model.\n\n### **Q4-memory consumption:**\n\nOur method is lightweight: on a single RTX 4090, training for both the diffusion model and the VAE stays below 10 GB of GPU memory, while inference requires only about 3 GB. We will add more details in the appendix further."}]

Author rebuttal:

Meta-review: Feature transformation refers to producing new features from original features using Mathematical operations.  The paper proposes Reward-Guided Diffusion Framework for Semi-Autoregressive Generative FT  DIFFT that casts feature transformation as a reward guuided  generation process  using  diffusion models of the transforms that need to be applied to the original features and that lead to low error in terms of performance on a specific task. Models are trained from scratch and independently for each task and dataset. The model consists of a VAE, a denoiser conditioned on a graph representation of a dataset, and a guidance model that takes downstream task performance as the reward. Experiments on several benchmarks show that the proposed method outperform similar method in that space of feature transformations. 

Reviewer sxKk mentioned the paper lacks clarity on which operations are used and that the work would fit better in a meta learning context for a single network that produce features for each specific task and dataset. Authors responded to this and the reviewer was satisfied, saying the paper is interesting to this niche area. 

Reviewer Z55B asked for statistical significance results which were provided in the rebuttal. Similar criticism of the multi-component nature of the work and the fact that a new training  is needed for each task/dataset was expressed , as well as more ablation study on the proposed method. ablations were provided and  explanation that the cost of the model provide number of features is small (10 K) cost of training per dataset is manageable. 

Reviewer mgFP is the most critic of the work and discussed the paper with the authors, with the main criticism being that there is no  coherence in the loss to train the present method since it consists of multiple stages and multiple components. Authors clarified that the training consists of two stages  in training and and test time compute to generate latents and the features. 

Reviewer FZ7e asked about the scalability and requirement in terms of space and memory of the method given its multiple components involved , also asked an ablation in terms of number of sampled used. Authors provided elements of responses in the rebuttal, unfortunately the reviewer did not engage in the discussion period , despite emails and pings on open-review from AC. 

Having read the paper and discussed it with SAC,  the idea of using generative model to generate the features is interesting and to the best of my knowledge novel. Nevertheless the complexity of the method and its applicability to tabular datasets that need retraining for each model renders the method rather complex and it would be appealing to fit it in the meta learning context as proposed by the reviewers.  The coherence of the loss brought by mgFP  is not problematic, since this is common in Deep Learning and the method is ablated and evaluated in the same setup of previous methods in that field, and the paper is rather empirical than theoretical. Given that papers with similar methodology and evaluations have been published in similar venues to neurips and that the method outperforms them, I recommend weak acceptance of the paper.
Decision: Accept (poster)