Iterative Feature Space Optimization through Incremental Adaptive Evaluation

Authors: Yanping Wu, Yanyong Huang, Zhengzhang Chen, Zijun Yao, Yanjie Fu, Kunpeng Liu, Xiao Luo, Dongjie Wang
Primary area: other topics in machine learning (i.e., none of the above)
Keywords: Automated Feature Optimization, Incremental Learning, Feature Space Evaluator

Abstract:
Iterative feature space optimization involves systematically evaluating and adjusting the feature space to improve downstream task performance. However, existing works suffer from three key limitations: 1) overlooking differences among data samples leads to evaluation bias; 2) tailoring feature spaces to specific machine learning models results in overfitting and poor generalization; 3) requiring
the evaluator to be retrained from scratch during each optimization iteration significantly reduces the overall efficiency of the optimization process. To bridge these gaps, we propose a gEneralized Adaptive feature Space Evaluator (EASE) to efficiently produce optimal and generalized feature spaces. This framework consists of two key components: Feature-Sample Subspace Generator and Contextual Attention Evaluator. The first component aims to decouple the information distribution within the feature space to mitigate evaluation bias. To achieve this, we first identify features most relevant to prediction tasks and samples most challenging for evaluation based on feedback from the subsequent evaluator. These identified feature and samples are then used to construct feature subspaces for next optimization iteration. This decoupling strategy makes the evaluator consistently target the most challenging aspects of the feature space. The second component intends to incrementally capture evolving patterns of the feature space for efficient evaluation. We propose a weighted-sharing multi-head attention mechanism to encode key characteristics of the feature space into an embedding vector for evaluation. Moreover, the evaluator is updated incrementally, retaining prior evaluation knowledge while incorporating new insights, as consecutive feature spaces during the optimization process share partial information. Extensive experiments on twelve real-world datasets demonstrate the effectiveness of the proposed framework. Our code and data are publicly available (https://anonymous.4open.science/r/EASE-1C51).

Decision: reject
Reviewer scores: 5, 5, 5

Reviews and discussion:
<thread>
creation date:
1729136668896

thread_type:
official_review

<thread_object>
creator: reviewer_wb9f

<content>
summary:
This paper proposes a method to iteratively find good feature space (specifically, a set of relevant features in the original features) to improve downstream task performance. For this, the authors propose a Feature-sample subspace generator that finds relevant features and difficult samples in each optimization step and a Contextual attention evaluator that evaluates the selected features so that it can improve the prediction performance. The experiments show that the proposed method outperformed existing iterative feature space optimization methods.

strengths:
- The feature selection is an important task in machine learning.
- The effectiveness of the proposed method was evaluated with many datasets including classification and regression tasks and many evaluation metrics.

weaknesses:
- The significance of the task of iterative feature space optimization is not clear. For example, even if you don't find a feature space (specifically, feature selection) iteratively, lasso or neural network-based embedding-based feature selection methods can select features that directly improve the performance of downstream tasks by optimizing a single objective function once. This paper may be trying to solve unnecessarily complex problems.
- The presentation quality and clarity of this paper are low. For example, in Eq. 1, the evaluator $M$ takes the feature space as input and calculates the loss with the label space. What is the loss between spaces? Isn't it correct to say data (feature) matrix and label vector, etc., rather than spaces? The word 'space' is associated with mathematical vector spaces, etc., which can be confusing. I also need help understanding Eq. 3. Is the score a scalar or a vector? Because the specific form of $F^t$ (whether it's a vector, a matrix, or something else) is not described, the output of $M$ is also unclear. Thus, the definition of Eq. 3 is ambiguous. 
Also, Eq. 2 in section 3 is different from Eq. 10, which is actually used. What is the intention of introducing a specific formula Eq. 2 at the problem-setting stage? I think that this makes the paper unnecessarily difficult to understand.I think the overall explanation of the proposed method needs to be improved.

questions:
- Since this paper is considering feature selection, wouldn't it be easier for readers to understand if the authors propose the method as a feature selection method rather than using the abstract term 'feature space optimization'? 
- In Figure 3, why is the proposed method faster than other comparison methods? Since the proposed method is based on neural networks (attention), I think it is more expensive.
</content>
</thread_object>

</thread>
<thread>
creation date:
1729929310486

thread_type:
official_review

<thread_object>
creator: reviewer_eoma

<content>
summary:
This paper designs a new framework (i.e., EASE) to enhance the optimization of feature spaces in machine learning tasks. EASE addresses common limitations in existing methods, such as evaluation bias, poor generalization, and inefficient training. It comprises two main components: the Feature-Sample Subspace Generator and the Contextual Attention Evaluator. The former mitigates evaluation bias by decoupling information within the feature space, while the latter captures evolving patterns incrementally for efficient evaluation. The framework is tested on twelve real-world datasets, demonstrating its effectiveness and efficiency over traditional methods.

strengths:
1. The paper effectively tackles prevalent issues in feature space optimization, such as bias, generalization, and training inefficiency, providing a more robust evaluation methodology.

2. The combination of Feature-Sample Subspace Generator and Contextual Attention Evaluator offers a comprehensive solution that both decouples complex interactions and captures evolving patterns in feature spaces.

3. The extensive experiments on multiple real-world datasets substantiate the superiority of EASE in terms of both accuracy and efficiency, enhancing the paper's credibility.

weaknesses:
1. The proposed method conduct the efficiency comparison experiments of different feature space evaluators across various datasets. It would be better to further analyze the time complexity of different feature space evaluators to demonstrate the efficiency of the proposed method.

2. The proposed method is applied to two iterative feature selection frameworks to validate its effectiveness and generalization capability, i.e., RFE and FLSR. However, both the two baselines are out-of-date. Are there any recent baselines can be applied to?

3. The used several datasets are relatively small, i.e., almost all of their sample sizes are small than 10 thousand. How about applied the proposed method on large-scale datasets?

4. I am a little concerned about the innovativeness of the proposed method, because Section 4.2 contextual attention evaluator seems to simply apply the self-attention mechanism.

questions:
See above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730556642259

thread_type:
official_review

<thread_object>
creator: reviewer_67bs

<content>
summary:
This paper proposes a general adaptive feature space evaluator named EASE, designed to optimize feature spaces through incremental updates and a contextual attention mechanism. By generating feature-sample subspaces and conducting incremental evaluations, this framework improves the efficiency and accuracy of feature selection, with its performance validated across multiple real-world datasets.

strengths:
1. The EASE framework includes two key components: a feature-sample subspace generator and a contextual attention evaluator. The feature-sample subspace generator creates feature subspaces relevant to downstream tasks, allowing the evaluator to focus on the most challenging subspaces in each iteration.
2. An incremental update strategy is introduced in this paper, which retains historical parameter weights and updates only critical parameters when new feature spaces appear, reducing the computational cost of retraining from scratch each time. Additionally, an Elastic Weight Consolidation (EWC) strategy is used to calculate Fisher information.

weaknesses:
1 The paper presents the innovative EASE framework. However, it provides limited implementation details, such as hyperparameter configurations, optimization strategies, and specific training processes. 
2 The paper compares EASE with some common models (such as GBDT and Random Forest) but lacks detailed comparisons with current state-of-the-art feature selection or feature space evaluation methods.
3 The theoretical analysis of EASE relies on specific assumptions about data distribution and feature relevance, which may not always align with real-world data, especially in datasets where features are weakly related or uncorrelated.

questions:
1. The complexity of feature subspace generation and multi-head attention mechanisms still makes the EASE framework computationally intensive during training. Due to the large number of parameters in multi-head attention, which require constant updating, the efficiency of EASE may be suboptimal in large-scale datasets or high-dimensional feature spaces.
2. The regularization parameters mentioned in this paper significantly impact the stability and adaptability of parameters in incremental updates, but detailed tuning strategies for different scenarios are lacking. Additionally, the size and sampling strategy of the feature-sample subspaces generated in different iteration steps could directly affect the accuracy and computational cost of final feature space evaluation.
3. While the EASE framework performs well experimentally, it lacks theoretical analysis to explain its generalizability across different feature spaces and tasks. For instance, the paper does not provide detailed theoretical support for whether incremental updates are effective across all feature space distributions, or whether the contextual attention mechanism is universally applicable.
</content>
</thread_object>

</thread>


Metareview:
metareview:
Thanks for your submission to ICLR.

This paper received borderline reviews, leaning to reject.  All three reviewers noted several issues with the manuscript, including i) limited implementation details / issues with presentation, ii) missing empirical comparisons (data sets, baselines), iii) concerns about the theoretical analysis, iv) novelty.  The rebuttal addressed some of these issues, but during the discussion phase, two of the three reviewers responded that they still felt the paper was not ready for publication.  The third reviewer did not respond.

Given the issues that persist after the discussion phase, and the relatively low scores, I think this paper needs some additional work before it is ready for publication.  Please keep in mind the comments of the reviewers when preparing a future version of the paper.