Range-limited Augmentation for Few-shot Learning in Tabular Data

Authors: Kyungeun Lee, Moonjung Eo, Hye-Seung Cho, Suhee Yoon, Sanghyu Yoon, Ye Seul Sim, Woohyung Lim
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Tabular representation learning; Contrastive learning; Few-shot learning; Data augmentation

Abstract:
Few-shot learning is essential in many applications, particularly in tabular domains where the high cost of labeling often limits the availability of annotated data. To address this challenge, we propose range-limited augmentation for contrastive learning in tabular domains. Our augmentation method shuffles or samples values within predefined feature-specific ranges, preserving semantic consistency during contrastive learning to enhance few-shot classification performance. To evaluate the effectiveness of our approach, we introduce FeSTa (Few-Shot Tabular classification benchmark), a benchmark consisting of 42 tabular datasets and 31 algorithms. On this benchmark, contrastive learning with our augmentation method effectively preserves task-relevant information and significantly outperforms existing approaches, including supervised, unsupervised, self-supervised, semi-supervised, and foundation models. In particular, our method achieves an average rank of 2.3 out of 31 algorithms in the 1-shot learning scenario, demonstrating its robustness and effectiveness when labeled data is highly limited. The benchmark code is available in the supplementary material.

Decision: reject
Reviewer scores: 6, 3, 5, 5

Reviews and discussion:
<thread>
creation date:
1730100652692

thread_type:
official_review

<thread_object>
creator: reviewer_nqqr

<content>
summary:
This paper introduces the limited range expansion of a small amount of learning in tabular data. This method aims at maintaining semantic consistency in the learning process, which is very important to improve the classification performance under the condition of limited labeled data. The authors have proved the effectiveness of their method through a new benchmark test, which is superior to the existing methods.

strengths:
Originality: The concept of range-limited augmentation for contrastive learning in tabular data is novel. The paper introduces a creative solution to the challenge of designing augmentations that preserve task-relevant information in a domain. The application of distribution-based range limiting to augmentation strategies is an innovative approach that leverages the inherent structure of tabular data, which is a creative combination of existing ideas in data augmentation and representation learning. The introduction of the FESTA benchmark is original as it addresses a gap in the evaluation of few-shot learning methods for tabular data, providing a new resource for the community.
Quality: The paper is well-structured, with a clear problem statement, detailed methodology, comprehensive experiments. The paper provides a rigorous evaluation of the proposed method through extensive experiments on a diverse set of datasets and algorithms.
Clarity: The paper is well-written, with a clear and logical flow that makes it easy to follow the authors' line of reasoning.
Significance: The paper addresses a significant problem in the field of machine learning, where labeled data is often scarce, especially in specialized domains like healthcare or finance. The proposed solution has the potential to impact a wide range of applications by enabling more effective learning from limited labeled data, which is a common scenario in many real-world scenarios. It provides a standardized way to evaluate and compare few-shot learning methods in tabular data, which could drive further research and development in this area.

weaknesses:
1. The paper provides empirical evidence of the effectiveness of the proposed method but does not offer a theoretical analysis of why range-limited augmentation works better than other methods in preserving task-relevant information.  Developing a theoretical framework to explain these observations could further strengthen the contribution.
2. In classifiation-related papers, confusion matrix is an important basis for judging a classifier, so it is suggested to add more blanks here.
3. There are some typos in the manuscripts. The presentation of this manuscript can be improved.

questions:
1. Can you provide a theoretical explanation for how effective a scope-limited augmentation method is in preserving task-related information compared to other augmentation methods?
2. What specific optimizations can be implemented to reduce the computational overhead associated with scope-constrained enrichment?
3. Will the benchmark code be exposed on a platform like GitHub to promote reproducibility?
4. Based on the findings of this study, what are the plans for future research?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730360035694

thread_type:
official_review

<thread_object>
creator: reviewer_kssf

<content>
summary:
This paper addresses a challenge in tabular data with approaches from few-shot learning. In particular, this paper proposes range-limited augmentation equipped with contrastive learning to build a model capable of tackling learning on tabular domains. The proposed approach provides augmentation by shuffling or sampling values within predefined feature-specific ranges, preserving semantic consistency. In addition to the proposed approach, this paper also introduces a benchmark to evaluate the efficacy of  the proposed approach compared to the counterparts so-called Few-Shot Tabular Classification (FESTA). Through experiments on the benchmark, the proposed approach with augmentation with contrastive learning the performance can be improved compared to the state-of-the-arts in few-shot learning.

strengths:
- This paper introduces an underexplored problem by focusing few-shot learning on tabular data. This is due to data unavailability in tabular domains, prohibiting training in a large scale manner. Thus, few-shot learning is a reasonable perspective in tackling such problems.
- This work also introduces a novel benchmark with 42 tabular datasets and 31 techniques for comparison. This benchmark has the potential to serve as a resource for future research efforts, providing a standardized comparison and facilitating the evaluation of new methods in tabular data analysis.

weaknesses:
- The problem scope is quite narrow and the proposed approach only works well in tabular domains. Additionally, the proposed augmentation approach demonstrates effectiveness primarily within tabular data domains and might not generalize to other types of data structures or formats.
- This paper has serious clarity issues in terms of problem introduction. The problem in this work is not elaborated well in the paper. The problem setup is quite general in which previous few-shot learning papers have discussed it. However, a specific focus problem setup on tabular domains is not described in the paper. Additionally, there is no clear problem description to differentiate among few-shot learning in several domains e.g., image, text, and tabular domains. This causes confusion in understanding the core problem in tabular domains compared to other modalities.
- Based on my understanding, the augmentation method is based on the ranged limited shuffling and sampling. However, this method is largely inspired by augmentation methods in the image domain, but here it is applied to a different domain. The strategy closely resembles what SimCLR introduced in its paper, with only a specific range constraint added to the technique. In SimCLR, we can interpret “range-limited sampling” as modifying the colors from a range between 0-255, and shuffling as cropping from a specific region. Therefore, the novelty of this approach sounds quite limited.
- The writing is quite poor. There are many irrelevant points in the paper e.g., lists of detailed baselines and augmentation. These sections do not provide meaningful insights or support the main contributions. As a result, the core contributions are obscured by unrelated information that does not directly connect to the main focus of the paper.

questions:
This paper missed explanation on how each method can be used to the problem at hands. For instance, XGBoost, CatBoost, LightBGM are not methods designed for few-shot learning. How are these methods modified to accommodate learning with a few data?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730537399760

thread_type:
official_review

<thread_object>
creator: reviewer_ceax

<content>
summary:
The paper proposed two things. First, the few-shot learning benchmark for tabular datasets with a focus on 1 and 5-shot learning. Second, they proposed a data augmentation method for tabular datasets by additionally considering the range of the feature value for the feature augmentation. The author shows that the proposed method is effective by combining it with contrastive learning.

strengths:
1. The overall paper is well-written and easy to read.

2. The author has conducted a new benchmark.

weaknesses:
1. The proposed method is only applicable to numerical values. This is a highly critical weakness since many tabular datasets consist of categorical values.

2. Limited novelty. The proposed idea is a data augmentation method with a range limitation. In this regard, the only novelty is the range limitation, which I think is sensible as an engineering technique yet not as novel as an academic paper to be accepted.

3. The focused problem is too narrow. I do believe few-shot learning for tabular datasets [1] is important, but the more important problem is the full-shot setup. I think in a full-shot setup, the most dominant methods are tree-based methods, where the proposed method is somewhat hard to utilize. Furthermore, I think the author should consider a broader range of few-shot if they want to focus on few-shot learning, e.g., 8, 16, 32, and 64-shot learning.

4. In a few-shot tabular learning setup, recent LLM-based methods have become effective. I think it would be interesting to combine the current method with the LLM-based, few-shot, tabular learning methods [2,3,4,5].

5. It would be more interesting to suggest a new self-supervised learning (or any learning) method that can better leverage the proposed augmentation. I believe this will enhance the novelty.

In summary, I recommend rejection. The weakness mentioned above is quite critical and needs to be addressed. 

Reference\
[1] STUNT: Few-shot Tabular Learning with Self-generated Tasks from Unlabeled Tables, ICLR 2023\
[2] LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Tasks, NeurIPS 2022\
[3] Language models are weak learners, NeurIPS 2023\
[4] Large Language Models Can Automatically Engineer Features for Few-Shot Tabular Learning, ICML 2024\
[5] Tabular Transfer Learning via Prompting LLMs, COLM 2024

questions:
Please refer to the weakness part above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730624368912

thread_type:
official_review

<thread_object>
creator: reviewer_qff3

<content>
summary:
The paper presents a novel data augmentation method called Range-Limited Augmentation, which is designed to improve few-shot learning for tabular data. The proposed approach involves shuffling or sampling feature values within predefined ranges to maintain semantic consistency and improve model performance. Also, the paper introduces a Few-Shot Tabular Classification Benchmark (FESTA), encompassing 42 tabular datasets and 31 algorithms to evaluate the effectiveness of their approach. The paper demonstrates that the proposed method achieves superior results in one-shot classification scenarios, outperforming existing supervised, unsupervised, and foundation models.

strengths:
1. The proposed range-limited augmentation is a unique solution to the challenge of maintaining semantic consistency in tabular data augmentations, which is crucial for effective contrastive learning.
2. The proposed FESTA benchmark offers a large-scale, diverse set of tabular datasets and learning paradigms, making it a valuable for future research opening.
3. The method shows substantial performance improvements, especially in 1-shot scenarios, and maintains competitive performance in 5-shot setups.
4. The analysis using mutual information and representation invariance score seems a reasonable choice to evaluate the quality of data augmentation.

weaknesses:
1. The proposed method seems straightforward when applying to numerical features, but it is unclear how it can be adapted to categorical data. Also, it is not clearly analyzed how the proposed method perform in homogeneous tabular dataset.
2. The proposed method is evaluated only on proposed FESTA benchmark, which may not cover the entire trend in tabular data learning. For example, the paper could include OpenML-CC18 dataset as preliminary benchmark.
3. The proposed method only considers classification task, where regression tasks are also important aspect of tabular data learning.

questions:
1. Could the author elaborate how the proposed range limited shuffling and range limited sampling applies to categorical features? 
2. Could the author provide empirical results on other popular OpenML dataset?
3. Could the proposed method be applied to regression tasks?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732076964498

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Dear Reviewers,

We would like to express our sincere gratitude to the reviewers for their thoughtful and constructive feedback on our manuscript. We greatly appreciate the time and effort each reviewer has taken to engage with our work, and we welcome further discussions that can help improve the quality of the manuscript.

Here, we summarize the strengths of our paper as highlighted in the reviews:

1. Novel contributions
    - Range-Limited Augmentation: A novel technique for contrastive learning in tabular data, preserving task-relevant information within feature-specific ranges. (NQQr, qfF3)
    - FeSTa Benchmark: A benchmark for few-shot learning on tabular data, including diverse datasets and methods, supporting standard evaluation and future research. (NQQr, qfF3, ceaX, ksSf)
2. Addressing a Key Challenge
    - Few-Shot Learning in Tabular Domains: Targeting the crucial issue of data scarcity in tabular fields like healthcare and finance, where privacy and cost limit data availability. (ksSf, NQQr)
3. Methodological Rigor and Performance
    - Robust Evaluation Metrics and Results: The proposed method demonstrates substantial performance gains, particularly in 1-shot scenarios, while remaining competitive in 5-shot setups. This effectiveness is further supported by empirical analysis for the quality of data augmentation (qfF3).
4. Clarity and Presentation
    - Clear and Structured: Well-organized, with clear problem definition, methods, and extensive evaluation across diverse datasets, making contributions easy to follow. (ceaX, NQQr)

We thank the reviewers for the reviews, and we will provide detailed responses to the specific weaknesses and questions in our official comments for each reviewer.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces a range-limited augmentation method for contrastive learning in tabular data and the FeSTa benchmark for few-shot learning on tabular datasets. The proposed method demonstrates improvements in preserving task-relevant information and achieving strong performance compared to existing methods. 

However, after the rebuttal period, most reviewers still kept their negative ratings and tended to reject this paper. There are still important remaining concerns, e.g., the technical novelty of the proposed method is somewhat limited. 

In the end, three reviewers give borderline and reject scores. Given the current weaknesses of the paper, the paper is rejected from the highly competitive ICLR conferences. The authors are encouraged to revise their manuscript according to the reviewers' comments and submit it to the next venue.