Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a meta-learning approach based on generation of new tasks from the columns of the training data. The authors's technique targets tabular data through modification of the task generation using randomization of the columns as well as an unsupervised validation technique. Through generation of such new tasks, the meta learning improves in it few shot learning performance. The proposed algorithm improves upon the state of the art notably. 
strength_and_weaknesses: Strengths
1. Good motivation of the problem.
2. Thorough literature survey.
3. Sound adaptation of task generation based meta-learning to tabular data. Thorough understanding of the tabular data domain. 
4. Good results

Weaknesses
1. Minor adaptation of the state of the art. Need to make clear why the proposed improvement is major.

clarity,_quality,_novelty_and_reproducibility: Clarity
The paper is well written and clear.

Novelty

The  work comes across as a minor improvement of the state of the art.
Reproducibility

The paper can certainly be reproduced because the work is described with sufficient clarity and code is provided.
Quality and Originality

The soundness of the results is commendable. The ideas are novel. I would like to be better convinced about the advance over the state of the art.
summary_of_the_review: The paper presents a technique that is well motivated by the needs of the target domain and addresses them well. The proposed technique achieves notable improvement over the state of the art. The advancement of the technique itself over the state of the art needs to be better established.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes an approach to few-shot learning with tabular data called STUNT, which creates synthetic meta-learning tasks from unlabeled tabular data by clustering randomly selected subsets of columns to form pseudo-labels. A pseudo-validation method is also proposed that can be used to perform early-stopping using the unlabeled data only. Experiments on a collection of tabular datasets representing a mix of numerical and categorical columns demonstrate that STUNT exhibits strong performance relative to a wide variety of baselines.
strength_and_weaknesses: Strengths
- The proposed method is simple and intuitive.
- Experiments are fairly extensive and compare against a variety of baselines on several datasets and runs over multiple random seeds.
- Limitations of the method are discussed with respect to low-shot learning.

Weaknesses
- Novelty is somewhat low as the method could be viewed as a minor modification to CACTUs.
- There are a few design decisions that are not well justified: the use of all columns for clustering in the pseudo-validation, the effect of marginal distribution masking versus other types of masking, and how the column masking ratio should be chosen.
clarity,_quality,_novelty_and_reproducibility: Clarity is overall good. The method is mathematically described with sufficient precision. The novelty of the method is somewhat low as many techniques for generating meta-learning tasks from unsupervised data have been explored recently. This is counterbalanced to some extent by the application area of tabular data, which is relatively underexplored. Quality is good and the claims of the paper appear to be supported by experiments. As for reproducibility, code is included and experiments are described in sufficient detail.

Minor comment: Eq (2) appears to be missing negations inside the exps.
summary_of_the_review: Post-rebuttal update: I would like to thank the authors for responding to my concerns. The paper has been improved quite a bit during the rebuttal phase. I appreciate the additional results on regarding the distribution masking, extra baselines as suggested by RsZ8N, and extensions to regression tasks as suggested by R7dUE. These changes have made the paper more complete, and I will update my rating accordingly.

---

This paper addresses an interesting problem, i.e. few-shot learning of tabular data, and proposes a simple yet effective solution to address it. The results appear to yield a significant improvement relative to baselines. The weakest point of the submission is novelty with respect to previous work on task generation from unsupervised data. However, the proposed method elegantly adapts previous work to the tabular setting and achieves good results.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes the Self-generated Tasks from UNlabeled Tables (STUNT) for few-shot tabular learning, where there are only a limited number of labeled training examples. STUNT applies unsupervised meta-learning to learn generalized knowledge, and an unsupervised validation strategy is also proposed. Experiments on various few-shot tabular data learning benchmarks validate the effectiveness of the method.

strength_and_weaknesses: This paper proposes a novel aspect of semi-supervised tabular data learning, which shows that unsupervised meta-learning is an effective tool for this task. The idea of task generation from unlabeled tables is novel and interesting. The experiments show the method could be generalized in various cases.

Here are some suggestions and questions for the paper:
1. There is another thread of unsupervised meta-learning such as [1] and [2]. The experiments in these papers show efficient unsupervised meta-learning works better than CACTUS and UMTRA. The authors need to discuss this method in the related work part and try to compare with them in experiments. [1] also shows the requirement of plenty of tasks in unsupervised meta-learning. How will the diversity of the generated tasks influence the convergence of the model?
2. The experiments show the unsupervised task works better than the contrastive strategy such as SubTab. Could the authors explain the reason why unsupervised meta-learning is better? Could we combine these strategies together?
3. The authors mentioned "We also exclude UMTRA (Khodadadeh
et al., 2019) from the baseline because it is not clear how to design the augmentation strategy for tabular data (UMTRA uses image augmentations)". Could the augmentation strategy in SubTab be applied with UMTRA and other unsupervised meta-learning methods?
4. In the step "adapting with labeled samples", does the model be fine-tuned on the labeled samples or only construct the prototypes directly?
5. How will the model performance changes w.r.t. different numbers of shots?

[1] Ye, Han, and Zhan. Revisiting Unsupervised Meta-Learning via the Characteristics of Few-Shot Tasks. TPAMI.
[2] Chen, Maji, and Learned-Miller, Shot in the dark: Few-shot learning with no base-class labels. CoRR.
clarity,_quality,_novelty_and_reproducibility: The main idea is clear. The usage of "few-shot" is a bit confusing since the method is introduced based on the semi-supervised learning setting. Although the authors apply STUNT on other few-shot applications, it should be made clear in the beginning of the paper.

summary_of_the_review: The paper is overall novel and interesting. Experiments show some convincing results. Some additional experiments and discussions are required to make the paper more solid. Please check the weakness part.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a new self-supervised training method for tabular learning, which leads to good performance improvements in the empirical analysis.  
strength_and_weaknesses: Strength:
  - This paper is well-written and easy to follow. One can understand the proposed method by simply looking at figure 1.
  - The proposed method is simple yet technically sound. 
  - The empirical analysis shows the proposed method leads to a good performance boost over the baselines. The results are reliable since it's calculated over a lot of random seeds.

Weakness:
  - Since the method is proposed for `tabular learning` and  `regression` tasks are very common in tabular learning, I am curious to see how this method performs on tabular regression tasks. 
clarity,_quality,_novelty_and_reproducibility: Clarity: Good.

Quality: Good.

Novelty: Good. 

Reproducibility: Code provided, hyperparameters clearly stated.
summary_of_the_review: This paper presents a novel self-supervised learning method for few-shot tabular classification tasks. The method is simple and effective. The empirical results are convincing.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: 
The authors proposed a simple yet effective approach for few-shot learning on tabular data, dubbed Self-generated Tasks from UNlabeled Tables (STUNT). Their key idea is to treat randomly chosen columns as a target label to self-generate diverse few-shot tasks to learn a good representation. 

Strengths
-------------
- Reviewer Gn88,Reviewer sZ8N : Good motivation of the problem, 
- Reviewer Gn88: Thorough literature survey. Sound adaptation of task generation based meta-learning to tabular data. 
- Reviewer Gn88, Reviewer 26Xs:  Thorough understanding of the tabular data domain. Limitations of the method are discussed with respect to low-shot learning.
- Reviewer Gn88, 26Xs, sZ8N,r 7dUE:  Good results. Experiments are fairly extensive and compare against a variety of baselines on several datasets and runs over multiple random seeds.
- Reviewer 26Xs, Reviewer 7dUE:: The proposed method is simple and intuitive.
-Reviewer sZ8N : The paper proposes a novel aspect of semi-supervised tabular data learning, which shows that unsupervised meta-learning is an effective tool for this task.
-Reviewer 7dUE: This paper is well-written and easy to follow. One can understand the proposed method by simply looking at figure 1.


Weaknesses 
-------------

- Reviewer 26Xs: Novelty is somewhat low as the method could be viewed as a minor modification to CACTUs.
- Reviewer 26Xs: There are a few design decisions that are not well justified: the use of all columns for clustering in the pseudo-validation, the effect of marginal distribution masking versus other types of masking, and how the column masking ratio should be chosen.




justification_for_why_not_higher_score: Four knowledgeable reviewers recommended acceptance, and one reviewer recommended rejection.  The authors addressed most of the reviewer's concerns, including answering clarifying questions, and providing additional results, including UMTRA and SES, and tabular regression tasks. 

On balance, the AC recommends accepting the paper due to its contribution to few-shot learning knowledge for tabular data,  which is interesting to the ICLR audience. 
justification_for_why_not_lower_score: Four knowledgeable reviewers recommended acceptance, and one reviewer recommended rejection.  The authors addressed most of the reviewer's concerns, including answering clarifying questions, and providing additional results, including UMTRA and SES, and tabular regression tasks. On balance, the AC recommends accepting the paper due to its contribution to few-shot learning knowledge for tabular data,  which is interesting to the ICLR audience. The main drawback of this paper is that it is limited to tabular data, so it might have a limited influence on few-shot learning for vision or speech, for example.
Decision: Accept: notable-top-25%