Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a meta-learning method that uses natural language to describe features. The natural language feature description can be encoded with pretrained models in the same way for different tasks, thus enabling the transferability of these features across tasks. The author conducted experiments using some statistical datasets that have categorical features and numerical labels and split these datasets into subsets for meta training, meta validation, and meta testing. Experiments show that the proposed method achieve faster adaptation to a new task with a few task-specific examples. 
strengths_and_weaknesses: Strengths:
1. The idea of describing features in natural language in the meta-learning setting looks novel and reasonable because such natural language features can be easily sharable.
2. The problem formulation and method description are very clear.

Weaknesses:
1. Despite the idea being general, the evaluation setting looks very restricted. The datasets are not commonly used in the existing meta-learning literature as far as I know. I would suggest the author justify why they chose these datasets, and test their method more broadly on other datasets too.
2. It's not clear what the tasks are in their datasets and how related they are. How different are the features of different tasks? The author should give some examples for readers to better understand the difficulty of the generalization across these tasks.
3. From the results in Table 4, it seems most of the improvement is from using pretrained models (comparing NN v.s. ours), which makes me question the contributions of other parts of the proposed method. 
4. Regarding technical details, the author averages the embeddings of different features (equation 3), which raises a concern about how representative the resulted vector can be if the number of features increases. In this work, it seems not a big issue because the number of features for each task is very small (4-5), but I question this if the proposed method is used for many real machine learning problems which usually have thousands of features.
questions: See the weaknesses above.
limitations: Yes, the author discussed the limitations quite well.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a meta-learning method that builds relationships across supervised learning tasks with different feature spaces using feature descriptions written in natural language. When different tasks may share similar or related feature descriptions, benefiting from the pre-trained language model (e.g., BERT), which contains different kinds of knowledge implicitly, the proposed method could improve the generalization performance on unseen tasks with small, labeled data. The authors also empirically demonstrate that the proposed method outperforms the existing meta-learning methods in real-world datasets.
strengths_and_weaknesses: Strengths:
1. The novel problem setting is reasonable and challenging, where instances have the same feature space in the same task and different feature description sets and feature spaces among different tasks.

2. Extensive experiments and ablation studies are provided to determine the effectiveness of the proposed method. 

3. The paper is well-written and easy to follow.

Weaknesses:
1. Despite the plausible setting, there are still a few points that the authors fail to explain elaborately. For example, whether the feature description sets or feature type sets exist intersection between meta-training datasets and meta-test datasets. The question is out of my curiosity about whether the meta-training model using feature descriptions could really generalize to those unseen tasks which contain entirely different feature description sets or even feature types from the meta-training tasks.

2. As the authors mentioned in the Limitation section, the proposed method exits several limitations. First, the method seems to depend on the number of meta-training datasets. Second, in this work, it is to be doubted whether the method could extend to those datasets, where instances contain numerical features or the corresponding labels are discrete. It could be better to find a dataset conforming to the characteristics described above to evaluate the effectiveness of the method.

3. A baseline that neural network with feature description is not meta-trained could be considered, which could evaluate the necessity of meta-training despite using the feature description.
questions: Whether the feature description sets or feature type sets exist intersection between meta-training datasets and meta-test datasets？
limitations: The  authors have addressed the limitations.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This work proposes to use the BERT representation of the textual description of the feature value as input feature to improve the model generalization, particularly in the meta-learning setup. Empirically, compared with baseline meta-learning methods on 2 datasets, the proposed method achieves better or competitive performances. 
strengths_and_weaknesses: The idea of the work is very interesting and intuitive. The paper is well written and easy to understand. The empirical evaluation shows some promising results. 

However, the empirical evaluation is done only on two relatively small datasets, which makes it difficult to judge how general the proposed method is in practice. It would be much more informative to try this method on large datasets.

questions: - For categorical features that do not come with descriptions, is there any systematic way to generate the feature description?
- Since the feature description is manually written, will the model quality be sensitive to the choice of the description?
limitations: The work only tests the proposed method in two relatively small datasets. Hence, it's hard to judge how general the conclusions are.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper considers the problem of heterogenous meta learning, where the different datasets potentially have different number and type of features.
To enable using the same model for all the datasets, the authors consider an approach where they use textual feature descriptions, which makes the model permutation invariant with respect to the features and also agnostic to the number of features.
This allows the same model to be used for all tasks, regardless of their composition of features, because new features can simply be encoded using a sentence encoder.
They show that they outperform other approaches on two datasets.
strengths_and_weaknesses: **Strengths**
1. The approach is very sound and in line with other recent approaches which propose to use descriptions of classes, features, and tasks.
1. The method was compared with a lot of other baselines which helps put the scores in context.
1. The gains seems to be consistent and strong on both meta-tasks considered.


**Weaknesses**
1. It looks like the only difference between the proposed method and a baseline (`MDK + B`) is the usage of the feature encoder (`Fig 1`), which is a 3 layer neural network. It looks like the authors agree with that as well (line 220). So the technical novelty (although guided by good intuition) seems to boil down to the addition of a single layer on top of a baseline.
1. I think it is important to experiment with different types of sentence encoders. Given that the descriptions they consider are very short, it's possible that simple word2vec vectors (bag-of-words averaging) could do the trick.
1. Another baseline could have been added, which fine-tunes the `NN` model on the downstream task.
questions: 1. Is `Ours + T` the same as `Ours + F` in Table 4?
1. How do you think the model would fare if you directly used the instance embeddings (`z`) to make the prediction through a function approximator which is fine-tuned on the support set rather using a GP. In other words, what if you did few-shot fine-tuning on the support set? I believe this would be equivalent to `NN` with few-shot fine-tuning.
limitations: Has been addressed in the supplementary PDF.
review_title: null

Meta-review: metareview: This paper presents a novel meta-learning approach based on learning a sentence encoder which maps feature descriptions to embeddings. The sentence encoder is shown to generalize to new tasks during the test phase, hence allowing few-shot learning. The main concern raised by the reviewers was about the use of only two datasets which are non-standard for evaluation meta-learning. However, as the authors note, the proposed approach requires using datasets where feature descriptions are available and hence the choice of datasets seems reasonable. The authors are encouraged to revise the paper to discuss how the approach might be generalized other setups in meta-learning.
confidence: Less certain
Decision: Accept