Sharing Knowledge for Meta-learning with Feature Descriptions
Authors: Tomoharu Iwata
Atsutoshi Kumagai
Venue: NeurIPS 2022 Accept (2022)
Keywords: Meta-learning
Few-shot learning

Abstract: Language is an important tool for humans to share knowledge. We propose a meta-learning method that shares knowledge across supervised learning tasks using feature descriptions written in natural language, which have not been used in the existing meta-learning methods. The proposed method improves the predictive performance on unseen tasks with a limited number of labeled data by meta-learning from various tasks. With the feature descriptions, we can find relationships across tasks even when their feature spaces are different. The feature descriptions are encoded using a language model pretrained with a large corpus, which enables us to incorporate human knowledge stored in the corpus into meta-learning. In our experiments, we demonstrate that the proposed method achieves better predictive performance than the existing meta-learning methods using a wide variety of real-world datasets provided by the statistical office of the EU and Japan.

Sharing Knowledge for Meta-learning with Feature Descriptions
Tomoharu Iwata
Atsutoshi Kumagai
NTT Communication Science Laboratories
NTT Computer and Data Science Laboratories
Kyoto, Japan
Tokyo, Japan
tomoharu. iiata.ata.. gychco wnt..ttttoo.oo.o jo
atsutoshi kumaggi ht@hho..................c .nt.tttt........................ co.
Abstract
Language is an important tool for humans to share knowledge. We propose a meta-learning method that shares knowledge across supervised learning tasks using feature descriptions written in natural language, which have not been used in the existing meta-learning methods. The proposed method improves the predictive performance unseen tasks with limited number of labeled data by meta- learning from various tasks. With the feature descriptions, we can find relationships across tasks even when their feature spaces are different. The feature descriptions are encoded using language model pretrained with a large corpus, which enables to incorporate human knowledge stored in the corpus into meta-learning. In our experiments, we demonstrate that the proposed method achieves better predictive performance than the existing meta-learning methods using wide variety of real-world datasets provided by the statistical office the EU and Japan.
Introduction
Humans can learn new tasks using descriptions written in natural language, which support us in improving the performance on the new tasks especially when have little experience since the descriptions help understand how experience on familiar tasks can be used for the new tasks. For example. Table T shows dataset from Eurostat, which is the statistical office of the European Union. Some people can guess the production value at basic price in Denmark on soft wheat and spelt in 2017 given Table using knowledge about countries, cereals, and agriculture. On the other hand, supervised learning methods using data shown in cannoot guess the value since some features, i.e., 'Denmark' and 'soft wheat and spelt', do not appear in the training data. Meta-learning methods have been proposed for sharing knowledge across supervised learning tasks to improve the performance on unseen tasks with small amount of training data. However, the existing meta- learning methods do not use feature descriptions written natural language although many real-world datasets contain feature descriptions as shown in Table
this paper, we propose a meta-leariing method that learns from datasets with feature descriptions to improve the generalization performance unseen tasks with small labeled data. With feature descriptions, we can share knowledge between different tasks with similar deseriptions even when their feature spaces are different. We transform the feature descriptions to sentence embeddings by a sentence encoder, e.g., Bidirectional Encoder Representations from Transformers (BERT) [3]. pretrained with large-scale corpus, by which can exploit huge amount of human knowledge accumulated in the corpus for meta-learning. The sentence embeddings reflect the meaning of the descriptions but they a feature encoder network to improve the generalization performance.
necessarily suitable for supervised learning tasks. We translate them by
We embed instances in task-specific space by instance encoder network using their feature values and descriptions translated into sentence embeddings. By adapting a Gaussian process (GP) [7 to
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Table 1: Example of a dataset with feature descriptions in Eurostat on economic accounts for agriculture. this paper, feature descriptions denote explanations about features for each instance, e.g., 'Production value at basic price' and "Wheat and spelt', and feature rype descriptions denote explanations about types of the features shown at the top row, e.g.. 'Agricultural indicator and 'List of products'. The 'Label' column comprises labels to be predicted.
Agricultural indicator
List of products
Unit of measure
Geo
Time
Label
Production value at basic price
Wheat and spelt
Million EURO
Belgium
2016
189.66
Production value at basic price
Wheat and spelt Barley
Million EURO
Belgium
2017
237.56
Production value at basic price
Million EURO
Italy
2016
158,82
Subsidies on products
Cereals (including seeds)
Million EURO
France
2016
6.19
Tax on products
Industrial crops
Million EURO
Germany
2016
17.00
small labeled data in the space, we obtain a task-specific prediction model. The neural networks for feature encoding, instance encoding, and prediction shared among all tasks, which enables to learn and share knowledge that is useful in different tasks. The parameters of the neural networks are trained by minimizing the expected (est error of the task-adapted prediction model using meta-training datasets based on a bilevel optimization framework. The meta-training datasets consist of labeled data from wide variety of tasks with feature descriptions.
The main contributions of this paper are as follows: 1) We newly present a meta-learning problem where feature descriptions are given for each task. 2) We propose simple yet effective neural network- based meta-learning method that improves the generalization performance with small labeled data using feature descriptions by learning from various datasets. 3) We empirically demonstrate that the proposed method outperforms the existing methods using a wide variety of real-world datasets provided by the statistical office of the EU and Japan.
Related work
Many meta-learning methods have been proposed 19 26 20). However, they usually assume that only pairs of feature and label values are given for each task, and do not use task information. Several existing meta-learning methods use task information, such as class descriptions and task descriptions. Class descriptions, e.g., image category name in natural language, have been used for few-shot (30 and zero-shot learning 5 21]. Task descriptions, e-g-, the brightness of images in the task, have been used for active meta-learning [9]. few-shot learning [3]. zero-shot domain adaptation [11. and natural language processing systems (8). However, they cannot use feature descriptions, Feature descriptions are helpful for understanding relationships between tasks especially when feature spaces are different across tasks. Most existing meta-learning methods assume feature spaces are the same for all tasks although tasks with different feature spaces can contain useful knowledge be shared. Heterogeneous meta-learning can handle tasks with different feature spaces [7]. However, it cannot use the information about features. Text information has been used in some applications with supervised learning settings, such as cross-lingual text classification [4), recommendation (27), and image classification [6]. However, they are inapplicable to meta- meta-learning. We use text information for sharing knowledge to improve performance on unseen tasks.
3 Proposed method
3.1 Problem formulation
In the meta-training phase, we are given D labeled meta-training datasets with feature descriptions, D = ([(x,n ydu Waa Wa)d where Xán = (fdnl)..., Ca Ja 1- is the set of feature descriptions, Na is
is the feature vector of the rith
instance, 3dn E IR is its label, Wa [[Wajele<)) the number of instances, Zdnj {l,.... Caj} is the categorical value of the jth feature, Caj the number of categories, Ja is the number of features, and Waje is the feature description of the cth category of the jth feature. For example, W.11 Prroduction value at basic price', Wa12 = -Subsidies on products', and w #221" = 'Wheat and spelt' are feature descriptions of the dataset in Table The feature description sets Wa ds the number of features Ja, and the number of categories Caj are different across tasks. In the meta-test phase, we are given a support set, which consists of a small number of
2

---

1)
W11
fsB
FEE
fis
Instance encoder
Instance
Large-scale
Vi
41
embedding
corpus
Feature descriptions
Sentence encoder
Sentence embedding
Feature values
Feature encoder
Feature embedding
Figure 1: Our encoder. 1) Feature descriptions {{ ((Wjel==1) are transformed to sentence embed-
ding vectors {(vialam ) using sentence encoder fsE pretrained with a large-scale corpus. 2) Feature values {r, } are transformed to feature embedding vectors {u,} by feature encoder fFe using the sentence embedding vectors. 3) Instance embedding vector z is obtained by encoding the average of the feature embedding vectors with instance encoder fIg. Shaded and unshaded circles represent observed and hidden variables, respectively.
labeled examples D {xx,, ) N I==1 with description W in an unseen task, which are different from the meta-training datasets. Our aim is improve the test label prediction performance test tasks. Although we explain the proposed method for categorical features and numerical labels for simplicity, can also use numerical features and categorical labels as described in Section 3.4
3.2
Model
With our model, instances with feature descriptions are embedded for each task as described in Section 3.2.I. and their labels are predicted in the task-specific embedding space as described in Section 3.2.
3.2.1 Instance embedding with feature descriptions
We embed each instance shown in Figure [ where æy E {1, , Cý} } is the jth feature value, / is the number of features, C is the number of categories of the jth feature. and wee is the feature description of the cth categorical value at the jth feature. Here, we omit the task and instance indices.
(x ...,, I.J) using feature descriptions W [(W as
First, each feature description Wje is transformed into sentence embedding vicc E M, by a pretrained sentence encoder, e.g., BERT,
jc SSE (wrs(wcc Wj
(1)
where fSE is the sentence encoder. With BERT pretrained using a huge number of text documents, we can incorporate knowledge the documents.
Second, feature embedding uy E RM, of the jth feature value is obtained using the sentence embedding and feature value,
u, fFE((Vzz),
where fFE is trainable feature encoder, such as feed-forwe forward neural network and transformer [25 By the feature encoder, we translate sentence embeddings, which are calculated based on the pretraining documents written for humans, to feature embeddings that are suitable for supervised learning tasks.
Third, instance emhedding z E R.M is obtained by averaging the feature embeddings,
J Н j=1
1
z f
f(x; W),
(3)
where fir a trainable instance encoder based on neural networks, and an encoder to output instance embedding given feature vector x and feature description W. Eq. (3) is permutation

---

Feature descriptions'
W
Instance Embedding
1)
2)
Support instances
Support labels
GP
f
Query
Task-adapted
Predicted
instance
x
2
GP
y
n=1 and query query label
Encoder
Mean and kernel functions
3)
Figure 2: Our task-specific prediction model. 1) Feature vectors of support {
instances are transformed instance embeddings {z, y and z by encoder f in Eq. (3 with feature descriptions W. 2) Using support instance embeddings {2,,)re) and their labels {v,) N the GP with mean function g and kernel function k is adapted.
Predicted query label ÿ obtained with the task-adapted GP and its instance embedding z. Shaded and unshaded circles represent observed and hidden variables, respectively. Encoder and mean g and kernel & functions are shared across tasks.
invariant neural network [32 since the summation is invariant even when the elements permuted, and our model can handle instances with any number of features. Neural networks, /FE and fIE, are shared across all tasks, by which we can store knowledge meta-learned in various tasks in the neural networks, and use it for unseen tasks. Although size J of input feature vector x can be different across tasks, size M, of instance embedding vector z the same for all tasks.
3.2.2 Task-specific prediction
The proposed model predicts label y of query instance x using support D = (xx, vn 1
n=t and
feature description W of the task, where x is the feature vector of the nth support instance, is its label, and N is the number of support instances. Here, we omit the task index. Figure 2 illustrates our model.
First, instance embeddings are obtained by f in Eq. for the query and support instances, f(x, W), and Nh = fx W). n = I,...,,N NS.
Second, the query label is predicted by Gaussian process regression with kernel function & and with neural network- mean function g using the instance embeddings. The predicted label adapted to the support set is given in closed form,
Dx W. () = 9k -g",
-g),
(4)
where k [k(22, 2i.,... k(z, [Ns) E I is the kernel vector between the query and sup- port instances, K E NN88 is the kernel matrix between the support instances, Knn' = k(Z), E R. E, g.
R is the label vector of the support instances, and E RR is the mean vector evaluated on the support instances. θ is parameters in model, which consists of neural network parameters in fFE, fIF, and g, and kemel parameters in h: Neural network y and kernel parameters in k are shared across all tasks. GP regression is nonparametrie nonlinear regression method successfully used in wide variety of applications, and it can output prediction adapted to the support set flexibly. Since the task-adapted GP prediction is given in a closed form shown in Eq. through the adaptation, and it has been used for meta- meta-learning [6, The prediction in Eq. (4) can be seen as fitting GP with a zero mean function to residual yd - g of prediction by neural network Neural network makes predictions based on the feature descriptions without using the support set. Even when the feature descriptions are not helpful for the prediction, the proposed model makes reasonable predictions by adapting them to the support set by GP. When labels can be predicted well by neural network y with the feature descriptions, since the residual becomes zero, the adaptation skipped. by which avoid the risk of overfitting to the small support set.
IuS....,..
we can efficiently perform backpropagation

---

Algorithm Meta-learning procedure of our model.
Input: Meta-training data D, number of support instances N number of query instances NQ
Output: Trained model parameters +
1: Initialize model parameters θ.
2: while End condition is satisfied do
3: Randomly select task index d from (1.... ................................ D}.
4: Randomly sample N instances from the dth task for support instances D
5: Randomly sample Nº instances from the dth task for query instances D& such that there is
no overlap between the support and query instances.
6: Calculate test error N (x.y))DD (y [x; DS, W, x))
7: Update model parameters θ using the gradient of the loss by stochastic gradient method.
8: end while
3.3
Meta-training
We train model parameters θ by minimizing the expected test mean squared error using an episodic training framework (18).
= arg min ES D9,W
Σ
Úx; W)))
(5)
NQ
(X
where ED DR, w represents the expectation and feature descriptions W. By using the query set, which is different from the support set used for adapting GP, we can evaluate the test error. Minimizing our meta-learning loss Eq. (5) is bilevel optimization problem. In the inner optimization, a task-specific prediction model is adapted by minimizing the loss on the support set based on GP as described in Section 3.2.2] In the outer optimization, neural network parameters shared among all tasks are estimated by minimizing the error on the query set when the task-adapted prediction model obtained in the inner optimization is used. Algorithm [ shows the meta-training procedure of the proposed model. The expectation in Eq. (5) is approximated by the Monte Carlo method, where a task, support and query instances are randomly sampled from the given meta-training datasets.
support D query set D (xx,, yn )
Jn=l
The computational complexity of calculating the test error for of support and query instances in Line 6 O(VQ + _v**) since we need the inverse of the kernel matrix whose size is S S In few-shot settings, which is our focus, the number of support instances N is small.
3.4 Variants
3.4.1 Feature type description
In addition feature descriptions (e.g., "Production value at basic price" and "Wheat and spelt" in
Table feature type descriptions (e.g.. "Agricultural indicator" and "List of products" in Table I are available some datasets. Let Wj be the feature type description of the jth feature. In this case, the feature embedding is obtained by taking the sentence embeddings of the feature description and feature type description as input of the feature encoder,
I
SSe( (se(w))),
(6)
instead of Eq. 2 This enables us to incorporate information of feature type descriptions for meta-learning.
3.4.2 Missing feature descriptions
Descriptions might be missing for part of features in some datasets. Let be features with descriptions, and x be those without descriptions in x. Instance embedding z is obtained using features with descriptions by encoder / in Eq. (3). z = fxx), by which we can incorporate observed descriptions in the embedding. The task-specific GP is adapted based on the concatenation of the instance embedding and onehot vector of features without deseriptions. particular, the kernel function is calculated by, A[zz, onehot(x)), (z,, onehot(x;))), where onehot represents the onehot
5

---

encoder, and [:-1] represents the concatenation of vectors. The prediction is performed by Eq. 44) using the kernel function with the concatenation, by which we can use the information on both of the features with and without descriptions for prediction, where the mean function takes instance embedding z as input.
3.4.3 Numerical feature values
When features numerical values, obtain the feature embedding by multiplying numerical value Ij and the sentence embedding of its feature type description w, as follows,
u, = fFe(fsE(w,)o)
(7)
3.4.4 Classification
In classification tasks, we can use the GP-based adaptation in Eq. 4 by representing labels by binary vectors as shown in [16], or we can use the Gaussian mixture model-based adaptation as in prototypical networks [20]. For the loss function in Eq. ( we can use the cross-entropy loss instead of the mean squared error.
4 Experiments
4.1 Data
For evaluating the proposed method, used the following data: e-Stat and Eurostat. These data contained statistical tables in various fields, including economy, health, education, sport, and crime. The e-Stat data were obtained from the official statistics of Japan using API The Eurostat data were obtained from the statistical office of the European Union using API A statistical table contained numerical values of statistics, which were used for labels, and categorical descriptions in natural language, which were used for features, as shown in Table We used a statistical table as a dataset task. When a statistical table comprised multiple units measurement, it was decomposed into multiple datasets based on the unit of measurement. We used datasets where the number of instances was between 50 and 1,000. The labels were normalized with zero mean and one standard deviation for each dataset. The statistics of the two data are shown in Table The descriptions were in Japanese in e-Stat, and in English in Eurostat. For e-Stat, we used the Japanese pretrained BERT For Eurostat, we used the English pretrained BERT 3 29 The BERT vector size was 768. For each of the e-Stat and Eurostat data, we sampled 700 datasets for meta-training, 100 for neta-validation, and 200 for meta-test, For each task, the number of support instances was 5, 10, or 20, and the number of query instances was 20. We evaluated the test mean squared error averaged over ten experiments with different splits of datasets. Note that did not use commonly-use meta-learning datasets, such as Omniglot [TT, Mini-imagenet [26), and datasets in Torchmeta 2 and they do not contain feature descriptions.
[3]] since
4.2 Comparing methods
We compared the proposed method (Ours) with the following methods: ridge regression (Ridge), Gaussian process regression (GP), heterogeneous meta-learning 7 (HML), neural networks with BERT sentence embeddings (NN), meta deep kernel learning [16] with shared categorical onehot vector features (MDK+C), meta deep kernel learning with word frequency vectors (MDK+W), meta deep kernel learning with BERT sentence embeddings (MDK+B), the proposed method without mean functions (Ours-M) and the proposed method with feature type descriptions (Ours+T). Table summarizes the comparing methods.
Ridge and GP methods were trained using support instances for each task. The hyperparameters, ie., the regularization weight in Ridge and the kernel parameters in GP, were optimized using the meta-training training datasets. We used RBF kernels for GP. The methods except for Ridge and GP had
httpe://ww.eestat.gg..ppp

---

Table 2: Statistics of e-Stat and Eurostat data.
e-Stat
Eurostat
Total number of tasks
5,488
2,103
Total number of instances
1,379.346
946,959
Average number of instances per task Average number of features per task Minimum number of features per task
251.3
450.3
5.4
4.5
3
2
Maximum number of features per task Average number of characters per feature description Total number of unique feature descriptions Number of feature descriptions that appear only in one task
10
4.9
16.1
8,059
11,388
3,881
2,278
Average number of tasks per feature description
12.6
18.8
Table 3: Comparing methods. "Meta-learning" represents that the method has neural networks shared among tasks, which are meta-traines using the meta-training datasets. Tasskadaptttioon represents that the method adapts its prediction to the given support set by minimizing the loss. 'Feature correspondence" represents that the method uses the correspondence features across tasks. 'Feature description' represents that the method uses feature descriptions in natural language. "Translation" represents that the method translates the BERT sentence embedding using a feature encoder for each description. 'Pretrained BERT" represents that the method uses the pretrained BERT. 'Mean function' represents that the method uses the mean function for task-specific GP adaptation. 'Fettree type description' represents that the method uses feature type descriptions.
Ridge GP HMI NN MDK+C MDK+W MDK+B Ours-M Ours Ours+T
Meta-learning Task adaptation
Feature correspondence
Feature description
V
Pretrained BERT
V
V
4
Translation Mcan function
V
V
V V
v
v
V
Feature type description
neural networks shared among all tasks, and they were meta-trained using the meta-training datasets. HML was a meta-learning method for tasks with different feature spaces, which did not use feature information. Note that since other existing meta-learning methods cannot handle tasks with different feature spaces, did not compare with them. In Ridge, GP, and HML, categorical feature values were transformed into onehot vectors for each task. NN was a neural network-based meta-learning method, where feature descriptions were used for its input. With NN, feature descriptions were embedded by BERT, translated by a feature encoder network, and taken to deep sets [32 as input for predicting labels. NN corresponds to the proposed method without GP adaptation to the support set for each task. In MDK+C, MDK+W and MDK+B, a task-specific GP was adapted using the support set, where RBF kernels were used with neural networks for input transformation. MDK+C used onehot vectors of categorical feature values that were shared in all tasks, MDK+C corresponds to the proposed method that uses feature descriptions for only identifying feature correspondences across different tasks. MDK+W used word frequency vectors for representing feature descriptions, where the vectors were embedded in 768-dimensional space by principal component analysis. MDK+W corresponds to the proposed method with word frequency vectors without pretrained BERT, MDK+B used BERT sentence embeddings, but they were not translated by feature encoder for each description in meta-learmin meta-learning, MDK+B corresponds to the proposed method without feature encoders. Since the feature type descriptions were unavailable by Eurostat API, we did evaluate Ours+T on Eurostat data.
4.3 Settings
In the proposed model, we used three-layered feed-forward neural network with 128 hidden and output units for the sentence, feature, and instance encoders, fSR, fFF, and fIF, and a three-layered feed-forward neural network with 128 hidden units and single output unit for mean function g. For the activation function, we used rectified linear unit ReLU(x) max(0, æ). For GP, we used RBF
7

---

Table 4: Average test mean squared error and its standard error with different numbers of support instances per task. Values bold typeface are not statistically significantly different the 5% level from the best performing method in each row according to paired t-test.
(a) e-Stat
#Support
10
20
Ridge
0.991 = 0.015
0.889
0,019
0,663 = 0.017
GP
0.921 0.016
0.807
0.018
0.626 = 0.016
HML
0.860 0.017
0.857 0.039
0.652 = 0.077
NN
0.462 0.018
0.462
0.018
0.462 = 0.018
MDK+C
0.607 0.017
0.583
0.013
0.555 = 0.008
MDK+W
0.470 0.016
0.413
0.022
0.331 + 0.009
MDK+B
0.444 0.013
0.406
0.013
0.350 + 0.007
Ours-M
0.658 + 0.025
0.490
0.020
0.366 + 0.011
Ours
0.448 + 0.014
0.383
0.013
0.320 + 0.010
Ours+T
0.417 = 0.011
0.375 0.012
0.333 = 0.015
(b) Eurostat
#Support
10
20
Ridge
0.969 + 0.021
0.925
0.015
0.790 = 0.017
GP
0.902 0.018
0.844 0.016
0.749 = 0.018
HML
0.961 0.018
0.934 = 0.029
0.963 = 0.025
NN
0.896 0.014
0.896 0.014
0.896 = 0.014
MDK+C
0.885 + 0.022
0.851
0.019
0.731 + 0.013
MDK+W
0.920 0.017
0.843
0.022
0.701 + 0.015
MDK
0.858 + 0.016
0.762
0.015
0.671 + 0.017
Ours-M
0.867 0.029
0.781
0.019
0.634 + 0.023
Ours
0.756 + 0.021
0.690
0.019
0.595 + 0.019
kernels, k(z, z²)
exp (-3 z z' + where a, 3, and were kernel parameters
to be meta- meta-trained. We optimized our models using Adam [10] with learning rate 10-3, batch dataset size 32, and dropout rate 0.1 [22]. The meta-validation datasets were used for early stopping, for which the maximum number of meta- mneta-training epochs was 5,000. In the comparing methods, we used the same neural network architectures and training procedures as the proposed method. We
implemented the proposed method with PyTorch [15]
4.4 Results
Table the test mean squared error with different numbers of support instances per task. The proposed method achieved the best performance in all cases. The performance the Ridge and GP methods was low since they were trained using only support instances in the meta-test datasets. The HML method did not perform well since it needs to meta-learn relationships among tasks using support instances without feature descriptions. This result indicates the effectiveness of the feature descriptions for meta-learning. The NN method predicts labels using only feature descriptions, where the prediction is not adapted the labeled support instances. Since the feature descriptions in natural language are often not quantitative, is difficult for the NN method to predict labels accurately. On the other hand, the proposed method achieved better performance than the NN method by minimizing the prediction error on the support instances. The MDK+C method uses feature descriptions for corresponding features among tasks, but does not use sentence embeddings. The MDK+C method performed worse than the proposed method, which implies sentence embeddings are important for meta-learning since they can quantify the relationships among features even when their descriptions are not the same. The MDK+W method underperformed the proposed method, which implies that incorporating knowledge effective for meta- meta-learning The error by the MDK+B method was higher than the proposed method, which indicates the effectiveness of the translation using the feature encoder for meta-learning, The proposed method without mean functions (Ours-M) performed worse than the NN method e-Stat data with five and ten support instances. is because feature descriptions are relatively important for task task-specific prediction when the number of support instances is small. With the proposed method
large-scale corpus stored by pretrained BERT is

---

0.9
0.8-
Ridge
0.9
0.7
GP
Emor
0.6
0.8
Ridge
0.5
0.7
200
600
200
600
#meta-train datasets
#meta-train datasets
(a) e-Stat
(b) Eurostat
Figure 3: Average test mean squared errors with different numbers of meta-training datasets. The bar shows the standard error.
0.95
0.8
0.90
0.85
0.80
0.5
0.75
0.4
0.70
0.2
Observed descriation
feature descriation
(a) e-Stat
(b) Eurostat
Figure 4: Average test mean squared errors with different rates of observed feature descriptions. The bar shows the standard error.
Table 5: Average mean squared error and its standard error by the proposed method with word2vec instead of BERT.
#Support
5
10
20
e-Stat
0.500 + 0.016
0.442
0.012
0.390 + 0.013
Eurostat
0.865 + 0.021
0.800 = 0.016
0.680 0.017
function is adapted by Gaussian processes using support instances for each task, which enables us to achieve better performance for all the cases. With the proposed method, the use of feature type descriptions (Ours+T) improved the performance when the number of support instances was five. The feature type descriptions can be helpful when support instances are small.
(Ours), the residual error of the neural network-basee
Figure 3 shows the test mean squared error with different numbers of meta-tras meta-training datasets. As the number of the meta-traing meta-training datasets increased, the performance of the proposed method increased. The proposed method achieved the best performance in all cases except for 100 meta-training datasets in Eurostat data. It is effective for the proposed method shows the test mean squared error with different rates of observed feature descriptions by the proposed method. The error decreased the observed rate increased especially in e-Stat data.
meta-training datasets. Figure 4
Table shows the test mean squared error by the proposed method using word2vee [12] for sentence embeddings instead of BERT. We used Japanese word2vec for e-Stat, and English word2vec for Eurostat, and the sentence embedding was obtained by the average of the word embeddings in the description. The error with word2vec higher than that with BERT (Ours in Table This result indicates that the meta-learning performance depends on the performance the sentence encoder.
Table shows the test mean squared error on tasks where more than half of the feature descriptions do not appear in the meta-training tasks. The proposed method also achieved lower errors in tasks that contain feature descriptions that do not appear in the meta-training data than the other methods.

---

Table 6: Average test mean squared errors on tasks where more than half of the feature descriptions do not appear in the meta-training tasks with five support instances per task.
Ridge
GP HML NN MDK+C MDK+W MDK+B Ours-M
Ours Ours+T
e-Stat
0.968 0.892 0.836 0.613
0.665
0.747
0.685
0.830
0.607
0.568
Eurostat 1.019
0.942 0.993 1.013
0.987
0.993
1.023
0.951
0.866
(a) Onchot
(b) BERT
(c) Ours
Figure 5: Visualization by t-SNE [24] of instances in task on e-Stat data. Each point is an instance, and color represents the label. Each instance is represented by (a) a onehot vector of categorical feature values, (b) a concatenation of BERT sentence embeddings of feature descriptions, and (c) an instance embedding with the proposed method.
Table 7: Meta-training time in seconds on e-Stat data with ten support instances per task.
Ridge GP HML
NN MDK+C MDK+W MDK+B Ours-M Ours Ours+T
3113.1 2695.5 92664.6 13642.3 21134.2 14469.3 11775.5 14661.8 16841.4 24483.3
the visualization instances. With the onehot vectors of categorical feature values (a), is difficult to represent the instances well. With the BERT sentence embeddings (b), instances with similar labels formed clusters, but the clusters were separated. With the proposed method (c), clusters with similar labels were located more closely than those with the BERT sentence embeddings by the feature and instance encoders, which helps label prediction in the instance embedding space.
With translation by the feature encoder, similar BERT sentence embeddings can be encoded to different feature embeddings for improving the meta-learning performance. For example, in Eurostat data, BERT embeddings of "Accommodation (10 or employees and self-employed persons)' and "Information and communication (10 or more employees and self-employed persons)' were similar (cosine=0.943) since the words in the parenthesis were the same, while their translated feature embeddings by the proposed method were different (cosine=0.003) BERT embeddings of 'Outwards' and 'Inwards' were similar (cosin=0.951), while their feature embeddings were different (cosine=0.025), since they were related but antonyms. BERT embeddings of 'Warmond Hoofddorp' and "Amersfoort- Hilversum' were different (cosine=0.288 280), while their feature embeddings were similar (cosine=0.901) since they are towns in the Netherland and related.
Table shows the average computation time in seconds for meta-training computers with 2.60GHz CPUs. The Ridge and GP methods were efficient since they did not have neural networks to be meta-trained. The computational time of the proposed method was shorter than that of the HML method.
Conclusion
We proposed a meta-learning method that effectively uses feature descriptions, and confirmed that the proposed method achieves better performance than the existing methods. Although our results are encouraging as an effective approach for sharing knowledge between various tasks meta-learning by incorporating human knowledge, there are some limitations to overcome. First, we need to empirically evaluate the variants of the proposed method, e.g., for classification tasks and numerical feature values. Second, we would like to use other types of neural networks for components in our model, such attention networks experiments.
25 since we used feed-forward neural networks our
10

---

References
[1] Y. Bengio, S. Bengio, and J. Cloutier. Learning a synaptic learning rule. In International Joint Conference Neural Networks, 1991.
[2] T. Deleu, T. Würfl, M. Samiei, J. P. Cohen, and Y. Bengio. Torchmeta: A meta-learning library for Pytorch. arXiv preprint
2019.
[3] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprini 0rXiv:/810.000 04805, 2018.
[4] C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning, pages 1126-1135, 2017.
[5] A. Frome, G. Corrado, J. Shlens, S. Bengio, J. Dean, M. Ranzato, and Mikolov. DeViSE:
A deep visual-semante embedding model. Advances in Neural Information Processing Systems, 26, 2013.
[6] X. He and Y. Peng. Fine-grained image classification via combining vision and language. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5994-6002, 2017.
[7] T. Iwata and A. Kumagai. Meta-learning from tasks with heterogeneous attribute spaces. Advances in Neural Information Processing Systems, 33:6053-6000 2020.
[8] T. Iwata and Y. Tanaka. Few-shot learning for spatial regression via neural embedding based Gaussian processes. Machine Learning, pages 1-19, 2021.
[9] J. Kaddour, S. Sæmundsson, et al. Probabilistic active meta-learning. Advances in Neural Information Processing Systems, 33:20813 20-222 2020.
[10] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
[11] B. M. Lake, R. Salakhutdinov, and J. B. Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 350(6266):1332-1333 2015.
[12] Mikolov, K. Chen, G. Corrado, and J. Dean. Efficient estimation of word representations in vector space. arXiv preprint
2013.
[13] J. Mu, P. Liang, and N. Goodman. Shaping visual representations with language for few-shot classification. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, pages 4823-4830, 2020.
[14] X. Ni, J.-T. Sun, J. Hu, and Z. Chen. Cross lingual text classification by mining multilingual topics from wikipedia. In Proceedings of the fourth ACM International Conference on Web Search and Data Mining, pages 375-384, 2011.
[15] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L.. Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems, 32, 2019.
[16] M. Patacchiola, J. Turner, E. J. Crowley, M. O'Boyle, and A. J. Storkey. Bayesian meta-learning for the few-shot setting via deep kernels. Advances in Neural Information Processing Systems, 33:16108-16118, 2020.
[17] C. Rasmussen and C. K. I. Williams. Gaussian Processes for Machine Learning. The MIT Press, 2005.
[18] S. Ravi and H. Larochelle, Optimization as model for few-shot learning. In International Conference on Learning Representations, 2017.
[19] Schmidhuber. Evolutionary principles in self-referential learning. on learning now to learn: Master's thesis, Technische Universitat Munchen, Germany, 1987.
The
11

---

[20] J. Snell, K. Swersky, and R. Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems, pages 4077-4087, 2017.
[21] R. Socher, M. Ganjoo, C. D. Manning, and A. Ng. Zero-shot learning through cross-modal transfer. Advances in Neural Information Processing Systems, 26, 2013.
[22] N. Srivastava, G. Hinton, A. Krizhevsky, Sutskever, and R. Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting, Journal of Machine Learning Research, 15())1192--1958 2014.
[23] E. Triantafillou, T. Zhu, V. Dumoulin, P. Lamblin, U. Evei, K. Xu, R. Goroshin, C. Gelada, K. Swersky, P-A. Manzagol, al. Meta-Dataset: A dataset of datasets for learning learn from few examples. In International Conference on Learning Representations, 2019.
[24] L. Van der Maaten and G. Hinton. Visualizing data using t-SNE. Journal of Machine Learning Research, 9(11), 2008.
[25] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, 30, 2017.
[26] O. Vinyals, C. Blundell, T. Lillicrap. D. Wierstra, et al. Matching networks for one shot learning. In Advances in Neural Information Processing Systems, pages 3630-3638, 2016.
[27] T. Wang and Y. Fu. Workshop on e- e-Commerce and NLP, pages 54-58, 2020.
collaborative filtering with BERT. In Proceedings of The 3rd
[28] O. Weller, N. Lourie, M. Gardner, and M. E. Peters. Learning from task descriptions. arXiv preprint arXiv:2011.0811 081115, 2020.
[29] T. Wolf, Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jermite, J. Plu, C. Xu, T. L. Scao, Gugger, M. Drame, Q. Lhoest, and A. M. Rush. Transformers: State-of- of-the-art natural language processing. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing: System Demonstrations, pages 38 45, 2020.
[30] C. Xing, N. Rostamzadeh, B. Oreshkin, and P. O. O Pinheiro. Adaptive cross-modal few-shot learning. Advances in Neural Information Processing Systems, 32, 2019.
[31] Y. Yang and Hospedales. Zero-shot domain adaptation via kernel regression on the Grass- mannian. arXiv preprint arXiv: 1507.007000080 2015.
[32] M. Zaheer, S. Kottur, S. Ravanbakhsh, B. Poczos, R. R. Salakhutdinov, and A. Smola. Deep
sets. In Advances in Neural Information Processing Systems, pages 3391-3401, 2017.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See the supplemental material.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See the supplemental material.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments...
12

---

(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or URL)? [No] The code is proprietary. The datasets and experimental procedures are described in Section [
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section [
(c) Did you report error bars (c.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasngg new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you deseribe any potential participant risks, with links Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "xxFXC9XmR5B", "reviewer": "Reviewer_CRpu", "summary": "This work proposes to use the BERT representation of the textual description of the feature value as input feature to improve the model generalization, particularly in the meta-learning setup. Empirically, compared with baseline meta-learning methods on 2 datasets, the proposed method achieves better or competitive performances. ", "questions": "- For categorical features that do not come with descriptions, is there any systematic way to generate the feature description?\n- Since the feature description is manually written, will the model quality be sensitive to the choice of the description?", "limitations": "The work only tests the proposed method in two relatively small datasets. Hence, it's hard to judge how general the conclusions are.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The idea of the work is very interesting and intuitive. The paper is well written and easy to understand. The empirical evaluation shows some promising results. \n\nHowever, the empirical evaluation is done only on two relatively small datasets, which makes it difficult to judge how general the proposed method is in practice. It would be much more informative to try this method on large datasets.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "nraSoK-9UW", "reviewer": "Reviewer_UMH4", "summary": "The paper proposes a meta-learning method that builds relationships across supervised learning tasks with different feature spaces using feature descriptions written in natural language. When different tasks may share similar or related feature descriptions, benefiting from the pre-trained language model (e.g., BERT), which contains different kinds of knowledge implicitly, the proposed method could improve the generalization performance on unseen tasks with small, labeled data. The authors also empirically demonstrate that the proposed method outperforms the existing meta-learning methods in real-world datasets.", "questions": "Whether the feature description sets or feature type sets exist intersection between meta-training datasets and meta-test datasets？", "limitations": "The  authors have addressed the limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The novel problem setting is reasonable and challenging, where instances have the same feature space in the same task and different feature description sets and feature spaces among different tasks.\n\n2. Extensive experiments and ablation studies are provided to determine the effectiveness of the proposed method. \n\n3. The paper is well-written and easy to follow.\n\nWeaknesses:\n1. Despite the plausible setting, there are still a few points that the authors fail to explain elaborately. For example, whether the feature description sets or feature type sets exist intersection between meta-training datasets and meta-test datasets. The question is out of my curiosity about whether the meta-training model using feature descriptions could really generalize to those unseen tasks which contain entirely different feature description sets or even feature types from the meta-training tasks.\n\n2. As the authors mentioned in the Limitation section, the proposed method exits several limitations. First, the method seems to depend on the number of meta-training datasets. Second, in this work, it is to be doubted whether the method could extend to those datasets, where instances contain numerical features or the corresponding labels are discrete. It could be better to find a dataset conforming to the characteristics described above to evaluate the effectiveness of the method.\n\n3. A baseline that neural network with feature description is not meta-trained could be considered, which could evaluate the necessity of meta-training despite using the feature description.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "NHc9-6Ss-zW", "reviewer": "Reviewer_D33L", "summary": "This paper proposes a meta-learning method that uses natural language to describe features. The natural language feature description can be encoded with pretrained models in the same way for different tasks, thus enabling the transferability of these features across tasks. The author conducted experiments using some statistical datasets that have categorical features and numerical labels and split these datasets into subsets for meta training, meta validation, and meta testing. Experiments show that the proposed method achieve faster adaptation to a new task with a few task-specific examples. ", "questions": "See the weaknesses above.", "limitations": "Yes, the author discussed the limitations quite well.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The idea of describing features in natural language in the meta-learning setting looks novel and reasonable because such natural language features can be easily sharable.\n2. The problem formulation and method description are very clear.\n\nWeaknesses:\n1. Despite the idea being general, the evaluation setting looks very restricted. The datasets are not commonly used in the existing meta-learning literature as far as I know. I would suggest the author justify why they chose these datasets, and test their method more broadly on other datasets too.\n2. It's not clear what the tasks are in their datasets and how related they are. How different are the features of different tasks? The author should give some examples for readers to better understand the difficulty of the generalization across these tasks.\n3. From the results in Table 4, it seems most of the improvement is from using pretrained models (comparing NN v.s. ours), which makes me question the contributions of other parts of the proposed method. \n4. Regarding technical details, the author averages the embeddings of different features (equation 3), which raises a concern about how representative the resulted vector can be if the number of features increases. In this work, it seems not a big issue because the number of features for each task is very small (4-5), but I question this if the proposed method is used for many real machine learning problems which usually have thousands of features.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "LXGKjsbz1xV", "reviewer": "Reviewer_Ay7F", "summary": "This paper considers the problem of heterogenous meta learning, where the different datasets potentially have different number and type of features.\nTo enable using the same model for all the datasets, the authors consider an approach where they use textual feature descriptions, which makes the model permutation invariant with respect to the features and also agnostic to the number of features.\nThis allows the same model to be used for all tasks, regardless of their composition of features, because new features can simply be encoded using a sentence encoder.\nThey show that they outperform other approaches on two datasets.", "questions": "1. Is `Ours + T` the same as `Ours + F` in Table 4?\n1. How do you think the model would fare if you directly used the instance embeddings (`z`) to make the prediction through a function approximator which is fine-tuned on the support set rather using a GP. In other words, what if you did few-shot fine-tuning on the support set? I believe this would be equivalent to `NN` with few-shot fine-tuning.", "limitations": "Has been addressed in the supplementary PDF.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n1. The approach is very sound and in line with other recent approaches which propose to use descriptions of classes, features, and tasks.\n1. The method was compared with a lot of other baselines which helps put the scores in context.\n1. The gains seems to be consistent and strong on both meta-tasks considered.\n\n\n**Weaknesses**\n1. It looks like the only difference between the proposed method and a baseline (`MDK + B`) is the usage of the feature encoder (`Fig 1`), which is a 3 layer neural network. It looks like the authors agree with that as well (line 220). So the technical novelty (although guided by good intuition) seems to boil down to the addition of a single layer on top of a baseline.\n1. I think it is important to experiment with different types of sentence encoders. Given that the descriptions they consider are very short, it's possible that simple word2vec vectors (bag-of-words averaging) could do the trick.\n1. Another baseline could have been added, which fine-tunes the `NN` model on the downstream task.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept