Distribution Embedding Networks for Generalization from a Diverse Set of Classification Tasks
Authors: Lang Liu
Mahdi Milani Fard
Sen Zhao
Venue: Accepted by TMLR ()

Abstract: We propose Distribution Embedding Networks (DEN) for classification with small data. In the same spirit of meta-learning, DEN learns from a diverse set of training tasks with the goal to generalize to unseen target tasks. Unlike existing approaches which require the inputs of training and target tasks to have the same dimension with possibly similar distributions, DEN allows training and target tasks to live in heterogeneous input spaces. This is especially useful for tabular-data tasks where labeled data from related tasks are scarce. DEN uses a three-block architecture: a covariate transformation block followed by a distribution embedding block and then a classification block. We provide theoretical insights to show that this architecture allows the embedding and classification blocks to be fixed after pre-training on a diverse set of tasks; only the covariate transformation block with relatively few parameters needs to be fine-tuned for each new task. To facilitate training, we also propose an approach to synthesize binary classification tasks, and demonstrate that DEN outperforms existing methods in a number of synthetic and real tasks in numerical studies.

Published in Transactions on Machine Learning Research (11/2022)
Distribution Embedding Networks for Generalization from Diverse Set of Classification Tasks
Lang Liu University of Washington
liu 60 edu
Mahdi Milani Fard Google Research
mmmslanfforddgggggee.com
Sen Zhao Google Research
senzhao caroolae@ggogle.com
Reviewed on OpenReview: https: // openreview. formm? id- FdrGGGCg00
Abstract
We propose Distribution Embedding Networks (DEN) for classification with small data. In the same spirit of meta-learning, DEN learns from a diverse set of training tasks with the goal to generalize to unseen target tasks. Unlike existing approaches which require the inputs of training and target tasks to have the same dimension with possibly similar distributions, DEN allows training and target tasks to live heterogeneous input spaces. This is especially useful for tabular-data tasks where labeled data from related tasks are scaree. DEN uses three-block architecture: covariate transformation block followed by distribution embedding block and then classification block. We provide theoretical insights to show that this architecture allows the embedding and classification blocks to be fixed after pre-training on a diverse set of (asks; only the covariate transformation block with relatively few parameters needs to be fine-tuned for each new task. To facilitate training, we also propose an approach synthesize binary classification tasks, and demonstrate that DEN outperforms existing methods in a number of synthetic and real tasks in numerical studies.
1 Introduction
While machine lcarning has made substantial progress in many technological and scientific applications, its success often relies heavily on large-scale data. However, in many real-world problems, it is costly or even impossible collect large training sets. For example, in online spam detection, at any time, we may only possoss dozens of freshly labeled spam results. In health sciences, we may only have clinical outcomes few hundred study subjects. Few-shot learning (FSL) has recently been proposed to tackle such small data problems. It has now gained huge attention in many applications such as image classification (Koch et al., 2015; Finn et al., 2017), sentence completion (Vinyals et al., 2016; Munkhdalai et al., 2018), and drug discovery (Altae-Tran Tran et al., 2017); see Wang et al. (2020) for a survey.
new framework
The core idea behind most of FSL methods to augment the limited traîning data with prior knowledge, e.g., images of other classes in image classification or similar molecules' assays in drug discovery. In meta-learning based FSL, such prior knowledge is formulated as set of related training tasks assumed to follow the same task distribution (Finn et al., 2017), with the goal that the trained model could quickly adapt to new tasks. In practice, however, given an arbitrary target task, the degree and nature of its relationship to the available auxiliary training data is often unknown. In this scenario, it is unclear if existing approaches can extract useful information from the training data and improve the performance on the target task. In fact, there is empirical evidence suggesting that learning from unrelated training tasks can lead to negative adaptation (Deleu & Bengio, 2018). In our numerical studies, we also observe similar behavior; existing FSL approaches perform poorly when training tasks are unrelated to the target task.

---

Published in Transactions on Machine Learning Research (11/2022)
Pre-training
Fine-tuning
Evaluation
Task
Target Task
Transformation
Transformation
Transformation
Task
Target Task
X
y
Embeddiig
Embedding
Embedding
X
Task 3
Query
Classification
Classification
Classification
X
Training and evaluation of DEN. We first pre-train DEN on training tasks with heterogeneous covariate spaces. We then fine-tune the transformation block on few labeled examples from the target task, and use the fine-tuned model for classification on the query set.
Figure 1:
In this paper, we investigate this important but under-studied setting few-shot meta-learning with possibly unrelated tasks. We specifically focus on classification tasks with tabular data. Unlike machine learning with image and text inputs, large datasets of related tasks are not available for generic tabular-data classification.
A key challenge in this setting that the input, in the form of covariate vectors for training and target tasks, can live in different spaces and follow different distributions with possibly different dimensions. Existing meta-learning techniques often assume a homogeneous input space across tasks and thus cannot be directly applied in such cases with heterogeneous covariate spaces.
this work, we propose Distribution Embedding Networks (DEN) for meta-learning on classification tasks with potentially heterogeneous covariate spaces. DEN consists of a novel turee-block architecture. It first calibrates the raw covariates via a transformation block. A distribution embedding block then applied form embedding vector serving the "summary" of the target task. Finally, classification block uses this task embedding vector along with the transformed query features to form prediction.
Since the tasks can be unrelated, we learn a different transformation block for each task to form task-invariant covariates for the rest of the network. Tn other words, the transformation block is task-dependert. We keep the fask-independent embedding and classification blocks fixed after pre-training, and use a few labeled examples from the larget lask (i.e., the support set) to fine-tune the lask-dependent transformation block. Since our setting is significantly more challenging than the standard few-shot meta-learning setting due the heterogeneity among training and target tasks, assume that we have access to slightly larger support. set compared to the FSL setting (e.g., 50 examples in total across all classes rather than 5 examples per class). We further assume that the support set follows the same distribution as the query set. To address the challenge of variable-length covariates, the classification block is built upon a Deep Sets architecture (Zaheer et al., 2017). Figure shows an overview of the architocture and training and evaluation mechanisms.
To summarize our main contributions: (T) We propose a method for meta-learning with possibly unrelated tabular-data training tasks an important setting that expands the application of meta-learning but has rarely been investigated in the literature; (II) We propose the bloce-block architecture, allowing the model large variety of lasks, and then fine-tuned on unrelated target task; we provide a
to be pre-trained scenario in which our three-block architecture can perform well; (III) Described in Section 5, we design a procedure to generate artificial tasks for pre-training, and empirically verify its effectiveness when testing on real tasks. This provides a principled way to generate training tasks and alleviates the cost collecting real training tasks. (IV) We compare DEN with various existing FSL approaches on both simulated and real tasks, showing improved performance in most of the tasks consider.
2 Related Work
There are multiple generic techniques applied the meta-learning problem in the literature (Wang al., 2020). The first camp learns similarities between pairs of examples (Koch et al., 2015; Vinyals et al.,
2

---

Published in Transactions on Machine Learning Research (11/2022)
2016; Bertinetto et al., 2016; Snell et al., 2017; Oreshkin et al., 2018; Wang et al., 2018; Sung et al., 2018; Satorras & Estrach, 2018; Liu et al., 2019; Mishra et al., 2018). For an unlabeled example on a new task, we use its similarity score with labeled examples of the given task for classification. The second camp of optimization-basee meta-lcarning aims to find a good starting point model such that it can quickly adapt to new lasks with a small number of labeled examples from the new task. This camp includes different. variants of MAML. (Finn et al., 2017; Lee & Choi, 2018; Finn et al., 2018; Grant et al., 2018; Rusu et al., 2019) and Meta-Learner LSTM (Ravi & Larochelle, 2017). More recently, Lee et al. (2020; 2021) proposed to learn task-specific parameters for the loss weight and learning rate for out-of- distribbutionn tasks. Their use of task-embedding is conceptually similar to DEN. The third camp is conceptually similar to topic modeling, such as Neural Statistician (Edwards & Storkey, 2017) and CNP (Garnelo et al., 2018), which learn a task specific (latent)) embedding for classification. The final camp utilizes memory (Santoro et al., 2016; Kaiser et al., 2017; Munkhdalai & Yu, 2017; Munkhdalai et al., 2018). Note that all the above methods assume that all training and target tasks are related and share the same input space.
A closely related problem is Domain Generalization (DG) which estimates functional relat ronship between the imput and output y given data from different domains (i.e., with different marginal P(æ)); see, e.g., Wang et al. (2021) and Zhou et al. (2021) for surveys. The core idea lies behind a large class of DG methods is to learn a domain-invant et al., 2018; Shen et al., 2018), which aligns the marginal P(x) and/or the conditional P(ylær) distributions across multiple domains. In a similar spirit, DEN first adapts to the task via the transformation block and then learns a task-invarian representation via the task- ies--indpeedeent embedding block.
feature representation (see, e.g., Muandet et al., 2013; Li et al., 2018a;b; Shankar
Learning from heterogeneous feature spaces has been studied in transfer learning, or domain adaptation (Dai et al., 2008; Yang et al., 2009; Duan et al., 2012; Li et al., 2014; Yan et al., 2017; Zhou et al., 2019); see Day & Khoshgoftaar (2017) for a survey. These approaches only focus on two tasks (source and target), and require the model to learn a transformation mapping to project the source and target tasks into the same space.
Unlike meta-learning and DG methods, DEN is applicable for tasks with heterogeneous covariates spaces. This phenomenon especially prevalent in tabular data tasks, where the number and definition of features could be vastly different across tasks. Iwata & Kumagai (2020) is among the first works that combine meta-learning with heterogeneous covariate spaces. Both their approach and DEN rely on pooling to handle variable-length inputs, using building blocks such as Deep Sets (Zaheer et al., 2017) and Set Transformers (Lee et al., 2019). There are several differences between their approach and DEN. Firstly, DEN uses a covariate transformation block, allowing it adapt to new tasks more efficiently. Secondly, their model is permutation invariant in covariates and thus restrictive in model expressiveness; while DEN does not have this restriction. Thirdly, we also provide theoretical insights and justification for our model architecture design.
3 Notations
Let Ti....,, TM be M training tasks. For each training task 1, we observe an ii.d. sample Dj = ((rr, (T,e)))) from some joint distribution Pr, where [n]] = {1....,n) n) and æt, E Rd is the covari- ate vector of the i-th example and yT, E [LT] is its associated label. We denote this sample in matrix form by (Xr, Br), where the j-th column of E R"T is the j-th covariate vector. We let XT, be the covariate sub-matrix corresponding to examples with label & for k E [LT]. When the context is clear, we drop the dependency on T for simplicity of the notation, e.g,, we write X X
Let S be a target task that is not contained in the training tasks. We are given a set of labeled examples (Xs, ys), where the sample size ns is small. We refer to it as the support set. The goal is to predict labels for unlabeled examples in the target task, which is called the query set. We denote T = (TI...., S}.
4 Distribution Embedding Networks
We first describe the model architecture of DEN for binary classification in Section 4.1, and then extend DEN for multi-class classification in Section 4.2. Finally, we provide insights into the model architecture and justify its design in Section 4.3.
3

---

Published in Transactions on Machine Learning Research (11/2022)
9F
Query batch
Trinnng Loss
X
Z
Transform (1)
Deep Sets Struetured Classifier (3) or (8)
x
z
Distribation Embedding (2) or (4)
Support batch:
88
Figure Block diagram of DEN for binary classification. During pre-training, for each gradient step we sample task I 3 and two batches of data from the task (X4, yf and (xf..f)) During fine-tuning, we treat the support set the support batch, using which derive distribution embedding to make predictions the query set, treated as the query batch.
4.1 Model Architecture for Binary Classification
To describe the model architecture of DEN for binary classification (illustrated Figure 2), consider data (X,9) in given task T. DEN can be decomposed into three major blocks: transformation, embedding and classification. We will describe these blocks in this section.
Transforming covariates with tesk-dependent transformation block. We first transform the co- variales via a transformation block, i.e.,
Z = cX),,
(1)
where : Rd + Rd is applied to each row. Specifically, we use a piecewise linear function' (PLF) for each covariate, i.c., c(x) = (c' (x'), "a.',...,cd,,
(.....,((()))) (x*)). PLFs can be optionally constrained to be monotonic, which would serve as form of regularization during training (Gupta al., 2016). Note that the transformation block is task-dependent its parameters need be re-trained for each new task. The goal is that, after applying the corresponding transformation to each task, the relatedness across tasks increases. In contrast, existing meta-learuing approaches usually do not have the transformation block and thus require the raw tasks to be related. This is conceptually similar to Muandet et al. (2013), where they consider the domain generalization problem and directly learn an invariant feature transformation by minimizing the dissimilarity across domains. To the contrary, we incorporate this block into a larger architecture and learn il in an end-to-end fashion.
One may instead consider other architectures than PLFs for the transformation block. We choose PLFs since they can implement compact oneddimensional non-lincarities and can thus be fino-tuned with small support set. Moreover, they are universal approximators: with enough keypoints, they can approximate any one-dimensional bounded continuous functions. In Section 5.3 we show that this PLF transformation is key to ensure good performance when training and target tasks have heterogencous covariate spaces. PLFs cannot model interactions between covariates, which is a sacrifice we may have to make in light of the small support set. We study in Section 5.3 the trade-offs between the flexibility of PLF and the size of the support set.
Summarizing task statistics with task-independent distribution embedding block. The second block in DEN is to learn a vector that summarizes the task distribution. This is similar to Garnelo al. (2018). Naively, could learn non-linear transformation é which embeds (z, y) into vector of smaller dimension. However, this would not work since the dimension of z can vary across tasks. In contrast, we embed the distribution P(z, y) of a given task into a vector using the transformed features Z in the following
See Appendix A for the precise definition.
4

---

Published in Transactions on Machine Learning Research (11/2022)
way. For all a. b E [d], we derive distribution embedding of P(2", 2", y) by
s (h(ZT.",, h,
(2)
where h is vector-valued trainable function, and the average is taken with respect to the training batch during training and support set during inference, i.e.,
h ( = Di= 11 4 1 yy = =k] D 1//4 = =}
and - Vr. Note that about 50 in our setting, so the empirical average close the population counterpart. Intuitively, decompose variable-length feature vector into smaller picces of fixed length 2, and use the h function to learn a pairwise embedding ga.b for each pair of the 2 features and b. This pairwise decomposition allows us to handle variable-length covariates. Note that the same function h is shared across all tasks, and it can be chosen as a few fully connected layers. The distribution embedding of P(z.8) y) is thus the set of embeddings all pairs s = (s*h)), a.bbEdd
Remark 4.1. The length 2 here arbitrary. We can use pieces of length r for any 2 1, i.e., obtain an embedding s' of P(z y) for all fi....,t E [d]. The larger r is the more expressive the model will be. We refer to the dependency order, and experiment with different values in Section 5.3.
Prediction with task-independent classification block. Given a query we first transform it via = = c and decompose the transformed features into sets of feature pairs. We then obtain the distribution embodding vector ga.b of each pair in (2). Finally, we obtain the prodicted logits using a Deop Sets architecture (Zaheer al., 2017):
q= 1 = v Σ s ([2 s0051)
a.bE(d)
where ip is vector-valued trainable function and vé is a real-valued trainable function. Both yp and te are shared across tasks and can be chosen as fully connected layers. The Deep Sets archilecture, which aggregates all possible pairs of covariates, is proven to be a universal approximator of set-input functions (Zaheer et al., 2017). We note that one may use other set input architectures to construct the classification block, c.g., Sct Transformer (Lee al., 2019).
4.2 Model Architecture for Multiclass Classification
For multiclass classification tasks, we modify the distribution embedding and classification blocks. Specifically, we modify the distribution embedding in (2) as
2h b Σ h (()))))
v(v))))
(4)
where v N + Rm is vector-valued, trainable function that is shared across tasks v(k) is hence vector encoding of class a-d h is vector-valued trainable function with input dimension m 2.
For the classification block, we modify the idca of Matching Net (Vinyals et al., 2016), which also similar to the modification adopted by Iwata & Kumagai (2020). This modification is suitable for tasks with different numbers of label classes. Tet & be in (3) without the last layer, i.e., 6(2, s) = Eabeta (([2a., s%1 (1%))) the penultimate layer embedding of the query set example. We then obtain its class scores by
((2, 8) $( ΣTh 1/y = k}
k)
(5)
Note that i in the above equation the index of support set examples (n in total). The score qe can thus be interpreted as the average dot-product of the penultimate layer embodding of the given query example with the penultimate embedding of the support set examples of class k. To obtain class probability, we apply a softmax q&.s
5

---

Published in Transactions on Machine Learning Research (11/2022)
4.3 Rationale for the Architectural Design
For further insights into the model architecture, consider the optimal Bayes classifier:
P(u k [ P( y P(y =
(6)
Σ P(æ y= P)
For a given task, if the conditional probability P(#: y y= k) belongs to the same family of distributions for all & [L], i..,, -y k ~ o(x; 0y), then the Bayes classifier can be constructed by estimating the paramcters O2 and Ply = k), and approximating the density Φ as
(x P(y
P (( 0.)
If, additionally, P( y-y-k k) belongs to the same family of distributions for all [4] and also all tasks, then the Bayes classifers for all lasks should have the same functional form; only the parameters Og and Pyy k) differ by task. In this case, we can simply pool the data from all tasks together estimate Os. Ply k) for cach task, and the task-independent function s
However, the distribution family dba;; 0&) may vary greatly across tasks. The task-dependent transformation block allows the transformed covariates to be in the same distribution family approximately. Then, we utilize the distribution embedding block to estimate the paramoters 0g and P(y = k). If all transformed covariates belong to the samne distribution family, then the function form to estimate the parameters 0₂ and P(y = k) should be identical for all tasks, which justifies our use of a fask--ndeppenden distribution embedding block. Finally, we use the task-independent classification block to approximate the task-independent function á and obtain score for each label. The effectiveness of DEN depends crucially the ability of the transformation block to align distribution families of covariates across tasks. We study in Section 5.3 the performance of DEN in relation to the flexibility of PLFs.
Finally, since the covariate dimension can vary across tasks, we decompose the covariate vector into sub-vectors of fixed length r and apply a Deep Sets architecture to these sub vectors. In fact, Proposition 4.3 shows that if we consider the following family of densities, then the Bayes classificr must be of the form (3).
Definition 4.2. Let [f(:; 0) Rn -> R] be a parametric family of functions (not necessarily densities). For any integer d 2 r, we say function g on IRd admits an f-erpansion if it factorizes as g(z) = [l. f(z Ohw), where { R") } is a set of parameters.
For instance, if zly = ~ NN(u,, Ia), then the conditional density p(zly= = 1) is proportional to
(2 112)=
II a=1
exp o
20²
which admits an f-expansion with r= and 0a = (4e,
Proposition 4.3. Let (z,y) be a random vector in Rd x [L] following some distribution P. Assume that the conditional density p(zly- = k) admits an f-crpansion for some parametric family of functions {f(: [[: on R with paramelers {0".), Then there erist functions vp and up such that
Ply = - 12) 1b
Σ g(z 0 TT)) EirrE
(8)
where ziir = (2) T = Ply = k), and vb and only depend on f.
The proof is included in Appendix A. Note that model (3) has exactly the same structure the optimal Bayes classifier in (8) with r = 2 and s representing the paramcters (0t } and marginal { Proposition 4.3 shows that, under appropriate conditions, our model class is expressive enough to include the optimal Bayes classifier. This justifies our choice of the distribution embedding (2) and the Deep Sets structure (3). DEN will consequently performed well when learning across tasks whose conditional distributions of the PLF transfranformed feature admits the same f-expansion. Heuristically, this means DEN is ideally applied meta-learning settings in which features across tasks can be transformed to have similar structure.

---

Published in Transactions on Machine Learning Research (11/2022)
Table Overview of training and evaluation.
Step
Input data
Transform
Embedding
Classification
Pre-training
Heterogeneous tasks ((oTT,.B,))))
Trained
Trained
Trained
Fine-tuning
Support set {( vs.s))
Trained cs
Fixed
Fixed
Evalustion
Query æ5,a and support set (s,,)
Fixed cs
Fixed
Fixed
4.4
Training and Inference
Figure 2 shows a high level summary of our thrce-block model architecture. The overall training and evaluation procedure is summarized in Table 1. Note that all sets of r imputs use the same h and functions through parameter sharing, which reduces the size of the model. For DEN applicd on tasks with d features, the feature transformation block has (dd parameters. An embedding block with embedding order r, L layers and H hidden nodes per layer has rH (2 1) parameters. A classification block with layers and H hidden nodes per layer for both the y and s functions has (r + 2) + 2HH2 parameters. This DEN model is roughly of the same size as 3L-layer deep H-node wide neural network. Empirically, we found that the computational complexily and resource requirement of DEN are similar to those of Vinyals et al. (2016) and Iwata & Kumagai (2020) given similar model sizes.
During pre-training, in each gradient step, we randomly sample a task T E {Te) and two batches A and B from 9r). These two batches are first transformed using PLFs in (1). We then use (2) or (4) to obtain a distribution embedding sy, taking the average with respect the examples in the support batch. Next, we use the distribution embedding to make predictions on the query batch using (3) or (5). Note that during training, sT is identical across examples within the same batch, but it could vary (even within the same task) across batches.
If PLDs can approximately transform the covariates so that they admit the same f-expansion, then the rest of the network is task- ieee--ndeeendent. Thus, after pre-training on {T,)e, for each new task S, we can fine-tune the transformation block (1), while keeping the weights of other blocks fixed. Because PLFs only have a small number of parameters, they can be trained on a small support set from the task S.
During inference, we first use the learned PLFs in (1) to transform covariates in both the support set and the query set. We then utilize the learned distribution embedding block to obtain 8s, where the average in (2) and (4) is taken over the whole support set. Finally, the embedding and PLF-transformed query set covariates are used to classify query set examples using (3) or (5).
5 Numerical Studies
In Section 5.1, use OpenML tabular datasets to demonstrate the performance of DEN real-world tasks. DEN achieves the best performance among a number of baseline methods. We then introduce in Section 5.2 approach to simulate binary classification tasks, which allows us to generate huge amount of pre-training examples without the need of collecting real tasks. Surprisingly, DEN (and several other methods) trained on simulated data can sometimes outperform those trained on real data. In Section 5.3, we examine the performance of DEN in relation to different architecture and hyper-parameter choices. The findings are: (a) PLF and fine-tuning are crucial when the training and test tasks are unrclated (c.g., when training tasks are simulated), whereas their effect is insignificant when the tasks are similar, and (b) the performance of DEN is relatively stable for small values of the dependency order r in Remark 4.1.
Bascline methods for comparison with DEN include Matching Net (Vinyals et al., 2016), Proto Net (Snell et al., 2017), TADAM (Oreshkin et al., 2018), PMN (Wang et al., 2018), Relation Net (Sung et al., 2018), CNP (Garnelo et al., 2018), MAML (Finn et al., 2017), BMAML (Finn et al., 2018), T-Net (Lee & Choi, 2018) and Twata Kumagai (2020). Hyperparameters of all mothods are chosen based on voss-validation on training tasks. Hyperparameters, model structures and implementation details are summarized in Appendix B. Note that we set the dependency order 2 if it is not stated explicitly. For MAML,, BMAMT. and T-net,
7

---

Published in Transactions on Machine Learning Research (11/2022)
Table 2: Average test AUC (standard error) and the percent of times that each method achieves the best. performance on 20 OpenML datascts 20 repeats.
Method
Average Test AUC (%)
% Best
Matching Net
50.11 (0.04)
0.00%
Proto Net
71.11 (0.72)
27.50%
PMN
56.11 (0.65)
0.75%
Relation Net
51.65 (0.28)
0.25%
CNP
58.01 (0.69)
7.00%
Iwata & Kumagai (2020)
70.35 (0.70)
26.75%
MAMT.
60.64 (0.82)
7.00%
T-Net
52.22 (0.41)
0.5%
DEN
70.12 (0.83)
30.25%
we fine-tune the last layer of the base model for epochs on the support set. For the rest of the methods, we train them with episodic training² (Vinyals et al., 2016). For methods that do not readily handle variable length inpuls, randomly repeat. features to make all tasks have the same length of inputs.
5.1 Results on OpenML Classification Tasks
5.1.1 Binary classification
We compare DEN with baseline methods on 20 OpenML binary classification datasets (Vanschoren et al., 2013) following the setup in Iwata & Kumagai (2020) (see a list of datasets in Appendix C). These datasets have examples ranging from 200 to 1,000,000 and features ranging from 2 to 25.
We pre-train DEN and baseline methods on the OpenML. datasets in the leave-one-dataset--ul fashion. That. is, for each of the 20 OpenMI. datasets chosen target task, pretrain the models the remaining 19 datasets. For the target task, randomly select 50 examples to form the support set, and use the rest of the dataset as the query set. We repeat the whole procedure 20 times, and report the average AUC and the percentage that each method achieves the best performance in Table 2 based on 20 test sets x 20 repeats. DEN has average AUC comparable with the best methods, and the highest frequency that achieves the best AUC. As a comparison, directly training task-specific linear models on the support set of each task gives an average test AUC of 57.15%. Directly training an hidden layer, 8 hidden nodes neural network on the support set gives average test AUC of 54.58%. With 2 hidden layers, AUC drops to 53.00% and with 3 hidden layers, 49.74%. DEN significantly outperforms those methods. These results demonstrate the effect of over-fitting for classical methods on small data, and the benefit of DEN over classical methods on those small-data problems. In Section 5.2, we will further describe an approach to generate binary classification training tasks through controlled simulation. DEN trained on simulated tasks, surprisingly, outperforms DEN trained on real tasks, and, in fact, achieves the best performance among all competing methods.
5.1.2 Multiclass classification
In addition to binary classification tasks, we also compare DEN with baseline methods on 8 OpenML multi-class classification datasets. These datasets have examples ranging from 400 to 1,000,000, features ranging from 5 to 23, and number of classes ranging from 3 to 7. We train DEN and baseline methods on the OpenML datascts in the same leave-one-dataset-ott fashion as in the binary classification. Results are summarized in Table 3. DEN achieved the best. performance, followed by Matching Net. We also compare against directly training a neural network (NN) on the support set, which achieved decent accuracy. But DEN remains the best methods in majority (61.6%) of cases.
2Code available at htw.://h.//.......oo.ooooooo/o/a
_embeddinggggggneen embeedinng_netwonks. networks.
8

---

Published in Transactions on Machine Learning Research (11/2022)
Table Average test accuracy (standard error) and the percent of times that each method achieves the best performance on OpenML, datascts x 20 repeats.
Method
Average Test Accuracy (%)
% Best
Direct NN
45.68 (1.56)
13.6%
Matching Net
46.36 (1.55)
13.8%
Proto Net
33.68 (1.80)
4.9%
PMN
24.77 (0.63)
0.0%
Relation Net
33.49 (1.82)
4.9%
CNP
18.77 (1.53)
1.1%
Iwata & Kumagai (2020)
24.93 (0.62)
0.0%
DEN
48.60 (1.51)
61.6%
5.2 Generate Training Tasks through Controlled Simulation
In this section, we describe an approach to generate binary classification pre-training tasks based on model aggregation. Comparing to pre-training meta-learning models on related real-world datasets, which could be expensive to collect, this synthetic approach can easily give us a huge amount of pre-training examples. We will show that meta-learning methods trained with simulated data can, surprisingly, sometimes outperform those trained with real data.
Specifically, we first take seven image classification datasets: CIFAR-10, CIFAR-100 (Krizhevsky, 2009), MNIST (LeCun et al., 2010), Fashion MNIST (Xiao et al., 2017), EMNIST (Cohen et al., 2017), Kuzushiji MNIST (KMNIST; Clanuwat et al., 2018) and SVHN (Netzer et al., 2011). On each dataset, we pick nine oqually spaced cutoffs and binarize the labels based on whether the class id below the cutoff. This gives nine binary classification tasks for each dataset with positive label proportion in (0.1, 0...,......... 0.9).
To generate covariates for each task, we build 50 convolution image classifiers of various model complexities (details in Appendix B) on each of the 63 tasks to predict the binary label. We take classification scores on the test set as covariates. With these covariates and associated labels, we construct 7 x 9 = 63 binary classification tasks {U....., T63}. Essentially, they are model aggregation tasks since we are combining 50 classifiers to make prediction. Note that the accuracy of those image classifiers ranges from below 0.6 over 0.99, giving rise covariates ranging widely in their signal-to-noise ratios.
Finally, to augment training data, we apply covariate sampling during pre-training. Tn each pre-training step, we first randomly sample an integer C' (1, 50} and task from the 63 aggregation tasks {U,...., Tea described above. Then, among the 50 convolution image classifiers we built, we randomly pick C' of them and use their classification scores as covariates to construct a sub-lask: Finally, DEN takes labeled examples from this sub-task and uses them for training in this step. We shall emphasize that although training tasks are built based on image classification datasets, we do nol use raw pixel values as covariates in pre-training.
OpenML binary classification with simulated training tasks. To illustrate the effectiveness of our data simulation approach, we use the models pre-trained on these data and evaluate them on the same 20 OpenML binary classification tasks in Section 5.1 after fine-tuning. Interestingly, as shown in Table 4, for 5 out of the methods considered, pre-training test. AUC. This suggests that the proposed approach to generate training tasks is not only convenient but also effective. Moreover, DEN pre-trained on the simulated data outperforms all methods (either pre-trained on the simulated data or OpenML data) significantly.
the simulated data gives statistically significantly better
5.3 Effect of Dependence Order, PLF, and Fine-tuning
We continue our exanination of DEN with pre-training on simulated data. In addition comparing DEN with competing methods, also study the performance of DEN in relation to its dependence order r, and

---

Published in Transactions on Machine Learning Research (11/2022)
Table 4: Average test AUC (standard error) and the percent of times that each method achieves the best. performance on 20 OpenML datasets 20 repcats. The models are pre-trained on simulated data. We also report the improvement in average test AUC compared to pre-training on OpenML data.
Method
Average Test AUC (%)
% Best
Improv. in Test AUC (%)
Matching Net
53.88 (0.46)
0.00%
3.77 (0.46)
Proto Net.
71.12 (0.65)
28.00%
0.01 (0.97)
PMN
59.04 (0.68)
1.25%
2.93 (0.94)
Relation Net
57.97 (0.59)
0.00%
6.32 (0.65)
CNP
60.95 (0.69)
2.50%
2.94 (0.98)
Twata & Kumagai (2020)
66.01 (0.74)
11.50%
-4.34 (1.02)
MAML
61.16 (0.66)
2.75%
0.52 (1.05)
T-Net
53.35 (0.46)
0.25%
1.13 (0.62)
DEN
74.13 (0.68)
53.75%
4.01 (1.07)
Table 5: Average test AUC (slandard error) on Nomao and Puzzles data.
Method
Nomao
Puzzles
Matching Net
73.18 (2.23)
62.65 (1.45)
Proto Net
80.56 (0.56)
73.77 (0.37)
TADAM
82.42 (0.35)
74.86 (0.25)
PMN
77.00 (3.13)
57.69 (1.53)
Relation Net
52.32 (1.61)
63.73 (1.79)
CNP
91.40 (0.36)
53.20 (0.12)
Iwata & Kunnaaa (2020)
66.85 (3.23)
61.32 (0.52)
MAMT.
78.92 (2.22)
54.92 (0.54)
BMAML
47.20 (3.40)
53.49 (1.88)
T-Net
61.42 (3.44)
54.55 (1.78)
DEN w/o PLF w/o FT
59.01 (1.04)
68.87 (0.38)
DEN w/o FT
94.42 (0.16)
70.74 (0.62)
DEN
95.21 (0.10)
78.11 (0.53)
the of PLF and fino-tuning. For DEN, we explore three options: 1) fine-tune the PLF layer for 10 epochs, 2) take the PLF layer directly from the last pre-training epoch without fine-tuning, 3) do not include a PLF layer in DEN and hence no fine-tuning at all.
5.3.1 Tasks with Heterogeneous Covariate Spaces
We study and demonstrate the importance of PLF when the training and test tasks are heterogeneous. Specifically, we use all 63 simulated tasks described in Section 5.2 for pre-training, and test the performance on two real datasets: Nomao and Puzzles. We give short description of each dataset in Section C.
For each dataset, repeat the whole procedure 20 times and report the average AUC and standard error in Table 5. It is clear that DEN outperforms other baseline methods significantly. More importantly, the results also show that fine-tuning and PLF greatly improve the performance of DEN. Since DEN is pre-trained on simulated tasks which are completely unrelated to the target task, this improvement demonstrates the importance of fine-tuning and PLF when the training and target tasks are heterogeneous.
We also examine the effect of the size of the support set on the fiexibility of the covariate transformation block. Figure 3 shows that with enough support set examples (e.g., Puzzles task), having more PL. keypoints
10

---

Published in Transactions on Machine Learning Research (11/2022)
0.95
Test AUC
0.90
Nomao
0.85
Puzzles
0.80
0.75
10
30
Number of PLF Keypoints
Figure 3: Average test AUC versus the number of PLF keypoints on Nomao and Puzzles data.
Table 6: Average test AUC (standard error) of DEN on Nomao data with different dependency order.
r 1
r. 2
r = 3
r
94.38 (0.46)
95.21 (0.10)
93.80 (0.25) 92.61 (0.55)
r 5
r 6
r 7=7 7
r
93.16 (0.45)
91.22 (0.74)
88.95 (2.02)
89.15 (1.34)
benefits the test performance due to the improved ability of lask-adaptation. However, the support set is small (e.g., Nomao task), having a flexible covariate transformation block could even be marginally harmful.
Finally, conduct an ablation study to examine the effect of dependency order in Remark 4.1 the test AUC the Nomao dataset. In particular, we examine DEN with E [8]. The results in Table 6 show that when r < 5, the test AUC is relatively stable, with the best performance achieved at r 2; whereas the test AUC is much worse and unstable for larger r.
5.3.2 Tasks with Homogeneous Covariate Spaces
We study the performance of DEN in the case when the training and target tasks are homogeneous. To ensure the task homogeneity, we use the model aggregation tasks described in Section 5.2 for both training and evaluation. Specifically, we use the 5 x 9 = 45 tasks described Section 5.2 derived from CIFAR-10, CIFAR-100, MNIST, Fashion MNIST and EMNIST to train DEN and other meta-learning methods. We then pick four test tasks from SVHN and KMNIST of different difficulties, where the average AUCs over 50 classificrs are 68.28 28, 78. 18,11%, 91.51%, and 87. $7.58%, respectively. Note that the training and test tasks are totally separated, but they likely follow similar distributions since the covariates for training and tests all classifier scores.
For each test task, we randomly select 100 sets of C classifiers among the 50 candidate classifiers, resulting in 100 aggregation sub-tasks. For each aggregation sub-task, we form a support set with 50 labeled examples and a disjoint query set with 8000 examples. We repeat the entire training and fine-tuning process for times, and report the average AUC and its standard error.
Table 7 shows the result of aggregating ensemble of C 25 classifiers. Table 8 shows the result where the number of classifiers C to be aggregated is sampled uniformly from [13, 25] and could vary across sub-tasks (100 aggregation sub-tasks x 5 repeats). To allow baseline methods to take varying number of covariates, we randomly duplicate some of the C' classifiers so that all inputs have 25 covariates.
We observe that DEN significantly outperforms other methods all tasks, and that DEN without PLF and/or without fine- tne-tuning is statistically no worse than DEN with fine-tuning on the PLF layer. This suggests that fine-tuning the PLF layer not necessary when the data distribution similar among tasks.
11

---

Published in Transactions on Machine Learning Research (11/2022)
Table 7: Average test AUC (standard error) when aggregating 25 classifiers.
Test AUC (%)
Method
Task 1
Task
Task 3
Task
Matching Net
70.01 (0.40)
82.13 (0.05)
95.29 (0.06)
93.82 (0.08)
Proto Net
90.95 (0.05)
89.84 (0.03)
98.07 (0.01)
97.40 (0.02)
TADAM
90.98 (0.05)
89.90 (0.03)
98.14 (0.01)
97.57 (0.02)
PMN
86.78 (0.10)
88.69 (0.03)
97.46 (0.02)
96.22 (0.05)
Relation Net
85.39 (0.15)
88.70 (0.02)
97.25 (0.02)
95.55 (0.08)
CNP
86.53 (0.09)
88.80 (0.02)
97.50 (0.02)
96.22 (0.05)
Twata & Kumagai (2020)
89.33 (0.05)
89.17 (0.02)
97.93 (0.01)
97.73 (0.02)
MAML
86.10 (0.11)
88.78 (0.03)
97.48 (0.02)
96.13 (0.06)
BMAML
71.38 (0.84)
85.96 (0.21)
97.04 (0.08)
95.39 (0.18)
T-Net
86.23 (0.10)
88.76 (0.03)
97.47 (0.02)
96.11 (0.06)
DEN w/o PLF w/o Fine- Tine-nuning
91.53 (0.03)
90.18 (0.02)
98.03 (0.01)
98.37 (0.01)
DEN w/o Fine-Tuining
91.76 (0.03)
90.20 (0.02)
98.18 (0.01)
98.41 (0.01)
DEN
91.80 (0.03)
89.77 (0.02)
97.38 (0.01)
97.23 (0.01)
Table 8: Average test AUC (standard error) when aggrogating variable mumber of classifiers
Test AUC (%)
Method
Task 1
Task
Task 3
Task
Matching Net
75.16 (0.37)
81.12 (0.08)
95.86 (0.04)
93.61 (0.08)
Proto Net.
90.02 (0.11)
89.65 (0.04)
97.94 (0.02)
97.57 (0.02)
TADAM
90.20 (0.10)
89.74 (0.04)
98.04 (0.01)
97.84 (0.01)
PMN
85.68 (0.24)
88.59 (0.04)
97.30 (0.03)
95.83 (0.08)
Relation Net
80.85 (0.65)
88.53 (0.04)
97.11 (0.04)
95.03 (0.12)
CNP
84.97 (0.28)
88.71 (0.04)
97.31 (0.04)
95.96 (0.07)
Iwata & Kumagai (2020)
89.26 (0.16)
89.05 (0.03)
97.82 (0.02)
97.75 (0.02)
MAML
85.39 (0.23)
88.53 (0.04)
97.32 (0.04)
95.86 (0.08)
BMAML
59.57 (1.07)
85.40 (0.24)
96.73 (0.10)
94.53 (0.23)
T-Net
85.51 (0.22)
88.55 (0.04)
97.35 (0.03)
95.94 (0.07)
DEN w/o PLF w/o Fine- Tane-Tunig
91.06 (0.09)
89.92 (0.03)
98.09 (0.01)
98.24 (0.01)
DEN w/o Fine-Tuning
91.17 (0.09)
89.95 (0.03)
97.95 (0.01)
98.10 (0.01)
DEN
91.29 (0.03)
89.83 (0.02)
97.60 (0.03)
97.47 (0.01)
6 Conclusion
In this work, we introduce a novel meta-learning method that can be applied to settings where both the distribution and number of covariates vary across tasks. This allows us to train the model on a wider range of training tasks and then adapt to variely of larget tasks. Most other meta-learning techniques do not readily handle such settings. In numerical studies, demonstrate that the proposed method outperforms number of meta-learning basclines.
The proposed model consists of three flexible building blocks. Each block can be replaced by more advanced structures to further improve its performance. DEN can also be combined with optimization based meta- learning methods, c.g., MAML. limitation of DEN that it requires calculating the embedding for dr different combinations of covariates, which is infeasible for high-dimensional tasks. A potential solution is random subset of these combinations. We leave the exploration of these options for future work.
12

---

Published in Transactions on Machine Learning Research (11/2022)
References
Han Altae-Tran, Bharath Ramsundar, Aneesh S Pappu, and Vijay Pande. Low data drug discovery with one-shot learning. ACS Central Science, 3(4), 2017.
Luca Bertinctto, João F. Henriques, Jack Valmadre, Philip H. S. Torr, and Andrea Vedaldi. Learning leed-forward one-shot learners. In NIPS, 2016.
Tarin Clanuwat, Mikel Bober-Irizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David Ha. Deep learning for classical Japanese literature. NeurIPS Workshop on Machine Learning for Creativity and Design, 2018.
Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre Van Schaik. EMNIST: Extending MNIST to handwritten letters. In IJCNN, 2017.
Wenyuan Dai, Yuqiang Chen, Guirong Xue, Qiang Yang, and Yong Yu. Translated learning: Transfer learning across different feature spaces. In NIPS, 2008.
Oscar Day and Taghi M. Khoshgoftaar. A survey on heterogeneous transfer learning. Journal of Big Data, 4, 2017.
Tristan Deleu and Yoshua Bengio. The effects of negative adaptation in model-agnostic meta-learning. arXiv preprinl, 2018.
Lixin Duan, Dong Xu, and Ivor W. Tsang. Learning with augmented features for heterogeneous domain adaptation. In ICML, 2012.
Harrison Edwards and Amos Storkey. Towards a neural statistician. In ICLR, 2017.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic mela-learning for fast adaptation of deep networks. In ICML, 2017.
Chelsea Finn, Kelvin Xu, and Sergey Levine. Probabilistic model-agnostic meta-learning. In NewrIPS, 2018
Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and S. M. Ali Eslami. Conditional neural processes. In ICML, 2018.
Erin Grant, Chelsea. Finn, Sergey Levine, Trevor Darrell, and Thomas Griffiths. Recasting gradient-based meta-learning as hierarchical Bayes, In ICLR, 2018.
Maya Gupta, Andrew Cotter, Jan Pfeifer, Konstantin Voevodski, Kevin Canini, Alexander Mangylov, Wojeiech Moczydlowski, and Alexander van Esbroeck. Monotonic calibrated interpolated look-up tables. Journal of Machine Learning Research, 17(109), 2016.
Tomoharu Iwata and Atsutoshi Kumagai. Meta-learning from tasks with heterogeneous attribute spaces. NewrIPS, 2020.
Lukasz Kaiser, Ofir Nachum, Aurko Roy, and Samy Bengio. Learning to remember rare events. In ICLR, 2017.
Gregory Koch, Richard Zemel, and Ruslan Salakhutdinov. Siamese neural networks for one-shot image rocognition. In ICML, 2015.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Yann LeCun, Corinna Cortes, and Christopher Burges. MNIST handwritten digit database. ATT Labs [Online]. Available: http://9gnnnlecun.n http://awnn..c.coo/.//// www./ooo/meeed/eeeeeeeeeeeeeeeee 2010.
Hae Beom Lee, Hayeon Lee, Donghyun Na, Sachoon Kim, Minscop Park, Eunho Yang, and Sung Ju Hwang. Learning to balance: Bayesian meta-learning for imbalanced and out-of-distribution tasks. In ICLR, 2020.
13

---

Published in Transactions on Machine Learning Research (11/2022)
Hayeon Lee, Eunyoung Hyung, and Sung Ju Hwang. Rapid neural architecture search by learning to generate graphs from datasets. In ICLR, 2021.
Juho Lee, Yoonho Lee, Jungtack Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set Transformer: A framework for attention-based permatation-iivariaat neural networks. In ICML, 2019.
Yoonho Lee and Seungjin Choi. Gradient-based meta-learning with learned layerwise metric and subspace. In ICML, 2018.
Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C Kot. Domain generalization with adversarial feature Icarning. In CVPR, 2018a.
Wen Li, Lixin Duan, Dong Xu, and Ivor W. Tsang. Learning with augmented features for supervised and semi-supervised heterogeneous domain adaptation. IEEE Transactions Pattern Analysis and Machine Intelligence, 36(6), 2014.
Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao. Deep domain generalization via conditional invariant adversarial networks. In ECCV, 2018b.
Yanbin Liu, Juho Lee, Minseop Park, Saehoon Kim, Funho Yang, Sungju Hwang, and Yi Yang. Learning to propagate labels: Transductive propagation network for few-shot learning. ICLR, 2019.
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta-learner. In ICLR, 2018
Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain generalization via invariant feature representation. In ICML, 2013.
Tsendsuren Munkhdalai and Hong Yu. Meta networks. In JCML, 2017.
Tsendsuren Munkhdalai, Xingdi Yuan, Soroush Mchri, and Adam Trischler. Rapid adaptation with condi- tionally shifted neurons. In ICML, 2018.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
Boris N. Oreshkin, Pau Rodríguez López, and Alexandre Lacoste. TADAM: task dependent adaptive metric for improved few-shot learning. In NearIPS, 2018.
Sachin Ravi and Hugo Larochelle. Optimization as model for few-shot learning. In ICLR, 2017.
Andrei A. Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-learning with latent embedding optimization. In ICLR, 2019.
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillierap. Meta-learning with memory-augmentted neural networks. In ICML, 2016.
Victor Garcia Satorras and Joan Bruna Estrach. Few-shot learning with graph neural networks. In ICLR, 2018.
Shiv Shankar, Vihari Piratla, Soumen Chakrabarti, Siddhartha Chaudhuri, Preethi Jyothi, and Sunita Sarawagi. Generalizing across domains via cross-gradient training. In ICLR, 2018.
Jian Shen, Yanru Qu, Weinan Zhang, and Yong Yu. Wasserstein distance guided representation learning for domain adaptation. In AAAT, 2018.
Jake Snell, Kevin Swersky, and Richard S. Zemel. Prototypical networks for few-shot learning. In NIPS, 2017.
Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip H. S. Torr, and Timothy M. Hospedales. Learning compare: Relation network for few-shot learning. In CVPR, 2018.
14

---

Published in Transactions on Machine Learning Research (11/2022)
Joaquin Vanschoren, Jan N. van Riju, Bernd Bischl, and Luis Torgo. OpenML: Networked science in machine learning. SIGKDD Eaplorations, 15(2), 2013.
Oriol Vinyals, Charles Blundell, Tim Lillicrap, Koray Karukcuoghu, and Daan Wierstra, Matching networks for shot learning. In NTPS, 2016.
Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, and Tao Qin. Generalizing to unseen domains: A survey on domain generalization. In IJCAI, 2021.
Yaqing Wang, Quanming Yao, James T Kwok, and Lionel M Ni. Generalizing from few examples: A survey on few-shot learning. ACM Computing Surveys, 53(3), 2020.
Yu-Xiong Wang, Ross B. Girshick, Martial Hebert, and Bharath Hariharan. Low-shot learning from imaginary data. In CVPR, 2018
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms. ArXiv preprinl, 2017.
Yuguang Yan, Wen Li, Michael K. P. Ng, Mingkui Tan, Hanrui Wu, Huaqing Min, and Qingyao Wu. Learning discriminative correlation subspace for heterogeneous domain adaptation. In IJCAI, 2017.
Qiang Yang, Yuqiang Chen, Gui-Rong Xue, Wenyuan Dai, and Yong Yu. Heterogeneous transfer learning for image clustering via the social web. In Joint Conference of ACL and IJCNLP, 2009.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabés Póezos, Ruslan Salakhuudinov, and Alexan- der J. Smola. Deep sets. In NIPS, 2017.
Joey Tianyi Zhou, Ivor W. Tsang, Sinno Jialin Pan, and Mingkui Tan, Multi-class heterogeneous domain adaptation. Journal of Machine Learning Research, 20, 2019.
Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy, Domain generalization: A survey. Xiv preprint, 2021.
15


Reviews:
[{"review_id": "QxDX4AulG4", "reviewer": "Reviewer_jvsS", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros\n\n- The motivation and the focus of the paper, which is to deal with input heterogeneity, is both sound and clear, and to my knowledge, this is the only work on meta-learning that focuses on dealing with heterogeneity in the tabular data. \n\n- The experimental comparison against existing methods as well as ablations of the proposed framework shows the effectiveness of the proposed framework as well as its individual components.\n\n- The theoretical analysis further justifies the choice of the architectural components (e.g. deep sets-based classifier).\n\n- The paper is overall well-written and well-structured, and both the methodology and the experiments section are easy to follow. \n\nCons\n\n- This is not the first work that deals with varying number of inputs, since [Lee et al. 20] and [Lee et al. 21] use set encoding to deal with datasets with different number and order of classes. Although these previous works do not tackle dealing with tabular data and does not use task-dependent embedding, the strategy used in [Lee et al. 20] and [Lee et al. 21] can be also utilized for tabular data with little modification. Thus, they should be acknowledged and be compared against. Maybe the proposed DEN could be compared against the L2B framework in [Lee et al. 20] in their any-shot, any-way classification setting since the goals for the two works are similar.  The task embedding using set encoding is further developed in [Lee et al. 21], and is used to generalize across datasets with different sizes and orders of classes, although experiments on NAS may be out of scope for this work. \n\n\n- The baselines used are somewhat dated, although there is one baseline from a work published in 2020. The authors should compare against more recent state-of-the-art meta-learning models from the recent years. Also, they are not direct competitors since they lack mechanisms to handle heterogeneous input distributions. \n\n[Lee et al. 20] Learning to Balance: Bayesian Meta-learning for Imbalanced and Out-of-distribution Tasks, ICLR 2020\n[Lee et al. 21] Rapid Neural Architecture Search by Learning to Generate Graphs from Datasets, ICLR 2021\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a meta-learning framework that can be used with heterogeneous covariate spaces, where the inputs for each task may be incompatible and/or have different dimensionalities. To handle such heterogeneity, the authors have proposed to learn a task-specific linear embedding for each task to send them to a more compatible, continuous space across tasks, while learning a task-independent distribution embedding block to summarize the task distribution and obtain a pairwise embedding of the set of features. Such a treatment of the features as a set allows to deal with variable length inputs that are dimension-wise incompatible. Finally, the authors propose to use a deep-set based classifier to deal with the input feature set. The authors provide theoretical analysis to justify the choice of architectural design, and further perform experiments on a tabular dataset against existing methods, which shows that the proposed framework outperforms them. \n\nContributions: \n- Proposal of a meta-learning framework that can deal with heterogeneous input spaces across tasks. \n- Experiments on a tabular data benchmark which shows the effectiveness of the proposed method. ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "- Please discuss [Lee et al. 20] and [Lee et al. 21], and modify the hierarchical set encoding in the two previous works into a flat set encoding and compare its performance on the tabular dataset benchmark, if possible.\n\n- If possible, please compare your methods on the any-shot any-way classification benchmark from [Lee et al. 20]. ", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "uPO9bCu127", "reviewer": "Reviewer_d2m2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "-Strengths\n  - The idea of Distribution Embedding Networks is quite interesting, and, to my knowledge, very different from many other meta-learning settings. \n  - The empirical performances are extremely strong\n\n-Weakness\n  - It's unclear where the power of the empirical performances is derived, some ablation studies are necessary. \n  - There seems to be no apparent reason why the study focuses on the tabular data. \n  - Proposition 4.3 seems to be a trivial result by reading the proofs in the appendix, also, the authors need to explain the significance of this result, there seems to be no mathematical significance of the result (which is fine), but then, there seems to practical significance either, what does the result guide us in using the method in practice?\n  - It's intuitively unconvincing that the method can be applied to arbitrary query tasks with arbitrary support tasks, yet it's not clear where the relation is specified, in other words, what are the application scenarios for this method? This also corresponds to the above question that there seems to be a missing link between the theoretical discussion and the practical side of the work. \n       ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a new Distribution Embedding Networks that focuses on the traditional tasks studied by meta learning: training with certain tasks and then generalize to new ones. Different from the focus studied by most of other meta-learning works, this paper focuses primarily on the tabular data, which might be the reason that this method manages to get a clear improvement over other existing methods. ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Please answer the questions above. ", "broader_impact_concerns": "no issues noted. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "HywM4Z07YE", "reviewer": "Reviewer_gQmZ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strong points\n1. This paper considered an important problem in meta representation learning: the tasks are not necessarily related. Then enforcing a meta-prior is not proper.\n2.  The proposed method is empirically validated by different open source datasets.\n\n### Weak points\n\n1. The proposed approach is merely a collection of different other approaches. Although TMLR does not emphasize the novelty, this paper is still strongly suggested to illustrate the motivation of choosing these components.\n2. Another main concern is why considering meta-representation learning in the context of tabular or simple image dataset. These datasets (particularly tabular dataset) could be solved by the “non-deep” regime such as kernel method or sparse coding. Based on this argument, the experiment or the proposed approach seems not convincing.\n3. Concerns in the proposed analysis.\n\n### Detailed comments\n\n1. A major concern is the motivation of the proposed approach. Please note the motivation of the problem itself is clear and well-appreciated.  In the context of uci dataset and simple image dataset, why do we need to learn a representation through deep neural networks?  If the data is UCI or simple image, we could learn the embedding though kernel or spare coding. The corresponding method could be significantly simplified. This reviewer feels rather confused about the motivation of the proposed approach or evaluated empirical results.\n\n2. The key to encode the task specific feature is equation (2-4), while this reviewer has multiple concerns:\n- In Eq(2), the task is represented by the label specific features and label average. While this seems a bit heuristic since the different labels across the tasks do not necessarily have the same label space. E.g, for two different binary classification, the same label average is not meaningful.\n- In Eq(3,4), the combination of learned embedding $Z$ and task specific embedding $S$ are concatenated, what is the support of combining information in this manner? Why not tensor product? \n- Besides, why introduce a,b in the embedding S, what is the specific benefit? This makes both high memory and computational complexity. It seems this paper did NOT report the comparison of  time and memory complexity.\n\n3. In Sec 4.3, the analysis is a bit ad-hoc, in Equation (6-7), the P(y) and P(x|y=k) are ground truth distribution. Unfortunately, in the context of this paper (few-shot settings), adopting the empirical term as an approximation of the ground truth distribution is incorrect. In fact, Sec 4.3 could not provide a clear explanation of the proposed approach. It seems an ad-hoc analysis without considering the few-shot or meta learning problems.\n\n\n4. In the empirical comparison, since the data is not complex or extremely high-dimensional, comparing with modern deep network based approaches seems unfair.\n\n\n\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper tracks an important problem in meta-learning: the tasks are not necessarily related. By designing the task specific representation, this paper jointly adopted the task specific representation and nature representation Z for the prediction. The empirical results on different UCI datasets and simple image datasets demonstrates the effectiveness of the proposed method.\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Indeed I really like the problem setup and the motivation, unfortunately the proposed approach seems not clearly addressing the problem. In fact, the proposed analysis seems ad-hoc and many components lack clear support or justifications. Based on these, I would suggest\n\n- A clear explanation and justification of each component, particularly why designing a meta-representation learning approach? If it is necessary, how to fully justify the learned representation is correct ? How to justify the encoding of the task specific feature $S$ exactly to capture the correct information?   The whole proposed approach lack clear **justifications**. \n\n- A revisit on the evaluated datasets. In general, the motivation of deep-meta learning is to learn a representation. In this paper, except accuracy, there is no other proper information/analysis about the representation learning. Why not a simple sparse-coding based approach on tabular datasets?\n\n- The derived theory could not support the proposed framework in few-shot settings. Authors are strongly suggested to make revisions on the theory. \n", "broader_impact_concerns": "Not applicable ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)