Distribution Embedding Network for Meta-Learning with Variable-Length Input
Authors: Lang Liu
Mahdi Milani Fard
Sen Zhao
Venue:  (2021)
Keywords: meta-learning
variable-length input
distribution embedding

Abstract: We propose Distribution Embedding Network (DEN) for meta-learning, which is designed for applications where both the distribution and the number of features could vary across tasks. DEN first transforms features using a learned piecewise linear function, then learns an embedding of the underlying data distribution after the transformation, and finally classifies examples based on the distribution embedding. We show that the parameters of the distribution embedding and the classification modules can be shared across tasks. We propose a novel methodology to mass-simulate binary classification training tasks, and demonstrate that DEN outperforms existing methods in a number of test tasks in numerical studies.

Under review as a conference paper at ICLR 2021
DISTRIBUTION EMBEDDING NETWORK FOR META- LEARNING WITH VARIABLE-LENGTH INPUT
Anonymous authors Paper under double-blind review
ABSTRACT
We propose Distribution Embedding Network (DEN) for meta-learning, which is designed for applications where both the data distribution and the number of features could vary across tasks. DEN first transforms features using learned piecewise linear function, then learns an embedding of the transformed data dis- tribution, and finally classifies examples based the distribution embedding. We show that the parameters the distribution embedding and the classifica- tion modules be shared across tasks. We propose novel methodology mass-simulate binary classification training tasks, and demonstrate that DEN out- performs existing methods in a number of test tasks numerical studies.
INTRODUCTION
Deep learning has made substantial progress in a variety of tasks in image classification (c.g.. He
et al. 2016), object detection (e.g.. Redmon & Farhadi] [017 He et al. 2017). machine translation (e.g.. Vaswani et al., 2017) and natural language understanding (e.g.. Devlin et al. 2019). These achievements rely on efficient gradient-based optimization algorithms (e.g.. Duchi et al. 2011
Sutskever et al. 2013: 3 Kingma Ba. 2015) as well as a large number of labeled examples to train highly flexible deep learning models. However, in many applications, it is prohibitively expensive or impossible to collect a large amount labeled training data, calling for techniques that can learn from small labeled data. Meta-learning aims to tackle the small data problem by training model on labeled data from number of related tasks, with the goal to learn model that can perform well on similar, but unseen future tasks with only a small amount of labeled training data.
In this work, we propose a meta-learning model for classification using Distribution Embedding Networks (DEN). Unlike many existing meta-learning algorithms that assume fixed feature set across tasks, DEN is designed for applications where both the distribution of features and the number of features could vary across tasks. For example, we may use DEN to learn the optimal aggregator of an ensemble of models to replace the naive majority vote, where in different aggregation tasks, the distribution of model outputs and the number of models in the ensemble could be different.
On a high level, DEN first applies a learned feature transformation that allows features to be trans- formed into the same distribution family across tasks. It then uses neural network learn an embedding of the transformed data distribution. Finally, given the learned distribution embedding, together with the transformed features, DEN classifies examples using a Deep Sets architecture (Za- heer et al. 2017), enabling it to be applied to variable-length inputs. To adapt the model on new task, we only update the feature transformations with relatively few parameters.
2 RELATED WORK
There are multiple generic techniques applied to the meta-learning problem in the literature. The first camp of approaches learn similarities between pairs of examples. When presented with new task with small set of labeled examples, these methods classify unlabeled data based on their similarities with labeled ones. These methods include Matching Net (Vinyals et al.] 2016) and Prototypical Net (Snell et al. 2017), which learn distance metric between examples. Siamese Net (Koch et al., 2015) and Relation Net (Sung et al. (018)) use twin towers to lear the relationship between examples. Learn Net (Bertinetto al. 2016) proposes having class specific weights for

---

Under review as a conference paper at ICLR 2021
the towers. Satorras & Estrach| (2018) learns the similarity metric using a Graph Neural Network, and Transductive Propagation Network (Liu et al., 2019) classifies all unlabeled data at once by exploring the manifold structure of the new class space.
The second camp of optimization based meta-learning aims to find a good starting point model, so that when presented with new task, the meta model can quickly adapt perform well the new task with a small number of gradient steps. MAML (Finn et al.) (017) designs a learning algorithm, such that the expected loss of the learned meta-model on new tasks after one gradient step is min- imized. Meta-Learner LSTM (Ravi & Larochelle 2017) modifies the classical gradient steps with learned gradient update weights, which are trained minimize the validation loss. More recently, LEO (Rusu et al.] (019) extends MAML and utilizes Relation Net to learn low-dimensional latent embedding of model parameters and performs optimization-
meta-learning from this space.
Another camp of methods use internal or external memory for meta-learning. They include MANN (Santoro et al.] 2016) and Meta Net (Munkhdalai & Yu 2017). which store the past knowledge in external memory and internal model activations, respectively. New examples are classified by retrieving relevant information from the memory.
Our proposal does not take the above three routes. Rather, DEN is conceptually similar to topic mod- eling that learns the latent context variable. For example, Neural Statistician (Edwards Storkey) 2017) considers the hierarchical generative process and uses variational autoencoder (Kingma & Welling 2014) to learn the latent vector that summarizes the dataset an unsupervised fashion. Similar proposals include variational homoencoder (Hewitt et al., 2018) and CNP (Garnelo et al., 2018). In comparison, DEN is a supervised procedure, which learns an embedding of the data distribution. We then utilize this distribution embedding for classification on unseen examples.
NOTATIONS
In this paper, we use bold upper case letters to denote matrices (c.g., X), bold lower case letters to denote vectors (e.g., x), italic lower case letters denote scalars (e.g.. z), normal text to denote random variables (e.g., x), and bold normal text to denote random vectors (e.g., x).
Let I,,..., Tas be M training tasks, following some task distribution P. In training task T, observe set of independent features and label pairs, (Xr,, MT.). where XT, [a ....................... 2 2 Raixd, is the dj dimensional real-valued feature matrix, कं, E IR* is the j-th feature of task T; and yr, E {0. )d, is the binary label vector. We assume that the label is binary for simplicity of presentation. Our proposed model can be trivially extended multiclass classification problems. We use Pr. to denote the joint distribution of (XT,> YT,)
We first train DEN on the training tasks [T,,..., Tw.. Given a new task S with a small set of labeled examples, we fine-tune the trained model on S using these labeled examples. The final model is then applied to unlabeled examples in S for classification. The set of labeled examples is called the support set and the set of unlabeled examples is called the query set.
DISTRIBUTION EMBEDDING NETWORK
To motivate our proposal, we first consider the problem of minimizing the risk on a single task T:
ÔT arg min (xT.vr~PP+ (XT: 0),y,>
(I)
where f(::0) is a model with parameter O and L is the loss function.
Lemma 1. Assume the joint distribution Pt has probability density (mass) function q(= m.). Then the optimizer θr is of the form ( where $2..f. is some deterministic function depending on the loss L,, the model f and the density q-
The proof Lemma can be found in Appendix A It suggests that, when the joint distribution Pt is in a parametric family, the dependency of 0 on the task T is through two parts: the functional form of the density g and the distribution parameter 7/r-
2

---

Under review as a conference paper at ICLR 2021
Now, if the joint distributions of all training tasks I....... UM} are in the same parametric family, then there exists a common function fi..f. such that 0r, Li. (MT,) for every E ( J. Hence, we may reparameterize f(zr:fL,f, (nu)) as fz 71:7), and learn the new model pa- rameter y from observations (Xr,: yr,) and some estimate vÿr,-- One advantage of this reparameter- ization is that the optimal choice of is now task lask-independent, so, once learned from training tasks, it can be transferred to new unseen tasks whose data distribution falls in the same parametric family. Moreover, since the data distributions of all tasks are in the same parametric family, we can learn shared model to estimate 7) for all T.
In practice, the data distribution could vary greatly across tasks. Hence, for each task U, propose to apply a transformation er to the original features and work with (xT())). Yy), allowing it to unify the distribution family of (XT): YT)-
To summarize, our proposal, DEN, consists of three building blocks. Firstly, we apply a transforma- distribution embedding module to obtain
tion layer сj to the original features. Secondly, distribution embedding 8т of the transformed data (cT(X)),)r Yr). This corresponds to the distribution parameter V/T discussed above. Finally, we use classification module output prediction based on the transformed feature er (æT) and the distribution embedding sy. Here the last two modules are shared across tasks.
4.1 LEARNING THE DISTRIBUTION EMBEDDING
To illustrate the idea of learning a compact distribution embedding, we first consider special case that XJ (yr k) No(UT, ΣT.x) for E {0. With the normality assumption, Pr is uniquely characterized by its first moment vectors E(xr((yr k)]. second moment matrices E[xtx] (yy = k)], and the label positive rate P(yr 1). Given an i.idd sample (Xt, (T) where X = j at] E Randd and yr E {0, 1)", we may estimate these quantities by sample mo- ments, leading to a compact distribution embedding
ST x
tt
at.
xd o..
24,0 C 24,0-
1.0:
at, ad
@ ad a1,,1
where of denotes the j-th feature vector of examples with label k o denotes the arithmetic average of its elements for vector æ, and @ is the element-wise product.
There are two notable characteristics in this distribution embedding. Firstly, every element in 8T is of average form over examples within the same class. This motivates layer in DEN when obtaining the distribution embedding, Secondly, the embedding sT only involves quantities of single features and pairs of features. As discussed in Section 4.1.2 this allows to decompose the whole distribution embedding vector of task lask-dependent dimension (e.g., 2dd +2 for Gaussian data) into sub- vectors of fixed dimension, opening up an opportunity to handle variable- length features using a Deep Sets architecture (Zaheer et al. 2017).
batch average
4.1.1 NON-GAUSSIAN FEATURES
In practice, the normality assumption is usually overly simplified. To address this issue, one simple strategy is to first apply transformation er Rd -> Rd each task that eT (xx))((y) k) follows a multivariate normal distribution. However, such transformations are not guaranteed to exist. is also challenging analytically construct such transformations, even if they exist.
Instead of analytically constructing the transformations cr, we propose to learn a piecewise linear function (PLF) & R + IR for each feature j and task T, i.e.,
K-1
x - kt/ a +
(of - -ots)) (K4 s s kk.
+ of (z; kj af)
Σ
1e kt - Kt,
1=1
(3)
3

---

Under review as a conference paper at ICLR 2021
where k/ == [ xt,x] E RK, K4,1 keypoints that spans the domain of x and o :- lat, f.,kl E RK is the parameter vector of the PLF, characterizing its output at each keypoint. The PLF can be optionally constrained to be monotonic with of S aT, 1++ for all i = .,.... K - 1, which serves as regularization and can be satisfied through parameter projections during training through, e.g.. projected SGD.
k4 <.. Ki,K. K. is the vector of predetermined
PLFs can implement compact one- dieeenmnniionl non-linearities that be learned with small sam- ple. They are universal approximators in this space: with enough keypoints, they can approximate any bounded continuous function. Although it is guaranteed that features transformed by PLFs satisfy the normality assumption, PLFs make it possible that transformed features (approximately) belong the same parametric family for all tasks.
Since this parametric family is not necessarily normal, the sample moments used in (2) are no longer appropriate. Fortunately, the next lemma indicates that, as long as this parametric family is an exponential family, the distribution embedding can be chosen as an average form.
Lemma 2. Let Q be a probability distribution in an exponential family with density q(u 7) :=
B(u) exp[>(n) S(u) - A(n)). Let Ut i.d.d Q. Then Ei=1 S(u,) is a sufficient statistic for 17.
According to Lemma 2 if the conditional distribution Pi(zrly; = k) belongs to the same exponen- tial family for all tasks T and k E {0, 1}, then there exists task-independent function S such that S ..) sufficient for Pr (zr lyr = k). Consequently, we may use ST == S (21,0), (2x,,)) yr as a distribution embedding of the joint probability Pr. where S can be encoded as a neural network.
4.1.2 VARIARLE-LENGTH FEATURES
To handle variable-length features, instead of learning the whole distribution embedding ar di- rectly, we decompose the joint distribution of (TT into smaller pieces: conditional probabilities z) ,...... z
for all r-subsets {i1,...,44 ir} C .l,....,l d} and the marginal y, where r hyperpa- rameter shared across tasks. The optimal choice of should depend on the distribution of (Zı, yT). For example, discussed in (2). when z follows multivariate normal distribution conditional on Yr. 2 can sufficiently characterize the joint distribution of (ZT. Yr). We will use 2 in all numerical studies in Section
Following the discussion of Lemma 2 in Section 1.1
we use a learnable model g, shared across tasks, to derive a distribution embedding vector for each subset (ir....,4y in} C {I....,d}
=T,
yu
(4)
where the average is taken with respect to training batch during training or the support set during testing. We will discuss in more detail in Section 4.3
An alternative formulation of sT is directly embed the joint distribution of (z#) ......... 2 y).
fr 9 ([= , 2 yr)
(5)
Both formulations in and 5 work well in numerical studies presented in Section 5
4.2 CLASSIFICATION WITH DISTRIBUTION EMBEDDING
With a distribution embedding s fi....tr for each subset (i1,...,4,} 1, we group the embedding ,1....57 with its associated transformed features (z ....,??? ,/r), and then use a Deep Sets structure Zaheer al. 2017) over the of all M-subsets of features:
Σ n /
((+ )))
ú = +b
(6)
For instance, they can he chosen as quantiles of the distribution of x
4

---

Under review as a conference paper at ICLR 2021
which is able to accommodate variable number of h and in turn accommodate variable number
of inputs. Based on Theorem 7 in Zaheer et al. (2017). J. 6 could approximate any permutation
invariant continuous function in h. Note that $1 Al is the average embedding, and all examples within the same batch/task share the same 8T i c (see details in Section 4.3)
4.3 TRAINING AND INFERENCE
Figure I shows high level summary of our model graph during training, where 1....., T are training tasks. In each gradient step. we first randomly sample task T; { 1,..... Tm) We then sample two disjoint batches A and B of training features and labels from (X T., yr, The
two batches of features are first transformed using PLFs in (3). We then use (4) or (5 to obtain a distribution embedding, taking the average with respect the sample in batch B. Next,
the
average distribution embedding to make predictions on batch using the Deep Sets formulation in O The of two batches is to reflect the scenario that during inference, the task, with which we classify query set
the support set
(i.e., batch B) to obtain the distribution embedding examples (i.e., batch A). This procedure is similar to the idea of episode-based training introduced in Vinyals et al. (2016). 16). Note that during training, 8T, is identical across examples within the same batch, but could vary (even within the same task) across batches.
Batch A x
Loss (I)
Zi
PLF (3) e.())
Deep Structured Classifier (6)
XP
2
Distribution Embedding (4) or (5)
Batch B
Figure 1:
Block diagram of the model graph during training. We resample batches and and Task I, after each gradient step.
PLFs are able to approximately transform the features into the same parametric family, then the rest of the network can be task-agnostic. Thus, after training u,...., T for each new task S, we learn a set of new PLFs (3) while keeping the weights of other layers fixed, Because PLFs only have a small number of parameters, they be trained on a small support (Xs, ys).
During inference, we first use learned PLFs in (3) to transform features in both the support set and the query set. We then utilize the learned distribution embedding module to obtain 8ş, where the average in (5) and 4 is taken over the whole support set. Finally, the embedding sş and PLF- transformed query set features are used to classify query set examples using (6).
NUMERICAL STUDIES
In this section, we numerically compare the two formulations of DEN, Conditional DEN (4) and Joint DEN (5) (with 2). with range of baseline methods, including Prototypical Net Snell et al. 2017). Relation Net (Sung et al.) (018)) and MAML (Finn et al. 2017) applied to a ReLU- activated deep neural network (DNN). All hyperparameters are chosen based on cross cross-valian coss-validation on training tasks, and are summarized in Appendix B We also compare our method against DNNs trained directly the support set (Direct DNN), whose hyperparameters were chosen based cross-validation on the support set
We will make the code of the experiments of Section 5.1 publicly available upon acceptance of this paper.

---

Under review as a conference paper at ICLR 2021
5.1 GENERATE TRAINING TASKS THROUGH CONTROLLED SIMULATION
To generate training tasks, we adopt a novel approach. We take seven multiclass image classification datasets: CIFAR-10, CIFAR-10U 100 (Krizhevsky 2009), MNIST (LeCun et al., 2010), Fashion M (Xiao et al. 2017), EMNIST (Cohen et al. 2017), Kuzushiji MNIST (KMNIST: Clanuwat et al. 2018) and Street View House Numbers VHN; Netzer et al. 2011). On each dataset, we pick nine equally spaced cutoffs and binarize the labels based on whether the class id below the cutoff or not. This gives rise to nine binary classification tasks for each dataset with positive label proportion in (0.1,0.2, ...., 0.9}. In summary, we collect x 9 = 63 binary classification tasks.
To generate features for DEN, we build 50 image classifiers on each task U, E IU,,..., Tea and take classification scores on the test set as features ØT, E (0, 1)500 The 50 convolutional image classifiers all have the structure Conv k) - MaxP (p) - Conv(f, k) - MaxPool (p) - ..... - (onv(f. k) + (unse(u) + Dense(u) + ... Dense(1), where the number of dense layers d E [1,4), the number of convoluted layers e E [0.3], 3), filters and units f, E [2, 511], kernel and pool sizes k; p E [2, and training epochs e E [1,24] are uniformly sampled. Note that these 50 classifiers range from linear classifiers ReLU-activated deep convolutional neural networks,
with accuracy ranging from below 0.6 to over 0.99.
Finally, augment the training data, we apply sub-sampling during training. In each training step. after selecting a task and two disjoint batches of training examples as discussed in Section 4.33 we randomly pick C 50 classifiers to construct a sub-task that aims to aggregate the classification scores of C' classifiers. Here C' could vary across training steps, and thus the joint distribution Pr would be different in different training steps, helping DEN learn from diverse set of Pr. By increasing the number of datasets, the number of label cutoffs and the number of classifiers, we could effectively generate a large number of training (sub-)tasks and examples for DEN.
5.2 META-LEARNING ON MODEL AGGREGATION
In this section, we study the performance of DEN in aggregating the outputs of an ensemble of classifiers. We also study the impact of fine-tuning PLF on the performance of DEN. As discussed in Section fine-tuning PLF is powerful way to make DEN adapt to wide variety of tasks with different input distributions. However, in the scenario when the distributions of training and target tasks are similar, fine-tuning PL.F can introduce overfitting that outweighs its benefit, especially when the support set is small compared to the number of features.
To introduce similar training and target tasks, we use 5 x 9 = 45 tasks derived from CIFAR- 10, CIFAR- 100, MNIST, Fashion MNIST and EMNIST to train DEN and other meta-learning methods. The model architectures and hyperparameters are presented in Appendix B Note that because the distributions of classifier outputs are similar, we learn a single set of PLIS during training across tasks for simplicity. We then pick four test tasks from SVHN and KMNIST (two from each dataset) of different difficulties. The average AUC across 50 classifiers are 68. 28%. 78.11%, 91.51%, and 87.58%, respectively. Given test task, randomly select 100 sets of C classifiers among the 50 candidate classifiers, which results in 100 aggregation sub-tasks for each of the four test tasks. For each aggregation sub-tasks, we form a support set with 50 labeled examples and a disjoint query set with 8000 examples. "Direct DNN" is trained directly on the 50 support set examples. For DNN with MAML, we fine-tune the last DNN layer for epochs on the support set. For DEN, we either fine-tune the first layer of C PLFs for 10 epochs, or take C' PLFs directly from the last training epoch without fine-tuning, The entire training and fine- mee-ttning process is repeated 5 times. For each test task, we report the average AUC score across 5 x 100 aggregation sub-tasks and its estimated standard error.
In Table I we present the result where we aggregate C' = 25 classifiers. Average and product rules are simple heuristics that make predictions based on the average and product of the 25 classifica- tion outputs, respectively. In Table 2, we show the result where the number of classifiers C' to be aggregated is sampled uniformly from [13, 25] across those 500 trials. To allow baseline methods to take varying number of features, we repeat and add random subset of (25 C) features that all inputs have 25 features. We observe that DEN significantly outperforms other methods in seven out of eight tasks.

---

Under review as a conference paper at ICLR 2021
Table I: Test % AUC (standard error) when aggregating 25 classifiers. Bold fonts indicate the best method at 95% significance level.
Method
Test AUC (%)
Task
Task
Task
Task 4
Average nule
86.68 (0.176)
88.80 (0.048)
97.54 (0.030)
96.27 (0.099)
Product rule
87.74 (0.168)
88.96 (0.046)
98.37 (0.009)
97.37 (0.044)
Direct DNN
85.95 (0.362)
89.53 (0.032)
97.88 (0.048)
97.56 (0.033)
Prototypical
91.06 (0.051)
89.95 (0.029)
98.19 (0.006)
97.51 (0.017)
Relation
83.26 (0.310)
88.71 (0.025)
(0.02)) (0.0233
95.64 (0.079)
DNN + MAML
86.05 (0.161)
88.74 (0.033)
97.42 (0.022)
96.02 (0.068)
Joint DEN w/o Fine-Tuining
91.77 (0.030)
90.26 (0.021)
98.15 (0.006)
98.43 (0.003)
Joint DEN w Fine-Tuining
91.87 (0.029)
90.04 (0.020)
97.55 (0.007)
97.51 (0.006)
Cond. DEN w/o Fine-Tuining
91.76 (0.032)
90.20 (0.021)
98.18 (0.005)
98.41 (0.003)
Cond. DEN w Fine- Fine-nuining
91.80 (0.030)
89.77 (0.021)
97.38 (0.007)
97.23 (0.006)
Table 2: Test % AUC (standard error) when aggregating variable number of classifiers. Bold fonts indicate the best method at 95% significance level.
Method
Test AUC (%)
Task 1
Task 2
Task 3
Task
Average rule
86.05 (0,437)
88.65 (0.080)
97.43 (0.059)
96.09 (0.139)
Product rule
87.06 (0.462)
88.80 (0.072)
98.34 (0.017)
97.27 (0.059)
Direct DNN
83.83 (0.595)
89.11 (0.057)
97.71 (0.050)
97.23 (0.107)
Prototypical
90.02 (0.107)
89.65 (0.041)
97.94 (0.015)
97.57 (0.016)
Relation
80.85 (0.646)
88.53 (0.038)
97.11 (0.039)
95.03 (0.123)
DNN + MAML
85.29 (0.229)
88.51 (0.044)
97.32 (0.036)
95.88 (0.085)
Joint DEN w/o Fine-Tuning
91.21 (0.098)
90.07 (0.029)
98.31 (0.007)
98.18 (0.006)
Joint DEN w Fine-Tuning
91.34 (0.030)
89.74 (0.022)
97.29 (0.030)
97.10 (0.004)
Cond. DEN w/o Fine- Fine-Tuning
91.17 (0.092)
89.95 (0.028)
97.95 (0.008)
98.10 (0.006)
Cond. DEN w Fine-Tuning
91.29 (0.030)
89.83 (0.022)
97.60 (0.025)
97.47 (0.004)
Results in Table 1 and 2 also show that in 15 out of 16 comparisons, DEN without fine-tuning is statistically no worse than DEN with fine-tuning on the PLF layer. This may suggest that fine-tuning the PLF layer is not necessary when the input distribution is similar among tasks.
5.3 META-LEARNING ON REAL DATASETS
Finally, apply DEN seven target tasks from three real datasets: Diamonds, Nomao and Puzzles. We give short description of each dataset below, and list the features Appendix D
With Diamonds data we use five features to classify whether the price of the diamond is above S5325, $2402, or $951. This results three binary classification tasks with positive label proportions of 25%, 50%, and 75%, respectively.
With Nomao data we use seven features to classify whether two business entities are identical. Positive examples account for 71 % of the data.
use six features to classify whether the number of units sold for each puzzle in a six months period is above 93, 45, or 24. The support set (with 155 puzzles) covers the first six months period, whereas the query set (with 367 puzzles) covers the second and third six months periods. The three cutoffs result in positive label proportions of 25%, 49% and 74% in the support set, and 23%, 51 % and 73% in the query set.
With Puzzles date
hhapss//archive.ic..ucii wtt.://.....uu///atttse//aaaaaa http:///wwwww.kgl...////////baaa

---

Under review as a conference paper at ICLR 2021
In this study, use all 63 tasks described in Section 5.1 during training. We train single Joint DEN and a single Conditional Den model for all seven tasks with C < 9, and fine-tune the PLF on 50 support set examples for Diamonds and Nomao tasks, and on 155 support set examples for Puzzles tasks. For other methods, we train a different model for each dataset (three in total), using suitable number of features. In the Nomao data, six of seven features contain missing values. For DEN, we learn missing value embedding via the PLF, whereas for other methods, we add six missing value indicator features, each corresponding to a feature with missing value.
We repeat the whole procedure for 20 times with different random seeds in initialization and selec- tion of training batches and report the average AUC and standard error in Tables 3 and 5 DEN significantly outperforms other methods in six out of seven tasks. Compared with other methods, DEN is especially impressive the Nomao task, in which all features are monotonie, and six of them include missing values. Direct DNN performs well in two of the Puzzles tasks with 155 sup- port set examples, indicating that the benefit of DEN diminishes with large support set. Note that fine-tuning the PLF greatly improved the performance of DEN in these seven tasks, which shows that fine-tuning the PLF is helpful when the training and target tasks have different distributions.
Table 3: Test % AUC (standard error) on Diamonds data. We binarize price based on S951, $2402 and $5325 to generate three tasks. Bold fonts indicate the best method at 95% significance level.
Method
Price > 8951
Price > $2402
Price > $5325
Direct DNN
65.82 (3.077)
90.62 (0.962)
65.37
Prototypical
82.10 (0.177)
91.49 (0. 195)
92.98 (0.182)
Relation
81.09 (1.568)
88.75 (1.605)
88.98 (1.177)
DNN + MAML
79.30 (1.315)
82.83 (1,541)
90.41 (0.970)
Joint DEN
99.58 (0.065)
99.49 (0.048)
99.24 (0.089)
Conditional DEN
97.90 (0.166)
99.67 (0.038)
99.47 (0.057)
Table 4: Test % AUC (standard error) on Nomao data. Bold fonts indicate the best method at 95% significance level.
Direct DNN 69.60 (2.372)
Prototypical
Relation
DNN + MAML
Joint DEN
Cond. DEN 95.21 (0.096)
80.56 (0.559) 52.32 (1.614)
78.92 (2.224)
95.61 (0.071)
Table 5: Test % AUC (standard error) on Puzzles data. We binarize number of units sold based on 24, 45 and 93 to generate three tasks. Bold fonts indicate the best method at 95% significance level.
Method
Sales >> 24
Sales 45
Sales 93
Direct DNN
64.37 (0.827)
78.44 (0.463)
79.83 (0.996)
Prototypical
69.22 (0.220)
73.77 (0.366)
81.39 (0.582)
Relation
62.40 (1.368)
63.73 (1.792)
63.83 (2.065)
DNN + MAML
53.84 (0.519)
54.92 (0.535)
56.57 (0.699)
Joint DEN
64.55 (0.844)
75.68 (0.600)
73,96 (0.279)
Conditional DEN
72.62 (0.512)
78.11 (0.533)
75.50 (0.240)
CONCLUSION
In this paper, we presented a novel meta-learning algorithm that can be applied to settings where both the distribution and number of input features vary across tasks. Most other meta-learning techniques do not readily handle such settings. In numerical studies, we demonstrated that an application of the proposed method on binary classification problems outperforms several meta-learning baselines.
The permutation invariance of the proposed structure is an interesting venue to explore in future work; see (Bloem-Reddy & Teh 2020) for thorough discussion on this topic. Furthermore, can
8

---

Under review as a conference paper at ICLR 2021
the Set Transformer (Lee et al. 2019) in place of the Deep Sets structure in the classifier block of DEN, and we leave the comparison to future work.
Our proposed method can be extended to multi-class classifications by revising the formulations in 4 and (5). DEN can also be combined with optimization based meta-lcarning methods, c.g., MAML,, to further improve its performance. Finally, although convenient and intuitive, the dimensional PLFs are not able to account for the associations between features. Improving the first layer of DEN may lead further performance improvement.
REFERENCES
Luca Bertinetto, João Henriques, Jack Valmadre, Philip Torr, and Andrea Vedaldi. Learning feed-forward one-shot learners. In Advances in Neural Information Processing Systems 29, pp- 523-531. 2016.
Benjamin Bloem-Redoe and Yee Whye Teh. Probabilistic symmetries and invariant neural net- works. Journal of Muchine Learning Reseurch, 21(90):1-61, 2020. URL http://
org/papers Sapers/v21/10
2-22.htm.
Tarin Clanuwat, Mikel Nober-Irizas, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David Ha. Deep learning for classical japanese literature, 2018 8.
Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre Van Schaik. Emnist: Extending mnist to handwritten letters. 2017 International Joint Conference on Neural Networks (IJCNN), 2017.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chupter of the Association for Computational Linguisties: Human Language Technologies, Volume 1 (Long and Short Papers), June 2019.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 22611)212212155) 2011.
Harrison Edwards and Amos Storkey. Towards neural statistician. In International Conference
Learning Representations, 2017.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. Proceedings of the 34th International Conference on Machine Learning. volume 70, pp. 1126-1135, 2017.
Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and S. M. Ali Eslami. Conditional neural processes. In Proceedings of the 35th International Conference on Machine Learning, volume 80, pp. 1704- 1713, 2018
Paul R. Halmos and L. Savage. Application of the radon-nikodym theorem to the theory of sufficient statistics. The Annals of Mathematical Statistics, 20(2):225-241, 1949.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2980-2988, 2017.
Luke B. Hewitt, Maxwell Nye, MAndreea Gane, Tommi Jaakkola, and Joshua B. Tenenbaum. The variational homoencoder: Learning learn high capacity generative models from few examples. In Proceedings of the 34th Conference on Uncertainty in Artificial Intelligence, pp. 978-987, 2018.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference Learning Representations, 2015.

---

Under review as a conference paper at ICLR 2021
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In International Conference on Learning Representations, 2014.
Gregory Koch, Richard Zemel, and Ruslan Salakhutdinov. Siamese neural networks for one-shot image recognition. In Proceedings of the 32nd International Conference on Machine Learning. volume 37, 2015.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online]. 2010.
Available: http://yamnleecn.coom http://aamm.ecuu..c//eebbbmiiii
Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set transformer: A framework for attention-based permutation-invariin neural networks. In Pro- ceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 3744-3753, Long Beach, California, USA, 09-15 Jun 2019. PMLR.
Yanbin Liu, Juho Lee, Minseop Park, Saehoon Kim, Eunho Yang, Sungju Hwang, and Yi Yang. Learning to propagate labels: Transductive propagation network for few-shot learning. In Inter- national Conference Learning Representutions, 2019.
Tsendsuren Munkhdalai and Hong Yu. Meta networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70, pp. 2554-2563, 2017.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Sachin Ravi and Hugo Larochelle. Optimization as model for few-shot learning. In International Conference on Learning Representations, 2017.
Joseph Redmon and Ali Farhadi. Yolo9000: Better, faster, stronger. In 20/7 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6517-6525, 2017.
Andrei A. Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-learning with latent embedding optimization. In International Conference on Learning Representations, 2019.
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta- learning with memory-augmentee neural networks. In Proceedings of The 33rd International Conference Machine Learning, volume 48, pp. 1842-1850, 2016.
Victor Garcia Satorras and Joan Bruna Estrach. Few-shot learning with graph neural networks. In International Conference on Learning Representations, 2018.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems 30, pp. 4077-4087. 2017.
Flood Sung, Yongxin Yang. Li Zhang, Tao Xiang, Philip H.S. Torr, and Timothy M. Hospedales. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE Con- ference Computer Vision and Pattern Recognition (CVPR), June 2018.
Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initial- ization and momentum in deep learning. In Proceedings of the 30th International Conference on Machine Learning, volume 28, pp. 1139-1147, 114,, 2013.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, F. ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems 30, pp. 5998-6008. 2017.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap. koray kavukcuoglu, and Daan Wierstra. Match- ing networks for one shot learning. In Advances Neural Information Processing Systems 29, pp. 3630-3638. 2016.
10

---

Under review as a conference paper at ICLR 2021
Han Xiao, Kashif Rasul, and Roland Vollgraf. Pashion-mmiss a novel image dataset for benchmark- ing machine learning algorithms. CoRR, abs/ 2017.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander Smola. Deep sets. In Advances Neural Information Processing Systems 30, pp. 1991-3401.2017
A APPENDIX: PROOF OF LEMMAS
For simplicity, we assume the joint distribution Pr has probability density ():.)). Define map L x - ++ by
Proof of Lemmal 7
C(O, n) == |L 0), = L(f(x; 0), V))(a ys n)dddy.
where L depends on the loss I., the model and the density Then the problem becomes
Ô arg min L(0, ())
m=
Hence, 0 is of the form (
Proof of Lemma
By independence, the joint probability of {u.}i=1
n
II ([ n) II B(,)) exp X(n)
S(u)) nA(n)
According to Fisher-Neyman factorization theorem (Halmos & Savage E 1==1 S(u:) is sufficient for 17.
1949), the statistic s(U)
B APPENDIX: MODEL STRUCTURES AND HYPERPARAMETERS
In this section, we report model structures and hyperparameters of all models we have considered in S5 We use cross-validation on 5 tasks derived from CIFAR-10, CIFAR-100, MNIST, Fashion MNIST and EMNIST to tune the hyperaprameters. Note that those 45 tasks were treated as training tasks in all experiments presented in Section
For DEN, Prototypical Net and Relation Net, we trained each of them for 200 epochs using Adam optimizer with batch size 256 and the TensorFlow default learning rate, 0.001. Their model-related hyperparameters were chosen based on 5-fold cross- vols-vaaiittin:: in each fold, we trained the model on all one training datasets and evaluated it on the remaining one" feeding with 500 supporting examples; we then selected the model with the highest AUC scores averaged over folds. For DNN with MAML, after training on the 36 tasks, we fine-tuned the last layer of the trained DNN on support set of size 500 of each validation task, then evaluated it the remaining examples of this validation task. For the DNN model directly trained on the support set, its hyperparameters were selected the support of each test task. So could vary from task task. We only report its model structure since there are many test tasks.
Based cross-validation, architectures across all of our experiments. We use m-Dense (u) to represent m consecutive ReLU- activated dense layers with units. With a bit abuse of notation, we use Dense(1) to represent sigmoid-an sigmoid-activated dense layer with 1 unit. We denote by Bat chAvg an average layer over examples batch, and denote by Pairrvg an average layer over pairs of features.
arrived at the following model architectures. We the following
Direct DNN: m-Dense u)) - Dense(1), where and chosen based the specific
test task and can vary from task to task.
'For example, in one of the five folds, we trained the models on 4 x 9 tasks derived from CIFAR-10 10, CIFAR- 100, MNIST and Fashion MNIST datasets, and evaluated the model on the 9 tasks derived from the EMNIST datasct
11

---

Under review as a conference paper at ICLR 2021
Prototypical Net: the embedding model is 2-Dense(32).
Relation Net: the embedding model is 2-Dense(32), and the relation model is 2-
(16se(16) + Dense(1).
DNN + MAML: 4-Dense(64) + (1)s 1200 training batches of size 256 where each batch contains 10 tasks, inner learning rate is 0.01, outer learning rate is 0.005, fine- tune 6 epochs.
Joint DEN: PLF with 10 calibration keypoints, the distribution embedding model is 3-Dense (64) BatchAvg, the Deep Sets classification model is 3-Dense PairÂvg + 3-Dense(64) + Dense(1).
Conditional DEN: PLF with 10 calibration keypoints, the distribution embedding model is 4-Dense (6) - Batch the Deep Sets classification model 4-Dense(64) PairAvg - (6De - Dense(1).
Remark that, during hyperparameter tuning, the maximum number of ReLU aectivcvee dense layers have tried is 12 and the maximum number of units we have tried is 64 for all models. The proposed Joint DEN and Conditional DEN tend to prefer larger models than the rest ones.
C APPENDIX: ADDITIONAL RESULTS FOR MODEL AGGREGATION EXPERIMENTS
In this section, we present results for Joint DEN and Conditional DEN with fine-tuning in model aggregation experiments. For comparison, also report results for Direct DNN and DNN with MAML.
C.1 FIXED NUMBER OF FEATURES
We consider a support set of size 500. For Joint DEN and Conditional DEN, we use cross- cross-validttion on the support set to tune the hyperparameters-tuue -tuiing epochs, initial learning rate and batch size. For DNN with MAML, we also tune the number tuning layers, egg., fine-tune the last one or two dense layers, in addition the above three hyperparameters.
Table 6: Test AUC (standard error) on aggregating 25 classifiers with 500 supporting examples. Bold fonts indicate the best method at 95% significance level.
Method
Test AUC (%)
Task
Task
Task
Task 4
Direct DNN
92.07 (0.038)
90.48 (0.029)
97.62 (0.012)
97.76 (0.008)
DNN MAMI.
91.94 (0.040)
90.22 (0.029)
98.13 (0.008)
98.13 (0.006)
Joint DEN
92.19 (0.030)
89.97 (0.022)
96.89 (0.030)
97.32 (0.004)
Conditional DEN
92.26 (0.030)
89.65 (0.022)
97.58 (0.025)
97.18 (0.004)
C.2 VARIABLE NUMBER OF FEATURES
We repeat the procedures in the above section for the case when the number of features vary.
D APPENDIX: DESCRIPTION OF REAL DATASETS
We use three datasets in our real data analysis.
With Diamonds data, use numeric carat, cut in ordinal scale, color in ordinal scale, clarity in dinal scale, numeric depth and numeric table to classify whether the price of diamond is above $951, above $2402, or above $5325, which corresponds the 25% qunatile, median and 75% quantile in the data. Four the six features, caret, cut, color and clarity are expected to be monotonic, whereas depth and table have unknown monotonicity.
12

---

Under review as a conference paper at ICLR 2021
Table 7: Test % AUC (standard error) aggregating variable number of classifiers with 500 sup- porting examples.
Method
Test AUC (%)
Task
Task
Task
Task 4
Direct DNN
90.79 (0.137)
89.87 (0.063)
97.92 (0.021)
97.66 (0.019)
DNN + MAML
89.63 (0.130)
89.51 (0.044)
97.92 (0.031)
97.72 (0.037)
Joint DEN
91.23 (0.077)
88.75 (0.031)
96.57 (0.010)
96.76 (0.007)
Conditional DEN
91.74 (0.079)
89.88 (0.029)
97.82 (0.038)
97.50 (0.007)
With Nomao data, we use fax trigram similarity score, street number trigram similarity score, phone trigram similarity score, clean name trigram similarity score, geocoder input address trigram similar- ity score, coordinates longitude trigram similarity score and coordinates latitude trigram similarity score classify whether two businesses identical. All of the features outputs from some other models, and are expected be monotonic. Six and seven of the features have missing values. Fax trigram similarity score is missing 97% of the time; phone trigram similarity score is missing 58% the time; street number trigram similarity score is missing 35% of the time; geocoder input address trigram similarity score is missing 0.1% % of the time, and both coordinates longitude trigram similarity score and coordinates latitude trigram similarity score are missing 55% of the time.
With Puzzles data, we has photo (whether the reviews has photo), amazon (whether the reviews were on Amazon), number of times users found the reviews to be helpful, total number of reviews, age of the reviews and number of words in the reviews classify whether the units sold for the respective puzzle is above 24, above 45 or above 93, Most of the features have unknown monotonicity their effect in the label.
13


Reviews:
[{"review_id": "sj3pP7UdjB3", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes Distribution Embedding Network (DEN), a meta-learning model for classification. DEN is designed for the setting where data distribution and the number of features can vary across tasks. It classifies examples based on an embedding of data distribution.\n\nThe network first embeds data X using a piecewise linear function (PLF). They construct distribution embeddings by aggregating all subsets of size r and inputting to a Deep Sets classifier that takes support set distribution embeddings and query datapoints.\n\nOn page 5, the paper says that each task gets its own PLF. How is the PLF trained on meta-test tasks? Do you temporarily split the support set into train and validation sets to minimize the loss? This is a critical detail and should be clearer.\n\nI think this method is more accurately described as a member of the neural process (NP) family. The overall structure is very similar to the conditional NP (CNP) model for classification [1]. Compared to CNP, the novel components are the PLF and their specific r-subset aggregation scheme (4, 5). However, the experiments do not sufficiently show how much each of these components contributes to performance. Rather than ProtoNets, RelationNets, MAML, etc., comparing against more relevant baselines such as PLF+CNP, r-subset + CNP, CNP+finetuning, etc. would have more clearly shown the efficacy of DEN.\n\nFurthermore, many works confirm that using attention instead of mean-pooling increases performance [2,3,4]. Since attention plays a similar role (encoding high-order interactions among items) to the proposed r-subset aggregation scheme, comparing these two aggregation methods would more firmly ground DEN in the context of existing work.\n\n[1] Garnelo, Marta, et al. \"Conditional neural processes.\"\n\n[2] Kim, Hyunjik, et al. \"Attentive neural processes.\"\n\n[3] Lee, Juho, et al. \"Set transformer: A framework for attention-based permutation-invariant neural networks.\"\n\n[4] Le, Tuan Anh, et al. \"Empirical evaluation of neural process objectives.\"\n\nminor\n\n-The proof of Lemma 1 defines a new loss, which is an expectation of the loss over the data distribution. I feel that both the claim and proof are obvious.\n\n-Section 4.1, just above (2): y_T \\in R^n → y_T \\in {0, 1}^n\n\n-Figure 1: link boxes in odd places.", "extra_scores": {}, "extra_text": {}}, {"review_id": "qJ6nidpWqX2", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe authors proposed a method for meta-learning that produces a distributional embedding of the task, and then uses this information to perform few-shot classification. The method decouples feature extraction from feature aggregation & few-shot reasoning within the given task. The authors conduct experiments to show that their method is competitive with a few other methods from the literature on a number of datasets.\n\n\nStrengths:\n* The proposed method is novel.\n* Based on the provided experiments, it appears to perform well.\n\nWeaknesses:\n* The motivation for this method is not very well explained. The authors discuss existing families of meta-learning methods & how their method does not fit into any of these, but it is not clearly explained what advantages theirs offers or where existing methods are lacking. Does the proposed method have any inductive biases or other desirable properties?\n* The experiments are hard to interpret. Prior work on meta-learning has introduced a few standardized datasets (for example, Omniglot, mini-Imagenet, and tiered-Imagenet), and there is a large body of work that has relied on these datasets for evaluation. It would be helpful to benchmark the proposed method on those datasets so that it can be more thoroughly compared to existing work, helping better assess the impact of this method.\n* Reproducibility / experimental details: Overall, I feel that it would be difficult to reproduce this work based on the content of the paper -- do the authors have plans to release the code? Several nontrivial details seem to be omitted -- for example, where do the features that are input to the proposed method come from? Is there any learned embedding / what happens on high dimensional problems?\n\nOverall, I would lean toward rejection based on my current understanding of this paper, but I would be willing to revise my score based on the authors' response & the other reviews.", "extra_scores": {}, "extra_text": {}}, {"review_id": "nRd5bR79WSD", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper tackles the problem of meta-learning, namely learning from labelled datasets across related tasks, aiming for adaptation to unseen tasks, with little data. The focus is made on varying distributions across data features as well as varying numbers of features.\n\nThe proposed architecture (DEN) contains three building blocks:\n1.\tA transformation layer modeled as a piecewise linear function, that balances data distributions across tasks\n2.\tA distribution embedding module that can take variable-length inputs by considering subsets of features, either in a conditional form or a joint one\n3.\tA classification module that aggregates the input features and the distribution embedding, taking the form of a Deep Sets (DS) network.\n\nOn the applicative side, a novel method for task generation is leveraged using binary classifiers. The proposed network is applied to a classifier aggregation task, as well as seven target tasks from real datasets.\n\nStrong points of the paper include: \n1.\tThe writing of the paper is very clear overall.\n2.\tThe paper tackles an important topic, and its focus points are valuable to the literature.\n3.\tThe experiments procedure presented in section 5.1 and 5.2 is rigorous.\n\nSome important explanations on modelization choices are missing from the current version of the paper, including the following ones:\n1.\tAuthors do not mention why the set representation and related topology are adapted to pairs of features and labels.\n2.\tAuthors resort to the DS framework; however, they do not motivate the use of invariant architectures for meta-learning to begin with, and the literature review on invariant networks is missing.\n3.\tThe invariance actually designed here is particularly unclear. I do not think the invariance the authors want to consider is h being permutation invariant in Equation 6, as is hinted at. Invariances should also be investigated in the other steps, for instance Eq. 4, and labels should be particularized.\n4.\tIn section 5.3, the fact that missing values are not handled similarly in DEN and in its competitors is not justified. Could you please clarify?\n\nI recommend a reject for this paper, on the grounds that the architecture design needs further analysis (as explained above).\n\nMinor details: \n1.\tIn Section 4.2, maybe recalling the definition of sufficient statistics would clarify lemma 2.\n2.\tTypo in Eq 4: r instead of M.\n3.\tThe notation g’ in Eq 5 could be misleading.", "extra_scores": {}, "extra_text": {}}, {"review_id": "cwOAXB2Tc2", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "############################################ \n\nSummary  \n \nThis paper provides an interesting model that can be used for a meta-learning situation where both data distribution and the number of features vary across tasks. The proposed model referred to as ‘Distributed Embedding Network (DEN)’ transforms features to belong to the same distribution family, learn distribution embedding with them and process variable-length inputs by using DeepSets. Through experiments on binary classification tasks, this paper shows applying the proposed method on the problems outperforming meta-learning baselines. \n \n############################################ \n\nReasons for score\n\nI vote for rejecting. This paper proposes a novel method that can handle varying data distribution and variable-length input for the meta-learning field. My main concerns are the clarity of the paper including the objective of each module design and that experiments are not enough to demonstrate the effectiveness of the proposed method. I hope the authors address my concerns during the rebuttal period.\n\n############################################\n\nStrong Points \n\n1. This paper proposes a novel method that encodes varying data distribution across tasks for the meta-learning field. \n2. Also, this paper can handle variable-length input across tasks.\n3. This paper demonstrates their potential to be developed by outperforming baselines on binary classification tasks.\n\n############################################\n\nWeak Points \n\n1. The paper said 'we compare ~ with a range of state-of-the-art baseline models' but, as I know, PrototypicalNet (2017), RelationNet(2018), MAML (2017) are not the SOTA baseline models. Also I think the baseline models used in the experiments are not enough strong. Could the authors show more results of Table 1 or 2 performed by recent meta-learning methods?  \n2. I feel the meaning of the same parametric family’ is rather ambiguous. Could the author describe the definition of ‘same parametric family’?\n3. Additionally, on the 3 page of the paper,  'new unseen tasks whose data distribution falls in the same parametric family.' is written. but, I think if unseen tasks such as tasks sampled from out-of-distribution are enough different from training tasks, it is difficult to use the shared parameters which are trained by training seen tasks. \n4. I wonder why the model with equation (2) can learn distribution. Please give me a more description including the role of pair-wise product between features in the equation (2). \n5. In the section 4.1.2, the reason decomposing the joint dist. into smaller pieces to handle variable-length features is unclear for me. Could the author give a more explanation for it?\n6. Even the paper said 'extending to multi-class classification is trivial', but I hope the experiment results on multi-class classification are included in the paper. Could the author show experiments on multi-class classification problems (e.g. 5way, 10way)?\n7. I think that the overall writing of the paper should be improved.\n\n    (1) I feel the introduction section is rather lacking in abstract level explanation of the proposed method. Adding a concept figure is one of options for a better explanation. \n\n    (2) Overall notations are rather complicated for me. Also, some notation looks written wrongly. Notation shapes of z of (5) and the sentence above (5) are different. Are they have different meanings? How about in case z of (4) and (5)? \n\n    (3) Before using DeepSet, it would be better to add a description of DeepSet.\n\n    (4) is \\psi in (6) DeepSet?\n8. The name ‘DEN’ is already used in the paper ‘Lifelong Learning with Dynamically Expandable Networks (ICLR2018)’[1]. I recommend this paper change the abbreviation.\n9. SetTransformer[2] is an advanced set encoding module outperforming DeepSet. I recommend using SetTransformer instead of DeepSet in the future.\n\n[1] Lifelong Learning with Dynamically Expandable Networks, ICLR2018\n\n[2] Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks, ICL2019\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper addresses a meta-learning method which works for cases where both the distribution and the number of features may vary across tasks. The method is referred to as 'distribution embedding network (DEN)' which consists of three building block. While the method seems to be interesting and contains some new ideas, all of reviewers agree that the description for each module in the model is not clear and the architecture design needs further analysis. In addition, experiments are not sufficient to justify the method. Without positive feedback from any of reviewers, I do not have choice but to suggest rejection. 

Decision: Reject