Chameleon: Learning Model Initializations Across Tasks With Different Schemas
Authors: Lukas Brinkmeyer
Rafael Rego Drumond
Randolf Scholz
Josif Grabocka
Lars Schmidt-Thieme
Venue:  (2021)
Keywords: Meta-Learning
Initialization
Few-shot classification

Abstract: Parametric models, and particularly neural networks, require weight initialization as a starting point for gradient-based optimization. Recent work shows that an initial parameter set can be learned from a population of supervised learning tasks that enables a fast convergence for unseen tasks even when only a handful of instances is available (model-agnostic meta-learning). 
Currently, methods for learning model initializations are limited to a population of tasks sharing the same schema, i.e., the same number, order, type, and semantics of predictor and target variables.
In this paper, we address the problem of meta-learning weight initialization across tasks with different schemas, for example, if the number of predictors varies across tasks, while they still share some variables. We propose Chameleon, a model that learns to align different predictor schemas to a common representation. 
In experiments on 23 datasets of the OpenML-CC18 benchmark, we show that Chameleon can successfully learn parameter initializations across tasks with different schemas, presenting, to the best of our knowledge, the first cross-dataset few-shot classification approach for unstructured data.

Under review as a conference paper at ICLR 2021
CHAMELEON: LEARNING MODEL INITIALIZATIONS ACROSS TASKS WITH DIFFERENT S SCHEMAS
Anonymous authors Paper under double-blind review
ABSTRACT
Parametric models, and particularly neural networks, require weight initialization as starting point for gradient-bate based optimization. Recent work shows that an initial parameter set can be learned from a population of supervised learning tasks that enables a fast convergence for unseen tasks even when only a handful of instances is available (model-agnosttic meta-learning). Currently, methods for learning model initializations are limited to a population of tasks sharing the same schema, i.e., the same number, order, type, and semanties of predictor and target variables. In this paper, we address the problem of meta-learning weight initialization across tasks with different schemas, for example, if the number predictors varies across tasks, while they still share some variables. We propose Chameleon, a model that learns to align different predictor schemas to a common representation. In experiments on 23 datasets of the OpenML-CC18 benchmark, we show that Chameleon can successfully learn parameter initializations across tasks with different schemas, presenting, to the best of our knowledge, the first cross-dataset few-shot classification approach for unstructured data.
INTRODUCTION
Humans require only a few examples correctly classify new instances of previously unknown objects. For example, it is sufficient to see handful of images of a specific type of dog before being able to classify dogs of this type consistently. In contrast, deep learning models optimized in classical supervised setup usually require vast number of training examples to match human performance. A striking difference is that a human has already learned to classify countless other objects, while parameters of neural network are typically initialized randomly. Previous approaches improved this starting point for gradient-bar based optimization by choosing more robust random initialization (He et al., 2015) or by starting from pretrained network (Pan & Yang, 2010). Still, models do not learn from only a handful of training examples even when applying these techniques. Moreover, established hyperparameter optimization methods (Schilling et al., 2016) are not capable parameter space. Few-shot
of optimizing the model initialization due to the high-dimensions classification aims correctly classifying unseen instances of novel task with only few labeled training instances given. This is typically accomplished by meta-learning across a set of training tasks, which consist of training and validation examples with given labels for a set of classes. The field has gained immense popularity among researchers after recent meta-learning approaches have shown that it possible learn weight initialization across different tasks, which facilitates a faster convergence speed and thus enables classifying novel classes after seeing only a few instances (Finn al., 2018). However, training a single model across different tasks is only feasible if all tasks share the same schema, meaning that all instances share one set of features in identical order. For that reason, most approaches demonstrate their performance on image data, which can be easily scaled to fixed shape, whereas transforming unstructured data to uniform schema is not trivial.
We want to extend popular approaches to operate invariant schema, i.c., independent of order and shape, making it possible to use meta-learning approaches on unstructured data with varying feature spaces, e.g., learning a model from heart disease data that can accurately classify a few-shot task for diabetes detection that relies similar features. Thus, require achema-invarian encoder that maps heart disease and diabetes data to one feature representation, which then can be used to train a single model via popular meta-learning algorithms like REPTILF. (Nichol et al., 2018b).

---

Under review as a conference paper at ICLR 2021
Original Task Schemaa
Transfomnel Task Schoma
Tosk A
YA
Dask
Chameleon
Model M
yD
Commnn Satern
C
O
Figure 1: Chameleon Pipeline: Chameleon aims to encode tasks with different schemas to shared representa- tion with an uniform feature space, which can then be processed by any classifier. The left block represents tasks the same domain with different schemas, The middle represents the aligned features in fixed schema.
We propose set-wise feature transformation model called CHAMELEON, named after a REPTILE capable of adjusting its colors according to the environment in which it is located. CHAMELEON projects different schemas to a fixed input space while keeping features from different tasks but of the same type or distribution in the same position, illustrated by Figure 1. Our model learns to compute a task-specific reordering matrix that, when multiplied with the original input, aligns the schema of unstructured tasks common representation while behaving invariant to the order of input features.
Our main contributions are as follows: (1) We show how our proposed method CHAMELEOW can learn to align varying feature spaces to a common representation. (2) We propose the first approach to tackle few-shot classification for tasks with different schemas. (3) In experiments on 23 datasets of the OpenML-CC18 benchmark (Bischl et al., 2017) collection, we demonstrate how current meta-learning approaches can successfully learn a model initialization across tasks with different schemas as long as they share some variables with respect to their type or semantics. (4) Although an alignment makes little sense to performed on top of structured data such as images which can be easily rescaled, we demonstrate how CHAMELEON can align latent embeddings of two image datasets generated with different neural networks.
2 RELATED WORK
Our goal is to extend recent few-shot classification approaches that make use of optimization- meta-learning by adding feature alignment component that casts different inputs to a common schema, presenting the first approach working across tasks with different schema. this section, we will discuss various works related to our approach.
Research on transfer learning (Pan Yang, 2010; Sung et al., 2018; Gligic et al., 2020) has shown that training a model on different auxiliary tasks before actually fitting it to the target problem can provide better results if training data is scarce. Motivated by this, few-shot learning approaches try to generalize novel tasks with unseen classes given only a few instances by first meta-learning across a set of training tasks (Duan et al., 2017; Finn et al., 201 7b; Snell et al., 2017). A task T consists of predictor data X,, a target Y, a predefined training/test split loss function C, Typically, an N-way K-shot problem refers to a few-shot learning problem where each task consists of N classes with K training samples per class.
(Xtaain Y,train Xlest Y and
Heterogeneous Transfer Learning tries to tackle a similar problem setting as described in this work. In contrast to regular Transfer Learning, the feature spaces of the auxiliary tasks and the actual task differ and are often non-ov non-overlapping (Day & Khoshgoftaar, 2017). Many approaches require co-occurence data i.e. instances that can found in both datasets (Wu et al., 2019: Qi et al., 2011), rely on jointly optimizing separate models for each dataset to propagate information (Zhao & Hoi, 2010; Yan et al., 2016), or utilize meta-features (Feuz & Cook, 2015). Oftentimes, these approaches operate structured data e.g. images and text with different data distributions for the tasks at hand (Li et al., 2019; He et al., 2019). These datasets can thus be embedded a shared space with standard models such as convolutional neural networks and transformer-bast based language models. However, none of these approaches are capable of training a single encoder that operates across a meta-datase of tasks with different schema for unstructured data.
2

---

Under review as a conference paper at ICLR 2021
Early approaches like (Fe-Fei et al., 2003) already investigated the few-shot learning setting by representing prior knowledge as a probability density function. In recent years, various works proposed new model- meta-learning approaches which rapidly improved the ofate-of-the-ar few-shot learning benchmarks. Most prominently, this includes methods which rely on learning an embedding space for non-parametric metric approaches during inference time (Vinyals al., 2016; Snell et al., 2017), and approaches which utilize an external memory which stores information about previously seen classes (Santoro et al., 2016; Munkhdalai & Yu, 2017). Several more recent meta-learning approaches have been developed which introduce architectures and parameterization techniques specifically suited for few-shot classification (Mishra et al., 2018: 8: Shi et al., 2019; Wang & Chen, 2020) while others try to extract useful meta-features from datasets to improve hyper-parameter optimization (Jomaa et al., 2019).
In contrast, Finn et al. (2017a) showed that an optimization-baseer approach, which solely adapts the learning paradigm can be sufficient for learning across tasks. Model Agnostic Meta-Learning (MAML)) describes model initialization algorithm that is capable of training an arbitrary model f across different tasks. Instead of sequentially training the model one task at time, it uses update steps from different tasks to find common gradient direction that achieves fast convergence. In other words, for each meta-learning update, we would need an initial value for the model parameters 6. Then, we sample batch of tasks T, and for each task 7 E T we find an updated version of 0 using N examples from the task by performing gradient descent with learning rate a as in: 0' t- 0 - a VV (Jo). The final update of ( with step size 3 will be:
t 3
Vo (+(or)
(1)
Finn et al. (2017a) state that MAML does not require learning an update rule (Ravi & Larochelle, 2016), or restricting their model architecture (Santoro et al., 2016). They extended their approach by incorporating a probabilistic component such that for a new task, the model is sampled from a distribution of models to guarantee a higher model diversification for ambiguous tasks (Finn et al., 2018). However, MAML requires to compute recond-order derivatives, resulting in computationally heavy approach. Nichol et al. (2018b) extend upon the first-order approximation given as an ablation by Finn et al. (2018), which numerically approximates Equation (1) by replacing the second derivative with the weights difference, s.t. the update rule used in REPTILE is given by:
t- 0
EO,- -0) 3 IT
1
which means can use the difference between the previous and updated version an approximation of the second-orde ooder derivatives to reduce computational cost. The serial version is presented in Algorithm (1).² All of these approaches rely a fixed schema, i.e. the same set of features with identical alignment across all tasks. However, many similar datasets only share a subset of their features, while oftentimes having a different order or representation e.g. latent embeddings for two different image datasets generated by training two similar architectures. Most current few-shot classification approaches sample tasks from a single dataset by selecting random subset of classes; although it is possible to train a single meta-model on two different image datasets as shown by Munkhdalai & Yu (2017) and Tseng et al. (2020) since the images can be scaled to a fixed size. Further research demonstrates that it is possible to learn single model across different output sizes (Drumond et al., 2020). Recently, a meta-dataset for few-shot classification of image tasks was also published to promote meta-learning across multiple datasets (Triantafillou et al., 2020). Optimizing a single model across various datascts requires a shared feature space. Thus, it is required to align the features which is achieved by simply rescaling all instances in the case of image data which is not trivial for unstructured data. Recent work relies on preprocessing images to a one-dimensional latent embedding with additional deep neural network. The authors Rusu et al. (2019) train Wide Residual Network (Zagoruyko & Komodakis, 2016) on the meta-training data of MinilmageNet (Vinyals et al., 2016) compute latent embeddings of the data which are then used for few-shot classification, demonstrating state-of-the Wet-the-rr art results.
Finding a suitable initialization for deep network has long been a focus of machine learning research. Especially the initialization of Glorot & Bengio (2010) and later He et al. (2015) which emphasize
Note that REPTILE. does not require validation instances during meta-learning.
3

---

Under review as a conference paper at ICLR 2021
the importance of scaled variance that depends on the layer inputs widely used. Similar findings are also reported by Cao et al. (2019). Recently, Dauphin & Schoenholz (2019) showed that is possible to learn suitable initialization by optimizing the norms of the respective weights. So far, none of these methods tried to learn a common initialization across tasks with different schema.
We propose novel feature alignment component named CHAMELEON, which enables ofte-offtheeart methods to learn how to work on top of tasks whose feature vector differ not only in their length but also their concrete alignment. Our model shares resemblance with scaled dot-product attention popularized by (Vaswani et al., 2017):
QKT vdk )
Attention(Q (Q, K V) = softmart
where Q, K and are matrices describing queries, keys and values, and dx is the dimensionality of the keys such that the softmax computes an attention mask which is then multiplied with the values V. In contrast to this, we pretrain the parametrized model CHAMELEON to compute soft permutation matrix which can realign features across tasks with varying schema when multiplied with V instead of computing simple attention mask.
Algorithm 1 REPTILE Nichol et al. (2018b)
Input: Meta-dataset T {XX], Y1, C1). (X1T), Y TT, ((T)), learning rate B Randomly initialize parameters 0 of model f
2: for iteration = 1, do
3: Sample task (,,
4: 8'
+ 0
5: for steps 1,2.... do
6: 0' + 0 V Cr
7: end for
8: 0 - 0 B(0' 8)
9: end for
10: return parameters 0 of model f
3 METHODOLOGY
3.1 PROBLEM SETTING
We describe a classification dataset with vector-shaped predictors (i.e., no images, time series etc.) by a pair (X,Y) E RNF x fo....,Cy. CJ.,, with predictors X and targets Y, where N denotes the number of instances, F the number of predictors and C' the number of classes. Let DF = UNEN NxF {o...., CJN be the space of all such datasets with F predictors and D = UPEN Dr be the space of any such dataset. Let us also denote the space of all predictor matrices with F' predictors by XF UNEN NxF and all predictor matrices by X ;= UFEN X'F. Then dataset 5 (X, Y) E D equipped with a predefined training/test split, i.e. the quadruplet T (Xaain Yurai X rest Y!SE) is called a task. A collection of such tasks T D called a meta- dataset. Similar to splitting a single data set into training and test part, one can split meta-dataset T = Ttrain U Trest The schema task 7 then describes not only the number and order, but also the semantics of predictor variables {Pr, Pr.: ........... Pryp in Xtrain
Consider a meta-dataset of correlated tasks T C D. such that the predictor variables {Pr, PPr. PrF } of any individual task 7 are contained in common set of predictor variables {Pr. (Pi.P2.-. P2..... Px ). Methods like REPTILE and MAML try to find the best initialization for a specific model, in this work referred to as 9. to operate has to share the same schema of common size K, where similar features shared across tasks are in the same position. A feature-order invariant encoder is needed map the data representation X, of tasks with varying input schema and feature length F,, to a shared latent representation X, with fixed feature length K:
set T of similar tasks. However, every task
X - Xx. X, R NXF. X, E RRNNK
4

---

Under review as a conference paper at ICLR 2021
Chameleon
Input X
Fr
F 8
11(xs)
Fr K
Output X-I((x+
Fr
Px
ConviD
(N*8*1)
CoaviD
(Nx16x
Cony
(N*K*1)
Soffmax
N
Figure The Chameleon Architecture: represents number of samples in T, Fr the number features in T. and is the number of features in the desired feature space. "Conv(a x x c)" is convolution operation with a input channels, filter size of b and kernel length c.
where N represents the number of instances in X,, Fr is the number of features of task which varies across tasks, and K is the size of the desired feature space. By combining this encoder with model ÿ that works on fixed input size K and outputs the predicted target e.g. binary classification, it is possible apply the REPTILE algorithm to learn initialization (jmit across tasks with different schema. The optimization objective then becomes the meta-los for the combined network f = I enc over set of tasks
argmin ginz
EET L.r (y ((x:t,,())
s.t.
05 = ((r)
((10(x77 (xtrain Virain Ir. S; gimit
(5)
where ginit the set of initial weights for the combined network consisting of enc with parameters Penc and model ý with parameters fg. and gfd) are the updated weights after applying the learning procedure A for iterations the task as defined in Algorithm for the inner updates of REPTILE. is important to mention that learning one weight parameterization across any heterogeneous set of tasks is extremely difficult since it is most likely impossible find one initialization for two tasks with vastly different number and type of features. By contrast, if two tasks share similar features, one can align the similar features to a common representation so that model directly learn across different tasks by transforming the tasks as illustrated in Figure 1.
3.2 CHAMELEON
Consider set of tasks where right stochastic matrix II, exists for each task that reorders predictor data X, into X, having the same schema for every task 7 T:
X, = X, [T, where
(6)
Φ111
ø1.K
F1,F
T1,1
T1,K
TN,i
UN.K
xN.I
UN...
TF..1
TF,KK
II.
Every Im.n represents the feature n of sample m. Every Tm.n represent how much of feature m (from samples in X,) should be shifted to position in the adapted input X,, Finally, every Fm... represent the new feature of sample m in X, with the adpated shape and size. order to achieve the same X, when permuting two features of a task X,, we must simply permute the corresponding rows in II, to achieve the same X-. Since II, is right stochastic matrix, the summation for every row IT is set be equal to in Σ; mj.a 1, so that each value IT, simply states how much feature is shifted to corresponding position. For example: Consider that task has features [apples, bananas, melons] and task features [lemons, bananas, apples). Both can be transformed to the same representation lapples, lemons, bananas, melons] by replacing missing features with zeros and reordering them. This transformation must have the same result for a and b independent of their feature order. In real life scenario, features might come with different names sometimes their similarity not clear to the human eye. Note that classic autoencoder is not capable of this as is not invariant to the order of the features. Our proposed component, denoted by t. takes task as
5

---

Under review as a conference paper at ICLR 2021
input and outputs the corresponding reordering matrix:
+ (X,, Benc) fi,
The function c is neural network parameterized by Oene- It consists of three ID-convolutions, where the last one is the output layer that estimates the alignment matrix via a softmax activation. The input is first transposed size [F3- x N] (where N is the number of samples) i.c., each feature is represented by a vector of instances. Each convolution has kernel length 1 (as the order of instances arbitrary and thus needs to be permutation invariant) and channel output size of 8, 16, and lastly K. The result is reordering matrix displaying the relation of every original feature to each of the K features the target space, Each of these vectors passes through softmax layer, computing the ratio of features in X, shifted to each position of X, Finally, the reordering matrix can multiplied with the input compute the aligned task defined in Equation (6). By using kernel length of in combination with the final matrix multiplication, the full architecture becomes permutation invariant in the feature dimension. Column-wise permuting the features of an input task leads to the corresponding row-wise permutation of the reordering matrix. Thus, multiplying both matrices results in the same aligned output independent of permutation. The overall architecture can be seen in Figure 2. The encoder necessary for training across tasks with different predictor vectors with REPTILE by optimizing Equation (5) then given as:
enc: X - Xx 15,.4(X, 4(X,, Oanc) = X, f
(8)
3.3 REORDERING TRAINING
Only joint-training the network ý o ene as described above, will not teach CHAMELEON denoted by & how to reorder the features to shared representation. That is why it is necessary to train & specifically with the objective reordering features (reordering training). optimize & align novel tasks by training exists such that maps T to the shared representation. In other words, we require a meta-datase that contains not only set of similar tasks T with different schema, but also the position for each feature in the shared representation given by a permutation matrix. If IT, is known beforehand for each E T, optimizing Chameleon becomes simple supervised classification task based on predicting the new position of each feature in T. Thus, we can minimize the expected reordering loss over the meta-dataset:
order to do so, we set of tasks for which the reordering matrix II,
Penc
argmin E (II,
Oone
where La is the softmax cross-entropy loss, 11, is the ground-truth (one-hot encoding of the new position for each variable), and ÎI, is the prediction. This training procedure can be seen in Algo- rithm (2). The trained CHAMELEON model can then be used to compute the II, for any unseen task T TET.
Algorithm 2 Reordering Training
Input: Meta-dataset T = In), (XIT), [[rr) latent dimension K, learning rate
1: Randomly initialize parameters dene of the CHAMELEON model
2: for training iteration = 1, 2, do
3: randomly sample 7 ~ T
4: Oone < Aenc - 7 V. (T,, (Xx,00enc)
5: end for
6: return Trained parameters lone of the CHAMELEON model
After this training procedure, use the learned weights initialization for & before optimizing ÿ o enc with REPTILE without further using L.p. Experiments show that this procedure improves our results significantly compared to only optimizing the joint meta-loss.
Training the CHAMELEON component to reorder similar tasks to a shared representation not only requires meta-dataset but one where the true reordering matrix II, is provided for every task. In application, this means manually matching similar features different training tasks so that novel tasks can be matched automatically. However, it is possible sample broad number of tasks from

---

Under review as a conference paper at ICLR 2021
Accuracy increase in neg. log scale
(a) Split
(b) No-Split
--02
-0.2
-044
-0.4
-0.6
-08
-0.8
(Reptlie Pad
(Reptile Padded)
(Proposed)
-1.0
grenc (Propesed)
-1.0
Oracle
tic
phoneme
analcat
jungle
electricity
wine
wil
vowel
lipd
wall
letter
Magic
pendigits gits
diabetes
resture
e
Sealon
segment
mffeat
wabc
tic analcat
electricity
jungle e
wine
wall
vowe!
blood
letter
phoneme e
pendigits
Gesture
vehic
Magic
banknote
wit
lipa
diabefes
abalone
mfeat
segment wdbc
Figure 3: Accuracy improvement for each method over Glorot initialization (Glorot & Bengio, 2010): The
difference is plotted in negative log scale to account for the varying performance scales across the different datasets (higher points better; A value of -1 is equivalent to the Glorot initialization). The graph (a) represents Split experiments while (b) depicts the No-Split experiments. Notice that the oracle has been omitted from the Split experiments since there is no true feature alignment for unseen features. The dataset axis is sorted by the performance of REPTILE on the base model to improve readability. All results are averaged over 5 runs.
single dataset by sampling smaller sub-tasks from it, selecting a random subset of features in arbitrary order for N random instances. Thus, is not necessary to manually match the features since these sub-tasks share the same II, apart from the respective permutation of the rows as mentioned above.
4 EXPERIMENTAL RESULTS
Baseline and Setup In order to evaluate the proposed method, we investigate the combined model ÿ o enc with the initialization for enc obtained by pretraining CHAMELEON as defined in Equation before using REPTILE to jointly optimize ý enc. We compare the performance with an initialization obtained by running REPTILE on the base model ý by training on tasks padded to a fixed size K ý is not schema invariant. Both initializations are then compared to the performance of model with random Glorot initialization (Glorot & Bengio, 2010) (referred to as Random). In all of our experiments,
measure the performance of a model and its initialization by evaluating the validation data of a task after performing three update steps on the respective training data. All experiments are conducted in two variants: In Split experiments, test tasks contain novel features in addition to features seen during meta-training. In contrast, test tasks in No-Split experiments only consist of features seen during meta-training. While the Split experiments evaluate the performance of the model when faced with novel features during meta-testing, the No-Splir experiments can be used to compare against a perfect alignment by repeating the baseline experiment with tasks that are already aligned (referred to as Oracle). A detailed description of the utilized models is found in Appendix B.
Deta-Datasees For our main experiments, utilize single dataset meta-dataset by sampling the training and test tasks from it. This allows us to evaluate our method on different domains without matching related datasets since fi, is naturally given for a subset of permuted features. Novel features can also be introduced during testing by splitting not only the instances but also the features of a dataset in train and test partition (Splir). Training tasks are then sampled by selecting a random subset of the training features arbitrary order for N instances. Stratified sampling guarantees that test tasks contain both features from train and test while sampling the instances from the test set only. For all experiments, 75% of the instances are used for reordering training of CIIAMELEON and joint-training of the full architecture, and 25% for sampling test tasks. For Split experiments, we further impose train-ter test split on the features (20% the features are restricted to the test split). Our work is built on top of REPTILE (Nichol et al., 2018b) but can be used in conjunction with any model-agnostic meta-learning method. We opted to use REPTILE since it does not require second-order derivatives, and the code is publicly available (Nichol et al., 201 8a) while also being easy to adapt our problem.
7

---

Under review as a conference paper at ICLR 2021
2
5
4
2 Random Reptile Padded 5f Untrain
o enc Proposed enc Frozen
y Random Reptile Padded
Oracle
end Proposed y ene Frezen
y o ene Untrain
Figure 4: Critical Difference Diagram for Split (Left) and No-Splir (Right) showing results of Wilcoxon signed-rank test with Holm's alpha correction and 5% significance level. Models are ranked by their performance and a thicker horizontal line indicates pairs that are not statistically different.
Main Results We evaluate our approach using the OpenML-CC18 benchmark (Bischl et al., 2017) from which we selected 23 datasets for few-shot classification. The details of all datasets utilized in this work are summarized in Appendix B. The results in Figure 3 display the model performance after performing three update steps on a novel test task to illustrate the faster convergence. The graph shows a clear performance lift when using the proposed architecture after pretraining it to reorder tasks. This demonstrates to the best of our knowledge the first few-shot classification approach, which successfully learns across tasks with varying schemas (contribution 2). Furthermore, in the No-Split results one can see that the performance of the proposed method approaches the Oracle performance, which suggests ideal feature alignment. When adding novel features during test time (Split) CHAMELEON is still able to outperform the other setups although with a lower margin.
Ablations We visualize the result of pretraining CHAMELEON on the Wine dataset (from OpenML- CC18) in Figure 6 to show that the proposed model is capable of learning the correct alignment between tasks. One can see that the component manages learn the true feature position in almost all cases. Moreover, this illustration does also show that CHAMELEON can be used to compute the similarity between different features by indicating which pairs are confused most often. For example, features two and four are showing a strong correlation, which is very plausible since they depict the free sulfur dioxide and total sulfur dioxide level of the wine. This demonstrates that our proposed architecture is able to learn an alignment between different feature spaces (contribution 1).
Furthermore, we repeat the experiments on the OpenML-CC18 benchmark in two ablation studies to measure the impact of joint-training and the proposed reordering training (Algorithm 2). First, we do not train CHAMELEOW with Equation 9, but only jointly train ý o enc with REPTILE to evaluate the influence of adding additional parameters to the network without pretraining it. Secondly, we use REPTILE only to update the initialization for the parameters of 1 while freezing the pretrained parameters of enc in order to assess the effect of joint-training both network components. These two variants are referred to as Untrain and Frozen. We compare these ablations to our approach by conducting Wilcoxon signed-rank test (Wilcoxon, 1992) with Holm's 's alpha correction (Holm, 1979). The results are displayed in the form of a critical difference diagram (Demšar, 2006; Ismail Fawaz al., 2019) presented in Figure 4. The diagram shows the ranked performance of each model and whether they are statistically different. The results confirm that our approach leads to statistically significant improvements over the random and REPTILE baselines when pretraining CHAMELEON. Similarly, our approach is also significantly better than jointly training the full architecture without pretraining CHAMELEON (UNTRAIN), confirming that the improvements do not stem from the increased model capacity. Finally, comparing the results to the FROZEN model shows improvements that are not significant, indicating that a near- near-optimal alignment was already found during pretraining. A detailed overview for all experimental results is given in Appendix C.
Latent Embeddings Experiments Learning align features is only feasible for unstructured data since this approach would not preserve any structure. However, it is a widespread practice among few-shot classification methods, and computer vision approaches in general, to use a pretrained model to embed image data into a latent space before applying further operations. We can use CHAMELEON to align the latent embeddings of image datasets that are generated with different networks. Thus it is possible to use latent embeddings for meta-training while evaluating on novel tasks that are not yet embedded in case the embedding network is not available, or the complexity of different datasets requires models with different capacities to extract useful features. We conduct an additional experiment for which we combine two similar image datasets, namely EMNIST-Digin and EMNIST-Letters (Cohen et al., 2017). Similar the work of Rusu al. (2019), we train one neural network on each dataset in order to generate similar latent embeddings with different schema, namely 32 and 64 latent features. Afterward, can sample training tasks from embedding while
8

---

Under review as a conference paper at ICLR 2021
0.36
1.0
Accuracy after 3 update step1
0.34
Feature in aligned task
10
0.8
0.32
Percent shifted
0.30
0.28
0.26
random
0.24
y-enc
0.22
y-enc (Ablation: Untrain)
y-enc (Ablation: Frozen]
Feature in original dataset
10
0
4000 8000 12000 16000 20000 Meta-epochs
Figure 6: Heat map of the feature shifts for the Wine data computed with CHAMELEON af- ter reordering-trainee twelve features of the original dataset in the correct order and the y-axis shows which position these fea- tures shifted when presented in permuted subsct.
Figure 5: Latent embedding results. Meta test ac- curacy on the EMNIST-iigis data set while training EMNIST-Letters. Each point represents the curacy on 1600 tasks after performing three update steps on the respective training data. Results are averaged over 5 runs.
The x-axis represents
evaluating tasks sampled from the other one. In the combined experiments, the full training is performed on the EMNIST-Letters dataset, while EMNIST-Digits used for testing. Splitting the features is not necessary the train, and test features are coming from different datasets. The results of this experiment are displayed in Figure 5. It shows the accuracy of EMNIST-Die Digitt averaged across 5 runs with 1,600 generated tasks per run during the REPTILE training EMNIST-Letters for the different model variants. Each test task is evaluated by performing 3 update steps on the training samples and measuring the accuracy of its validation data afterward. One can see that proposed approach reports a significantly higher accuracy than the REPTILE baseline after performing three update steps on task (contribution 4). Thus, showing that CHAMELEON is able transfer knowledge from one dataset to another. Moreover, simply adding CHAMELEON without pretraining it to reorder tasks (Untrain) does not lead to any improvement. This might be sparked by using a CHAMELEON component that has a much lower number of parameters than the base network. Only by adding the reordering-trainee
the model manages to converge to a suitable initialization. In contrast to our experiments on the OpenML datasets, freezing the weights of CHAMELEON after pretraining also fails to give an improvement, suggesting that the pretraining did not manage to capture the ideal alignment, but enables learning during joint-training. Our code is available at BLIND-REVIEW.
CONCLUSION
In this paper, we presented, to the best of our knowledge, the first approach to tackle few-shot classification for unstructured tasks with different schema. Our model component CHAMELEON capable of embedding tasks common representation by computing matrix that can reorder the features. For this, we propose novel pretraining framework that is shown to learn useful permutations across tasks in supervised fashion without requiring actual labels. In experiments on 23 datasets of the OpenML-CC18 benchmark, our method shows significant improvements even when presented with features not seen during training. Furthermore, by aligning different latent embeddings we demonstrate how a single meta-model can be used to learn across multiple image datasets each embedded with distinct network.
REFERENCES
Bernd Bischl, Giuseppe Casalicchio, Matthias Feurer, Frank Hutter, Michel Lang. Rafacl G Manto- vani, Jan N van Rijn, and Joaquin Vanschoren. Openml benchmarking suites and the openml100, 2017.

---

Under review as a conference paper at ICLR 2021
Weipeng Cao, Muhammed JA Patwary, Pengfei Yang, Xizhao Wang, and Zhong Ming. An initial study on the relationship between meta features dataset and the initialization of nnrw. In 2019 International Joint Conference Neural Networks (IJCNN), pp. 1-8. IEEE, 2019.
Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre Van Schaik. Emnist: Extending mnist to handwritten letters, 2017.
Yann N Dauphin and Samuel Schoenholz. Metainit:
Initializing learning by learning to initialize. In Advances in Neural Information Processing Systems, pp.
2019.
Oscar Day and Taghi M Khoshgoftaar. survey on heterogeneous transfer learning. Journal of Big Data, 4(1)229, 2017.
Janez Demšar. Statistical comparisons of classifiers over multiple data sets. Journal of Machine learning research, 7(Ja 2006.
Rafael Rego Drumond, Lukas Brinkmeyer, Josif Grabocka, and Lars Schmidt-Thiens Thieme. Hidra: Head initialization across dynamic targets for robust architectures. In Proceedings of the 2020 SIAM International Conference on Data Mining, pp. 397-405. SIAM, 2020.
Yan Duan, Marcin Andrychowicz. Bradly Stadie, OpenAl Jonathan Ho, Jonas Schneider, Ilya Sutskever, Pieter Abbeel, and Wojeiech Zaremba. One-shot imitation learning. In Advances in neural information processing systems, pp. 1087-1098, 2017.
Li Fe-Fei et al. A bayesian approach to unsupervised one-shot learning of object categories. In Proceedings Ninth IEEE International Conference on Computer Vision, pp. 1134 114--144 1141. IEEE, 2003.
Kyle D Feuz and Diane J Cook. Transfer learning across feature-rich heterogeneous feature spaces via feature-space remapping (fsr). ACM Transactions on Intelligent Systems and Technology (TIST), (1):1-27, 2015.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1226-1135. JMLR. org, 2017a.
Chelsea Finn, Tianhe Yu, Tianhao Zhang, Pieter Abbeel, and Sergey Levine. One-shot visual imitation learning via meta-learning arXiv preprint arXiv: 04905, 2017b.
Chelsea Finn, Kelvin Xu, and Sergey Levine. Probabilistic model-agnostic meta-learning. In Advances in Neural Information Processing Systems, pp. 9537-9548, 2018.
Luka Gligic, Andrey Kormilitzin, Paul Goldberg, and Alejo Nevado-Holgado, Named entity recog- nition electronic health records using transfer learning bootstrapped neural networks. Neural Networks, 121:132-132 2020.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256, 2010.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034, 2015.
Xin He, Yushi Chen, and Pedram Ghamisi. Heterogeneous transfer learning for hyperspectral image classification based on convolutional neural network. IEEE Transactions on Geoscience and Remote Sensing, 58(5):3246-3222 -263,, 2019.
Sture Holm. simple sequentially rejective multiple test procedure. Scandinavian journal of statistics, pp. 65-70, 1979.
Hassan Ismail Fawaz, Germain Forestier, Jonathan Weber, Lhassane Idoumghar, and Pierre-Alas Alain Muller. Deep learning for time series classification: a review. Data Mining and Knowledge Discovery, 33(4):117-963, 2019.
10

---

Under review as a conference paper at ICLR 2021
Hadi S Jomaa, Josif Grabocka, and Lars Schmidt-Thiee Thieme. Dataset2vec: Learning dataset meta- features. arXiv preprint arXiv:1905.1106: 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.600 (ee.i141.66660 29800, 2014.
Haoliang Li, Sinno Jialin Pan, Renjie Wan, and Alex C Kot. Heterogeneous transfer learning via deep matrix completion with adversarial kernel embedding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp.
2019.
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta- learner. In International Conference on Learning Representations, 2018.
Tsendsuren Munkhdalai and Hong Yu. Meta networks. In Proceedings of the 34th International Conference on Machine Learning-Volume Volume, pp. 2554-2563, 2017.
Alex Nichol, Joshua Achiam, and John Schulman. Supervised reptile. https///gitubb.compppnnil supervisedrreptile, 2018a.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. CoRR, abs/1 tass/803.02999, 2018b. URL http://arxiv.org/abb// 005p:///...rrraaa///00009999999
Sinno Jialin Pan and Qiang Yang. survey on transfer learning. IEEE Transactions on knowledge and data engineering, 22(10):1345-133 2010.
Guo-Jun Qi, Charu Aggarwal, and Thomas Huang. Towards semantic knowledge propagation from text corpus to web images. In Proceedings of the 20th international conference on World wide web, pp. 297-306, 2011.
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. 2016.
Andrei A. Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-learning with latent embedding optimization. International Conference on Learning Representutions, 2019. URL
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta- learning with neemory-augmente neural networks. In International conference on machine learning, pp. 1842-18551 2016.
Nicolas Schilling, Martin Wistuba, and Lars Schmidt- Tehmie Scalable hyperparameter optimization with products of gaussian process experts. In Join! European conference on machine learning and knowledge discovery in databases, pp- 33-48. Springer, 2016.
Jing Shi, Jiaming Xu, Yiqun Yao, and Bo Xu. Concept learning through deep reinforcement learning with mnemory-augmentee neural networks. Neural Networks, 110:47 - 54, 2019. ISSN 0893-6080. doi: https://doi.org// www.:///..cmmee2201....... article/pi/50083360000 83031137
URL http://wwww.scinneciietttttcc/ce
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems, pp. 4077-4087, 2017.
Flood Sung, d Li Yang, Yongxin an Zhang, Tao Xiang, Philip HS Torr, and Timothy M. Hospedales. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1199-1208, 2018
Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evei, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, and Hugo Larochelle. Meta- dataset: A dataset of datasets for learning to learn from few examples. In International Conference on Learning Representations, 2020. URL
Hung- Yu Tseng, Hsin- Ying Lee, Jia-Bin Huang, and Ming- -suan Yang. Cross-domain few-shot classification via learned feature- wise transformation. In International Conference on Learning Representations, 2020. URI http:://wpppneeeeeeee/e//nnnn//s

---

Under review as a conference paper at ICLR 2021
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, al. Matching networks for one shot learning. In Advances in neural information processing systems, pp. 3630-3638, 2016.
Qian Wang and Ke Chen. Multi-label zero-shot human action recognition via joint latent ranking embedding. Neural Networks, 122:1 23, 2020. ISSN 0893-6080. doi: http:://dii.or//10..o neunet.2019.09.029, URL
19.
Frank Wilcoxon. Individual comparisons by ranking methods. In Breakthroughs in statistics, pp- 196 202. Springer, 1992.
Hanrui Wu, Yuguang Yan, Yuzhong Ye, Huaqing Min, Michael K Ng, and Qingyao Wu. Online heterogeneous transfer learning by knowledge transition. ACM Transactions on Intelligent Systems and Technology (TIST), 10(3): 2019.
Yuguang Yan, Qingyao Wu, Mingkui Tan, and Huaqing Min. Online heterogeneous transfer learning by weighted offline and online classifiers. European Conference on Computer Vision, pp- 467-474. Springer, 2016.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint 175.0:1605.07146 2016.
Peilin Zhao and Steven CH Hoi. Otl: framework of online transfer learning. In Proceedings of the 27th International Conference International Conference on Machine Learning, pp. 1231-1238, 2010.
12

---

Under review as a conference paper at ICLR 2021
A APPENDIX INNER TRAINING
We visualize the inner training for one of the experiments in Figure 7. It shows two exemplary snapshots the inner loss when training on a sampled task with the current initialization ( before meta-learning and after 20,000 meta-epochs. It is compared to the test loss of the model when it is trained on the same task starting with the random initialization. For this experiment, models were trained until convergence, Note that both losses are not identical in meta-epoch 0 because the CHAMELEON component already pretrained. The snapshots show the expected REPTILE behavior, namely faster convergence when using the currently learned initialization compared to random one.
Meta-epoch 0
Meta-epoch 20,000
2.0
2.0
Chameleon
Scratch
1.5
1.5
Loss
1.0
1.0
0.5
0.5
100
200
300
0
100
200
300
Epochs
Epochs
Figure 7: Snapshots visualizing the inner training. Validation eossopy task sampled wal---rboo-avaae navigation data set during inner training starting from the current initialization (blue) and random initialization (red).
B APPENDIX EXPERIMENTAL DETAILS
The features of each dataset are normalized between O and 1. The Split experiments are limited to the 21 datasets which have more than four features in order perform feature split. We sample 10 training and 10 validation instances per label for new task, and 16 tasks per meta-batch. The number of classes in task is given by the number of classes of the respective dataset, as shown in Table 1. During the reordering traiiingtraiingg phase and the inner updates of reptile, specified in line 6 of Algorithm (1), we use ADAM optimizer (Kingma & Ba, 2014) with an initial learning rate of 0.0001 and 0.001 respectively. The meta-updates of REPTILE E are carried out with learning rate B of 0.01. The reordering-traintee phase is run for 4000 epochs. All results reported in this work are averaged over 5 runs.
OpenML-CC18 All experiments on the OpenML-CC16 benchmark are conducted with the same model architecture. The base model a feed-forward neural network with two dense hidden layers that have 16 neurons each. CHAMELEON consists of two ID-convolutions with 8 and 16 filters respectively and final convolution that maps the task the feature-length K, as shown in Figure We selected dataasets that have to 33 features and minimum number of 90 instances per class. limited the number of features and model capacity because this work seeks to establish a proof of concept for learning across data with different schemas. In contrast, very high-dimensional data would require tuning a more complex CHAMELEON architecture. The details for each dataset are summarized in Appendix 1. When sampling task in Split, sample between 40% and 60% of the respective training features. For test tasks in Split experiments 20% the features are sampled from the set of test features to evaluate performance similar tasks with partially novel features. For each
13

---

Under review as a conference paper at ICLR 2021
experimental run, the different variants tested the same data split, and sample 1600 test tasks beforehand, while the training tasks are randomly sampled each epoch. All experiments are repeated five times with different instance and, in the case of Split, different feature splits, and the results are averaged.
Latent Embeddings Both networks used for generating the latent embeddings consist of two convolutional and two dense hidden layers with 64 neurons each, but the number of neurons in the output layer is 32 for EMNIST- EMNIST-Digits and 64 for EMNIST-Letters For these experiments, the CHAMELEON component still has two convolutional layers with 8 and 16 filters, while we use larger base network with two ferw-forward layers with 64 neurons each. All experimental results are averaged over five runs.
Dataset
Instances
Features
Classes
Full Name phoneme cme vowel analcatdata-dmfft
phonem
5404
2
eme
1473
24
3
vowel
990
27
11
analcat
797
21
6
tic
958
27
2
tic-tac-toe
banknote
1372
2
banknote-authentiiation
wdbe
569
30
2
wdbcc
diabetes
768
8
2
diabetes
segment
2310
16
7
segment
Magic
19020
10
2
MagicTelescope
blood
748
4
2
blood-taasfusion-sevvee-cccm
wall
5456
24
4
wall-robot-navigation
wilt
4839
5
2
wilt pendigits
pendigits
10992
16
10
Gesture
9873
32
5
GesturePhaaSSgmeenttioPPPrroee
abalone
4177
10
3
abalone
jungle
44819
6
3
letter
20000
16
26
letter ilpd wine-quality
ilpd
583
11
2
wine
6497
11
5
mfeat
2000
6
10
mfeat-mmorpollggicll
electric
45312
14
2
electricity
vehicle
846
18
4
vehicle
Embedded
Datasets
EDigits 280,000
32*
10
EMNIST-Digits EMNIST-Letters
ELetter
145,600
64*
28
Table 1: Information for the 23 OpenMI-CCC18 dataset used in this paper. * These datasets were embedded using embedding neural network (see Apendix B).
C APPENDIX TABLES WITH EXPERIMENTS RESULTS
The following tables show the detailed results of our experiments on the OpenML-CC18 datasets for Split and NoSplit settings. The tables contain the loss and accuracy for the the base model trained from a random initialization and with REPTILE, and our proposed model ÿ enc with the additional ablation studies Untrain and Frozen:
14

---

Under review as a conference paper at ICLR 2021
Loss
Dataset
Random
g (Reptile Padded)
y ene (Untrain)
g enc (Proposed)
o ene (Prozen)
(Oracle)
segmen
2.157
0.003
1.409 0.020
1.203 0.056
0.928
0.022
0.901 0.030
0.940 0.030
jungle
1,324
:: 0.004
1,079 1 0.002
1,086 1: 0.002
1,081 = 0.002
1,077 1: 0.002
1.023 1: 0.003
wine
1.851
0.005
1.580 0.003
1.567 0.002
1,506 0.006
1,513 0.012
1.512 # 0.009
wilt
0.848
0.005
0.631 0.002
0.653 0.005
0.555 0.008
0.549 :: 0.007
0.541 1: 0.004
cmc
1.327 #: 0.003
1 1,086 +: 0.003
1.057 #: 0.007
1.039 0.002
1.042 0.003
1.035 :: 0.007
electr
0.869
0.004
0.686 0.004
0.683 0.002
0.639 0.007
0.641 0.007
0.655 0.008
letter
3.426 =: 0.001
3.150 +: 0.020
3.033 a: 0,017
2.909 = 0.024
2,689 a: 0.031
2.377 a: 0.031
phonem
0.858 0.002
0.665 0.005
0.684 0.004
0.665 0.005
0.668 0.004
0.577 + 0.005
vehiel
1.624 0.004
1,310 0.020
1.227 0.008
1.214 0.033
1.199 0.037
1.063 1: 0.012
mfeut
2,535 = 0.005
1.681 + 0.031
1 1,86- # 0.036
1,370 0.027
1.405 0.026
1.359 #: 0.049
ilpd
0.831 0.006
0.626 0.004
0.615 0.003
0.603 0.004
0,611 0.003
0.605 0.006
Gestur
1,809 =: 0.002
1 1,499 0.006
1.437 = 0,006
1,419 0.002
1,421 0.004
1.398 a: 0.006
MagicT
0.853 0.002
0.649 0.003
0,652 0.008
0.604 0.007
0.603 + 0.004
0.590 + 0.007
tic
0,871 0.002
0.698 0.001
0.694 0.001
0.690 0.000
0.690 0.001
0.610 a 0.004
bankno
0.840 = 0.009
0.639 + 0.004
0.654 = 0.006
0.616 0.001
0.621 + 0.002
0.569 # 0.003
diabet
0.851 0.003
0.623 0.002
0.638 0.004
0.605 0.002
0.598 0.003
0.600 0.004
wdbc
0.823 = 0.010
0.311 + 0.026
0.221 = 0,014
0.158 0.007
0,194 0.014
0.197 + 0.013
blood
0.845 0.004
0.681 0.003
0.688 0,002
0.660 + 0.003
0.659 + 0.002
0.647 + 0.001
vowel
2.641 0.003
2.315 0.015
1.912 0.016
1.821 0.023
1.843 0.021
1.671 + 0.029
pendig
2,545 = 0.004
2.189 + 0.006
2.169 # 0.010
2.107 = 0.020
2,099 + 0.021
1.088 # 0.034
wall
1.688 0.003
1,360 0.011
1083 0.014
0.972 0.014
0,986 0.007
0.868 0.016
abalon
1,311 = 0.003
0.871 = 0.005
0.894 0.009
0.828 0.004
0.834 0.005
0.823 # 0.007
analca
2,062 0.002
1.801 0.000
1.794 0.001
1.806 0.002
1.806 0.001
1.827 # 0.004
Accuracy
Dataset
Random
3 (Reptile Padded)
o emc (Untrain)
enc (Proposed)
enc (Frozen)
(Oricle)
segmen
0.147 =: 0.001
0.419 :: 0.005
0.496 :: 0.015
0.595 =: 0.012
0.619 a: 0.015
0.605 1: 0.009
jungle
0.335
0.002
0.395 0.003
0,382 0.003
0.393 0.004
0.396 0.002
0.460 0.003
wine
0,201 =: (1.002
0.264 1: 0.003
11,773 :: 0.003
0.314 0.005
0.308 1 0.007
0.312 a: 0.009
wilt
0.504 0.002
0.628 0.002
0.601 0.009
0,718 0.005
0.720 :: 0.005
0.724 :: 0.003
0.331 0.002
0.386 0.004
0.422 0.008
0.448 0.003
0.446 0.002
0.461 1 0.008
electr
0.499 =: 0.002
0,548 +: 0.010
0,559 # 0.004
0.626 0.011
0.625 + 0.007
0.603 :: 0.011
letter
0.039 0.000
0.078 0.005
0.005
0.153 0.006
0.204 0.006
0,282 1: 0.012
phonem
0.504 =: 01.003
0.594 +: 0.012
0,561 = 0.005
0.600 0.008
0,597 0.008
0,702 a: 0.001
vehicl
0.255 0.001
0,366 0.010
0,413 0.010
0,418 0.022
0.434 = 0.027
0,523 + 0.009
mfizut
0,104 01.002
0.354 0.006
0.398 0.008
0.428 0.012
0,431 0.009
0,447 1: 0.012
ilpd
0,506 = 0.003
0.654 + 0.005
0.659 = O. 0.004
0.670 0.00S
0.662 0.006
0.669 + 0.006
Gestur
0.202 0.002
0.310 0.002
0,350 0.006
0.368 0.003
0,364 0.004
0,383 0.002
MagicT
0.503 = 0.002
0.611 0.002
0,601 = 0.012
0.662 0.007
0,661 0.005
0.672 +: 0.004
tic
0.502 0.002
0.504 0.003
0,510 0,003
0,533 0.001
0,534 + 0.005
0.666 + 0.005
bankno
0.506 01.055
0.634 0.005
0.622 0.003
0.629 0.044
0.626 0.004
0.652 a: 0.003
diabet
0.505 = 0.004
0.656 + 0.004
0.639 = 0.007
0.674 0.002
0.673 0.003
0.675 # 0.006
wdbc
0.521 0.007
0.882 0.008
0,906 0.008
0.937 0.003
0,918 0.007
0.919 0.006
blood
0.502 = 0.001
0.579 = 0.012
0.558 = 0.010
0.613 = 0.007
0.615 + 0.003
0.636 + 0.004
vowel
0.092 0.001
0.143 0.007
0,303 0.007
0,346 0.010
0.336 0.008
0.391 + 0.013
pendig
0.102 0.001
0.180 0.003
0.193 0.004
0.222 0.011
0,227 0.009
0.646 + 0.010
wall
0.254 =: 0.001
0.324 + 0.012
0.494 = 0.007
0,576 0.009
0.562 + 0.007
0.631 # 0.010
abalon
0.339 0.003
0.566 0.002
0,554 0.007
0.594 0.004
0.587 0.004
0.593 0.005
analca
0.166 = 0.001
0.170 0.000
0.170 0.002
0.172 0.002
0,171 0.002
0.179 + 0.002
Table 2: Loss and accuracy scores of each model variant for the No-Split experiments. The values depict the mean and standard deviation across runs for each dataset with 1600 sampled test tasks per run. Best results are holdfaced (excluding OR ORACLE).
15

---

Under review as a conference paper at ICLR 2021
Loss
Dataset
Random
2 (Reptile Padded)
enc (Untrain)
f ene (Proposed)
i ene (Frozen)
vowel
2.640
2.313 0.007
1,969
1,933 0.013
1.911 0.016
wdhe
0.826 0.014
0.264 0.031
0.167 0.010
0.162 0.002
0.170 0.006
jungle
1.332 0.004
1.142 0.014
1,899 0.002
1.099 0.005
1.093 0.004
phonem
0.856 0.003
0.769 0.034
0.719 0.005
0.720 0.010
0.716 0.009
wine
1.855 0.002
1.596 0.004
1.582 0.005
1,546 0.018
1.547 0.013
analca
2,061 0.001
1.802 0.002
1.794 0.001
1.796 0.001
1.798 0.002
MagicT
0.851 0.004
0.673 0.009
0.662 0.002
0.629 0.007
0.630 0.004
diabet
0.850 0.004
0.675 0.009
0.677 0.008
0.646 0.012
0.655 0.014
letter
3,426 0.001
3.160 0.017
3.058 0.009
2.980 0.031
2.782 0.032
ilpd
0.840 0.002
0.692 0.005
0.689 0.008
0.694 0.004
0.694 0.004
Gestur
1.813 0.003
1.514 0.006
1,429 0.004
1,413 0.005
1.466 0,003
mfcat
2.531 0.005
1.591 0.067
1.417 0.01O
1.627 0.053
1.620 0.059
wilt
0.844 0.003
0.721 0.034
0.652 0.007
0,633 0.026
0.671 0.019
wall
1.640 0.004
1.356 0.002
1.081 0.009
0.993 0.010
1.003 0.009
segmen
2.166 0.002
1.388 0.061
1.147 0.021
0.799 + 0.024
0.840 0.020
1.327 0.001
1.098 0.003
1.086 0.003
1.076 + 0.011
1.082 0.004
pendig
2.548 0.003
2.210 0.016
2.195 0.015
2.123 0.009
2,038 0.196
electr
0.865 0.003
0.691 0.005
0.686 0.001
0.642 0.005
0.646 0.007
vehicl
1.624 0.005
1.289 0.008
1.221 0.004
1.193 0.018
1.225 0.004
abalon
1.313 0.004
0.971 0.025
0.929 0.004
0.894 1 0.014
0.910 0.003
tic
0.870 0.003
0.703 0.003
0.695 0.001
0.696 0.002
0.694 0.001
Accuracy
Dataset
Random
2 (Reptile Padded)
(Untrain)
(Proposed)
Ú (Frozen)
vowel
0.092 0.001
0.144 0.003
0.288 = 0.007
0.311 : 0.009
0.311 0.007
wdbe
0.522 0.009
0.901 0.011
0.937 = 0.004
0,942 1 0.003
0.935 0.004
jungle
0.001
0.359 0.011
0.385
0.378 0.009
0.388 0.005
phonem
0.503 0.002
0.504 0.021
0.502 = 0.017
0.529 = 0.004
0.533 0.024
wine
0.201 0.002
0.248
0.265
analea
0.167 0.001
0.172 0.003
0.173 0.002
0.002
MagicT
0.502 0.002
0.010
0.586
0.010
0.634 0.004
diabet
0.605
letter
0.007
0.181 0.006
ilpd
0.501 0.002
0.571 0.004
0.579 0.003
0.583 0.006
0.580 0.005
Gestur
0.200
0.306 0.003
0.361 0.003
0,375 0.004
0.372 0.004
mfeatt
0.001
0.377
0.425 0.023
0.336 0.027
0.335 0.015
wilt
0.504 0.004
0.563 0.043
0.598 0.011
0.643 0.034
0.589 0.030
wall
0,252 0.002
0.330 0.004
0.487 0.005
0,553 0.004
0.543 0.005
segmen
0.148 0.002
0.414 0.022
0.501 0.013
0.638 0.009
0.617 0.013
0.333 0.001
0.371 0.004
0.394 0.006
0.415 0.013
0.408 0.006
pendig
0.102 0.001
0.173 0.005
0.181 0.008
0.229 0.002
0.257 0.079
electr
0.500 0.002
0.545 0.006
0.551 0.002
0,622 0.007
0.616 0.009
vehicl
0.254 0.002
0.369 0.008
0.397 0.007
0.420 0.015
0.397 0.010
abalon
0.338 0.002
0.513 0.016
0.532 0.003
0.556 0.010
0.543 0.003
tic
0.501 0.002
0.501 0.001
0.506 0.003
0.514 0.002
0.515 0.003
Table Loss and accuracy scores of each model variant Split experiments. The values depict mean and standard deviation across 5 rums for each dataset with 1600 sampled test tasks per run. Best results are boldfaced.
16

---

Under review as a conference paper at ICLR 2021
D PROBLEM SETTING: GENERAL MULTI-TASK LEARNING.
We describe a classification dataset with vector-shaped predictors (i.e., no images, time series etc.) by a pair (X, Y) E RNXF x {0, ..., CJ., with predictors X and targets Y. where N denotes the number of instances, F the number of predictors and C' the number of classes. Let De = UNEN
I XxF {O.... C.. be the space of all such datasets with F predictors and D = U FEN Dr be the space of any such dataset. Let also denote the space of all predictor matrices with predictors by Xy : UNeN RNF and all predictor matrices by X := UPEN Xy. Then dataset 7 (X, Y) E DD equipped with a predefined training/test split, i.e. the quadruplet T = (Xqain Y X X Yrst) is called a task. A collection of such tasks T C 'D is called a meta- dataset. Similar to splitting a single data set into training and test part, one can split a meta-dataset T = T Ü T
Consider a meta-dataset of correlated tasks T D, such that the predictor variables (Pr, PPm..r...... Prr of any individual task T are contained in common set of predictor variables (P1, (Pr.P2.....1 J. As elucidated in the previous section, our goal is construct an encoder that learns to match these predictors and map the features of any task T E T into a shared latent space R.K.
enc: X - XK, X E E - X e EERRN R
(10)
This encoder can be combined with parametric model of fixed input size BR + {0, 1] (e.g. neural network or SVM) such that for the joint model ý o enc an initialization (init can be learned via MAML or REPTILE across all tasks, even when those may not have the same predictor vector. Just as with MAMI,, this initialization facilitates rapid convergence of the combined model ÿ o enc on any new, previously unseen task T E T More explicitly, the ultimate goal is to minimize the meta test loss
L( == Er, I:- (yrest ÿ enc (x A(w)))
(11)
here Ly is the task specific loss (e.g. miss-classification rate) of the model on the test data of Tr, using the updated parameters of")) The latter the updated parameters of the joint model ý enc which are obtained by minimizing I, on the training data (Xtrain yuain) of T7 via some learning iterative learning algorithm A (e.g. Gradient Descent) for iterations.
g(a) AA X Y.rain I.,, 9 enc; gimil
(12)
MAML and REPTILE are solving sub-problems when the number F of features is fixed and the predictors of all tasks are the same and aligned, i.e., the same predictor always occurs at the same position within the predictor vector, thus the identity be used as predictor encoder. This problem alternatively can be described as a supervised learning problem with a multivariate or structured target.
17


Reviews:
[{"review_id": "vCAikM3KaKF", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n--------------\nThe paper proposes a trainable way to re-order or recover the ordering of features from sets of examples, and use it as a way to build a common feature space (or embedding) for a neural net, the (initial) parameters of which can be trained by Reptile.\nExperiments show that such initial parameters enable faster training (inside of an episode) than untrained weights.\n\nPros\n------\n- The paper shows it is possible to recover information about the identity of coordinates in the input space, through a learned transformation, on several unstructured datasets. The similarity between such representations of individual coordinates can help identify similar features, either in a given dataset or across datasets.\n\n\nCons\n--------\nThe paper is overall really hard to follow, statements are often confusing or misleading. For instance:\n- The introduction suggests a multi-modal learning paradigm, where different tasks could have access to data in different input spaces, some of them common. However, the paper then seems to consider individual coordinates in the input space only, and focuses on mapping shuffled subsets of these coordinates back to their initial position.\n- There is confusion about the \"tasks\", which sometimes correspond to one of the OpenML datasets, and sometimes to individual few-shot episodes from one of these datasets.\n- Concepts like \"schema\" and \"predictors\" are never properly introduced or defined.\n- The description of the \"chameleon\" (alignment) component mentions \"order-invariant\" and \"permutation invariant\" several times, but it is quite unclear whether it refers to the the order of the examples within the data set (or episode) or the order in which the features are represented.\n\nThe paper uses few-shot learning vocabulary and techniques, including Reptile, but the methodology seems completely different from the few-shot learning literature. In particular:\n- There does not appear to be a split between meta-training and meta-test classes within a dataset, or meta-training datasets and meta-testing ones, except for the EMNIST experiment. Even then, the pre-training of the \"chameleon\" alignment module seems to involve using examples of the meta-test classes.\n- The reported evaluation metric is really unusual: they report the improvement (and sometimes accuracy) after 3 steps of gradient descent from within an episode, which is somewhat related to the quality of the meta-learned weights, but no other metric that would be comparable to existing literature, which makes it especially hard to assess the results.\n\nThe principle of the alignment module seems similar to (soft) attention mechanisms, in that there is a softmax trained to highlight which parts of an input vector should be emphasized (or selected) at a given point in the processing (here, in the aligned feature space). However, the literature on attention is not reviewed. \n\nMany design choices are not addressed clearly, neither in how they were made, or the impact of these choices, especially regarding the architecture of the alignment module:\n- It is a linear transformation (before the softmax), though parameterized by 3 matrices. An alternative would have been a 3-layer neural net, similar to attention networks.\n- The parameterization of the first matrix makes the number of parameters depend on N, the number of examples in a given task. This could be quite limiting to be restrained to tasks of exactly N examples, especially if both the support (mini-train) and query (mini-test or valid) parts of an episode need to have exactly N examples.\n- There is also no discussion of the  value or impact of or K, the size of the chosen embedding space).\n\nRecommendation\n--------------------------\nI recommend to reject this submission.\n\nArguments\n------------------\nThe main idea in the paper, learning alignments of various input spaces into a common embedding space through an attention mechanism, has merit and may  work reasonably.\nHowever, both the algorithm and the experimental set up are described in a quite confused way, and not well justified or grounded. The reported results are not comparable with few-shot learning literature, nor multi-modal training or feature imputation, and do not make a convincing case. \n\nQuestions\n---------------\nAs I understand it, the \"Chameleon\" architecture itself simply consists in 3 matrix multiplications (Nx8, 8x16, 16xK), which would be equivalent to the length-1 1D convolutions, is that correct? It may be more straightforward to explain that way, as $enc(X) = X M_1 M_2 M_3 X^T$.\nAlso, should the 2nd and 3rd convolutions be labeled \"8x16x1\" and \"16xKx1\" respectively? As far as I can tell, only the first Conv1D should have a dependency on N.\n\nAdditional feedback\n---------------------------\nIn Figure 2, the \"reshape\" operation should be \"transpose\" instead.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "mP5SR2rcqQC", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "- Summary and contributions\n    - In this work, the authors tried to solve the problem of ``heterogeneous'' meta-learning where each task resides in a different feature space from the other tasks. They introduced a feature transformation or re-ordering matrix to align the features. While I agree with the authors that this problem is of significance in the meta-learning community, the solution in this work, depending on the ground-truth of re-ordering matrix, is trivial and impractical. \n\n- Strengths: \n    - The problem investigated in this paper, i.e., meta-learning tasks in heterogeneous feature spaces, is important to the field of meta-learning. \n    - The paper is well written and easy to follow. \n\n- Weaknesses:\n    - The primary concern about this paper is its technical contribution, being limited and impractical. To align tasks in incommensurable feature spaces, projecting them into a common feature space has been a common practice. Please kindly see related works on heterogeneous transfer learning. The major challenge lies in the supervision needed to train the alignment matrix or function. The ground-truth feature alignment matrix is almost impractical to collect, if the dimension of features is super large and we have no knowledge of the semantic correspondence between two features from two tasks. \n    - The empirical results are also not convincing.\n         - Why is only Glorot initialization compared in Figure 3? What has been widely adopted is some better initialization strategies, including (He initialization). \n        - From both Figure 3 and Figure 4, and also the results in Appendix C, I see little improvement of the proposed over Frozen. This means that most benefits of the feature alignment come from the supervised training part where a ground-truth alignment matrix is required to train $\\Phi$, while the matrix is even infeasible to have in practical settings. \n        - In Line 6 of the section \"Ablations\", the authors mentioned that features 2 and 3 are showing a strong correlation, but I cannot see why in Figure 6. Maybe it is features 2 and 4?\n\n- Minor:\n    Line 2 in Section 4: Equation (9) does not exist…", "extra_scores": {}, "extra_text": {}}, {"review_id": "jYUeZe3aezJ", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Previous meta-learning approaches typically focus on tasks that share the same input types, e.g. images.\nThis paper addresses the problem of meta-learning weight initialization across tasks with different types of input features. \nIt proposes Chameleon model that learns to align input features from different tasks by learning a permutation matrix for each task, and shows that Chameleon can successfully learn good initialization.\n\n\nStrength:\n- It identifies and tackles a new important problem in meta-learning: meta-learning on tasks with different input features. \n- The proposed approach is simple but shows improvements over the baseline method.\n\nWeakeness:\n- Supervised training for the permutation matrix is necessary for the model to perform well.\n- Experimental results section can be more detailed. Given that Algorithm 2 is the major part of the method, how is the reordering training procedure constructed? How is the target permutation matrix determined? are there / what are the shared features between different tasks?\n- Would be great if experiments are done on one or two more datasets to strengthen the result.\n\nAdditional Comments:\n- How many features are used? How would the performance change if there are more/fewer features?\n- typo: Equation (9) is mentioned several times\n\nI believe this paper proposed a new interesting problem in meta-learning and provided a simple effective model to address the problem.\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "YR_o8utp3K6", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper aims to perform meta-learning across tasks that have different input data types by learning separate task-specific encoders, and then aligning the features produced by these encoders before making predictions.\n\nPros:\nSharing information across tasks with different input types is a relevant problem\nCons:\nPrecise problem statement and method very unclear\nExperiments are only on toy datasets\n\nDetailed Comments:\n\nIt is not clear from the abstract / introduction what is meant by “schema.” From the abstract: “for example, if the number of predictors varies across tasks, while they still share some variables.” Does this refer to the number of classes in a few-shot problem? What variables are shared? Classes, or input features? Later in the intro: “training a single model across different tasks is only feasible if all tasks share the same schema, meaning that all instances share one set of features in identical order.” These definitions of schema do not seem to be the same. Schema also does not seem to be defined in Section 3. At the beginning of that section it says, “every task has to share the same schema of common size K” which seems to indicate “schema” is the number of features and then a few lines later, “ tasks with varying input schema and feature length F” which seems to indicate “schema” is *not* the number of features.\n\n\nIn the related work section, few-shot learning did not begin in 2017 as might be suggested by the citations. It would be good to recognize the earlier works in this area, such as \nFei-Fei, L. et al. A bayesian approach to unsupervised one-shot learning of object categories. 2003\nA Bayesian framework for concept learning. PhD thesis, Massachusetts Institute of Technology, 1999.\nFor few-shot learning with deep learning, Matching Networks should arguably be cited: Vinyals, Oriol, et al. Matching networks for one shot learning. 2016.\nThe original MAML paper actually proposed the first-order version of MAML, Nichol et al. was not the first to propose this.\n\nI don’t understand how the method works when the features are learned and not given. For example, the encoder for EMNIST-Digits produces 32 features, while the encoder for EMNIST-Letters produces 64 features. If the meta-training tasks are drawn from only EMNIST-Digits, then how can the “re-ordering” matrix be learned from EMNIST-Digits such that it can re-order features from EMNIST-Letters? At the most basic level, based on Figure 2, the matrix \\Pi would have to have different dimensionality for each dataset. Even if they were the same dimensionality, how is the feature ordering supervision performed in this case?\n\nIn the “main results”, if you sub-sample features, how do you know that the sub-sampled features have enough information to perform the classification task? \n\nIt would be helpful to have an experiment on a less-toy dataset, both to demonstrate that the problem of “mis-aligned features” exists in more complex data, and that the method can address it. \n\nOverall, this paper is extremely confusing. I do not understand the problem statement or how the method is trained in the learned feature case. In my view, the clarity of this paper needs to be significantly improved to consider acceptance. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "S8A1K3_qCUU", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Chameleon: Learning Model Initializations Across Tasks With Different Schemas\n\n  \nThe paper provides an interesting direction in the few-shot classification field. In particular, it proposes a model that learns to align different predictor schemas to a common representation. The paper also demonstrates how current meta-learning approaches can successfully learn a model initialisation across tasks with different schemas as long as they share some variables with respect to their type or semantics.\n\nThe paper takes on an interesting facet of few-shot classification: An encoder model that aligns to different predictor schemas to a common representation. It tackles the problem by using 1D convolution (three of them) to transform the input features to the K-features target space and learning the alignment from the data itself. Comprehensive experiments have been done with quantitative results and analysis, to show the effectiveness of the proposed approach and the results are convincing and the code is provided to determine the reproducibility of the results.\n\nOverall performance is quite good however, it would be a good study to have an analysis of the different datasets as to how balanced/unbalanced they are, how it affects the performance, the nature of the features etc. Also, I would like the author to discuss how suitable/adaptable this approach will be for multi-label tasks and what kind of modifications (if any) are to be made.\n\nThe idea of encoding different predictor schemas to a common representation is quite interesting and comprehensive experiments and supporting ablation study has been made.   \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: After carefully going through the reviews and rebuttal, and looking at the content of the paper as well, I feel there are some issues with the current manuscript. As also pointed out by AnonReviewer5 and AnonReviewer2, the text lacks clarity. From specifically defining what a schema is, to being more explicit about the limitation of the work. 
I understand that the authors are interested in a largely unexplored setting, and hence there might not be a lot of prior work to cement the evaluation protocol. Particularly because of this I think such papers need to be upfront and clear not only in what is the setting and what is the evaluation but also what are the limitations and open problems. 

I do agree that there is value in this direction of research, and that the idea of re-ordering the features using attention (which I have to agree it is reminiscent of Bahdanau et al., ICLR 2015 -- though the semantics of it and its purpose makes it novel here) might be a way forward. But I do think for the paper to make an impact (and be ICLR ready) it needs more work both in the writing and maybe on the experimental side as well (consider some more complex task, or be more explicit on what is the common aspect between tasks in the distribution that can allow chameleon to work)
Decision: Reject