Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: Summary:
This paper aims to perform meta-learning across tasks that have different input data types by learning separate task-specific encoders, and then aligning the features produced by these encoders before making predictions.

Pros:
Sharing information across tasks with different input types is a relevant problem
Cons:
Precise problem statement and method very unclear
Experiments are only on toy datasets

Detailed Comments:

It is not clear from the abstract / introduction what is meant by “schema.” From the abstract: “for example, if the number of predictors varies across tasks, while they still share some variables.” Does this refer to the number of classes in a few-shot problem? What variables are shared? Classes, or input features? Later in the intro: “training a single model across different tasks is only feasible if all tasks share the same schema, meaning that all instances share one set of features in identical order.” These definitions of schema do not seem to be the same. Schema also does not seem to be defined in Section 3. At the beginning of that section it says, “every task has to share the same schema of common size K” which seems to indicate “schema” is the number of features and then a few lines later, “ tasks with varying input schema and feature length F” which seems to indicate “schema” is *not* the number of features.


In the related work section, few-shot learning did not begin in 2017 as might be suggested by the citations. It would be good to recognize the earlier works in this area, such as 
Fei-Fei, L. et al. A bayesian approach to unsupervised one-shot learning of object categories. 2003
A Bayesian framework for concept learning. PhD thesis, Massachusetts Institute of Technology, 1999.
For few-shot learning with deep learning, Matching Networks should arguably be cited: Vinyals, Oriol, et al. Matching networks for one shot learning. 2016.
The original MAML paper actually proposed the first-order version of MAML, Nichol et al. was not the first to propose this.

I don’t understand how the method works when the features are learned and not given. For example, the encoder for EMNIST-Digits produces 32 features, while the encoder for EMNIST-Letters produces 64 features. If the meta-training tasks are drawn from only EMNIST-Digits, then how can the “re-ordering” matrix be learned from EMNIST-Digits such that it can re-order features from EMNIST-Letters? At the most basic level, based on Figure 2, the matrix \Pi would have to have different dimensionality for each dataset. Even if they were the same dimensionality, how is the feature ordering supervision performed in this case?

In the “main results”, if you sub-sample features, how do you know that the sub-sampled features have enough information to perform the classification task? 

It would be helpful to have an experiment on a less-toy dataset, both to demonstrate that the problem of “mis-aligned features” exists in more complex data, and that the method can address it. 

Overall, this paper is extremely confusing. I do not understand the problem statement or how the method is trained in the learned feature case. In my view, the clarity of this paper needs to be significantly improved to consider acceptance. 

review_title: Confusing presentation of problem statement and method

final_score: rating: 3: Clear rejection
review_content: Summary
--------------
The paper proposes a trainable way to re-order or recover the ordering of features from sets of examples, and use it as a way to build a common feature space (or embedding) for a neural net, the (initial) parameters of which can be trained by Reptile.
Experiments show that such initial parameters enable faster training (inside of an episode) than untrained weights.

Pros
------
- The paper shows it is possible to recover information about the identity of coordinates in the input space, through a learned transformation, on several unstructured datasets. The similarity between such representations of individual coordinates can help identify similar features, either in a given dataset or across datasets.


Cons
--------
The paper is overall really hard to follow, statements are often confusing or misleading. For instance:
- The introduction suggests a multi-modal learning paradigm, where different tasks could have access to data in different input spaces, some of them common. However, the paper then seems to consider individual coordinates in the input space only, and focuses on mapping shuffled subsets of these coordinates back to their initial position.
- There is confusion about the "tasks", which sometimes correspond to one of the OpenML datasets, and sometimes to individual few-shot episodes from one of these datasets.
- Concepts like "schema" and "predictors" are never properly introduced or defined.
- The description of the "chameleon" (alignment) component mentions "order-invariant" and "permutation invariant" several times, but it is quite unclear whether it refers to the the order of the examples within the data set (or episode) or the order in which the features are represented.

The paper uses few-shot learning vocabulary and techniques, including Reptile, but the methodology seems completely different from the few-shot learning literature. In particular:
- There does not appear to be a split between meta-training and meta-test classes within a dataset, or meta-training datasets and meta-testing ones, except for the EMNIST experiment. Even then, the pre-training of the "chameleon" alignment module seems to involve using examples of the meta-test classes.
- The reported evaluation metric is really unusual: they report the improvement (and sometimes accuracy) after 3 steps of gradient descent from within an episode, which is somewhat related to the quality of the meta-learned weights, but no other metric that would be comparable to existing literature, which makes it especially hard to assess the results.

The principle of the alignment module seems similar to (soft) attention mechanisms, in that there is a softmax trained to highlight which parts of an input vector should be emphasized (or selected) at a given point in the processing (here, in the aligned feature space). However, the literature on attention is not reviewed. 

Many design choices are not addressed clearly, neither in how they were made, or the impact of these choices, especially regarding the architecture of the alignment module:
- It is a linear transformation (before the softmax), though parameterized by 3 matrices. An alternative would have been a 3-layer neural net, similar to attention networks.
- The parameterization of the first matrix makes the number of parameters depend on N, the number of examples in a given task. This could be quite limiting to be restrained to tasks of exactly N examples, especially if both the support (mini-train) and query (mini-test or valid) parts of an episode need to have exactly N examples.
- There is also no discussion of the  value or impact of or K, the size of the chosen embedding space).

Recommendation
--------------------------
I recommend to reject this submission.

Arguments
------------------
The main idea in the paper, learning alignments of various input spaces into a common embedding space through an attention mechanism, has merit and may  work reasonably.
However, both the algorithm and the experimental set up are described in a quite confused way, and not well justified or grounded. The reported results are not comparable with few-shot learning literature, nor multi-modal training or feature imputation, and do not make a convincing case. 

Questions
---------------
As I understand it, the "Chameleon" architecture itself simply consists in 3 matrix multiplications (Nx8, 8x16, 16xK), which would be equivalent to the length-1 1D convolutions, is that correct? It may be more straightforward to explain that way, as $enc(X) = X M_1 M_2 M_3 X^T$.
Also, should the 2nd and 3rd convolutions be labeled "8x16x1" and "16xKx1" respectively? As far as I can tell, only the first Conv1D should have a dependency on N.

Additional feedback
---------------------------
In Figure 2, the "reshape" operation should be "transpose" instead.

review_title: The idea of learning to re-align input spaces in a common feature space has merit, but the experimental protocol is unusual and results not convincing

final_score: rating: 4: Ok but not good enough - rejection
review_content: - Summary and contributions
    - In this work, the authors tried to solve the problem of ``heterogeneous'' meta-learning where each task resides in a different feature space from the other tasks. They introduced a feature transformation or re-ordering matrix to align the features. While I agree with the authors that this problem is of significance in the meta-learning community, the solution in this work, depending on the ground-truth of re-ordering matrix, is trivial and impractical. 

- Strengths: 
    - The problem investigated in this paper, i.e., meta-learning tasks in heterogeneous feature spaces, is important to the field of meta-learning. 
    - The paper is well written and easy to follow. 

- Weaknesses:
    - The primary concern about this paper is its technical contribution, being limited and impractical. To align tasks in incommensurable feature spaces, projecting them into a common feature space has been a common practice. Please kindly see related works on heterogeneous transfer learning. The major challenge lies in the supervision needed to train the alignment matrix or function. The ground-truth feature alignment matrix is almost impractical to collect, if the dimension of features is super large and we have no knowledge of the semantic correspondence between two features from two tasks. 
    - The empirical results are also not convincing.
         - Why is only Glorot initialization compared in Figure 3? What has been widely adopted is some better initialization strategies, including (He initialization). 
        - From both Figure 3 and Figure 4, and also the results in Appendix C, I see little improvement of the proposed over Frozen. This means that most benefits of the feature alignment come from the supervised training part where a ground-truth alignment matrix is required to train $\Phi$, while the matrix is even infeasible to have in practical settings. 
        - In Line 6 of the section "Ablations", the authors mentioned that features 2 and 3 are showing a strong correlation, but I cannot see why in Figure 6. Maybe it is features 2 and 4?

- Minor:
    Line 2 in Section 4: Equation (9) does not exist…
review_title: The technical contribution is limited and impractical.

final_score: rating: 6: Marginally above acceptance threshold
review_content: Chameleon: Learning Model Initializations Across Tasks With Different Schemas

  
The paper provides an interesting direction in the few-shot classification field. In particular, it proposes a model that learns to align different predictor schemas to a common representation. The paper also demonstrates how current meta-learning approaches can successfully learn a model initialisation across tasks with different schemas as long as they share some variables with respect to their type or semantics.

The paper takes on an interesting facet of few-shot classification: An encoder model that aligns to different predictor schemas to a common representation. It tackles the problem by using 1D convolution (three of them) to transform the input features to the K-features target space and learning the alignment from the data itself. Comprehensive experiments have been done with quantitative results and analysis, to show the effectiveness of the proposed approach and the results are convincing and the code is provided to determine the reproducibility of the results.

Overall performance is quite good however, it would be a good study to have an analysis of the different datasets as to how balanced/unbalanced they are, how it affects the performance, the nature of the features etc. Also, I would like the author to discuss how suitable/adaptable this approach will be for multi-label tasks and what kind of modifications (if any) are to be made.

The idea of encoding different predictor schemas to a common representation is quite interesting and comprehensive experiments and supporting ablation study has been made.   

review_title: Official Blind Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: Previous meta-learning approaches typically focus on tasks that share the same input types, e.g. images.
This paper addresses the problem of meta-learning weight initialization across tasks with different types of input features. 
It proposes Chameleon model that learns to align input features from different tasks by learning a permutation matrix for each task, and shows that Chameleon can successfully learn good initialization.


Strength:
- It identifies and tackles a new important problem in meta-learning: meta-learning on tasks with different input features. 
- The proposed approach is simple but shows improvements over the baseline method.

Weakeness:
- Supervised training for the permutation matrix is necessary for the model to perform well.
- Experimental results section can be more detailed. Given that Algorithm 2 is the major part of the method, how is the reordering training procedure constructed? How is the target permutation matrix determined? are there / what are the shared features between different tasks?
- Would be great if experiments are done on one or two more datasets to strengthen the result.

Additional Comments:
- How many features are used? How would the performance change if there are more/fewer features?
- typo: Equation (9) is mentioned several times

I believe this paper proposed a new interesting problem in meta-learning and provided a simple effective model to address the problem.



review_title: Official review

Meta-review: After carefully going through the reviews and rebuttal, and looking at the content of the paper as well, I feel there are some issues with the current manuscript. As also pointed out by AnonReviewer5 and AnonReviewer2, the text lacks clarity. From specifically defining what a schema is, to being more explicit about the limitation of the work. 
I understand that the authors are interested in a largely unexplored setting, and hence there might not be a lot of prior work to cement the evaluation protocol. Particularly because of this I think such papers need to be upfront and clear not only in what is the setting and what is the evaluation but also what are the limitations and open problems. 

I do agree that there is value in this direction of research, and that the idea of re-ordering the features using attention (which I have to agree it is reminiscent of Bahdanau et al., ICLR 2015 -- though the semantics of it and its purpose makes it novel here) might be a way forward. But I do think for the paper to make an impact (and be ICLR ready) it needs more work both in the writing and maybe on the experimental side as well (consider some more complex task, or be more explicit on what is the common aspect between tasks in the distribution that can allow chameleon to work)
Decision: Reject