Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The stated contributions of the paper are: (1) a method for performing few-shot learning and (2) an approach for building harder few-shot learning datasets from existing datasets. The authors describe a model for creating a task-aware embedding for different novel sets (for different image classification settings) using a nonlinear self-attention-like mechanism applied to the centroid of the global embeddings for each class. The resulting embeddings are used per class with an additional attention layer applied on the embeddings from the other classes to identify closely-related classes and consider the part of the embedding orthogonal to the attention-weighted-average of these closely-related classes. They compare the accuracy of their model vs others in the 1-shot and 5-shot setting on various datasets, including a derived dataset from CIFAR which they call Hierarchical-CIFAR.

Overall, while we like the concepts/ideas and the problem is definitely important, we were not enthusiastic about the paper. First, we found the write up to be cryptic, involving very long unclear statements. It read as if the authors were writing for themselves and not for ICLR general audience. Beyond the writing style, we found the paper to have:

* Inadequate description of the model, including mathematical inaccuracies.
* Inadequate description of Hierarchical-CIFAR, motivation, and evaluation.

Description of model:
------------------------------
The manuscripts describe the presented approach as metric learning but make no use of a distance function in the different spaces they map to. Instead, the manuscript defines an inner product over embeddings to compute similarities.

The manuscript describe their “self-attention operation” as a dynamic set-to-set operation. While the usual definition of self-attention is permutation invariant, the definition presented here is not and thus cannot be accurately described as a mapping between sets. Specifically, in the usual presentation of self-attention the only sharing of information between different set elements is during the outer product of the key and query vectors. The use of a BLSTM between “neighboring elements” of the set of prototype vectors violates this assumption and induces a lack of permutation invariance. This makes the method sensitive to permutations of classes, which does not make sense for predicting unordered classes.

In sections 2.2.3 and 2.2.4 the material is presented twice but slightly differently. For example, the definition of $b_k$ inline before equation 3 differs from equation 6 later in the text.

Equation 7 is incorrect and should not exclude the current class from the denominator.

The description of how to classify new points after equation 6 is poorly explained. The description of what happens when $h_V$ is a “BLTSM” [sic (should be BLSTM)] is noninformative.

The manuscript describes two dimension sizes $H$ and $M$ but the definition of $attn$ requires that $H = M$.

Description of new dataset and evaluations:
------------------------------------------------------------
One of the stated contributions of the manuscript is a methodology to build harder few-shot learning datasets. Section 4.2 is the only place in the text that appears to address this point, but is unclear where either new finer-grained or coarser-grained labels are coming from (new manual annotation or otherwise). The manuscript “leave[s] out the detail of its construction for simplicity”, but it is unclear what is being done here in the first place.

The manuscript does not detail tuning competing methods on the new dataset and so it is unclear whether it is a fair comparison.

The manuscript presents evaluations without any discussion of differences in performance between datasets or the 1-shot/5-shot settings. For example, their method is significantly better on CUB on 1-shot but not so much on 5-shot, on the other hand it is not significantly better on 1-shot for H-CIFAR and CIFAR-HS but then becomes better than the rest with 5-shot.

Additional comments/corrections
---------------------------------------------
There were numerous typos and grammatical errors that were present in the manuscript that did not directly impact this evaluation but should be fixed in the future.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: The authors propose a new neural network model, called as Dissimilarity Network, to improve the few-shot learning accuracy.
Overall the idea is well motivated that by emphasizing the difference among classes, the model can achieve more accurate predictions for classes where only limited data points are available for training.
However, the paper is not quite well written.
Firstly, much of the work is built upon previous work including attention mechanisms, episodic training for few-shot learning. Such components are the core of this work because the attention mechanisms implement the class-awareness, and the episodic training facilitates the LSTM structure. Yet these are not well explained and not much context is provided, thus making the paper hard to follow.
Secondly, some terms are fairly overloaded, or not clearly defined. For example, the “prior” as mentioned in both the abstract and the introduction doesn’t refer to the commonly interpreted term as in the Bayesian settings, but rather as a hand-waiving term to indicate the model design. Also, the terms, “score”, “metric”, “dissimilarity” are mentioned in the paper but the paper is not really learning the metric, to my understanding. Thus the details of the paper is quite hard to grasp. 
Lastly, the idea of designing the global embedding and the task aware embedding is interesting but shouldn’t really be restricted to few-shot learning. It would be interesting to test the idea on general classification tasks, for example in a simple cross validation settings.
Thus I think the paper would be stronger if the above are addressed and it’s not ready for publishing yet in its current form.

Below are some more detailed comments:
1)	In the abstract, the “newly introduced dataset H-CIFAR” is not precise to me; my understanding is that the paper proposes such an experiment design for testing how well a classifier can predict the labels with hierarchy. The current writing refers to that the authors comprises a completely new dataset with new labels.
2)	In the last sentence of the second paragraph in Introduction, the question is asked “what prior” should be reasonable. Since the authors didn’t really add any priors in a Bayesian settings but rather designed an architecture, I suggest to reword something like “how to explicitly encode hierarchies into the model structure”.
3)	In Section 2.1, some more description for “episodic training” would be nice: why should it be used? How is it used and why it makes sense in the few-shot learning context?
4)	In Section 2.2, it would be nice to add the mathematical definition of “prototype”.
5)	In Section 2.2.1, it would be nice to define “H”.
6)	In Section 2.2.2, is M required to be fixed given it’s episodic training? Also it would be nice to add more details about the attention mechanism.
7)	In the result section, it would be nice to discuss when the proposed method is doing better than other methods, for example RelationNet, as well as when it’s worse since different datasets show different results.

review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: In this paper the authors propose a metric based model for few-shot learning. The goal of the proposed technique is to incorporate a prior that highlight better the dissimilarity between closely related class prototype. Thus, the proposed paper is related to prototypical neural network (use of prototype to represent a class) but differ from it by using inner product scoring  as a similarity measure instead of the use of euclidean distance. There is also close similarity between the proposed method and matching network 


overall, the paper does not  highlight the novelty of their proposed method especially prototypical network and matching network. Thus, the related work session is so general and does not tackle the close models in details. The experiments do not provide convincing evidence of the correctness of the proposed approach.

Several parts are unclear/incomprehensible:
(1) The Introduction is confusing and does not demonstrate the problem that the paper is trying to solve. Specifically, the described intuition (Mill’s method of difference) is not convincing 
(2) the first sentence of  the section “Our work.”  (page 1) is long and unclear …  “In this paper, we propose a model that focuses on the differences in the support set of closely related classes in assigning the class label to a new instance in the novel task.”
(3) the use of the two level of embedding is confusing and not clear. Figure 1 is also confusion and not clear.

the correctness of the proposed approach is not proved by the conducted experiment and does not provide convincing and fair comparison with SoA techniques:
(1) The experiments do not provide the details of the used architecture compared to your baseline.  (how many layers are used in both embedding systems)
(2) In Table 1 you are using the results reported by Chen et al. (2019) did you use his framework (Resnet or 4 layers CNN)

**Minor comments** 
The definition of the embedding function f = (f_g o f_f)  (in line 1 page 5) is not consistent with the domain of each function  f_g is defined on R^H x R^H.

K is not defined (last line page 3)

review_title: Official Blind Review #3

Meta-review: Main content:

[Blind review #3] The authors propose a metric based model for few-shot learning. The goal of the proposed technique is to incorporate a prior that highlight better the dissimilarity between closely related class prototype. Thus, the proposed paper is related to prototypical neural network (use of prototype to represent a class) but differ from it by using inner product scoring  as a similarity measure instead of the use of euclidean distance. There is also close similarity between the proposed method and matching network.

[Blind review #2] The stated contributions of the paper are: (1) a method for performing few-shot learning and (2) an approach for building harder few-shot learning datasets from existing datasets. The authors describe a model for creating a task-aware embedding for different novel sets (for different image classification settings) using a nonlinear self-attention-like mechanism applied to the centroid of the global embeddings for each class. The resulting embeddings are used per class with an additional attention layer applied on the embeddings from the other classes to identify closely-related classes and consider the part of the embedding orthogonal to the attention-weighted-average of these closely-related classes. They compare the accuracy of their model vs others in the 1-shot and 5-shot setting on various datasets, including a derived dataset from CIFAR which they call Hierarchical-CIFAR.

--

Discussion:

All reviews agree on a weak reject.

--

Recommendation and justification:

While the ideas appear to be on a good track, the paper itself is poorly written - as one review put it, more like notes to themselves, rather than a well-written document to the ICLR audience.
Decision: Reject