Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: Overview  

This paper proposes a new objective function called Instance Cross Entropy (ICE) for metric learning. Compared to the triplet loss (or contrastive loss) and its many variants, the distance between points in feature space is defined to be the dot/inner product, rather than computing the euclidian distance between feature vectors. Since the L2 norm of the final features is constrained  to be 1, the dot product represents the cosine distance between 2 feature vectors. Compared to the softmax/categorical loss, this objective has the advantage that there is no need to learn a per-class weight vector in the output softmax layer and therefore this method can be used even when the number of classes during training is unbounded. This is a very useful design feature, since the output softmax layer grows linearly with the size of the classes and can quickly become prohibitively large. Furthermore, mini batches for training can be randomly sampled without requiring expensive negative data mining strategies which are necessary when using the triplet loss. This is another very useful property. 

Overall, I think the ideas presented are interesting and the paper provides a useful summary of existing approaches for metric learning. However, I think the technical presentation needs to be improved significantly before  the paper can be accepted for publication. I found the paper quite difficult to follow. I had to re-read the motivation and the description of the loss function many times before being able to understand what is going on and how the proposed loss compares and contrasts with existing approaches. I also think some of the terminology and notation used in the paper is very confusing and should be updated to help the reader get to the main argument quickly. 

Comments


1. Title: What exactly does instance cross entropy (ICE) mean? And how is it different from categorical cross entropy (CCE)? Isn’t categorical cross-entropy also calculating the entropy between the predicted distribution for *each instance* and the ground truth? I think the authors should reconsider the name for the proposed loss and choose a more descriptive name for the algorithm.

2.  I also found the use of the term “matching distribution” a bit confusing since it has a well-defined meaning in statistics. However ML is full of overloaded terms and I understand if the authors want to keep this description. 

3. None of the references in the main text have brackets around them. I think this makes the paper appear very cluttered and makes parsing each paragraph quite difficult. I would highly recommend the paper be reformatted and the authors use brackets around references. 

4. In the abstract, the authors mention that the proposed method has a “clear probabilistic interpretation”. From my understanding, I see ICE as a blend of the categorical cross-entropy loss and the triplet loss, retaining useful properties of each without increasing the complexity of the loss computation. However, I do not see a clear probabilistic interpretation of the loss function. The softmax computation yields a probability distribution like output given a query and an anchor point, which is hand-designed in the objective function. I  think the authors should more clearly motivate what the probabilistic angle for the loss function is. 

5. I found Figure 1 to be very difficult to interpret. This is a missed opportunity, since this figure alone could communicate some of the key ideas in the paper to the reader. However, there are many details that have not been explicitly mentioned. What do the colours (white, blue, yellow) mean? The figure shapes probably mean classes. What do the terms p_* mean? What do i,j mean in Figure 1 e and how are they different from the previous figures. These details should all be present in the title for the figure. Even with these details, the figure could still be a bit more explicit. I found the yellow arrow with cross entropy and the notion of “ground truth” in the figure difficult to follow. 

6. How was the constrained optimisation performed? In Equation 6, we see that there is a constraint associated with each example in the mini batch which says the norm of the feature vector should be equal to 1. However I could not find any implementation details of how this constraint was satisfied. Did the authors use a Lagrangian formulation? I think the paper is irreproducible without this detail.  

Summary

I think this is an interesting proposal to combine the useful features of the softmax and triplet losses. However, I think the technical presentation needs to be improved significantly in order for the paper to be accepted for publication. 

review_title: Official Blind Review #4

final_score: rating: 8: Accept
review_content: The paper proposes a method to measure the difference between an
estimated instance-level matching distribution and its ground-truth
one, based on instance cross entropy (ICE).
The goal is to learn an embedding that captures the semantic
similarities among samples. 
In particular, with ICE they try to maximize the matching probability
of an instance with similar instances (same class).
The authors also use sample re-weighting into ICE and show the benefits
of the approach against other state-of-the-art methods on three
datasets. The authors performed several experiments with convincing
results.

The positive aspect of the paper is introducing this instance-based
measure which is shown to perform well on 3 challenging datasets.

It is not clear how is the scaling parameter (s) determined. Are there
any guidelines for fixing its value?
The authors should explain in more detail how is the non-linear
transformation achieved.

The algorithm goes through all the examples of a class on each
mini-batch, which seems a computational expensive procedure. The paper
will benefit if time results are reported for different approaches.

The paper is sometimes difficult to follow and needs a careful
revision. 

review_title: Official Blind Review #3

final_score: rating: 1: Reject
review_content: The paper proposes a method inspired by the categorical cross entropy (CCE). In a similar way as many metric learning approaches, a softmax approach based on cross entropy is used to enforce similar examples to have smaller distances than dissimilar distances. 
Nonetheless, instead of considering a centroid-based deep metric learning approach (e.g. like Snell et al. (2017)), only one anchor is considered for each training example, and the distance of an anchor with one of its positive is learned that its distance is smaller than the distance with examples belonging to different categories.

I vote for reject for the following reasons:
- The paper is too similar to NCA and S-NCA (introduced in Section 2.2). In the same way as ICE, S-NCA also considers learning l2-regularized representations. The main difference with S-NCA is the way negative examples are sampled. 
S-NCA proposes a framework based on augmented memory, ICE proposes a sampling strategy similar to [A] and used in Nickel et al. (2018) to subsample negative pairs.
- Another contribution of ICE is the use of some hyperparameter s in Equations (11) and (12). This hyperparameter s plays the role as the inverse of the temperature and is in fact learned in ICE. S-NCA also plays with the temperature but does not learn it. 
On the other hand, learning the temperature in a similar way has been proposed in the deep metric learning literature (e.g. TADAM).

On the other hand, the paper has some nice contributions:
- The main "novelty" of ICE seems to be the analysis in Section 3.4 of the partial derivatives and their impact on the sample reweighting. Although the explanation is simple, it helps understand what's happening during optimization.
- The experimental results on different transfer learning benchmarks seem convincing.


[A] Jean et al. On usingvery large target vocabulary for neural machine translation. ACL 2015
review_title: Official Blind Review #2

Meta-review: The paper proposes a new objective function called ICE for metric learning.

There was a substantial discussion with the authors about this paper. The two reviewers most experienced in the field found the novelty compared to the vast existing literature lacking, and remained unconvinced after the discussion. Some reviewers also found the technical presentation and interpretations to need improvement, and this was partially addressed by a new revision.

Based on this discussion, I recommend a rejection at this time, but encourage the authors to incorporate the feedback and in particular place the work in context more fully, and resubmit to another venue.
Decision: Reject