Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: As you describe in § 4, this isn't the focus of your work, but I'm wondering how you might incorporate best mining practices into your approach.
review_title: hard negative mining

final_score: rating: 7: Good paper, accept
review_content: The paper modifies the KISSME algorithm in two ways: by incorporating a dimensionality reduction step and by integrating it in a loss function for learning deep networks. Experiments show that the latter solution is more stable than standard approaches to metric learning with deep nets.

The paper contribution could be good, but the paper could use some polish. After carefully reading the authors' response to my original questions, I have to say that I am still confused about several details.

Lemma 1 states that "log(delta) identifies a Mahalanobis metric ... as M = Proj(...)". I still think that the wording of the Lemma is imprecise; the authors should probably simply say that log(delta) results in an approximation of a certain metric, up to a constant term.  The word "identifies" definitely does not mean "approximates", which, since the word "approximation" is contained in the proof of the Lemma, is what is happening here. The authors have promised to fix this issue in the final version by changing the wording to "log(\delta) determines a Mahalanobis metric", but "determines" means that log(\delta) is a metric, whereas it seems to only *approximate* a metric.

Note also that the proof contains the sentence "the Mahalanobis distance should approximate". It is quite odd to find the verb "should" in a mathematical proof. Ultimately, we are not sure what is being proved since the statement of the lemma is imprecise.

Following up on this discussion, about the relationships between Eq. 4 and 6, the authors comment that "While, the form of Eq.6 may suggest that it is an approximation to Eq.4, as discussed above, it is indeed the form of metric obtained in the latent space".  I agree that, taken in isolation, this is a valid definition of some metric, but then, at the very least, the paper links this up with the rest of the discussion in a very confusing manner. Note that Eq. 6 is obtained "using lemma 1", which contains the world "approximation" in the proof.

While none of these issues necessarily invalidates the *method* and the *results* of the paper, they make the *premises* a little tenuous and confusing.

A second contribution of the paper is to incorporate KISSME in metric learning using deep networks and pairwise loss. Here the authors clarify satisfactorily the difference between standard metric learning in deep networks and their approach in their answers. I would recommend putting the comparison with the baselines upfront to make this message clearer to the reader (e.g. by moving Section 4.2 before Section 4.1).






















Eq. (5) and (6) are supposed to "reflect better" Eq. (1) than Eq. (4). In fact, Eq. (6) is a regularised version of Eq. (4) (by the low-dimensional bottleneck), so it can only be a worse approximation of Eq. (1). Probably the authors mean that the regularisation can make Eq. (6) preferable to Eq. (4) on the test data.

I would also like to understand better how the method is integrated in deep learning.  As noted by the authors,  with CNNs one can simply learn a projection matrix L as the last FC layer and define the metric to be L'L. Learning can use the usual Siamese scheme with pairwise or triplet losses. I am rather confused how the proposed algorithm relates to this approach according (Section 3). My understanding is that one first fixes L and trains the rest of the network optimising the pairwise loss Eq. (14). Once this is done, one updates L as the square root of the matrix M obtained by the modified KISSME algorithm. Is that correct? If so, the authors should compare to the obvious baseline of optimising L directly as part of back-propagation while training the network, as this is a much more direct approach.

I am also confused about the dimensionality reduction role of matrix W. In Eq. (5) W^T is used to take the data down from dimension D to dimension d. However, when applied to a CNN, the dimensionality reduction is assumed to be incorporated in the network itself, such that the dimension of the CNN representation is d from the outset. Hence, in this case the modified KISSME algorithm does not need to perform any dimensionality reduction. Is that so? Can the authors give us an intuition of what their algorithm brings on top of standard Siamese learning in this case?

Experiments:

Comparison with state of the art on the Car dataset is not very meaningful as every method appear to use a different underlying deep net. The paper uses GoogLeNet, which is significantly better than VGG-M used by Liu et al (2016). Hence it is not surprising that the authors obtain a better performance.

I am more interested in understanding the comparison with the KISSME baseline. How is this baseline applied to this data? Is there any fine tuning of the CNN at all? If not, this may be enough to explain the difference with the proposed JDR-KISSME.

There is one last experiment on CIFAR after conclusion and references. Should we consider that as well?
review_title: A potentially useful contribution in metric learning with deep nets, but the paper lacks clarity and polish

final_score: rating: null
review_content: Hi, 

it would be very useful if you could comment on the following two points, sorry for posting these only late.
Many thanks

In the first experiment, the authors compare to related metric learning work. They  claim that the proposed approach "achieves the state-of-the-art verification accuracies", but this seems unjustified as the CNNs used to generate the features by the authors is different from the one used in related work. Thus it seems to me that no conclusions on the quality of the proposed approach as compared to others can be drawn from these experiments.

The main motivation of the paper to base itself on KISSME is scalability, see introduction. However, for the case where a CNN is trained it seems that it is likely to be in a large-scale data and iterative training regime, and it is not obvious that KISSME would be more efficient than other metric learning objectives based on a pairwise or triplet loss.




review_title: Experimental comparison and scalability

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper presents extensions of the KISSME metric learning method.
First, a subspace learning component is integrated, which avoids the need of ad-hoc pre-processing in the case of high-dimensional data.
Second, the approach is integrated with a CNN model that derives the features from an input image before it is entered into the Mahalanobis distance computation.

The related work section would benefit from a disucssion of previous work that also learns low-rank Mahalanobis metrics, such as e.g. [A]. One might argue that the referenced work, and similar ones, solves a non-convex objective and is therefore of less interest. This argument, however, seems weak at the moment that CNN training is integrated, since this also involves hihgly non-convex optimisation, yet leads to excellent results. A discussion on this point would be useful to include in the paper.

In the first experiment, the authors compare to related metric learning work. They  conclude that the proposed approach is superior, but this seems unjustified as the CNNs used to generate the features by the authors is different from the one used in related work. Thus it seems to me that no conclusions on the quality of the metric learning approach can be drawn from these experiments.

In the second experiment the authors show that a baseline where a factorisation of the PSD matrix M is absorbed in the low-rank projection matrix W leads to worse results than the proposed over-parametrised method. It would be useful if the authors could provide an analysis of why this might be the case; the current text does not seem to provide an explanation or hyposthesis. Is the same cost function used by the proposed method and the pairwise and triplet baseline ?

I found the title of the paper slightly misleading, as the contribution of the paper is not to provide an extension to deep feature training: this has been done extensively in the past. In my understanding, the contribution lies in the proposed (overparameterized) formulation of learning a low-rank Mahalanobis metric for the KISSME objective function. Certainly, the extension to deep feature training is interesting, but relatively straightforward as compared to the main contribution. The title could reflect this more accurately.

The main motivation of the paper to base itself on KISSME is scalability, see introduction. However, for the case where a CNN is trained it seems that it is likely to be in a large-scale data and iterative training regime, and it is not obvious that KISSME would be more efficient than other metric learning objectives based on a pairwise or triplet loss. Therefore, comparison with other metric learning  objective functions in both performance and run time would be very useful to get a complete picture of which methods are most effective under what conditions. 


[A] Guillaumin, M.; Mensink, T.; Verbeek, J. & Schmid, C. Face recognition from caption-based supervision IJCV, 2012, 96, 64-82

review_title: Low-rank Mahalanobis metric learning with the KISMME objective function

final_score: rating: 7: Good paper, accept
review_content: KISSME was a promising metric learning approach that got lost in the deep learning flood. This is a shame since it had very good results at the time, but it was hindered by the quirks of the PCA pre-processing stage. In a nutshell, this paper reimagines this approach using deep convnets, gets good results on modern benchmark datasets and does not require meticulous preprocessing tricks. It therefore deserves to be published, since it rescues a promising approach from obscurity and does a competent job of illustrating its potential in a modern context, with all the necessary discussion of related work on old (LMNN, MMC, PCCA) and new (deep, triplet-loss based) metric learning approaches.
review_title: reviving a promising metric learning approach through the lens of deep learning

final_score: rating: null
review_content: Technical content:

There are several imprecisions in the paper which I would like the authors to clarify:

Lemma 1 states that "log(delta) identifies a Mahalanobis metric ... as M = Proj(...)". While clearly the projection operator results, by definition of its range, in a metric, there is no clear definition of what "identifies" means. My best interpretation of "identify" is that log(delta) results in a quantity that is roughly similar to M.

Eq. (5) and (6) are supposed to "reflect better" Eq. (1) than Eq. (4). In fact, Eq. (6) is a regularised version of Eq. (4) (by the low-dimensional bottleneck), so it can only be a worse approximation of Eq. (1). Probably the authors mean that the regularisation can make Eq. (6) preferable to Eq. (4) on the test data.

I would also like to understand better how the method is integrated in deep learning.  As noted by the authors,  with CNNs one can simply learn a projection matrix L as the last FC layer and define the metric to be L'L. Learning can use the usual Siamese scheme with pairwise or triplet losses. I am rather confused how the proposed algorithm relates to this approach according (Section 3). My understanding is that one first fixes L and trains the rest of the network optimising the pairwise loss Eq. (14). Once this is done, one updates L as the square root of the matrix M obtained by the modified KISSME algorithm. Is that correct? If so, the authors should compare to the obvious baseline of optimising L directly as part of back-propagation while training the network, as this is a much more direct approach.

I am also confused about the dimensionality reduction role of matrix W. In Eq. (5) W^T is used to take the data down from dimension D to dimension d. However, when applied to a CNN, the dimensionality reduction is assumed to be incorporated in the network itself, such that the dimension of the CNN representation is d from the outset. Hence, in this case the modified KISSME algorithm does not need to perform any dimensionality reduction. Is that so? Can the authors give us an intuition of what their algorithm brings on top of standard Siamese learning in this case?

Experiments:

Comparison with state of the art on the Car dataset is not very meaningful as every method appear to use a different underlying deep net. The paper uses GoogLeNet, which is significantly better than VGG-M used by Liu et al (2016). Hence it is not surprising that the authors obtain a better performance.

I am more interested in understanding the comparison with the KISSME baseline. How is this baseline applied to this data? Is there any fine tuning of the CNN at all? If not, this may be enough to explain the difference with the proposed JDR-KISSME.

There is one last experiment on CIFAR after conclusion and references. Should we consider that as well?








review_title: Some questions on technical content and experiments

Meta-review: The paper proposes two extensions of the KISSME metric learning method: (i) learning dimensionality reduction together with the metric; (ii) incorporating it into deep neural networks. The contribution is rather incremental, and the results are at the level of the prior art in deep metric learning, so in its current form the paper is not ready to be accepted.
Decision: Reject