Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: Can the authors comment on whether the proposed objective is affected by class imbalanced data sets?
i.e. many more samples for some classes than others. 
review_title: class imbalance

final_score: rating: 4: Ok but not good enough - rejection
review_content: The authors propose a semi-supervised technique for neural networks which includes two objectives: (1) the neural net embeddings of two samples with identical labels is constrained to be closer than the embeddings of samples with different labels (2) the embedding of an unlabeled example is constrained to be close to the embeddings of the closest labeled sample (and far away from the other ones).

While the authors list a number of previous works, they do not relate them very well with their approach, and actual differences appear unclear. In particular, the approach seems rather incremental with respect to (Hadsell et al. 2006), the way the neighbors are chosen being the main difference; in that respect, (Weston et al, 2008 or 2012) (which could be viewed as an application of DrLim from Hadsell et al, applied to semi-supervised learning) is even closer to the approach proposed here.

There is also no balancing constraint in the proposed approach, which was known to be crucial in all these models from the 2000s.

Concerning the experimental results, reported performance are very good; given the approach is very close to previous work, it is hard to know if good performance come from better neural net architectures or something else. This should be clarified.

In summary, the novelty is not clearly defined in this paper; differences with existing literature should be highlighted. Experimental results are very good, but it is hard to know where comes the difference in performance with previous (very related) work.
review_title: Not clearly enough related to previous work.

final_score: rating: 6: Marginally above acceptance threshold
review_content: This work presents an embedding approach for semi-supervised learning with neural nets, in the presence of little labeled data. The intuition is to learn a metric embedding that forms “clusters” with the following desiderata: two labeled examples from the class should have a smaller distance in this embedding compared to any example from another class & a given unlabeled example embedding will be closer to all of the embeddings of *some* label (i.e. that a given unlabeled example will be “matched” to one cluster). The paper formulates these intuitions as two differentiable losses and does gradient descent on their sum.

It’s unclear to me how different is this work from the sum of Hoffer & Ailon (2015) (which is eq. 3) and Grandvalet & Bengio (2004) (seems to be related to eq. 4). Would be nice if the authors not only cited the previous work but summarized the actual differences.
In Section 5.1, the authors say that Szegedy et al. (2015) use random noise in the targets -- is that actually true? I think only soft targets are used (which are not noisy).

Does the choice of \lambda_{1,2} make a difference?

How is k for k-NN actually chosen? Is there a validation set?

Figure 1 would benefit from showing where the labeled examples were at the beginning of training (relative to each other / rest of the data).

The submission seems overall OK, but somewhat light on actual data-driven or theoretical insights. I would’ve liked experiments showing the influence of data set sizes at the very least, and ablation experiments that showed the influence of each of the corresponding losses.



review_title: null

Meta-review: This paper studies semi-supervised learning by combining ideas from metric embedding and entropy. The model attempts to embed samples with visible labels into corresponding clusters, while embedding samples without labels close to some of the cluster centroids. Strong numerical results are reported on small-scale semi-supervised image tasks. 
 
 The reviewers acknowledged the effectiveness of the method and the strong numerical results, but expressed concerns about the lack of originality with respect to previous works. 
 
 The AC found the work pleasantly well presented and strikingly simple, yet with excellent numerical performance. These two qualities make this work actually refreshing relative to the ever-increasing complexity of current deep learning architectures. In particular, this seems to be the *proper* way to perform semi-supervised learning. 
 However, the model appears extremely similar to [Grandvalet & Bengio, '04], not just in the entropy-minimization regularization for unlabeled examples, but also for the cross-entropy minimization of labeled examples and the semi-supervised objective. The only difference I perceive is that [G & B '04] present a generic recognition model p( z | x), whereas the present submission uses a classification layer of the form 
 (1) p(z | x ) = softmax( -|| Phi(x) - Phi(y_k) ||^2 ), 
 where the y_k are randomly chosen examples of class respectively k. p(z | x) is thus random relative to the choice of 'anchors' y_k. 
 Thus, the present model is rather a particular case, unless I misunderstood. In that case, the main question is to understand how much the excellent numerical performance relies on this particular choice of classification layer. The numerical section should consider a baseline where (1) is replaced by a 'standard' classification softmax of the form p(z | x ) = softmax( < v_k, Phi(x) > ), where v_k are class-specific parameters. 
 
 For these reasons,we recommend inviting this submission to the workshop track, and we encourage the authors to address the previous remark.
Decision: Invite to Workshop Track