Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The paper discusses a new strategy for deep semi-supervised learning that seems related to the deep label propagation method of Iscen et al. 2019, but is more scalable and has a different loss function. 

Each example is associated with a representation vector v_i and a label y_i. The authors' approach essentially works by alternating between two steps:

(1) Representation learning: Updating the v_i (and other model parameters) where the loss is an addition of two terms: 
-standard supervised loss
-term that encourages points with similar labels to have similar v_i

(2) Label Propagation: Uses the representations v_i to compute nearest neighbors and propagate labels. The authors approach takes O(NM) where N is total number of points and M is number of labeled points and can be parallelized to O(NM/P).  This is in contrast to Iscen et al. 2019 which takes O(N^2). 

Experiments show that the authors' approach performs consistently better on ImageNet than existing approaches. With suboptimal preprocessing, it also performs comparable / slightly better than UDA (Xie et al. 2019) (The authors speculate it could do better with the preprocessing that UDA uses)

I am not from this area but found the paper well written and easy to understand. 

review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: The authors propose a local label propagation approach for large-scale semi-supervised learning. The approach learns a representation that tries to minimize a combination of the cross-entropy loss on the labeled data and a negative inner-product-based likelihood between the propagated pseudo-label and other examples with the same true label. The pseudo-labels on the unlabeled data are then calculated with a weighted k-NN scheme, where the weights take a heuristic correction of a soft similarity. Some further computational speedup is done with a memory cache described in an earlier work (Wu 2018b). Experimental results seem significantly superior to the competitors. The design choices are mostly justified with ablation studies.

The whole idea is interesting and the results are promising. From the current manuscript, my remaining concerns are

(1) How much contribution has readily been done by (Wu 2018a, b), and how much is the original design of the authors? From Section 3 (and without reading (Wu 2018a, b)), I cannot find a clear answer to this question. Currently it appears that the additional contribution over Wu's works is marginal.

(2) It is not clear to me how the proposed approach reaches the asserted efficiency over global label propagation approaches. In particular, each P(v_i)*Z in Equation (2) is O(N) to compute. Each w_j(v) in Equation (5) is O(K) to compute after getting all P(v_i)*Z, and then there are N (or at least N-M) such w_j(v) needed. So the total complexity is naively O(N (N-M) K). Even ignoring the K as a small constant, I cannot see how LLP is O(NM). Some running time profiling of LLP versus global LP might be helpful.

(3) For label propagation methods, it is important to understand whether the pseudo-labels are accurate and/or whether the methods might be mis-guided by the pseudo-labels. Is there any evidence on whether the pseudo-labels are accurate (absolutely, or with respect to the confidence)?

(4) For hyper-parameter selection, there is a "Learning rate is initialized to 0.03 and then dropped by a factor of 10 whenever validation performance saturates." But it is not clear how the validation set is formed, and what performance is measured. Is it a performance based on a labeled validation set (and if so, how large is the set) or unlabeled one?

I read the rebuttal. While it does not change my assessment, I thank the authors for clarifying some issues.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: The paper introduces an approach for semi-supervised learning based on local label propagation. The idea is to leverage the geometric structure in the embedding space, such that data near to each other in the embedding space should have the same labels. The labels of the K-nearest labeled examples are weighted to form the propagated pseudo label of the data point. And the objective aims to match the propagated pseudo label and the predicted label from the classification model. An extra term is added to the objective to force data points with similar pseudo labels to get close to each other in the embedding space. The local propagation strategy makes the method scalable compared to similar methods in the literature. The method is tested on different experimental setups and show superior performance than the state of the art baselines. 

I like the idea of learning a consistent embedding space for prediction and label propagation seems interesting and novel to my knowledge. However, the paper uses the technique of local aggregation in [2] to replace the label propagation in [1], which makes it less novel.

Also, the experiments seem to be extensive with additional analysis to the behavior of the proposed method. I noticed that the authors used the unsupervised training proposed in [2] for the first 10 epochs. I wonder how important it is to have this initialization and would like to see ablation studies on whether using this as initialization or not.

[1]Iscen, Ahmet, Giorgos Tolias, Yannis Avrithis, and Ondrej Chum. "Label propagation for deep semi-supervised learning." In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5070-5079. 2019
[2]Zhuang, Chengxu, Alex Lin Zhai, and Daniel Yamins. "Local aggregation for unsupervised learning of visual embeddings." arXiv preprint arXiv:1903.12355 (2019).
review_title: Official Blind Review #1

Meta-review: The paper introduces an approach for semi-supervised learning based on local label propagation. While reviewers appreciate learning a consistent embedding space for prediction and label propagation, a few pointed out that this paper does not make it clear how different it is from preview work (Wu et al, Iscen et al., Zhuang et al.), in addition to complexity calculation, or pseudo-label accuracy. These are important points that weren’t included to the degree that reviewers/readers can understand, and reviewers seem to not change their minds after authors wrote back. This suggests the paper can use additional cycles of polishing/editing to make these points clear. We highly recommend authors to carefully reflect on reviewers both pros and cons of the paper to improve the paper for your future submission. 

Decision: Reject