Local Label Propagation for Large-Scale Semi-Supervised Learning

1 INTRODUCTION . Deep neural networks ( DNNs ) have achieved impressive performance on tasks across a variety of domains , including vision ( Krizhevsky et al. , 2012 ; Simonyan & Zisserman , 2014 ; He et al. , 2016a ; 2017 ) , speech recognition ( Hinton et al. , 2012 ; Hannun et al. , 2014 ; Deng et al. , 2013 ; Noda et al. , 2015 ) , and natural language processing ( Young et al. , 2018 ; Hirschberg & Manning , 2015 ; Conneau et al. , 2016 ; Kumar et al. , 2016 ) . However , these achievements often heavily rely on large-scale labeled datasets , requiring burdensome and expensive annotation efforts . This problem is especially acute in specialized domains such as medical image processing , where annotation may involve performing an invasive process on patients . Semi-supervised learning ( SSL ) seeks to learn useful representations from limited amounts of labeled data , leveraging it in conjunction with extensive unlabeled data . SSL has shown significant promise ( Liu et al. , 2018 ; Iscen et al. , 2019 ; Zhai et al. , 2019 ; Miyato et al. , 2018 ; Tarvainen & Valpola , 2017 ; Lee , 2013 ; Grandvalet & Bengio , 2005 ; Qiao et al. , 2018 ; Xie et al. , 2019 ) . However , gaps to supervised performance levels still remain significant , and many recent SSL methods rely on techniques whose efficiency scales poorly with dataset size and thus can not be readily applied to many real-world machine learning problems ( Liu et al. , 2018 ; Iscen et al. , 2019 ) . Here , we propose a novel SSL algorithm that is specifically adapted for use with large sparsely-labeled datasets . This algorithm , termed Local Label Propagation ( LLP ) , learns a nonlinear embedding of the input data , and exploits the local geometric structure of the latent embedding space to help infer useful pseudo-labels for unlabeled datapoints . LLP borrows the framework of non-parametric embedding learning , which has recently shown utility in unsupervised learning ( Wu et al. , 2018b ; Zhuang et al. , 2019 ) , to first train a deep neural network that embeds labeled and unlabeled examples into a lower-dimensional latent space . LLP then propagates labels from known examples to unknown datapoints , weighting the likelihood of propagation by the local density of known examples . The neural network is then optimized to categorize all datapoints according to their pseudo-labels ( with stronger emphasis on true known labels ) , while simultaneously encouraging datapoints sharing the same ( pseudo- ) labels to aggregate in the latent embedding space . The resulting embedding thus gathers both labeled images within the same class and unlabeled images sharing statistical similarities with the labeled ones . Through iteratively applying the propagation and network training steps , the LLP algorithm builds a good underlying representation for supporting downstream tasks , and trains an accurate classifier for the specific desired task . We apply LLP in the context of object categorization in the ImageNet dataset ( Deng et al. , 2009 ) , learning a high-performing network while discarding most of the labels . We show that LLP substantially outperforms previous scalable semi-supervised algorithms ( Zhai et al. , 2019 ; Miyato et al. , 2018 ; Tarvainen & Valpola , 2017 ; Lee , 2013 ; Grandvalet & Bengio , 2005 ; Qiao et al. , 2018 ) across a wide variety of training regimes , and that LLP-trained features support improved transfer to Places205 , a large-scale scene-recognition task . We also present analyses that provide insights into the learning procedure and justification of key parameter choices . 2 RELATED WORK . Below we describe conceptual relationships between our work and recent related approaches , and identify relevant major alternatives for comparison . Deep Label Propagation . Like LLP , Deep Label Propagation ( Iscen et al. , 2019 ) ( DLP ) also iterates between steps of label propagation and neural network optimization . In contrast to LLP , the DLP label propagation scheme is based on computing pairwise similarity matrices of learned visual features across all ( unlabeled ) examples . Unlike in LLP , the DLP loss function is simply classification with respect to pseudo-labels , without any additional aggregation terms ensuring that the pseudolabeled and true-labeled points have similar statistical structure . The DLP method is effective on comparatively small datasets , such as CIFAR10 and Mini-ImageNet . However , DLP is challenging to apply to large-scale datasets such as ImageNet , since its label propagation method is O ( N2 ) in the number N of datapoints , and is not readily parallelizable . In contrast , LLP is O ( NM ) , where M is the number of labeled datapoints , and is easily parallelized , making its effective complexity O ( NM/P ) , where P is the number of parallel processes . In addition , DLP uniformly propagates labels across the embedding space , while LLP ’ s use of local density-driven propagation weights specifically exploits the geometric structure in the space , improving pseudo-label inference . Deep Metric Transfer and Pseudolabels . The Deep Metric Transfer ( Liu et al. , 2018 ) ( DMT ) and Pseudolabels ( Lee , 2013 ) methods both use non-iterative two-stage procedures . In the first stage , the representation is initialized either with a self-supervised task such as non-parametric instance recognition ( DMT ) , or via direct supervision on the known labels ( Pseudolabels ) . In the second stage , pseudo-labels are obtained either by applying a label propagation algorithm ( DMT ) or naively from the pre-trained classifier ( Pseudolabels ) , and these are then used to fine-tune the network . As in DLP , the label propagation algorithm used by DMT can not be applied to large-scale datasets , and does not specifically exploit local statistical features of the learned representation . While more scalable , the Pseudolabels approach achieves comparatively poor results . A key point of contrast between LLP and the two-stage methods is that in LLP , the representation learning and label propagation processes interact via the iterative training process , an important driver of LLP ’ s improvements . Self-Supervised Semi-Supervised Learning . Self-Supervised Semi-Supervised Learning ( Zhai et al. , 2019 ) ( S4L ) co-trains a network using self-supervised methods on unlabeled images and traditional classification loss on labeled images . Unlike LLP , S4L simply “ copies ” self-supervised learning tasks as parallel co-training loss branches . In contrast , LLP involves a nontrivial interaction between known and unknown labels via label propagation and the combination of categorization and aggregation losses , both factors that are important for improved performance . Consistency-based regularization . Several recent semi-supervised methods rely on dataconsistency regularizations . Virtual Adversarial Training ( VAT ) ( Miyato et al. , 2018 ) adds small input perturbations , requiring outputs to be robust to this perturbation . Mean Teacher ( MT ) ( Tarvainen & Valpola , 2017 ) requires the learned representation to be similar to its exponential moving average during training . Deep Co-Training ( DCT ) ( Qiao et al. , 2018 ) requires the outputs of two views of the same image to be similar , while ensuring outputs vary widely using adversarial pairs . Very recently , Unsupervised Data Augmentation ( UDA ) ( Xie et al. , 2019 ) achieves state-of-the-art performance by incorporating a substantially more complex data augmentation scheme into the data-consistency framework , and employing computationally expensive but practically impactful details such as the use of very large batch size during optimization . These methods all use unlabeled data in a “ point-wise ” fashion , applying the proposed consistency metric separately on each . They thus differ significantly from ( and are thus likely complementary to ) LLP , or indeed any method that explicitly relates unlabeled to labeled points . LLP benefits from training a shared embedding space that aggregates statistically similar unlabeled datapoints together with labeled ( putative ) counterparts . As a result , adding more unlabeled images consistently increases the performance of LLP , unlike for ( e.g . ) MT . 3 METHODS . We first give an overview of the LLP method . At a high level , LLP learns a model fθ ( ⋅ ) from labeled examples XL = { x1 , . . . , xM } , their associated labels YL = { y1 , . . . , yM } , and unlabeled examples XU = { xM+1 , . . . , xN } . fθ ( ⋅ ) is realized via a deep neural network whose parameters θ are network weights . For each input x , fθ ( x ) generates two outputs ( Fig . 1 ) : an “ embedding output ” , realized as a vector v in a D-dimensional sphere , and a category prediction output ŷ . In learning fθ ( ⋅ ) , the LLP procedure repeatedly alternates between two steps : label propagation and representation learning . First , known labels YL are propagated fromXL toXU , creating pseudo-labels YU = { yM+1 , . . . , yN } . Then , network parameters θ are updated to minimize a loss function balancing category prediction accuracy evaluated on the ŷ outputs , and a metric of statistical consistency evaluated on the v outputs . In addition to pseudo-labels , the label propagation step also generates [ 0,1 ] -valued confidence scores ci for each example xi . For labeled points , confidence scores CL = { c1 , . . . , cM } are automatically set to 1 , while for pseudo-labeled examples , confidence scores CU = { cM+1 , cM+2 , ... , cN } are computed from the local geometric structure of the embedded points , reflecting how close the embedding vectors of the pseudo-labeled points are to those of their putative labeled counterparts . The confidence values are then used as loss weights during representation learning . Representation Learning . Assume that datapoints X = XU ∪ XL , labels and pseudo-labels Y = YU ∪ YL , and confidences C = CU ∪ CL are given . Let V = { v1 , . . . , vN } denote the set of corresponding embedded vectors , and Ŷ = { ŷ1 , . . . , ŷN } denote the set of corresponding category prediction outputs . In the representation learning step , we update the network embedding parameters by simultaneously minimizing the standard cross-entropy loss LC ( Y , Ŷ ) between predicted and propagated pseudo-labels , while maximizing a global aggregation metric LA ( V ∣Y ) to enforce overall consistency between known labels and pseudo-labels . The definition of LA ( V ∣Y ) is based on the non-parametric softmax operation proposed in ( Wu et al. , 2018b ; a ) . Specifically , we define the joint probability of any two embedding vectors vi and vj as : P ( vi , vj ) = exp ( vTi vj/τ ) Z , Z = N ∑ k=1 N ∑ l=1 exp ( vTk vl/τ ) ( 1 ) where temperature τ ∈ ( 0 , 1 ] is a fixed hyperparameter . Using this definition , we can get the probability of vi and the conditional probability of vj given vi as : P ( vi ) = N ∑ j=1 P ( vi , vj ) = ∑Nj=1 exp ( vTi vj/τ ) Z , P ( vj∣vi ) = P ( vi , vj ) P ( vi ) = exp ( vTi vj/τ ) ∑Nl=1 exp ( vTi vl/τ ) ( 2 ) Additionally , for S ⊂ X , its probability given vi is P ( S∣vi ) = ∑vj∈S P ( vj∣vi ) . We then define the aggregation metric as the ( negative ) log likelihood of the examples whose pseudo-labels are also y , the pseudo-label of the current example v : LA ( v ) = −log ( P ( A∣v ) ) , where A = { xi∣yi = y } . Optimizing LA ( v ) encourages the embedding corresponding to a given datapoint to selectively become close to embeddings of other datapoints with the same pseudo-label ( Fig . 1 ) . The cross-entropy and aggregation loss terms are scaled on a per-example basis by the confidence score , and an L2 weight regularization penalty is added . Thus , the final loss for example x is : L ( x∣θ ) = c ⋅ [ LC ( y , ŷ ) + LA ( v ) ] + λ∥θ∥22 , where λ is a regularization hyperparameter . Label Propagation . To understand how LLP generates pseudo-labels YU and confidence scores CU , it is useful to start from the weighted K-Nearest-Neighbor classification algorithm ( Wu et al. , 2018b ) , in which a “ vote ” is obtained from the top K nearest labeled examples for each unlabeled example x , denoted NK ( x ) . The vote of each xi ∈ NK ( x ) is weighted by the corresponding probabilities P ( vi∣v ) . Assuming S classes , the total weight for pseudo-labeled v as class j is thus : wj ( v ) = ∑ i∈I ( j ) P ( vi∣v ) , where I ( j ) = { i∣xi ∈ NK ( v ) , yi = j } ( 3 ) Therefore , the probability pj ( v ) that datapoint x is of class j , the associated inferred pseudolabel y , and the corresponding confidence c , may be defined as : pj ( v ) = wj ( v ) /∑Sk=1 wk ( v ) , y = arg maxj pj ( v ) , and c = py ( v ) . Although intuitive , weighted-KNN introduces a positive correlation between the local density of a labeled example and the number of unlabeled examples whose pseudo-labels are propagated from this labeled example . Moreover , if the labeled examples within one category have higher densities than other categories , there will be more unlabeled examples pseudolabled as this category . To avoid this correlation , we additionally penalize the labeled examples with higher densities through dividing the KNN weight of a labeled example by its local density . To formalize this penalization , we divide P ( vi∣v ) in the definition of wj ( v ) with a local densityweighted probability : P L ( vi∣v ) = P ( vi∣v ) /ρ ( vi ) where ρ ( vi ) = ∑ j∈NT ( vi ) P ( vj , vi ) ( 4 ) where NT ( vi ) are T nearest unlabeled examples and denominator ρ ( vi ) is a measure of the local embedding density . For consistency , we replace NK ( v ) with NLK ( v ) , which contains the K labeled neighbors with highest locally-weighted probability , to ensure that the votes come from the most relevant labeled examples . The final form of the LLP propagation weight equation is thus : wj ( v ) = ∑ i∈I ( j ) P ( vi∣v ) ∑k∈NT ( vi ) P ( vk , vi ) , where I ( j ) = { i∣i ∈ NLK ( v ) , yi = j } ( 5 ) The intuition behind the local density weighting idea is further quantitatively explored in §5 . Memory Bank . Both described steps implicitly require access to all the embedded vectors V at every step . However , recomputing V becomes intractable for bigger dataset . We address this issue by approximating realtime V with a memory bank V̄ that keeps a running average of the embeddings . As this procedure is taken from Wu et al . ( 2018b ) , we refer readers there for a detailed description .

Summary: The authors propose a local label propagation approach for large-scale semi-supervised learning. The approach learns a representation that tries to minimize a combination of the cross-entropy loss on the labeled data and a negative inner-product-based likelihood between the propagated pseudo-label and other examples with the same true label. The pseudo-labels on the unlabeled data are then calculated with a weighted k-NN scheme, where the weights take a heuristic correction of a soft similarity. Some further computational speedup is done with a memory cache described in an earlier work (Wu 2018b). Experimental results seem significantly superior to the competitors. The design choices are mostly justified with ablation studies.