Local Label Propagation for Large-Scale Semi-Supervised Learning

1 INTRODUCTION . Deep neural networks ( DNNs ) have achieved impressive performance on tasks across a variety of domains , including vision ( Krizhevsky et al. , 2012 ; Simonyan & Zisserman , 2014 ; He et al. , 2016a ; 2017 ) , speech recognition ( Hinton et al. , 2012 ; Hannun et al. , 2014 ; Deng et al. , 2013 ; Noda et al. , 2015 ) , and natural language processing ( Young et al. , 2018 ; Hirschberg & Manning , 2015 ; Conneau et al. , 2016 ; Kumar et al. , 2016 ) . However , these achievements often heavily rely on large-scale labeled datasets , requiring burdensome and expensive annotation efforts . This problem is especially acute in specialized domains such as medical image processing , where annotation may involve performing an invasive process on patients . Semi-supervised learning ( SSL ) seeks to learn useful representations from limited amounts of labeled data , leveraging it in conjunction with extensive unlabeled data . SSL has shown significant promise ( Liu et al. , 2018 ; Iscen et al. , 2019 ; Zhai et al. , 2019 ; Miyato et al. , 2018 ; Tarvainen & Valpola , 2017 ; Lee , 2013 ; Grandvalet & Bengio , 2005 ; Qiao et al. , 2018 ; Xie et al. , 2019 ) . However , gaps to supervised performance levels still remain significant , and many recent SSL methods rely on techniques whose efficiency scales poorly with dataset size and thus can not be readily applied to many real-world machine learning problems ( Liu et al. , 2018 ; Iscen et al. , 2019 ) . Here , we propose a novel SSL algorithm that is specifically adapted for use with large sparsely-labeled datasets . This algorithm , termed Local Label Propagation ( LLP ) , learns a nonlinear embedding of the input data , and exploits the local geometric structure of the latent embedding space to help infer useful pseudo-labels for unlabeled datapoints . LLP borrows the framework of non-parametric embedding learning , which has recently shown utility in unsupervised learning ( Wu et al. , 2018b ; Zhuang et al. , 2019 ) , to first train a deep neural network that embeds labeled and unlabeled examples into a lower-dimensional latent space . LLP then propagates labels from known examples to unknown datapoints , weighting the likelihood of propagation by the local density of known examples . The neural network is then optimized to categorize all datapoints according to their pseudo-labels ( with stronger emphasis on true known labels ) , while simultaneously encouraging datapoints sharing the same ( pseudo- ) labels to aggregate in the latent embedding space . The resulting embedding thus gathers both labeled images within the same class and unlabeled images sharing statistical similarities with the labeled ones . Through iteratively applying the propagation and network training steps , the LLP algorithm builds a good underlying representation for supporting downstream tasks , and trains an accurate classifier for the specific desired task . We apply LLP in the context of object categorization in the ImageNet dataset ( Deng et al. , 2009 ) , learning a high-performing network while discarding most of the labels . We show that LLP substantially outperforms previous scalable semi-supervised algorithms ( Zhai et al. , 2019 ; Miyato et al. , 2018 ; Tarvainen & Valpola , 2017 ; Lee , 2013 ; Grandvalet & Bengio , 2005 ; Qiao et al. , 2018 ) across a wide variety of training regimes , and that LLP-trained features support improved transfer to Places205 , a large-scale scene-recognition task . We also present analyses that provide insights into the learning procedure and justification of key parameter choices . 2 RELATED WORK . Below we describe conceptual relationships between our work and recent related approaches , and identify relevant major alternatives for comparison . Deep Label Propagation . Like LLP , Deep Label Propagation ( Iscen et al. , 2019 ) ( DLP ) also iterates between steps of label propagation and neural network optimization . In contrast to LLP , the DLP label propagation scheme is based on computing pairwise similarity matrices of learned visual features across all ( unlabeled ) examples . Unlike in LLP , the DLP loss function is simply classification with respect to pseudo-labels , without any additional aggregation terms ensuring that the pseudolabeled and true-labeled points have similar statistical structure . The DLP method is effective on comparatively small datasets , such as CIFAR10 and Mini-ImageNet . However , DLP is challenging to apply to large-scale datasets such as ImageNet , since its label propagation method is O ( N2 ) in the number N of datapoints , and is not readily parallelizable . In contrast , LLP is O ( NM ) , where M is the number of labeled datapoints , and is easily parallelized , making its effective complexity O ( NM/P ) , where P is the number of parallel processes . In addition , DLP uniformly propagates labels across the embedding space , while LLP ’ s use of local density-driven propagation weights specifically exploits the geometric structure in the space , improving pseudo-label inference . Deep Metric Transfer and Pseudolabels . The Deep Metric Transfer ( Liu et al. , 2018 ) ( DMT ) and Pseudolabels ( Lee , 2013 ) methods both use non-iterative two-stage procedures . In the first stage , the representation is initialized either with a self-supervised task such as non-parametric instance recognition ( DMT ) , or via direct supervision on the known labels ( Pseudolabels ) . In the second stage , pseudo-labels are obtained either by applying a label propagation algorithm ( DMT ) or naively from the pre-trained classifier ( Pseudolabels ) , and these are then used to fine-tune the network . As in DLP , the label propagation algorithm used by DMT can not be applied to large-scale datasets , and does not specifically exploit local statistical features of the learned representation . While more scalable , the Pseudolabels approach achieves comparatively poor results . A key point of contrast between LLP and the two-stage methods is that in LLP , the representation learning and label propagation processes interact via the iterative training process , an important driver of LLP ’ s improvements . Self-Supervised Semi-Supervised Learning . Self-Supervised Semi-Supervised Learning ( Zhai et al. , 2019 ) ( S4L ) co-trains a network using self-supervised methods on unlabeled images and traditional classification loss on labeled images . Unlike LLP , S4L simply “ copies ” self-supervised learning tasks as parallel co-training loss branches . In contrast , LLP involves a nontrivial interaction between known and unknown labels via label propagation and the combination of categorization and aggregation losses , both factors that are important for improved performance . Consistency-based regularization . Several recent semi-supervised methods rely on dataconsistency regularizations . Virtual Adversarial Training ( VAT ) ( Miyato et al. , 2018 ) adds small input perturbations , requiring outputs to be robust to this perturbation . Mean Teacher ( MT ) ( Tarvainen & Valpola , 2017 ) requires the learned representation to be similar to its exponential moving average during training . Deep Co-Training ( DCT ) ( Qiao et al. , 2018 ) requires the outputs of two views of the same image to be similar , while ensuring outputs vary widely using adversarial pairs . Very recently , Unsupervised Data Augmentation ( UDA ) ( Xie et al. , 2019 ) achieves state-of-the-art performance by incorporating a substantially more complex data augmentation scheme into the data-consistency framework , and employing computationally expensive but practically impactful details such as the use of very large batch size during optimization . These methods all use unlabeled data in a “ point-wise ” fashion , applying the proposed consistency metric separately on each . They thus differ significantly from ( and are thus likely complementary to ) LLP , or indeed any method that explicitly relates unlabeled to labeled points . LLP benefits from training a shared embedding space that aggregates statistically similar unlabeled datapoints together with labeled ( putative ) counterparts . As a result , adding more unlabeled images consistently increases the performance of LLP , unlike for ( e.g . ) MT . 3 METHODS . We first give an overview of the LLP method . At a high level , LLP learns a model fθ ( ⋅ ) from labeled examples XL = { x1 , . . . , xM } , their associated labels YL = { y1 , . . . , yM } , and unlabeled examples XU = { xM+1 , . . . , xN } . fθ ( ⋅ ) is realized via a deep neural network whose parameters θ are network weights . For each input x , fθ ( x ) generates two outputs ( Fig . 1 ) : an “ embedding output ” , realized as a vector v in a D-dimensional sphere , and a category prediction output ŷ . In learning fθ ( ⋅ ) , the LLP procedure repeatedly alternates between two steps : label propagation and representation learning . First , known labels YL are propagated fromXL toXU , creating pseudo-labels YU = { yM+1 , . . . , yN } . Then , network parameters θ are updated to minimize a loss function balancing category prediction accuracy evaluated on the ŷ outputs , and a metric of statistical consistency evaluated on the v outputs . In addition to pseudo-labels , the label propagation step also generates [ 0,1 ] -valued confidence scores ci for each example xi . For labeled points , confidence scores CL = { c1 , . . . , cM } are automatically set to 1 , while for pseudo-labeled examples , confidence scores CU = { cM+1 , cM+2 , ... , cN } are computed from the local geometric structure of the embedded points , reflecting how close the embedding vectors of the pseudo-labeled points are to those of their putative labeled counterparts . The confidence values are then used as loss weights during representation learning . Representation Learning . Assume that datapoints X = XU ∪ XL , labels and pseudo-labels Y = YU ∪ YL , and confidences C = CU ∪ CL are given . Let V = { v1 , . . . , vN } denote the set of corresponding embedded vectors , and Ŷ = { ŷ1 , . . . , ŷN } denote the set of corresponding category prediction outputs . In the representation learning step , we update the network embedding parameters by simultaneously minimizing the standard cross-entropy loss LC ( Y , Ŷ ) between predicted and propagated pseudo-labels , while maximizing a global aggregation metric LA ( V ∣Y ) to enforce overall consistency between known labels and pseudo-labels . The definition of LA ( V ∣Y ) is based on the non-parametric softmax operation proposed in ( Wu et al. , 2018b ; a ) . Specifically , we define the joint probability of any two embedding vectors vi and vj as : P ( vi , vj ) = exp ( vTi vj/τ ) Z , Z = N ∑ k=1 N ∑ l=1 exp ( vTk vl/τ ) ( 1 ) where temperature τ ∈ ( 0 , 1 ] is a fixed hyperparameter . Using this definition , we can get the probability of vi and the conditional probability of vj given vi as : P ( vi ) = N ∑ j=1 P ( vi , vj ) = ∑Nj=1 exp ( vTi vj/τ ) Z , P ( vj∣vi ) = P ( vi , vj ) P ( vi ) = exp ( vTi vj/τ ) ∑Nl=1 exp ( vTi vl/τ ) ( 2 ) Additionally , for S ⊂ X , its probability given vi is P ( S∣vi ) = ∑vj∈S P ( vj∣vi ) . We then define the aggregation metric as the ( negative ) log likelihood of the examples whose pseudo-labels are also y , the pseudo-label of the current example v : LA ( v ) = −log ( P ( A∣v ) ) , where A = { xi∣yi = y } . Optimizing LA ( v ) encourages the embedding corresponding to a given datapoint to selectively become close to embeddings of other datapoints with the same pseudo-label ( Fig . 1 ) . The cross-entropy and aggregation loss terms are scaled on a per-example basis by the confidence score , and an L2 weight regularization penalty is added . Thus , the final loss for example x is : L ( x∣θ ) = c ⋅ [ LC ( y , ŷ ) + LA ( v ) ] + λ∥θ∥22 , where λ is a regularization hyperparameter . Label Propagation . To understand how LLP generates pseudo-labels YU and confidence scores CU , it is useful to start from the weighted K-Nearest-Neighbor classification algorithm ( Wu et al. , 2018b ) , in which a “ vote ” is obtained from the top K nearest labeled examples for each unlabeled example x , denoted NK ( x ) . The vote of each xi ∈ NK ( x ) is weighted by the corresponding probabilities P ( vi∣v ) . Assuming S classes , the total weight for pseudo-labeled v as class j is thus : wj ( v ) = ∑ i∈I ( j ) P ( vi∣v ) , where I ( j ) = { i∣xi ∈ NK ( v ) , yi = j } ( 3 ) Therefore , the probability pj ( v ) that datapoint x is of class j , the associated inferred pseudolabel y , and the corresponding confidence c , may be defined as : pj ( v ) = wj ( v ) /∑Sk=1 wk ( v ) , y = arg maxj pj ( v ) , and c = py ( v ) . Although intuitive , weighted-KNN introduces a positive correlation between the local density of a labeled example and the number of unlabeled examples whose pseudo-labels are propagated from this labeled example . Moreover , if the labeled examples within one category have higher densities than other categories , there will be more unlabeled examples pseudolabled as this category . To avoid this correlation , we additionally penalize the labeled examples with higher densities through dividing the KNN weight of a labeled example by its local density . To formalize this penalization , we divide P ( vi∣v ) in the definition of wj ( v ) with a local densityweighted probability : P L ( vi∣v ) = P ( vi∣v ) /ρ ( vi ) where ρ ( vi ) = ∑ j∈NT ( vi ) P ( vj , vi ) ( 4 ) where NT ( vi ) are T nearest unlabeled examples and denominator ρ ( vi ) is a measure of the local embedding density . For consistency , we replace NK ( v ) with NLK ( v ) , which contains the K labeled neighbors with highest locally-weighted probability , to ensure that the votes come from the most relevant labeled examples . The final form of the LLP propagation weight equation is thus : wj ( v ) = ∑ i∈I ( j ) P ( vi∣v ) ∑k∈NT ( vi ) P ( vk , vi ) , where I ( j ) = { i∣i ∈ NLK ( v ) , yi = j } ( 5 ) The intuition behind the local density weighting idea is further quantitatively explored in §5 . Memory Bank . Both described steps implicitly require access to all the embedded vectors V at every step . However , recomputing V becomes intractable for bigger dataset . We address this issue by approximating realtime V with a memory bank V̄ that keeps a running average of the embeddings . As this procedure is taken from Wu et al . ( 2018b ) , we refer readers there for a detailed description .

Summary: The paper introduces an approach for semi-supervised learning based on local label propagation. The idea is to leverage the geometric structure in the embedding space, such that data near to each other in the embedding space should have the same labels. The labels of the K-nearest labeled examples are weighted to form the propagated pseudo label of the data point. And the objective aims to match the propagated pseudo label and the predicted label from the classification model. An extra term is added to the objective to force data points with similar pseudo labels to get close to each other in the embedding space. The local propagation strategy makes the method scalable compared to similar methods in the literature. The method is tested on different experimental setups and show superior performance than the state of the art baselines.