Data driven semi-supervised learning

1 Introduction . In recent years machine learning has found gainful application in diverse domains . A major bottleneck of the currently used approaches is the heavy dependence on expensive labeled data . Advances in cheap computing and storage have made it relatively easier to store and process large amounts of unlabeled data . Therefore , an important focus of the present research community is to develop general domain-independent methods to learn effectively from the unlabeled data , along with a small amount of labels . Achieving this goal would significantly elevate the state-of-the-art machine intelligence , which currently lags behind the human capability of learning from a few labeled examples . Our work is a step in this direction , and provides algorithms and guarantees that enable fundamental techniques for semi-supervised learning to provably adapt to problem domains . Graph-based approaches have been popular for learning from unlabeled data for the past two decades [ Zhu and Goldberg , 2009 ] . Labeled and unlabeled examples form the graph nodes and ( possibly weighted ) edges denote the feature similarity between examples . The graph therefore captures how each example is related to other examples , and by optimizing a suitably regularized objective over it one obtains an efficient discriminative , nonparametric method for learning the labels . There are several well-studied ways to define and regularize an objective on the graph [ Chapelle et al. , 2010 ] , 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . and all yield comparable results which strongly depend on the graph used . A general formulation is described as follows , variations on which are noted under related work . Problem formulation Given sets L and U of labeled and unlabeled examples respectively , and a similarity metric d over the data , the goal is to use d to extrapolate labels in L to U . A graph G is constructed with L + U as the nodes and weighted edges W with w ( u , v ) = g ( d ( u , v ) ) for some g : R≥0 → R≥0 . We seek labels f ( · ) for nodes u of G which minimize a regularized loss function l ( f ) = α ∑ v∈L l̂ ( f ( v ) , yv ) + βH ( f , W ) + γ ‖f‖ 2 , under some constraints on f . The objective H captures the smoothness ( regularization ) induced by the graph ( see Table 1 for examples ) and l̂ ( f ( v ) , yv ) is the misclassification loss ( computed here on labeled examples ) . The graph G takes a central position in this formulation . However , the majority of the research effort on this problem has focused on how to design and optimize the regularized loss function l ( f ) , the effectiveness of which crucially depends on G. There is no known principled study on how to build G and prior work largely treats this as a domain-specific art [ Chapelle et al. , 2010 ] . Is it possible to acquire the required domain expertise , without involving human experts ? In this work we provide an affirmative answer by formulating graph selection as data-driven design . More precisely , we are required to solve not only one instance , but multiple instances of the underlying algorithmic problem that come from the same domain [ Gupta and Roughgarden , 2016 , Balcan , 2020 ] . We show learning a near-optimal graph over commonly used infinite parameterized families is possible in both online and distributional settings . In the process we generalize and extend data-driven learning techniques , and obtain practical methods to build the graphs with strong guarantees . In particular , we show how the techniques can learn several parameters at once , and also learn a broader class of parameters than previously known . Our contributions and key challenges . We present a first theoretically grounded work for graphbased learning from limited labeled data , while extending general data-driven design techniques . Data-driven algorithm design . Firstly , for one dimensional loss functions , we show a novel structural result which applies when discontinuities ( for loss as function of the algorithm parameter ) occur along roots of exponential polynomials with random coefficients with bounded joint distributions ( previously known only for algebraic polynomials in Balcan et al . [ 2020b ] ) . This is crucial for showing learnability in the Gaussian graph kernels setting . Secondly , Balcan et al . [ 2020b ] only applies when the discontinuities occur along algebraic curves with random coefficients in just two dimensions . By a novel algebraic and learning theoretic argument we are able to analyze higher ( arbitrary constant number of ) dimensions , making the technique much more generally applicable . Semi-supervised learning . We examine commonly used parameterized graph families , denoted by general notation G ( ρ ) , where ρ corresponds to a semi-supervised learning algorithm . We consider online and distributional settings , providing efficient algorithms to obtain low regret and low error respectively for learning ρ . Most previously studied settings involve polynomially many discontinuities for loss as function of the hyperparameter ρ on a fixed instance , implying efficient algorithms , which may not be the case for our setting . To resolve this , we describe efficient semi-bandit implementations , and in particular introduce a novel min-cut and flow recomputation algorithm on graphs with continuously changing edge weights which may be of independent interest . For the distributional setting , we provide asymptotically tight bounds on the pseudodimension of the parameter learning problem . Our lower bounds expose worst case challenges , and involve precise constructions of problem instances by setting node similarities which make assigning labels provably hard . Our techniques are extremely general and are shown to apply for nearly all combinations of optimization algorithms ( Table 1 ) and parametric graph families ( Definition 1 ) . Related work Semi-supervised learning is a paradigm for learning from labeled and unlabeled data ( Zhu and Goldberg [ 2009 ] ) . It resembles human learning behavior more closely than fully supervised and fully unsupervised models ( Zhu et al . [ 2007 ] , Gibson et al . [ 2013 ] ) . A popular approach for semi-supervised learning is to optimize a graph-based objective . Several methods have been proposed to predict labels given a graph including st-mincuts ( Blum and Chawla [ 2001 ] ) , soft mincuts that optimize a harmonic objective ( Zhu et al . [ 2003 ] ) , label propagation ( Xiaojin and Zoubin [ 2002 ] ) , and many more ( Shi and Malik [ 2000 ] , Belkin et al . [ 2006 ] ) . All algorithms have comparable performance provided the graph G encodes the problem well [ Zhu and Goldberg , 2009 ] . However , it is not clear how to create the graph itself on which the extensive literature stands , barring some heuristics ( Zhu et al . [ 2005 ] , Zemel and Carreira-Perpiñán [ 2004 ] ) . Sindhwani et al . [ 2005 ] construct warped kernels aligned with the data geometry , but the performance may vary strongly with warping and it is not clear how to optimize over it . We provide the first techniques that yield provably near-optimal graphs . Gupta and Roughgarden [ 2016 , 2017 ] define a formal learning framework for selecting algorithms from a family of heuristics or setting hyperparameters . It is further developed by Balcan et al . [ 2017 ] and noted as a fundamental algorithm design perspective [ Blum , 2020 ] . It has been successfully applied to several combinatorial problems like integer programming and clustering [ Balcan et al. , 2018a , 2019 , 2018c ] and for giving powerful guarantees like adversarial robustness , adaptive learning and differential privacy [ Balcan et al. , 2018b , 2020a , c , Vitercik et al. , 2019 , Balcan et al. , 2020e , d ] . Balcan et al . [ 2018b , 2020b ] introduce general data-driven design techniques under some smoothness assumptions . We extend the techniques to significantly broader problem settings , and investigate the structure of graph-based label learning formulation to apply the new techniques . 2 Setup and definitions . We are given some unlabeled points U ⊂ X and labeled points L ⊂ X ×Y , such that |L|+ |U | = n. One constructs a graph G by placing ( possibly weighted ) edges w ( u , v ) between pairs of data points u , v which are ‘ similar ’ , and labels for the unlabeled examples are obtained by optimizing some graphbased score . We have an oracle O which on querying provides us the labeled and unlabeled examples , and we need to pick graph G ( ρ ) from some family G of graphs , parameterized using a parameter ρ ∈ P . We commit to using some graph labeling algorithm A ( G , L , U ) ( abbreviated as AG , L , U ) which provides labels for examples in U , and we should pick a ρ such that A ( G ( ρ ) , L , U ) results in small error in its predictions on U . More formally , for a loss function l : Y × Y → [ 0 , 1 ] and a target labeling τ : U → Y , we need to find argminρ∈P lA ( G ( ρ ) , L , U ) : = ∑ U l ( AG ( ρ ) , L , U ( u ) , τ ( u ) ) . We will now describe some graph families G and algorithms AG , L , U . We assume there is a feature based similarity function d : X × X → R≥0 , a metric which monotonically captures pairwise similarity . Commonly used parametric methods to build a graph using the similarity function follow . Definition 1 . Graph kernels.1 a ) Threshold graph , G ( r ) . Parameterized by a threshold r , we set w ( u , v ) = I [ d ( u , v ) ≤ r ] . b ) Polynomial kernel , G ( α̃ ) . w ( u , v ) = ( d̃ ( u , v ) + α̃ ) d for fixed degree d , parameterized by α̃ . c ) Gaussian RBF or exponential kernel , G ( σ ) . w ( u , v ) = e−d ( u , v ) 2/σ2 , parameterized by σ . Remark 1 . Another popular family of graphs used in practice is the k nearest neighbor graphs , where k ∈ { 0 , 1 , . . . , n− 1 } , n is the number of nodes in the graph , is the parameter . Even though k-NN graphs may result in different graphs the ones considered in the paper , learning how to build an optimal graph over the algorithm family G ( k ) is much simpler . Online learning of the parameter k in this setting can be recognized as an instance of learning with experts advice for a finite hypothesis class ( Section 3.1 of Shalev-Shwartz et al . [ 2011 ] ) , where an upper bound of O ( √ T log n ) is known for the Weighted Majority algorithm . Online-to-batch conversion provides generalization guarantees in the distributional setting ( Section 5 of Shalev-Shwartz et al . [ 2011 ] ) . We remark that our algorithm families need more sophisticated analysis due to continuous ranges of the algorithm parameters . 1With some notational abuse , we have d as the integer polynomial degree , and d ( · , · ) as the similarity function . Common choices are setting d ( u , v ) as the Euclidean norm and d̃ ( u , v ) as the dot product when u , v ∈ Rn . The threshold graph adds ( unweighted ) edges to G only when the examples are closer than some r ≥ 0 . We refer to this setting by the unweighted graph setting , and the others by the weighted graph setting . The similarity function d̃ ( u , v ) in Definitions 1b increases monotonically with similarity of examples ( as opposed to the other two ) . Once the graph is constructed using one of the above kernels , we can assign labels using some algorithm AG , L , U . A popular , effective approach is to optimize a quadratic objective 12 ∑ u , v w ( u , v ) ( f ( u ) − f ( v ) ) 2. f may be discrete , f ( u ) ∈ { 0 , 1 } corresponds to finding a mincut separating the oppositely labeled vertices [ Blum and Chawla , 2001 ] , or f ∈ [ 0 , 1 ] may be continuous and we can round f to obtain the labels [ Zhu et al. , 2003 ] . These correspond to the mincut and harmonic function algorithms respectively from Table 1 . We also need some well-known definitions from prior work ( Appendix A ) . In particular , we use dispersion from [ Balcan et al. , 2020b ] . The sequence of random loss functions l1 , . . . , lT is β-dispersed for the Lipschitz constant L if , for all T and for all ≥ T−β , E [ maxρ , ρ′∈C , ‖ρ−ρ′‖2≤ ∣∣ { t ∈ [ T ] | lt ( ρ ) − lt ( ρ′ ) > L ‖ρ− ρ′‖2 } ∣∣ ] ≤ Õ ( T ) .

Summary: This work presents novel theoretical tools for data-driven algorithm design. In particular, the authors generalise a dispersion-based analysis from intervals to axis-aligned paths of arbitrary dimension.   Using these new results, the authors study graph-based semi-supervised learning from a data-driven perspective and achieve strong bounds in the online and PAC setting. Their goal is to perform well on a fixed distribution of semi-supervised problem instances (or minimising the regret in the online setting) by selecting the most appropriate graph from a parametrised graph family (in particular: threshold graphs and graphs with edge weights given by the polynomial or RBF kernel). Many additional results, such as first experiments and lower bounds, are discussed.