Towards Sharper Generalization Bounds for Structured Prediction

n ) . In each scenario , applications are provided to suggest that these conditions are easy to be satisfied . 1 Introduction . Structured prediction [ 18 , 12 , 17 ] covers a wide range of machine learning fields , such as computer vision , natural language processing , and computational biology . Several examples of structured prediction problems include part-of-speech tagging , dependency parsing , named entity recognition and machine translation in natural language understanding , image segmentation and objection recognition in computer vision , and protein folding in computational biology . An essential property of structured prediction is that the output space admits some structure , such as strings , graphs , trees , or sequences [ 18 , 12 , 13 ] . Meanwhile , another property common to the above tasks is that the natural loss function in each case admits a decomposition along with the output substructures . The complex output structure and corresponding loss function make structured prediction different from the widely studied binary or multi-class classification problems . The design of structured prediction algorithms have been thrived for many years , including conditional random fields [ 30 ] , structured support vector machines [ 61 ] , kernel-regression algorithm [ 19 ] , searchbased structured prediction [ 20 ] , maximum-margin markov network [ 60 ] , image segmentation [ 43 ] , part-of-speech-tagging [ 27 ] , and machine translation [ 68 ] . Compared to the prosperity and development of structured prediction algorithm designing , the theoretical analysis of structured prediction appears to be not sufficiently well-documented [ 13 ] , especially the studying on sharper generalization bounds . However , it is known that the theoretical study of structured prediction algorithms is essential [ 18 , 12 , 13 , 1 ] . Several theoretical studies of structured prediction consider a specific algorithm and a simple loss such as Hamming loss [ 60 , 16 , 14 ] . Recently , increasing work has considered the analysis of general losses . These works can be roughly cast into four categories : PAC-Bayesian approach , factor graph ∗Corresponding Author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . decomposition approach , implicit embedding approach , and stability approach . [ 22 , 45 , 26 , 4 ] provide PAC-Bayesian guarantees for arbitrary losses through the analysis of randomized algorithms using count-based hypotheses . [ 12 , 13 , 53 , 52 , 8 , 11 , 58 , 7 ] use implicit loss embedding to construct a connection between discrete output of structured prediction and regression , to investigate the generalization performance of structured prediction . [ 18 , 3 , 51 ] present generalization bounds for general losses based on factor graph decomposition approach , which is proposed in the seminal work [ 18 ] . [ 41 , 42 , 40 ] use the stability tool to investigate the generalization bounds and provide interesting theoretical insights that structure prediction is possible to generalize from a few large examples , potentially even just one . A special case of structured prediction problem worth mentioning is multi-class classification [ 34 , 33 , 36 , 55 , 50 , 65 , 48 , 49 , 44 , 31 ] , whose scoring function can be seen as a factor graph with both factor node size and substructure size equal to one . Although the aforementioned works have provided generalization bounds from different perspectives , there are still some problems urgently to be solved in structured prediction . ( 1 . ) Existing generalization bounds mostly have a slow convergence rate with respect to ( w.r.t . ) the sample size n. Specifically , in PAC-Bayesian approach , [ 45 , 26 , 4 , 22 ] provide the generalization bounds of order O ( 1√ n ) . In implicit embedding approach , [ 12 , 13 , 52 , 11 , 58 , 7 ] provide the convergence rate of order O ( 1 n1/4 ) , and [ 53 ] of order O ( 1√ n ) . In the factor graph decomposition approach , [ 18 , 51 ] present the generalization upper bounds of order O ( 1√ n ) . Also , in stability approach , [ 41 , 42 , 40 ] show the O ( 1√ n ) order bounds . Therefore , this raises a question : can the convergence rate of structured prediction achieves faster order O ( 1n ) ? ( 2 . ) In structured prediction , the number of possible labels is potentially infinite [ 18 ] . Thus it is important to obtain upper bounds that have a lower order dependency on the cardinality of possible labels . The factor graph decomposition approach can provide the explicit dependency on the properties of the factor graph and help us to explicitly show the dependency on the number of possible labels , which sheds new light on the role played by the graph in generalization [ 18 ] . Therefore , our analysis is based on the factor graph decomposition [ 18 , 3 , 51 ] . However , [ 3 ] focuses on the lower bound , [ 51 ] studies the specific surrogate loss , and the upper bounds in [ 18 ] show a square-root dependency on the cardinality of possible labels . If we consider the vast number of possible labels , the results in [ 18 ] may be divergent and can not explain the good performance of structured prediction algorithms in practice . Thus , can the upper bound of structured prediction presents a lower order dependency on the cardinality of possible labels or even no dependency ? ( 3 . ) Although some work in multi-class classification [ 36 , 55 , 65 ] , a special structured prediction problem , have shown the faster convergence rate of orderO ( 1n ) w.r.t . the sample size n , it is unknown in more difficult and complex structured prediction problems . Additionally , we naturally want to know whether the generalization bounds of structured prediction can combine ( 1 ) with ( 2 ) to show faster convergence rates simultaneously ? This paper intends to answer the three interesting questions . We consider the general loss and present novel margin-based theoretical analysis for structured prediction from three perspectives : Lipschitz continuity , smoothness , and space capacity condition . We first assume the loss function used in structured prediction is Lipschitz continuous and try to answer question ( 2 ) . Under this condition , we improve the generalization bounds of structured prediction from a square-root dependency to a logarithmic dependency on the label set cardinality , but a slow convergence rate of order O ( 1√ n ) w.r.t . the sample size n. Then , we assume the loss is smooth and intend to answer questions ( 1 ) and ( 3 ) . Under this assumption , we obtain sharper bounds that not only have the faster convergence rate of order O ( 1n ) on the sample size n but also have the logarithmic dependency on the label set cardinality . Furthermore , we consider the space capacity-dependent assumptions : logarithmic covering number and polynomial covering number assumptions , which are commonly used in learning theory [ 32 , 69 , 56 ] . This setting attempts to answer questions ( 1-3 ) . Under this condition , we show that the bound can present no dependency on the label set cardinality , simultaneously with faster convergence rates thanO ( 1√ n ) . In the above three perspectives , we all provide applications to suggest that these conditions are easy to be satisfied in practice . This paper is organized as follows . In Section 2 , we introduce the notations and definitions relevant to our discussion . In Section 3 , we present the main results , deriving a series of new learning guarantees for structured prediction . Section 4 concludes this paper . 2 Preliminaries . Notations . Let P be a probability measure defined on a sample space X ×Y with X ∈ Rd being the input space and Y being the output space . A key aspect of structured prediction is that the output space may be sequences , graphs , images , or lists [ 18 ] . We thus assume that Y can be decomposed into l substructures : Y = Y1 × · · · × Yl , where Yi is the set of possible labels that can be assigned to substructure i . Take the simple webpage collective classification task for example [ 59 ] , each Yi is a webpage label and Y is a joint label for an entire website . If we assume each Yi ∈ { 0 , 1 } , the number of possible labels to Y is exponential in the number of substructures l , i.e. , |Y| = 2l . Factor graphs and scoring function . In structured prediction , we aim to learn a scoring function f = X ×Y → R. Let F be a family of the scoring function . For each f ∈ F , we define the predictor f̂ as : for each x ∈ X , f̂ ( x ) = arg maxy∈Y f ( x , y ) . Moreover , we assume that the scoring function f can be decomposed as a sum [ 61 , 60 , 30 , 18 ] , which is standard in structured prediction and can be formulated via the notation of factor graph [ 18 ] . A factor graph G is a bipartite graph , and is represented as a tuple G = ( V , H , E ) , where V = { 1 , ... , l } is a set of variables nodes , H is a set of factor nodes , and E is a set of undirected edges between a variable node and a factor node . Let N ( h ) be the set of variable nodes connected to the factor h by an edge and Yh be the substructure set cross-product Yh = ∏ k∈N ( h ) Yk . Based on the above notations , the scoring function f ( x , y ) can be decomposed as a sum of functions fh [ 18 ] , with an element x ∈ X and an element yh ∈ Yh as arguments : f ( x , y ) = ∑ h∈H fh ( x , yh ) . In this paper , we consider the more generally setting : for each example ( xi , yi ) , the corresponding factor graph may be different , that is , G ( xi , yi ) = Gi = ( [ li ] , Hi , Ei ) . A special case of this setting is that , for example , when the size li of each example is allowed to vary and where the number of possible labels |Y| is potentially infinite . Figure 1 shows different examples of factor graphs . Learning . In order to measure the success of a prediction , we use a loss function L : Y × Y → R+ to measure the dissimilarity of two elements of the output space Y . A distinguishing property of structured prediction is that the natural loss function admits a decomposition along the output substructures [ 18 ] , such as the Hamming loss , defined as : L ( y , y′ ) = 1l ∑l k=1 Iyk 6=y′k for two outputs y , y′ ∈ Y , with y = ( y1 , ... , yl ) and y = ( y′1 , ... , y′l ) , or edit-distance loss in natural language and speech processing applications , or other losses [ 18 , 10 ] . Given a training set S = { ( x1 , y1 ) , ... , ( xn , yn ) } of size n being independently drawn from the underlying distribution P , our goal is to learn a hypothesis f ∈ F with good generalization performance from the sample S by optimizing the loss L. The performance is measured by expected risk , defined as R ( f ) : = E ( x , y ) ∼P [ L ( f̂ ( x ) , y ) ] . However , the above mentioned loss function f → L ( f̂ ( x ) , y ) is typically the 0− 1 loss of f , which is hard to handle in machine learning . Therefore , one usually consider surrogate losses [ 12 , 13 , 18 ] . Margin and loss function class . We now introduce the definition of standard margin and the surrogate loss . The margin of any scoring function f for an example ( x , y ) is defined as : ρf ( x , y ) = f ( x , y ) −max y′ 6=y f ( x , y′ ) . We consider the scoring function f of the form f ( x , y ) = 〈w , Ψ ( x , y ) 〉 , where Ψ is a feature mapping from X × Y to RN such that Ψ ( x , y ) = ∑ h∈H Ψh ( x , yh ) due to the decomposition of f [ 18 ] . And we define the following scoring function space : F = { x 7→ 〈w , Ψ ( x , y ) 〉 : w ∈ RN , ‖w‖p ≤ Λp } , ( 1 ) where ‖w‖p = ( ∑N i=1 |wi|p ) 1 p . Then , the corresponding surrogate loss function class is defined as : Lρ = { ` ρ ( x , y , f ) : = ` ( ρf ( x , y ) ) : f ∈ F } , where ` ρ is the surrogate loss of L. We assume ` ρ is bounded by M , which implies that 0 ≤ L ( f̂ ( x ) , y ) = L ( f̂ ( x ) , y ) ρf ( x , y ) ≤0 ≤ ` ( ρf ( x , y ) ) ≤M since L is positive , the cost will only occur when the margin function value is negative and the surrogate loss ` is larger than L. Additionally , we define the empirical risk of any scoring function f as R̂ ( ` ( ρf ) ) : = 1 n n∑ i=1 ` ρ ( xi , yi , f ) , and the expected risk as R ( ` ( ρf ) ) : = E ( x , y ) ∼P [ ` ρ ( x , y , f ) ] . Clearly , we have R ( f ) ≤ R ( ` ( ρf ) ) .

Summary: The authors consider the problem of bounding the error between the true risk and the empirical risk for structured prediction with a large number of possible labels. By adding fairly weak assumptions that are satisfied for common cases, they improve the dependence on the number of labels $d$ from $O(\sqrt{d})$ to $O(\log d)$ and/or the rate of convergence from $1/\sqrt{n}$ to $1/n$. They then analyze the generalization risk when there may be an infinite number of labels but the family of scoring functions has small complexity as defined by covering number.