Title: Structured Prediction with Stronger Consistency Guarantees
Venue: NeurIPS 2023
Accepted: yes
Keywords: structured prediction

consistency

learning theory

natural language processing
Abstract: We present an extensive study of surrogate losses for structured prediction supported by *$H$-consistency bounds*. These are recently introduced guarantees that are more relevant to learning than Bayes-consistency, since they are not asymptotic and since they take into account the hypothesis set $H$ used. We first show that no non-trivial $H$-consistency bound can be derived for widely used surrogate structured prediction losses. We then define several new families of surrogate losses, including *structured comp-sum losses* and *structured constrained losses*, for which we prove $H$-consistency bounds and thus Bayes-consistency. These loss functions readily lead to new structured prediction algorithms with stronger theoretical guarantees, based on their minimization. We describe efficient algorithms for minimizing several of these surrogate losses, including a new *structured logistic loss*.

Structured Prediction with Stronger Consistency Guarantees

Abstract

We present an extensive study of surrogate losses for structured prediction supported by H -consistency bounds . These are recently introduced guarantees that are more relevant to learning than Bayes-consistency, since they are not asymptotic and since they take into account the hypothesis set H used. We first show that no nontrivial H -consistency bound can be derived for widely used surrogate structured prediction losses. We then define several new families of surrogate losses, including structured comp-sum losses and structured constrained losses , for which we prove H -consistency bounds and thus Bayes-consistency. These loss functions readily lead to new structured prediction algorithms with stronger theoretical guarantees, based on their minimization. We describe efficient algorithms for minimizing several of these surrogate losses, including a new structured logistic loss .

1 Introduction

In most applications, the output labels of learning problems have some structure that is crucial to consider. This includes natural language processing applications, where the output may be a sentence, a sequence of parts-of-speech tags, a parse tree, or a dependency graph. It also includes image annotation, image segmentation, computer vision, video annotation, object recognition, motion estimation, computational photography, bioinformatics, and many other important applications.

Several algorithms have been designed in the past for structured prediction tasks, including Conditional Random Fields (CRFs) [Lafferty et al., 2001a, Gimpel and Smith, 2010], StructSVMs [Tsochantaridis et al., 2005a], Maximum-Margin Markov Networks (M3N) [Taskar et al., 2003a], kernel-regression-based algorithms [Cortes et al., 2005, 2007], Voted CRF and StructBoost [Cortes et al., 2016], search-based methods [Daumé III et al., 2009, Doppa et al., 2014, Lam et al., 2015, Chang et al., 2015, Ross et al., 2011] and a variety of deep learning techniques [Jurafsky and Martin, 2009, Vinyals et al., 2015a, Nadeau and Sekine, 2007, Zhang et al., 2008, Wu et al., 2016, Lucchi et al., 2013, Vinyals et al., 2015b], see Appendix A for a more comprehensive list of references and discussion.

Structured prediction tasks inherently involve a natural loss function based on substructures, which could be the Hamming loss, the n -gram loss, the edit-distance loss, or some other sequence similaritybased loss or task-specific structured loss. Many of the algorithms previously mentioned overlook this inherent structured loss by simply minimizing the cross-entropy loss. In contrast, the surrogate loss functions minimized by algorithms such as CRFs [Lafferty et al., 2001a, Gimpel and Smith, 2010], M3N [Taskar et al., 2003a], StructSVMs [Tsochantaridis et al., 2005a] or Voted CRF and StructBoost [Cortes et al., 2016] do take into account the natural structured loss of the task. But are these structured prediction loss functions consistent? What guarantees can we rely on when minimizing them over a restricted hypothesis set that does not include all measurable functions? Can we derive non-asymptotic guarantees?

This paper deals precisely with these theoretical problems in structured prediction.

Previous work. We include a detailed discussion on consistency in structured prediction in Appendix A. Here, we briefly discuss previous work by Osokin et al. [2017]. To our knowledge, this is one of the only studies proving Bayes-consistency for a family of loss functions in structured prediction (see also [Nowak et al., 2020] and other related references in Appendix A). The surrogate losses the authors proposed are the following quadratic losses (see also [Zhang, 2004]) defined for any function h mapping X × Y to R and any loss function glyph[lscript] between output labels by

However, the authors only consider the hypothesis set of linear scoring functions. Moreover, the feature vector in their setting only depends on the input x and ignores the label y . In many applications such as natural language prediction, however, it is critical to allow for features that depend both on the input sequence and the output sequence, parse tree, or dependency graph. Finally, in this formulation, the structured prediction problem is cast as a regression problem. Thus, as shown below, the loss function derived is non-standard, even in the binary classification case, where glyph[lscript] = glyph[lscript] 0 -1 is the zero-one loss and Y = { y 1 , y 2 } . In this simple case, L quad ( h, x, y 1 ) can be expressed as

This is not a typical formulation since it incorporates the magnitude of individual scores. In contrast, in standard binary classification scenario, only the difference between scores matters.

Structure of the paper. We present an extensive study of surrogate losses for structured prediction supported by H -consistency bounds . These are recently introduced guarantees that are more relevant to learning than Bayes-consistency, since they are not asymptotic and since they take into account the hypothesis set H used. We first show that no non-trivial H -consistency bound or even Bayesconsistency can be derived for widely used surrogate structured prediction losses (Section 3). We then define several new families of surrogate losses, including structured comp-sum losses (Section 4) and structured constrained losses (Section 5), for which we prove H -consistency bounds and thus Bayes-consistency. These loss functions readily lead to new structured prediction algorithms with stronger theoretical guarantees, based on their minimization. We also describe efficient gradient computation algorithms for several of these surrogate losses, including a new structured logistic loss (Section 6).

2 Preliminaries

Learning scenario. We consider the standard structured prediction scenario with the input space X and output space Y = { 1 , . . . , n } . The output space may be discrete objects with overlapping structures, such as sequences, images, graphs, parse trees, lists, or others. We assume that the output can be decomposed into l substructures. The substructures could represent words or tokens for example, or other subsequences along a sequence, resulting in the decomposition of the output space Y as Y = Y 1 ×/uni22EF × Y l . Here, each Y j represents the set of possible labels or classes that can be assigned to the j -th substructure.

Scoring function. Structured prediction is typically formulated via scoring functions that map X × Y to R , which assign a score to each possible class y ∈ Y . Let H be a family of such scoring functions. For any h ∈ H , we denote by h ( x ) its prediction for the input x ∈ X , which is the output y ∈ Y that maximizes the score h ( x, y ) , that is, h ( x ) = argmax y ∈ Y h ( x, y ) , with a fixed deterministic strategy to break ties in selecting the label with the highest score. For simplicity, we choose the label with the highest index under the natural ordering of labels as the tie-breaking strategy. We denote by H all the family of all measurable scoring functions.

Generalization error and target loss. Given a distribution D over X × Y and a loss function L ∶ H × X × Y → R , the generalization error of a hypothesis h ∈ H and the best-in-class generalization error are defined as follows:

In structured prediction, the goal is to select a hypothesis h ∈ H with small generalization error with respect to a target loss function L ( h, x, y ) , which can be written as L ( h, x, y ) = glyph[lscript] ( h ( x ) , y ) for some non-negative auxiliary function glyph[lscript] ( y ′ , y ) with any y ′ , y ∈ Y . glyph[lscript] is assumed to be symmetric, that is, glyph[lscript] ( y ′ , y ) = glyph[lscript] ( y, y ′ ) . This is a natural assumption since all instances of glyph[lscript] that we are familiar with in structured prediction admit this property. A significant characteristic of structured prediction is that the target loss function can be decomposed along the substructures Y k . As an example, we may use the Hamming loss as the target loss function, which is defined as L ham ( h, x, y ) = glyph[lscript] ham ( h ( x ) , y ) , where glyph[lscript] ham ( y ′ , y ) = 1 l ∑ l j = 1 1 y ′ j ≠ y j with any y ′ j , y j ∈ Y j ; we can also use the zero-one loss as the target loss function, which is defined as L 0 -1 ( h, x, y ) = glyph[lscript] 0 -1 ( h ( x ) , y ) , where glyph[lscript] 0 -1 ( y ′ , y ) = 1 y ′ ≠ y . Note that L 0 -1 can be viewed as a special case of L ham when l = 1 . We denote by glyph[lscript] max = max y ′ ,y ∈ Y glyph[lscript] ( y ′ , y ) the maximal value of a target loss function. Without loss of generality, we assume that glyph[lscript] max ≤ 1 , which can be achieved by normalizing the function glyph[lscript] .

Consistency guarantees and surrogate loss. Optimizing the target loss functions in structured prediction for many choices of the hypothesis sets is NP-hard because they are not convex. One common method to address this issue is to resort to surrogate loss functions. Different surrogate loss functions readily lead to different structured prediction algorithms. A natural learning guarantee for such surrogate losses is Bayes-consistency , which guarantees that minimizing the generalization error for a surrogate loss L sur over H all also leads to the minimization of generalization error for the target loss L .

Definition 1 (Bayes-consistency) . A surrogate loss L sur is Bayes-consistent in structured prediction, if for any target loss glyph[lscript] , hypothesis h n ∈ H all and any distribution,

Bayes-consistency is an asymptotic guarantee and does not take into account typical hypothesis sets used in structured prediction algorithms, such as linear models or neural networks. To tackle these issues, recent work by Awasthi, Mao, Mohri, and Zhong [2022a,b] propose a stronger consistency guarantee, referred to as H -consistency bounds , which are bounds relating the estimation error of the target loss to the estimation error of a surrogate loss (see also [Awasthi et al., 2021a,b, Mao et al., 2023h,e,f, Zheng et al., 2023, Mao et al., 2023b,g,d, Mohri et al., 2023, Mao et al., 2023c,a, Awasthi et al., 2023a,b]):

Definition 2 ( H -consistency bounds) . Given a subset of the hypothesis class H ⊆ H all , a surrogate loss L sur admits a H -consistency bound in structured prediction, if for some non-decreasing function f ∶ R + → R + , a bound of the following form holds for any target loss glyph[lscript] , hypothesis h ∈ H and any distribution:

As pointed out by Awasthi et al. [2022a,b], H -consistency bounds are the state-of-the-art consistency guarantees for surrogate losses. They are much stronger and more informative than Bayes-consistency, since they account for hypothesis sets H adopted and provide a quantitative, non-asymptotic relation between surrogate losses and target losses. H -consistency bounds can imply Bayes-consistency when taking H to be H all . In the next sections, we will present an extensive study of surrogate losses for structured prediction supported by H -consistency bounds .

Conditional regret and minimizability gap. We denote by p ( x ) = ( p ( x, 1 ) , . . . , p ( x, n )) the conditional distribution of Y given X = x . Then, the conditional error of a hypothesis h for a loss function L , denoted by C L ( h, x ) , can be expressed as

We further define the best-in-class conditional error and the conditional regret as C ∗ L ( H , x ) = inf h ∈ H C L ( h, x ) and ∆ C L , H ( h, x ) = C L ( h, x ) -C ∗ L ( H , x ) respectively. The generalization error can then be rewritten as R L ( h ) = E x [ C L ( h, x )] .

Akey quantity appearing in H -consistency bounds is the minimizability gap M L ( H ) , which measures the difference between the best-in-class generalization error and the expected best-in-class conditional error for a loss function L and a hypothesis set H : M L ( H ) = R ∗ L ( H ) -E x [ C ∗ L ( H , x )] . This is an inherent quantity that we cannot hope to minimize or estimate. As shown by Steinwart [2007,

Theorem 3.2], the minimizability gaps vanish M L ( H all ) = 0 for the family of all measurable functions. More generally, the minimizability gaps vanish when the best-in-class error coincides with the Bayes-error, that is, R ∗ glyph[lscript] ( H ) = R ∗ glyph[lscript] ( H all ) [Awasthi et al., 2022b, Mao et al., 2023h].

The following result characterizes the best-in-class conditional error and the conditional regret for a target loss L , which will be helpful for proving H -consistency bounds in structured prediction. We denote by H ( x ) the set of all possible predictions on a input x generated by hypotheses in H : H ( x ) = { h ( x )∶ h ∈ H } . The proof is given in Appendix B.

Lemma 3. The best-in-class conditional error and the conditional regret for a target loss L in structured prediction can be expressed as follows:

3 Structured max losses

In this section, we examine the loss functions associated to several prominent structured prediction algorithms. We show that, while they are natural, none of them is Bayes-consistent, which implies that they cannot be supported by H -consistency bounds either. More generally, we consider the following family of surrogate loss functions proposed in [Cortes, Kuznetsov, Mohri, and Yang, 2016], which we refer to as structured max losses :

where Φ u ∶ R → R + is an upper bound on v ↦ u 1 v ≤ 0 for any u ∈ R + . In this formulation, different choices of Φ u can lead to different structured prediction algorithms. Specifically, as shown by Cortes et al. [2016], the following choices of Φ u ( v ) recover many well-known algorithms:

Φ u ( v ) = max ( 0 , u ( 1 -v )) : StructSVM [Tsochantaridis et al., 2005b].

Φ u ( v ) = max ( 0 , u -v ) : Max-Margin Markov Networks (M3N) [Taskar et al., 2003b].

Φ u ( v ) = log ( 1 + e u -v ) : Conditional Random Field (CRF) [Lafferty et al., 2001b].

Φ u ( v ) = ue -v : StructBoost [Cortes et al., 2016].

The following gives a general negative result for L max that holds under broad assumptions.

Theorem 4 ( Negative results of L max ) . Assume that n > 2 and that Φ u ( v ) is convex and nonincreasing for u = 1 . Then, the max structured loss L max is not Bayes-consistent.

The proof is included in Appendix C. It is straightforward to see that the assumption of Theorem 4 holds for all the choices of Φ u listed above. Thus, the theorem rules out consistency guarantees for any of the loss functions associated to the structured prediction algorithms mentioned above: StructSVM, M3N, CRF, Structboost. Furthermore, Theorem 4 provides negative results for a broad and generalized family of loss functions, collectively referred to as structured max loss. This extends the scope of existing research, as previous works had only addressed the inconsistency of specific instances within the structured max loss category, such as that of M3N [Osokin et al., 2017, Ciliberto et al., 2016, Nowak et al., 2020].

4 Structured comp-sum losses

In this section, we first analyze the V oted CRF loss function, which incorporates the auxiliary loss function glyph[lscript] in the CRF loss and which has been used in several previous studies. Next, we introduce a new family of loss functions for structured predictions that we prove to admit strong consistency guarantees.

4.1 Voted Conditional Random Field (VCRF)

We first study a family of surrogate losses called Voted Conditional Random Field (VCRF) , which corresponds to the structured prediction algorithm defined in [Cortes et al., 2016]:

This loss function has also been presented as the softmax margin [Gimpel and Smith, 2010] or the reward-augmented maximum likelihood [Norouzi et al., 2016]. It can be viewed as the softmax variant of the M3N loss. Indeed, the loss function for M3N can be written as follows:

If we replace the maximum function with the softmax, we obtain

Next, we show that, as with the loss function for M3N, the VCRF loss function L VCRF is inconsistent. Theorem 5 ( Negative result of L VCRF ) . The Voted Conditional Random Field L VCRF is not Bayesconsistent.

The proof is included in Appendix D. The key observation in the proof is that the conditional error of VCRF loss function can be reduced to a specific form when the target loss function L decouples, which can lead to a different Bayes classifier from that of the target loss function.

To the best of our knowledge, no prior studies in the literature have explored the consistency of the VCRF loss formulation. The most closely related discussions center around a specialized instance of the multi-class logistic loss (also referred to as Conditional Random Field in that context), in which glyph[lscript] ( y ′ , y ) disappears within the framework of the Voted Conditional Random Field. The previous works by Osokin et al. [2017], Ciliberto et al. [2016], Nowak et al. [2020] point out that the multi-class logistic loss cannot be consistent in structured prediction due to the absence of the target loss function within its formulation. Instead, our result shows that, even when integrating the target loss glyph[lscript] ( y ′ , y ) within its formulation, the Voted Conditional Random Field cannot be consistent.

Along with Theorem 4, these results rule out consistency guarantees for commonly used surrogate loss functions in structured prediction.

4.2 Structured comp-sum loss functions

In this section, we define a family of new loss functions for structured prediction that are not only Bayes-consistent but also supported by H -consistency bounds. These are loss functions that can be viewed as the generalization to structured prediction of loss functions defined via a composition and a sum, and that have been referred to as comp-sum losses in [Mao et al., 2023h]. Thus, we will refer to them as structured comp-sum losses . They are defined as follows:

where glyph[lscript] ( y ′ , y ) = 1 -glyph[lscript] ( y ′ , y ) , Φ 1 ∶ R + → R + is a non-decreasing auxiliary function and Φ 2 ∶ R → R + a non-decreasing auxiliary function. This formulation (8) can also be viewed as a weighted comp-sum loss, if we interpret glyph[lscript] (⋅ , y ) as a weight vector.

Specifically, we can choose Φ 2 ( v ) = e v and Φ 1 ( v ) = log ( v ) , Φ 1 ( v ) = v -1 , Φ 1 ( v ) = 1 α ( 1 -1 v α ) , α ∈ ( 0 , 1 ) and Φ 1 ( v ) = 1 -1 v , which leads to new surrogate losses for structured prediction defined in Table 1. These surrogate losses are novel strict generalization of their counterparts in the standard multi-class classification case where glyph[lscript] = glyph[lscript] 0 -1 . More precisely, when glyph[lscript] = glyph[lscript] 0 -1 , L comp log coincides with the logistic loss [Verhulst, 1838, 1845, Berkson, 1944, 1951]; L comp exp coincides with the sumexponential loss [Weston and Watkins, 1998, Awasthi et al., 2022b]; L comp gce coincides with the

generalized cross-entropy loss [Zhang and Sabuncu, 2018]; and L comp mae coincides with the mean absolute error loss [Ghosh et al., 2017].

We will show that these structured comp-sum losses benefit from H -consistency bounds in structured prediction, when H is a symmetric and complete hypothesis set. A hypothesis set H is symmetric if there exists a family F of real-valued functions such that {[ h ( x, 1 ) , . . . , h ( x, n )]∶ h ∈ H } = {[ f 1 ( x ) , . . . , f n ( x )]∶ f 1 , . . . , f n ∈ F } for any x ∈ X . Thus, the choice of the scoring functions does not depend on the order of the categories in Y . A hypothesis set H is complete if it can generate scores that span R , that is, { h ( x, y )∶ h ∈ H } = R for any ( x, y ) ∈ X × Y . As shown by Awasthi et al. [2022b] and Mao et al. [2023h], these assumptions are general and hold for common hypothesis sets used in practice, such as the family of linear hypotheses and that of multi-layer feed-forward neural networks, and of course that of all measurable functions.

Theorem 6 ( H -consistency bound of L comp ) . Assume that H is symmetric and complete. Then, for any target loss glyph[lscript] , any hypothesis h ∈ H and any distribution, we have

where Γ ( t ) = 2 √ t when L comp = L comp log or L comp exp ; Γ ( t ) = 2 √ n α t when L comp = L comp gce ; and Γ ( t ) = nt when L comp = L comp mae .

Theorem 6 represents a consolidated result for the four structured comp-sum losses, with the proofs for each being presented separately in Appendix E. The key step of the proof is to upper bound the conditional regret of the target loss (Lemma 3) by that of a surrogate loss. To achieve this, we upper bound the best-in-class conditional error by the conditional error of a carefully selected hypothesis h µ ∈ H . The resulting softmax S µ of this hypothesis only differs from the original softmax S corresponding to h on two labels. Theorem 6 admits as special cases the H -consistency bounds of Mao et al. [2023h] given for standard multi-class classification ( glyph[lscript] = glyph[lscript] 0 -1 ) and significantly extends them to the general structured prediction scenario.

Let us emphasize that our proof technique is novel and distinct from the approach used in [Mao et al., 2023h], which only applies to the special case where glyph[lscript] is the zero-one loss and cannot be generalized to any target loss glyph[lscript] . In their proof, the authors choose h µ based on individual scores h ( x, y ) , rather than the softmax. Consequently, when glyph[lscript] ≠ glyph[lscript] 0 -1 , as is common in structured prediction, the resulting optimization problem of µ can be very intricate and a closed-form expression of the optimization solution cannot be derived. However, our new proof method overcomes this limitation. By viewing the softmax of hypothesis as a unit and introducing a pseudo-conditional distribution q , we are able to solve a simple constrained optimization problem on µ within structured prediction scenario.

By Steinwart [2007, Theorem 3.2], the minimizability gaps M L comp , H and M L , H vanish for the family of all measurable functions. Therefore, when H = H all , the H -consistency bounds provided in Theorem 6 imply the Bayes-consistency of these structured comp-sum losses.

Corollary 7. The structured comp-sum loss L comp is Bayes-consistent for L comp = L comp log , L comp exp , L comp gce , and L comp mae .

In fact, Theorem 6 provides stronger quantitative bounds than Bayes-consistency when the minimizability gaps vanish, which suggests that if the estimation error of the structured comp-sum loss R L comp ( h ) -R ∗ L comp , H is reduced to glyph[epsilon1] , the estimation error of the target loss R L ( h ) -R ∗ L , H is upper bounded by 2 √ glyph[epsilon1] for structured logistic loss and structured sum-exponential loss, 2 √ n α glyph[epsilon1] for structured generalized cross-entropy loss, and nglyph[epsilon1] for structured mean absolute error loss.

5 Structured constrained loss functions

In this section, we introduce another new family of surrogate losses for structured prediction that we prove to admit H -consistency bounds. We will present a novel generalization of the constrained losses [Lee et al., 2004, Awasthi et al., 2022b] to structured prediction. Thus, we refer to them as structured constrained losses and define them as follows:

with the constraint that ∑ y ∈ Y h ( x, y ) = 0 and Φ u ∶ R → R + is an upper bound on v ↦ u 1 v ≤ 0 for any u ∈ R + . In standard constrained loss formulation, a single-variable function Φ ( v ) that defines a margin-based loss is used. In (10), the single-variable function Φ ( v ) is generalized to being a function of two variables Φ u ( v ) , which depends on both the target loss and the scores, to accommodate the structured prediction scenario. Specifically, we can choose Φ u ( v ) = ue -v , Φ u ( v ) = u max { 0 , 1 -v } 2 , Φ u ( v ) = u max { 0 , 1 -v } , Φ u ( v ) = u min { max { 0 , 1 -v / ρ } , 1 } , which lead to new surrogate losses for structured prediction defined in Table 2. These surrogate losses are novel generalization of their corresponding counterparts [Lee et al., 2004, Awasthi et al., 2022b] in standard multi-class classification, where glyph[lscript] = glyph[lscript] 0 -1 . As with structured comp-sum losses, we will show that these structured constrained losses benefit from H -consistency bounds in structured prediction as well, for any symmetric and complete hypothesis set H .

Theorem 8 ( H -consistency bound of L cstnd ) . Assume that H is symmetric and complete. Then, for any target loss glyph[lscript] , hypothesis h ∈ H and any distribution, we have

The proof is included in Appendix F. As for Theorem 8, the key part of the proof is to upper bound the conditional regret of the target loss (Lemma 3) by that of a surrogate loss. Here too, we introduce a pseudo-conditional distribution q , which can be viewed as a weighted distribution of the original one, p ( x ) , with weights given by the target loss function. Then, we upper bound the best-in-class conditional error by the conditional error of a carefully selected hypothesis h µ ∈ H .

As shown by Steinwart [2007, Theorem 3.2], for the family of all measurable functions, the minimizability gaps vanish: M L cstnd , H = 0 and M L , H = 0 . Therefore, when H = H all , the H -consistency bounds provided in Theorem 6 imply the Bayes-consistency of these structured constrained losses.

Corollary 9. The structured constrained loss L cstnd is Bayes-consistent for L cstnd = L cstnd exp , L cstnd sq -hinge , L cstnd hinge , and L cstnd ρ .

As with the cases of structured comp-sum losses, Theorem 8 provides in fact stronger quantitative bounds than Bayes-consistency. They show that that if the estimation error of the structured constrained loss R L comp ( h )-R ∗ L comp , H is reduced to glyph[epsilon1] , the estimation error of the target loss R L ( h )-R ∗ L , H is upper bounded by 2 √ glyph[lscript] max glyph[epsilon1] for L cstnd exp , 2 √ glyph[epsilon1] for L cstnd sq -hinge and glyph[epsilon1] for L cstnd hinge and L cstnd ρ .

It is important to note that we can upper bound the minimizability gap by the approximation error, or finer terms depending on the magnitude of the parameter space as in [Mao et al., 2023h]. Furthermore, our H -consistency bounds (Theorems 6 and 8) can be used to derive finite sample learning bounds for a hypothesis set H . These bounds depend on the Rademacher complexity of the hypothesis set and the loss function, as well as an upper bound on the minimizability gap for the surrogate loss.

6 Optimization of L comp log and L comp exp

In this section, we show that the gradient of the structured logistic loss L comp log can be computed efficiently at any point ( x i , y i ) and therefore that this loss function is both supported by H -consistency bounds and is of practical use. We similarly show that for L comp exp in Appendix G.2.

Fix the labeled pair ( x i , y i ) and h ∈ H . Observe that L comp log ( h, x i , y i ) can be equivalently rewritten as follows:

where glyph[lscript] i = ∑ y ′ ∈ Y glyph[lscript] ( y ′ , y i ) , and Z h,i = ∑ y ∈ Y e h ( x i ,y ) . Note that glyph[lscript] i does not depend on h and can be pre-computed. Modulo normalization, this quantity is the average similarity of y i to Y , if we interpret glyph[lscript] = 1 -glyph[lscript] as a similarity. While Y may be very large, this can be often computed straightforwardly for most loss functions glyph[lscript] . For example, for the Hamming loss, for sequences of length l , we have

Thus, in this case, glyph[lscript] i does not depend on i and is a universal constant. Similarly, glyph[lscript] i can be shown to be a constant for many other losses.

Hypothesis set. For the remaining of this section, to simplify the presentation, we will consider the hypothesis set of linear functions H = { x ↦ w ⋅ Ψ ( x, y )∶ w ∈ R d } , where Ψ is a feature mapping from X × Y to R d . Note that a number of classical structured prediction algorithms adopt the same linear hypothesis set: StructSVM [Tsochantaridis et al., 2005b], Max-Margin Markov Networks (M3N) [Taskar et al., 2003b], Conditional Random Field (CRF) [Lafferty et al., 2001b], Voted Conditional Random Field (VCRF) [Cortes et al., 2016]. Our algorithms can also be incorporated into standard procedures for training neural network architectures (see [Cortes et al., 2018], Appendix B).

Markovian features. We will also assume Markovian features, as is common in structured prediction. Features used in practice often admit this property. Furthermore, in the absence of any such assumption, it is known that learning and inference in general are intractable. We will largely adopt here the definitions and notation from [Cortes et al., 2016] and will consider the common case where Y is a set of sequences of length l over a finite alphabet ∆ of size r . Other structured problems can be treated in similar ways. We will denote by ε the empty string and for any sequence y = ( y 1 , . . . , y l ) ∈ Y , we will denote by y s ′ s = ( y s , . . . , y s ′ ) the substring of y starting at index s and ending at s ′ . For convenience, for s ≤ 0 , we define y s by y s = ε .

We will assume that the feature vector Ψ admits a Markovian property of order q , that is it can be decomposed as follows for any ( x, y ) ∈ X × Y :

for some position-dependent feature vector function ψ defined over X × ∆ q ×[ l ] . We note that we can write Ψ = ∑ p k = 1 ˜ Ψ k with ˜ Ψ k = ( 0 , . . . , Ψ k , . . . , 0 ) . In the following, abusing the notation, we will simply write Ψ k instead of ˜ Ψ k . Each Ψ k corresponds to a Markovian feature vector based only on k -grams, p is the largest k . Thus, for any x ∈ X and y ∈ Y , we have

For any k ∈ [ 1 , p ] , let ψ k denote the position-dependent feature vector function corresponding to Ψ k . Also, for any x ∈ X and y ∈ ∆ l , define ̃ ψ by ̃ ψ ( x, y s s -p + 1 , s ) = ∑ p k = 1 ψ k ( x, y s s -k + 1 , s ) . Observe then that we can write

Gradient computation. Adopting the shorthand w for h , we can rewrite the loss at ( x i , y i ) as:

Thus, the gradient of L comp log at any w ∈ R d is given by

where q w is defined for all y ∈ Y by q w ( y ) = e w ⋅ Ψ ( x i ,y ) Z w with Z w = ∑ y ∈ Y e w ⋅ Ψ ( x i ,y ) . Note that the sum defining these terms is over a number of sequences y that is exponential in r and that the computation appears to be therefore challenging. The following lemma gives the expression of the gradient of L comp log and helps identify the most computationally challenging terms.

Lemma 10. For any w ∈ R d , the gradient of L comp log can be expressed as follows:

Proof. Using the decomposition of the feature vector, we can write:

This completes the proof.

In light of this result, the bottleneck in the gradient computation is the evaluation of Q w ( z , s ) and L ( z , s ) for all s ∈ [ l ] and z ∈ ∆ p . In previous work [Cortes, Kuznetsov, Mohri, and Yang, 2016, Cortes, Kuznetsov, Mohri, Storcheus, and Yang, 2018], it was shown that the quantities Q w ( z , s ) can be determined efficiently, all together, by running two single-source shortest-distance algorithms over the (+ , ×) semiring on an appropriate weighted finite automaton (WFA). The overall time complexity of the computation of all quantities Q w ( z , s ) , z ∈ ∆ p and s ∈ [ l ] , is then in O ( lr p ) , where r = ∣ ∆ ∣ .

We now analyze the computation of L ( z , s ) for a fixed z ∈ ∆ p and s ∈ [ l ] . Note that, unlike Q w ( z , s ) , this term does not depend on w and can therefore be computed once and for all, before any gradient computation. The sum defining L ( z , s ) is over all sequences y that admit the substring z at position s .

Rational losses. In Appendix G.1, we also give an efficient algorithm for the computation of the quantities L ( z , s ) in the case of Markovian losses. Here, we present an efficient algorithm for their computation in the important case of rational losses . This is a general family of loss functions based on rational kernels [Cortes, Haffner, and Mohri, 2004] that includes, in particular, n -gram losses, which can be defined for a pair of sequences ( y, y ′ ) as the negative inner product of the vectors of n -gram counts of y and y ′ .

Our algorithm bears some similarity to that of Cortes et al. [2018] for the computation of the gradient of the VCRF loss function. It is however distinct because the structured prediction loss function we are considering and our definition of rational loss are both different. We will adopt a similar notation and terminology. Recall that for any sequence y , we denote by y i the symbol in its i th position and by y j i = y i y i + 1 /uni22EF y j the substring of y starting at position i and ending at j . We denote by E A the set of transitions of a WFA A . Let U be a weighted finite-state transducer (WFST) over the (+ , ×) semiring over the reals, with ∆ as both the input and output alphabet. Then, we define the rational loss associated to U for all y, y ′ ∈ ∆ ∗ by glyph[lscript] ( y, y ′ ) = U ( y, y ′ ) .

Let Y denote a WFA over the (+ , ×) semiring accepting the set of all sequences of length l with weight one and let Y i denote the WFA accepting only y i with weight one. Then, by definition, the weighted transducer Y ○ U ○ Y i obtained by composition maps each sequence y in ∆ l to y i with weight U ( y, y i ) . The WFA Π 1 ( Y ○ U ○ Y i ) derived from that transducer by projection on the input (that is by removing output labels) is associating to each sequence y weight U ( y, y i ) . We use weighted determinization [Mohri, 1997] to compute an equivalent deterministic WFA denote M . As shown by Cortes et al. [2015][Theorem 3], M can be computed in polynomial time. M

0

1

a:a/1

b:b/1

2

a:a/1

b:b/1

3/1

a:a/1

b:b/1

0

1

a:a/1

2

b:b/1

3/1

a:a/1

admits a unique path labeled with any sequence y ∈ ∆ l and the weight of that path is U ( y, y i ) . The weight of that accepting path is obtained by multiplying the weights of its transitions and that of the final state.

We now define a deterministic p -gram WFA N that accepts all sequences y ∈ ∆ l with each of its states ( z ′ , s ) encoding a ( p -1 ) -gram z ′ read to reach it and the position s in the sequence y at which it is reached. The transitions of N are therefore defined as follows with weight one:

The initial state is ( glyph[epsilon1], 0 ) and the final states are those with the second element of the pair (the position) being l . Note that, by construction, N is deterministic. Then, the composition (or intersection) WFA N ○ M still associates the same weight as M to each input string

(

ε

, 0)

(a, 1)

a:a/1

(b, 1)

b:b/1

(a, 2)

a:a/1

(b, 2)

b:b/1

a:a/1

b:b/1

Figure 2: Illustration of the WFA N for ∆ = { a, b } , p = 2 and l = 2 . y ∈ ∆ l . However, the states in that composition help us compute L ( z , s ) . In particular, for any z ∈ ∆ p and s ∈ [ l ] , let E ( z , s ) be the set of transitions of N ○ M constructed by pairing the transition

(( z p -1 1 , s -1 ) , z p , ω ( z , s ) , ( z p 2 , s )) in N with a transition ( q M , z p , ω, q ′ M ) in M . They admit the following form:

The WFA N ○ M is deterministic as a composition of two deterministic WFAs. Thus, there is a unique path labeled with a sequence y ∈ ∆ l in N ○ M and y admits the substring z ending at position s iff that path goes through a transition in E ( z , s ) when reaching position s . Therefore, to compute L ( z , s ) , it suffices for us to compute the sum of the weights of all paths in N ○ M going through a transition in E ( z , s ) . This can be done straightforwardly using the forward-backward algorithm or two single-source shortest-distance algorithm over the (+ , ×) semiring [Mohri, 2002a], one from the initial state, the other one from the final states. Since N ○ M is acyclic and admits O ( l ∣ ∆ ∣ p ) transitions, we can compute all the quantities L ( z , s ) , s ∈ [ l ] and z ∈ ∆ p , in time O ( l ∣ ∆ ∣ p ) .

7 Conclusion

Our detailed study revealed shortcomings in commonly used surrogate loss functions and algorithms for structured prediction, prompting the introduction of new, strongly consistent alternatives. These findings not only enhance the theoretical and algorithmic foundations of structured prediction but also pave the way for the development of practical and effective solutions. In upcoming work, we will report an extensive empirical analysis of our algorithms. Our work provides tools and insights for future algorithm design in this domain, promising advancements in both theory and application.

References

E. Ahmad, L. Brogat-Motte, P. Laforgue, and F. d'Alché Buc. Sketch in, sketch out: Accelerating both learning and inference for structured prediction with kernels. arXiv preprint arXiv:2302.10128 , 2023.

Allauzen and M. Mohri. An efficient pre-determinization algorithm. In O. H. Ibarra and Z. Dang, editors, Implementation and Application of Automata, 8th International Conference, CIAA 2003, Santa Barbara, California, USA, July 16-18, 2003, Proceedings , volume 2759 of Lecture Notes in Computer Science , pages 83-95. Springer, 2003.

Allauzen and M. Mohri. An optimal pre-determinization algorithm for weighted transducers. Theor. Comput. Sci. , 328(1-2):3-18, 2004.

Awasthi, N. Frank, A. Mao, M. Mohri, and Y. Zhong. Calibration and consistency of adversarial surrogate losses. In Advances in Neural Information Processing Systems , 2021a.

Awasthi, A. Mao, M. Mohri, and Y. Zhong. A finer calibration analysis for adversarial robustness. arXiv preprint arXiv:2105.01550 , 2021b.

Awasthi, A. Mao, M. Mohri, and Y. Zhong. H-consistency bounds for surrogate loss minimizers. In International Conference on Machine Learning , 2022a.

Awasthi, A. Mao, M. Mohri, and Y. Zhong. Multi-class H-consistency bounds. In Advances in neural information processing systems , 2022b.

Awasthi, A. Mao, M. Mohri, and Y. Zhong. Theoretically grounded loss functions and algorithms for adversarial robustness. In International Conference on Artificial Intelligence and Statistics , pages 10077-10094, 2023a.

Awasthi, A. Mao, M. Mohri, and Y. Zhong. DC-programming for neural network optimizations. Journal of Global Optimization , 2023b.

Belanger and A. McCallum. Structured prediction energy networks. In International Conference on Machine Learning , pages 983-992, 2016.

Belanger, B. Yang, and A. McCallum. End-to-end learning for structured prediction energy networks. In International Conference on Machine Learning , pages 429-439, 2017.

Belharbi, R. Hérault, C. Chatelain, and S. Adam. Deep neural networks regularization for structured output prediction. Neurocomputing , 281:169-177, 2018.

Berkson. Application of the logistic function to bio-assay. Journal of the American Statistical Association , 39:357--365, 1944.

Berkson. Why I prefer logits to probits. Biometrics , 7(4):327--339, 1951.

Blondel. Structured prediction with projection oracles. In Advances in neural information processing systems , 2019.

Brogat-Motte, A. Rudi, C. Brouard, J. Rousu, and F. d'Alché Buc. Learning output embeddings in structured prediction. arXiv preprint arXiv:2007.14703 , 2020.

A. Cabannes, F. Bach, and A. Rudi. Fast rates for structured prediction. In Conference on Learning Theory , pages 823-865, 2021.

Cabannnes, A. Rudi, and F. Bach. Structured prediction with partial labelling through the infimum loss. In International Conference on Machine Learning , pages 1230-1239, 2020.

Chang, A. Krishnamurthy, A. Agarwal, H. Daumé III, and J. Langford. Learning to search better than your teacher. In Proceedings of ICML , 2015.

L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille. Semantic image segmentation with deep convolutional nets and fully connected crfs. arXiv preprint arXiv:1412.7062 , 2014.

L.-C. Chen, A. Schwing, A. Yuille, and R. Urtasun. Learning deep structured models. In International Conference on Machine Learning , pages 1785-1794, 2015.

Choi, O. Meshi, and N. Srebro. Fast and scalable structural svm with slack rescaling. In Artificial Intelligence and Statistics , pages 667-675, 2016.

Ciliberto, L. Rosasco, and A. Rudi. A consistent regularization approach for structured prediction. In Advances in neural information processing systems , 2016.

Ciliberto, F. Bach, and A. Rudi. Localized structured prediction. In Advances in Neural Information Processing Systems , 2019.

Ciliberto, L. Rosasco, and A. Rudi. A general framework for consistent structured prediction with implicit loss embeddings. The Journal of Machine Learning Research , 21(1):3852-3918, 2020.

Collobert, J. Weston, L. Bottou, M. Karlen, K. Kavukcuoglu, and P. Kuksa. Natural language processing (almost) from scratch. Journal of machine learning research , 12(ARTICLE):2493-2537, 2011.

Corro. On the inconsistency of separable losses for structured prediction. arXiv preprint arXiv:2301.10810 , 2023.

Cortes, P. Haffner, and M. Mohri. Rational kernels: Theory and algorithms. J. Mach. Learn. Res. , 5:1035-1062, 2004.

Cortes, M. Mohri, and J. Weston. A general regression technique for learning transductions. In L. D. Raedt and S. Wrobel, editors, Machine Learning, Proceedings of the Twenty-Second International Conference (ICML 2005), Bonn, Germany, August 7-11, 2005 , volume 119 of ACM International Conference Proceeding Series , pages 153-160. ACM, 2005.

Cortes, M. Mohri, and J. Weston. A General Regression Framework for Learning String-to-String Mappings. In Predicting Structured Data . MIT Press, 2007.

Cortes, V. Kuznetsov, and M. Mohri. Ensemble methods for structured prediction. In Proceedings of the 31th International Conference on Machine Learning, ICML 2014, Beijing, China, 21-26 June 2014 , volume 32 of JMLR Workshop and Conference Proceedings , pages 1134-1142. JMLR.org, 2014a.

Cortes, V. Kuznetsov, and M. Mohri. Learning ensembles of structured prediction rules. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, ACL 2014, June 22-27, 2014, Baltimore, MD, USA, Volume 1: Long Papers , pages 1-12. The Association for Computer Linguistics, 2014b.

Cortes, V. Kuznetsov, M. Mohri, and M. K. Warmuth. On-line learning algorithms for path experts with non-additive losses. In Proceedings of COLT , 2015.

Cortes, V. Kuznetsov, M. Mohri, and S. Yang. Structured prediction theory based on factor graph complexity. In Advances in Neural Information Processing Systems , 2016.

Cortes, V. Kuznetsov, M. Mohri, D. Storcheus, and S. Yang. Efficient gradient computation for structured output learning with rational and tropical losses. In Advances in Neural Information Processing Systems , 2018.

Daumé III, J. Langford, and D. Marcu. Search-based structured prediction. Machine Learning , 75 (3):297-325, 2009.

Domke. Learning graphical model parameters with approximate marginal inference. IEEE transactions on pattern analysis and machine intelligence , 35(10):2454-2467, 2013.

R. Doppa, A. Fern, and P. Tadepalli. Structured prediction via output space search. JMLR , 15(1): 1317-1350, 2014.

Dragone, S. Teso, and A. Passerini. Neuro-symbolic constraint programming for structured prediction. arXiv preprint arXiv:2103.17232 , 2021.

Edunov, M. Ott, M. Auli, D. Grangier, and M. Ranzato. Classical structured prediction losses for sequence to sequence learning. arXiv preprint arXiv:1711.04956 , 2017.

Finocchiaro, R. Frongillo, and B. Waggoner. An embedding framework for consistent polyhedral surrogates. In Advances in neural information processing systems , 2019.

Ghosh, H. Kumar, and P. S. Sastry. Robust loss functions under label noise for deep neural networks. In Proceedings of the AAAI conference on artificial intelligence , 2017.

Gimpel and N. A. Smith. Softmax-margin crfs: Training log-linear models with cost functions. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics , pages 733-736, 2010.

Graber and A. Schwing. Graph structured prediction energy networks. In Advances in Neural Information Processing Systems , 2019.

Graber, O. Meshi, and A. Schwing. Deep structured prediction with nonlinear output transformations. In Advances in Neural Information Processing Systems , 2018.

Gygli, M. Norouzi, and A. Angelova. Deep value networks learn to evaluate and iteratively refine structured outputs. In International Conference on Machine Learning , pages 1341-1351, 2017.

R. Hershey, J. L. Roux, and F. Weninger. Deep unfolding: Model-based inspiration of novel deep architectures. arXiv preprint arXiv:1409.2574 , 2014.

Huang, W. Xu, and K. Yu. Bidirectional lstm-crf models for sequence tagging. arXiv preprint arXiv:1508.01991 , 2015.

Jaderberg, K. Simonyan, A. Vedaldi, and A. Zisserman. Deep structured output learning for unconstrained text recognition. arXiv preprint arXiv:1412.5903 , 2014.

Jang, S. Mo, and S. Ahn. Diffusion probabilistic models for graph-structured prediction. arXiv preprint arXiv:2302.10506 , 2023.

Jiang, M. Zhang, W.-J. van Hoeve, and Y. Xue. Constraint reasoning embedded structured prediction. Journal of Machine Learning Research , 23(345):1-40, 2022.

Jurafsky and J. H. Martin. Speech and Language Processing (2nd Edition) . Prentice-Hall, Inc., 2009.

Kim, C. Denton, L. Hoang, and A. M. Rush. Structured attention networks. In International Conference on Learning Representations , 2017.

Kunisch and T. Pock. A bilevel optimization approach for parameter learning in variational models. SIAM Journal on Imaging Sciences , 6(2):938-983, 2013.

Lafferty, A. McCallum, and F. Pereira. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of ICML , 2001a.

D. Lafferty, A. McCallum, and F. C. Pereira. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In International Conference on Machine Learning , pages 282-289, 2001b.

Lam, J. R. Doppa, S. Todorovic, and T. G. Dietterich. H c-search for structured prediction in computer vision. In CVPR , 2015.

Larsson, A. Arnab, S. Zheng, P. Torr, and F. Kahl. Revisiting deep structured models for pixel-level labeling with gradient-based inference. SIAM Journal on Imaging Sciences , 11(4):2610-2628, 2018.

Lee, Y. Lin, and G. Wahba. Multicategory support vector machines: Theory and application to the classification of microarray data and satellite radiance data. Journal of the American Statistical Association , 99(465):67-81, 2004.

Li and R. Zemel. Mean-field networks. arXiv preprint arXiv:1410.5884 , 2014.

Liu, Y. Jiang, N. Monath, R. Cotterell, and M. Sachan. Autoregressive structured prediction with language models. arXiv preprint arXiv:2210.14698 , 2022.

Long, E. Shelhamer, and T. Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3431-3440, 2015.

Lu and B. Huang. Structured output learning with conditional generative flows. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 5005-5012, 2020.

Lucchi, L. Yunpeng, and P. Fua. Learning for structured prediction using approximate subgradient descent with working sets. In Proceedings of CVPR , 2013.

D. Manning and H. Schütze. Foundations of Statistical Natural Language Processing . The MIT Press, Cambridge, Massachusetts, 1999.

Mao, C. Mohri, M. Mohri, and Y. Zhong. Two-stage learning to defer with multiple experts. In Advances in neural information processing systems , 2023a.

Mao, M. Mohri, and Y. Zhong. H-consistency bounds: Characterization and extensions. In Advances in Neural Information Processing Systems , 2023b.

Mao, M. Mohri, and Y. Zhong. Principled approaches for learning to defer with multiple experts. arXiv preprint arXiv:2310.14774 , 2023c.

Mao, M. Mohri, and Y. Zhong. Predictor-rejector multi-class abstention: Theoretical analysis and algorithms. arXiv preprint arXiv:2310.14772 , 2023d.

Mao, M. Mohri, and Y. Zhong. H-consistency bounds for pairwise misranking loss surrogates. In International conference on Machine learning , 2023e.

Mao, M. Mohri, and Y. Zhong. Ranking with abstention. In ICML 2023 Workshop The Many Facets of Preference-Based Learning , 2023f.

Mao, M. Mohri, and Y. Zhong. Theoretically grounded loss functions and algorithms for scorebased multi-class abstention. arXiv preprint arXiv:2310.14770 , 2023g.

Mao, M. Mohri, and Y. Zhong. Cross-entropy loss functions: Theoretical analysis and applications. In International Conference on Machine Learning , 2023h.

Meister, T. Vieira, and R. Cotterell. Best-first beam search. Transactions of the Association for Computational Linguistics , 8:795-809, 2020.

Mohri, D. Andor, E. Choi, M. Collins, A. Mao, and Y. Zhong. Learning to reject with a fixed predictor: Application to decontextualization. arXiv preprint arXiv:2301.09044 , 2023.

Mohri. Finite-state transducers in language and speech processing. Computational Linguistics , 23 (2):269-311, 1997.

Mohri. Generic glyph[epsilon1] -removal algorithm for weighted automata. In S. Yu and A. Paun, editors, Implementation and Application of Automata, 5th International Conference, CIAA 2000, London, Ontario, Canada, July 24-25, 2000, Revised Papers , volume 2088 of Lecture Notes in Computer Science , pages 230-242. Springer, 2000.

Mohri. Semiring Frameworks and Algorithms for Shortest-Distance Problems. Journal of Automata, Languages and Combinatorics , 7(3):321-350, 2002a.

Mohri. Generic glyph[epsilon1] -removal and input glyph[epsilon1] -normalization algorithms for weighted transducers. Int. J. Found. Comput. Sci. , 13(1):129-143, 2002b.

Mohri. Weighted automata algorithms. In Handbook of Weighted Automata , pages 213-254. Springer, 2009.

Mohri and M. Riley. Weighted determinization and minimization for large vocabulary speech recognition. In G. Kokkinakis, N. Fakotakis, and E. Dermatas, editors, Fifth European Conference on Speech Communication and Technology, EUROSPEECH 1997, Rhodes, Greece, September 22-25, 1997 , pages 131-134. ISCA, 1997.

Mohri, A. Rostamizadeh, and A. Talwalkar. Foundations of Machine Learning . MIT Press, second edition, 2018.

Nadeau and S. Sekine. A survey of named entity recognition and classification. Linguisticae Investigationes , 30(1):3-26, January 2007.

Norouzi, S. Bengio, N. Jaitly, M. Schuster, Y. Wu, D. Schuurmans, et al. Reward augmented maximum likelihood for neural structured prediction. In Advances In Neural Information Processing Systems , 2016.

Nowak, F. Bach, and A. Rudi. Sharp analysis of learning with discrete losses. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 1920-1929, 2019.

Nowak, F. Bach, and A. Rudi. Consistent structured prediction with max-min margin markov networks. In International Conference on Machine Learning , pages 7381-7391, 2020.

Nowak, A. Rudi, and F. Bach. On the consistency of max-margin losses. In International Conference on Artificial Intelligence and Statistics , pages 4612-4633, 2022.

Nowak-Vila, F. Bach, and A. Rudi. A general theory for structured prediction with smooth convex surrogates. arXiv preprint arXiv:1902.01958 , 2019.

Osokin, F. Bach, and S. Lacoste-Julien. On structured prediction theory with calibrated convex surrogate losses. In Advances in Neural Information Processing Systems , 2017.

Pan, P. Liu, Y. Yan, T. Yang, and Y. Yang. Adversarial localized energy network for structured prediction. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 5347-5354, 2020.

Patel, P. Dangati, J.-Y. Lee, M. Boratko, and A. McCallum. Modeling label space interactions in multi-label classification using box embeddings. In International Conference on Learning Representations , 2022.

K. Pillutla, V. Roulet, S. M. Kakade, and Z. Harchaoui. A smoother way to train structured prediction models. In Advances in Neural Information Processing Systems , 2018.

Ross, G. J. Gordon, and D. Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of AISTATS , 2011.

G. Schwing and R. Urtasun. Fully connected deep structured networks. arXiv preprint arXiv:1503.02351 , 2015.

Steinwart. How to compare different loss functions and their risks. Constructive Approximation , 26(2):225-287, 2007.

Stoyanov, A. Ropson, and J. Eisner. Empirical risk minimization of graphical model parameters given approximate inference, decoding, and model structure. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics , pages 725-733, 2011.

Sutskever, O. Vinyals, and Q. V. Le. Sequence to sequence learning with neural networks. In Advances in neural information processing systems , 2014.

S. Tai, R. Socher, and C. D. Manning. Improved semantic representations from tree-structured long short-term memory networks. arXiv preprint arXiv:1503.00075 , 2015.

Taskar, C. Guestrin, and D. Koller. Max-margin Markov networks. In NIPS , 2003a.

Taskar, C. Guestrin, and D. Koller. Max-margin markov networks. In Advances in neural information processing systems , 2003b.

Tsochantaridis, T. Joachims, T. Hofmann, and Y. Altun. Large margin methods for structured and interdependent output variables. JMLR , 6:1453-1484, Dec. 2005a.

Tsochantaridis, T. Joachims, T. Hofmann, Y. Altun, and Y. Singer. Large margin methods for structured and interdependent output variables. Journal of machine learning research , 6(9), 2005b.

Tu and K. Gimpel. Learning approximate inference networks for structured prediction. arXiv preprint arXiv:1803.03376 , 2018.

Tu and K. Gimpel. Benchmarking approximate inference methods for neural structured prediction. arXiv preprint arXiv:1904.01138 , 2019.

Tu, R. Y. Pang, and K. Gimpel. Improving joint training of inference networks and structured prediction energy networks. arXiv preprint arXiv:1911.02891 , 2019.

F. Verhulst. Notice sur la loi que la population suit dans son accroissement. Correspondance mathématique et physique , 10:113--121, 1838.

F. Verhulst. Recherches mathématiques sur la loi d'accroissement de la population. Nouveaux Mémoires de l'Académie Royale des Sciences et Belles-Lettres de Bruxelles , 18:1--42, 1845.

Vinyals, L. Kaiser, T. Koo, S. Petrov, I. Sutskever, and G. Hinton. Grammar as a foreign language. In Proceedings of NIPS , 2015a.

Vinyals, A. Toshev, S. Bengio, and D. Erhan. Show and tell: A neural image caption generator. In Proceedings of CVPR , 2015b.

Wang, S. Fidler, and R. Urtasun. Proximal deep structured models. Advances in Neural Information Processing Systems , 2016.

Weston and C. Watkins. Multi-class support vector machines. Technical report, Citeseer, 1998.

Wu, M. Schuster, Z. Chen, Q. V. Le, M. Norouzi, W. Macherey, M. Krikun, Y. Cao, Q. Gao, K. Macherey, J. Klingner, A. Shah, M. Johnson, X. Liu, Łukasz Kaiser, S. Gouws, Y. Kato, T. Kudo, H. Kazawa, K. Stevens, G. Kurian, N. Patil, W. Wang, C. Young, J. Smith, J. Riesa, A. Rudnick, O. Vinyals, G. Corrado, M. Hughes, and J. Dean. Google's neural machine translation system: Bridging the gap between human and machine translation. CoRR , abs/1609.08144, 2016. URL http://arxiv.org/abs/1609.08144 .

Zhang, L. Sun, and W. Li. A structured prediction approach for statistical machine translation. In Proceedings of IJCNLP , 2008.

Zhang. Statistical analysis of some multi-category large margin classification methods. Journal of Machine Learning Research , 5(Oct):1225-1251, 2004.

Zhang and M. Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In Advances in neural information processing systems , 2018.

Zheng, G. Wu, F. Bao, Y. Cao, C. Li, and J. Zhu. Revisiting discriminative vs. generative classifiers: Theory and implications. arXiv preprint arXiv:2302.02334 , 2023.

Zheng and A. Pronobis. From pixels to buildings: End-to-end probabilistic deep networks for large-scale semantic mapping. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 3511-3518, 2019.

Zheng, S. Jayasumana, B. Romera-Paredes, V. Vineet, Z. Su, D. Du, C. Huang, and P. H. Torr. Conditional random fields as recurrent neural networks. In Proceedings of the IEEE international conference on computer vision , pages 1529-1537, 2015.

Contents of Appendix

A Related work

Structured prediction and neural networks. A variety of deep learning techniques have been used for structured prediction tasks, including a unified neural network architecture for natural language processing [Collobert et al., 2011], energy model-based structured prediction including structured prediction energy networks (SPENs) and various inference methods [Belanger and McCallum, 2016, Tu and Gimpel, 2018, Larsson et al., 2018, Tu and Gimpel, 2019, Tu et al., 2019, Graber and Schwing, 2019, Pan et al., 2020], attention networks incorporating richer structural distributions [Kim et al., 2017], tree-structured long short-term memory (LSTM) networks [Tai et al., 2015], sequence to sequence learning [Sutskever et al., 2014, Edunov et al., 2017], memory-reduced variant of best-first beam search [Meister et al., 2020], end-to-end learning for SPENs [Belanger et al., 2017], conditional generative flow [Lu and Huang, 2020], proximal methods [Wang et al., 2016], CRF using deep features [Jaderberg et al., 2014, Huang et al., 2015, Chen et al., 2014, Schwing and Urtasun, 2015, Chen et al., 2015], non-iterative feed-forward predictors [Stoyanov et al., 2011, Domke, 2013, Kunisch and Pock, 2013, Hershey et al., 2014, Li and Zemel, 2014, Belharbi et al., 2018, Zheng et al., 2015], deep value network [Gygli et al., 2017], fully convolutional networks [Long et al., 2015], constraint reasoning tool [Dragone et al., 2021, Jiang et al., 2022], multi-label box model[Patel et al., 2022], probabilistic deep networks[Zheng and Pronobis, 2019, Jang et al., 2023], autoregressive methods [Liu et al., 2022], nonlinear output transformations and embedding [Graber et al., 2018, Brogat-Motte et al., 2020], smoothing methods [Pillutla et al., 2018] and structural training [Choi et al., 2016, Ahmad et al., 2023]. Let us also mention ensemble algorithms for structured prediction algorithms [Cortes, Kuznetsov, and Mohri, 2014a,b], which can be used to combine several algorithms for this problem.

Consistency in structured prediction. Here, we discuss in detail previous work on consistency in structured prediction [Osokin et al., 2017, Ciliberto et al., 2016, Blondel, 2019, Nowak et al., 2020, 2022, Ciliberto et al., 2019, 2020, Nowak-Vila et al., 2019, Nowak et al., 2019, Cabannes et al., 2021, Cabannnes et al., 2020, Corro, 2023].

Osokin et al. [2017], Nowak et al. [2020] and Nowak et al. [2022] pointed out that the max-margin Markov networks (M3N), or more generally structural SVMs may not be Bayes-consistent. Instead, Osokin et al. [2017] proposed the first Bayes-consistent surrogate loss in the structured prediction setting, the quadratic surrogate (QS) loss. A general theory of QS was further developed in [NowakVila et al., 2019, Nowak et al., 2019]. However, as pointed out in Section 1, the quadratic surrogate loss formulation casts the structured prediction problem as a regression problem and is not a typical formulation even in the binary classification case.

Nowak et al. [2020] proposed a consistent method called max-min margin Markov networks (M4N) derived from first principles for binary SVM. However, this method is restricted to SVM-type loss functions. Instead, we propose broad families of surrogate losses, which can be naturally derived from common multi-class losses including the logistic loss.

Nowak et al. [2022] addressed the inconsistency of Max-Margin loss in structured prediction by introducing the notion of Restricted-Max-Margin, where the maximization is performed over a subset of the original domain. Their method is based on an implicit embedding [Finocchiaro et al., 2019]; a general framework for structured prediction has been further developed by Ciliberto et al. [2020]. However, these methods are only applied to polyhedral-type surrogates which are not as smooth as the logistic loss. Thus, the resulting surrogate losses may not be favorable from the optimization point of view. Instead, our novel families of surrogate losses are very general and can be smooth, including a new structured logistic loss, for which we describe efficient gradient computation algorithms.

Ciliberto et al. [2016] focused on a least squares surrogate loss function and corresponding framework. In this framework, the structured prediction problem is cast as a regression problem. They derived a regularization approach to structured prediction from the least squares surrogate loss and proved the Bayes-consistency of that approach. Ciliberto et al. [2019] focused on a local structure-adapted framework for structured prediction. They proposed a novel structured prediction algorithm that adaptively leverages locality in the learning problem. Ciliberto et al. [2020] developed a general framework for structured prediction based on implicit embedding. Their methods lead to polyhedraltype surrogates losses that benefit from Bayes-consistency.

On the other hand, our work presents an extensive study of surrogate losses for structured prediction supported by H -consistency bounds. Different from the surrogate loss studied in the previous work,

the formulations of our proposed surrogate losses including structured comp-sum losses and structured constrained losses are completely novel and do not cast structured prediction problems as a regression problem. Furthermore, we prove stronger consistency guarantees that imply Bayes-consistency for these new proposed families of surrogate loss.

Other related work on structured prediction includes: projection-based losses for structured prediction [Blondel, 2019]; fast convergence rates for general structured prediction problems [Cabannes et al., 2021]; a unified framework for dealing with partial labelling [Cabannnes et al., 2020]; and an analysis of the inconsistency of separable negative log-likelihood losses for structured prediction [Corro, 2023].

B Proof of Lemma 3

Lemma 3. The best-in-class conditional error and the conditional regret for a target loss L in structured prediction can be expressed as follows:

Proof. By the definition, the conditional L -risk can be expressed as follows:

Since { h ( x ) ∶ h ∈ H } = H ( x ) , the best-in-class conditional error can be expressed as follows:

which proves the first part of the lemma. By the definition,

C Proofs for structured max losses

Theorem 4 ( Negative results of L max ) . Assume that n > 2 and that Φ u ( v ) is convex and nonincreasing for u = 1 . Then, the max structured loss L max is not Bayes-consistent.

Proof. For the structured max loss L max , the conditional L max -risk can be expressed as follows:

Take glyph[lscript] ( y ′ , y ) = 1 y ≠ y ′ to be the zero-one loss. Since glyph[lscript] ( y ′ , y ) = 1 for any y ≠ y ′ , the conditional L max -risk can be reformulated as follows:

Consider the distribution that supports on a singleton domain { x } . Take y 1 ≠ y 2 ∈ Y such that y 1 ≠ n and y 2 ≠ n . We define the conditional distribution as p ( x, y 1 ) = p ( x, y 2 ) = 1 2 and p ( x, y ) = 0 for other y ∈ Y . Then, by using the fact that Φ 1 ( v ) is convex and non-increasing, we obtain

where the equality can be achieved by h ∗ ∈ H , defined as h ∗ ( x, 1 ) = h ∗ ( x, 2 ) = . . . = h ∗ ( x, n ) . Therefore, h ∗ is a Bayes classifier of the structured max loss. Note that h ∗ ( x ) = n . However, by Lemma 3, in such a case, the Bayes classifier h ∗ glyph[lscript] of the target loss satisfies that

Thus, we obtain h ∗ ≠ h ∗ glyph[lscript] . Therefore, L max is not Bayes-consistent.

D Proofs for Voted Conditional Random Field

Theorem 5 ( Negative result of L VCRF ) . The Voted Conditional Random Field L VCRF is not Bayesconsistent.

Proof. For the Voted Conditional Random Field L VCRF , the conditional L VCRF -risk can be expressed as follows:

Consider the distribution that supports on a singleton domain { x } . Note that R L VCRF = C L VCRF is convex with respect to h ( x, y ) , y = 1 , . . . , n . To find the global minimum, we will differentiate C L VCRF with respect to h ( x, y ) for any y ∈ Y and setting the derivatives to zero. Thus, we obtain for any y ∈ Y ,

Using the fact that ∑ y ′ ≠ y e glyph[lscript] ( y,y ′ )+ h ( x,y ′ )-h ( x,y ) = ∑ y ′ ∈ Y e glyph[lscript] ( y,y ′ )+ h ( x,y ′ )-h ( x,y ) -e glyph[lscript] ( y,y )+ h ( x,y )-h ( x,y ) to further simplify the LHS of (16), we obtain for any y ∈ Y ,

Consider a target loss function L such that e glyph[lscript] ( y,y ′ ) = Φ y Φ y ′ , that is glyph[lscript] ( y, y ′ ) = log ( Φ y ) + log ( Φ y ′ ) , where Φ y is a function mapping from Y to R + . For this special choice of the target loss function, the expression of glyph[lscript] ( y, y ′ ) decouple and (17) can be simplified to

Therefore, for the Bayes classifier h ∗ of Voted Conditional Random Field, by (18), we have

which implies that

However, by Lemma 3, in such a case, the Bayes classifier h ∗ glyph[lscript] of the target loss satisfies that

Thus, we obtain h ∗ ≠ h ∗ glyph[lscript] in general. Indeed, consider the case where p ( x, y ) = Φ 2 y ∑ n k = 1 Φ 2 k , y ∈ Y . Then, h ∗ ( x ) = argmax y ′ ∈ Y Φ y ′ ∑ n k = 1 Φ 2 k = argmax y ′ ∈ Y Φ y ′ ≠ argmin y ′ ∈ Y Φ y ′ = h ∗ glyph[lscript] ( x ) when { Φ y ∶ y ∈ Y } are not equal. Therefore, L VCRF is not Bayes-consistent.

E Proofs for structured comp-sum losses

E.1 Structured logistic loss

Theorem 11 ( H -consistency bound of L comp log ) . Assume that H is symmetric and complete. Then, for any target loss glyph[lscript] , hypothesis h ∈ H and any distribution,

Proof. For the comp-sum structured loss L comp log , the conditional L comp log -risk can be expressed as follows:

where we denote by q ( x, y ′ ) = ∑ y ∈ Y p ( x, y ) glyph[lscript] ( y ′ , y ) ∈ [ 0 , 1 ] and S ( x, y ) = e h ( x,y ) ∑ y ′′ ∈ Y e h ( x,y ′′ ) ∈ [ 0 , 1 ] with the constraint that ∑ y ∈ Y S ( x, y ) = 1 . Let y min = argmin y ′ ∈ Y ∑ y ∈ Y p ( x, y ) glyph[lscript] ( y ′ , y ) , where we choose the label with the highest index under the natural ordering of labels as the tie-breaking strategy. For any h ∈ H such that h ( x ) ≠ y min and x ∈ X , by the symmetry and completeness of H , we can always find a family of hypotheses { h µ ∶ µ ∈ [-S ( x, y min ) , S ( x, h ( x ))]} ⊂ H such that S µ ( x, ⋅) = e hµ ( x, ⋅) ∑ y ′ ∈ Y e hµ ( x,y ′ ) take the following values:

Note that S µ satisfies the constraint:

By (20) and using the fact that H ( x ) = Y when H is symmetric, we obtain

Differentiating with respect to µ yields the optimal value µ ∗ = q ( x, h ( x )) S ( x, h ( x ))-q ( x,y min ) S ( x,y min ) q ( x,y min )+ q ( x, h ( x )) . Plugging in that value gives:

Differentiating with respect to S shows that the minimum is attained for S ( x, h ( x )) = S ( x, y min ) , which gives:

Since the function t ↦ t 2 4 is convex, by Jensen's inequality, we obtain for any hypothesis h ∈ H and any distribution,

which leads to

E.2 Structured sum-exponential loss

Theorem 12 ( H -consistency bound of L comp exp ) . Assume that H is symmetric and complete. Then, for any target loss glyph[lscript] , hypothesis h ∈ H and any distribution,

Proof. For the comp-sum structured loss L comp exp , the conditional L comp exp -risk can be expressed as follows:

where we denote by q ( x, y ′ ) = ∑ y ∈ Y p ( x, y ) glyph[lscript] ( y ′ , y ) ∈ [ 0 , 1 ] and S ( x, y ) = e h ( x,y ) ∑ y ′′ ∈ Y e h ( x,y ′′ ) ∈ [ 0 , 1 ] with the constraint that ∑ y ∈ Y S ( x, y ) = 1 . Let y min = argmin y ′ ∈ Y ∑ y ∈ Y p ( x, y ) glyph[lscript] ( y ′ , y ) , where we choose the label with the highest index under the natural ordering of labels as the tie-breaking strategy. For any h ∈ H such that h ( x ) ≠ y min and x ∈ X , by the symmetry and completeness of H , we can always find a family of hypotheses { h µ ∶ µ ∈ [-S ( x, y min ) , S ( x, h ( x ))]} ⊂ H such that S µ ( x, ⋅) = e hµ ( x, ⋅) ∑ y ′ ∈ Y e hµ ( x,y ′ ) take the following values:

Note that S µ satisfies the constraint:

By (22) and using the fact that H ( x ) = Y when H is symmetric, we obtain

(differentiating with respect to q ( x,y min )+ √ q ( x, h ( x ))

(differentiating with respect to S to minimize, minimum is attained when S ( x, h ( x )) = S ( x, y min ) = 1 2 )

Since the function t ↦ t 2 4 is convex, by Jensen's inequality, we obtain for any hypothesis h ∈ H and any distribution,

which leads to

E.3 Structured generalized cross-entropy loss

Theorem 13 ( H -consistency bound of L comp gce ) . Assume that H is symmetric and complete. Then, for any target loss glyph[lscript] , hypothesis h ∈ H and any distribution,

Proof. For the comp-sum structured loss L comp gce , the conditional L comp gce -risk can be expressed as follows:

where we denote by q ( x, y ′ ) = ∑ y ∈ Y p ( x, y ) glyph[lscript] ( y ′ , y ) ∈ [ 0 , 1 ] and S ( x, y ) = e h ( x,y ) ∑ y ′′ ∈ Y e h ( x,y ′′ ) ∈ [ 0 , 1 ] with the constraint that ∑ y ∈ Y S ( x, y ) = 1 . Let y min = argmin y ′ ∈ Y ∑ y ∈ Y p ( x, y ) glyph[lscript] ( y ′ , y ) , where we choose the label with the highest index under the natural ordering of labels as the tie-breaking strategy. For any h ∈ H such that h ( x ) ≠ y min and x ∈ X , by the symmetry and completeness of H , we can always find a family of hypotheses { h µ ∶ µ ∈ [-S ( x, y min ) , S ( x, h ( x ))]} ⊂ H such that S µ ( x, ⋅) = e hµ ( x, ⋅) ∑ y ′ ∈ Y e hµ ( x,y ′ ) take the following values:

Note that S µ satisfies the constraint:

By (24) and using the fact that H ( x ) = Y when H is symmetric, we obtain

α α (differentiating with respect to µ to optimize, optimum µ ∗ = q ( x, h ( x )) 1 1 -α S ( x, h ( x ))-q ( x,y min ) 1 1 -α S ( x,y min ) q ( x,y min ) 1 1 -α + q ( x, h ( x )) 1 1 -α ) ≥ 1 αn α [ 2 α ( q ( x, y min ) 1 1 -α + q ( x, h ( x )) 1 1 -α ) 1 -α -q ( x, y min ) -q ( x, h ( x ))] (differentiating with respect to S to minimize, minimum is attained when S ( x, h ( x )) = S ( x, y min ) = 1 n ) ≥ ( q ( x, h ( x )) -q ( x, y min )) 2 4 n α ( ( a 1 1 -α + b 1 1 -α 2 ) 1 -α -a + b 2 ≥ α 4 ( a -b ) 2 , ∀ a, b ∈ [ 0 , 1 ] , 0 ≤ a + b ≤ 1 ) = (∑ y ∈ Y p ( x, y ) glyph[lscript] ( h ( x ) , y ) - ∑ y ∈ Y p ( x, y ) glyph[lscript] ( y min , y )) 2 4 n α = ∆ C L , H ( h, x ) 2 4 n α . (by Lemma 3 and H ( x ) = Y )

Since the function t ↦ t 2 4 n α is convex, by Jensen's inequality, we obtain for any hypothesis h ∈ H and any distribution,

which leads to

E.4 Structured mean absolute error loss

Theorem 14 ( H -consistency bound of L comp mae ) . Assume that H is symmetric and complete. Then, for any target loss glyph[lscript] , hypothesis h ∈ H and any distribution,

Proof. For the comp-sum structured loss L comp mae , the conditional L comp mae -risk can be expressed as follows:

where we denote by q ( x, y ′ ) = ∑ y ∈ Y p ( x, y ) glyph[lscript] ( y ′ , y ) ∈ [ 0 , 1 ] and S ( x, y ) = e h ( x,y ) ∑ y ′′ ∈ Y e h ( x,y ′′ ) ∈ [ 0 , 1 ] with the constraint that ∑ y ∈ Y S ( x, y ) = 1 . Let y min = argmin y ′ ∈ Y ∑ y ∈ Y p ( x, y ) glyph[lscript] ( y ′ , y ) , where we choose the label with the highest index under the natural ordering of labels as the tie-breaking strategy. For any h ∈ H such that h ( x ) ≠ y min and x ∈ X , by the symmetry and completeness of H , we can always find a family of hypotheses { h µ ∶ µ ∈ [-S ( x, y min ) , S ( x, h ( x ))]} ⊂ H such that S µ ( x, ⋅) = e hµ ( x, ⋅) ∑ y ′ ∈ Y e hµ ( x,y ′ ) take the following values:

Note that S µ satisfies the constraint:

By (26) and using the fact that H ( x ) = Y when H is symmetric, we obtain

∆ C L comp , H ( h, x ) = C L comp ( h, x ) -C ∗ L comp ( H , x ) ≥ C L comp ( h, x ) -inf µ ∈ R C L comp ( h µ , x ) = sup µ ∈[-S ( x,y min ) , S ( x, h ( x ))] { q ( x, y min )[-S ( x, y min ) + S ( x, h ( x )) -µ ] + q ( x, h ( x ))[-S ( x, h ( x )) + S ( x, y min ) + µ ]} = q ( x, y min ) S ( x, h ( x )) -q ( x, h ( x )) S ( x, h ( x )) (differentiating with respect to µ to optimize, optimum µ ∗ = -S ( x, y min ) ) ≥ 1 n ( q ( x, y min ) -q ( x, h ( x ))) (differentiating with respect to S to minimize, minimum is attained when S ( x, h ( x )) = 1 n ) = ∑ y ∈ Y p ( x, y ) glyph[lscript] ( h ( x ) , y ) - ∑ y ∈ Y p ( x, y ) glyph[lscript] ( y min , y ) n = ∆ C L , H ( h, x ) n (by Lemma 3 and H ( x ) = Y )

Therefore, we obtain for any hypothesis h ∈ H and any distribution,

which leads to

F Proofs for structured constrained losses

F.1 Structured constrained exponential loss

Theorem 15 ( H -consistency bound of L cstnd exp ) . Assume that H is symmetric and complete. Then, for any target loss glyph[lscript] , hypothesis h ∈ H and any distribution,

Proof. Denote by q ( x, y ′ ) = ∑ y ∈ Y p ( x, y ) glyph[lscript] ( y ′ , y ) ∈ [ 0 , glyph[lscript] max ] . For the constrained structured loss L cstnd exp , the conditional L cstnd exp -risk can be expressed as follows:

Let y min = argmin y ′ ∈ Y ∑ y ∈ Y p ( x, y ) glyph[lscript] ( y ′ , y ) , where we choose the label with the highest index under the natural ordering of labels as the tie-breaking strategy. For any h ∈ H such that h ( x ) ≠ y min and x ∈ X , by the symmetry and completeness of H , we can always find a family of hypotheses { h µ ∶ µ ∈ R } ⊂ H such that h µ ( x, ⋅) take the following values:

Note that the hypotheses h µ satisfies the constraint:

Since ∑ y ∈ Y h ( x, y ) = 0 , there must be non-negative scores. By definition of h ( x ) as a maximizer, we must thus have h ( x, h ( x )) ≥ 0 . By (28) and using the fact that H ( x ) = Y when H is symmetric, we obtain

Since the function t ↦ t 2 4 glyph[lscript] max is convex, by Jensen's inequality, we obtain for any hypothesis h ∈ H and any distribution,

which leads to

F.2 Structured constrained squared-hinge loss

Theorem 16 ( H -consistency bound of L cstnd sq -hinge ) . Assume that H is symmetric and complete. Then, for any target loss glyph[lscript] , hypothesis h ∈ H and any distribution,

Proof. Denote by q ( x, y ′ ) = ∑ y ∈ Y p ( x, y ) glyph[lscript] ( y ′ , y ) ∈ [ 0 , glyph[lscript] max ] . For the constrained structured loss L cstnd sq -hinge , the conditional L cstnd sq -hinge -risk can be expressed as follows:

Let y min = argmin y ′ ∈ Y ∑ y ∈ Y p ( x, y ) glyph[lscript] ( y ′ , y ) , where we choose the label with the highest index under the natural ordering of labels as the tie-breaking strategy. For any h ∈ H such that h ( x ) ≠ y min and x ∈ X , by the symmetry and completeness of H , we can always find a family of hypotheses { h µ ∶ µ ∈ R } ⊂ H such that h µ ( x, ⋅) take the following values:

Note that the hypotheses h µ satisfies the constraint:

Since ∑ y ∈ Y h ( x, y ) = 0 , there must be non-negative scores. By definition of h ( x ) as a maximizer, we must thus have h ( x, h ( x )) ≥ 0 . By (30) and using the fact that H ( x ) = Y when H is symmetric, we obtain

Since the function t ↦ t 2 is convex, by Jensen's inequality, we obtain for any hypothesis h ∈ H and any distribution,

which leads to

F.3 Structured constrained hinge loss

Theorem 17 ( H -consistency bound of L cstnd hinge ) . Assume that H is symmetric and complete. Then, for any target loss glyph[lscript] , hypothesis h ∈ H and any distribution,

Proof. Denote by q ( x, y ′ ) = ∑ y ∈ Y p ( x, y ) glyph[lscript] ( y ′ , y ) ∈ [ 0 , glyph[lscript] max ] . For the constrained structured loss L cstnd hinge , the conditional L cstnd hinge -risk can be expressed as follows:

Let y min = argmin y ′ ∈ Y ∑ y ∈ Y p ( x, y ) glyph[lscript] ( y ′ , y ) , where we choose the label with the highest index under the natural ordering of labels as the tie-breaking strategy. For any h ∈ H such that h ( x ) ≠ y min and x ∈ X , by the symmetry and completeness of H , we can always find a family of hypotheses { h µ ∶ µ ∈ R } ⊂ H such that h µ ( x, ⋅) take the following values:

Note that the hypotheses h µ satisfies the constraint:

Since ∑ y ∈ Y h ( x, y ) = 0 , there must be non-negative scores. By definition of h ( x ) as a maximizer, we must thus have h ( x, h ( x )) ≥ 0 . By (32) and using the fact that H ( x ) = Y when H is symmetric, we obtain

Therefore, we obtain for any hypothesis h ∈ H and any distribution,

which leads to

F.4 Structured constrained ρ -margin loss

Theorem 18 ( H -consistency bound of L cstnd ρ ) . Assume that H is symmetric and complete. Then, for any target loss glyph[lscript] , hypothesis h ∈ H and any distribution,

Proof. Denote by q ( x, y ′ ) = ∑ y ∈ Y p ( x, y ) glyph[lscript] ( y ′ , y ) ∈ [ 0 , glyph[lscript] max ] . For the constrained structured loss L cstnd ρ , the conditional L cstnd ρ -risk can be expressed as follows:

Let y min = argmin y ′ ∈ Y ∑ y ∈ Y p ( x, y ) glyph[lscript] ( y ′ , y ) , where we choose the label with the highest index under the natural ordering of labels as the tie-breaking strategy. For any h ∈ H such that h ( x ) ≠ y min and x ∈ X , by the symmetry and completeness of H , we can always find a family of hypotheses { h µ ∶ µ ∈ R } ⊂ H such that h µ ( x, ⋅) take the following values:

Note that the hypotheses h µ satisfies the constraint:

Since ∑ y ∈ Y h ( x, y ) = 0 , there must be non-negative scores. By definition of h ( x ) as a maximizer, we must thus have h ( x, h ( x )) ≥ 0 . By (34) and using the fact that H ( x ) = Y when H is symmetric, we obtain

Therefore, we obtain for any hypothesis h ∈ H and any distribution,

which leads to

G Efficient gradient computation and inference

Here, we describe efficient algorithms for the computation of the gradients for the loss functions L comp log and L comp exp . We also briefly discuss an efficient algorithm for inference.

G.1 Efficient gradient computation for L comp log

We first present an efficient algorithm for the computation of the quantities L ( z , s ) in the important case of rational losses, next in the case of Markovian losses.

Rational losses. Rational losses form a general family of loss functions based on rational kernels [Cortes et al., 2004] that includes, in particular, n -gram losses, which can be defined for a pair of sequences ( y, y ′ ) as the negative inner product of the vectors of n -gram counts of y and y ′ .

Our algorithm bears some similarity to that of Cortes et al. [2018] for the computation of the gradient of the VCRF loss function. It is however distinct because the structured prediction loss function we are considering and our definition of rational loss are both different. We will adopt a similar notation and terminology. Recall that for any sequence y , we denote by y i the symbol in its i th position and by y j i = y i y i + 1 /uni22EF y j the substring of y starting at position i and ending at j . We denote by E A the set of transitions of a WFA A .

Let U be a weighted finite-state transducer (WFST) over the (+ , ×) semiring over the reals, with ∆ as both the input and output alphabet. Then, we define the rational loss associated to U for all y, y ′ ∈ ∆ ∗ by glyph[lscript] ( y, y ′ ) = U ( y, y ′ ) .

Let Y denote a WFA over the (+ , ×) semiring accepting the set of all sequences of length l with weight one and let Y i denote the WFA accepting only y i with weight one. Then, by definition, the weighted transducer Y ○ U ○ Y i obtained by composition maps each sequence y in ∆ l to y i with weight U ( y, y i ) . The WFA Π 1 ( Y ○ U ○ Y i ) derived from that transducer by projection on the input (that is by removing output labels) is associating to each sequence y weight U ( y, y i ) . We use weighted determinization [Mohri, 1997] to compute an equivalent deterministic WFA denote M . As shown by Cortes et al. [2015][Theorem 3], M can be computed in polynomial time. M

0

1

a:a/1

b:b/1

2

a:a/1

b:b/1

3/1

a:a/1

b:b/1

0

1

a:a/1

2

b:b/1

3/1

a:a/1

admits a unique path labeled with any sequence y ∈ ∆ l and the weight of that path is U ( y, y i ) . The weight of that accepting path is obtained by multiplying the weights of its transitions and that of the final state.

We now define a deterministic p -gram WFA N that accepts all sequences y ∈ ∆ l with each of its states ( z ′ , s ) encoding a ( p -1 ) -gram z ′ read to reach it and the position s in the sequence y at which it is reached. The transitions of N are therefore defined as follows with weight one:

The initial state is ( glyph[epsilon1], 0 ) and the final states are those with the second element of the pair (the position) being l . Note that, by construction, N is deterministic. Then, the composition (or intersection) WFA N ○ M still associates the same weight as M to each input string y ∈ ∆ l . However, the states in that composition help us compute L ( z , s ) . In particular, for any z ∈ ∆ p and s ∈ [ l ] , let E ( z , s ) be the set of transitions of N ○ M constructed by pairing the transition

(

ε

, 0)

(a, 1)

a:a/1

(b, 1)

b:b/1

(a, 2)

a:a/1

(b, 2)

b:b/1

a:a/1

b:b/1

(( z p -1 1 , s -1 ) , z p , ω ( z , s ) , ( z p 2 , s )) in N with a transition ( q M , z p , ω, q ′ M ) in M . They admit the following form:

The WFA N ○ M is deterministic as a composition of two deterministic WFAs. Thus, there is a unique path labeled with a sequence y ∈ ∆ l in N ○ M and y admits the substring z ending at position

s iff that path goes through a transition in E ( z , s ) when reaching position s . Therefore, to compute L ( z , s ) , it suffices for us to compute the sum of the weights of all paths in N ○ M going through a transition in E ( z , s ) . This can be done straightforwardly using the forward-backward algorithm or two single-source shortest-distance algorithm over the (+ , ×) semiring [Mohri, 2002a], one from the initial state, the other one from the final states. Since N ○ M is acyclic and admits O ( l ∣ ∆ ∣ p ) transitions, we can compute all the quantities L ( z , s ) , s ∈ [ l ] and z ∈ ∆ p , in time O ( l ∣ ∆ ∣ p ) .

Markovian loss. We consider adopting a Markovian assumption, which is commonly adopted in natural language processing [Manning and Schütze, 1999]. We will assume that glyph[lscript] can be decomposed as follows for all y, y ′ ∈ ∆ l : glyph[lscript] ( y, y ′ ) = ∏ l t = 1 glyph[lscript] t ( y t t -p + 1 , y ′ ) . Thus, we can write:

To efficiently compute L ( z , s ) , we will use a WFA representation similar to the one used by Cortes et al. [2016, 2018] and, for convenience, will adopt a similar notation. L ( z , s ) coincides with a flow computation in a WFA A that we now define. A has the following set of states:

with I A = ( ε, 0 ) its single initial state, F A = {( y l l -p + 1 , l )∶ y ∈ ∆ l } its set of final states, and a transition from state ( y t -1 t -p + 1 , t -1 ) to state ( y t -1 t -p + 2 b, t ) with label b and weight ω ( y t -1 t -p + 1 b, t ) = glyph[lscript] t ( y t -1 t -p + 1 b, y i ) , that is the following set of transitions:

By construction, A is deterministic. The weight of a path in A is obtained by multiplying the weights of its constituent transitions. In view of that, L ( z , s ) can be seen as the sum of the weights of all paths in A going through the transition from state ( z p -1 1 , s -1 ) to ( z p 2 , s ) with label z p .

For any state ( y t t -p + 1 , t ) ∈ Q A , we denote by α (( y t t -p + 1 , t )) the sum of the weights of all paths in A from the initial state I A to ( y t t -p + 1 , t ) and by β (( y t t -p + 1 , t )) the sum of the weights of all paths from ( y t t -p + 1 , t ) to a final state. Then, L ( z , s ) is given by

Since A is acyclic, α and β can be computed for all states in linear time in the size of A using a single-source shortest-distance algorithm over the (+ , ×) semiring or the so-called forward-backward algorithm. Thus, since A admits O ( l ∣ ∆ ∣ p ) transitions, we can also compute all quantities L ( z , s ) , s ∈ [ l ] and z ∈ ∆ p , in time O ( lr p ) .

G.2 Efficient gradient computation for L comp exp

In this section, we provide a brief overview of the gradient computation for L comp exp , which is similar to the approach used for L comp log .

Note that the loss L comp exp on a given point ( x i , y i ) can be expressed as follows:

The gradient of L comp exp is therefore given by

An efficient computation of these terms is not straightforward since the summations run over an exponential number of sequences for y . However, we will leverage the Markovian property of the features to design an efficient computation. This approach is similar to what we demonstrated earlier for L comp log . We start with identifying the most computationally challenging terms by rewriting the expression of the gradient of L comp exp in the following lemma.

Lemma 19. For any w ∈ R d , the gradient of L comp exp can be expressed as follows:

where Q ′ w ( z , s ) = ∑ y ∶ y s s -p + 1 = z e w ⋅ Ψ ( x i ,y ) , C w ( z , s ) = ∑ y ∶ y s s -p + 1 = z glyph[lscript] ( y, y i ) e -w ⋅ Ψ ( x i ,y ) and N w = ∑ y ∈ ∆ l glyph[lscript] ( y, y i ) e -w ⋅ Ψ ( x i ,y ) .

Proof. Using the decomposition of the feature vector, we can write:

This completes the proof.

It was shown by Cortes et al. [2016, 2018] that all of the quantities Q ′ w ( z , s ) for z ∈ ∆ p and s ∈ [ l ] and Z w can be computed efficiently in time O ( lr p ) , where r = ∣ ∆ ∣ . Thus, the remaining bottleneck in the gradient computation suggested by Lemma 19 is the evaluation of the quantities C w ( z , s ) for z ∈ ∆ p and s ∈ [ l ] and N w . As with the loss function L comp log discussed in the previous section, we will analyze the computation of these quantities first in the case of rational losses, next in that of Markovian loss.

Rational losses. We will adopt the same notation as in the case of the L comp log loss with the same definition of a rational loss : glyph[lscript] is a rational loss if there exists a WFST over the (+ , ×) semiring over the reals with ∆ as both the input and output alphabet such that for all y, y ′ ∈ ∆ ∗ , we have glyph[lscript] ( y, y ′ ) = U ( y, y ′ ) .

Our algorithm is also somewhat similar to the one described for the L comp log loss or that of Cortes et al. [2018] for the computation of the gradient of the VCRF loss function. There are, however, several differences here too because the quantities computed and thus the automata operations required are distinct.

Exactly as in the case of L comp log loss, we first define a deterministic WFA M over the (+ , ×) semiring that can be computed in polynomial time and that admits a unique path labeled with any sequence

y ∈ ∆ l , whose weight is U ( y, y i ) . Next, as in [Cortes et al., 2018], we define a deterministic WFA A such that

The set of states Q A of A are defined as Q A = {( y s s -p + 1 , s )∶ y ∈ ∆ l , s = 0 , . . . , l } , with I A = ( ε, 0 ) its single initial state, F A = {( y l l -p + 1 , l )∶ y ∈ ∆ l } its set of final states, and with a transition from state ( y s -1 s -p + 1 , s -1 ) to state ( y s -1 s -p + 2 a, s ) with label a and weight, that is, the following set of transitions:

Then, by definition of composition or intersection [Mohri, 2009], the WFA ( M ○ A ) is deterministic and admits a unique path labeled with any given y ∈ ∆ l whose weight is ( M ○ A )( y ) = M ( y )⋅ A ( y ) = glyph[lscript] ( y, y i ) e -w ⋅ Ψ ( x i ,y ) .

Now, N w coincides with the sum of the weights of all accepted paths in this WFA. Thus, since ( M ○ A ) is acyclic, it can be computed in time linear in the size of ( M ○ A ) , that is its number of transitions. For any s ∈ [ l ] and z ∈ ∆ p , C w ( z , s ) is the sum of the weights of all paths in

(

M

○

A

)

admitting

z

labeled with a sequence as a substring ending at position

states in the composition

y

s

. The

(

M

○

A

)

help us compute

(

ε

, 0)

(a, 1)

a:a/exp(-w·

ψ

(x,

ε

a,1))

(b, 1)

b:b/exp(-w·

ψ

(x,

ε

b,1))

(a, 2)

a:a/exp(-w·

ψ

(x,aa,2))

(b, 2)

b:b/exp(-w·

ψ

(x,ab,2))

a:a/exp(-w·

ψ

(x,ba,2))

b:b/exp(-w·

ψ

(x,bb,2))

C w ( z , s ) . As in the case of the L comp log loss, for any z ∈ ∆ p and s ∈ [ l ] , we define E ( z , s ) as the set of transitions of ( M ○ A ) constructed by pairing a transition ( q M , z p , ω M , q ′ M ) in M with the transition (( z p -1 1 , s -1 ) , z p , ω ( z , s ) , ( z p 2 , s )) in A . They admit the following form:

The WFA ( M ○ A ) is deterministic as a composition of two deterministic WFAs. Thus, there is a unique path labeled with a sequence y ∈ ∆ l in ( M ○ A ) and y admits the substring z ending at position s iff that path goes through a transition in E ( z , s ) when reaching position s . Therefore, to compute C w ( z , s ) , it suffices for us to compute the sum of the weights of all paths in C w ( z , s ) going through a transition in E ( z , s ) . This can be done straightforwardly using the forward-backward algorithm or two single-source shortest-distance algorithm over the (+ , ×) semiring [Mohri, 2002a], one from the initial state, the other one from the final states. Since ( M ○ A ) is acyclic and admits O ( l ∣ ∆ ∣ p ) transitions, we can compute all the quantities C w ( z , s ) , s ∈ [ l ] and z ∈ ∆ p , in time O ( l ∣ ∆ ∣ p ) .

Markovian loss. Here, we adopt the Markovian assumption and assume that glyph[lscript] can be decomposed as follows for all y, y ′ ∈ ∆ l : glyph[lscript] ( y, y ′ ) = ∏ l t = 1 glyph[lscript] t ( y t t -p + 1 , y ′ ) . Thus, the quantity C w ( z , s ) can be written as:

Then, we can proceed as in the Markovian loss case for the loss function L comp log except that instead of the WFA A used there, we define here a similar WFA A ′ . The only difference is that the weight ω ( y t -1 t -p + 1 b, t ) = glyph[lscript] t ( y t -1 t -p + 1 b, y i ) for the WFA A is replaced with ω ′ ( y t -1 t -p + 1 b, t ) = glyph[lscript] t ( y t -1 t -p + 1 b, y i ) e -w ⋅ ̃ ψ ( x i ,y t -1 t -p + 1 b,t ) . With the same argument, we can compute all quantities C w ( z , s ) , s ∈ [ l ] and z ∈ ∆ p , in time O ( lr p ) . The quantity N w can also be efficiently computed in time O ( lr p ) since it is the sum of the weights of all paths in A ′ .

G.3 Efficient Inference

We focused on the problem of efficient computation of the gradient. Inference is also a key problem in structured prediction since the label with a highest score must be determined out of an exponentially large set of possible ones. However, for the linear hypotheses considered in the previous sections, this problem can be efficiently tackled since it can be cast as a shortest-distance problem in a directed acyclic graph, as in [Cortes, Kuznetsov, Mohri, and Yang, 2016].

More generally, an efficient gradient computation, efficient inference and other related algorithms can benefit from standard weighted automata and transducer optimization algorithms such as glyph[epsilon1] -removal [Mohri, 2000, 2002b] and determinization [Mohri, 1997, Mohri and Riley, 1997, Allauzen and Mohri, 2003, 2004] (see also the survey chapter [Mohri, 2009]).

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies $\\mathcal{H}$-consistency of surrogate losses for structured prediction. The authors show that non classic surrogate losses are not $\\mathcal{H}$-consistent thus not Bayes-consistent. They propose two families of $\\mathcal{H}$-consistent losses as extensions to existing losses and algorithms for two special cases of loss.\n\nSTRENGTHS:\nThe manuscript is well-written with few typos. The theoretical results are sound and novel. Practical optimization methods are provided to show that the proposed losses are not only consistent but also computationally feasible.\n\nWEAKNESSES:\nI don't see obvious weaknesses in the manuscript. A possible drawback might be the lack of empirical results.\n\nQUESTIONS:\nI have a few minor concerns as follows.\n\n1. The inconsistency results for each individual classic loss in Section 3 are not new but the authors seem to give a novel unified view of four losses. Does this include Fenchel-Young losses?\n\n2. The consistency bounds in Theorem 6 and 8 depend on a quantity defined by the minimizability gap, which is difficult or impossible to minimize or estimate. I understand that a non-asymptotic bound is desirable but isn't it implicit since we may not obtain an estimated upper bound for a special case unlike what we can do with a generalization bound based on Rademacher complexities?\n\n3. Can you discuss and compare $\\mathcal{H}$-consistency with Fisher consistency since there is a requirement for Fisher consistency in terms of a comparison inequality? (Nowak et al., 2020) (Blondel, 2019)\n\n4. Section E presents proofs for each of the four structured comp-sum losses individually but Theorem 6 is stated for general structured comp-sum losses. Does the conclusion hold or did I miss something?\n\n5. Line 270, $\\bar{\\ell}_i$ instead of $\\ell_i$.\n\n6. I am confused with the definitions about Markovian features in lines 287-294. Specifically, what's the definition of $p$? Based on the context, I can infer that $\\mathbf{\\Psi}$, $\\mathbf{\\psi}$, $\\mathbf{\\Psi}_k$, $\\mathbf{\\psi}_k$, $\\tilde{\\psi}$ all refer to a vector in $\\mathbb{R}^d$. But what is the number of padding zeros in the definition of $\\tilde{\\mathbf{\\Psi}}_k$ in line 290?\n\n7. Due to the theoretical nature of this work, experiments may not be necessary. But is it possible to compare the proposed consistent losses with other consistent losses empirically (Nowak et al., 2020) (Blondel, 2019)?\n\nLIMITATIONS:\nLimitations are mentioned together with introduction of methods. Potential negative societal impacts are not discussed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work extensively studies surrogate losses for structured predictions supported by H-consistency bounds. It first shows several negative results for some widely used surrogate losses in structured predictions: no non-trivial H-consistency bound can be derived. Then it provides two new families of surrogate losses that are supported by H-consistency bounds (which imply Bayes consistency): structured comp-sum losses and structured constrained losses. Finally, efficient algorithms are proposed for some of these new surrogate losses.\n\nSTRENGTHS:\nThis work presents a solid theoretical study in the field of structured predictions. First, the paper shows that structured max losses (which include loss functions associated with several prominent structured prediction algorithms in the literature) are not Bayes consistent, which implies they cannot be supported by H-consistency bounds either (Theorem 4). Then, it shows that voted Conditional Random Field losses (which have been presented in some works of structured predictions) are not Bayes consistent either (Theorem 5). Moving to positive results, the paper provides two new families of surrogate losses that are supported by H-consistency bounds (which imply Bayes consistency): structured comp-sum losses and structured constrained losses (Theorem 6, 8, Corollary 7, 9). Finally, it presents efficient algorithms for minimizing several of the proposed surrogate losses.\n\nThe level of originality exhibited in the research was noticeable, demonstrating a study (H-consistency bounds in structured predictions) that has not been extensively explored before. The quality of the work is good, with meticulous comparisons and contrasts with previous works (including some prominent ones) in structured predictions (Line 134-145, 157-162) and detailed analysis, supporting the authors' arguments convincingly. The paper is well-written at large (some suggestions for improvements below). The work is significant; it offers valuable insights into an under-researched topic (H-consistency bounds in structured predictions), and the implications of the findings could stimulate new directions for future research in structured predictions.\n\nWEAKNESSES:\n1. The current manuscript does not have a conclusion section. Given that the results are already impressive, the authors should have shortened Section 6 and added a conclusion section to improve the readability further.\n2. Because the authors showed the gradient of the structured logistic loss can be computed efficiently and claimed practical use, the authors should consider including some experiments for demonstration.\n\nQUESTIONS:\n1. When you say \"no non-trivial H-consistency bounds can be derived\", what are the \"trivial H-consistency bounds\"?\n2. How are the results of this work related to those in several works by Ciliberto et al., 2016, 2019, and 2020?\n\nLIMITATIONS:\nI could not find the location where the limitations of the work were explicitly discussed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors study surrogate losses for structured prediction problems. They show that surrogate losses proposed in previous work are not Bayes-consistent, i.e. a sequence of hypotheses which minimises the surrogate loss may not minimise the target loss. They then introduce two families of surrogate losses, namely structured comp-sum losses and structured constrained loss functions, which generalise two corresponding existing families to the structured prediction setting, and show that these admit $\\\\mathcal{H}$-consistency bounds, a stronger form of consistency that also implies Bayes-consistency. Last, they derive practically efficient algorithms for minimising the surrogate losses they have introduced, under certain settings.\n\nSTRENGTHS:\n__Paper highlights shortcomings of existing theory:__ The authors prove that existing surrogate loss functions, namely the structured max loss and the structured voted conditional random field loss, are not Bayes-consistent. Therefore, minimising one of these surrogates is not guaranteed to also minimise the corresponding target losses. This result is likely of interest to the wider community.\n\n__Introduction of new structured losses supported by theory:__ This work introduces two classes of surrogate losses, namely the structured comp-sum and the structured constrained loss, which they show to be $\\\\mathcal{H}$-consistent. This implies Bayes consistency, a guarantee that was missing from existing structured surrogate losses. In fact, $\\\\mathcal{H}$-consistency is a stronger guarantee than Bayes-consistency, since, as the authors point out, it is not asymptotic and accounts for the hypothesis set $\\\\mathcal{H}$ in question.\n\n__Theoretically sound and technically precise:__ Although I was not able to check the entirety of the derivations in the Appendix (I did not verify the proofs of Theorem 6 and Theorem 8 closely enough to be fully confident about these), other parts which I did check closely looked sound and technically precise to me. Also, generally, I found the paper to be carefully written and I also found that the notation was precise and clear (though I thought Section 6 could have been better organised to improve readability, and that the paper could have benefited from a conclusion section).\n\nWEAKNESSES:\n__The exposition of the paper could be improved:__ While I found that the notation and definitions in the paper are clear and precise, I found that from Section 6 onwards, the paper has tougher to follow, and the exposition was far denser and less clear than in the preceding sections. I think that the paper could benefit by organising the key results in lemmas and propositions, and deferring more some of the details from this section into the appendix.\n\nQUESTIONS:\nPlease see a list of questions and suggestions for improvement I have about the paper, organised roughly in order of appearance:\n\n- __Line 83:__ The authors say “naturally $\\\\ell$ is symmetric”. Does it have to be symmetric, or are they assuming it to be symmetric?\n\n- __Lines 95-97:__ The phrasing here could perhaps be clarified to “which guarantees that minimizing the generalization error for a surrogate loss $L_{\\\\text{sur}}$ over $\\\\mathcal{H}_{\\\\text{all}}$ also leads to the minimization of generalization error for the target loss $L$.”\n\n- __Definition 1:__ This appears to have typos. First $f_n$ appears in eq. 3 but it is not defined. Second, $h$ appears in the definition but not in eq. 3. Do the authors mean $h_n$ in the definition and $h_n$ instead of $f_n$ in eq. 3?\n\n- __Definition 2:__ How is $\\\\mathcal{H}$ defined (in line 107)? Is this a subset of the full hypothesis set? If so, the authors could clarify this by saying “Given a subset of the hypothesis class $\\\\mathcal{H} \\\\subseteq \\\\mathcal{H}_{\\\\text{all}}$, a surrogate loss…”.\n\n- __Equation below line 116:__ Should the expression on the right hand side have the conditional distribution $p(y | x)$ rather than the joint $p(x, y)$?\n\n- __Comment on notation:__ In line 117 the authors could use the notation $\\\\mathcal{C}_{L, \\\\mathcal{H}}^*(x)$ to make this consistent with the notation under eq. 79.\n\n- __Line 122:__ Typo, “hypothesis set” not “hypothesis sets”.\n\n- __Line 129:__ Typo, the “possible predictions” instead of “possible prediction”.\n\n- __Lemma 3:__ Similarly to the above comment for line 116, should the lemma involve the conditional $p(y | x)$ rather than the joint $p(x, y)$? Furthermore, the authors don’t seem to be using Lemma 3 (or referring to it) in the rest of the main text. In this case, I would suggest removing it and deferring it to the appendix.\n\n- __Theorem 4:__ I have two questions regarding Theorem 4:\n    - First, in line 650, Appendix C, you refer to $h^*$ as “the Bayes classifier” of the structured max loss. I think a more accurate statement would be that $h^*$ is “__a__ Bayes classifier”, because there exist many choices of $h$ which minimise the generalisation error. For example, consider $h^*(x, 1) = h^*(x, 2) = 1$ and $h^*(x, y) = 0$ for all other $y > 2$. This also minimises the generalisation error, and coincides with the Bayes classifier of the target loss.\n\n    - Second, the proof of this result seems to highlight a way in which the problematic classifiers (i.e. the classifiers which are optimal for the surrogate loss, but not for the target loss) are far fewer than those which are optimal for the target loss. In particular, is it correct to say that the set of classifiers which optimise the surrogate loss but not the target loss are those for which $h(x, y_1) = h(x, y_2) = \\\\dots = h(x, y_n)$, and this set is much smaller than the set of classifiers which optimise both the surrogate as well as the target loss (that is the set of classifiers for which $h(x, y_i) > h(x, y_3), \\dots, h(x, y_n)$ for either $i = 1$ or $i = 2$). Can the authors comment on this point? For example, could one learn $h$ with an algorithm that involves a randomisation step (e.g. randomised initialisation), that results in convergence to the Bayes-optimal predictor of the target loss with high probability?\n\n- __Line 663:__ In Appendix D, in the proof for Theorem 5, the authors introduce the quantity $\\\\Phi_y$. Is this simply a function from $\\\\mathcal{Y}$ to $\\\\mathbb{R}$?\n\n- __Theorem 5:__ The proof of Theorem 5 relies on an argument (line 663) where the authors consider a loss $\\ell(y’, y)$ that decouples (i.e. factorises) into a term that depends solely on $y$ only and another term which depends solely on $y’$. From this, they show that the VCRF loss is not Bayes-consistent. Some common losses, such as the zero-one $\\\\ell_{0-1}$ loss do not decompose in this way. When constrained to such losses, is the VCRF loss Bayes-consistent? How critical is the factorisation requirement in the proof?\n\nLIMITATIONS:\nIn my assessment, I do not see any substantial limitations of this work which have not been addressed by the authors. However, I would appreciate the authors’ clarification on the questions I have raised above regarding the argument in the proofs of Theorems 4 and 5.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n* The paper studies (Fisher, or “Bayes”) consistency in structured prediction.  In particular, the focus is on non-asymptotic, quantitative bounds for common and not-so-common (i.e., new) surrogate losses, which requires different proof techniques and an approach based on “H-consistency”.\n* Thm 4, Thm 5 provide a few of the main results in the paper, showing that commonly used losses in structured prediction are not Bayes-consistent\n* Thm 6, Thm 8 (and companion corollaries) provide the other main results in the paper, showing that a few new-ish but less tractable (though still convex) losses are H-consistent and Bayes-consistent — the losses in question here are the so-called “comp-sum” and “structured constrained” losses\n* Finally, lem 10 shows that the latter (comp-sum) losses may be computed in polynomial time — but revealing something of a natural statistical/computational trade-off, i.e., the consistent losses evidently require more compute than the non-consistent ones\n\nSTRENGTHS:\n* The paper gives a quite detailed study on the statistical + computational aspects of H-consistency, for commonly (and not-so-commonly) used losses in structured prediction\n* In particular, the paper highlights losses that are Bayes-consistent (though coming at the price of computability)\n\nWEAKNESSES:\nI have just a couple questions / comments:\n* It seems the content of thms 4, 5 — on the lack of consistency — is also present in previous works (e.g., those by Ciliberto et al.).  Can you please elaborate on why these results in your paper are novel?\n* Usually Fisher consistency is defined relative to a target loss function.  What is the target loss function here?\n* I think the primary avenue the paper could be (significantly) strengthened is via an empirical analysis — the authors could illustrate the consistency (and computational costs) of the comp-sum, structured constrained losses through real-world examples.  That would really tie together the whole message of the paper.\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nNo they have not, but I’m not sure that’s necessary here.  This is an abstract / general theory paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}