SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Anqi Mao
Courant Institute New York, NY 10012 aqmao@cims.nyu.edu
We present an extensive study of surrogate losses for structured prediction supported by  $\mathcal{H}$ -consistency bounds. These are recently introduced guarantees that are more relevant to learning than Bayes-consistency, since they are not asymptotic and since they take into account the hypothesis set  $\mathcal{H}$  used. We first show that no nontrivial  $\mathcal{H}$ -consistency bound can be derived for widely used surrogate structured prediction losses. We then define several new families of surrogate losses, including structured comp-sum losses and structured constrained losses, for which we prove  $\mathcal{H}$ -consistency bounds and thus Bayes-consistency. These loss functions readily lead to new structured prediction algorithms with stronger theoretical guarantees, based on their minimization. We describe efficient algorithms for minimizing several of these surrogate losses, including a new structured logistic loss.
In most applications, the output labels of learning problems have some structure that is crucial to consider. This includes natural language processing applications, where the output may be a sentence, a sequence of parts-of-speech tags, a parse tree, or a dependency graph. It also includes image annotation, image segmentation, computer vision, video annotation, object recognition, motion estimation, computational photography, bioinformatics, and many other important applications.
Several algorithms have been designed in the past for structured prediction tasks, including Conditional Random Fields (CRFs) [Lafferty et al., 2001a, Gimpel and Smith, 2010], StructSVMs [Tsochantaridis et al., 2005a], Maximum-Margin Markov Networks (M3N) [Taskar et al., 2003a], kernel-regression-based algorithms [Cortes et al., 2005, 2007], Voted CRF and StructBoost [Cortes et al., 2016], search-based methods [Daumé III et al., 2009, Doppa et al., 2014, Lam et al., 2015, Chang et al., 2015, Ross et al., 2011] and a variety of deep learning techniques [Jurafsky and Martin, 2009, Vinyals et al., 2015a, Nadeau and Sekine, 2007, Zhang et al., 2008, Wu et al., 2016, Lucchi et al., 2013, Vinyals et al., 2015b], see Appendix A for a more comprehensive list of references and discussion.
Structured prediction tasks inherently involve a natural loss function based on substructures, which could be the Hamming loss, the  $n$ -gram loss, the edit-distance loss, or some other sequence similarity-based loss or task-specific structured loss. Many of the algorithms previously mentioned overlook this inherent structured loss by simply minimizing the cross-entropy loss. In contrast, the surrogate loss functions minimized by algorithms such as CRFs [Lafferty et al., 2001a, Gimpel and Smith, 2010], M3N [Taskar et al., 2003a], StructSVMs [Tsochantaridis et al., 2005a] or Voted CRF and StructBoost [Cortes et al., 2016] do take into account the natural structured loss of the task. But are these structured prediction loss functions consistent? What guarantees can we rely on when minimizing them over a restricted hypothesis set that does not include all measurable functions? Can we derive non-asymptotic guarantees?
This paper deals precisely with these theoretical problems in structured prediction.
Previous work. We include a detailed discussion on consistency in structured prediction in Appendix A. Here, we briefly discuss previous work by Osokin et al. [2017]. To our knowledge, this is one of the only studies proving Bayes-consistency for a family of loss functions in structured prediction (see also [Nowak et al., 2020] and other related references in Appendix A). The surrogate losses the authors proposed are the following quadratic losses (see also [Zhang, 2004]) defined for any function  $h$  mapping  $\mathcal{X} \times \mathcal{Y}$  to  $\mathbb{R}$  and any loss function  $\ell$  between output labels by
$$
\forall (x, y) \in \mathcal {X} \times \mathcal {Y}, \quad \mathrm {L} ^ {\text {q u a d}} (h, x, y) = \sum_ {y ^ {\prime} \in \mathcal {Y}} \left[ \ell \left(y ^ {\prime}, y\right) + h \left(x, y ^ {\prime}\right) \right] ^ {2}. \tag {1}
$$
However, the authors only consider the hypothesis set of linear scoring functions. Moreover, the feature vector in their setting only depends on the input  $x$  and ignores the label  $y$ . In many applications such as natural language prediction, however, it is critical to allow for features that depend both on the input sequence and the output sequence, parse tree, or dependency graph. Finally, in this formulation, the structured prediction problem is cast as a regression problem. Thus, as shown below, the loss function derived is non-standard, even in the binary classification case, where  $\ell = \ell_{0-1}$  is the zero-one loss and  $\mathcal{Y} = \{y_1, y_2\}$ . In this simple case,  $\mathsf{L}^{\mathrm{quad}}(h, x, y_1)$  can be expressed as
$$
\mathsf {L} ^ {\text {q u a d}} (h, x, y _ {1}) = \sum_ {y ^ {\prime} \in \mathcal {Y}} \left[ \ell_ {0 - 1} \left(y ^ {\prime}, y _ {1}\right) + h \left(x, y ^ {\prime}\right) \right] ^ {2} = h \left(x, y _ {1}\right) ^ {2} + \left(1 + h \left(x, y _ {2}\right)\right) ^ {2}. \tag {2}
$$
This is not a typical formulation since it incorporates the magnitude of individual scores. In contrast, in standard binary classification scenario, only the difference between scores matters.
Structure of the paper. We present an extensive study of surrogate losses for structured prediction supported by  $\mathcal{H}$ -consistency bounds. These are recently introduced guarantees that are more relevant to learning than Bayes-consistency, since they are not asymptotic and since they take into account the hypothesis set  $\mathcal{H}$  used. We first show that no non-trivial  $\mathcal{H}$ -consistency bound or even Bayes-consistency can be derived for widely used surrogate structured prediction losses (Section 3). We then define several new families of surrogate losses, including structured comp-sum losses (Section 4) and structured constrained losses (Section 5), for which we prove  $\mathcal{H}$ -consistency bounds and thus Bayes-consistency. These loss functions readily lead to new structured prediction algorithms with stronger theoretical guarantees, based on their minimization. We also describe efficient gradient computation algorithms for several of these surrogate losses, including a new structured logistic loss (Section 6).
Learning scenario. We consider the standard structured prediction scenario with the input space  $\mathcal{X}$  and output space  $\mathcal{Y} = \{1, \dots, n\}$ . The output space may be discrete objects with overlapping structures, such as sequences, images, graphs, parse trees, lists, or others. We assume that the output can be decomposed into  $l$  substructures. The substructures could represent words or tokens for example, or other subsequences along a sequence, resulting in the decomposition of the output space  $\mathcal{Y}$  as  $\mathcal{Y} = \mathcal{Y}_1 \times \dots \times \mathcal{Y}_l$ . Here, each  $\mathcal{Y}_j$  represents the set of possible labels or classes that can be assigned to the  $j$ -th substructure.
Scoring function. Structured prediction is typically formulated via scoring functions that map  $\mathcal{X} \times \mathcal{Y}$  to  $\mathbb{R}$ , which assign a score to each possible class  $y \in \mathcal{Y}$ . Let  $\mathcal{H}$  be a family of such scoring functions. For any  $h \in \mathcal{H}$ , we denote by  $\mathsf{h}(x)$  its prediction for the input  $x \in \mathcal{X}$ , which is the output  $y \in \mathcal{Y}$  that maximizes the score  $h(x, y)$ , that is,  $\mathsf{h}(x) = \operatorname*{argmax}_{y \in \mathcal{Y}} h(x, y)$ , with a fixed deterministic strategy to break ties in selecting the label with the highest score. For simplicity, we choose the label with the highest index under the natural ordering of labels as the tie-breaking strategy. We denote by  $\mathcal{H}_{\mathrm{all}}$  the family of all measurable scoring functions.
Generalization error and target loss. Given a distribution  $\mathcal{D}$  over  $\mathcal{X} \times \mathcal{Y}$  and a loss function  $L: \mathcal{H} \times \mathcal{X} \times \mathcal{Y} \to \mathbb{R}$ , the generalization error of a hypothesis  $h \in \mathcal{H}$  and the best-in-class generalization error are defined as follows:
$$
\mathcal {R} _ {\mathrm {L}} (h) = \underset {(x, y) \sim \mathcal {D}} {\mathbb {E}} [ \mathrm {L} (h, x, y) ] \quad \text {a n d} \quad \mathcal {R} _ {\mathrm {L}, \mathcal {H}} ^ {*} = \inf  _ {h \in \mathcal {H}} \mathcal {R} _ {\mathrm {L}} (h).
$$
In structured prediction, the goal is to select a hypothesis  $h \in \mathcal{H}$  with small generalization error with respect to a target loss function  $\mathsf{L}(h,x,y)$ , which can be written as  $\mathsf{L}(h,x,y) = \ell(\mathsf{h}(x),y)$  for some non-negative auxiliary function  $\ell(y',y)$  with any  $y',y \in \mathcal{Y}$ .  $\ell$  is assumed to be symmetric, that is,  $\ell(y',y) = \ell(y,y')$ . This is a natural assumption since all instances of  $\ell$  that we are familiar with in structured prediction admit this property. A significant characteristic of structured prediction is that the target loss function can be decomposed along the substructures  $\mathcal{Y}_k$ . As an example, we may use the Hamming loss as the target loss function, which is defined as  $\mathsf{L}_{\mathrm{ham}}(h,x,y) = \ell_{\mathrm{ham}}(\mathsf{h}(x),y)$ , where  $\ell_{\mathrm{ham}}(y',y) = \frac{1}{l}\sum_{j=1}^{l}\mathbb{1}_{y_j' \neq y_j}$  with any  $y_j',y_j \in \mathcal{Y}_j$ ; we can also use the zero-one loss as the target loss function, which is defined as  $\mathsf{L}_{0-1}(h,x,y) = \ell_{0-1}(\mathsf{h}(x),y)$ , where  $\ell_{0-1}(y',y) = \mathbb{1}_{y' \neq y}$ . Note that  $\mathsf{L}_{0-1}$  can be viewed as a special case of  $\mathsf{L}_{\mathrm{ham}}$  when  $l = 1$ . We denote by  $\ell_{\max} = \max_{y',y \in \mathcal{Y}}\ell(y',y)$  the maximal value of a target loss function. Without loss of generality, we assume that  $\ell_{\max} \leq 1$ , which can be achieved by normalizing the function  $\ell$ .
Consistency guarantees and surrogate loss. Optimizing the target loss functions in structured prediction for many choices of the hypothesis sets is NP-hard because they are not convex. One common method to address this issue is to resort to surrogate loss functions. Different surrogate loss functions readily lead to different structured prediction algorithms. A natural learning guarantee for such surrogate losses is Bayes-consistency, which guarantees that minimizing the generalization error for a surrogate loss  $\mathsf{L}_{\mathrm{sur}}$  over  $\mathcal{H}_{\mathrm{all}}$  also leads to the minimization of generalization error for the target loss  $\mathsf{L}$ .
Definition 1 (Bayes-consistency). A surrogate loss  $\mathsf{L}_{\mathrm{sur}}$  is Bayes-consistent in structured prediction, if for any target loss  $\ell$ , hypothesis  $h_n \in \mathcal{H}_{\mathrm{all}}$  and any distribution,
$$
\left(\mathcal {R} _ {\mathrm {L} _ {\text {s u r}}} \left(h _ {n}\right) - \mathcal {R} _ {\mathrm {L} _ {\text {s u r}}, \mathcal {H} _ {\text {a l l}}} ^ {*} \xrightarrow {n \rightarrow + \infty} 0\right) \Longrightarrow \left(\mathcal {R} _ {\mathrm {L}} \left(h _ {n}\right) - \mathcal {R} _ {\mathrm {L}, \mathcal {H} _ {\text {a l l}}} ^ {*} \xrightarrow {n \rightarrow + \infty} 0\right). \tag {3}
$$
Bayes-consistency is an asymptotic guarantee and does not take into account typical hypothesis sets used in structured prediction algorithms, such as linear models or neural networks. To tackle these issues, recent work by Awasthi, Mao, Mohri, and Zhong [2022a,b] propose a stronger consistency guarantee, referred to as  $\mathcal{H}$ -consistency bounds, which are bounds relating the estimation error of the target loss to the estimation error of a surrogate loss (see also [Awasthi et al., 2021a,b, Mao et al., 2023h,e,f, Zheng et al., 2023, Mao et al., 2023b,g,d, Mohri et al., 2023, Mao et al., 2023c,a, Awasthi et al., 2023a,b]):
Definition 2 (H-consistency bounds). Given a subset of the hypothesis class  $\mathcal{H} \subseteq \mathcal{H}_{\mathrm{all}}$ , a surrogate loss  $\mathsf{L}_{\mathrm{sur}}$  admits a H-consistency bound in structured prediction, if for some non-decreasing function  $f: \mathbb{R}_+ \to \mathbb{R}_+$ , a bound of the following form holds for any target loss  $\ell$ , hypothesis  $h \in \mathcal{H}$  and any distribution:
$$
\mathcal {R} _ {\mathrm {L}} (h) - \mathcal {R} _ {\mathrm {L}, \mathcal {H}} ^ {*} \leq f \left(\mathcal {R} _ {\mathrm {L} _ {\text {s u r}}} (h) - \mathcal {R} _ {\mathrm {L} _ {\text {s u r}}, \mathcal {H}} ^ {*}\right). \tag {4}
$$
As pointed out by Awasthi et al. [2022a,b],  $\mathcal{H}$ -consistency bounds are the state-of-the-art consistency guarantees for surrogate losses. They are much stronger and more informative than Bayes-consistency, since they account for hypothesis sets  $\mathcal{H}$  adopted and provide a quantitative, non-asymptotic relation between surrogate losses and target losses.  $\mathcal{H}$ -consistency bounds can imply Bayes-consistency when taking  $\mathcal{H}$  to be  $\mathcal{H}_{\mathrm{all}}$ . In the next sections, we will present an extensive study of surrogate losses for structured prediction supported by  $\mathcal{H}$ -consistency bounds.
Conditional regret and minimizability gap. We denote by  $p(x) = (p(x,1),\ldots ,p(x,n))$  the conditional distribution of  $Y$  given  $X = x$ . Then, the conditional error of a hypothesis  $h$  for a loss function  $\mathsf{L}$ , denoted by  $\mathcal{C}_{\mathsf{L}}(h,x)$ , can be expressed as
$$
\mathcal{C}_{L}(h,x) = \underset {y|x}{\mathbb{E}}\left[\ell (\mathsf{h}(x),y)\right] = \sum_{y\in \mathcal{Y}}p(x,y)\ell (\mathsf{h}(x),y).
$$
We further define the best-in-class conditional error and the conditional regret as  $\mathcal{C}_{\mathsf{L}}^{*}(\mathcal{H},x) = \inf_{h\in \mathcal{H}}\mathcal{C}_{\mathsf{L}}(h,x)$  and  $\Delta \mathcal{C}_{\mathsf{L},\mathcal{H}}(h,x) = \mathcal{C}_{\mathsf{L}}(h,x) - \mathcal{C}_{\mathsf{L}}^{*}(\mathcal{H},x)$  respectively. The generalization error can then be rewritten as  $\mathcal{R}_{\mathsf{L}}(h) = \mathbb{E}_x[\mathcal{C}_{\mathsf{L}}(h,x)]$
A key quantity appearing in  $\mathcal{H}$ -consistency bounds is the minimizability gap  $\mathcal{M}_{\mathrm{L}}(\mathcal{H})$ , which measures the difference between the best-in-class generalization error and the expected best-in-class conditional error for a loss function  $\mathsf{L}$  and a hypothesis set  $\mathcal{H}$ :  $\mathcal{M}_{\mathrm{L}}(\mathcal{H}) = \mathcal{R}_{\mathrm{L}}^{*}(\mathcal{H}) - \mathbb{E}_x[\mathcal{C}_{\mathrm{L}}^{*}(\mathcal{H}, x)]$ . This is an inherent quantity that we cannot hope to minimize or estimate. As shown by Steinwart [2007,
Theorem 3.2], the minimizability gaps vanish  $\mathcal{M}_{\mathrm{L}}(\mathcal{H}_{\mathrm{all}}) = 0$  for the family of all measurable functions. More generally, the minimizability gaps vanish when the best-in-class error coincides with the Bayes-error, that is,  $\mathcal{R}_\ell^* (\mathcal{H}) = \mathcal{R}_\ell^* (\mathcal{H}_{\mathrm{all}})$  [Awasthi et al., 2022b, Mao et al., 2023h].
The following result characterizes the best-in-class conditional error and the conditional regret for a target loss  $\mathsf{L}$ , which will be helpful for proving  $\mathcal{H}$ -consistency bounds in structured prediction. We denote by  $\mathsf{H}(x)$  the set of all possible predictions on a input  $x$  generated by hypotheses in  $\mathcal{H}$ :  $\mathsf{H}(x) = \{\mathsf{h}(x) : h \in \mathcal{H}\}$ . The proof is given in Appendix B.
Lemma 3. The best-in-class conditional error and the conditional regret for a target loss  $\mathsf{L}$  in structured prediction can be expressed as follows:
$$
\mathcal{C}^{*}_{\mathsf{L},\mathcal{H}}(x) = \min_{y^{\prime}\in \mathsf{H}(x)}\sum_{y\in \mathcal{Y}}p(x,y)\ell (y^{\prime},y)
$$
$$
\Delta \mathcal {C} _ {\mathrm {L}, \mathcal {H}} (h, x) = \sum_ {y \in \mathcal {Y}} p (x, y) \ell (\mathsf {h} (x), y) - \min  _ {y ^ {\prime} \in \mathsf {H} (x)} \sum_ {y \in \mathcal {Y}} p (x, y) \ell (y ^ {\prime}, y).
$$
3 Structured max losses
In this section, we examine the loss functions associated to several prominent structured prediction algorithms. We show that, while they are natural, none of them is Bayes-consistent, which implies that they cannot be supported by  $\mathcal{H}$ -consistency bounds either. More generally, we consider the following family of surrogate loss functions proposed in [Cortes, Kuznetsov, Mohri, and Yang, 2016], which we refer to as structured max losses:
$$
\forall (x, y) \in \mathcal {X} \times \mathcal {Y}, \quad \mathrm {L} ^ {\max } (h, x, y) = \max  _ {y ^ {\prime} \neq y} \Phi_ {\ell \left(y ^ {\prime}, y\right)} \left(h (x, y) - h \left(x, y ^ {\prime}\right)\right), \tag {5}
$$
where  $\Phi_u\colon \mathbb{R}\to \mathbb{R}_+$  is an upper bound on  $v\mapsto u1_{v\leq 0}$  for any  $u\in \mathbb{R}_{+}$ . In this formulation, different choices of  $\Phi_u$  can lead to different structured prediction algorithms. Specifically, as shown by Cortes et al. [2016], the following choices of  $\Phi_u(v)$  recover many well-known algorithms:
-  $\Phi_u(v) = \max(0, u(1 - v))$ : StructSVM [Tsochantaridis et al., 2005b].
-  $\Phi_u(v) = \max(0, u - v)$ : Max-Margin Markov Networks (M3N) [Taskar et al., 2003b].
-  $\Phi_u(v) = \log (1 + e^{u - v})$ : Conditional Random Field (CRF) [Lafferty et al., 2001b].
-  $\Phi_u(v) = ue^{-v}$ : StructBoost [Cortes et al., 2016].
The following gives a general negative result for  $\mathsf{L}^{\mathrm{max}}$  that holds under broad assumptions.
Theorem 4 (Negative results of  $\mathsf{L}^{\mathrm{max}}$ ). Assume that  $n > 2$  and that  $\Phi_u(v)$  is convex and nonincreasing for  $u = 1$ . Then, the max structured loss  $\mathsf{L}^{\mathrm{max}}$  is not Bayes-consistent.
The proof is included in Appendix C. It is straightforward to see that the assumption of Theorem 4 holds for all the choices of  $\Phi_u$  listed above. Thus, the theorem rules out consistency guarantees for any of the loss functions associated to the structured prediction algorithms mentioned above: StructSVM, M3N, CRF, Structboost. Furthermore, Theorem 4 provides negative results for a broad and generalized family of loss functions, collectively referred to as structured max loss. This extends the scope of existing research, as previous works had only addressed the inconsistency of specific instances within the structured max loss category, such as that of M3N [Osokin et al., 2017, Ciliberto et al., 2016, Nowak et al., 2020].
4 Structured comp-sum losses
In this section, we first analyze the Voted CRF loss function, which incorporates the auxiliary loss function  $\ell$  in the CRF loss and which has been used in several previous studies. Next, we introduce a new family of loss functions for structured predictions that we prove to admit strong consistency guarantees.
4.1 Voted Conditional Random Field (VCRF)
We first study a family of surrogate losses called Voted Conditional Random Field (VCRF), which corresponds to the structured prediction algorithm defined in [Cortes et al., 2016]:
$$
\forall (x, y) \in \mathcal {X} \times \mathcal {Y}, \mathrm {L} ^ {\mathrm {V C R F}} (h, x, y) = - \log \left[ \frac {e ^ {h (x , y)}}{\sum_ {y ^ {\prime} \in \mathcal {Y}} e ^ {h (x , y ^ {\prime}) + \ell (y , y ^ {\prime})}} \right] = \log \left[ \sum_ {y ^ {\prime} \in \mathcal {Y}} e ^ {\ell (y, y ^ {\prime}) + h (x, y ^ {\prime}) - h (x, y)} \right].
$$
This loss function has also been presented as the softmax margin [Gimpel and Smith, 2010] or the reward-augmented maximum likelihood [Norouzi et al., 2016]. It can be viewed as the softmax variant of the M3N loss. Indeed, the loss function for M3N can be written as follows:
$$
\mathsf {L} (h, x, y) = \max  _ {y ^ {\prime}} \max  (0, \ell \left(y ^ {\prime}, y\right) + h (x, y ^ {\prime}) - h (x, y)). \tag {6}
$$
If we replace the maximum function with the softmax, we obtain
$$
\mathrm {L} (h, x, y) = \log \left[ \sum_ {y ^ {\prime} \in \mathcal {Y}} e ^ {\max  \left(0, \ell \left(y ^ {\prime}, y\right) + h \left(x, y ^ {\prime}\right) - h (x, y)\right)} \right] = \log \left[ \sum_ {y ^ {\prime} \in \mathcal {Y}} \max  \left(1, e ^ {\ell \left(y ^ {\prime}, y\right) + h \left(x, y ^ {\prime}\right) - h (x, y)}\right) \right]. \tag {7}
$$
Next, we show that, as with the loss function for M3N, the VCRF loss function  $\mathsf{L}^{\mathrm{VCRF}}$  is inconsistent.
Theorem 5 (Negative result of  $\mathsf{L}^{\mathrm{VCRF}}$ ). The Voted Conditional Random Field  $\mathsf{L}^{\mathrm{VCRF}}$  is not Bayes-consistent.
The proof is included in Appendix D. The key observation in the proof is that the conditional error of VCRF loss function can be reduced to a specific form when the target loss function  $L$  decouples, which can lead to a different Bayes classifier from that of the target loss function.
To the best of our knowledge, no prior studies in the literature have explored the consistency of the VCRF loss formulation. The most closely related discussions center around a specialized instance of the multi-class logistic loss (also referred to as Conditional Random Field in that context), in which  $\ell(y', y)$  disappears within the framework of the Voted Conditional Random Field. The previous works by Osokin et al. [2017], Ciliberto et al. [2016], Nowak et al. [2020] point out that the multi-class logistic loss cannot be consistent in structured prediction due to the absence of the target loss function within its formulation. Instead, our result shows that, even when integrating the target loss  $\ell(y', y)$  within its formulation, the Voted Conditional Random Field cannot be consistent.
Along with Theorem 4, these results rule out consistency guarantees for commonly used surrogate loss functions in structured prediction.
4.2 Structured comp-sum loss functions
In this section, we define a family of new loss functions for structured prediction that are not only Bayes-consistent but also supported by  $\mathcal{H}$ -consistency bounds. These are loss functions that can be viewed as the generalization to structured prediction of loss functions defined via a composition and a sum, and that have been referred to as comp-sum losses in [Mao et al., 2023h]. Thus, we will refer to them as structured comp-sum losses. They are defined as follows:
$$
\forall (x, y) \in \mathcal {X} \times \mathcal {Y}, \quad \mathrm {L} ^ {\operatorname {c o m p}} (h, x, y) = \sum_ {y ^ {\prime} \in \mathcal {Y}} \bar {\ell} \left(y ^ {\prime}, y\right) \Phi_ {1} \left(\sum_ {y ^ {\prime \prime} \in \mathcal {Y}} \Phi_ {2} \left(h \left(x, y ^ {\prime \prime}\right) - h \left(x, y ^ {\prime}\right)\right)\right), \tag {8}
$$
where  $\overline{\ell}(y', y) = 1 - \ell(y', y)$ ,  $\Phi_1 \colon \mathbb{R}_+ \to \mathbb{R}_+$  is a non-decreasing auxiliary function and  $\Phi_2 \colon \mathbb{R} \to \mathbb{R}_+$  a non-decreasing auxiliary function. This formulation (8) can also be viewed as a weighted comp-sum loss, if we interpret  $\overline{\ell}(\cdot, y)$  as a weight vector.
Specifically, we can choose  $\Phi_2(v) = e^v$  and  $\Phi_1(v) = \log(v)$ ,  $\Phi_1(v) = v - 1$ ,  $\Phi_1(v) = \frac{1}{\alpha} \left(1 - \frac{1}{v^\alpha}\right)$ ,  $\alpha \in (0,1)$  and  $\Phi_1(v) = 1 - \frac{1}{v}$ , which leads to new surrogate losses for structured prediction defined in Table 1. These surrogate losses are novel strict generalization of their counterparts in the standard multi-class classification case where  $\ell = \ell_{0-1}$ . More precisely, when  $\ell = \ell_{0-1}$ ,  $L_{\mathrm{log}}^{\mathrm{comp}}$  coincides with the logistic loss [Verhulst, 1838, 1845, Berkson, 1944, 1951];  $L_{\mathrm{exp}}^{\mathrm{comp}}$  coincides with the sum-exponential loss [Weston and Watkins, 1998, Awasthi et al., 2022b];  $L_{\mathrm{gce}}^{\mathrm{comp}}$  coincides with the