SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 1: Illustration of the WFA  $\overline{\mathcal{Y}}$  for  $\Delta = \{a,b\}$  and  $l = 3$  and the WFA  $\mathcal{Y}_i$  where  $y_{i} = aba$
[2015][Theorem 3],  $\mathcal{M}$  can be computed in polynomial time.  $\mathcal{M}$
admits a unique path labeled with any sequence  $y \in \Delta^l$  and the weight of that path is  $\mathcal{U}(y, y_i)$ . The weight of that accepting path is obtained by multiplying the weights of its transitions and that of the final state.
We now define a deterministic  $p$ -gram WFA  $\mathcal{N}$  that accepts all sequences  $y \in \Delta^l$  with each of its states  $(\mathbf{z}', s)$  encoding a  $(p - 1)$ -gram  $\mathbf{z}'$  read to reach it and the position  $s$  in the sequence  $y$  at which it is reached. The transitions of  $\mathcal{N}$  are therefore defined as follows with weight one:
$$
\mathsf {E} _ {\mathcal {N}} = \left\{\left(\left(y _ {s - p + 1} ^ {s - 1}, s - 1\right), a, 1, \left(y _ {s - p + 2} ^ {s - 1} a, s\right)\right): y \in \Delta^ {l}, a \in \Delta , s \in [ l ] \right\}.
$$
The initial state is  $(\epsilon, 0)$  and the final states are those with the second element of the pair (the position) being  $l$ . Note that, by construction,  $\mathcal{N}$  is deterministic. Then, the composition (or intersection) WFA  $\mathcal{N} \circ \mathcal{M}$  still associates the same weight as  $\mathcal{M}$  to each input string  $y \in \Delta^l$ . However, the states in that composition help us compute  $\mathsf{L}(\mathbf{z}, s)$ . In particular, for any  $\mathbf{z} \in \Delta^p$  and  $s \in [l]$ , let  $\mathsf{E}(\mathbf{z}, s)$  be the set of transitions of  $\mathcal{N} \circ \mathcal{M}$  constructed by pairing the transition
$\left((\mathbf{z}_1^{p - 1},s - 1),z_p,\omega (\mathbf{z},s),(\mathbf{z}_2^p,s)\right)$  in  $\mathcal{N}$  with a transition  $(q_{\mathcal{M}},z_p,\omega ,q_{\mathcal{M}}^{\prime})$  in  $\mathcal{M}$ . They admit the following form:
Figure 2: Illustration of the WFA  $\mathcal{N}$  for  $\Delta = \{a,b\}$ ,  $p = 2$  and  $l = 2$ .
$$
\mathsf {E} (\mathbf {z}, s) = \left\{\left(\left(q _ {\mathcal {N}}, q _ {\mathcal {M}}\right), z _ {p}, \omega , \left(q _ {\mathcal {N}} ^ {\prime}, q _ {\mathcal {M}} ^ {\prime}\right)\right) \in \mathsf {E} _ {\mathcal {N} \circ \mathcal {M}}: q _ {\mathcal {N}} = \left(\mathbf {z} _ {1} ^ {p - 1}, s - 1\right) \right\}. \tag {14}
$$
The WFA  $\mathcal{N} \circ \mathcal{M}$  is deterministic as a composition of two deterministic WFAs. Thus, there is a unique path labeled with a sequence  $y \in \Delta^l$  in  $\mathcal{N} \circ \mathcal{M}$  and  $y$  admits the substring  $\mathbf{z}$  ending at position  $s$  iff that path goes through a transition in  $\mathsf{E}(\mathbf{z}, s)$  when reaching position  $s$ . Therefore, to compute  $\mathsf{L}(\mathbf{z}, s)$ , it suffices for us to compute the sum of the weights of all paths in  $\mathcal{N} \circ \mathcal{M}$  going through a transition in  $\mathsf{E}(\mathbf{z}, s)$ . This can be done straightforwardly using the forward-backward algorithm or two single-source shortest-distance algorithm over the  $(+, \times)$  semiring [Mohri, 2002a], one from the initial state, the other one from the final states. Since  $\mathcal{N} \circ \mathcal{M}$  is acyclic and admits  $O(l|\Delta|^p)$  transitions, we can compute all the quantities  $\mathsf{L}(\mathbf{z}, s)$ ,  $s \in [l]$  and  $\mathbf{z} \in \Delta^p$ , in time  $O(l|\Delta|^p)$ .
Our detailed study revealed shortcomings in commonly used surrogate loss functions and algorithms for structured prediction, prompting the introduction of new, strongly consistent alternatives. These findings not only enhance the theoretical and algorithmic foundations of structured prediction but also pave the way for the development of practical and effective solutions. In upcoming work, we will report an extensive empirical analysis of our algorithms. Our work provides tools and insights for future algorithm design in this domain, promising advancements in both theory and application.
I. Tsochantaridis, T. Joachims, T. Hofmann, and Y. Altun. Large margin methods for structured and interdependent output variables. JMLR, 6:1453-1484, Dec. 2005a.
I. Tsochantaridis, T. Joachims, T. Hofmann, Y. Altun, and Y. Singer. Large margin methods for structured and interdependent output variables. Journal of machine learning research, 6(9), 2005b.
L. Tu and K. Gimpel. Learning approximate inference networks for structured prediction. arXiv preprint arXiv:1803.03376, 2018.
L. Tu and K. Gimpel. Benchmarking approximate inference methods for neural structured prediction. arXiv preprint arXiv:1904.01138, 2019.
L. Tu, R. Y. Pang, and K. Gimpel. Improving joint training of inference networks and structured prediction energy networks. arXiv preprint arXiv:1911.02891, 2019.
P. F. Verhulst. Notice sur la loi que la population suit dans son accroissement. Correspondance mathématique et physique, 10:113-121, 1838.
P. F. Verhulst. Recherches mathématiques sur la loi d'accroissement de la population. Nouveaux Mémoires de l'Académie Royale des Sciences et Belles-Lettres de Bruxelles, 18:1-42, 1845.
O. Vinyals, L. Kaiser, T. Koo, S. Petrov, I. Sutskever, and G. Hinton. Grammar as a foreign language. In Proceedings of NIPS, 2015a.
O. Vinyals, A. Toshev, S. Bengio, and D. Erhan. Show and tell: A neural image caption generator. In Proceedings of CVPR, 2015b.
S. Wang, S. Fidler, and R. Urtasun. Proximal deep structured models. Advances in Neural Information Processing Systems, 2016.
J. Weston and C. Watkins. Multi-class support vector machines. Technical report, Citeseer, 1998.
Y. Wu, M. Schuster, Z. Chen, Q. V. Le, M. Norouzi, W. Macherey, M. Krikun, Y. Cao, Q. Gao, K. Macherey, J. Klingner, A. Shah, M. Johnson, X. Liu, Łukasz Kaiser, S. Gouws, Y. Kato, T. Kudo, H. Kazawa, K. Stevens, G. Kurian, N. Patil, W. Wang, C. Young, J. Smith, J. Riesa, A. Rudnick, O. Vinyals, G. Corrado, M. Hughes, and J. Dean. Google's neural machine translation system: Bridging the gap between human and machine translation. CoRR, abs/1609.08144, 2016. URL http://arxiv.org/abs/1609.08144.
D. Zhang, L. Sun, and W. Li. A structured prediction approach for statistical machine translation. In Proceedings of IJCNLP, 2008.
T. Zhang. Statistical analysis of some multi-category large margin classification methods. Journal of Machine Learning Research, 5(Oct):1225-1251, 2004.
Z. Zhang and M. Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In Advances in neural information processing systems, 2018.
C. Zheng, G. Wu, F. Bao, Y. Cao, C. Li, and J. Zhu. Revisiting discriminative vs. generative classifiers: Theory and implications. arXiv preprint arXiv:2302.02334, 2023.
K. Zheng and A. Pronobis. From pixels to buildings: End-to-end probabilistic deep networks for large-scale semantic mapping. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3511-3518, 2019.
S. Zheng, S. Jayasumana, B. Romero-Paredes, V. Vineet, Z. Su, D. Du, C. Huang, and P. H. Torr. Conditional random fields as recurrent neural networks. In Proceedings of the IEEE international conference on computer vision, pages 1529-1537, 2015.
A Related work 18
B Proof of Lemma 3 19
C Proofs for structured max losses 19
D Proofs for Voted Conditional Random Field 20
E Proofs for structured comp-sum losses 21
E.1 Structured logistic loss 21
E.2 Structured sum-exponential loss 22
E.3 Structured generalized cross-entropy loss 23
E.4 Structured mean absolute error loss 25
F Proofs for structured constrained losses 26
F.1 Structured constrained exponential loss 26
F.2 Structured constrained squared-hinge loss 27
F.3 Structured constrained hinge loss 29
F.4 Structured constrained  $\rho$ -margin loss 30
G Efficient gradient computation and inference 31
G.1 Efficient gradient computation for  $\mathsf{L}_{\mathrm{log}}^{\mathrm{comp}}$  31
G.2 Efficient gradient computation for  $\mathsf{L}_{\mathrm{exp}}^{\mathrm{comp}}$  32
G.3 Efficient Inference 35
Structured prediction and neural networks. A variety of deep learning techniques have been used for structured prediction tasks, including a unified neural network architecture for natural language processing [Collobert et al., 2011], energy model-based structured prediction including structured prediction energy networks (SPENs) and various inference methods [Belanger and McCallum, 2016, Tu and Gimpel, 2018, Larsson et al., 2018, Tu and Gimpel, 2019, Tu et al., 2019, Graber and Schwing, 2019, Pan et al., 2020], attention networks incorporating richer structural distributions [Kim et al., 2017], tree-structured long short-term memory (LSTM) networks [Tai et al., 2015], sequence to sequence learning [Sutskever et al., 2014, Edunov et al., 2017], memory-reduced variant of best-first beam search [Meister et al., 2020], end-to-end learning for SPENs [Belanger et al., 2017], conditional generative flow [Lu and Huang, 2020], proximal methods [Wang et al., 2016], CRF using deep features [Jaderberg et al., 2014, Huang et al., 2015, Chen et al., 2014, Schwing and Urtasun, 2015, Chen et al., 2015], non-iterative feed-forward predictors [Stoyanov et al., 2011, Domke, 2013, Kunisch and Pock, 2013, Hershey et al., 2014, Li and Zemel, 2014, Belharbi et al., 2018, Zheng et al., 2015], deep value network [Gygli et al., 2017], fully convolutional networks [Long et al., 2015], constraint reasoning tool [Dragone et al., 2021, Jiang et al., 2022], multi-label box model[Patel et al., 2022], probabilistic deep networks[Zheng and Pronobis, 2019, Jang et al., 2023], autoregressive methods [Liu et al., 2022], nonlinear output transformations and embedding [Graber et al., 2018, Brogat-Motte et al., 2020], smoothing methods [Pillutla et al., 2018] and structural training [Choi et al., 2016, Ahmad et al., 2023]. Let us also mention ensemble algorithms for structured prediction algorithms [Cortes, Kuznetsov, and Mohri, 2014a,b], which can be used to combine several algorithms for this problem.
Consistency in structured prediction. Here, we discuss in detail previous work on consistency in structured prediction [Osokin et al., 2017, Ciliberto et al., 2016, Blondel, 2019, Nowak et al., 2020, 2022, Ciliberto et al., 2019, 2020, Nowak-Vila et al., 2019, Nowak et al., 2019, Cabannes et al., 2021, Cabannnes et al., 2020, Corro, 2023].
Osokin et al. [2017], Nowak et al. [2020] and Nowak et al. [2022] pointed out that the max-margin Markov networks (M3N), or more generally structural SVMs may not be Bayes-consistent. Instead, Osokin et al. [2017] proposed the first Bayes-consistent surrogate loss in the structured prediction setting, the quadratic surrogate (QS) loss. A general theory of QS was further developed in [Nowak-Vila et al., 2019, Nowak et al., 2019]. However, as pointed out in Section 1, the quadratic surrogate loss formulation casts the structured prediction problem as a regression problem and is not a typical formulation even in the binary classification case.
Nowak et al. [2020] proposed a consistent method called max-min margin Markov networks (M4N) derived from first principles for binary SVM. However, this method is restricted to SVM-type loss functions. Instead, we propose broad families of surrogate losses, which can be naturally derived from common multi-class losses including the logistic loss.
Nowak et al. [2022] addressed the inconsistency of Max-Margin loss in structured prediction by introducing the notion of Restricted-Max-Margin, where the maximization is performed over a subset of the original domain. Their method is based on an implicit embedding [Finocchiaro et al., 2019]; a general framework for structured prediction has been further developed by Ciliberto et al. [2020]. However, these methods are only applied to polyhedral-type surrogates which are not as smooth as the logistic loss. Thus, the resulting surrogate losses may not be favorable from the optimization point of view. Instead, our novel families of surrogate losses are very general and can be smooth, including a new structured logistic loss, for which we describe efficient gradient computation algorithms.
Ciliberto et al. [2016] focused on a least squares surrogate loss function and corresponding framework. In this framework, the structured prediction problem is cast as a regression problem. They derived a regularization approach to structured prediction from the least squares surrogate loss and proved the Bayes-consistency of that approach. Ciliberto et al. [2019] focused on a local structure-adapted framework for structured prediction. They proposed a novel structured prediction algorithm that adaptively leverages locality in the learning problem. Ciliberto et al. [2020] developed a general framework for structured prediction based on implicit embedding. Their methods lead to polyhedral-type surrogates losses that benefit from Bayes-consistency.
On the other hand, our work presents an extensive study of surrogate losses for structured prediction supported by  $\mathcal{H}$ -consistency bounds. Different from the surrogate loss studied in the previous work,
the formulations of our proposed surrogate losses including structured comp-sum losses and structured constrained losses are completely novel and do not cast structured prediction problems as a regression problem. Furthermore, we prove stronger consistency guarantees that imply Bayes-consistency for these new proposed families of surrogate loss.
Other related work on structured prediction includes: projection-based losses for structured prediction [Blondel, 2019]; fast convergence rates for general structured prediction problems [Cabannes et al., 2021]; a unified framework for dealing with partial labelling [Cabannnes et al., 2020]; and an analysis of the inconsistency of separable negative log-likelihood losses for structured prediction [Corro, 2023].
B Proof of Lemma 3
Lemma 3. The best-in-class conditional error and the conditional regret for a target loss  $L$  in structured prediction can be expressed as follows:
$$
\mathcal {C} _ {\mathrm {L}, \mathcal {H}} ^ {*} (x) = \min  _ {y ^ {\prime} \in \mathrm {H} (x)} \sum_ {y \in \mathcal {Y}} p (x, y) \ell \left(y ^ {\prime}, y\right)
$$
$$
\Delta \mathcal {C} _ {\mathrm {L}, \mathcal {H}} (h, x) = \sum_ {y \in \mathcal {Y}} p (x, y) \ell (\mathsf {h} (x), y) - \min  _ {y ^ {\prime} \in \mathcal {H} (x)} \sum_ {y \in \mathcal {Y}} p (x, y) \ell (y ^ {\prime}, y).
$$
Proof. By the definition, the conditional L-risk can be expressed as follows:
$$
\mathcal {C} _ {\mathrm {L}} (h, x) = \sum_ {y \in \mathcal {Y}} p (x, y) \ell (\mathrm {h} (x), y). \tag {15}
$$
Since  $\{\mathsf{h}(x):h\in \mathcal{H}\} = \mathsf{H}(x)$ , the best-in-class conditional error can be expressed as follows:
$$
\mathcal {C} _ {\mathrm {L}, \mathcal {H}} ^ {*} (x) = \min  _ {y ^ {\prime} \in \mathsf {H} (x)} \sum_ {y \in \mathcal {Y}} p (x, y) \ell (y ^ {\prime}, y),
$$
which proves the first part of the lemma. By the definition,
$$
\Delta \mathcal {C} _ {\mathrm {L}, \mathcal {H}} (h, x) = \mathcal {C} _ {\mathrm {L}} (h, x) - \mathcal {C} _ {\mathrm {L}, \mathcal {H}} ^ {*} (x) = \sum_ {y \in \mathcal {Y}} p (x, y) \ell (\mathsf {h} (x), y) - \min  _ {y ^ {\prime} \in \mathsf {H} (x)} \sum_ {y \in \mathcal {Y}} p (x, y) \ell (y ^ {\prime}, y).
$$
C Proofs for structured max losses
Proof. For the structured max loss  $\mathsf{L}^{\max}$ , the conditional  $\mathsf{L}^{\max}$ -risk can be expressed as follows:
$$
\mathcal {C} _ {\mathrm {L} ^ {\max }} (h, x) = \sum_ {y \in \mathcal {Y}} p (x, y) \max  _ {y ^ {\prime} \neq y} \Phi_ {\ell \left(y ^ {\prime}, y\right)} \left(h (x, y) - h (x, y ^ {\prime})\right).
$$
Take  $\ell(y', y) = \mathbb{1}_{y \neq y'}$  to be the zero-one loss. Since  $\ell(y', y) = 1$  for any  $y \neq y'$ , the conditional  $\mathsf{L}^{\max}$ -risk can be reformulated as follows:
$$
\mathcal {C} _ {\mathrm {L} ^ {\max }} (h, x) = \sum_ {y \in \mathcal {Y}} p (x, y) \max  _ {y ^ {\prime} \neq y} \Phi_ {1} (h (x, y) - h (x, y ^ {\prime})).
$$
Consider the distribution that supports on a singleton domain  $\{x\}$ . Take  $y_{1} \neq y_{2} \in \mathcal{Y}$  such that  $y_{1} \neq n$  and  $y_{2} \neq n$ . We define the conditional distribution as  $p(x, y_{1}) = p(x, y_{2}) = \frac{1}{2}$  and  $p(x, y) = 0$  for other  $y \in \mathcal{Y}$ . Then, by using the fact that  $\Phi_{1}(v)$  is convex and non-increasing, we obtain
$$
\begin{array}{l} \mathcal {R} _ {\mathrm {L} ^ {\max }} (h) = \mathcal {C} _ {\mathrm {L} ^ {\max }} (h, x) = \frac {1}{2} \max  _ {y ^ {\prime} \neq y _ {1}} \Phi_ {1} (h (x, y _ {1}) - h (x, y ^ {\prime})) + \frac {1}{2} \max  _ {y ^ {\prime} \neq y _ {2}} \Phi_ {1} (h (x, y _ {2}) - h (x, y ^ {\prime})) \\ = \frac {1}{2} \Phi_ {1} \left(h (x, y _ {1}) - \max  _ {y ^ {\prime} \neq y _ {1}} h (x, y ^ {\prime})\right) + \frac {1}{2} \Phi_ {1} \left(h (x, y _ {2}) - \max  _ {y ^ {\prime} \neq y _ {2}} h (x, y ^ {\prime})\right) (\Phi_ {1} (v) \text {i s n o n - i n c r e a s i n g}) \\ \geq \Phi_ {1} \left(\frac {1}{2} h (x, y _ {1}) - \frac {1}{2} \max  _ {y ^ {\prime} \neq y _ {2}} h (x, y ^ {\prime}) + \frac {1}{2} h (x, y _ {2}) - \frac {1}{2} \max  _ {y ^ {\prime} \neq y _ {1}} h (x, y ^ {\prime})\right) (\Phi_ {1} (v) \text {i s c o n v e x}) \\ \geq \Phi_ {1} (0) \quad (\Phi_ {1} (v) \text {i s n o n - i n c r e a s i n g}) \\ \end{array}
$$
where the equality can be achieved by  $h^* \in \mathcal{H}$ , defined as  $h^*(x, 1) = h^*(x, 2) = \ldots = h^*(x, n)$ . Therefore,  $h^*$  is a Bayes classifier of the structured max loss. Note that  $\mathsf{h}^*(x) = n$ . However, by Lemma 3, in such a case, the Bayes classifier  $h_\ell^*$  of the target loss satisfies that
$$
\mathsf{h}_{\ell}^{*}(x) = \operatorname *{argmin}_{y^{\prime}\in \mathcal{Y}}\sum_{y\in \mathcal{Y}}p(x,y)\ell (y^{\prime},y) = \operatorname *{argmin}_{y^{\prime}\in \mathcal{Y}}(p(x,y_{1})\mathbb{1}_{y^{\prime}\neq y_{1}} + p(x,y_{2})\mathbb{1}_{y^{\prime}\neq y_{2}}) = y_{1}\text{or} y_{2}.
$$
Thus, we obtain  $\mathsf{h}^* \neq \mathsf{h}_\ell^*$ . Therefore,  $\mathsf{L}^{\max}$  is not Bayes-consistent.
D Proofs for Voted Conditional Random Field
Proof. For the Voted Conditional Random Field  $L^{\mathrm{VCRF}}$ , the conditional  $L^{\mathrm{VCRF}}$ -risk can be expressed as follows:
$$
\mathcal {C} _ {\mathrm {L} ^ {\mathrm {V C R F}}} (h, x) = \sum_ {y \in \mathcal {Y}} p (x, y) \log \left(\sum_ {y ^ {\prime} \in \mathcal {Y}} e ^ {\ell (y, y ^ {\prime}) + h (x, y ^ {\prime}) - h (x, y)}\right).
$$
Consider the distribution that supports on a singleton domain  $\{x\}$ . Note that  $\mathcal{R}_{\mathrm{L}^{\mathrm{VCRF}}} = \mathcal{C}_{\mathrm{L}^{\mathrm{VCRF}}}$  is convex with respect to  $h(x,y), y = 1, \ldots, n$ . To find the global minimum, we will differentiate  $\mathcal{C}_{\mathrm{L}^{\mathrm{VCRF}}}$  with respect to  $h(x,y)$  for any  $y \in \mathcal{Y}$  and setting the derivatives to zero. Thus, we obtain for any  $y \in \mathcal{Y}$ ,
$$
p (x, y) \frac {- \sum_ {y ^ {\prime} \neq y} e ^ {\ell \left(y , y ^ {\prime}\right) + h \left(x , y ^ {\prime}\right) - h (x , y)}}{\sum_ {y ^ {\prime} \in \mathcal {Y}} e ^ {\ell \left(y , y ^ {\prime}\right) + h \left(x , y ^ {\prime}\right) - h (x , y)}} + \sum_ {y ^ {\prime} \neq y} p (x, y ^ {\prime}) \frac {e ^ {\ell \left(y ^ {\prime} , y\right) + h (x , y) - h \left(x , y ^ {\prime}\right)}}{\sum_ {y ^ {\prime \prime} \in \mathcal {Y}} e ^ {\ell \left(y ^ {\prime} , y ^ {\prime \prime}\right) + h \left(x , y ^ {\prime \prime}\right) - h (x , y ^ {\prime})}} = 0. \tag {16}
$$
Using the fact that  $\sum_{y' \neq y} e^{\ell(y, y') + h(x, y') - h(x, y)} = \sum_{y' \in \mathcal{Y}} e^{\ell(y, y') + h(x, y') - h(x, y)} - e^{\ell(y, y) + h(x, y) - h(x, y)}$  to further simplify the LHS of (16), we obtain for any  $y \in \mathcal{Y}$ ,
$$
p (x, y) = \sum_ {y ^ {\prime} \in \mathcal {Y}} p (x, y ^ {\prime}) \frac {e ^ {\ell \left(y ^ {\prime} , y\right) + h (x , y) - h \left(x , y ^ {\prime}\right)}}{\sum_ {y ^ {\prime \prime} \in \mathcal {Y}} e ^ {\ell \left(y ^ {\prime} , y ^ {\prime \prime}\right) + h \left(x , y ^ {\prime \prime}\right) - h \left(x , y ^ {\prime}\right)}} = \sum_ {y ^ {\prime} \in \mathcal {Y}} p (x, y ^ {\prime}) \frac {e ^ {\ell \left(y ^ {\prime} , y\right) + h (x , y)}}{\sum_ {y ^ {\prime \prime} \in \mathcal {Y}} e ^ {\ell \left(y ^ {\prime} , y ^ {\prime \prime}\right) + h \left(x , y ^ {\prime \prime}\right)}}. \tag {17}
$$
Consider a target loss function  $\mathsf{L}$  such that  $e^{\ell(y, y')} = \Phi_y \Phi_{y'}$ , that is  $\ell(y, y') = \log(\Phi_y) + \log(\Phi_{y'})$ , where  $\Phi_y$  is a function mapping from  $\mathcal{Y}$  to  $\mathbb{R}_+$ . For this special choice of the target loss function, the expression of  $\ell(y, y')$  decouple and (17) can be simplified to
$$
p (x, y) = \sum_ {y ^ {\prime} \in \mathcal {Y}} p (x, y ^ {\prime}) \frac {\Phi_ {y} \Phi_ {y ^ {\prime}} e ^ {h (x , y)}}{\sum_ {y ^ {\prime \prime} \in \mathcal {Y}} \Phi_ {y ^ {\prime}} \Phi_ {y ^ {\prime \prime}} e ^ {h (x , y ^ {\prime \prime})}} = \frac {\Phi_ {y} e ^ {h (x , y)}}{\sum_ {y ^ {\prime} \in \mathcal {Y}} \Phi_ {y ^ {\prime}} e ^ {h (x , y ^ {\prime})}}. \tag {18}
$$
Therefore, for the Bayes classifier  $h^*$  of Voted Conditional Random Field, by (18), we have
$$
\frac {\Phi_ {y} e ^ {h ^ {*} (x , 1)}}{p (x , 1)} = \dots = \frac {\Phi_ {y} e ^ {h ^ {*} (x , n)}}{p (x , n)}
$$
$$
\mathsf {h} ^ {*} (x) = \operatorname * {a r g m a x} _ {y ^ {\prime} \in \mathcal {Y}} h ^ {*} (x, y ^ {\prime}) = \operatorname * {a r g m a x} _ {y ^ {\prime} \in \mathcal {Y}} e ^ {h ^ {*} (x, y ^ {\prime})} = \operatorname * {a r g m a x} _ {y ^ {\prime} \in \mathcal {Y}} \frac {p (x , y ^ {\prime})}{\Phi_ {y ^ {\prime}}}.
$$
However, by Lemma 3, in such a case, the Bayes classifier  $h_{\ell}^{*}$  of the target loss satisfies that
$$
\mathsf{h}_{\ell}^{*}(x) = \operatorname *{argmin}_{y^{\prime}\in \mathcal{Y}}\sum_{y\in \mathcal{Y}}p(x,y)\ell (y^{\prime},y) = \operatorname *{argmin}_{y^{\prime}\in \mathcal{Y}}\sum_{y\in \mathcal{Y}}p(x,y)(\log (\Phi_{y}) + \log (\Phi_{y^{\prime}})) = \operatorname *{argmin}_{y^{\prime}\in \mathcal{Y}}\Phi_{y^{\prime}}.
$$
Thus, we obtain  $\mathsf{h}^* \neq \mathsf{h}_\ell^*$  in general. Indeed, consider the case where  $p(x,y) = \frac{\Phi_y^2}{\sum_{k=1}^{n} \Phi_k^2}, y \in \mathcal{Y}$ . Then,  $\mathsf{h}^*(x) = \operatorname{argmax}_{y' \in \mathcal{Y}} \frac{\Phi_{y'}}{\sum_{k=1}^{n} \Phi_k^2} = \operatorname{argmax}_{y' \in \mathcal{Y}} \Phi_{y'} \neq \operatorname{argmin}_{y' \in \mathcal{Y}} \Phi_{y'} = \mathsf{h}_\ell^*(x)$  when  $\{\Phi_y : y \in \mathcal{Y}\}$  are not equal. Therefore,  $L^{\mathrm{VCRF}}$  is not Bayes-consistent.
E Proofs for structured comp-sum losses
E.1 Structured logistic loss
Theorem 11 (H-consistency bound of  $\mathsf{L}_{\log}^{\mathrm{comp}}$ ). Assume that  $\mathcal{H}$  is symmetric and complete. Then, for any target loss  $\ell$ , hypothesis  $h \in \mathcal{H}$  and any distribution,
$$
\mathcal {R} _ {\mathrm {L}} (h) - \mathcal {R} _ {\mathrm {L}, \mathcal {H}} ^ {*} \leq 2 \left(\mathcal {R} _ {\mathrm {L} _ {\log} ^ {\text {c o m p}}} (h) - \mathcal {R} _ {\mathrm {L} _ {\log} ^ {\text {c o m p}}, \mathcal {H}} ^ {*} + \mathcal {M} _ {\mathrm {L} _ {\log} ^ {\text {c o m p}}, \mathcal {H}}\right) ^ {\frac {1}{2}} - \mathcal {M} _ {\mathrm {L}, \mathcal {H}}. \tag {19}
$$
Proof. For the comp-sum structured loss  $\mathsf{L}_{\log}^{\mathrm{comp}}$ , the conditional  $\mathsf{L}_{\log}^{\mathrm{comp}}$ -risk can be expressed as follows:
$$
\begin{array}{l} \mathcal {C} _ {\mathrm {L} _ {\log} ^ {\mathrm {c o m p}}} (h, x) = - \sum_ {y \in \mathcal {Y}} p (x, y) \sum_ {y ^ {\prime} \in \mathcal {Y}} \bar {\ell} \left(y ^ {\prime}, y\right) \log \left(\frac {e ^ {h \left(x , y ^ {\prime}\right)}}{\sum_ {y ^ {\prime \prime} \in \mathcal {Y}} e ^ {h \left(x , y ^ {\prime \prime}\right)}}\right) \\ = - \sum_ {y ^ {\prime} \in \mathcal {Y}} \log \left(\frac {e ^ {h (x , y ^ {\prime})}}{\sum_ {y ^ {\prime \prime} \in \mathcal {Y}} e ^ {h (x , y ^ {\prime \prime})}}\right) \sum_ {y \in \mathcal {Y}} p (x, y) \bar {\ell} (y ^ {\prime}, y) \\ = -\sum_{y^{\prime}\in \mathcal{Y}}\log (\mathfrak{S}(x,y^{\prime}))\overline{q} (x,y^{\prime}), \\ \end{array}
$$