Title: Trading off Consistency and Dimensionality of Convex Surrogates for Multiclass Classification
Venue: NeurIPS 2024
Accepted: yes
Keywords: Loss Functions

Consistency

Property Elicitation
Abstract: In multiclass classification over $n$ outcomes, we typically optimize some surrogate loss $L: \mathbb{R}^d \times\mathcal{Y} \to \mathbb{R}$ assigning real-valued error to predictions in $\mathbb{R}^d$. In this paradigm, outcomes must be embedded into the reals with dimension $d \approx n$ in order to design a consistent surrogate loss. Consistent losses are well-motivated theoretically, yet for large $n$, such as in information retrieval and structured prediction tasks, their optimization may be computationally infeasible. In practice, outcomes are typically embedded into some $\mathbb{R}^d$ for $d \ll n$, with little known about their suitability for multiclass classification. We investigate two approaches for trading off consistency and dimensionality in multiclass classification while using a convex surrogate loss. We first formalize partial consistency when the optimized surrogate has dimension $d \ll n$. 
We then check if partial consistency holds under a given embedding and low-noise assumption, providing insight into when to use a particular embedding into $\mathbb{R}^d$. Finally, we present a new method to construct (fully) consistent losses with $d \ll n$ out of multiple problem instances. Our practical approach leverages parallelism to sidestep lower bounds on $d$.

Trading off Consistency and Dimensionality of Convex Surrogates for Multiclass Classification

Abstract

In multiclass classification over n outcomes, we typically optimize some surrogate loss L : R d × Y → R assigning real-valued error to predictions in R d . In this paradigm, outcomes must be embedded into the reals with dimension d ≈ n in order to design a consistent surrogate loss. Consistent losses are well-motivated theoretically, yet for large n , such as in information retrieval and structured prediction tasks, their optimization may be computationally infeasible. In practice, outcomes are typically embedded into some R d for d ≪ n , with little known about their suitability for multiclass classification. We investigate two approaches for trading off consistency and dimensionality in multiclass classification while using a convex surrogate loss. We first formalize partial consistency when the optimized surrogate has dimension d ≪ n . We then check if partial consistency holds under a given embedding and low-noise assumption, providing insight into when to use a particular embedding into R d . Finally, we present a new method to construct (fully) consistent losses with d ≪ n out of multiple problem instances. Our practical approach leverages parallelism to sidestep lower bounds on d .

1 Introduction

Multiclass classification, due to its combinatorial and discontinuous nature, is intractable to optimize directly, which drives machine learners to optimize some nicer surrogate loss . To ensure these surrogates properly 'correspond' to the discrete classification task, we seek to design consistent surrogates. If one uses a consistent surrogate loss, in the limit of infinite data and model expressivity, one ends up with the same classifications as if one had solved the original intractable problem directly with probability 1 .

Surrogate losses form the backbone of gradient-based optimization for classification tasks. Optimizing a surrogate is easier than direct optimization, but a large dimension d of the surrogate loss L : R d ×Y → R can make gradient-based optimization intractable. Therefore, previous literature has operated under the premise that the prediction dimension d should be as low as possible, subject to consistency for the classification task [Ramaswamy and Agarwal, 2016, Finocchiaro et al., 2024, 2020]. For multi-class classification over n outcomes, the lower bound on d is n -1 [Ramaswamy and Agarwal, 2016].

Bo Waggoner

These previous works implicitly focus on a binary approach to consistency: a surrogate is either consistent for every possible label distribution, or it is not consistent. But there is a way out: lower bounds on the surrogate dimension d rely on edge-cases that rarely show up in reality [Ramaswamy and Agarwal, 2016]. As a result, practitioners are often willing to trade-off the guarantee of consistency in order to improve the computational tractability of optimization. However, we currently lack rigorous analysis tools to analyze many of the partially-consistent surrogates commonly used in practice. Thus, unlike previous works, our work focuses on this more realistic paradigm of partial consistency. We apply our unique approach to rigorously analyze a popular surrogate construction that encompasses methods such as one-hot and binary encoding. Our approach allows for fine-grained control of the trade-off between consistency and dimension.

Prior works have informally brushed upon the proposed partial-consistency paradigm, without rigorous study. For example, Agarwal and Agarwal [2015] impose a low-noise assumption to construct a surrogate for classification with d = log( n ) . However, their work does not provide any way to control the consistency-dimension trade-off. Similarly, Struminsky et al. [2018] characterize the excess risk bounds of inconsistent surrogates, which teaches us about the learning rates for inconsistent surrogates, but not under which distributional assumptions we can recover consistency guarantees.

Using different techniques than both of these approaches, we seek to understand the tradeoffs of consistency, surrogate prediction dimension, and number of problem instances through the use of polytope embeddings which are common in the literature [Wainwright et al., 2008, Blondel et al., 2020]. When embedding outcomes into d ≪ n dimensions, we first show there always exists a set of distributions where hallucinations occur: where the report minimizing the surrogate leads to a prediction ˆ y such that the underlying true distribution has no weight on the prediction; that is, Pr [ Y = ˆ y ] = 0 (Theorem 3). Following this, we show that every polytope embedding is partially consistent under strong enough low-noise assumptions (Theorem 5). Finally, we demonstrate through leveraging the embedding structure and multiple problem instances that the mode (in particular, a full rank ordering) over n outcomes embedded into a n 2 dimensional surrogate space is elicitable over all distributions via O ( n 2 ) problem instances (Theorem 10). This alternative approach to recovering consistency is parallelizable, detangling the complexity of gradient computation of one high-dimensional surrogate.

2 Background and Notation

Let Y be a finite label space, and throughout let n = |Y| . Define R Y + to be the nonnegative orthant. Let ∆ Y = { p ∈ R Y + | ∥ p ∥ 1 = 1 } be the set of probability distributions on Y , represented as vectors. We denote the point mass distribution of an outcome y ∈ Y by δ y ∈ ∆ Y . Let [ d ] := { 1 , . . . , d } . In general, we denote a discrete loss by ℓ : Y × Y → R + with outcomes denoted by y ∈ Y and a surrogate loss by L : R d ×Y → R with surrogate reports u ∈ R d and outcomes y ∈ Y . The surrogate must be accompanied by a link ψ : R d →Y mapping the convex surrogate model's predictions back into the discrete target space, and we discuss consistency of a pair ( L, ψ ) with respect to the target ℓ .

For ϵ > 0 , we define an epsilon ball via B ϵ ( u ) = { x ∈ R d | ∥ u -x ∥ 2 < ϵ } and B ϵ := B ϵ ( ⃗ 0) . Given a closed convex set C ⊂ R d , we define a projection operation onto C via Proj C ( u ) := arg min x ∈C ∥ u -x ∥ 2 . Given a closed convex set C ⊂ R d and u ∈ R d , we let the set-pointwise distance to be defined as ∥ u - C∥ 2 := ∥ u -Proj C ( u ) ∥ 2 . Full tables of notation are found in Appendix A.

2.1 Property Elicitation, Consistency, and Prediction Dimension

Consistency is generally a difficult condition to work with directly. Hence, we will use the notion of calibration , which is equivalent to consistency in our setting with finite outcomes. Our approach follows from the property elicitation literature, which allows us to abstract away from the feature space

X and focus on the conditional distributions over the labels, p = Pr[ Y | X = x ] ∈ ∆ Y [Bartlett et al., 2006, Tewari and Bartlett, 2007, Zhang, 2004, Ramaswamy and Agarwal, 2016, Steinwart, 2007]. In this approach, the central object of study is a property which maps label distributions to reports that minimize the loss.

Definition 1 (Property, Elicits, Level Set) . Let R be an arbitrary report set. For P ⊆ ∆ Y , a property is a set-valued function Γ : P → 2 R \ { ∅ } , which we denote Γ : P ⇒ Y . A loss L : R×Y → R elicits the property Γ on P if

If L elicits a property, it is unique and we denote it prop[ L ] . The level set of Γ for report r is the set Γ r := { p ∈ P | r = Γ( p ) } . If prop[ L ] = Γ and | Γ( p ) | = 1 for all p ∈ P , we say that L is strictly proper for Γ .

In this work, R = Y for target losses and R = R d for surrogate losses.

Once a model is optimized wrt. a surrogate L , it predicts reports in the surrogate space, R d . Then, to map surrogate reports to discrete labels, the surrogate loss must be paired with a link, ψ : R d →Y . Intuitively, a surrogate and link pair ( L, ψ ) are calibrated with respect to a target loss ℓ , if the optimal expected surrogate loss when making the incorrect classification (by ψ ) is strictly greater than the optimal surrogate loss.

Definition 2 ( ℓ -Calibrated Loss) . Given discrete loss ℓ : Y ×Y → R + , surrogate loss L : R d ×Y → R , and link function ψ : R d →Y . We say that ( L, ψ ) is ℓ -calibrated over P ⊆ ∆ Y if, for all p ∈ P ,

If P is not specified, then we are discussing calibration over ∆ Y . In general, when ( L, ψ ) is ℓ -calibrated over P such that P ⊂ ∆ Y , we say partial calibration holds with respect to P .

Our analysis crucially relies on the ability to specify P when invoking the definition of calibration. This is because the surrogates we analyze break the d = n -1 lower bound on the dimension of any consistent surrogate loss. So the surrogates will not be calibrated over the whole simplex ∆ Y . To aid in our analysis, we use a condition that shows that converging to a property value implies calibration for the target loss itself [Agarwal and Agarwal, 2015].

Definition 3 ( ℓ -Calibrated Property) . Let P ⊆ ∆ Y , Γ : P ⇒ R d , discrete loss ℓ : Y ×Y → R + , and ψ : R d →Y . We will say (Γ , ψ ) is ℓ -calibrated for all p ∈ P and all sequences in { u m } in R d if,

Theorem 1 ([Agarwal and Agarwal, 2015, Theorem 3]) . Let ℓ : Y × Y → R + and P ⊆ ∆ Y . Let Γ : P ⇒ R d and ψ : R d →Y be such that Γ is elicitable and (Γ , ψ ) is an ℓ -calibrated property over P . Let L : R d ×Y → R be a convex function for all y ∈ Y and strictly proper for Γ i.e. prop[ L ] = Γ and | Γ( p ) | = 1 for all p ∈ P . Then, ( L, ψ ) is ℓ -calibrated over P .

̸

Finally, we present the 0-1 loss that we analyze, which is the target loss for multiclass classification. Definition 4 (0-1 Loss) . We denote the 0-1 loss by ℓ 0 -1 : Y × Y → { 0 , 1 } such that ℓ 0 -1 ( y, ˆ y ) := y =ˆ y . Observe γ mode ( p ) := prop[ ℓ 0 -1 ]( p ) = { y ∈ Y| y ∈ arg max y p y } .

✶ 3 Polytope Embedding and Existence of Calibrated Regions

Often, discrete outcomes are embedded in continuous space onto the vertices of the simplex via one-hot encoding, or the vertices of the unit cube via binary encoding [Seger, 2018]. Generalizing, we introduce an approach to surrogate construction inspired by Wainwright et al. [2008] and Blondel et al. [2020] that encompasses the aforementioned embedding methods. This construction utilizes embeddings onto the vertices of arbitrary low-dimensional polytopes φ : Y → R d . Then, an embedding scheme naturally induces a large class of loss functions L G φ defined by the embedding, any G -Bregman Divergence, and a link function ψ φ .

Our analysis begins by defining a condition stronger than inconsistency that arises when embedding into d < n -1 dimensions for multiclass classification. To this end, we introduce the notion of

hallucination as a means to characterize the 'worst case' behavior of a surrogate pair (§ 3.2). In a positive manner, we characterize the calibration regions of various embeddings (§ 3.3), which are sets P ⊆ ∆ Y such that our surrogate and link pair ( L G φ , ψ φ ) are ℓ -calibrated over P . We refer the reader to the Appendix B for omitted full proofs.

3.1 Polytope Embedding Construction

A Convex Polytope P ⊂ R d , or simply a polytope, is the convex hull of a finite number of points u 1 , . . . , u n ∈ R d . An extreme point of a convex set A , is a point u ∈ A such that if u = λy +(1 -λ ) z with y, z ∈ A and λ ∈ [0 , 1] , then y = u and/or z = u . We shall denote by vert( P ) a polytope's set of extreme points. A polytope can be expressed by the convex hull of its extreme points, i.e. P = conv (vert( P )) [Brondsted, 2012, Theorem 7.2]. Additional definitions pertaining to polytopes are used for proofs that are omitted to the appendix, we refer the reader to (§ B.1) for said definitions.

We propose the following embedding procedure that allows one to construct surrogate losses with almost any polytope, and any Bregman divergence.

Construction 1 (Polytope Embedding) . Given Y outcomes, |Y| = n , choose a polytope P ⊂ R d such that | vert( P ) | = n . Choose a bijection between Y and vert( P ) . According to this bijection, assign each vertex a unique outcome so that { v y ∈ R d | y ∈ Y} = vert( P ) . Then the polytope embedding φ : ∆ Y → P is φ ( p ) := ∑ y ∈Y p y v y , which is the sum of p -scaled vectors

Following the work of Blondel [2019] and their proposed Projection-based losses, we use the extremely general class of Bregman divergences (Definition 5) and a polytope embedding φ to define an induced loss L G φ (Definition 6).

Definition 5 (Bregman Divergence) . Given a strictly convex function G : R d → R , D G ( u, v ) := G ( v ) -[ G ( u ) + ⟨ dG v , u -v ⟩ ] is a Bregman divergence where dG v denotes a subgradient of G at v . For this work, we shall always assume that dom( G ) = R d .

Definition 6 ( ( D G , φ ) Induced Loss) . Given a Bregman divergence D G and a polytope embedding φ , we say ( D G , φ ) induces a loss L G φ : R d ×Y → R + defined as

Note, that for any fixed y ∈ Y , L G φ ( u, y ) is convex with respect to u ∈ R d .

We show that for any p ∈ ∆ Y , the report that uniquely minimizes the expectation of the loss L G φ is φ ( p ) , the embedding point of p . Furthermore, the polytope P contains all of, and only the minimizing reports in expectation under L G φ .

Proposition 2. For a given induced loss L G φ , the unique report which minimizes the expected loss is u ∗ := arg min u ∈ R d E Y ∼ p [ L G φ ( u, Y )] = φ ( p ) such that u ∗ ∈ P . Furthermore, every ˆ u ∈ P is a minimizer of E Y ∼ ˆ p [ L G φ ( u, Y )] for some ˆ p ∈ ∆ Y .

We now define the maximum a posteriori (MAP) link, which will be used in conjunction with an induced loss L G φ to form a surrogate pair for the 0-1 loss. The MAP link projects surrogate predictions onto the polytope P , then links to the nearest vertex of P , and is commonly used in the literature [Tsochantaridis et al., 2005, Blondel, 2019, Xue et al., 2016]. Since the MAP link performs a projection, one may ask if this is computationally challenging; fortunately this operation is computationally feasible due to the convexity of the polytope [Blondel, 2019].

Definition 7 (MAP Link) . Let φ be a polytope embedding from ∆ Y to P . The MAP link ψ φ : R d → Y is defined as ψ φ ( u ) = arg min y ∈Y || Proj P ( u ) -v y || 2 . The level set of the link for y is ψ φ y = { u ∈ R d | y = ψ φ ( u ) } . We break ties arbitrarily but deterministically.

3.2 Hallucination Regions

Since our polytope embedding violates surrogate dimension bounds, calibration for 0-1 loss will not hold for all distributions. In particular, we show there always exists some distribution p such that p y = 0 yet E Y ∼ p [ L G φ ( u, Y )] is minimized at some u such that ψ φ ( u ) = y . This implies a 'worst case' inconsistency where the reported outcome could never actually occur with respect to our embedding of n events via φ into vert( P ) .

b

a

f

e

d

a

ad

cd

ab

ab

bc

C

b

d

Definition 8 (Hallucination) . Given ( L, ψ ) such that L : R d × Y → R + , |Y| = n , d < n , and ψ : R d →Y , we say that a hallucination occurs at a surrogate report u ∈ R d if, for some p ∈ ∆ Y , u ∈ arg min ˆ u ∈ R d E Y∼ p [ L (ˆ u, Y )] and ψ ( u ) := y but p y = 0 . We denote by H ⊆ P ⊂ R d as the hallucination region as the elements of P at which hallucinations can occur.

We express the subspace of the surrogate space where hallucinations can occur as the hallucination region denoted by H . In Theorem 3, we characterize the hallucination region for any polytope embedding while using the surrogate pair ( L G φ , ψ φ ) and show that H is never empty.

Theorem 3. For any given pair ( L G φ , ψ φ ) and ℓ 0 -1 with embedding dimension d < n -1 ; it holds that H = ∪ y ∈Y conv (vert( P ) \ { v y } ) ∩ ψ φ y and furthermore H ̸ = ∅ .

̸

Sketch. Fix y ∈ Y . We abuse notation and write vert( P -y ) := vert( P ) \ { v y } . Observe conv (vert( P -y )) ∩ ψ φ y ⊆ H since any point in this set can be expressed as a convex combination without needing vertex v y implying there is a distribution embedded by φ to said point which has no weight on y . We seek to show that H ⊆ ∪ y ∈Y conv (vert( P -y )) ∩ ψ φ y . Assume there exists a point u / ∈ conv (vert( P ) \ v y ) ∩ ψ φ y such that there exists some p ∈ ∆ Y where φ ( p ) = u , p y = 0 , and ψ φ ( u ) = y . Since ψ φ ( u ) = y and u / ∈ conv (vert( P -y )) ∩ ψ φ y , it must be the case that u / ∈ conv (vert( P -y )) . However, that implies that u is strictly in the vertex figure and thus must have weight on the coefficient for y . Thus, forming a contradiction that p y = 0 which implies that H ⊆ ∪ y ∈Y conv (vert( P -y )) ∩ ψ φ y . Finally, using Helly's Theorem [Rockafellar, 1997, Corollary 21.3.2] we show that ∩ y ∈Y conv (vert( P ) \ v y ) = ∅ , which implies the non-emptiness of H as well.

Theorem 3 suggests that using machine learning in high-risk settings such as medical and legal applications while violating the known n -1 dimensional bound for surrogate losses in multiclass classification is inherently ill-advised without human intervention given the possibility for hallucinations. Furthermore, hallucinations may be forced by the target loss, as in the case of Hamming loss (see Appendix C). In these cases practitioners should carefully consider the choice of target loss. We conjecture that hallucinations are common for many structured prediction losses. However this is not a concern in our primary loss of study of multi-class classification.

3.3 Calibration Regions

Ideally, we would like calibration to hold over the entire simplex since that would imply minimizing surrogate risk would always correspond to solving the target problem regardless of the true underlying

distribution. We observe that the mode's embedded level sets in the polytope overlap (see Figure 1L), which is unsurprising given that we are violating the lower bounds on surrogate prediction for the mode and hence calibration does not hold over the entire simplex. Since | 2 Y \ { ∅ }| is a finite set, we know that the number of unique mode level sets is finite. Although every point in the polytope is a minimizing report for some distribution, if multiple distributions with non-intersecting mode sets are embedded to the same point, there is no way to define a link function that is correct in all cases. However, if the union of mode sets for the p 's mapped to any u ∈ P is a singleton, regardless of the underlying distribution † , a link ψ would be calibrated over the union if it mapped u to the mentioned singleton. Given ( L, ψ ) , φ , and a target loss ℓ , we define strict calibrated regions as the points for which calibration holds regardless of the actual distribution realized, which are possible at said points.

Definition 9 (Strict Calibrated Region) . Suppose we are given ( L, ψ ) , φ , and a target loss ℓ . We say R ⊆ P is a strict calibrated region via ( L, ψ ) with respect to ℓ if ( L, ψ ) is ℓ -calibrated for all p ∈ φ -1 ( R ) := { p ∈ ∆ Y : φ ( p ) ∈ R } .

For any y ∈ Y , we define R y := R ∩ ψ y . We let R Y := ∪ y ∈Y R y .

By violating lower bounds, we are in a partially consistent paradigm where surrogate reports do not necessarily correspond to a unique distribution p . However, strict calibration regions allow us to check whether or not the loss is calibrated for the distribution p generating the data - even without explicit access to p . One simply has to check whether the report u is in R Y .

In Theorem 4, regardless of one's chosen P , we show that there always exists a non-zero Lebesgue measurable strict calibration region and that ( L G φ , ψ φ ) is calibrated for the 0-1 loss overall distributions embedded into the strict calibration region. This result shows that our surrogate and link construction for any d , always yields discernible calibration regions - lending support to the practical use and study of these surrogates.

Theorem 4. Let D G be a Bregman divergence, φ be any polytope embedding, ψ φ be the MAP link, and L G φ be the loss induced by ( D G , φ ) . There exists a P ⊆ ∆ Y with non-zero Lebesgue measure and φ ( P ) ⊆ R Y via ( L G φ , ψ φ ) with respect to ℓ 0 -1 .

Although strict calibration regions R y exist for each outcome y ∈ Y via the polytope embedding, tightly characterizing strict calibration regions is non-trivial. Since the level sets of elicitable properties are convex within the underlying simplex, characterizing the strict calibration regions becomes a collision detection problem, which is often computationally hard.

4 Restoring Inconsistent Surrogates via Low-Noise Assumptions

Looking towards application, we refine our results on the existence of strict calibration regions by examining a low-noise assumption, which provides an interpretable calibration region (§ 4.1). We show which low-noise assumptions imply calibration when embedding 2 d outcomes into d dimensions and d ! outcomes into d dimensions (§ 4.2). We refer the reader to Appendix B for omitted proofs.

4.1 Calibration via Low Noise Assumptions

We demonstrate that every polytope embedding leads to calibration under some low-noise assumption. Our results enable practictioners to choose the dimension d , unlike in previous works. Following previous work [Agarwal and Agarwal, 2015], we define a low noise assumption to be a subset of the probability simplex on the label distribution parameterized by ˆ α : Θ ˆ α = { p ∈ ∆ Y | max y ∈Y p y ≥ 1 -ˆ α } where ˆ α ∈ [0 , 1] . This noise assumption can be understood as Massart noise [Massart and Nédélec, 2006] in the multiclass setting.

Given α ∈ [0 , 1] and y ∈ Y , we define the set Ψ y α = { (1 -α ) δ y + αδ ˆ y | ˆ y ∈ Y} . With an embedding φ onto P , we define the set P y α := φ (conv (Ψ y α )) , a scaled version of P anchored at v y , that moves vertices (1 -α ) proportionally towards y , (Figure 2R).

Theorem 5. Let D G be a Bregman divergence, φ be any polytope embedding, and L G φ be the loss induced by ( D G , φ ) . There exists an α ∈ [0 , . 5) such that for the link ψ φ,α ( u ) = arg min y ∈Y ∥ u -P y α ∥ 2 , ( L G φ , ψ φ,α ) is ℓ 0 -1 -calibrated over the distributions Θ α := { p ∈ ∆ Y | max y ∈Y p y ≥ 1 -α } .

̸

̸

Proof. Part 1 (Choosing α ∈ [0 , . 5) ) : By Theorem 4, there exists an ϵ > 0 such that B ϵ ( v y ) ∩ P ⊆ R y for all y ∈ Y . Given that vert( P ) are unique points, there exists a sufficiently small ϵ ′ > 0 such that B ϵ ′ ( v ) ∩ B ϵ ′ (ˆ v ) = ∅ for all v, ˆ v ∈ vert( P ) where v = ˆ v . Let ϵ ′′ = min( ϵ, ϵ ′ ) . For any y ∈ Y , observe the set conv (Ψ y α ) , defined using any α ∈ [0 , . 5) , is a scaled-down translated unit simplex and that for all p ∈ conv (Ψ y α ) ⊂ ∆ Y it holds that y = mode ( p ) .

We shall show that for some sufficiently small α ∈ [0 , . 5) , P y α is a scaled down version of P positioned at the respective vertex v y . Furthermore, we shall show that P y α ⊂ B ϵ ′′ ( v y ) ∩ P ⊆ R y for all y ∈ Y . Observe that by linearity of φ ,

and hence, P y α is a scaled version of P positioned at v y . Hence for some sufficiently small α , (1 -α ) v y + αv ˆ y ∈ B ϵ ′′ ( v y ) for all ˆ y and hence P y α ⊆ B ϵ ′′ ( v y ) ⊆ R y . With said sufficiently small α , define ψ φ,α and the respective sets conv (Ψ y α ) for each y ∈ Y . Using the previous α , define the set Θ α as well.

Part 2 (Showing Calibration) : Recall, by Proposition 2, for any p ∈ ∆ Y , u = φ ( p ) minimizes the expected surrogate loss E Y∼ p [ L G φ ( u, Y )] . For any fixed y ∈ Y , observe that conv { (1 -α ) δ y + αδ ˆ y | ˆ y ∈ Y} = { p ∈ ∆ Y : p y ≥ 1 -α } ⊂ ∆ Y and hence, by Proposition 2, ∪ y ∈Y P y α contains all of the minimizing surrogate reports with respect to Θ α . By our choice of α and the construction of ψ P α , every u ∈ ∪ y ∈Y P y α is linked to the proper unique mode outcome since ∪ y ∈Y P y α ⊆ R Y . Assuming a low-noise condition where p ∈ Θ α , any u / ∈ ∪ y ∈Y P y α is never optimal for any low-noise distribution. In such cases, we project the point to the nearest P y α as a matter of convention. Given that calibration is a result pertaining to minimizing reports, this design choice is non-influential. Finally, since every ∪ y ∈Y P y α ⊆ R Y , by the definition of strict calibration region, it holds that ( L G φ , ψ φ,α ) is ℓ 0 -1 -calibrated for Θ α .

4.2 Embedding into the Unit Cube and Permutahedron under Low-Noise

In this section, we demonstrate embedding onto the unit cube and the permutahedron [Blondel et al., 2020, Seger, 2018]. We show that by embedding 2 d outcomes into a d dimensional unit cube P □ , ( L G φ , ψ P □ ,α ) is calibrated over Θ α for all α ∈ [0 , 1 2 ) . Furthermore, we found that by embedding d ! outcomes into a d dimensional permutahedron P w , ( L G φ , ψ P w ,α ) is calibrated for Θ α for α ∈ [0 , 1 d ) . Theorem 6 enables us to simultaneously study the aforementioned embeddings.

̸

Theorem 6. Let D G be a Bregman divergence, φ be any polytope embedding, and L G φ be the loss induced by ( D G , φ ) . Fix α ∈ [0 , . 5) and with it define Θ α . If for all y, ˆ y ∈ Y such that y = ˆ y it holds that P y α ∩ P ˆ y α = ∅ , then ( L G φ , ψ φ,α ) is ℓ 0 -1 -calibrated for Θ α where ψ φ,α ( u ) = arg min y ∈Y ∥ u -P y α ∥ 2 .

̸

Proof. Pick an α such that for all y, ˆ y ∈ Y , P y α ∩ P ˆ y α = ∅ . Define Θ α and ψ φ,α accordingly. For p ∈ Θ α and some y ∈ Y , say a sequence { u m } converges to prop[ L G φ ]( p ) = φ ( p ) ∈ P y α , where the equality follows from Proposition 2. Given that each P y α is closed and pairwise disjoint, there exists some ˆ ϵ > 0 such that for all y, ˆ y ∈ Y where y = ˆ y , it also holds that ( P y α + B ˆ ϵ ) ∩ ( P ˆ y α + B ˆ ϵ ) = ∅ where + denotes the Minkowski sum. Since { u m } converges to φ ( p ) , there exists some N ∈ N such that for all n ≥ N , ∥ u n -φ ( p ) ∥ 2 < ˆ ϵ . By the definition of ψ φ,α , any u n where n ≥ N will be mapped to y , the correct unique report given that prop[ L G φ ]( p ) ∈ P y α . Hence, (prop[ L G φ ] , ψ φ,α ) is ℓ 0 -1 -calibrated property with respect to Θ α . Finally, since L G φ is strictly proper for prop[ L G φ ] , by Theorem 1, we have that ( L G φ , ψ φ,α ) is ℓ 0 -1 -calibrated for Θ α .

Unit Cube Define a unit cube in d -dimensions by P □ := conv ( {-1 , 1 } d ) . Binary encoding outcomes into the elements of {-1 , 1 } d (the vertices of a unit cube) is a commonly used method in practice (e.g., [Seger, 2018, Yu and Blaschko, 2018]). We show that calibration holds under a low noise assumption of Θ α when α < . 5 .

Corollary 7. Let φ be an embedding from 2 d outcomes into the vertices of P □ in d -dimensions and define an induced loss L G φ . Fix α ∈ [0 , . 5) and define Θ α . ( L G φ , ψ P □ ,α ) is ℓ 0 -1 -calibrated for Θ α .

d

C

b

a

f

e

d

a

b

Corollary 7 suggests that binary encoding is an appropriate methodology when one has a prior over the data that the mode of the label distribution Pr[ Y | X = x ] is greater than half for all x ∈ X . Interestingly, the bound of α is not dependent on the dimension of d . We now present a result for embedding outcomes into a factorially lower dimension via the permutahedron. Intuitively, ranking can be recast as a multiclass classification problem, in which case the outcomes are orderings of the d possible labels.

Permutahedron Let S d express the set of permutations on [ d ] . The permutahedron associated with a vector w ∈ R d is defined to be the convex hull of the permutations of the indices of w , i.e., P w := conv ( { π ( w ) | π ∈ S d } ) ⊂ R d . The permutahedron may serve as an embedding from d ! outcomes into d -dimensions; it is a natural choice for embedding full rankings over d items.

Corollary 8. Let φ be an embedding from d ! outcomes into the vertices of P w in d dimensions such that w = (0 , 1 βd , 2 βd , . . . , d -1 βd ) ∈ R d where β = d -1 2 . Fix α ∈ [0 , 1 d ) . Then ( L G φ , ψ P w ,α ) is ℓ 0 -1 -calibrated over Θ α .

The calibration region in Corollary 8 show that consistency in Θ α shrinks exponentially in d . Unless one has a prior that the data follows some form of a power distribution, Corollary 8 suggests not to factorially embed outcomes.

5 Elicitation in Low Dimensions with Multiple Problem Instances

The tools developed in previous sections now enable us to address the setting in which we require full consistency, P = ∆ Y , but also desire surrogate prediction dimension d ≪ n -1 . We side-step the n -1 lower bound by utilizing multiple problem instances and aggregation of the outputs. Although cumulatively we have a larger surrogate prediction dimension than n -1 , each individual problem instance has a less than n -1 surrogate prediction dimension. This approach is well-motivated since it allows for distributed computing of separate, smaller models which leads to faster convergence overall since in general optimization is at least poly ( d ) . Previous work such as Ramaswamy et al. [2014] has explored the consistency of multiclass problem reductions; however, we take a different, geometrically motivated, approach.

Definition 10. Extending Definition 1, we say a loss and link pair ( L, ψ ) , where L : R d ×Y → R and ψ : R d → Y , elicits a property Γ : P ⇒ Y on P ⊆ ∆ Y if ∀ p ∈ P , Γ( p ) = ψ (arg min u ∈ R d E Y ∼ p [ L ( u, Y )]) .

Definition 11 ( ( n, d, m ) -Polytope Elicitable) . Suppose we are given a property γ : P ⇒ Y such that P ⊆ ∆ Y and |Y| = n finite outcomes. Say we have m unique polytope embeddings { φ j : ∆ Y → R d } j m =1 where d < n -1 , and a set of induced losses { L G φ j } j m =1 and links ψ j : R d →B j defined wrt. φ j , where B j is an arbitrary report set. For each j ∈ [ m ] , assume the pair ( L G φ j , ψ j ) elicits the property Γ j : P ⇒ B j . If there exists a function Υ : B 1 ×··· × B m ⇒ Y such that for any p ∈ ∆ Y it holds that Υ(Γ 1 ( p ) , . . . , Γ m ( p )) = γ ( p ) , we say that γ is ( n, d, m ) -Polytope Elicitable over P .

a

C

C

b

d

a

CQ

d

b

Equivalently, we will also say that the pair ( { ( L G φ j , ψ j ) } j m =1 , Υ) ( n, d, m ) -Polytope elicits the property γ with respect to P .

We shall express a d -cross polytope by P ⊕ := conv ( { π (( ± 1 , 0 , . . . , 0)) | π ∈ S d } ) where ( ± 1 , 0 , . . . , 0) ∈ R d . Observe that a d -cross polytope has 2 d vertices. For any vertex of a d-cross polytope v ∈ vert( P ⊕ ) , we shall say that ( v, -v ) forms a diagonal vertex pair.

Lemma 9. Say we are given a cross-polytope embedding φ : ∆ 2 d → P ⊕ and induced loss L G φ . Let ( v a i , v b i ) , be the i th diagonal pair (i.e. φ ( δ a i ) = v a i ). Define the property Γ φ : ∆ 2 d → B element-wise by

Furthermore define the link ψ P ⊕ : R d →B with respect to each diagonal pair as

Then ( L G φ , ψ P ⊕ ) elicits Γ φ .

The following theorem states that by using multiple problem instances, based on Lemma 9, we can Polytope-elicit the mode. Algorithm 1 outlines how to aggregate the individual solutions to infer the mode. We defer the proof to Appendix B.

Theorem 10. Let d ≥ 2 . The mode is (2 d, d, m ) -Polytope Elicitable for some m ∈ [2 d -1 , d (2 d -1)] .

Algorithm 1 Elicit mode via comparisons and the d -Cross Polytopes

Learn a model h j : X → R d for each instance ( L G φ , ψ P ⊕ j ) ∈ M

For some fixed x ∈ X , collect all B j ← ψ P ⊕ j ( h j ( x )) where B j ∈ B Report R ← FindMaxes † ( B , . . . , B )

Although Theorem 10 states that the mode is (2 d, d, m ) -Polytope Elicitable for some m ∈ [2 d -1 , d (2 d -1)] , it does not state how we select said { ( L G φ j , ψ P ⊕ j ) } j m =1 problem instances in an optimal manner. Unfortunately, selecting the min number of problem instances reduces to a a minimum set cover problem which is computationally hard. Even so, through a greedy approach, one can choose

problem instances that are log approximate optimal relative to the true best configuration. In practice using real data, given that these are asymptotic results, we may have conflicting logic for the provided individual reports. In Appendix D, we discuss an approach of how to address this in practice.

6 Discussion and Conclusion

This work examines various tradeoffs between surrogate loss dimension, restricting the region of consistency in the simplex when using the 0-1 loss, and number of problem instances. Since our analysis is based on an embedding approach commonly used in practice, our work provides theoretical guidance for practitioners choosing an embedding. We see several possible future directions. The first is a deeper investigation into hallucinations. Future work could investigate the size of the hallucination region in theory, and the frequency of reports in the hallucination region in practice. Another direction would be to construct a method that efficiently identifies the strict calibration regions and the distributions embedded into them. This would provide better guidance on whether or not a particular polytope embedding aligns with one's prior over the data. Another possible direction would be to explore whether concepts from this paper could be applied to the underlying problem of cost-sensitive multiclass classification. Finally, another direction is to identify other properties that can be elicited via multiple problem instances while also reducing the dimension of any one instance.

Broader Impacts: Our work broadly informs the selection of loss functions for machine learning. Thus our work may influence practitioners' choice of loss function. Of course, such loss functions can be used for ethical or unethical purposes. We do not know of particular risks of negative impacts of this work beyond risks of machine learning in general.

Acknowledgments and Disclosure of Funding

We thank Rafael Frongillo for discussions about hallucinations, which led to the exploration of many of the ideas in this work and Amzi Jeffs for discussions regarding convex geometry. This material is based upon work supported by the National Science Foundation under Award No. 2202898 (JF).

References

Arpit Agarwal and Shivani Agarwal. On consistent surrogate risk minimization and property elicitation. In Conference on Learning Theory , pages 4-22. PMLR, 2015.

Arindam Banerjee, Xin Guo, and Hui Wang. On the optimality of conditional expectation as a bregman predictor. IEEE Transactions on Information Theory , 51(7):2664-2669, 2005.

Peter L Bartlett, Michael I Jordan, and Jon D McAuliffe. Convexity, classification, and risk bounds. Journal of the American Statistical Association , 101(473):138-156, 2006.

Mathieu Blondel. Structured prediction with projection oracles. Advances in neural information processing systems , 32, 2019.

Mathieu Blondel, André FT Martins, and Vlad Niculae. Learning with fenchel-young losses. The Journal of Machine Learning Research , 21(1):1314-1382, 2020.

Arne Brondsted. An introduction to convex polytopes , volume 90. Springer Science & Business Media, 2012.

Jessie Finocchiaro, Rafael Frongillo, and Bo Waggoner. Embedding dimension of polyhedral losses. In Conference on Learning Theory , pages 1558-1585. PMLR, 2020.

Jessie Finocchiaro, Rafael Frongillo, and Bo Waggoner. Unifying lower bounds on prediction dimension of consistent convex surrogates. arXiv preprint arXiv:2102.08218 , 2021.

Jessie Finocchiaro, Rafael M Frongillo, and Bo Waggoner. An embedding framework for the design and analysis of consistent polyhedral surrogates. Journal of Machine Learning Research , 25(63): 1-60, 2024.

Peter M Gruber. Convex and discrete geometry , volume 336. Springer, 2007.

Jean-Baptiste Hiriart-Urruty and Claude Lemaréchal. Fundamentals of convex analysis . Springer Science & Business Media, 2004.

Pascal Massart and Élodie Nédélec. Risk bounds for statistical learning. 2006.

Harish G Ramaswamy and Shivani Agarwal. Convex calibration dimension for multiclass loss matrices. The Journal of Machine Learning Research , 17(1):397-441, 2016.

Harish G Ramaswamy, Balaji Srinivasan Babu, Shivani Agarwal, and Robert C Williamson. On the consistency of output code based learning algorithms for multiclass learning problems. In Conference on Learning Theory , pages 885-902. PMLR, 2014.

R Tyrrell Rockafellar. Convex analysis , volume 11. Princeton university press, 1997.

Cedric Seger. An investigation of categorical variable encoding techniques in machine learning: binary versus one-hot and feature hashing, 2018.

Ingo Steinwart. How to compare different loss functions and their risks. Constructive Approximation , 26(2):225-287, 2007.

Kirill Struminsky, Simon Lacoste-Julien, and Anton Osokin. Quantifying learning guarantees for convex but inconsistent surrogates. Advances in Neural Information Processing Systems , 31, 2018.

Ambuj Tewari and Peter L Bartlett. On the consistency of multiclass classification methods. Journal of Machine Learning Research , 8(5), 2007.

Ioannis Tsochantaridis, Thorsten Joachims, Thomas Hofmann, Yasemin Altun, and Yoram Singer. Large margin methods for structured and interdependent output variables. Journal of machine learning research , 6(9), 2005.

Martin J Wainwright, Michael I Jordan, et al. Graphical models, exponential families, and variational inference. Foundations and Trends® in Machine Learning , 1(1-2):1-305, 2008.

Yexiang Xue, Zhiyuan Li, Stefano Ermon, Carla P Gomes, and Bart Selman. Solving marginal map problems with np oracles and parity constraints. Advances in Neural Information Processing Systems , 29, 2016.

Jiaqian Yu and Matthew B Blaschko. The lovász hinge: A novel convex surrogate for submodular losses. IEEE transactions on pattern analysis and machine intelligence , 42(3):735-748, 2018.

Tong Zhang. Statistical analysis of some multi-category large margin classification methods. Journal of Machine Learning Research , 5(Oct):1225-1251, 2004.

A Notation tables

B Polytopes, Omitted Proofs, and Results

B.1 Polytopes

A Convex Polytope P ⊂ R d , or simply a polytope, is the convex hull of a finite number of points u 1 , . . . , u n ∈ R d . An extreme point of a convex set A , is a point u ∈ A such that if u = λy +(1 -λ ) z with y, z ∈ A and λ ∈ [0 , 1] , then y = u and/or z = u . We shall denote by vert( P ) a polytope's set of extreme points. A polytope can be expressed by the convex hull of its extreme points, i.e. P = conv (vert( P )) [Brondsted, 2012, Theorem 7.2].

Wedefine the dimension of P via dim( P ) := dim(affhull( P )) where affhull( P ) denotes the smallest affine set containing P . A set F ⊆ P is a face of P is there exists a hyperplane H ( y, α ) := { u ∈ R d | ⟨ u, y ⟩ = α } such that F = P ∩ H and P ⊆ H + such that H + ( y, α ) := { u ∈ R d | ⟨ u, y ⟩ ≤ α } . Let F i ( P ) where i ∈ [ d -1] denote set of faces of dim i of a polytope P . A face of dimension zero is called a vertex and a face of dimension one is called an edge. We define the edge set of a polytope P by E ( P ) := { conv (( v i , v j )) | ( v i , v j ) ⊆ ( vert( P ) 2 ) , conv (( v i , v j )) ∈ F 1 ( P ) } . We define the neighbors of a vertex v by ne( v ; P ) := { ˆ v ∈ vert( P ) | conv (( v, ˆ v )) ∈ E ( P ) } . We will denote conv (( v, ˆ v )) ∈ E ( P ) by as e v, ˆ v and ne( v ; P ) by ne( v ) when clear from context.

B.2 Omitted Proofs from § 3

Proposition 2. For a given induced loss L G φ , the unique report which minimizes the expected loss is u ∗ := arg min u ∈ R d E Y ∼ p [ L G φ ( u, Y )] = φ ( p ) such that u ∗ ∈ P . Furthermore, every ˆ u ∈ P is a minimizer of E Y ∼ ˆ p [ L G φ ( u, Y )] for some ˆ p ∈ ∆ Y .

Proof. By [Banerjee et al., 2005, Theorem 1], the minimizer of E Y ∼ p [ L G φ ( u, Y )] is ∑ y ∈Y p y v y = φ ( p ) . Thus, by the construction of the polytope embedding, it holds that u ∗ = φ ( p ) . Since Bregman divergences are defined with respect to strictly convex functions, u ∗ uniquely minimizes E Y ∼ p [ L G φ ( u, Y )] .

Conversely, every ˆ u ∈ P is expressible as a convex combination of vertices; hence, by the definition of φ , for some distribution, say ˆ p ∈ ∆ Y , it holds ˆ u = φ (ˆ p ) . Therefore, it holds that ˆ u minimizes E Y ∼ ˆ p [ L G φ ( u, Y )] .

Theorem 3. For any given pair ( L G φ , ψ φ ) and ℓ 0 -1 with embedding dimension d < n -1 ; it holds that H = ∪ y ∈Y conv (vert( P ) \ { v y } ) ∩ ψ φ y and furthermore H ̸ = ∅ .

Proof. Choose a y ∈ Y . We abuse notation and write vert( P ) \ v y := vert( P ) \ { v y } . Observe all u ∈ conv (vert( P ) \ v y ) ∩ ψ φ y can be expressed as a convex combination of vertices without needing vertex v y . The coefficients of said convex combination express a p ∈ ∆ Y that is embedded to the point u ∈ P where p y = 0 . Yet, by Proposition 2, said u is an expected minimizer of L G φ with respect to p . Given the intersection with ψ φ y and by Definition 8, it holds that ∪ y ∈Y conv (vert( P ) \ v y ) ∩ ψ φ y ⊆ H .

We now shall show that H ⊆ ∪ y ∈Y conv (vert( P ) \ v y ) ∩ ψ φ y . Fix y ∈ Y . Assume there exists a point u / ∈ conv (vert( P ) \ v y ) ∩ ψ φ y such that there exists some p ∈ ∆ Y where φ ( p ) = u , p y = 0 , and ψ φ ( u ) = y . Since ψ φ ( u ) = y and u / ∈ conv (vert( P ) \ v y )) ∩ ψ φ y , it must be the case that u / ∈ conv (vert( P ) \ v y ) . However, that implies that u is strictly in the vertex figure and thus must have weight on the coefficient for y . Thus, forming a contradiction that p y = 0 which implies that H = ∪ y ∈Y conv (vert( P ) \ v y ) ∩ ψ φ y .

̸

To show non-emptiness of H , we shall use Helly's Theorem (Rockafellar [1997], Corollary 21.3.2). W.l.o.g, assign an index such that Y = { y 1 , . . . , y d , y d +1 , . . . , y n } . Observe the elements of the set {Y \ y i } n i =1 each differ by one element. W.l.o.g, pick the first d +1 elements of the previous set. Observe | ∩ d +1 i =1 Y \ y i | = |Y \ { y 1 , . . . , y d , y d +1 }| = n -( d +1) > 0 . Hence, by Helly's theorem and uniqueness of y i 's, ∩ y ∈Y conv (vert( P ) \ v y ) = ∅ .

Pick a point u ′ ∈ ∩ y ∈Y conv (vert( P ) \ v y ) . Since ψ φ is well-defined, u ′ will be linked to some outcome y ′ ∈ Y and thus u ′ ∈ conv (vert( P ) \ v y ′ ) ∩ ψ φ y ′ ⊂ H . Yet, u ′ can be expressed as a

convex combination which does not use v y ′ since it lies in ∩ y ∈Y conv (vert( P ) \ v y ) . Thus, by using Proposition 2 and by the definition of Hallucination (Def. 8), we have that H ̸ = ∅ .

Lemma1 (Proposition 1.2.4) . [Hiriart-Urruty and Lemaréchal, 2004] If φ is an affine transformation of R n and A ⊂ R n is convex, then then the image φ ( A ) is also convex. In particular, if the set A is a convex polytope, the image is also a convex polytope.

Lemma 2. Let D G be a Bregman divergence, φ be any polytope embedding, ψ be the MAP link, and L G φ be the loss induced by ( D G , φ ) . Assume the target loss is ℓ 0 -1 . If a point is in a strict calibrated region such that u ∈ R y for some y ∈ Y , it is necessary that u ∈ conv ( { v y } ∪ ne( v y )) \ conv (ne( v y )) .

Proof. If u ∈ R y and u ∈ P \ ( conv ( { v y } ∪ ne( v y )) \ conv (ne( v y )) ) , then u can be expressed as a convex combination which has no weight on the coefficient for v y . Hence, there exists a distribution embedded into u where y would not be the mode, thus violating the initial claim that u ∈ R y .

Lemma 3. Let D G be a Bregman divergence, φ be any polytope embedding, ψ be the MAP link, and L G φ be the loss induced by ( D G , φ ) . For any u ∈ e ( v i ,v j ) ∈ E ( P ) , it holds that | φ -1 ( u ) | = 1 .

Proof. Observe, the two vertices of an edge define the convex hull making up the edge and hence, by (Gruber [2007] ,Theorem 2.3) the two vertices are affinely independent. Therefore, all elements of the edge have a unique convex combination which are expressed by the convex combinations of the edge's vertices. Given the relation of the embedding φ and convex combinations of vertices expressing distributions, it holds that | φ -1 ( u ) | = 1 .

Lemma 4. Let D G be a bregman divergence, φ be a polytope embedding, and L G φ be the induced loss by ( D G , φ ) . For all y ∈ Y , it holds that dim( φ ( mode y )) = dim( P ) ≥ 2 .

Proof. By the construction of φ , we know that dim( P ) ≥ 2 . Fix y ∈ Y . By Lemma 3, we know that any edge connected from v y and ˆ v ∈ ne( v y ) , the distributions embedded into the half of the line segment closer to v y , y is in the mode. By Lemma 1, we know that φ ( γ mode y ) is a convex set. Thus, the convex hull of the half line segments is part of φ ( γ mode y ) . Since each vertex has at least dim( P ) neighbors, it holds that dim( φ ( γ mode y )) = dim( P ) .

Theorem 4. Let D G be a Bregman divergence, φ be any polytope embedding, ψ φ be the MAP link, and L G φ be the loss induced by ( D G , φ ) . There exists a P ⊆ ∆ Y with non-zero Lebesgue measure and φ ( P ) ⊆ R Y via ( L G φ , ψ φ ) with respect to ℓ 0 -1 .

̸

Proof. Recall that γ mode ( p ) := prop[ ℓ 0 -1 ]( p ) = mode ( p ) . Fix y ∈ Y . For contradiction, assume for any ˆ y ∈ Y where y = ˆ y , it holds that B ϵ ( v y ) ∩ φ ( γ mode ˆ y ) = ∅ for all ϵ > 0 . By Lemma 3, it holds that conv ( { v y } ∪ m v y ,α ) ⊆ φ ( γ mode y ) where m v y ,α := { (1 -α ) v y + αv | v ∈ ne( v y ) } defined by any α ∈ (0 , . 5) . Furthermore, the elements of ∪ m ∈ m vy,α conv ( { v y } ∪ { m } ) have one distribution embedded onto it where y is the only valid mode thus, we know that φ ( mode ˆ y ) ∩ ∪ m ∈ m vy,α conv ( { v y } ∪ { m } ) = ∅ . Since φ ( γ mode ˆ y ) ⊂ P is closed and convex, there must exist some non-negative min distance between φ ( γ mode ˆ y ) and v y which we shall denote by d v . For any ϵ ∈ (0 , d v y ) , we can define B ϵ ( v y ) such that B ϵ ( v y ) ∩ φ ( γ mode ˆ y ) = ∅ , forming a contradiction.

̸

For each v y ∈ vert( P ) define a d v y and let ϵ ′ ∈ ∩ v y ∈ vert( P ) (0 , d v y ) . By the construction of P and the definition of ψ φ , there exists a ϵ ′′ > 0 such that for all u ∈ B ϵ ′′ ( v y ) it holds that ψ ( u ) = y and B ϵ ′′ ( v y ) ⊂ ψ φ y . For any y ∈ Y , we know that B min { ϵ ′ ,ϵ ′′ } ( v y )) ∩ P ⊆ R y by the construction of our epsilon ball. We claim φ -1 ( B min { ϵ ′ ,ϵ ′′ } ( v y ) ∩ P ) is a set of distributions for which calibration holds.

For p ∈ ∆ Y such that φ ( p ) ∈ B min { ϵ ′ ,ϵ ′′ } ( v y ) ∩ P for some v y ∈ vert( P ) , suppose a sequence { u m } converges to prop[ L G φ ]( p ) = φ ( p ) (equality by Proposition 2). By construction of B min { ϵ ′ ,ϵ ′′ } ( v y ) ∩ P , ψ φ ( φ ( p )) = y ∈ mode ( p ) and hence, a minimizing report for ℓ 0 -1 ( y ; p ) . Furthermore, since B min { ϵ ′ ,ϵ ′′ } ( v y ) ⊂ ψ φ φ -1 ( v y ) , all elements within B min { ϵ ′ ,ϵ ′′ } ( v y ) link to y . Since { u m } converges to prop[ L G φ ]( p ) , there exists some N ∈ N and n ≥ N , such that ∥ u n -φ ( p ) ∥ 2 <

min { ϵ ′ , ϵ ′′ } , meaning that E Y∼ p [ ℓ 0 -1 ( ψ φ ( u m ) , Y )] → min y ∈Y E Y∼ p [ ℓ 0 -1 ( y, Y )] . Hence, for any v y ∈ vert( P ) , (prop[ L G φ ] , ψ φ ) is ℓ 0 -1 -calibrated property with respect to φ -1 ( B min { ϵ ′ ,ϵ ′′ } ( v y ) ∩ P ) . Furthermore, by the construction of B min ϵ ′ ,ϵ ′′ ( v y ) for each v y ∈ vert( P ) , we have that L G φ is strictly for prop[ L G φ ] . Thus, by Theorem 1, ( L G φ , ψ φ ) is ℓ 0 -1 -calibrated for at least the distributions P = ∪ v y ∈ vert( P ) φ -1 ( B min { ϵ ′ ,ϵ ′′ } ( v y ) ∩ P ) as well as φ ( P ) ⊆ R Y . Furthermore, since B min { ϵ ′ ,ϵ ′′ } for each v y ∈ vert( P ) is non-empty, we have that P ̸ = ∅ .

B.3 Omitted Proofs from § 4

Corollary 7. Let φ be an embedding from 2 d outcomes into the vertices of P □ in d -dimensions and define an induced loss L G φ . Fix α ∈ [0 , . 5) and define Θ α . ( L G φ , ψ P □ ,α ) is ℓ 0 -1 -calibrated for Θ α .

Proof. W.l.o.g, say the outcome y 1 ∈ Y is embedded into ✶ [ d ] ∈ vert( P □ ) . Say α = . 5 . Observe that

̸

and that 1 ≥ (1 -α ) ± α ≥ 0 for any α ∈ (0 , . 5) . Hence, for any α ∈ (0 , . 5) it holds that P y 1 0 . 5 = conv ( { 0 , 1 } d ) and furthermore P y 1 α ⊂ P y 1 0 . 5 ⊂ R d > 0 . By symmetry of P □ and the linearity of φ , for any α ∈ (0 , . 5) and y ∈ Y , we have that P y α is a strict subset of the orthant that contains v y . Hence, for all y, ˆ y ∈ Y such that y = ˆ y , it holds that P y α ∩ P ˆ y α = ∅ . Thus by Theorem 6, ( L G φ , ψ P □ ,α ) is ℓ 0 -1 -calibrated for Θ α where α ∈ (0 , . 5) .

Corollary 8. Let φ be an embedding from d ! outcomes into the vertices of P w in d dimensions such that w = (0 , 1 βd , 2 βd , . . . , d -1 βd ) ∈ R d where β = d -1 2 . Fix α ∈ [0 , 1 d ) . Then ( L G φ , ψ P w ,α ) is ℓ 0 -1 -calibrated over Θ α .

Proof. Let ∆ d := conv ( { ✶ i ∈ R d | i ∈ [ d ] } ) and observe P w ⊂ ∆ d since for all π , ∥ π · w ∥ 1 = ∥ w ∥ 1 = 1 . Observe that P w can be symmetrically partitioned into d ! regions with disjoint interiors, one for each permutation π ∈ S d via ∆ π d := { u ∈ ∆ d | u 1 ≤ ·· · ≤ u d } . Fix π ∈ S d and w.l.o.g assume π is associated with the constraints ∆ π w := { u ∈ ∆ w | u 1 ≤ · · · ≤ u d } implying that π ( w ) = ( 0 βd , 1 βd , . . . , d -1 βd ) . Let α = 1 d and define Θ α . With respect to Θ α , let y := φ -1 ( π ( w )) ∈ Y and ˆ y := φ -1 (ˆ π ( w )) ∈ Y such that ˆ π ∈ S d . Thus the set Ψ y α := { (1 -1 d ) δ y +( 1 d ) δ ˆ y | ˆ y ∈ Y} is mapped via φ to the following points

within the permutahedron.

We shall show that P y α ⊆ ∆ π d . If this were not true, there would exists an element of w π, ˆ π ∈ φ (Ψ y α ) such such that for some pair of adjacent indices, say i, i +1 ∈ [ d -1] , w π, ˆ π i > w π, ˆ π i +1 . For sake of contradiction, fix i ∈ [ d -1] and assume there exists a ˆ π ∈ S d such that w π, ˆ π i > w π, ˆ π i +1 . Observe that

any element of ˆ π ( w ) can be expressed by j βd using some j ∈ { 0 , 1 , . . . , d -1 } . Thus,

̸

for some j, ˆ j ∈ { 0 , 1 , . . . , d -1 } where j = ˆ j .

Case 1 : ( j < ˆ j ): The smallest value possible for ˆ j -j is 0 -( d -1) however, 1 -d ≯ 1 -d .

Case 2 :( j > ˆ j ): The smallest value possible for ˆ j -j is 1 however, 1 -d ≯ 1 .

̸

Hence, P y α ⊆ ∆ π d and specifically, there can exists an extreme point of P y α that lies on the boundary of ∆ π d as shown in Case 1 . However, if α ∈ (0 , 1 d ) , every extreme point of P y α moves closer to π ( w ) (besides the extreme point itself already on π ( w ) ) and therefore P y α lies strictly within ∆ π d . By symmetry of P w and the linearity of φ , this would imply that for all y ′ , y ′′ ∈ Y such that y ′ = y ′′ it holds that P y ′ α ∩ P y ′′ α = ∅ . Thus by Corollary 6, ( L G φ , ψ P w ,α ) is ℓ 0 -1 -calibrated for Θ α where α ∈ (0 , 1 d ) .

B.4 Omitted Proofs from § 5

Lemma 9. Say we are given a cross-polytope embedding φ : ∆ 2 d → P ⊕ and induced loss L G φ . Let ( v a i , v b i ) , be the i th diagonal pair (i.e. φ ( δ a i ) = v a i ). Define the property Γ φ : ∆ 2 d → B element-wise by

Furthermore define the link ψ P ⊕ : R d →B with respect to each diagonal pair as

Then ( L G φ , ψ P ⊕ ) elicits Γ φ .

Proof. W.l.o.g, fix a diagonal pair ( v a , v b ) and let v a := ✶ 1 and v b := -✶ 1 . Define the embedding φ accordingly. We will show that the following is true for all distributions mapped via φ to u ∈ P ⊕ .

First, fix p ∈ ∆ 2 d . Recall, by Proposition 2, the minimizing report for L G φ in expectation is u = φ ( p ) ∈ P ⊂ R d . We will prove the forward direction of the first and second lines. Then the reverse directions follow from the contrapositives.

Case 1 , = ⇒ : Assume for contradiction that p < p and || φ ( p ) -v || < || φ ( p ) -v || . Then

By the definition of a d -cross polytope P ⊕ := conv ( { π (( ± 1 , 0 , . . . , 0)) | π ∈ S d } ) and the orthogonal relation between vertices, to express a u ∈ P ⊕ as a convex combination of vertices, each diagonal pair of vertices coefficients solely influence the position along a single unit basis vector. Hence, due to the definition of φ , we have u 1 = ✶ 1 · p a -✶ 1 · p b < 0 since we have assumed that p a < p b . Hence -u 1 < u 1 < 0 , a contradiction.

Case 2 , = ⇒ : Assume p a > p b and || φ ( p ) -v a || 2 < || φ ( p ) -v b || 2 . By symmetry with case 1, all the inequalities are reversed, leading to the contradiction that -u 1 > u 1 > 0 .

Case 3 : ( p = p

Hence ( L G φ , ψ φ ) elicits Γ φ

a b ): Follows from the if and only ifs of cases 1 and 2. .

Theorem 10. Let d ≥ 2 . The mode is (2 d, d, m ) -Polytope Elicitable for some m ∈ [2 d -1 , d (2 d -1)] .

Proof. We will elicit the mode via the intermediate properties, Γ φ j , defined in Lemma 9. First we construct a set of embeddings so that we guarantee that all the φ j 's allow comparison between any pair of outcome probabilities. For example, for each unique pair ( a, b ) j ∈ ( Y 2 ) define an embedding: φ j ( δ a ) = ✶ 1 and φ j ( δ b ) = -✶ 1 , and embed every other remaining report r ∈ Y \ { a, b } arbitrarily. Since ( L G φ , ψ P ⊕ ) elicits Γ φ , minimizing each L G φ j with a separate model yields us comparisons via the link ψ P ⊕ . To find the set r ∈ Y such that p r is maximum, we use a sorting algorithm that uses pairwise comparisons, such as bubble sort. Hence with Υ as Algorithm 1, we have that Υ( { L G φ j , ψ P ⊕ } ) = mode ( p ) .

Assuming there exist φ j s such that there is no redundancy in comparison pairs between each Γ φ j , we would need only d (2 d -1) d = 2 d -1 problem instances. Hence, we establish our lower bound on the needed number of problem instances.

C Hamming Loss Hallucination Example

Hamming loss ℓ : Y × Y → R + is defined by ℓ ( y, ˆ y ) = ∑ d i =1 ✶ y i =ˆ y i where Y = {-1 , 1 , } d . Suppose d = 3 and we have the following indexing over outcomes Y := { y 1 ≡ (1 , 1 , 1) , y 2 ≡ (1 , 1 , -1) , y 3 ≡ (1 , -1 , 1) , y 4 ≡ ( -1 , 1 , 1) , y 5 ≡ ( -1 , -1 , 1) , y 6 ≡ (1 , -1 , -1) , y 7 ≡ ( -1 , 1 , -1) , y 8 ≡ ( -1 , -1 , -1) } .

Let us define the following distribution

such that ϵ > 0 .

For all ϵ ∈ [0 , 1 12 ) , the minimizing report in expectation is y 1 = (1 , 1 , 1) . However, p ϵ, 1 = 0 and thus, a hallucination would occur under a calibrated surrogate and link pair.

̸

D Linking under Multiple Problem Instances

As stated in § 5, when using real data, given that these are asymptotic results, we may have conflicting logic for the provided individual reports. In this section, we provide an approach such that the algorithm still reports information in the aforementioned scenario and will reduce to Algorithm 1 asymptotically. We build a binary relation table M ∈ { 0 , 1 } n × n with the provided reports. Based on M , we select a largest subset of S ⊆ Y such that when M is restricted to rows and columns corresponding to the elements of S , denoted by M S , we have that M S is reflexive, antisymmetric, transitive, and strongly connected implying M S has a total-order relation defined over its elements. Having a total-order relation infers the mode can be found via comparisons. The algorithm returns ( R,S ) , where R is the mode set with respect to the elements of S .

Algorithm 2 Elicit mode via comparisons and the d-Cross Polytopes over well-defined partial orderings

Learn a model h j : X → R d for each instance ( L G φ j , ψ P ⊕ j ) ∈ M

For some fixed x ∈ X , collect all B j ← ψ P ⊕ ( h j ( x )) where B j ∈ B

Build M ∈ { 0 , 1 } n × n binary relation table with provided { B } as such j j j j m =1

Label rows top to bottom by y 1 , . . . , y n and columns left to right by y 1 , . . . , y n .

For all ( · , p y i , p y k ) ∈ B j , if p y i ≤ p y k set M [ i, k ] = 1 and 0 otherwise.

Select largest subset S ⊆ Y such that M S is reflexive, antisymmetric, transitive, and strongly connected.

Report ( R,S ) ← FindMaxElements-ofS ( M ; S )

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS paper checklist" ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Any claimed result in the abstract is proved within this work.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Yes, our paper discuss how these results are asymptotic.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Yes, we thoroughly introduce every necessary definition and past result necessary to understand the assumptions that hold under our results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The results of this work have rigorous proofs presented next to the results or referenced clearly in the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification:This paper does not include experiments requiring code.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: The paper does not include experiments

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: None of our conducted work for this paper violates the code of ethics presented. Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: In the body of our paper, we provide a broader impace section.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The answer NA means that the paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: The paper does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: the paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies surrogate loss design and the trade-off between surrogate consistency and loss dimension. The contributions are three-fold: (1) the characterization of the hallucination region, where the decoded prediction from the surrogate loss minimizer gives a class with no target probability mass, indicating a completely \"irrational\" prediction; (2) the construction of the (\"weakly but reasonably\" inconsistent) calibrated surrogate and link under the low-noise setup; (3) the decomposition of property eliciation into multiple elicitation problems with low dimensions.\n\nSTRENGTHS:\n+ **Very well-motivated problem**: Consistent/calibrated surrogate losses, the main topic of this paper, are sometimes too restrictive because the (in)consistency analysis hinges on \"far-fetched\" distributions---as argued by the authors, and as can be seen in some counterexamples to show the inconsistency of the well-known Crammer-Singer hinge loss. If we can remove such \"pathological\" situations from the entire distribution space, we would have a nicer characterization of loss functions. This is the central motivation of this paper.\n+ **Interesting instances to show calibrated surrogates under the low-noise condition**: This paper mainly investigates two situations to convince the benefits of the relaxed notion of calibration: the unit cube and permutahedron. Both make sense with reasonably practical scenarios and provide the first attempt to show the benefits of incorporating the knowledge of the noise level in loss function design.\n\nWEAKNESSES:\n+ **The result with the unit cube may need $d \\\\geq n$**: At first sight, the statement of Corollary 7 (the calibrated link design for the unit cube) does not have any restriction on $\\\\alpha$ (say, any dependency on the embedding $d$), unlike Corollary 8 (the calibrated link design for the permutahedron). When I look at its proof, I suspect we need the condition $d \\\\geq n$ for this because the proof leverages that \"$P\\_\\\\alpha^y$ is a strict subset of the orthant that contains $v\\_y$\" at l.500. Otherwise, it is strange because we can choose an arbitrarily small embedding dimension $d$.\n+ **The trade-off could be seen only for the unit cube case**: The main contribution of this paper is to showcase the trade-off between the consistency and the embedding dimension, as suggested by the title. However, we may not see a clear trade-off in the unit-cube case. This is also related to the above point: Once we have $d \\\\geq n$, there may not be a clear trade-off for $\\\\alpha$ and $d$. Given this, we are not very sure how universally we can observe the trade-off across different elicitation problems.\n\nNevertheless, I don't think these points are significant enough to undermine the contributions of this paper. I expect the authors to address them, which leads to the presentation of the contributions in a more fair/precise manner.\n\nQUESTIONS:\nHere are the major questions.\n+ In the proof sketch of Theorem 3, it is not very straightforward to see $\\\\mathrm{conv}(\\\\mathrm{vert}(P\\_{-y})) \\\\cap \\\\psi\\_y^\\\\varphi \\\\ne \\\\varnothing$. This is important to ensure that $\\\\mathcal{H} \\\\ne \\\\varnothing$ (before invoking Helly's theorem). Even if this is merely a sketch, it is better to discuss it in my opinion.\n+ Figure 1 (especially the left one) is challenging to understand. Although I can guess that each vertex of the rectangle corresponds to $v\\_y$, I cannot see what each area \"ad\", \"cd\", etc. mean exactly. Either the figure or caption should be refined. Figure 1R (and its caption) is slightly more digestible.\n+ In Theorem 5, the constructed link contains the \"point-set\" distance $\\\\|u - P\\_\\\\alpha^y\\\\|\\_2$. What does it mean exactly? It appears several times afterward (for example, in Theorem 6). I guess we can recover the statements without significant modifications if we modify the link definition slightly.\n\n----\nSome minor points:\n\n- In the last paragraph of the introduction, it is a bit too sudden to talk about the mode for unfamiliar readers. You may say a few words about the relationship between the mode and multiclass classification.\n- In Definition 1, the very general report space is introduced, which I'm not sure is really necessary or not. Subsequently, the discussion is only based on the case $\\\\mathcal{R} = \\\\mathbb{R}^d$.\n  - By the way, the domain of a surrogate loss $L$ should be $\\\\mathcal{R} \\\\times \\\\mathcal{Y}$ in Definition 1, not $\\\\mathcal{Y} \\\\times \\\\mathcal{Y}$.\n- Throughout the paper, \"mode\", \"prop\", and \"vert\" look better with \\\\mathrm, not with \\\\text.\n- In Construction 1, $v\\_y$ is not formally introduced (while I could catch its meaning). In my opinion, this should be formally introduced because it appears repeatedly in the paper.\n- At l.171, \"To show that ...\" does not form a grammatically complete sentence.\n- At l.174, what does it mean by \"the vertex figure\"?\n- At l.194--196, the sentence is not easy to understand mainly because it contains two \"if.\" Could you rephrase it without multiple \"if\"?\n- At l.201, the level set $\\\\psi\\_y$ for the general link is not defined as far as I see, even though the level set for $\\\\psi^\\\\varphi$ is defined in Definition 7.\n- At l.201, $R\\_\\\\mathcal{Y}$ probably matches $R$. Is it true? I'm asking out of curiosity.\n- At l.209, what does it mean by \"a robust sense\"?\n- At l.213, the last \"via $(L,\\\\psi)$ with respect to $\\\\ell_{0-1}$\" is relevant to the definition of the strict calibration region $R\\_\\\\mathcal{Y}$. However, if you append \"via [...]\" in this way directly after $R\\_\\\\mathcal{Y}$, it is a bit hard to understand at first sight.\n- At l.229, $\\\\delta\\_y$ is not defined (though I catch the meaning).\n- At l.233, the notation $\\\\psi\\_\\\\alpha^\\\\varphi$ should be changed if possible because this notation with the subscript/superscript is confusing with the level set notation $\\\\psi\\_y^\\\\varphi$.\n- At l.250, $=$ (in the beginning) should be $\\\\subseteq$ in my understanding.\n- At l.251 and l.267, the link notation $\\\\psi\\_\\\\alpha^P$ should be $\\\\psi\\_\\\\alpha^\\\\varphi$. In regard to this, the notation of $\\\\psi\\_\\\\alpha^{P^\\\\square}$ and $\\\\psi\\_\\\\alpha^{P^w}$ in Section 4.2 are better to be consistent with $\\\\psi\\_\\\\alpha^\\\\varphi$ if possible---the superscript stands for the embedding function, not the embedded space.\n- At l.269, \"$P\\_\\\\alpha^y$ is [...] pairwise disjoint\" should be complemented with \"in $y$\" to be clearer.\n- At l.274, the strict properness is not introduced. Actually, I think the usual calibration suffices in this context.\n- In the appendix, can you make the theorem numbers consistent with those used in the main part? You can do this by using the restatable package.\n\nLIMITATIONS:\nThis work has discussed the limitations. Specifically, there is computational hardness to break down an elicitation problem into multiple low-dimensional problems in Section 5.\n\nThis work does not have potential negative societal impacts because the contributions of this paper are relevant only to general machine learning theory.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-06-13"}

comments: SUMMARY:
This paper proposes a method called polytope embedding, which embeds multiclass predictions onto real numbers. The paper studies the properties of this embedding, like hallucination and calibration. Further, with low-noise assumptions, the authors showed more calibration results for their embedding in some cases like embedding into the unit cube.

STRENGTHS:
The topic of this paper on how to design a consistent surrogate loss seems interesting. The results proved in the paper also relate to topics that people care about like hallucination and calibration.

WEAKNESSES:
Even though each result seems interesting, the structure of this paper is not very clear to me. I am open to different arguments but I think it is hard for readers to understand what the authors are trying to show with this paper.

QUESTIONS:
I am wondering what exactly the authors' definition of partial consistency is. How exactly are the methods trading off consistency and dimensionality? If the authors mean consistency by hallucination and calibration, which theorem shows how dimension affects these properties.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper examines the trade-off between consistency and dimensionality in multi-class classification. It has been known that the lower bound on the dimension for consistent surrogate losses under any distribution is $n - 1$, wheren $n$ is the dimension of the input space. The authors propose the notion of partial consistency, which permits the establishment of surrogate losses at much lower dimensions than the input dimension. This method allows for the consistency of lower-dimensional surrogate losses under a low-noise condition and the construction of dimensionally reduced consistent surrogate losses across multiple problem instances.\n\nSTRENGTHS:\n1. The paper is the first study to explore the control of the trade-off between consistency and dimension in multi-class classification.\n\n2. The paper is well-written and clear, even though it is theoretically dense.\n\nWEAKNESSES:\n1. The paper demonstrates the existence of distributions under which consistency holds for low-dimensional surrogates but does not offer much guidance on how to identify such distributions for a given surrogate loss.\n\n2. The paper restricts its study and analysis to asymptotic guarantees.\n\nQUESTIONS:\n1. Theorem 5 establishes the existence of an $\\alpha \\in [0, 0.5)$ across distributions where consistency is maintained. Is it possible to determine or estimate the value of $\\alpha$ for a specific surrogate loss to guarantee consistency?\n\n2. The max hinge loss, formulated by Crammer and Singer (2001), is inconsistent when $\\alpha = 0$ and consistent when $\\alpha = 0.5$. Do the findings of this paper potentially include such a result as a particular example?\n\n3. The paper examines the asymptotic guarantees of Bayes consistency. Could some of these results be extended to incorporate non-asymptotic consistency guarantees, such as excess error bounds or H-consistency bounds?\n\n4. The low-noise condition mentioned in Line 228 can be considered as the multi-class version of Massart's noise condition. Is it possible to extend the analysis to encompass the broader Tsybakov noise condition?\n\n5. Could the authors provide further elaboration on how to choose an embedding in practice, based on the theoretical results?\n\n6. The results in the paper apply to multi-class classification. Is there any possibility of generalizing them to other scenarios, such as ranking?\n\n7. The use of $\\mathcal{H}$ to denote the hallucination region in line 164 could lead to confusion, as $\\mathcal{H}$ is commonly associated with the hypothesis set. Is there an alternative notation that could be used to avoid this ambiguity?\n\n8. The notion of partial consistency is mentioned in both the abstract and introduction, yet it is not formally defined in the main text. It seems that partial consistency suggests that surrogate reports may not correspond to a single distribution. A formal definition would likely enhance reader comprehension.\n\n9. The approach of using multiple problem instances and aggregating their outputs is intriguing. However, might this method be vulnerable to data imbalances across different problem instances? I am eager to hear the authors' opinions on this.\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}

comments: SUMMARY:
In this paper, the problem of constructing consistent multiclass surrogate losses for the 0-1 loss while reducing the dimension of the scoring function is studied. The concept of partial consistency, which can be dated back to the study of multiclass SVM, is used as a crucial part of this work. It is first revealed that any losses with scoring function’s dimension less than #class number-1 will lead to severe misclassification. Then low noise assumption is used to enable the trade off between (partial) consistency and the scoring function’s dimension number. An attempt of recovering full consistency with several scoring functions of lower dimensions is also made.

STRENGTHS:
1. A clear trade off between the strictness of low-noise assumption and scoring function’s dimension number is quantified.

2. The analysis of hallucinations is also enlightening, which provides an interesting insight of why some well-suited models (with some losses) can make wrong predictions even with no real-world evidence.

3. The clear presentation and layout of the results make them easy for readers to understand.

WEAKNESSES:
While the dimension of predictor is reduced and thus the computational cost of training can be promisingly reduced, the computational cost of the inference stage may increase compared with that of the traditional #class number-dimensional scoring functions. Is there any remedy for this problem?

QUESTIONS:
Please see the weaknesses.

LIMITATIONS:
Please see the weaknesses.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-29