Moment Distributionally Robust Probabilistic Supervised Learning
Authors: Yeshu Li
Brian D Ziebart
Venue: Submitted to ICLR 2023 (2023)
Keywords: probabilistic supervised learning
distributionally robust optimization
proper scoring rules

Abstract: Probabilistic supervised learning assumes the groundtruth itself is a distribution instead of a single label, as in classic settings. Common approaches learn with a proper composite loss and obtain probability estimates via an invertible link function. Typical links such as the softmax yield restrictive and problematic uncertainty certificates. In this paper, we propose to make direct prediction of conditional label distributions from first principles in distributionally robust optimization based on an ambiguity set defined by feature moment divergence. We derive its generalization bounds under mild assumptions. We illustrate how to manipulate penalties for underestimation and overestimation. Our method can be easily incorporated into neural networks for end-to-end representation learning. Experimental results on datasets with probabilistic labels illustrate the flexibility, effectiveness, and efficiency of this learning paradigm.

Under review as a conference paper at ICLR 2023
MOMENT DISTRIBUTIONALLY ROBUST PROBABILISTIC SUPERVISED LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Probabilistic supervised learning assumes the groundtruth itself is distribution instead of a single label, as in classic settings. Common approaches learn with a proper composite loss and obtain probability estimates via an invertible link function. Typical links such as the softmax yield restrictive and problematic un- certainty certificates. In this paper, we propose to make direct prediction of condi- tional label distributions from first principles in distributionally robust optimiza- tion based on an ambiguity set defined by feature moment divergence. We derive its generalization bounds under mild assumptions. We illustrate how to manipulate penalties for underestimation and overestimation. Our method can be easily incor- porated into neural networks for end-to-end representation learning. Experimental results on datasets with probabilistic labels illustrate the flexibility, effectiveness, and efficiency of this learning paradigm.
INTRODUCTION
The goal of classical supervised learning is point bel domain given features-usuall) without justifying the confidence. The outcome distribution of an event can be inherently uncertain and more desirable than point predictions in some scenarios. For example, weather predictions that express the uncertainty of events such as rain occurring are more sensible than binary-valued predictions, while uniform distribution prediction for the outcome of a fair dice roll is more sensible than speculating an integral value randomly. On one hand, the pre- dicted distribution quantifies label uncertainty and is thus more informative than point prediction, which is widely studied in weakly supervised learning (Yoshida et al. 2021). boosting (Friedman al. 2000) and optimal treatment (Leibovici et al. 2000). On the other hand, the ground truth naturally comes with multiple targets, possibly with different importances. For instance, there can be multiple emotions in a human face image, there are different gene expression levels over a period of time in biological experiments, and many annotators might disagree over a highly ambiguous instance. the above settings, each predefined label is part of the ground truth as long as it has a positive probability in the true distribution. Hence, it is natural to probabilistic labels in both training and inference when the ground truth is no longer point. In the literature, the task of pre- dicting full distributions from features is called probabilistic supervised learning (Gressmann al., 2018).
single target from the la-
A probabilistic supervised learning task comes with probabilistic loss functional quantitatively measuring the utility of the prediction (Bickel 2007). Williamson et al. (2016) propose a com- posite multiclass loss that separates properness and convexity. They illuminate the connection be- tween classification calibration (Tewari & Bartleu 2007) and properness (Gneiting & Raftery [007]] Dawid 2007). representing Fisher consisteney for classification and probability estimation respec- tively. A proper loss is minimized when predictions match the true underlying probability, which implies classification calibration, but not vice versa. Among proper losses, the logarithmic loss (Good 1952) severely penalizes underestimation of rare outcomes and assessing the "surprise" of the predictor an information-theoredt sense, the Brier score score-originally proposed for evaluat- ing weather forecasts (Brier 1950) useful for assessing prediction calibration, and the spherical scoring rule (Bickel, (007)) is used when a distribution with lower entropy is desired. A single proper loss is sometimes not sufficient for scenarios that elicit optimistic pessimistic predictions for decision making with practical concerns (Elsberry 2002) Chapman 2012). For example, under- estimating disastrous events may provide very low utility, motivating more pessimistic predictions.

---

Under review as a conference paper at ICLR 2023
Therefore it is desirable for proper loss to be flexible in its penalties for deviated predictions that combine statistical properties of multiple losses.
Deep neural networks typically adopt the softmax function to predict a legal distribution. However, softmax intentionally renormalizes the logits and therefore assumes that it follows a logistic distri- bution (Bendale & Boult 2016). It is poor at calibration, uncertainty quantification and robustness against overfitting (Joo et al. 2020). The inverse the canonical link function in Williamson et al. (2016) can be used to recover probabilities but commonly resembles softmax (Zou al., 20088.
In this paper, we propose a probabilistic supervised learning method from first principles in dis- tributionally robust optimization (DRO) for general proper losses that realize desired prediction properties. Instead of specifying parametric distribution, it starts with a minimax learning problem which the predictor non-parametrically minimizes the the most adverse risk among all distribu- tions in an ambiguity set defined by empirical feature moments. The ambiguity set represents our uncertainty about the underlying distribution. By strong duality, we show that the primal DRO prob- lem is equivalent to regularized empirical risk minimization (ERM) problem. The regularization results naturally from the ambiguity set instead of being explicitly imposed. The ERM form also allows us to derive generalization bounds and make inferences from unseen data. We illustrate set of solutions for general proper losses satisfying certain mild conditions and an efficient algorithm for weighted sum of two common strictly proper losses. We conduct experiments real-world datasets by adapting our method to end-to-end differentiable learning. We defer all technical proofs to the appendix.
Contributions. Our contributions are summarized as follows. (1) We propose a distributionally robust probabilistic supervised learning method. (2) We characterize the solutions the proposed method and present an efficient algorithm for specific losses. (3) We incorporate our method into neural networks and perform extensive empirical study on real-world data.
1.1 RELATED WORK
Model assessment of probabilistic models via predictive likelihood has been studied in Bayesian
models (Gelman et al., 2014). probabilistic forecasting (Gneiting & Raftery| (007). machine learn- ing (Masnadi-Shire & Vasconcelos 2009), conditional density estimation (Sugiyama al. 2010),
information theory (Reid & Williamson 12011) and representation learning (Dubois et 2000). A
comprehensive framework for probabilistic supervised learning can be found in Gressmann et al. 20188.
Techniques developed explicitly tackle multiclass probabilistic classification include multiclass logistic regression (Collins et al. 2002), support vector machines (Lyu et al.) 20019; Wang et al., 2019, learning from noisy labels (Zhang et al. 2021), weakly supervised learning (Yoshida al.
2021. and neural networks (Papadopoulos) 2013 Gast & Roth, 2018). Multilabel classification, aimed at predicting multiple classes with equal importance, has been analyzed by Cheng et al. (2010) and Geng (2016) in a general probabilistic setting. Note that confidence calibration (Guo ct al. 2017) has different objective from probabilistic supervised learning.
Fisher consistency results have been established for classification losses (Tewari & Bartlett 2007 structured losses (Ciliberto et al., 2016) Nowak et al. 2020), proper losses (Williamson et al. 2016)
and Munchel-YYung losses (Blondel et al. 2020).
The emerging field of DRO has led to learning methods with ambiguity sets defined by feature moments (Farnia & Tse 2016; Mazuelas al. 2020), ø-divergence (Duchi & Namkoong 2019)
and the Wasserstein distance (Shaficezadeh- Abadeh et al. 2019). The moment-basse ambiguity set
adopted in this work originates from maximum entropy (Cortes et al.] 2015)) Mazuelas et al. 2022
with similar work studying classification (Asif et al. 2015 Fathony et al. 2016) and structured
prediction (Fathony et al. 2018a b).
2

---

Under review as a conference paper at ICLR 2023
2 PRELIMINARIES
2.1 NOTATIONS
We adopt the following notations by convention. A bold letter æ denotes vector whereas a normal letter æ represents scalar. 2; or (æ); stands for the i-th coordinate of æ. We denote random variables with capitalization (e.g, X or X) and sets with calligraphic capitalization (e.g, X, A). We | means the absolute value of scalar or the cardinality of set, depending on the context. The p norm of vector is defined as ||
denote by [n] the set (1.2....,
(E, 1//p The indicator function of a subset S of a set X' is a mapping Ls X - {0, 1 such that 1s(x) =
S and ls)( = 0 otherwise. 1(-) is adopted for events that ((S) = 1 if event S occurs and I(S) = 0 otherwise. We write 82 as the Dirac point measure at 2 E 2. A probability simplex of (d -iddieensila vectors is represented as A whose superscript is omitted when the context is clear. We denote by P(2) the set of all probability distributions on a set 2.
2.2 PROBABILISTIC LOsS FUNCTIONALS
A loss function measures the quality of a pre- diction associated with an event. Scoring rules are widely adopted to assess probabilistic pre- dictions, but can be naturally translated to loss functions by appropriate negation and normal- ization. To illustrate some examples, sider a decision problem which y E y is an outcome and Py E P()) is predicted distribution over y. We denote by py (Py (PY())) a vector of probabilities.
0-1
log
Brier
1.2
spherical
expected loss
con-
1.0-
0.8-
0.6-
The zero-one loss is defined for determinis- tic prediction so that penalty of 1 is in- curred whenever y' and y differ: foi (v' y) I(y' A y). It extends probabilistic predic- tions as
0.1
0.0
[y(1)
Figure 1: The expected value of four loss func- tions for three classes with Q) Qy (2) = (3) = 0.2. (2) (3 varies. Each loss is normalized to cross (1, 0) and (0.5, 0.5) according to the binary case with a hard label. Best viewed in color.
0.6 and
& 1- Py (y The cost- sensitive loss for multiclass classification is similarly defined with confusion cost matrix ()) EvEYPY (i)C C E R x1
() as (1)
The multiclass Brier loss, based on the Brier score or quadratic scoring rule, measures the mean squared difference between Py and y: (P) Ey (PP I1(y' = =²))a
The logarithmic loss, also called log- log-likelihood loss, incurs a rapidly increasing penalty as the predicted probability the target event approaches zero: ( (.Py,1) y) _ In Py (v)-
The spherical scoring rule can be interpreted the spherical projection of the true belief onto the prediction vector. To use it as a loss function, define Esp (Py
(v)) /yr/2
For ease exposition, we define L(P, Q) = Σ,, Qv(y))(Py,)) 4) where E(:,)) P()) x x R.
a probabilistic loss function as illustrated above. A loss is called proper if TLQ Q) < L(P Q)
for all IP, Q. and called strictly proper if Q is the unique minimizer of T.(, Q). Figure provides a graphical comparison of the above losses for prediction with three classes. We can inter that the zero-one loss an improper loss.
2.3 PROBABILISTIC SUPERVISED LEARNING
We study the probabilistic supervised learning task where we are given training samples (xx(1, (2)), (x(2), (2)), ..., (x("), (())),g())) (')) drawn i.id. from distribution IP on the joint space
In the literature, the zero-one loss is sometimes defined as for (Py, v) = T(y & arg max, Pr(v))), which proper, but discontinuous and not strictly proper.
3

---

Under review as a conference paper at ICLR 2023
X' 3. in which X' is feature space and y/ is univariate finite discrete label space. A proba- bilistic multiclass loss function L P(() x P())) + R+ is given. The goal of ERM is to learn from the samples a mapping h X' P()) to minimize the empirical L-risk of h:
h E: arg min R (h) == pero P(Ix)) MCH
(I)
where pemp represents the empirical distribution and is a hypothesis space. Here we assume may be accompanied with probabilistic label by aggregating instances with the same (f), In this way, both learning and inference are accomplished in the general setting subsuming classical supervised learning.
3 METHOD
We now present our formulation for learning with general multiclass probabilistic losses. We provide theoretical results of consistency and generalization. We study the solution for general proper losses in our formulation and develop efficient algorithm for two typical proper losses.
3.1 FORMULATION
We consider a continuous proper loss L to be optimized under the unknown distribution paree We assume that class-sensitiv feature function X' y Rd that maps data point to d- dimensional feature vector is given. Examples include the multi-vector representation and class- dependent TF-IDF scores. Choosing good &f is representation learning problem, but discuss in Section 3.4] it is not a concern once our method is incorporated into neural networks as a layer. Intuitively, the elements of the vector Φ(x, y) can be regarded as scores indicating how well the label matches with the feature x. For example, with linear hypothesis hw (x. y) = (w, (xx, y)). a good parameter vector w* should yield
will
(w", ((x v)) (w", ((xxx'xx))
P(x, 8) > P(x, v)).
Instead of specifying parametric form of predictions, adopt minimax statistical learning formulation:
min max Eex [L. (Prix: Qrix)],
PriXEP()) QEA(P
where A(Pemp) := (Q Q P(X V) A PTT Qx A Epenp (o( Bg ( < E). The ambiguity set is different from that in Wiesemann al. (2014) and Farnia & Tse (2016) due to the inequality and feature mapping respectively. The minimization over the function space is re- placed by directly minimizing over P())) for each x E X. The probabilistic predictions chosen to minimize the worst-case risk evaluated on a set of distributions in an ambiguity set defined by the empirical distribution pemp and feature mapping Φb The ambiguity set ([Pemp) includes distribu- tions that share the same marginal on X' and are no more than away from Tpemp in terms of feature moment divergence. Note that given any feature function Φ, the ambiguity set is compact convex set. Conceptually, we restrict the support of Q on X' to be the same as the empirical distribution for convenience in both algorithm design and theoretical analysis.
Minimizing the worst-case risk by allowing a certain amount of label uncertainty makes this method inherently robust. It can also be shown to be equivalent dual dual-norm regularized ERM problem:
Proposition 1 (Lii et al. 2.22)). The distributionally robust probabilistic supervised learning prob- lem based on moment divergence in Eq. (2) can be rewritten as
min Egpar min max (FYIx, Qvix) + OT QT ((X,Y) Exporo $(X. +
(3)
8
B
Q
xix
Lan L.. 10, x
where O E PRP the vector of Lagrangian multipliers and [|-I1- is the dual norm of [|-I]-
We give a proof sketch here. Both P(V) and A(P) are non-empty closed convex sets. Since we assume I. is continuous and proper, know that L(:, @ is quasi-convex for every Q and I.(P.)) is
4

---

Under review as a conference paper at ICLR 2023
concave for every P by definition. Eq. (2) is therefore quasi quasi-comvex-conaayy problem and strong duality holds (Sion 1958). The regularization is obtained via Lagrangian and Fenchel conjugate.
is well-known that continuous proper losses quasi-convex, such the Brier score, the logarith- mic score, the spherical score, the Winkler's ' score, the ranked probability score, etc. However, some improper (possibly discrete and non-convex) losses can be quasi-convex in the predicted distribution (e.g., the zero-one loss). contrast, surrogate classification losses are usually convex in a parameter space that easy work with, for example, the multiclass hinge loss Weston & Watkins (1998).
Lwwl (1b, y) Eg' zy max{0, (0, 1 + By Vy.. and the multiclass logistic loss (Nelder & Wedderburn bug (1, ( y)
1972
EEy exp (4hyr thy. where 2b D is vector class scores.
From a game theoretic point of view, formulation in Eq. (2) is equivalent two-player zero- sum game in which the predictor player chooses a distribution to minimize the expected game payoff' while the adversary player chooses one to maximize the game value while constrained satisfy certain statistical properties of training data (Grünwald et al. 2004). In the dual problem (Eq. (3).
the Lagrange multipliers parameterize the payoff function for an augmented game and provide new payoff function for unseen data to construct predictors.
3.2 STATISTICAL PROPERTIES
well known that minimizing strictly proper losses leads to Fisher consistent probability estimation (Williamson al. 2016). However, minimization of the surrogate risk in Eq. (3) may induce sub-optimal classifier because of misalignment between the surrogate loss Laav and the original loss L. Fisher consistency provides desirable statistical implications for a surrogate loss such that minimizing it yields an estimator that also minimizes the original loss.
The adversarial surrogate loss Lady endowed with an additional regularization term. It reduces to a Fenchel- Young loss (Blondel et al., (2020) when the ambiguity radius is zero. A conclusion of consistency can drawn based on Nowak et al. 22000; Blondel et (2020) and our assumption that the groundtruth is probabilistic:
Corollary (QLi et al. 2022 When = O, Ludd is Fisher consistent with respect to L. Namely, for any x,
P Pive Y|a
arg min L(P
Fy
the Bayes optimal probabilistic prediction made by Oinee the solution in Eq. 3 under pire, The prediction made by O is pe y E arg minp maxq L (Pylx. Qyix) + Exy9y OT OT (X)).
The consistency result guarantees that the learned probabilistic prediction rules yield Bayes opti- mal risk ERM with proper losses in the ideal setting with true distributions and all measurable functions. Also note that the conclusion holds for all quasi-convex losses.
Basic generalization bounds related to true risk for DRO methods can be derived from measure centration. This approach depends on the choice of ambiguity sets and may have a dimensionality issue. It is also not appropriate for ambiguity Thus, we take an alternate approach following Farnia & Tse (2016) to prove excess out-of-sample risk bounds. We assume by making mild assumptions on boundedness on features and losses:
defined by low-order moments in this paper.
to ensure boundedness of 0 *. We establish the following theorem
Theorem (LLie et al. 2022)). Given n. samples, a non-negative multiclass probabilistic loss I.::.) such that [I(..'] K, feature function ((::-) such that [|<(;--)) < B and a positive ambiguity
level e > 0, then, for any O < < 1, with a probability at least 6, the following excess true worst-case risk bound holds:
4KB
In(4/6)
max (E
12500mm)-
max (EA((
(0
Evn
1
2V
2
(4)
where Ozmp and Oiee are the optimal parameters learned in Eq. tion pomp and true distribution ppruue respectively. The original risk of 0 under Q is 12((0) =
3 under the empirical distribu-
Egx,y po L(Py X, Qylx).
5

---

Under review as a conference paper at ICLR 2023
Theorem improves the results Asif et al. 2015)) and Fathony et al. (2016) that only show quali- tative bounds. Under positive regularization, this bound explains the rate of uniform convergence of the true worst-case risk of the estimator Emp learned through the empirical distribution peop to the true worst-case risk of the ideal estimator 0 learned under parce, Although the empirical estimator is obtained based on a finite set of samples, Theorem justifies the roles which the ambiguity set A(-), the feature function -),,), the loss function -(;-)) and the ambiguity parameter play in upper bounding the excess out-of-samppl worst-case risk. Intuitively, a larger will reject more hy- potheses that are sensitive with larger dual norms, whereas the worst-case risk scales with the range
of loss and feature functions.
3.3 ALGORITHM
Since L(.--) is a continuous quasiconvex-concave function, saddle point in Eq. 3 given 0 must have zero derivative with respect to P and Q:
Σ Cria(v))0((P u) /0P y (1/) + Z =0
(5)
((Pyla 0T (f u) + Zizvi= 0,
(6)
where ZevI= is the Lagrange multipliers for the simplex constraint Σ, (o) 1, similarly for Z Note that Z is constant for all y given æc If is local, e.g., E(P is independent of (1/) for y/ * y and if ((-,)) )) is monotone in y (ÿ) > 0 (without simplex constraints) with range R. which is the case for the logarithmic loss, Eq. (6) always has solution and the system of
equations for all along with the simplex constraint Σ, ( has unique solution. With few
assumptions on the boundedness of and OT Eq. (6) is ill-posed. Given Py from Eq. (6). the
solution 1 to Eq. (5) exists iff
dE(Py 1) (1)
1
06(P 1)/0Pr1
ae(Py)=: 1'1)/01P1
1 0
singular. By assuming locality and positiveness, there exists a unique solution Q2 One benefit of the proposed method that users only need focus on solve Eq. 6 and Eq. (5) for proper losses while Williamson et al. (2016) additionally require canonical link function for convexity.
Next we show how the system of equations can always be solved with specific losses. We consider an additive combination of the multiclass Brier loss and the logarithmic loss, both of which are continuous strictly proper losses. As indicated by Figure [ these losses differ primarily it goes to zero and
how The Brier
they penalize the ground truth label's prediction probability loss exhibits quadratic growth. The logarithmic loss has a vertical asymptote for labels considered increasingly unlikely the point of impossibility by the predictor. They have different penalties for underestimation and overestimation of the desired prediction. A trade-off between the log loss and the Brier loss thus provides flexibility to control the cost for misalignment between the prediction and the observation. See appendix for discussion on including the ranked probability score and other specific losses.
We employ this kind of loss in our DRO method and present an efficient algorithm that can be implemented in practice. With only slight loss of generality and for computational consideration, we assume a fixed positive weight on the log loss. To begin with, the mixture loss is
Emix (PP v) In Py + 3(1 (v) + Σ P2 (1))))
with derivative
Ommi a / = -1/Pr 2/3 2,TP
Scalar 3 weights the contribution of the Brier loss, to this additive combination, controlling the sensitivity of the predictor to underestimation. The adversarial surrogate of this mixture loss is

---

Under review as a conference paper at ICLR 2023
Fisher consistent as direct corollary. Methods that solely mix the predictions of classifiers designed for logarithmic loss minimization and Brier loss optimization, may be appealing for their simplicity, but demonstrably sub-optimal. For example, with the logistic loss, logistic regression provides a natural parametric form for the predictor, that equates loss minimization with data likelihood maximization.
Although the Brier loss is not local, the additional sum of quadratic terms Eg Py (v)) ) is constant across all y. Therefore Eq. 66) has closed form expression in terms of the Lambert W function. Furthermore, the sum over y for all (y) will cancel out, leaving terms only dependent on the
same y. So Eq. (5) is simplified into an expression of Q in terms of IP. Normalizing Q solves Zr, yielding the following proposition:
Proposition The DRO method for a probabilistic loss based on logarithmic loss, and Brier loss has solution y for the predictor parameterized by 0 defined by the following systems of equations:
VX E EXR Pr|x(v) = = + 0 ((x - W2 4(x.))))
(7)
where C constant dependent O and but independent of y. W() the principal branch of the Lambert W function. The corresponding adversary Y2yxx is defined as
2,5Tp
(1)
Z
P (v)
1 Ey 28P (v)/ 2,5Tp y (v))
(1) vix
and Zp
1+ 231
1 (H)
Ey Py p (1) (1))
Now show how solve Eq. 7 with simplex constraints to obtain Te Yix given for any
Algorithm Distributionally robust learning for probabilis- tic supervised learning with mixture of logistic and Brier losses
X. Let C = ft(t) = 0T "xx(x v) Int - 23t be function of = P (v). By definition, ʃ(.) is a monotoni- cally decreasing function with do- main R.- and range R. Its in- verse mapping f (-) monotoni- cally decreasing with domain IR and range + Therefore, let gC)) E, (C) = E, ( accord- ing to the intermediate value theo- rem, there exists C e R such that g( ()) 1. Because of their monotonicity, we can find C and (-) as solution to Equation
Input: d, 1x,Y, 3, learning rate y
Output:
Initialize 0 to be random vector
repeat
for all x E X do
C, C, (1xx <- Bisection(x, () by Compute Q (+1x) by Eq.
end for Compute 0L. /00 by (
0 t 0 7 /00 until convergence
7 via bisection method. Once Prix is obtained, we can find Qy simply by substitution. After that, the sub aub-gradient,
/ 00 A Expnn (Bay (ø(X, Y)]
Epen [Ø(X, Y)]) + 101,/000
can be leveraged to optimize The above steps are summarized in Algorithm |
3.4 DIFFERENTIABLE LEARNING
By taking advantage of deep neural networks, our method will be able to jointly optimize data representation and the Lagrange multipliers:
min Eppo Lmai (0, pemp f
0.6
enjoying the benefits end-to-end representation learning without manually looking for a good feature mapping Φb More off sfe-tff mini-hatch training tools could be leveraged as well.
7

---

Under review as a conference paper at ICLR 2023
We show how to make of DRO method loss layer in neural network training. A net- work for supervised learning typically has a linear classification layer in the end without activation. Assume the penultimate layer outputs <(x)) the last layer will output a (D1m-iimnsionnl vector (0(D")) TT)z))) This is essentially equivalent to adopting multivector representation to construct Φ. Specifically, given x E Rd and y E [[>]]] the resulting feature vector v = Φ(x. y) E R4y1 satisfies Uyd-d+i = z; for E [d] and vj = 0 otherwise. Therefore taking (ax) as the input is sufficient for to compute yy and L In this way, our method is the loss layer without learnable parameters, which backpropagates the sub-derivative of loss with respect to <b(x)) to the linear classification layer:
<b(x)) = ((((l))T((()))))))))))))))))))
E Suor 44
Pyix) E @ (a))))))
emp
Recall and pemp are the probability vectors for Q and Ipemp, The sub-gradient with respect to is added to the classification layer.
EXPERIMENTS
In the experiments, we consider as the performance measure the L-risk (h), also called the expected generalization loss. The mixture loss fiix of the log loss and Brier loss is adopted. The normalized generalization loss [1-3] Re (h) is estimated based on the test set distribution Pxx. X,Y
We compare our adversarial learning approach against neural network models with the softmax and the spherical softmax function as the final normalization layer (Laha al., 2018). All the baseline methods are able to make use of probabilistic labels in both training and testing. We adopt three- layer neural network for all the methods, who share the same number of parameters. To make a more fair comparison, baselines compute the target loss Lmix based on their probability outputs applied to the logits.
set
0 such that the final classification layer is unregularized. The
We implement all the methods using PyTorch (Paszke al. 2019). We use Adam (Kingma & Ba 2014) for optimization. The number of hidden units is set to 50. The number of training steps set to 500 with batch size of 64. We set 3 = 1. Default values used for unmentioned hyperparameters.
We conduct experiments several real-world datasets, including core15k (uygulu al. 2002), flags (Gonçalves et al. 2013), Stackex chess (Charte et al. 2015). Gposit i &ive and Gnegat Seattveeeeeseee (Xu et al.) 2016), having statistics reported in Table The ground truth labels in these dataset are either originally probabilistic or converted to a uniform distribution for multi-label classification datasets. At the beginning of each run, we randomly choose 80% of the dataset as the training set and the remaining 20% for evaluation. We further take 20% of the training set as the validation set determine the best parameter for final testing.
Table 1: Dataset statistics and normalized generalization losses with 95% confidence intervals on each dataset. The best results are indicated in bold. f indicates statistical significance with paired t-test (p 0.05).
Datasel
corel5k
CheqativePSst
flags
OpositivePseAac
Stackex.chess
5000
1392
194
519 4
1672
D
374
227
Features
499
440
440
585
Softmax
2.738 +: 0.013
0.306
0.011
1.294
0.017
0.329
0.014
2.565
0.031
Sphericul
2.907 1 0.010)
0.307
0.012
1.324
0.037
0.339 1 0.0161
2.700
0.043)
Ours
2.738 + 0.012
0.306 0.011
1.294 0.017
0.329 1 0.014
2.555
0.037
We repeat the above process 10 times for each dataset on a laptop with a 2.7 GHz Quad-Core Intel Core i7 CPU. All the methods take less than 1 minute per run in wall time. The results in Table show that our proposed method either has the best performance or achieves similar performance to the best method with statistical significance in most of the adopted datasets.
For sensitivity analysis, we fix random split of the Stackex chess dataset and vary 3 with other settings unchanged. The experiments are repeated 10 times. As shown in Figure the expected loss
8

---

Under review as a conference paper at ICLR 2023
Safmman
30
5x 10
10
Rix 10
7.6
Figure 2: Normalized generalization losses with different coefficients or noise levels. Left: varying in [0.1, 10.0]. Right: varying probability of contamination in [0, 0.8). The X axes of the left subfigure is in logarithmic scale. Best viewed in color.
of our method on the test set is slightly better than baselines. For better illustration, we cut [0.1, 0.5] off the x-axis because the softmax and our method are indistinguishable without scaling.
Additionally, we study the robustness of our approach by introducing noise to the training set of the stackex.chess dataset, repeated 10 times. To this end, for each instance x, with probability Pnoise. we replace the ground truth by a random distribution from P()). We vary Pnoise from 0 to 0.8. As seen in Figure 2] our method is slightly better when Proise 0.4. All the methods become vulnerable for large Paoise possibly because of the backbone neural network model.
DISCUSSION AND CONCLUSION
We proposed a moment-based distributionally robust learning framework for probabilistic super- vised learning under mild assumptions, showed its equivalence to dual-norm regularization for surrogate loss, presented its out-of-sample guarantees, developed efficient algorithms for typical continuous proper losses, incorporated the proposed method into differentiable learning and con- ducted experiments on several real-world datasets. We aimed to shed light on this more general supervised learning setting (Gressmann al. (2088) and provide a more expressive way of quanti- fying prediction uncertainty. A drawback of the proposed method is that solving the saddle-point problem can be difficult for some complicated losses while neural networks equipped with a soft- max layer makes use of automatic differentiation to avoid facing this issue. Interesting directions for future investigation include generalizing the learning framework to conditional density estimation and considering ambiguity sets defined by higher-order moments.
REFERENCES
Kaiser Asif, Wei Xing, Sima Behpour, and Brian D Ziebart. Adversarial cost-sensitive classification. In UAL, pp. 92- 2015.
Abhijit Bendale and Terrance E Boult, Towards open set deep networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1563-1572, 2016.
J Eric Bickel. Some comparisons among quadratic, spherical, and logarithmic scoring rules. Deci- sion Analysis, 4(2):49 65, 2007.
Mathieu Blondel. André FT Martins, and Vlad Niculae. Learning with fenchel- losses. Jour- nal of Machine Learning Research, 21(55):1-69) 2020.

---

Under review as a conference paper at ICLR 2023
Glenn W Brier. Verification of forecasts expressed in terms of probability. Monthly weather review, 78(1):1-3, 1950.
Lee Chapman. Probabilistic road weather forecasting, In Proceedings of the 16th SIRWEC Confer- ence, Helsinki, Finland, May 2012, 2012.
Francisco Charte, Antonio J Rivera, María J del Jesus, and Francisco Herrera. Quinta: A question tagging assistant to improve the answering ratio in electronic forums. In Ieee eurocon 2015- international conference computer tool (eurocon), pp- 1-6. IEEE, 2015.
Weiwei Cheng, Eyke Hüllermeier, and Krzysztof Dembezynski. Bayes optimal multilabel classi- fication via probabilistic classifier chains. In Proceedings of the 27th international conference on machine learning (ICMI.-10), pp. 279-286, 2010.
Carlo Ciliberto, Lorenzo Rosasco, and Alessandro Rudi. A consistent regularization approach for structured prediction. In Advances in neural information processing systems, pp. 4412-4420, 2016.
Michael Collins, Robert E Schapire, and Yoram Singer. Logistic regression, adaboost and bregman distances. Machine Learning, 48(1-3):253225 2803 2002.
Corinna Cortes, Vitaly Kuznetsov, Mehryar Mohri, and Umar Syed. Structural maxent models. In International Conference Machine Learning, pp. 391-399. PMLR, 2015.
A Philip Dawid. The geometry of proper scoring rules. Annals of the Institute of Statistical Mathe- matics, 59( 1):77-93, 2007.
Yann Dubois, Douwe Kiela, David Schwab, and Ramakrishna Vedantam. Learning optimal repre- sentations with the decodable information bottleneck. Advances in Neural Information Processing Systems, 33:18674 10118867--888 18690, 2020.
John Duchi and Hongseok Namkoong. Variance-based regularization with convex objectives. The Journal of Machine Learning Research, 20(1):2450-2500 2019.
Pinar Duygulu, Kobus Bararrd, Joao FG de Freitas, and David A Forsyth. Object recognition as machine translation: Learning a lexicon for a fixed image vocabulary. In European conference on computer vision, pp. 97-112. Springer, 2002.
Russell L Elsberry. Predicting hurricane landfall precipitation: Optimistic and pessimistic views from the symposium on precipitation extremes. Bulletin of the American Meteorological Society, 83(9):1333-13399 2002.
2002.
Farzan Farnia and David Tse. A minimax approach to supervised learning. In Advances in Neural Information Processing Systems, pp. 4240 4248, 2016.
Rizal Fathony, Angi Liu, Kaiser Asif, and Brian Ziebart. Adversarial multiclass classification: A risk minimization perspective. In Advances in Neural Information Processing Systems, pp. 559-567, 2016.
Rizal Fathony, Sima Behpour, Xinhua Zhang, and Brian Ziebart. Efficient and consistent adversarial bipartite matching. In International Conference Machine Learning, pp. 1457-1466, 2018a.
Rizal Fathony, Ashkan Rezaei, Mohammad Ali Bashiri, Xinhua Zhang, and Brian Ziebart. Dis- tributionally robust graphical models. Advances in Neural Information Processing Systems, 31, 2018b.
Jerome Friedman, Trevor Hastie, and Robert Tibshirani. Additive logistic regression: statistical view of boosting (with discussion and a rejoinder by the authors). The annals of statisties, 28(2): 337-407, 2000.
Jochen Gast and Stefan Roth. Lightweight probabilistic deep networks. In Proceedings the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3369-3378, 2018.
Andrew Gelman, Jessica Hwang, and Aki Vehtari. Understanding predictive information criteria for bayesian models. Statistics and computing, 24(6)99771006, 2014.
10

---

Under review as a conference paper at ICLR 2023
Xin Geng. Label distribution learning, IEEE Transactions Knowledge and Data Engineering, 28 (7):1734-1748, 2016.
Tilmann Gneiting and Adrian E Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American statistical Association, 102(477)355-337) 2007.
Eduardo Corrèa Gonçalves, Alexandre Plastino, and Alex A Freitas. A genetic algorithm for opti- mizing the label ordering multi-labe classifier chains. In 2013 JEEE 25th International Con- ference on Tools with Artificial Intelligence, pp. 469-476. IEEE, 2013.
Good. Rational decisions. Journal of the Royal Statistical Society. Series B (Meshodological), 14 (1):107-114, 1952.
Frithjof Gressmann, Franz J Király, Bilal Mateen, and Harald Oberhauser. Probabilistic supervised learning. arXiv preprint 18Xiv:1801.0000 000.00053........ 2018.
Peter D Grünwald, A Philip Dawid, et al. Game theory, maximum entropy, minimum discrepancy and robust bayesian decision theory. the Annals of Statistics, 32(4):1367-1433, 2004.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning, pp. 1321-1330 PMLR, 2017.
Taejong Joo, Uijung Chung, and Min-Gwan Sco. Being bayesian about categorical probability. In International Conference Machine Learning, pp. 4950-4961. PMLR, 2020.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:
2014.
Anirban Laha, Saneem Ahmed Chemmengath, Priyanka Agrawal, Mitesh Khapra, Karthik Sankara- narayanan, and Harish G Ramaswamy. On controllable sparse alternatives softmax. Advances in neural information processing systems, 31, 2018.
Leonard Leibovici, Michal Fishman, Henrik C Schonheyder, Christian Riekehr, Brian Kristensen, Ilana Shraga, and Steen Andreassen. A causal probabilistic network for optimal treatment of bacterial infections. IEEE Transactions Knowledge and Data Engineering, 12(4))517-528, 2000.
Yeshu Li, Danyal Saeed, Xinhua Zhang, Brian Ziebart, and Kevin Gimpel. Moment distributionally robust tree structured prediction. Advances in Neural Information Processing Systems, 35, 2022.
Shengfei Lyu, Xing Tian, Yang Li. Bingbing Jiang, and Huanhuan Chen. Multiclass probabilistic classification vector machine. IEEE Transactions on Neural Networks and Learning Systems, 2019.
Hamed Masnadi-Shirall and Nuno Vasconcelos. On the design of loss functions for classification: theory, robustness to outliers, and savageboost. In Advances in neural information processing systems, pp. 1049-1056, 2009.
Santiago Mazuelas, Andrea Zanoni, and Aritz Pérez. Minimax classification with 0-1 loss and performance guarantees. Advances in Neural Information Processing Systems, 33:302-312, 2020.
Santiago Mazuelas, Yuan Shen, and Aritz Pérez. Generalized maximum entropy for supervised classification. IEEE Transactions Information Theory, 2022.
John Ashworth Nelder and Robert WM Wedderburn. Generalized linear models. Journal of the
Royal Statistical Society: Series A (General), 135(3):370-385 1972.
Alex Nowak, Francis Bach, and Alessandro Rudi. Consistent structured prediction with max-min margin markov networks. In International Conference on Machine Learning, pp. 7381 7381-7391. PMLR, 2020.
Harris Papadopoulos. Reliable probabilistic classification with neural networks. Neurocomputing, 107:59-68, 2013.
11

---

Under review as a conference paper at ICLR 2023
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems, 32: 8026-8037, 2019.
Mark D Reid and Robert C Williamson. Information, divergence and risk for binary experiments. The Journal of Machine Learning Research, 12:731-817, 2011.
Soroosh Settfeezaadh-Abbadde Daniel Kuhn, and Peyman Mohajerin Esfahani. Regularization via mass transportation. Journal of Machine Leurning Research, 20(103):1-68, 2019.
Shai Shalev-Shwartz and Shai Ben-David. Understanding machine learning: From theory to algo- rithms. Cambridge university press, 2014.
Maurice Sion. On general minimax theorems. Pacific Journal of mathematics, 8(1):171-16,, 1958.
Masashi Sugiyama, Ichiro Takeuchi, Taiji Suzuki, Takafumi Kanamori, Hirotaka Hachiya, and Daisuke Okanohara. Conditional density estimation via least-squares density ratio estimation. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statis- tics, pp. 781-788, 2010.
Ambuj Tewari and Peter L Bartlett. On the consistency of multiclass classification methods. Journal of Machine Learning Research, 8(May):1007-10)) 2007.
Xin Wang, Hao Helen Zhang, and Yichao Wu. Multiclass probability estimation with support vector machines. Journal of Computational and Graphical Statistics, 2863)5586-595, 2019.
Jason Weston and Chris Watkins. Multi-class support vector machines. Technical report, Citeseer, 1998.
Wolfram Wiesemann, Daniel Kuhn, and Melvyn Sim. Distributionally robust convex optimization. Operations Research, 6266):135-1 2014.
Robert Williamson, Elodie Vernet, and Mark Reid. Composite multiclass losses. Journal of Machine Learning Research, 17:1-52, 2016.
Jianhua Xu, Jiali Liu, Jing Yin, and Chengyu Sun. A multi-label feature extraction algorithm via maximizing feature variance and feature-label dependence simultancously. Snowledge-Basse Systems, 18-17118 2016.
Shuhei M Yoshida, Takashi Takenouchi, and Masashi Sugiyama. Lower-bounded proper losses for weakly supervised classification. In International Conference Machine Learning, pp. 12110- 12120. PMLR, 2021.
Mingyuan Zhang, Jane Lee. and Shivani Agarwal. Learning from noisy labels with no change the training process. In International Conference on Machine Learning, pp. 12468-12478. PMLR, 2021.
Hui Zou, Ji Zhu, and Trevor Hastie. New multicategory boosting algorithms based on multicategory fisher-consistent losses. The Annals of Applied Statistics, 2(4):1290, 2008.
A TECHNICAL PROOFS
Proposition 1. The distributionally robust probabilistic supervised learning problem based on mo- ment divergence Eq. Q can be rewritten as
min Epay min max L (Pyix, Qyix) + OT (Earix Φ(X,8) - -Epor Fix
$(x, + +
L.a10.1m Fix!
where O E RDD is the vector of Lagrangian multipliers and [|=I|- is the dual norm of [|-I]-
12

---

Under review as a conference paper at ICLR 2023
Proof. Recall the primal problem
min where A(Pem)) == (Q PyIXEPO) QEA((M)))
max
Eox [L (Prix. Qrxxx)]
P Qx A [[penr [Ø( (b(.-))- 10( -11 S Se).
Note the feature function Φb( ) fixed and given. The constraint sets P())) and (Pemp) are convex.
The objective function L(I,, Q) is quasi-convex in P by (Williamson et al. 2016 and concave in Q because is affine in Q Therefore strong duality holds by Sion's ss minimax theorem (Sion) 1958):
max QEA((EmR)
min (P()))
Eox [L (Prix, Qr1x)]
Let C(u)= -(u: | Egen (l)) S Ee}. Rewrite the problem with this constraint:
sup min E pem [L (Frjx: Qrix)] - Ic(u)
Quu
P
S.t. E Brr by Φ(X (),Y),
where Te(+) is the indicator function with Te( 0 if C and +o0 otherwise. The simplex constraints and Q are omitted.
The dual problem by relaxing the equality constraint is
sup min min Epprr L (I),,, Qrix)] lc(u) + O O Q k(X,)) UTu,
Q.u
where 0 is the vector of Lagrange multipliers.
Given æ. optimization of Q2 and py can be done independently. Again by strong duality, we can rearrange the terms:
min Egpr min max L (Py x, Qylx) + OTE 4(x 1xXY Y) + sup -Ic(w) - 0T
0
Q
The associated dual norm
||-1|~ of the norm ||-|| defined as
|| := sup(=Tæ x ||>||| - 1}:
based on which we are able to simplify the optimization over u as
sup Ic(u) T sup
TTu = sup - 0 (Ep - Ee) = - - Φ(.) + =||0||..
wEC
e:jel|s1
Plugging it back the dual problem, we have
min Egau min max L (Pyix: Qyix) + oT (Barix
Eparo 6(X,))) + =||0||.. xix
Corollary 2. When 0, Leddy Fisher consistent with respect L. Namely, for any ac
p O Y|æ
Carg min min L(Py pprre
Py
is the Bayes optimal probabilistic prediction made by O the solution in Eq. under Jiree
Proof. Our setting differs from Nowak et al. (2020) in the fact that we use a distribution as the ground truth. By defining y" (1) as the gold standard probabilistic prediction and y the set of all possible probabilistic predictions in Proposition C.2 in Nowak et al. (2020), we have
T Y
E Conv(arg min L[Py|a, Pi))))
Py
Because L is assumed continuous proper, any convex combination of minimizers is also minimizer. Therefore,
E arg min L(Py Ia )
Y
Pplm
13

---

Under review as a conference paper at ICLR 2023
Theorem 3. Given samples, non-negative multiclass probabilistic loss L(.;-) such that
[L(,,)) I K. a feature function ((:,)) such that [| I B and a positive ambiguity level
>> O. then, for any 0 < 8 S L, with probability at least 1 ó, the following excess true worst- case risk bound holds:
4KB Evn
ln(4/))
R5(
( (mie) I ((B())
max (E
max
2V
where Comp and One are the optimal parameters learned in Eq. 3 under the empirical distribu- tion pemp and true distribution Iprc,, respectively, The original risk of 0 under Q is Rb(0)
E P p L(Py x. Qrix) with prediction 19g E arg ming maxq L (Prix. Qyix) +
Eggix 0T (X),).
Proof. Define the adversarial surrogate risk of 0 with respect to as
I (0) Exx min max L (Pyix, Qx(xx) OT (Eœrix a (X,Y) Expix (xX, Y)) + =||0||--
Let 0 E arg ming R (0) and @cmp E arg ming RS (0) be the optimal parameters learned with " and xxx respectively.
Given æ, define the decoded prediction by as
FY arg min max L (Fy)x- Qx BTE $(X,1² Y). Q
Let the original risk of loss under some distribution Q be
R == Eax L (P& Crix)
According to Proposition]] I for any fixed P, have similarly
Eox (PP Qvxx) A min Epgr max L (Pr Qrix) + OBo .(X.8) - -Eeas o(x,1)) + +c|0].. vix
max QEA(FTM)
0
We start by looking at the worst-case risk of Ous and cmp
max (5((
QEA((EM))
min Epm max L Gimp Qrix) + oT (Baari 2(X, Epmr (xX Y)) + +||0|||
Lpgn max L (PYiN QYlx) + Ozmp (B2y x 2 (X E ((X,)) + s 0
where the last inequality holds because Oump is not necessarily a minimizer. Similarly for 0
max QEA((FRR)
R5 Equn max L ( Qrix) 0 (Boy 6(X,8)
EFer
vix 6(X, Y)) + e 0
Q
On the other hand,
(ptixx Qvix) + 0ine (Eorix (xX, Y) Epme (XX, Y)) + 0
Epg max 1
min Eage min max T. (Py),, (xlx) + (T $(X V) R Φ(X, Y)) + e 0|
P
Q
min min Epn max I. (Py)x, Qyix) + +0T Φ(X, @
Ennr xix ((X,Y) + c 0|.
< min Egr max L (p Qrix) (E $(X,) Y) Eggn Φ (xx, Y)) + + fix
R4(0 ) QEA((rM)
14

---

Under review as a conference paper at ICLR 2023
where the first equality holds according to the definition of Oiree The above two inequalities imply the equality:
max (E.((PMM)
(f = Eppn max I. Q
(l)::
Qrix) + O (Ber $ V) Ezr ((x, + e meeell.-
Therefore,
( (cmp))
(
max (Pim))
max ((POn))))
< (o max L (Poixx
QrIx) + Oump (Ber)x (XX
-ppn Φ(X, + +=100mmp empll (lppl+)
(Epge max L
PT Qrxx) Oime (Bapix Φ5 (x Y)
Epun rix Φ Y))
0 Il+).
Q
(10)
The main idea is thus to uniform convergence bound. Firstly, by substituting Q ppiree note that
min max L (Pyy Qrix) + OT (R (6x Y) Ryure ( Y)) 2 min T. (PYIx rix 2 0.
P
We can get an upper bound of the norm of any optimal solution 0 or @imp as follows:
0 + ellOmeell ||- S Repi (0 S Rige (0) Ragyy I (Py Qr1x) SK 1imelll S K
Let 1/ X, Y) @TT Y) and th (1b(1, Y)) Define
f(0, PP) == Exx min max L (Pyix, Qr(x) + OTTLEeii $(X,)) E $(X,8))
Q
A Epx max @
L (Pe (F Qxxx) OT (Bavix Φ(x,1) - -Epy $(X, P))
E E max L PP
f Qr1x) (BQvix (1 Y) Epyix (xx V))
g b
Let qu E A be the probability vector of Qy and ey be the standard basis vector with Y-th entry equal 1. We have that for any (ær, Y),
g(b, )(m,))) S Conv((g= ey qu E A)) O
g(th ((n
max
1
er 2,
ya
where ,(2,Y) is the Dirac point measure. g(bb, IP) is therefore 2-Lipschitz with respect to the & norm. As per the assumption, [| S B. This further implies that
f(O1, )(..)))))) /(2,, (02,0(ez.)))) va)) < 4KB
K 8
v0, 02, 3 32 Y1, Yo ||
Wi
1,2.
E
We then follow the proof of Theorem 3 in Farnia & Tse 2016). According Theorem 26.12 in Shalev- Shaler-Shwwatz& & Ben-Dave (2014), by uniform convergence, for any d E (0, 2]. with probability at least 1 A
7.
4KB Evnn
In (4/0))
f00 Parue)
f(0mmp: pempp)
1
2
According to the definition of O the following inequality holds:
f(0" emp: pemp) + =l100 II. f(0 pemp) ell 0 e0 0.
15

---

Under review as a conference paper at ICLR 2023
Since due do not depend samples, according to the Hoeffding's inequality, with probability 1 6/2.
2KB
In(4/6)
fl Ti PPmp)
(fiiu pplnee
E EvN
Applying the union bound to the above three inequations, with probability 1 5, we have
4KB
3 In (4/6)
[ [(0mmp: (ptne) + =e|00mppl - f(O [prmiee - E =l S
+
EV'
As stated by Inequation 10 conclude with the following excess risk bound:
4KB
In (444))
max (E(())))
(5(0 cmpp)
- max QE.((Pim))
(5
1
2
2
Proposition 4. The DRO method for probabilistic loss based on logarithmic loss, and 3 Brier loss has a solution pyyj for the predictor parameterized by 0 defined by the following systems of equations:
VX EEX,CCERVVEE Prix(U) =Txx(C exp(C 0* ((x,1) v) Wo(23c d(x.1))).
where C constant dependent on O and but independent of y. W()) the principal branch of the Lambert W function. The corresponding adversary L x is defined as
2 Yix (1) + Zrr Py P (v)
1 E, 23P x (1)/(1))))++++))))1 2,51 (v))
Q1)x(1)= =
and Zpyix
1 2014
Ey Py (1)((1 2 11 (1)
1|x
Proof. Recall the saddle-point optimality condition:
Σ Qr v)) + Zpy -0
((PY, v) oT (T + Zer =0
Dependence æ is omitted when context is clear. Substituting lmix yields:
(y)( (y) 2,3) + 2,3 (V) + Zzy =0 0
(n) + B(1 22 Σ P+ + (T + Zar 0.
Note that : B (1/) + Zay is constant across all y's given 0, æ. Thus for fixed 0, æ. we have for some Cr 0,
C 0. + (x v) =
Vy E y.
which is equivalent
25F (y)e 28Pr (v) 0(
By the definition of the Lambert W function,
23Py (v) W (
Since 2 0. (.aaaa))) > 0, the principal branch Wo of the Lamber W function is always applica- ble. Also by the formula (x)) (()) have
Py(v) exppCC... (()()v)) Wo Co +0" ((x.)))) (0000x00)))))))
Vy.
16

---

Under review as a conference paper at ICLR 2023
Let Py- (for given 0) be solution to this set of equations that also satisfies PT 1. By
Eq. ( the optimal Q satisfies
2,31P y (1) + Zay
2,317 + Py(())
(v)
Ft(1) + +23
1+ ++23FP
Zey must be chosen to properly normalize (g):
2/P
Σ
Zpy Σ
+ +
+a+20 28
y
y (y)
M
(T)
1 Σn
2 (M)
+0+28
1 Ey ++
2
Zpr
4
Σn
Ey (a
Py-(v)
+a+26 1a
Both Zizy and Qÿ (y) are positive because Py E P(V) is solution.
B MORE LOSSES
The discrete ranked probability vector assumes ordering relationship in V, i... y (1. []]]. The score can be written as
6p (Pr
P =)
The mixture loss of the log loss, Brier loss and ranked probability loss can be written as
f (P In (v) 3(1 (y) +
Σ
(r)) +
Py(J) - T(i y))
y
Substituting the loss into Eq. O6) yields
|
Qr Py()) 23) + 233 + 20
Py(j) + Zp, - 2a(11) 1-- 4+11
Σ
(i Qr) =0
j=y==
i=y+1
(11)
-In + 3(1 - 21 + Σ
P2. (2)(1)) (T 4) Zor
+ - y +
Py 2112 2
Pr
2 ()))= =0 =1
(12)
==
(J) is constant across all y. By absorbing them into constant C. we also observe that the equation for y only depends on Py(y') for y < y in the term ΣØ Σ j= (r). Therefore, PY (v) can be found in increasing order of from 1 to [
Notice that Σ -
Given Py, consider Eq. in matrix form:
Py (1) 2,3
2x
4a
2(y) 1)ax
!
Qr
C C2
0
-1 -1/Px
23
2a
2((y 2)(x
i
Q2)
0
0
0
1/Prr(()))))) 233
i
Cr
Opi 1.
1
1
Zev
This is not unreduced Hessenberg matrix. However, notice that Zty increases, Qy ((D?)) also increases by the penultimate equation. This turn increases Qy (DI-1) according to the third from last equation. Therefore, the solution @ without the simplex constraint increases monotonically as Ziy increases. We can use bisection method again to find the Q that also satisfies the simplex constraint.
17


Reviews:
[{"review_id": "mMjwo814IT", "reviewer": "Reviewer_FSDY", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes an algorithm to compute a DRO objective where the uncertainty set is defined with deviations in moments of the representation. The algorithm allows one to compute the minimax DRO loss using a primal-dual formulation and reasoning about the stationary points.", "summary_of_the_review": "While I think the ability to compute minimax loss is great, I am unclear about some details and what the value of the paper is.", "strength_and_weaknesses": "The derivation in the paper is theoretically interesting.\n\nMy main concern is this:\n\nWhile the paper guarantees $\\theta$ is learned well with $\\phi$ fixed, it is unclear to me why I can learn a representation with the given objective. The reason is that changing $\\phi$ changes the uncertainty set for the setup DRO problem! Without the ability to use a fixed uncertainty set from some known restriction, what is the value of the proposed method? The experiments seem to support this concern as the authors say \"All the methods become vulnerable for large pnoise possibly because of the backbone neural network model.\"\n\n\nWeaknesses/Questions:\n1. Both corrollary 2 and theorem 3.2, along with the stated uncertainty set are from ((Li et al., 2022). Why do the authors claim \" We propose a distributionally robust probabilistic supervised learning method\"? The main contribution seems to be the algorithm.\n2. What can go wrong when let $\\beta=0$?\n3. Theoretically, it seems like when learning $\\phi$, it seems possible to end up in a minima where $\\phi=0$. Can the authors explain why this won't happen when the objective is optimized?\n4. Is mixing log-likelihood and brier score used in an application? Can the authors point to a case?\n5. The experiments in table 1 do not seem to favor the proposed method much; softmax is better or similar.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "lHhlUG2kbC", "reviewer": "Reviewer_ARxu", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper is concerned with probabilistic predictions: Instead of predicting a class, the prediction is a distribution over the label space. The paper proceeds by using a probabilistic loss functional, and consider an adversarial training approach. The proposed method is a minimax problem, with perturbations in a moment set that is specified using the feature function. The paper then provides the dual formulation and integrates this formulation into a gradient descent algorithm (Algorithm 1).", "summary_of_the_review": "- The paper lacks mathematical rigor, which severely hinders my understanding and appreciation of the results. \n- There is a significant overlap with Li et al (2022) which requires clarification.\n- The experiment does not demonstrate the benefit of the proposed method", "strength_and_weaknesses": "The paper lacks clarity and mathematical rigor:\n1. Before Proposition 1 in the main text, the authors claim that “the ambiguity set ($\\mathcal{A}(\\mathbb{P}^{\\mathrm{emp}}$) is a compact convex set.” However, no proof is provided.\n2. In the proof of Proposition 1, page 13, line 6, the authors invoke Sion’s minimax theorem. However, the authors did not verify that the loss function is lower-semicontinuous in $\\mathbb{P}_{Y|X}$ and upper-semicontinuous in $\\mathbb{Q}$.\n3. Also, in the proof of Proposition 1, in the middle of page 14, the authors claimed that “Again by strong duality, we can rearrange…”. However, no justification for strong duality has been provided. \nFurther, I feel that the use of $I_{\\mathcal{C}}(\\cdot)$ is unnecessary. In fact, the indicator function is non-smooth, non-convex, which complicates the whole process of applying duality results.\n4. The notations of $\\min_{\\mathbb{P}}$ is really confusing in the statement of Proposition 1. The expectation is already taken with respect to $P_x^{emp}$, so what is the $\\min_{\\mathbb{P}}$ in equation (3) concerning? Shouldn't $\\mathbb{P}$ be $\\mathbb{P}_{Y|X}$?\n5. The results of this paper seem to follow from Li et al. (2022) [Proposition 1, Corollary 2 and Theorem 3], however, up to today (Oct 23), I could not find the manuscript of Li et al. (2022) on arXiv or on any public domain (openreview). At this point, it is difficult for me to judge the novelty of the manuscript.\n6. Using the setup of this paper, any distribution can be conveniently represented as a matrix because the space $\\mathcal{Y}$ is discrete and the space $\\mathcal{X}$ can be reduced to a finite set supported on the training data. From this perspective, the problem becomes uninteresting because, in the end, we are only perturbing the elements of a matrix. The problem only becomes interesting, in my opinion, when $\\mathcal{Y}$ is continuous or we allow perturbations of the marginal distribution on $X$.\n7. The numerical experiments can not demonstrate that the proposed method is superior than existing approaches. The experiments are also conducted with $\\varepsilon = 0$, which implies no moment robustness.\n8. The authors should make clear how a probabilistic prediction is made at test time, when only $x$ is given and there is no information on $y$.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "JW5GsSdH1d", "reviewer": "Reviewer_QSPn", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper considers distributionally robust probabilistic supervised learning. The ambiguity set is constructed to include distributions that share the same marginal with the empirical distribution and are no more than ε away from the empirical in terms of first-order feature moment divergence. The strong duality is shown such that the primal DRO problem is equivalent to a regularized ERM problem. The authors characterize the solutions to the proposed method and present an efficient algorithm for specific losses. Moreover, neural network representations are incorporated and the DRO problem can serve as a differential layer to enable end-to-end differentiable learning. \n\n", "summary_of_the_review": "Overall this paper is well-written and considers a problem that can be of broad interest to the machine-learning community. The results are standard -- such as the duality with regularized ERM and the risk bounds. However, the theoretical results look weak and most importantly, the authors didn't explain their technical contribution clearly, as compared with the work (Li et al., 2022). Moreover, the result in Corollary 2 is only limited to the case epsilon=0, which is the standard ERM, not the DRO problem. \nAlso considering that the numerical results are also limited and do not compare with many baseline methods, I think this is more like a borderline paper.\n", "strength_and_weaknesses": "strength:\n- A DRO version of probabilistic supervised learning is proposed under first-order moment ambiguity sets. \n- The DRO problem can be incorporated into neural networks and perform end-to-end differentiable learning\n\n\nWeaknesses\n- The theoretical result is weak, and the technical contribution is not clear -- in all theorems and Propositions, the reference (Li et al., 2022) is cited, but there is no clear justification for the differences with (Li et al., 2022) \n- The empirical study on real-world data is not enough, especially the baseline methods compared are very limited, only neural network models with the softmax and the spherical softmax function.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "7GPNFD98n5", "reviewer": "Reviewer_u3yA", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors present a method for learning a model that outputs probabilities wherein the output probability is the output of an optimization over an uncertainty set rather than, say the result of a softmax. They show how to learn a model using this approach and compare the method to simple alternatives on several datasets.", "summary_of_the_review": "Overall, I found the technical aspects of this paper clear and well-written, and the methods and evaluations reasonable. With that said, I am still very unclear on the motivation for this method, especially given that if failed to outperform simple softmax on any of the evaluation problems. My concerns are below:\n\nMajor:\n\n1. I did not follow the motivation for this method and strongly recommend that authors work to make the intuition clearer in the introduction.    Further, the experiments did nothing to make this motivation clearer. If anything, I left the paper wondering why I would use this method when a simple softmax appears to perform nearly identically. I recommend design their experiments to highlight and provide intuition for the differences between existing approaches and the proposed method and, if no differences exist, explain why a negative result is interesting. If there are no differences and no value in a negative result then it is hard for me to see the significance of the method.\n\nMinor:\n\n2. Define \"proper\" in the intro.\n\n3. Contributions: \"Extensive\" seems strong here. Instead of saying \"we performed evaluations\", describe what your evaluations show and how that contributes to our knowledge.", "strength_and_weaknesses": "See below.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject