Tailoring to the Tails: Risk Measures for Fine-Grained Tail Sensitivity
Authors: Christian Fröhlich
Robert Williamson
Venue: Accepted by TMLR ()

Abstract: Expected risk minimization (ERM) is at the core of many machine learning systems. This means
that the risk inherent in a loss distribution is summarized using a single number - its average.
In this paper, we propose a general approach to construct risk measures which exhibit a
desired tail sensitivity and may replace the expectation operator in ERM. Our method
relies on the specification of a reference distribution with a desired tail behaviour, which is
in a one-to-one correspondence to a coherent upper probability. Any risk measure, which is
compatible with this upper probability, displays a tail sensitivity which is finely tuned to the
reference distribution. As a concrete example, we focus on divergence risk measures based
on f-divergence ambiguity sets, which are a widespread tool used to foster distributional
robustness of machine learning systems. For instance, we show how ambiguity sets based on
the Kullback-Leibler divergence are intricately tied to the class of subexponential random
variables. We elaborate the connection of divergence risk measures and rearrangement
invariant Banach norms.

Published in Transactions on Machine Learning Research (01/2023)
Tailoring to the Tails: Risk Measures for Fine-Grained Tail Sensitivity
Christian Fröhlich
Deportment of Computer Science University of Tübingen
Robert C. Williamson
bob. hol.wlllaanssonneeeidttubeennnee
Deportment of Computer Science and Tübingen Al center
Universily of Tübingen
Reviewed on OpenReview: opennevvee.. net/ formm id= UntUoeLuuu
Abstract
Expected risk minimization (ERM) at the core of many machine learning systems. This means that the risk inherent in a loss distribution is summarized using single number its average. In this paper, we propose general approach to construct risk measures which exhibit desired tail sensitivity and may replace the expectation operator in ERM. Our method relies on the specification of reference distribution with a desired tail behaviour, which is in one-to-one correspondence to coherent upper probability. Any risk measure, which is compatible with this upper probability, displays a tail sensitivity which finely tuned the reference distrribution. As a concrete example, we focus on divergence risk measures based on f-divergence ambiguity sets, which are a widespread tool used to foster distributional robustness of machine learning systems. For instance, show how ambiguity sets based the Kullback-Leibe divergence are intimately tied to the class of subexpo- nential random variables. We claborate the connection between divergence risk mcasures and rearrangement invariant Banach norms.
1 Introduction
A plain language summary of our work and diagram showing key conceptual elements are in Appendix A.5
Expected risk minimization is at the core of machine learning systems. This means that the risk inherent in loss distribution is sunmarized using single number its average. Yet this does not capture the tail behaviour of the distribution. Since the expectation is insensitive to such behaviour, two distributions may have the same expectation, although they exhibit very different tails. In the context of machine learning, this can be problematic: extreme losses correspond to extremely bad predictions, or, for instance in the context. of reinforcement learning, even lead disastrous consequences. In many settings, there appear to be legitimate motivations for sacrificing some average performance in order to avoid rare but extreme losses. A docision maker with this attitude is said to be risk averse.
Another issue to consider is that the empirical distribution perfectly corresponds to the 'true' distribution only in the limit of infinite data. With finite data, the observed tails could be much lighter than the 'true' tail of the loss distribution, since extreme events are by definition infrequently observed. This suggests that a cautious approach to pretend that the tail heavier than actually observed. This approach has been studied under the research theme of distributional robustness, where expected risk minimization has been replaced by the distributionally robust forsst--fii f-divergence risk measure objective:
R(X) =
(1)

---

Published in Transactions on Machine Learning Research (01/2023)
where X is the random loss variable, the Q are probability measures and the f-divergence with divergence function f is defined as
dr(Q, /
dP.
if Q is absolutely continuous with respect to P. Here, the decision maker takes worst-case altitude with respect to a set of probability measures, which are contained in an f-divergence ball of radius e, centered at the base distribution P. These probability measures are said to constitute an ambiguity set. decision maker, who takes a worst-case stance towards an ambiguity set, is ambiguity averse. In practice, we take P to be the empirical distribution, that is, the training data. Intuitively, the function f controls how much reweighting do is allowed. As a consequence, we guard against possibly different tail behaviour of the 'true' distribution.
The goal of this paper is to construct tail-sensitive coherent risk measures (Section 3.4 where the tail sensitivity can be controlled by the machine learning engineer. Such risk measures are then potential re- placements for the expectation in ERM and can model both risk and ambiguity aversion, depending on the decision maker's intention. For instance, the f-divergence risk measure in (I a coherent risk measure, and we demonstrate how the function f corresponds exactly to choice of tail scnsitivity. Consequently, the choice of f determines the structure of the divergence ball, that is, the tail behaviour of the reweightings it contains. Due their popularity focus on f-divergence risk measures. However, we also explicate its relation to other coherent risk measures with the same fundamental function, which corresponds (to some extent) to the same tail sensitivity. To investigate these relations, we view coherent. risk measures from the perspective of rearrangemend invorianl Banach norms.
means to achieve distributional robustness in machine learning, we
For each class of specified tail behaviour (for instance, exponential tails, Gaussian tails etc.), subject a suitable equivalence relation, show how the following objects be derived:
An f-divergence risk measure as which can be employed replace the expectation in the risk minimization loop.
A Lorentz and Mareinkiewicc norm and related coherent risk measures, which are further such replacement candidates.
A utility-based shortfall risk measure, which is conver risk measure, with potential applications for instance in reinforcement learning.
The main guiding thread is the family of Orlicz norms, which offer fine-grained control over tail risk via the specification of a convex function. The class of such functions is approximately as rich as the class of f-divergences. The f-divergence risk measure
[ then arises as the natural exlension of what we call an Orlicz regret measure. The thcory of Orlicz spaces is well-devloped, and not only underlies divergence risk measures but also offers a general approach to obtaining concentration inequalities and maximal inequalities (Edgar& & Sucheston, [192), but appears that the machine learning community is not aware of this link and thus cannot exploit the rich mathematical literature the subject.
1.1 Related Work
We weave together idcas from different literatures, from coherent risk mcasures & Römisch 2007 and rearrangement invariant Banach spaces (Krein al. 1982 Bennett Sharpley 1988
Pfug (Artznrr et al. 1999
Rubshtein al. 2016) to distributional robustness (Rahimian & Mehrotra, 2022).
Our goal is to construct risk measures which are sensitive to the hcaviness of the tails. The importance of heavy-tailed phenomena has been popularized by Taleb (2077) and they have been studied gemerally by Nair et al. (2022) and Resnick (2007). In terms of applications, heavy-tailed phenomena have been investigated in oconomics [Nrdhaus 20122 Ibragimov al., 2015), carth sciences (Cavanaugh et al. 2015 Merz et al.) 2022). neuroscience (Roberts et al.) 2015)) and many other areas. Within machine learning, Gurbuzbalaban et al. h021h have identified heavy tails in the iterates of stochastic gradient descent. Empirical risk mininization
2

---

Published in Transactions on Machine Learning Research (01/2023)
in the context of a heavy-tailed loss distribution has been studied by Brownlees et al (2015) and Hsu & Sabato (2016). Vladimirova et al. (019) have demonstrated the heavy-tailed nature of prior distributions in Bayesian deep learning.
We are inspired by the work of Dommel & Pichler 20211
themselves building on Ahmadi-Javid (2012)
and Ahmadi-Javid & Pichler (2017), who connected f-divergence risk measures to Orlicz norms. Ambiguity
sets based on f-divergences have received much attention in the machine learning literature so far, see for
instance (Ben-Tal & Nemirovski, 1999), [Bayraksan & Love 2015): NNamoong & Duchi 2016), (Shapiro, 2017), Namkoong & Duchi, 2017), (Hu et al,, 2018) and (Duchi et al. 2021). In mathematical finance, the special case of the coherent entropic risk measure, which corresponds to the choice of Kullback-Leibler
divergence for dy in (]), was studied by Föllmer & Knispel (2011). Closely related is the convex entropic risk measure, an empirical variant of which has been proposed by Li et al. (2021) in machine learning. Standard
material on Orlicz spaces can be found in (Krasnoselskii & Rutickii 1961], (Bennett & Sharpley 1988)
EEdgrr & Sucheston 1992). (Kosmol & (iiller-Wichhrdss 2011), (Pick al. 2013) and (Rubshtein et al. 2016).
1.2 Contributions
We define the Orlicz regret measure, which is an asymmetric variant of an Orlicz norm, and show that its natural extension yields the f-divergence risk measure. We embed the Orlicz regret measure and the divergence risk measure in the mathematical framework of rearrangement invariant Banach function spaces and study their fundamental functions. Subsequently we explicate how the fundamental function is, via its one-to-one relation to the envelope function, intricately linked to the tail scnsitivity of risk mcasure. We show how one can therefore construct corresponding Marcinkiewicz norm, Orlicz norm (hence, divergence risk measure) and a Lorentz norm, which share this tail sensitivity. We illuminate the connections and -nonveequivalenees between these norms. Overall, we do not attempt to establish that one among these three familics of norms is 'superior', but rather to analyze their relations and implications. Finally, we link our approach tail sensitivity to Orlicz deviation inequalities, of which many well-known concentration inequalities are special cases.
1.3
Use in Machine Learning
In supervised learning problem, we may wish to replace the expectation operator in ERM by a risk measure:
arg min mig E[(([(X)())))) - arg min R(((f(x),
feF
fer
for some risk mcasure R, a function f from some hypothesis space F, a loss function E, input X and ground truth labels Y. Similarly, in reinforcement learning problems, a risk measure may replace the expectation operator in the maximization of expected discounted sum of returns. Correspondingly, in empirical risk minimization, the risk measure can be applied to the empirical loss distribution.
Possible motivations for replacing the expectation with risk measure are manifold: in machine learning, risk measures have been employed to express risk-averse attitude (Tamar et al. 2015 Dabney et al.) 2018
Singh et al.] 2020 Urpí et al. 2021; Vijayan & Prashanth 2022 or to achieve distributional robustness [Namkoong & Duchi 2016 Duchi et al.) 2021 Zhang et al. 2021). Moreover, consider the case where the individual losses correspond to individual humans: in this context, it has been argued that risk measures
can promote subgroup fairness (Hashimoto et al., 2018 Williamson & Menon 2019).
We restrict ourselves to theoretical analysis
there is ample experimental evidence for the suitability of risk measures in different settings in the aforementioned works. In addition, Chouzenoux et al. (2019) and Fröhlichh
& Williamson (2022) provide general treatments on risk measures in machine learning, including experimental evidence. Our focus is therefore on drawing connections and deepening our theoretical understanding, For aspects regarding practical computation see Section
A question remains: which risk measure to choose in a given problem setting? The focus of this paper is give the reader guidance and intuition to answer this question for themselves, relative to the problem at hand.
3

---

Published in Transactions on Machine Learning Research (01/2023)
Specifically, focus is the tail sensitivity of risk measures. In the previously mentioned problem settings aiming for risk aversion, distributional robustness or fairness - the tail behaviour of the loss random variable deserves special attention. For instance, a tail event in a fairness context is an individual who receives an extreme loss. Our goal is therefore tailoring to the tails instead of pandering to the masses
1.4 Notation and Assumptions
Throughout, we assume an atomless probability space F, P), where the Lebesgue measure and we take Ω [0. 1]. We denote the set of probability measures Q which are absolutely continuous with respect to P as Q. Let M denote the class of Lebesgue measurable functions from Ω to the reals R, i.e. random variables; however, we have no regard for non-measurable functions throughout and thus often drop writing X E M. By writing P-a.e. we mean "almost everywhere with respect to the mcasure p We denote by L the Lebesgue spaces, i.c. the set of functions with finite p-th moment:
|X { (66 IXIP dP)= ess sup(X)
15pc0
C= | < co).
==00
where ess sup(X)= = inf (A2 0: P([w E El: (X XJ) 03. We write R = oo) for the nonnegative real line with included and = (-oc, 0). By X+ max(0, X) we denote the positive part of random dQ For random variable X with cumulative distribution variable. We write Radon-Nikodym derivatives as ar.
function Fx(x) = P([w Q X(w) < =)), we define the quantile function Px as the generalized inverse of the distribution function: Fx (q) = Vu 0: Fx(A) < q). For expectations Ep[X] with respect the base measure P, we often drop the subscript and simply write B[X]. We write indicator functions XA and convex indicator functions i:::
1 WEA 0 A
0
WEA A DO
Xa(w)=
(a(w)
w A
By an increasing function we mcan non-dccreasing function; similarly, a decrcasing function is non- increasing. We write 6(0+) for the right-hand limit given by lim d(x) <(0+). When defining a function f R- R U [ox), abuse notation and implicitly take this to imply that f(x) oc for 0. The convex conjugate of function f RR R U {oo} is the function R U (o0}:
g(v) = sup ry f(z).
zER
Throughout, and g will denote conjugate pair of divergence functions (Definition 3.1 and O and T will denote a conjugate pair of Young functions (Definition 2.1
2 Orlicz Norms and Spaces
The motivation behind Orlicz norms is to measure the size of a function in a way which is sensitive to tail behaviour. Often they are introduced as generalizations of L spaces. As a first example, the variance indeed is sensitive to tails and therefore does not exist (is infinite) for many random variables.
Definition 2.1. A function + R + R+oof is called Young function if it is left- increasing, convex, salisfies (0) = 0 and is nontrivial (not identical to the constant zero or infirily function).
We emphasize that here increasing means non-decreasing. Since & is convex, continuity is guaranteed the interior of its effective domain and therefore the left-continiity concerns only the point
by = sup < x0).
That "pandering to the masses" is the opposite "tailoring to the tails" was suggested to by Peter Kootsookos.
4

---

Published in Transactions on Machine Learning Research (01/2023)
Note that it is possible that (()) 0 for 0. We denote the convex conjugate of o as v R - R U that is:
(v) = sup - - (xx),
.20
where the rest.riction to the nonnegative half line is a consequence our assumption that + takes on +00 on the negative half line. is a Young function, then so is v. In the following, and w will always refer pair of conjugate Young functions.
Definition 2.2. The Luzembury norm of the random variable is defined as:
X
| inf A A>0:
0:
Ed
A
which is the gauge of the set Iz E M: (+( 1}. Definition 2.3. The Orlicz norm of the random variable X is defined
|| sup e L <[ A.EM.
Observe that the conjugate v appears here. Confusingly, the Luxemburg norm Luxemburg 1955 is some- times simply called Orlicz norm in the literature. In fact, these norms are equivalent by factor of 2:
|| 4 - 1 s 2
and thus they are finite for the same set of functions. Therefore it is inconsequential whether one takes the Luxemburg or Orlicz norm in the following definition.
Definition 2.4. The Orlicz space L is given by:
c = Ex 1 < co}.
For any Young function 4, the Orlicz space _ is a Banach space, iee. complete normed vector space. It is furthermore a Banach lattice, that is, it is monotone in the sense that:
X S (Y] P-a.e. = | < ||1118.
Important in the following development will be an identical expression for the Orlicz norm, which is also called the Amemiya norm (Hudzik & Maligranda| 2000):
X
||
inf t>0
1
E&
(2)
This representation is the key to compute Orlicz norms in practice.
3 Orlicz Regret Measures
The problem with Orlicz norms in setting is that, in virtue of satisfying the properties of norm, they treat losses and gains in the same way. We follow the machine learning convention that losses correspond to positive values, hence gains are negative values. Since we aim for an asymmetric setup, we introduce Orlicz regret measures as the asymmetric variants of Orlicz norms and link them to f-divergence risk measures.
3.1 f f-iiveggences
First, we define the function class which we consider in the following and their associated f-divergences. Definition 3.1. A divergence function is proper, lower semi-contin semi-continuous, convex function R- R U (00), which satisfies f(1) 0, (0) and the supercoercivity condition lim [(x)

---

Published in Transactions on Machine Learning Research (01/2023)
The conditions that f(0) < and limg-x2 [(]] oo play an important role. They are non-standard restrictions that we impose here and will be further discussed.
We denote the convex conjugate of f as throughout the paper. It has the following properties. Proposition 3.2. The conjugate g is finăte on E, conver and increasing.
Proof. The statement, which relies on the supercoercivity of J, is well known and proof is included in Appendix A.1.1
]
Definition 3.3. Given a divergence function f, the f f-divergence between probability measures Q and is defined as:
d)(O.P)= = /
dQ
dQ
d Epf
52
dP
if Q is absolutely continuous with respect to P, and toc otherwise.
dQ A simple shift of perspective, which will be important in the following, is to view ap (w) as single random variable rather than as a fraction of two densitics.
We write & for the set of divergence functions I and By for the subset of divergence functions f, which are also valid Young functions. We call elements of By Young divergence functions. Since we assumed nonnegativity, the only difference to Young function is that general divergence function f might be posilive and decreasing for 0 x <. To given divergence function J, can associate canonical Young function f by setting:
f(x) -w)=(1r)
Lo ((x)) 1>1,
05051
as in (Dommel & Pichler 2021p. We write for the convex conjugate of f. We will see (Proposition 3.11p that it stands in a close relationship to the original f. For various levels of risk aversion E > 0, we define the shorthand notation (x) = (z) and ge (3) = Eg(ey), which is the convex conjugate of fe- The value of is not of thcoretical importance: if f is a divergence function, so is fe.
divergence, where fKL = x log(x) - (x-1). It can also be obtained as f(x) = x log(æ), but the affine shift guarantees nonnegativity without altering the divergence. We complete its definition implicitly by setting (K)(O) lim fKL(2) 1 < co. The induced f-divergence is
Example 3.4. prominent example the Kullback-Leiin
/ log
dQ dP
(Q, =
aQ.
The conjugate of SKL. is (y) exp(y) 1. Example 3.5. The x² divergence is given by fx²(x) (xr - -1)2.
Then the divergence is
d2,(P /
[Q
dP-1. 1
dP
8
Note that the conjugate of fx² is gx2 max(-2, v) + maxx(22,)) 22, negative half line. In some tables, the reader will find that g(v) y + v²/4, which is the conjugate if the effective domain of f(x)= (x - 1)² is R. The difference is crucial in our asymmetric setting.
/4, since take to be infinite on the
Example 3.6. Consider the pathological divergence function
05051
=(.))=
0 1>1
We later observe that the corresponding divergence risk measure in is simply the expectation E.
Forr simplicity, we shall the "risk aversion level" thronghout. However, divergence risk measure may also capture the arguably distinct phenomenon of ambiguity aversion.

---

Published in Transactions on Machine Learning Research (01/2023)
Example 3.7 (Shapiro (017)). ) Given some [0, 1), we slightly modify the previous example
Scvar, (101
The corresponding divergence risk measure in is the conditional value risk CVara. which is the expectation in the upper (1 a--tail:
CVar, = 1 [" Fx dq.
For continuous Fx, this can be written in the more suggestive form ("tail-conditioall expectation");
CVara(X) = E [ (a)].
In the machine learning community, CVara has recently received significant attention, see for instance Takeda Sugiyama 2008), Fan et al. 2017) and (Curi et al. 2020
3.2 Definition and Functional Properties
We now define the Orlicz regret measure, which has similar propertics to norm, yet is fundamentally asymmetric in that it treats losses and gains differently. We will find that it has a very close relation to the divergence risk measure in 11
Definition 3.8. Given a divergence function f E 3 with conjugate 9, the Orlicz regret measure of X E M is defined as:
1 Bg (7)
Vy(X) inf 150
Intuitively, it is g here that plays the role of the Young function of an Orlicz norm. In contrast to the Orlicz norm in ( there is no use of an absolute value here; hence the behaviour of on R (R+) determines the treatment of gains (losses).
Proposition 3.9. For any divergence function f E B, the regret V, has the following properties VX, Y E M::
VI. (A(X) AV2(X) VAER+
(positive homogeneity)
V2. V,(X + Y) 5 VX() - V2()) (subadditicity)
V3. X S P-a.e + Vx(X) VY) (monotonicity)
V4. V,(X) 2 [
(aversity)
V5. If X 2 0, then V2(X) = if and only X
(nonnegotive point-seporating)
Proof. The proof is included in Appendix A.1
It will prove convenient lo define Orlicz regret measures for varying levels of risk aversion e. By a simple scaling, this parameter can be hidden in the function itself. Thus we can express the regret either by hiding the dependence on e or by making explicit.
Proposition 3.10. With fz( = (x) and its conjugate ge (v) }g(Ey), we have
)
X
X
Vg, (X) inf t
1 Egs
inf ē - Eg E>
Proof. See Appendix A.1.3
Recall that 7 is the convex conjugate of J, where f is the Young function canonically associated to
7

---

Published in Transactions on Machine Learning Research (01/2023)
Proposition 3.11. Let f E & divergence function with conjugate 9. Then it holds that g :RR B Ufoo), defined as:
g(v) = f0 0
5 0 y y>0 0.
((())
is the conjugale of I and it is volid Young function when restricted to the domain R+
Proof. The proof is in Appendix A.1.4
We use 11 to denote the Orlicz norm, where the Young function is the restriction of g the nonnegative domain R A coherent regret measure (we define the term generally in Section 3.4) canonically induces norm Pichler 2013
Proposition 3.12. Let f 3 a divergence function. Then setting
|x = x)((
defenes norm the space Vo {X M 1|x||,, x0). In fact it holds that || || and V. f9 Furthermore, Vg is finâte the space Vg-
Proof.
X
|| = VV(II) = inf
+ Eg
820
This is the Amemiya expression of the Orliez norm with Young function g. as g only ever evaluated the nonnegative domain. Therefore it holds that || ||xI|g. The finiteness of Vg Vg. that is, X E V Vx(X)= < oo, follows from (Pichler 2013 Prop. 5), essentially from monotonicity of Vg, J
Corollary 3.13. Let f E & a divergence function. Then the norms || llv, and || llv, are identical.
This is due the inherent asymmetry in our story: care about loss tails, that is, right tails. Consider the meaning of f( d)): the behaviour of f(x) for 0 < < 1 concerns the situation where Q underestinates with respect to the base measure. In contrast, f(x) for x 2 specifies the penalty that overestimation incurs. Due to the absolute value, the Orlicz norm is based only on the treatment of losses. Since V, and V2 treat losses the same way, the potentially different bchaviour for S x < disappears in the corm-perspectiv. To capture this desired view, we have imposed the constraint that f())) < co: underestination may only be punished "finitely", hence we can (up to equivalence) neglect underestimation in our approach.
Furthermore, the chosen risk aversion level of the regret mcasure is not essential, as the resulting norms equivalent for any choice of risk aversion level.
Proposition 3.14. For any risk aversion level, the norms induced by the regret measures are equivalent. Let < E1 E2. We have
E2 _ 1|x| 81
||ll, 5 <||X|
V
Proof. The proof is in Appendix A.1.5
Remark 3.15. Vg is indeed the largest vector space, on which Vg is finite. However, this does not mean that X & Vg = (X) oo. This is due to the gain-loss asynmetry: an X with a heavy left tail might have (X) < co, but Vy([X])= = oc and therefore X V. Since we are concerned with right tails, this subtle distinction is inconsequential and we can focus the natural domain Vg.
8

---

Published in Transactions on Machine Learning Research (01/2023)
3.3 Envelope Representation
Due to subadditivity and positive homogeneily, the Orlicz regret measure Vg possesses an envelope repre- sentation (or dual representation). This insightful representation will, in our case, illuminate the nature of an underlying tail reweighting mechanism in Section 4 As prelude to the full dual representation, we first consider nonnegative random variables. We denote by C the cone of nonnegative random variables in an Orlicz space L
Proposition 3.16. Let f E 8 a divergence function and I be the conjugate of the associated Young divergence function f. For nonnegative X, we have:
(x(X) = sup [[ Ze ct 22 22 220 = VX E7.
Proof. From Proposition 3.12 is simply the definition of the Orlicz norm ||
know that for nonnegative X, we have Vg(X) || Then the statement
The general envelope representation is similar.
Proposition 3.17. Let f E B divergence function. Then the following envelope representation holds
V sup( Ze c¹ 22 0, [2()) 1 E L..
Proof. The proof is in Appendix A.1.6
We refer to the Z over which the supremum ranges dual variobles. While the dual variables from L the primal random variables, for which V, is finite, are from L To understand the mcaning of envelope representations more generally, we turn to the general theory of coherent regret and risk measures.
3.4 Coherent Regret and Risk Measures
The framework of coherent risk measures Artzner al 1999 Delbaen 2002 Föllmer & Weber 2015 aims to provide tail-sensitive risk control. The goal is to summarize the risk inherent in financial position, conceptualized as a probability distribution over losses and gains. Here, there is a fundamental asymmetry: exceeding the expected loss is worse than the converse, hence the focus is on the right tail of the distribution.
In the search for possible replacements of the expectation as the tool to mcasure the risk of distribution, the question of which properties suitable replacement must satisfy arises. Artzner al. 1999) proposed the following influential axioms for coherent risk measures VX,Y
C1. R(AX) = XR(X) V R
(positive homogeneity)
C2. R(X + Y) < R(X) + R(Y)
(subadditivity)
C3. X < Y P-a.e = R(X)= < R(Y) (monotomicity)
C4. R(X + +) R(X)+c E (translation equivariance)
These axioms can be motivated from a financial viewpoint, but also from broader considerations regarding decision making under uncertainty; closely related are the works of Gilboa & Schmeidler (989 in economics and Walley [191 in improcise probability.
The divergence risk measure 1 satisfies all of these and is therefore coherent risk measure. In rastr the Orlicz regret measure satisfies Cl C2 and but not translation equivariance (C4p. The Orlicz regret mcasure can be thought of as "asymmetric norm" (hence not norm), where translation equivariance
We translate the axioms loss-hased orientation. Also, Artzner al. 1999 worked with finite hence they stated with the quantification Vi as opposed P-a.e.

---

Published in Transactions on Machine Learning Research (01/2023)
is undesirabl. The class of such functionals has been captured by Rockafellar & Uryasev 2013 the coherent regret mensure.
Coherent regret measures and coherent risk measures are directly characterized by their envelope represen-
tations. C1
and C2 together with an appropriate closedness assumption, imply that such an a support function.
Let. 3 and its conjugate.
Corollary 28), Arai 2010 Theorem 1)), A proper functional R L R+ U { which satisfies Ci C2 C$ and is order lower Souer-semicontimuuu allows the envelope represenlation
Proposition 3.18 Biagini & Frittelli 2009
R(X) sup E| ZEZ
Z c4, where 2, = ZE c =Z 0 P.a.e..
f==ZZe
with some set Z, called the envelope of R, of nonnegative random variables. If is furthermore translation equivariant C4p, then Z - {z c. E(Z) 1}.
Thus the monotonicity of the Orlicz regret measure can be directly observed from its envelope representation. It is, however, lacking translation equivariance. For the use of coherent regret mcasures in risk-sonsitive regression, see [Rockaeellrr & Uryasev 2013
3.5 From Regret to Risk
An Orlicz regret measure is in some sense like norm, but which distinguishes positive and negative orientation. However, it lacks the property of translation equivariance, which is desirable for replacement of the expectation as an aggregation operator. Yet there is a simple way to canonically construct a coherent risk measure from a coherent regret measure. In this way, the f-divergence risk measure arises from the Orlicz regret measure. The f-divergence risk measure defined as:
(X) == sup [Eo[X] : Q Q. ()
For reasons of consistency, we use the conjugate of f, which is g, in the subscript. Indeed, the conjugate is the appropriate object here, the following proposition shows. A dual variable Z which satisfies Z 2 0 and E[Z] =1 is valid probability density, hence we introduce
(z(A) = f Z(w) dP(w),
which is guaranteed to be probability measure.
Proposition 3.19. The divergence risk measure (X) is given by an infomal convolution of the Orlicz regret measure Vg,, Let X L Then:
Rg inf HER
Vg, -1)
(4)
X
inf "ER,1>0
t
1 4 Ege
X
inf AER,CS0
E ū + Eg
(5)
Rockafellar & Uryasev 2013) further demanded the aversity condition V(X) > B[X] for X 0. For consistency, weaken aversity to weak inequality. Then it is in fact consequence of "rearrangement invariance". see Proposition 1.3
We
only consider rearrangement invariant colurent regret and risk measures throughout the paper. We remark that Rockafellar & Uryasev (0033) do not use the term "coherent regret measure" itself, but instcad define regular measures regret; when adding call colerent by analogy coherent risk measures
positive homogeneity,
8 SSee Blagini Frittelll) 2009) for this technical property. We also refer (Cheridito & 2009
discussion of risk
measures on Orkez hearts, which are particularly interesting subspaces of Orlicz spaces.
10

---

Published in Transactions on Machine Learning Research (01/2023)
Its envelope representation is:
(X) ssup E[XZ] Z E , 220,22220001221111112054,
=sup ( [Eez Z e 2
sup [Eaz : c 102 2, S 5 (Y) VY E c
(6)
Proof. The proof is in Appendix A.1.7
We typically write Rg with e = in the following. The infimal convolution in (4 ensures translation equivariance the resulting functional. The combination of monotonicity and translation equivariance then guarantees that the dual variables are legitimate probability measures, that is, Z 2 0 and E|Z] = (see for instance (Fröhlich & Williamson 2022) An interesting perspective also provided by (6) the envelope is the set of linear risk measures, ic. expectations, which are L'-ppintwise dominated by the regret measure. In the language of the imprecise probability literature, this process of projecting from Vg to Ry by forming supremum over the set of dominated expectations is known as natural ertension (Walley [991 Troffaes & De Cooman 2014h. This provides the rationale for interpreting the envelope as an ambiguity set and embeds the divergence risk measure in the field imprecise probability.
The infimum-based representation in under various technical assumptions, has been around in the literature (Ben-Taa & Nemirovski 1999 Bayraksan & Love 2015 Shapiro 2017 Dommel & Pichler 2021) However, derivation from the Orlicz regret mcasure provides new perspective and intuition for the result; remark that Dommel & Pichler (2021), themsel inspired by |Ahmadi-Javid (2012) and Ahmmdi--
Javid & Pichler (2017), have investigated the divergence risk measure in the context of Orlicz spaces, but without reference to an undorlying Orlicz regret measure.
Example 3.20 [Bayraksan & Love (0155)) With fovar, = 110,1/(1-a)] obtain the well-known infimal convolution expression of the conditional value at risk:
CVara
inf μ + E[((X - -1)+],
WER
where the special case a 0 recovers the expectation.
Similar to the Orlicz regret measure, the divergence risk induces a norm by setting ||X||R, == Rg (IXI), with the induced space R, (X+ E M: R2(IX) so). Then the following holds:
Proposition 3.21. The norms || [|R, and || llv, are equivalent. This implies that the spaces Vy and Rg are identical.
Proof. Since 11v || Theorem 4.5 in Dommel & Pichler 2021 is applicable. The authors did assume finiteness of f, but this is not required in the proof of the statement.
As consequence, the divergence risk measure is finite on V/ = RR and this is the largest vector space, on which it is finite (cf. Remark 3.15
Corollary 3.22 (Dommel & Pichler 2021 ). Let f E 3. Then the divergence risk measure is equivalent to its corresponding Young dévergence risk measure, that is, -1, is equivalent to ll,,-
Proof. The statement holds since ||. IIn, and || I|,,,,, are equivalent and || 1lv,, = 11 11-, (Corollary 3.13
Corollary 3.23 DDomell & Pichler (2021) Consider fea Seg 3, that is, variants of the same underlying divergence function differrnt risk oversion thresholds. Then || -111,,,, and || I|R are equivalent.
Proof. This readily follows from the equivalence of the underlying Orlicz regret measures.
11

---

Published in Transactions on Machine Learning Research (01/2023)
4 Rearrangement Invariant Banach Norms and Fundamental Functions
We now embed the Orlicz regret measure and the divergence risk measure in the framework of rearrangement invariant Banach function spaces, which provides more insight into the envelope representation and allows us to investigate equivalence relationships of the induced spaces.
A desirable property of a coherent regret or risk measure is rearrangement invariance, which is known as law invariance the risk measure literature. The characterizing property is that a rearrangement invariant regret (risk) measure depends only on the distribution of the random variable, not on the order which outcomes are associated with clementary events w. To capture this notion, we define the distribution function px IR+ [0, of a random variable X E M
mx(A) = P([W :Q: [X(w] > 사Y).
For nonnegative random variables, this decreasing (non-increasing) and right-continuon function is just the survival function Fx- We say that random variables X and Y are equimeasurable if their distribution functions are the same, that is, μx (A) = My (A) AA 2 0. For each X E M, the right-continuous generalized inverse of its distribution function X* [0, 1] RR+,
X'(w) = inf 0: (x S ww),
is equimeasurable with X itself. We call X the decreasing rearrongement of X. In the language of probability theory, X is just the quantile function backwards: (²( F -t). As an intuition, X is the continuous analogue of sorting list of values in decreasing order.
4.1 A Primer on Rearrangement Invariant Spaces
Definition 4.1. A Banach space (X, []) is called rearrangement invariont if (Rubshtein et al. 2016
R1. X < YY| and Y E X, then X E & and || S ||Y|].
R2. If X and Y are equimeosurable and Y e X, then E X' and ||||| ||||].
We call such norm || rearrangement invoriant (r.i.).
From their definitions, is clear that ||- llv, and ||. RR, are r.i. norms and therefore V, and Rg are r.i. spaces, In general, any coherent regret measure or risk measure, which satisfies R2 induces legitimate ri. norm by taking absolute values.
Example 4.2. The Lebesgue spaces CP, 1 p oo, are r.i. spaces, which are in particular Orlicz spaces.
Interestingly, any r.i. space is in between L and - due to the following embedding theorem.
Proposition 4.3 Bennett & Sharpley 1988 p. 77)). Let X be any ri space. Then:
- CC
and the norms are the relationshi
X < 11 S c V
Finally, we need the concept of the associate norm, which linked envelope representations.
6This is the standard terminology in the rearrangement invariant Banach space litcrature; it is not to be confused with the cumulative distribution function Fx- Under the assumption that [|]| 1, otherwise renorm first with the multiplicative factor 1/1
12

---

Published in Transactions on Machine Learning Research (01/2023)
Definition 4.4. Let (x -1,1)-1xx) Banach space, where || ||x satisfies the monotonicity condition
R.
Its
associale norm || || is defined as:
|X sup { [x(w) de 11 s 1, <1,
EX}
The associale space is X" = (X E M: 1| < oo).
If I |Ix is furthermore an r.. norm (satisfics
R2
then the associate norm can be expressed in terms of
decreasing rearrangements.
Proposition 4.5 [ennett & Sharpley 1988
p. 60)). Let || |lx be on r. norm. Then its associate norm
|X|| - sup {L"" X (w) dw 11 S 11,
This remarkable result asserts that for r.i. norm, the associate relationship can be understood in terms of certain admissible quantile rewcightings; recall Z (w) = Fiz((w) (w)
Example 4.6 (Bennelt & Sharpley (1988 p. 275)). For a pair of conjugate Young functions and V, consider _ with the Luxemburg norm associate norm.
Its associate space is L with the Orlicz norm || || as the
Example 4.7 (Bennett & Sharpley 1988
p. 10)). The Lebesgue spaces L": let 1 S p S oo and the norm
|l 1l... Its associate space is L" where 1/p + 1/q 1, and the associate norm is |l Iloe. Note that p 1 is paired with ==
4.2 Fundamental Functions of Orlicz Regrets and Risk Measures
Rearrangement invariant norms have widely varying tail sensitivity. For instance, the L norm (expectation) is tail-insensitive whereas the L norm (variance) is rather tail-scnsitive. To "stratify" the space of all rearrangement invariant norms, Bennett & Sharpley (1988) and Pröhlich & Williamson 2022 emphasize the importance of the fundamental function, one-dinensional function, which characterizes indeed fundamental properties of the norm. For instance, its derivative at the origin can be used to characterize norm equivalences.
Definition 4.8. Let X be an r.í. space with r.ú. norm ||. lxx. Let Ar S Q be an arbitrary Lebesgue measurable set with measure P(A:)= =t. The fundomental function Φx : [0, + R+ is defined as:
xx((1) =xxx)==lxx+1l1xx 1|xx
Due to rearrangement invariance, the choice of A, is arbitrary. We will also speak of the fundamental function of a coherent regret (risk) measure, which due to nonnegativity of indicator functions just coincides with the fundamental function of the inducod norm.
Proposition 4.9 Bennett & Sharpley
1988 p. 67)). For any r.d. space X, Φx satisfies:
Φx is increasing.
t (x(1)/t decreasing.
6x is continuous except perhaps at 0 and (x = 0.
A function Φx satislying these properties is called guasiconcave. Every increasing concave function, which vanishes only at the origin, is quasiconcave, but a quasiconcave function need not be concave in general.
Example 4.10. The || lle. norm has fundamental function dz- (t) = t. In contrast, the || Ile> norm (supremuum) has dz> (t) = X(o.1]: which immediately jumps to 1, hence it is not continuous at 0. It holds that any fundamental function lics pointwise in between those of L and e (ef. also Proposition 4.3
We denote the fundamental function of the Orlicz regret as Φv, (t) Vo(XA.) IIxx,llv, and similarly, the fundamental function of the divergence risk measure on, (1) Rg(xa,) 11xa,11R, We have that
13

---

Published in Transactions on Machine Learning Research (01/2023)
ØR (1) due to translation equivariance of Ry, which implics Ry(c) Vc R (Rockafellar & Uryasev 2013) We begin by characterizing the fundamental function of the Orlicz regret measure. We use the fact that left-continuous function f possesses generalized left-continuous inverse (y) = sup(2: f(x) < y).
Proposition 4.11 [ossml & Millle-Wicharde (2011)) ). Let f B. Then:
Φv, (t) (t)
is the fundamental function of the Orlicz regret measure and it is concave.
Proof. The proof is in Appendix A.2.1
Proposition 4.12. Let f B. Then Φv, and oR, both have the following properties (denote by & either of them):
FF1. ((0+) == lim (t) 0.
FF2. If f is finite Rt, then ('(0) lim4jo o(t) 00. Otherwise, d0()) oc.
Proof. The proof is in Appendix A.2.2
These properties have direct consequences for norm equivalence relationships. Corollary 4.13. Let f E B. Then _ VV Rg. where g is the conjugate of f.
Proof. As the smallest rearrangement invariant space, ~ V Rg holds. But it is indeed strict subset: from Haroske 2006 Prop. 3.4 ii)) we know that Rg C L.0 holds if and only if limeoo कτ) < cc. But
limago (0() = 0 and therefore limmgo क0) xc.
Corollary 4.14. Let f 3. f is finite, then Vg = RRg & Ct. In contrast, if f not finite, then V, = Rg L
Proof. As the largest rearrangement invariant space, Vg Rg C L holds. However, we have from Theorem 31 in (Fröhlich & Williamson 2022), that all rearrangement invariant norms with d' d'(0) < are equivalent to C.. In contrast, if o' (0( = oo, the norm cannot be equivalent to L (Fröhlich & Williamson 2022 Corollary 32) and thus Rg C.
These corollaries are well-known in Orliez space theory, but it is instructive to see how they easily follow from properties of the fundamental function.
A general formula for the fundamental function øn, the divergence risk measure scems infeasible How- ever, we observe that in the case of a Young divergence function, has a simple expression.
Proposition 4.15. Let f By. Then the fundomental function of the divergence risk measure is
OR, (t) min 1,tf (t)
that is, a capped version of the fundamental function of the corresponding regret measure.
Proof. The proof is in Appendix A.2.3
Example 4.16. For the Young KL divergence, where fKI( = (x log(x) - x + 1). X(1,00s the fundamental function
1(1/t- 1
(l)
(1/t-1)
W
Tt. easy write down equation, but one which typically difficult to solve.
14

---

Published in Transactions on Machine Learning Research (01/2023)
1.0-
0.6
Young KL
Young
OVaraa
0.4
Young KL
0.2
0.2-
Young x CVanaa
0.0-
0.0-
0.8
1.0
0.00
0.01
0.02
0.03
0.04
0.05
The fundamental functions of the Young KL, the Young x² and the CVara=0.2 divergence risk measures, all with 1. In practice, this will be extreme choice of E. The grey dotted line is the identity, Left: the full fundamental function. Right: the fundamental function for 0 S z S 0.05. Here, the difference in tail behaviour is better visible: the fundamental function of the Young KT. divergence risk measure is extremely steep near the origin, whereas that of CVara=0.2 close to the identity, which corresponds to the expectation. All of these fundamental functions are continuous at the origin.
Figure 1:
where W is the Lambert W function.
Example 4.17. For the Young x2 divergence, where fx=(x) = (x 112 X(1,c)); the fundamental function is
(x² min { {1,7+ vij.
Example 4.18. For CVara, the fundamental function is Φcvar. (t) min {1, [ For a =0, ée(() t.
We illustrate these fundamental functions in Figure
The fundamental function characterizes important aspects of an r.i. norm. Of particular interest is d'(0), is exemplified in Corollary 4.14 Furthermore, we have the following result which will be useful in Section 6
Proposition 4.19. Let o: [0. + R+ be an increasing concave function with s(0) 0 and <(0+) 0.
Then there crists an Orlicz norm || 1 which has & as its fundamental function and a supercoercive Young function as the conjugate of V.
Proof. The proof is in Appendix A.2.4
4.3 Equivalences of Orlicz Norms
Through the framework of r.i. Banach spaces, associate each divergence risk measure with its natural space, which it is finite. However, many non-identical divergence risk measures may induce the same space. Then, however, they are equivalent. Recall that two r.i. norms || llx, and || llx, are said to be equivalent if
Ec d' 0: c. x-I|I - || s é' || VX E.M
Two r.i. norms are equivalent if and only their respective spaces coincide (Bennett & Sharpley 7). As an example, we have seen that the parameter does not matter up to equivalence (Corollary 3.23
1988
Why are we interested in norm (non)-equivallences? Whether two norms equivalent or not prima facie seems like very coarse criterion for considering them similar. After all, the binary equivalence notion does not tell us how large the multiplicative constants are. Equivalence means that the norms are finite on the same set of functions: yet in practice, when computing the norm of an empirical loss distribution, we will never encounter infinities, since all real-world samples are bounded (from ) In this way, unbounded tails are a useful, asymptotic idealization used to model the frequency of extreme events.
In a sense, those divergence risk measures with the same space share the same tail sensitivity; although we will later argue that sharing the same fundamental function provides coarser and simpler way to capture
15

---

Published in Transactions on Machine Learning Research (01/2023)
tail sensitivity. Consider what the risk assessment R(X) means: the decision maker considers the tail risk in X to be unbearable, that is, there is no finite certainty equivalent (gain), which in exchange would make the risk bearable. For example, irrespective of the choice of E > 0, decision maker who uses the KL divergence risk mcasure would assess R(X) = oo for a random variable X which is not subexponential, (see Example 5.10) and thus prefer any subex ponbntaallt random variable over it. Thus, investigating equivalence relationships between spaces reveals different attitudes towards tail risk. In general, the more tail-sensitive risk measure, the smaller the induced space.
We briefly consider equivalence of Orlicz norms, require these results in what follows. Definition 4.20 (Rubshtein et al. 20066 p. 208)). Let 91-92 be Young functions. We write 9h > 92 when
(2(x) S (ax),
Vx 1220
for some b 0, a 0.
We write when both 9i > 92 and 92 > 91 holds. We see immediately that I f~ Se for any 8> 0 if f E Br.
Proposition 4.21 (Rubshtein et al. (0016 p. 209)). Let gn, d2 be Young fanctions. The Jollowing stalements are equivalent:
E1. 91 > 92-
E2. L C _
E3. 3e > 0: ΦV (t) S C0V,, (t), Vt E [0, 1].
E4. 3e 0: || S (ll
Example 4.22. The norm induced by the (Young) x² divergence risk measure is equivalent to the L norm, which has fundamental function d2 (t) Vt. Observe that
Vt < t(1 v1/t) Vt E [0,
and ((t) = 1(1 + v1/t) is the fundamental function of the x2 Orliez regret measure. Therefore, the natural r.i. space for the x" divergence risk measure is that random variables X for which [X] has a variance.
Example 4.23. CVara is equivalent to the expectation for all a E 10. As a + 1, it yiclds _ however, and the risk measure the essential supremuu..
Example 4.24. Let h be a Young function, which need not be divergence function, e.g. Mxx = x². Then f(z) = (h(z) - h(1)) X(1, is valid Young divergence function which is furthermore equivalent to h. As an example, h(x) x² is cquivalent to f(x) = ((x - 1) X[1.00); the x² Young divergence function.
Proof. Obviously f is a valid Young divergence function. Consider the definition of the Luxemburg norm
|| inf { x ::
X
M(1)) XP1.00 $1
h
to see that Ch = C
5 Tail-Specific Marcinkiewicz and Lorentz Norms
Our overarching goal is to show how r.i. norms (and hence, coherent risk mcasures) can be constructed which exhibit. a desired tail sensitivity; thereby yielding natural spaces of "tail classes", on which the norm is finite, for instance the space of subexponential random variables. In this section we advance on this idea by showing how tail sensitivity connected to the fundamental function and the envelope function. Intuitively, desired tail sensitivity is a very close correspondence with the fundamental function. For each thus constructed fundamental function, consider the smallest and largest compatible coherent. risk measure.
16

---

Published in Transactions on Machine Learning Research (01/2023)
In the next section, we derive the corresponding divergence risk measure via the same approach, which is then situated in between these extremes or coincides with either of them.
In the language of the imprecise probability literature, the fundamental function specifies coherent upper probability (Walley 1991), which is the restriction of the risk measure to indicator functions. In classical precise probability theory, one can take either the expectation operator or a probability measure as the primitive notion, since applying an expectation to indicator functions yields a probability and vice versa, expectation is uniquely defined via integration. In the more general theory of imprecise probability, this one-to-one correspondence does not hold anymore: fundamental function can in general be extended in multiple ways to a coherent risk measure However, the fundamental function still characterizes fundamental properties of a norm. For the relation of imprecise probability and coherent risk measures we refer to (Vicig, 2008) and (Fröhlich & Williamson 2022). Our approach is to specify a tail sensitivity via the envelope function, which is one-to-one relationship with the fundamental function. The specification of an envelope function least upper bound on the tail behaviour of the random variables, which are contained in the space. In some cases, the envelope function is itself in the space; in others it is a natural limit object, which is "just outside" the space.
Recall that for an r.i. norm, we have the associate relationship:
| ssup (L- x=(w)Z((w de | $1, E x
(7)
By specifying a set Z = 1z" |1 S 1}, we have fully specified the norm in terms of which quantile reweightings are admissible. Conversely, any set of nonnegative and decreasing Z leads to a legitimate r.i. norm (see e.g. Fröhlich & Williamson 2022 Section 4.7). Thus, r.i. norm is fully identified by its set of nonnegative and decreasing functions Z' on [0. 1]. However, the fundamental function, which is a far simpler object than a whole set of 2", already stratifies the space of r.i. norms in useful way regarding tail sensitivity. To see this more clearly, we need to introduce the concept of envelope functions (Haroske) 2006).
Definition 5.1 (Haroske (2006]). ). Let X be any r.i. space. Define Ex : (0, 1] - RR U {oo} by:
Ex(t) xup(X* (t) || $ 13.
Ex is right-continuuus and decreasing, hence Ex Ex-
Recall that since (t) F (1 t), the tail behaviour is encoded in the behaviour $ O.
Proposition 5.2 (Haroske 22006 p. 53)). The envelope function in a one-to-one relation with the fun-
damental function via
1 Ex(t) = (x)²) Vt E (0.
Since multiplicative factors do not matter up to norm equivalence, we shall be concerned with equivalence classes of envelope functions and fundamental functions. That is, similar to E3 we call two envelope functions
equivalent if
3c, 0: e- Ex, (L) S (x) s' Ex, E (0,1).
and likewise for fundamental functions. Intuitively, the envelope function constrains the tail behaviour of random variables in the space. It is however more instructive to change perspective and consider the associate space. We need the following technical result.
Proposition 5.3 Rubshtein et al. (2016 p. 135)). Let X' be an r.i. space with fundamental funclion éx- Then the fundamental function of the associate space X' is oxx (f) (x((]
We may refer to ox as the associate fundamental function. Consequently, the envelope function in the associate space is:
(x sup(z(()) (f) |[] 1).
Ex((t) (1) t
In expectational cases, there only a single extension, see Fröhlich Williamson 2022 Section 4).
17

---

Published in Transactions on Machine Learning Research (01/2023)
Thus, the envelope function constrains the tail behaviour of the dual variables Z" in and it is the least such upper bound, the supremuum. All in all, we have one-to-one relations between 0x,0x, .Ex and Ex; which object is easiest to specify depends on motivation and perspective. In what follows, we often write simply & instead of 6x for the primal fundamental function to prevent excessive subscripts. Essentially, the choice of either of these objects choice of a coherent upper probability underlying an r.i. norm. Also observe that
@x
0'(0) lim 430
lim (x) +j0
The derivative of at the origin has been used to characterize norm equivalences (see for instance Corollar- ies 4.13 4.14 and (Fööhlich & Williamson, 2022 Section 4.9)). Via the envelope function perspective we see that it corresponds to constraint on the tail behaviour of the dual variables.
The envelope function naturally suggests the following approach to the construction of tail-sensitive r.i. norms: specify a constraint on the tail behaviour of dual or primal variables by specifying an envelope function (Ex or Ex²) as the decreasing rearrangement of a reference distribution (e.g. as the decreasing rearrangement of an arbitrary random variable with exponential distribution); then construct an r.i. norm with the corresponding fundamental function. For example, when constraining the quantiles of primal variables X to be bounded from above by the quantiles of an exponential distribution, will arrive at the class of subexponential variables and the KL divergence risk measure.
For any concave fundamental function, we may consider the smallest and the largest r.i. norm, with the Orlicz norm in between. The largest norm is known as the Lorentz norm and the smallest is the Marcinkiewicc norm.
5.1 The Marcinkiewicz Norm
Assume we have specified a legitimate Ex and thus a quasiconcave 6(t) = 1/Ex(t), since a fundamental functions must always be quasiconcave (Proposition 4.9). A first (unsuccessful) atlempt to define a norm, which captures this constrained tail behaviour, is define the following.
Definition 5.4 (Krenn al. (1982 ) Let @(t) (/Ex(t) be quasiconeave fundamental function. We define the Marcinkieuricz quasi-norm
X = sup (t)) ocrsi
This naturally expresses the constraint on the quantile behaviour of X, compared to the reference Ex Intuitively:
M < < co le: x c. Ex-(()) Ee (0.
Here, Ex is the decreasing rearrangement of reference distribution, c.g. exponential. Unfortunately, lar, is not norm, as it does not satisfy the triangle inequality. However, we can define the closely related Marcinkiewicz norm. By integrating the decreasing rearrangement, we obtain the mazimal function Bennett
& Sharpley 1988 pp. 52-53):
(²) = 1
('' (2) dw, t> 0.
Since X*(1) = (1 - (), we find that x*)(1) - CV thus x**(0) is the essential supremun of [X| and (²) = [[]]===]]]] [.
Definition 5.5. Given any quasiconcane fundamental function as:
funciiooo o, the Mareinkiewicz norm || IIa,,,, is defined
|| = sup (((t)*((1)) 0-151
sup
(0(1) CVar)--(XX))-
oct51
We denote the Marcinkiewicz space as Ms = (X E M: |||,,, oo).
18

---

Published in Transactions on Machine Learning Research (01/2023)
This norm also has fundamental function ø. Intuitively, the Marcinkiewicz norm measures the growth of the CVar's, but weighted with the fundamental function. Tn the risk measure community, the Marcinkiewica norm has been unknowingly re-discovered by Pichler 22013] Section 5). The difference between Il,M, and II-IIM, is the use of the decreasing rearrangement (backwards quantiles) vs. the maximal function (backwards CVar's). Subaddivity of the maximal function yields a legitimate norm in the latter case. However, the quasi-norm and the norm are fact equivalent many cases of interest. Recall that Ex Ex,, since Ex decreasing, and the maximal function Ex
Proposition 5.6 (Krenn et al. (1822)) ) Let @ be quasiconcave. The following are equivalent:
M1. Ys > 1: supocrs1 Ex (t/s) ox (6)
<
M2. I- IlM,, and || ll are equivalent: 3K > 00: (X|,, S &1IXI1 S SK- -1
M3. ||óx (t)/tllm, = |[Ex||M, 00.
M4. Ex and Ex are equivalent: 3K 0: Ex(1) (x kxx((), vi (0,1).
M5. Let Φcv (1) 1/Ex(() |1- 11.1 and || MMMocarr equivalent.
Proof. The proof is Appendix A.3.1
A necessary, but not sufficient criterion for these conditions to hold is that ox be strictly increasing. Intuitively, the conditions amount to requiring that the CVar's of the envelope function do not grow too rapidly.
The Marcinkiewicz norm naturally expresses a tail sensitivity based on CVar's. If any of the conditions in Proposition 5.6 are fulfilled, does not matter up to equivalence whether the approach to specify a tail sensitivity is based on CVar's or quantiles of reference distribution: the same space results.
To summarize, approach is follows: specify envelope function Ex the decreasing rearrangement of reference distribution with desired tail behaviour. This amounts to choice of coherent upper probability in form of the fundamental function Φx. Then consider the corresponding Marcinkiewiez quasi- norm, which is lypically equivalent to the Marcinkiewica norm (we later demonstrate that the equivalence "often" " holds). In any case, the Marcinkiewicz norm occupies a special status among all r.i. norms, which are compatible with specified fundamental function. To state the result, we need define its associato norm, the Lorentz norm.
5.2 The Lorentz Norm
Definition 5.7. Given any concave fundamental function 6, the Lorentz norm || IIx, is defined as:
|| = ''' (w) dó(w)
= =xx(0)0(++)+ + S x"(w)o((w) dw.
We denote the Lorentz space as Ag = (X M: ||814, < <o)
The Lorentz norm is fully specified by its concave fundamental function, which indeed is ø. It has intuitive the derivative of the fundamental function is used to reweight the docrcasing rearrangement. Since & is concave, o' is decreasing and thus more weight is put on worse outcomes. Furthermore, Φ is not continuous at the origin, i.e. d(0+) > 0, then the essential supremum (*) receives the fixed weight d(0+). We remark that in finance and insurance, the Lorentz norm appears as the norm induced by spectral risk measure (Acerbi 2002), also called distortion risk measure (Wang 2000), which plays a major role in financial risk assessment.
interpretation:
19

---

Published in Transactions on Machine Learning Research (01/2023)
In fact, the Marcinkiewicz norm || -aae where (x) t/o(t), , is the associate norm to the Lorentz norm Ag. The converse associate relation also holds, but with a technical subtlcty regarding (quasi)concavity (see Section 5.3)
Example 5.8. C is Lorentz space with dc. (t) = t. Diametrically opposed, _ is also Lorentz space with &g (t) = X(o.1)- In the case of L and _ the Marcinkiewicz, Orlicz and Lorentz norm are in fact identical FFööhlich & Williamson
2022 Section 4). In contrast, for instance, L is not Lorentz space, but
Orlicz space.
Given concave fundamental function, the Marcinkiewiez norm yields the smallest (most optimistic) and the Lorentz norm the largest (most pessimistic) risk assessment.
Theorem 5.9 (Bennett & Sharpley 1988 p. 72)). Let X' be an r.i. space with concave fundamental function
Φ. Then the following embedding holds:
As C
and the norms are in the relationship:
| S 1||x||x S || V
We offer the following intuition for the result: both the Marcinkiewicz and the Lorentz norm are in a special, parsimonious one-to-one correspondence the fundamental function Let ø be concave and continuous at the origin. Then the Lorentz norm is defined using only the single dual variable ø', which controls the reweighting (and it is the only r.i. norm for which single dual variable suffices). As to the Marcinkiewicz = 1. In this sense, the Marcinkiewicz norm is exactly
norm, we have the extremal assessment [| x tuned to the associate fundamental function. Thus, the Marcinkiewicz and the Lorentz norm demarcate the space of r.i. norms, which are compatible with a specified fundamental function.
We build intuition for the use of the Marcinkiewicz and Lorentz norms with the following example of the spaces Lexp and LlogI. In this example we consider the class of suberponential random variables and its "dual" (ail class, the random variables with finite differential entropy.
Example 5.10 (Bennett & Sharpley (1988 pp. 243-254)) Lexps LlogL: Consider the envelope function that expresses subexponential tails. The decreasing rearrangement of exponential distribution, which is shifted by 1, is (f) = log(t); the reason for the shift will become clear later. We take this to be the envelope function (t) = 1 - -log(t) and define the Marcinkiewicz quasi-norm:
sup o<t<1 log(t) (t)...
Note that (()) = 1/(1 - log(t)) is quasiconcave. In this case, it is easy to verify that condition M1 in Proposition 5.6 holds, and therefore we may equivalently consider the Marcinkiewicz norm:
|| = sup 1 1 log(t)
X*(t).
octs1
Observe that the CVar's of an exponential distribution (not shifted) are in fact
CV (IxI) = log(1 a
a) Va E 10,
We call random variable suberponential if || oo and denote the space Leep- It turns out, we observe in the next section, that an equivalent approach to capture this tail class is via an Orlicz norm, which then yields the following definition: random variable X is subexponential if 3E > 0 such that:
/ exp (A[X(w))) du Elexp(A[XI]] < <0,
that is, the moment- -ennerttnnnneaiingggg function of [X] is finite for some 0. This typical textbook definition of subexponential tails ( (Vershynin] 2018). In this case, we have equivalence of the two definitions of Texp-
20

---

Published in Transactions on Machine Learning Research (01/2023)
We will later observe that this is due to the fact that the Marcinkiewicz space here coincides with the Orlicz space with the same fundamental function.
The space Lexp is in fact the associate space of the Lorentz space LlogL, defined by:
|| Llogt. == L/ (w) dorlogt(w), (Loogt (1) t log(11)) /t).
This is distinguished space: the space LlogI, consists of those random variables whose CVar's are integrable:
|L <0 = <00 L' CVara((X)) da 00;
which is a very weak condition, hence the space is very "large". In fact, we have the remarkable embedding situation:
- & Lexp _ & LlogI & C², Vp E (1,
Thus, Texp is smaller than any L space, whereas its associate LlogL, larger than any L space, p E (1, oo). This behaviour can be intuited from the fundamental functions as 1 0: the fundamental function of LlogL, is very moderate, while that of Lexp is extremely steep near the origin (see Appendix A.3.2
Furthermore, LlogL consists of those random variables with finite differential entropy 10
S (x(W) log((X(())) dP(ww) < X0;
| <00
which, if X = 4Q dP is legitimate density, is just the KL divergence of Q to the base measure P. Hence we observe that the KL divergence risk measure is finite on Lexp (a very "small" space) and the feasible dual variables are from LlogL (a very "large" space). This means that almost arbitrary tail reweightings are allowed, which entails an extremely tail-sensitive behaviour of the KL. divergence risk measure.
We remark that, confusingly, according to some authors a subexponential random variable is one which has heavier tails than an exponential random variable. Such terminology appears counter-intuitive to us and conflicts with the definition of a sub- Gaussian random variable.
5.3 Quasiconcavity and Concavity
The question remains whether any random variable Y is valid reference distribution as an envelope function, with the identification d(t) = 1/YY*(t) for < S 1. We always set <(0) = 0. Then is continuous at if and only if limego Y (*) is unbounded. Throughout, we shall only consider such Y* (with the instructive exception of Example 6.5). The critical issue is that o must be quasiconcave. For this, need that ø is nonnegative and increasing, which is satisfied for any Y, as Y* is nonnegative and decreasing. Also, we require that + (tt)/t be decrcasing:
(t)) t
1 xx (t)
V-(1 1/o(1)
which is decreasing if and only if t (t)) tt is increasing. Hence, any nonzero Y which ensures that t + (+) t is incrcasing yiclds a legitimate envelope function. Observe that since (t) = -t) this is equivalent lo requiring that ox be quasiconcave (note that ++ (1)tt is decreasing). Therefore it is also equivalent requiring that the envelope function of the associate space X" is decreasing, which of course is necessary for any legitimate r.i. norm. We make observation, which relates to the embedding theorem.
Proposition 5.11. Assume Y is differentiable on (0, 1). Then for ( ox (1) Y (1) L to be increasing it is necessory that Y (t) S (1)
Obseevv the content Bennett Sharpley 1988 p. 224-2445 and combine with the equivalence the Orlicz functions (x log(a))x(1.co) and x log(2) (x 1).
21

---

Published in Transactions on Machine Learning Research (01/2023)
Proof. The proof is in Appendix A.3.3
Recall that L is the largest of all r.i. spaces and has envelope function (t) 7. Since the constraint (*(1) =1 (equivalently, ((1) = 1) can be imposed without loss of generality, we see that unsurprisingly, the envelope function of L is the largest of all legitimate envelope functions.
Working with the maximal function is even simpler: by setting @Cvar (oo 1/Y* for any nonnegative and nonzero Y, the resulting & is guaranteed to be quasiconcave, since t + y (t)-t= (' dw is increasing. Example 5.12. Taking the exponential distribution, Y*(t) - log(t)) fails since t + - log(t)) t is not increasing on (0, 1]; it violates the condition in Proposition 5. However, when shifting the distribution to have minimamm value 1, i.e. (*) = - log(!), we get + log(1)) l, which is increasing on (0, 1]. Of course, such a shift insignifioutt for tail behaviour.
With regard to the quasiconcavity vs. concavity distinction, we remark that it is not of essential importance. For instance, the Lorentz norm requires concave in its definition. In this case, one use the least concave majorant A of ø, since quasiconcave function & is always equivalent to its least concave majorant di: _ō S Φ S o (Rubshtein et al. 2016 131). In fact, any r.i. space with quasiconcave fundamental function can be equivalently renormed to have its least concave majorant as the fundamental function (Bennett & Sharpley 1988 p. 71); (Pick et al. 2013 p. 268), so the quasiconcave concave distinction not essential.
Finally, we note that even di is concave, it may be that ox is only quasi-concave. The associate relationship of the Marcinkiewicz and Lorentz space is preserved using the least concave majorant then (Rubshtein al. 2016 p. 147).
6 Tail-Specific Divergence Risk Measures
In the previous section, we have constructed Marcinkiewicz and Lorentz norms, which capture desired tail behaviour. In this section, we proceed with the construction of tail-specific Orlicz norms, which yield corre- sponding divergence risk measures. We focus on the construction of Orlicz norms for a specific tail sensitivity; from this, equivalent Orlicz regret and divergence risk measures can easily be derived (see Example 4.24 and recall Corollary 3.21).
Remark 6.1 Rubshteii et al. (2016] p. 162, p. 209)). Let and 5 be equivalent concave fundamental func- tions. Then As and Az are equivalent; Mo and Mj are equivalent; and the Orlicz norms with fundamental functions & and & are equivalent.
We follow the same logic as in the previous section, beginning with the specification of a fundamental function or, via its one-to-one correspondence, an envelope function. this section, we denote by and I a conjugate pair of Young functions, which need not be divergence functions. Intuitively, they play the role of f and g. respectively. The corresponding fundamental functions of the Orlicz spaces are of (t) and Φv (t) (t), where we use the L to emphasize that it corresponds to the Luxemburg norm in L. Then:
| sup {L" (w(w)Z((w) dw Z 2
0, D. ( 2
$1} EXEL"
In fact, the Orlicz norm is the associate norm to the Luxemburg norm, that is:
| sup (L x (w) dw 2 , 2.
AX
(9)
To achieve desired tail sensitivity, we aim to constrain the tail behaviour of dual variables Z* in an Orlicz norm (a divergence risk measure, respectively). For example, for the LlogL divergence risk measure, we allow dual variables to be from Lexp- Converscly, for the Lexp (KL) divergence risk mcasure, we allow dual variables from LlogL, which merely need to have finite differential entropy. We demote the envelope function in _ of the Luxemburg norm, indexed by the Young function Φ, as:
F = sup 1| 1 1).
22

---

Published in Transactions on Machine Learning Research (01/2023)
By specifying this envelope function based on the decreasing rearrangement of some reference distribution y", setting F/ = Y we have the least. upper bound on the tail behaviour of the dual variables in (9) in terms of Y and thereby delimit possible quantile reweightings. The more extreme our envelope function, the higher the reweighting can be. We assume that the envelope function is finite for all t E (0, 1] and unbounded for & = (otherwise there would be no tails).
Proposition 6.2. Choose E : (0. + lR+t as the decrensing rearrungement of some reference distribution, for which E (0+) = oc, (t) (t)-t is increasing and concave on (0, 1] and which satisfies Φy (0+))) 0. We set (0) = 0. Then there erists
Orlicz norm Il which has fundamental function Φv- The conjugate Young function of V, which is D induces L with envelope function E Furthermore, & is finite and supercoercive and therefore Va is finile.
Proof. Let do be the fundamental function of I1 that is, (t) = t :::ia111//0 (1/t). Denote by ( t/ow(t) the fundamental function of the Luxemburg norm with Young function Φ. We know that
1 1 (t)
(t
2b(()
1/0v(1)
by Proposition 5.1 and Proposition
5.3 recall that the Orlicz norm |1 is in an associate relationship to
11.. Therefore:
(v(() -000==EE((0)) (t) = to (1/tt, e (0, 1]
(1/xx) - (x). Ve E (c)
By specifying a legitimate (t), we thus obtain 4 (x) for x 2 (1). Of course, the behaviour of & on < x < (1) is immaterial for tail sensitivity. As we have seen before, Orlicz norms have concave fundamental functions (Proposition 4.11), hence we have to ensure that (t) (t) t is a valid concave fundamental function. Given that we specified some F%6 which yields concave @it)= F. (t) 1 and thereby specifies the behaviour of f on x 2 (1), can always find compatible 4, so that 4 is indeed legitimate Young function? The answer is affirmative and a constructive approach is in the proof of Proposition 4.19 For the supercoercivity of 4, see the proof of FF1 in Proposition 4.12 For the finiteness of 4, observe that (0+) = xo = dy (0) = co and then see FF2 in Proposition
4.12
Of course, the primal approach is also feasible, where the envelope of the Orlicz norm in - is specified as (t) = sup(2"(t) || - 1 = 11/0 (t), hence we then demand that (t(() is concave and o = O. If we at least have a quasiconcave Φ, that is, if + ou == (t) t increasing, we may take its equivalent least, concave majorant (see Section 5.3 Thus, to achieve tail sensitivity we merely need that t @v = (t()) -t is increasing, since t + de (t)/t (t) (t) is decreasing for any Y.
We have seen that specifying least upper bound on the tail behaviour of primal or dual variables yields a corresponding Orlicz space. Assume that we have therefore obtained a Young function tb and an equivalent Young divergence function f E 3 via Example 4.24 To further motivate our approach, observe that if Z $ L no finite e will suffice for to satisfy the f-divergence constraint in the envelope representation of the divergence risk measure. The converse direction is more subtle due to the Ag condition (see Definition A.6). Thus, our approach is about the stracture of the divergence ball, essentially given by the tail constraint, not its particular size, which of course is also significant in practical implementations.
Example 6.3. - = LlogL: we choose the dual envelope (t) to be the decreasing rearrangement of random variable with exponential distribution, shifted by 1, that is, (t) = - -log(t). As desired, + oz (t) = (t) / is increasing and concave and de (0+) = 0. Then Fb(1/x) = 1 1og(1/x) क (x),
x 2 1. This is satisfied by the Young function
(xx)
x
0 < x <
(exp(x11 1) I 2 1,
which is in fact equivalent (in the sense of Scction 4.3) obtain the LlogT. Orlicz norm as ||. 110-
the function ((x) exp(x) 1. Therefore we
23

---

Published in Transactions on Machine Learning Research (01/2023)
Example 6.4. L = Lexp: assume that we want to construct Orlicz norm where the primal variables X are constrained to be subexponential. This can simply be obtained from the previous example by duality. To provide more intuition, we derive it in another way. When choosing the primal envelope function as Φy (t) = (t) = 1/(1 - log(t)) based on a shifted exponential distribution, we face the problem that this is only quasiconcave. However, we have the equivalence the concave function L (+> log(1/e+1))
-- log(() log(1/1+1) 2 1 log(() WE (0.
Setting (1) (1) = 1Φ (1((() yields an unwieldy form for ʃ innolling the Lambert W function instead of (x)) = x log(z) - (z - 1)) albeit equivalent. Instead, we consider the Luxemburg norm || l|
with envelope function (t) = (t) and choose (t) = log(1/t + 1), where the fundamental function
is (Rubshtein et al. 2016 p. 177):
(4 v v-1((11)) (1/f) = -011/)==og(1111 F WE (0, 1].
Thus we find that (y)) = exp(y) 1 satisfies the equation. Recall that the Luxemburg and Orlicz norms with the same Young function are equivalent. follows:
X 1 < < - YE > 0: [[exp(A[X[)] - < oo,
which is a standard definition of a subexponential random variable (Vershynin 2018 cf. S
Example 6.5. _ = C); Expectation and CVar,: in contrast to the assumption of Proposition 6.2 that (0+) = 00, assume we want to allow only bounded quantile reweightings, i.e. dual variables from L For instance, let (t) = t, (0, 1], so that all quantiles of the dual variables are bounded from above by 1. Therefore 1 = Φ (1/t), V E (0, I], whence it follows up to equivalence that.
0
0
((x)
OC
I r>1. 1,
which is also divergence function. Let g(x) = (x)) for x 2 0 and g(x) = 0 for x < O, which is the convex conjugate of 4. In fact, g(x) r).. Then || = X]] and (X) E[X*]. This is nothing more but the L - -0 associate relation: the function tb is the Young function whose Orlicz space is _ thus its associate space is L.
Similarly, when allowing reweightings up to 1/(1- a), i.e. (t) 1/(1 ((1--00 a), we have V2(X) ==
and Rg(X)) CVara(X) via the infimal convolution in Section 3.5 This does not change the primal space, which is still Ct, whose associate space is ~ The situation is clarified by the envelope representation
CV) =sup| {L X" (w))Z((() dw E|Z]
1,05 < 2 1//1
Example 6.6. Pareto, 22: The decreasing rearrangement of Pareto distribution with shape parameter p (typically denoted a) is (*) = - Set the dual envelope function F1((1) = -_ Then ((x) = xº. For instance, p 2 yields the x² divergence risk measure (up to equivalence, see Example 4.22). Of course, _ _ and L is then just L where q is the dual exponent. Since 2 is self dual, that is, 1/2 + 1/2 1,
we have that the x² risk measure is finite L The same procedure naturally works for any p oo, as +++ t 7 t is increasing, giving the space L..
An interesting case is p 1: Then (t) 1/t, but t Et(() t is not continuous at 0. Then o <-1(x) x and (xx) = I, which is not supercoercive. In fact, x(x) = x is the Young function whose Orlicz norm is the expectation E[| []. By duality, we would obtain the essential supremum as ||. 11 where
0 x $ x>1.
v(x)
x
24

---

Published in Transactions on Machine Learning Research (01/2023)
Due to involved technical complications, however, we have excluded such norms (risk measures, respectively) from our framework and assumed supercoercivity. In contrast, as p 0c, we obtain ( 1 and we recover the expectation (CV - as in the previous example 6.5
A curious phenomenon can be observed: when constraining the quantiles of dual variables by Parcto distribution with p- = 2, we obtain the space L However, the Parelo distribution itself only has a finite variance when p > 2. Similarly, p=3 3 yields the L space, but the Pareto distribution only has a Kurtosis for p > (and so on). This means that the random variable Y whose decreasing rearrangement we used to construct the Orlicz space via the envelope function E = Y is itself not in the space L although it is the least upper bound on the decreasing rearrangements {t Z* |12 S 1] (recall Definition 5.1 Thus, the envelope function is natural limit object. For example, of those distributions which have a finite variance is the Pareto case is the pathological = 1: the Pareto distribution only has an L norm (an expectation) for > 1. In this way, the extremely heavy-tailed Parcto p = 1 distribution with envelope function t + 1/t is as extreme as it gets, as the correspouding Orlicz space is the largest of all r.i. spaces (recall Proposition 4.3).
might say informally that the "limit" 2 distribution. An interesting special
The question of when the reference distribution is itself in the space, ie. E Y E L has surprisingly simple and insightful answer, which is related to M3 in Proposition 5.6
Proposition 6.7. Let de (resp. o4) be the fundamental function of L (resp. )*) and ou (t)/t= Eg. Then
||| do (1)/411 = |1 < oo if and only if Moe = L..
Proof.
||e + 20 (t)/t|11 = 11e r + (1/0114 inf ED t $ (1(t))
$1).
(10)
From Pick et al.
2013 p. 412) we have that the last expression is finite, then Moe = C Conversely,
assume Mo. that E = Y E c
Then we obtain from M3 in Proposition 5.6 together with the embodding theorem 5.9
To put this into words, the random variable whose decreasing rearrangement (backwards quantiles) we used to specify the envelope function E is itself in the space _ if and only if - coincides with its corresponding Marcinkiewicz space, that is, with the shared fundamental function do- On the other hand, we know from Proposition 5.6 that & + Et = (t((//t E Mo. if and only if the envelope approach based quantiles is equivalent to that based on CVar's; formally, if the Marcinkiewicz quasi-norm | |Mag and the Marcinkiewicz norm ||- IlMe, coincide (we use the "dual" of Proposition 5.6 Due to the embedding theorem, / + (1)/t Mà, necessary, but not sufficient condition for 1 (o L². We clarify this with some examples.
Example 6.8. _ Lexp ((y) = exp(y) 2 -x) log(x 1). Then:
YE > 0: +| de (t)/Ell < /''' exp(AAog(1/x+) 1)) - -I< 1<
is finite, for instance with X = 1/e. Therefore E E Lexps that is, an exponential distribution is itself in the space of subexponential random variables.
Example 6.9. _ LlogL: ((x) (x log(z) - x 1) X[1, We have that (t) = 1/(1 1a1--ogh -log(t)) up to equivalence, since L v Lexp- Then condition MI is (here & Φr):
1 - log(t)
Vs 1: sup
0<t<1 log(t/s) 1.
log(() 1. Hence, t t (x(t)/t & LlogL. Consequently, LlogL But for any s >> 1, we obtain that limyjo l-log(t/a) does not coincide with the Marcinkiewicz space Mo,, and morcover the envelope function Eb grows so rapidly 1.0 0 that the Marcinkiewiez quasi-norm does not coincide with the Marcinkiewiez norm.
25

---

Published in Transactions on Machine Learning Research (01/2023)
Example 6.10. L L:: let < 0o and ((z) = x², as in the Pareto example 6.6 Then ||t H dv (1)/4ll 0o is easily checked by using (1).. Thus, for instance the x² divergence risk measure is not finite on the Marcinkiewicz space with fundamental function o(t) = vt (here 7 = LV = =)). However, we have that | + (tt//t|l,,, < oo, that is, the conditions of Proposition 5.6 apply. Therefore the Marcinkiewicz quasi-norm and the Marcinkiewics norm are equivalent, but their induced space is strictly larger than the Orlicz space (the L" space). In fact, this larger space known as the " "weak L space", which used in the theory of interpolation of operators. Thus, we find that the Pareto distribution of parameter p, even if it does not have a finite L" norm, is the natural representative and member of the weak L" space, as it is the least upper bound on the quantiles of random variables in L". For p = 1 and - co, the Marcinkiewicz and the Orlicz space coincide
In summary, in this section we have proposed a way to construct an Orlicz norm with a desired tail sonsitivity based on the specification of the least, upper bound on the quantile behaviour of random variables in the space. Due to the equivalence relation in Example 4.24 this approach can then be used to construct a corresponding divergence risk measure, where the tail sensitivity is finely tuned to the reference distribution, which is the envelope function.
6.1 Utility-Based Shortfall Risk
Until now, we have always considered coherent risk measures to encode a specific tail sensitivity. However, there is another related class, the conver risk measures which have gained much attention recently in financial risk measurement. A potential problem is that due to positive homogencity, R(AX) = XR(X), the risk assessment scales just like the loss variable itself. This might. be undesirable due to hazards arising from very large risks. Thus it may be desirable to construct risk measures which are not positively homogeneous, so that the risk assessment can grow faster in the presence of large risks. A convex risk measure, which has received much attention in finance particularly as a tool to measure heavy-tailed risk (Giesecke et al.] 2008). is the baslel shortfall risk (UBSR) (Fölleer & Schied] 2002). Tn the context of machine learning it has not seem widespread use yet; but, for example, (Shen et al. (2014) apply it a tool for risk-sensitive reinforcement learning. Furthermore, Li et al. (2021) apply special case of the UBSI to practical machine learning problems and show that it can flexibly tune the impact of individual losses. In Appendix A.4 we discuss the UBSR, which has been linked to Orlicz norms. We make this connection yet more explicit; consequently, the insights from the previous section can then be directly applied to construct a utility-based shortfall risk measure with a specific tail sensitivity. For instance, the choice in Li al. (2021) corresponds to subexponential tail sensitivity.
7 Tail-Specific Orlicz Deviation Inequalities
Few papers in the machine learning literature have made explicit use of Orliez space theory; see for instance (Andoni et al. 2018), (Song et al. 2019) and (Chamakh et al. 2020). In contrast, concentration inequalities are widely used in machine learning in the context of statistical learning. Many concentration inequalities are in fact merely special cases of a general Orlicz deviation inequality; the standard textbook [Boucheron iet al. 22013, p. 45) on concentration inequalitics only montions Orlicz spaces in passing. We refer to (van der Vaart & Wellner 1996] Part 2.2) for the general approach; concrete cases of tail sensitivity which have been studied in the literature are the sub-Gaussian and the sub-Exponential norm (see e.g. (Vershynin) 2018)) the sub- Webbull norm (Vladimirova et al. 2020), the Bernstein-Orlicz norm (van de Geer & Lederer 2013 the Bennett-Orlicz norm (Wellner 2017 and more (Chamakh et al. 2021 Consequently, all of these give rise to equivalent divergence risk measures.
We bricfly present the Orlicz deviation incquality, since we believe that this general vicwpoint may be helpful to the machine learning community. Furthermore, we show how it naturally fits within our approach to tail sensitivity basod on the envelope function (hence, the fundamental function).
Ths is due to the special situation that for L and _ all r.i. norms equivalent.
12For positive values of their paramcter t, the objective in (Li all 2021 is the convex entropic risk measure Föllmer &
Knispel 2011) on the empirical distribution. Note that the convex entropic risk measure is special case of the UBSR (llesecke
el. 2008
26

---

Published in Transactions on Machine Learning Research (01/2023)
Assume finite Young function & and X 2 0. Then, applying Markov's incquality to the nonnegative and increasing function - (x/1Ix11 yields Ovan der Vaart & Wellner 1996 Part 2.2):
P(X 2 x) & P ( X (1)) 2 V (221|1X(1))
Vx > 0
E ( X [6)
Vr 0
Vx 0,
which follows from the definition of the Luxemburg norm In summary, for finite Young function v we obtain:
p(X2,)
(11)
For each finite Young function V, we therefore obtain a deviation inequality. In fact, we observe that it bounds the tail behaviour by the distribution function of the envelope function. Recall that the distribution function is (A( P([w Ω: [Y(w] >}) and that its generalized inverse is Y*. For simplicity, we shall assume that jy is continuous and invertible. Of course, Jyy μу holds. Since confusion could arise, we distinguish the fundamental function of the Orlicz norm of from the fundamental function of the Luxemburg norm o where the subscript indicales the Young function.
Proposition 7.1 (Rubshtein et al. 2016 p. 177)). The fundamentaa function of the Luzemburg norm ||- is given by:
05(1) ((1(1/)) og())'
(0.1)
For given Luxemburg norm, we can directly relate the deviation incquality to the envelope function. Proposition 7.2. With the envelope function Y*(t) Ftt()) 1/)4(t), where we assume (*(1) 1 (without loss of generality) and finite V we obtain the deviation inequality:
P(X 2) S ur- x 1211 X20. 0.
Proof. Just observe that
ot()) (11(1
(zz)
Vr 2 1,
(t)
(+(I)
pY (x)
and use
11
To intuit this result, recall that the envelope function constrains the quantile behaviour of random variables in an Orlicz space. Thus, it is unsurprising that the corresponding distribution function plays analogous role. Conversely, we offer the following constructive approach to obtain a deviation inequality with a desired tail scnsivity based on a reference distribution.
Proposition 7.3. Choose E (0. - R as the decreasing rearrangement of some reference distribution, where (0+) = xc, E 1, and for which L + og((1) (1) E (1) -t is increasing and concave (0, and which satisfies (0+) = O. We set o% = O. With the envelope function (*) = (t) 1/04 (t), the following Orlicz deviation inequality holds:
P(X 2x) x) Hy
x 2 X 0.
Proof. We apply the "dual" of Proposition 6.2 and obtain an Orlicz norm ||| 118 with fundamental function Φ6. Observe that by the associate relationship
t (t)
(b(1) (t())
The associate of the Orlicz norm || 11.11 is the Luxemburg norm || 11 with envelope EJ. Also, we get from Proposition 6.2 that y is finite Young function and therefore the deviation inequality holds due
Proposition 7.2
27

---

Published in Transactions on Machine Learning Research (01/2023)
Again, if (t) (t) t is at least quasiconcave, for which it suffices that (t (t) t is increasing (ef. Section 5.3) we may use the equivalent least concave majorant in the construction to obtain a deviation inequality, which is equivalent up to constant multiplicative factor.
Example 7.4. The sub- Eibonnnnnetti deviation incquality: let (t) log((t) (Example 5.10 Then:
P(X 2 x) S exp( exxp(=x/|1>> + 1), x 2 || I X2 0.
where the Young function vxx) for x E (0, and v(x) exp(x 1) for 2 satisfies the equation V (1/t) (t) for t e (0,
Example 7.5. The Pareto deviation inequality: let F (t) = 11p/ < <<<>> xc. Then v(x)) = x. and:
1
P(X 2
x2 |X 2 0.
(x/1IXI1))
For p = 2, this gives Chebyshev's incquality.
8 Application
We have demonstrated how, given a specific tail-sensitivity in the form of a fundamental function, one can canonically construct three r.i. norms: the extremal Marcinkiewicz and Lorentz norms, and the Orlicz norm in between. In some cases, the Orlicz norm is equivalent to one or even both of them. This raises the question of whether one of these norms is superior. To give some practical advice, we consider corresponding risk measures and how to compute with them. These are then candidates to replace the expectation in expected risk minimization, in order to obtain a specified tail sensitivity.
The downside of the Marcinkiewicz norm for such practical use that it lacks the property of translation equivariance (44. even when restricted to nonnegative random variables and constants only. However, [Fröh- lich & Williamson 2022) recently defined an equivalent norm, which is induced by translation equivariant risk measure 13
Proposition 8.1. Fröhlich & Williamson 2022 Theorem 4.16) Given any concave fundamental function é, which is continuous at 0, the posilive translation equicariant Marcinkiewicz norm is defined as:
1 r(1 [|X + (1 (1-1-0(1 t)) (V
| = sup u<t<1
It holds that || Ila,, and ||. IlTM, are equövalent. A corresporuiigg coherend risk measure, which induces this norm, is:
1 1 (1 [[X] + (1-1-0(1-1) CV)
TM,(X) sup 0<1<1
The ||. I1TM, norm is the smallest r.d. norm with fundamental function o, which is induced by a coherent risk measure.
Proof. We observe that this norm (risk measure, respectively) is supremum over convex combinations of the expectation and CVar,, where the weightings depend on the fundamental function to achieve the desired
tail-scnsitivity. The class of risk measures closed under convex combinations and taking suprema, hence TMg is a legitimate coherent risk measure, which is obviously r.i. by definition. By taking absolute values, the norm || [|r,, is induced. For the remaining statements, see Fröhlich & Williamson 2022 Theorem 4.16).
13 An r.j. norm can be translation equivariant due absolute homogencity. Therefore Fröhlich & Williamson 2022
have defined the related property of positive translation equivariance (PTE) for norm: norm called PTE IX C
such that X x ||X+ +c|| = |[ + Indeed, the norm in Proposition 8.1 turns out the smallest. PTE r.1. with given fundamental function Fröhlich Williamson, 2022 Theorem 4.16).
28

---

Published in Transactions on Machine Learning Research (01/2023)
Thus, the TM4 risk measure is distinguished in the sense that the induced norm is the smallest of all r.i. norms which are induced by risk measures of fundamental function Φ, thereby providing the smallest (most optimistic) risk assessment.
A spectral risk measure, which induces a Lorentz norm, extends the fundamental function in the most cautious (pessimistic) possible way. The Lorentz norm IlA, corresponds to the spectral risk measure Ræ:
Rx(X) = L/ F (1-w) do(w).
Thus, having specified a desired tail sensitivity, using the coherent risk measures TM₁ and RS are the extremal choices and they hence provide a maximal "model uncertainty" regarding the choice of risk measure. Example 8.2.
Fröhlich & Williamson 2022 Let <(t) = (1 t)?. Then consider
Rx(X)= = MaxVar(X) = Xx,,X [max(XX (Xi, X2)] (M,(X) = Dutch(X) E [max [[]])
~ means that X, and X, are independent and identically distributed. These risk measures are known as Max Va [Cherny Madan 2009) and the Dutch risk measure (Van Heerwaarden & Kaas, 1992).
where X1, X2 X2
The Orlicz norm (the Orliez regret and risk measure, respectively) is situaled in between these two ends of optimism and pessimism. However, refrain from making general recommendation to which of the three candidates (or even another risk measure with the given fundamental function) to use in practice. We make some remarks regarding the choice.
Spectral risk measures are embedded in rich literature, from economics (Quiggin 2012), insurance [Wang 2000 to philosophy (Buchak 2013). and can be motivated from various perspectives tool for rational decision making. However, the computation of spectral risk mcasures is non-trivial.
We refer to (Pandey et al. 2021 for the estimation of spectral risk measures from samples and
to (Holland & Haress 2022 1, Leqi et al. 2022) and (Mehta et al. 2022) for learning schemes,
which may be used in practice to replace ERM with spectral risk measures. Bäuerle & Glauner
(2021) study the minimization of spectral risk measures the context of Markov decision processes. Some spectral risk measures, like E. CVar,, or convex combinations thereof easily computed and optimized; a useful form is that of Example 3.20 To improve the stability of optimizing CVar, in batch setting, Curi et al. (2020) have proposed an adaptive sampling method; see also (Soma & Yoshida 2020 and (Holland & Haress 2021
The TM4 risk measure, related lo the Marcinkiewicz norm, is perhaps not intuitive. However, it is easy to compute in a risk minimization loop. As of now, see little motivation for employing it in practice.
The f-divergence risk measure situated in between, albeit in many cases equivalent one of the above, observed in Scction Specifically, highly tail-sensitive divergence risk mcasure is typically equivalent to the corresponding Marcinkiewicz norm. An advantage is that divergence risk measure naturally comes with corresponding deviation inequality, we established in Section 7 Also, some prominent spectral risk measures like E or CVar, are divergence risk measures. The infimum-based representation 5 allows practical computation. For instance, when training a neural network, single line suffices to implement.
5 a framework such as PyTorch. We also refer Chouzenoux et al. (2019) who propose suitable accelerated projected gradient algorithm.
On the other hand, the utildty-based shortfall risk has different use cases such as reinforcement learning, as it lacks positive homogeneity and is thus not suitable in a standard supervised learning risk minimization
14e name Vax is due our loss based orientation here; the authors originally called it MinVar due to sign flip.
29

---

Published in Transactions on Machine Learning Research (01/2023)
loop. Finally, use cases for the Orlicz regret measure may appear in risk-sensitive regression; refer to Rockafellar & Uryasev 2013 and Rockafellar et al. (2008) for regression based on error measwres E, which are based on regret measures V via the unique correspondence B(X) = (()))(((X)))) For instance, quantile regression is associated with the CVara Orlicz regret mcasure.
9 Conclusion
In this work, have brought together concepts such the fundamental function and envelope function from the theory of r.i. Banach function spaces with the framework of coherent risk mcasures, in particular divergence risk measures. As a consequence, we have obtained & general approach to fine-tune the tail sensitivity of a risk measure, which then functions as a more cautious replacement for the expectation in ERM, or more generally, can be used to assess the risk inherent in a loss distribution.
We have demonstrated how lail sensitivity related to the fundamental function. To first approximation, those risk measures with the same fundamental function share the same tail sensitivity; on a finer level, however, it might be the case that the spaces which different such risk measures are finite do not coincide. Nonetheless, the fundamental function offers a useful and simple stratification of the space of coherent risk measures. In this way, the choice of fundamental function indeed appears more fundamental than the exact choice of risk measure; Marcinkiewicz, Orlicz, Lorentz or a still different one.
Abundant opportunities for future work arise. For instance, quantitative comparison of the (positive and their corresponding risk measures
translation equivariant) Marcinkiewicz, Orlicz and Lorentz practical problems has not been conducted yet. Also, cstimatability from finite samples is open problem: conjecture that the more tail-sensitive a risk measure, the harder it is to estimate it from finite samples. For instance, can one obtain convergence rates based on properties of the fundamental function?
Broader Impact Statement
We study risk measures, which add fail sensilivity to empirical risk minimization. As a consequence, the use of such risk measures emphasizes highly negative outcomes and thus may reduce inequality in the loss distribution. However, this will certainly not fully remedy the problem of disparate impact that machine learning systems
have, for instance individuals. Moreover, the choice of tail sensitivity and risk measure is an additional ethical choice for the machine learning engineer to make, which must be explicitly stated and about which disagreement is to be expected.
Acknowledgments
This work was was funded by the Deutsche Forschunsgeemeinschaat (DFG, German Research Foundation) under Germany's Excellence Strategy EXC number 2064/1 Project number 390727645. The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Christian Fröhlich. Thanks to Rabanus Derr for many helpful discussions and comments. We thank one anonymous reviewer in particular for providing helpful comments.
References
Carlo Acerbi. Spectral measures of risk: A coherent representation of subjective risk aversion. Journal of Banking & Finance, 26(7):1505 -511188551118, 2002.
Amir Ahmadi-Javid. Entropic value-at-risk: A new coherent risk measure. Journal of Oplimizalion Theory and Applications, 155(3):1105-1123 2012.
Amir Ahmadi-Javid and Alois Pichler. An analytical study of norms and Banach spaces induced by the entropic value-at-risk. Mathematics and Pinancial Economics, 11(4):527 550, 2017.
Alexandr Andoni, Chengyu Lin, Ying Sheng, Peilin Zhong, and Ruiqi Zhong. Subspace embedding and linear regression with Orlicz norm. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th
30

---

Published in Transactions on Machine Learning Research (01/2023)
International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 224-233. PMI.R 10-15 Jul 2018.
Takuji Arai. Convex risk measures on Orlicz spaces: inf-convolution and shortfall. Mothematics and Financial Economics, 3(2):73 88,, 2010.
Takuji Arai. Good deal bounds induced by shortfall risk. STAM Journal on Financial Mathematics, 2(1): 1 21, 2011.
Philippe Artzner, Freddy Delbaen, Jean-Mare Eber, and David Heath. Coherent measures of risk. Mathe- matical Finance, 9(3):203-228 1999.
Drumi D. Bainov and Pavel S. Simeonov. Integral inequalities and applications. Springer Science & Business Media, 2013.
Nicole Bäuerle and Alexander Glauner. Minimizing spectral risk measures applied to Markov decision processes. Mathemalical Methods of Operalions Research, 94(1):35 69, 2021.
Güzzn Bayraksan and David K. Love. Data-driven stochastic programming using Phi-divergences. In Dionne M. Aleman and Aurélie C. Thiele (eds.), The Operations Research Revolution, chapter 1, pp- 1-19. 2015.
Fabio Bellini and Valeria Bignozzi. On elicitable risk measures. Quantitative Finance, 15(5):725-73)) 2015.
Fabio Bellini and Emanuela Rosazza Gianin. On Haezendonck risk measures. Journal of Banking & Finance, 32(6))986-994, 2008.
Aharon Ben-Tai and Arkadi Nemirovski. Robust solutions of uncertain linear programs. Operations Research Letters, 25(1):1-13, 1999.
Colin Bennett and Robert Sharpley. Interpolation of Operators. Academic Press, 1988.
Sara Biagini and Marco Frittelli. On the extension of the Namioka- Klee theorem and on the Fatou property for risk measures. In Freddy Delbaen, Miklós Rásonyi, and Christophe Stricker (ods.), Optimality and risk -modern trends in mathematical finance, pp. 1-28. Springer, 2009.
Stéphane Boucheron, Gábor Lugosi, and Pascal Massart. Concentration inequalities: A nonasymplolic theory of independence. Oxford university press, 2013.
Christian Brownlees, Emilien Joly, and Gábor Lugosi. Empirical risk minimization for heavy-tailed losses. The Annals of Statistics, 43(6):2507-2555 2536, 2015.
Lara Buchak. Risk and rationality. Oxford University Press, 2013.
Nicholas R. Cavanaugh, Alexander Gershunov, Anna K. Panorska, and Tomasz J. Kozubowski. The proba- bility distribution of intense daily precipitation. Geophysical Research Letters, 42(5):1560-1567, 2015.
Linda Chamakh, Emmanuel Gobet, and Zoltán Szabó. Orlicz random Fourier features. Journal of Machine Learning Research, 21145)::1-37, 2020.
Linda Chamakh, Emmanuel Gobet, and Wenjun Liu. Orlicz norms and concentration inequalitics for 3-heavy tailed random variables. HAL preprint, hal- hal-03175697u3, 2021.
Patrick Cheridito and Tianhui Li. Risk measures on Orlicz hearts. Mathematical Finance, 19(2):189 10(2):199-114 214, 2009.
Alexander Cherny and Dilip Madan. New measures for performance evaluation. The Review of Financial Studies, 22(7):2571-2666, 2009.
Emilie Chouzenoux, Henri Gérard, and Jean- Christophe Pesquet. General risk measures for robust machine learning. arXio preprint arxiv: 1904. 707, 2019.
31

---

Published in Transactions on Machine Learning Research (01/2023)
Yunan Cui, Lifen Duan, Henryk Hudzik, and Marek Wisla. Basic theory of p-Amemiya norm in Orlicz extreme points and rotundity in Orlicz spaces endowed with these norms. Nonlinear Analysis: Theory, Methods & Applications, 69(5-6):1796 1816, 2008.
spaces (1 S S oo):
Scbastian Curi, Kfir Y. Lovy, Stefanic Jegelka, and Andreas Krause. Adaptive sampling for stochastic risk- averse learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Newral Information Processing Systems, volume 33, pp. 1186-1047. Curran Associates, Inc., 2020.
Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos. Implicit quantile networks for distribu- tional reinforcement learning. In International conference on machine learning, volume 80, pp. 1096-1105. PMLR, 2018.
Freddy Delbaen. Coherent risk measures on general probability spaces. In Klaus Sandmann and Philipp J. Schönbucher (eds.), Advances in Pinance and Stochastics: Essays in Honour of Dieter Sondermann, pp. 1-37. Springer, 2002.
Paul Dommel and Alois Pichler. Convex risk measures based on divergence. Pure and Applied Functional Analysis, 6(6):1157-118)) 2021.
John C. Duchi, Peter W. Glymn, and Hongseok Namkoong, Statistics of robust optimization: A generalized empirical likelihood approach. Mathematics of Operations Research, 46(3):946-969, 2021.
Gerald A. Edgar and Louis Sucheston. Stopping times and directed processes. Cambridge University Press, 1992.
Yanbo Fan, Siwei Lyu, Yiming Ying, and Baogang Hu. Learning with average top-k loss. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (ods.), Advances in Newral Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Hans Föllmer and Thomas Knispel. Eutropic risk measures: Coherence vs. convexily, model ambiguity and robust large deviations. Stochastics and Dynomics, 11 11(02003):333 351, 2011.
Hans Föllmer and Alexander Schied. Convex measures of risk and trading constraints. Finance and Stochas- tics, 6(4):429-44)) 447, 2002.
Hans Föllmer and Alexander Schied. Stochaslic Finance. de Gruyter, 2016.
Hans Föllmer and Stefan Weber. The axiomatic approach to risk measures for capital determination. Annual Review of Financial Economics, 7(1):301-33) 337, 2015.
Christian Fröhlich and Robert C. Williamson. Risk measures and upper probabilities: Coherence and strat- ification. Xix preprint arxiv:2206. 211:22266.0118,, 2022.
Kay Giesecke, Thorsten Schmidt, and Stefan Weber. Mcasuring the risk of large losses. Journal of Investment Management (JOIM), Fourth Quarter, 2008. URL
Ttzhak Gilboa and David Schmeidler. Maxmin expected utility with non-unique prior. Journal of Mothe- matical Economies, 18(2):141 153, 1989.
Mert Gurbuzbalaban, Umut Simsckli, and Lingjiong Zhu. The heavy-tail phenomenon in SGD. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 3964-3975. PMLR, 18-24 28-24 Jul 2021.
Dorothee D. Haroske. Envelopes and sharp embeddings of function spaces. Chapman and Hall/CRC, 2006.
Tatsunori Hashimoto, Megha Srivastava, Hongsook Namkoong, and Percy Liang. Fairness without demo- graphics in repeated loss minimization. In International Conference on Machine Learning, pp. 1929-1938. PMLRR 2018.
32

---

Published in Transactions on Machine Learning Research (01/2023)
Matthew Holland and El Mchdi Haress. Learning with risk-averse feedback under potentially heavy tails. In Arindam Banerjee and Kenji Fukumizu (eds.), Proceedings of The 24th International Conference Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research, pp. 892 900. PMLR, 13-15 Apr 2021.
Matthew Holland and Mehdi Haress. Spectral risk-based learning using unbounded losses. In International Conference on Artificial Intelligence and Statistics, pp.
PMLR, 2022.
Daniel Hsu and Sivan Sabato. Loss minimization and parameter estimation with heavy tails. The Journal of Machine Learning Research, 17(1):543 582, 2016.
Weihua Hu, Gang Niu, Issci Sato, and Masashi Sugiyama. Does distributionally robust supervised learning give robust classifiers? In Jemnifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 2037. PMLR, 10 15 Jul 2018.
Henryk Hudzik and Lech Maligranda. Amemiya norm equals Orlicz norm in general. Indagationes Mothe- malicae, 11 14():5573 585, 2000.
Marat Ibragimov, Rustam Ibragimov, and Johan Walden. Heavy-tailed distributions and robustness in eco- nomics and finance. Springer, 2015.
Peter Kosmol and Dieter Wichars. Optimization in function spaces. De Gruyter, 2011.
M. A. Krasnoselskii and YA. B. Ruticki.. Conver functions and Orlicz spaces. P. Noordhoff, Groningen, 1961.
Selim Grigor'evich Krein, Jurii Ivanovvich Petunin, and Evgenii Mikhailovich Semenov. Interpolation of Linear Operators. American Mathematical Society, 1982.
Liu Leqi, Audrey Huang, Zachary Lipton, and Kamyar Azizzadenesheli. Supervised learning with general risk functionals. In Kamalika Chaudhuri, Stefanic Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 12570-12592. PMLR, 17-23 Jul 2022.
Tian Li, Ahmad Beirami, Maziar Sanjabi, and Virginia Smith. Tilted empirical risk minimization. In International Conference on Learning Representations, 2021.
Wilhelmus J. J. Luxemburg. Banach function spaces. PhD thesis, Technische Hogeschool te Delft, Nether- lands, 1955.
Ronak Mehta, Vincent Roulet, Krishna Pillutla, Lang Liu, and Zaid Harchaoui. Stochastic optimization for spectral risk measures. arXie preprint arXiv:2212.051449 2022.
Bruno Mcrz, Stefano Basso, Svenja Fischer, David Lun, Günter Blöschl, Ralf Merz, Björn Guse, Alberto Viglione, Sergiy Vorogushyn, Elena Macdonald, et al. Understanding heavy tails of flood peak distribu- tions. Water Resources Research, (60))e2221000005500 2022.
Jayakrishnan Nair, Adam Wierman, and Bert Zwart. The fundamentals of heavy toils: Properties, emergence, and estimation. Cambridge University Press, 2022.
Hongscok Namkoong and John C. Duchi. Stochastic gradient methods for distributionally robust optimiza- tion with f-divergences. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016.
Hongscok Namkoong and John C. Duchi. Varianco-basel regularization with convex objectives. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advonces in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
33

---

Published in Transactions on Machine Learning Research (01/2023)
William D. Nordhaus. Economic policy in the face of severe tail events. Journal of Public Economic Theory, 14(2):197-22) 219,,,,9--21, 2012.
Ajay Kumar Pandey, L.A. Prashanth, and Sanjay Bhat. Estimation of spectral risk measures. In Pro- ceedings of the AAAT Conference Artificial Intelligence, volume 35, pp. 22166-12173, 2021.
Georg Ch. Pflug and Werner Römisch. Modeling, measuring and managing risk. World Scientific, 2007.
Alois Pichler. The natural Banach space for version independent risk measures. Insurance: Mathematics and Economics, 53(2):405-415, 2013.
Luboš Pick, Alois Kufner, Oldřich John, and Svatopluk Fucík. Function spaces, de Gruyter, 2013.
John Quiggin. Generalized expected utility theory: The ronk-dependent model. Springer Science & Business Media, 2012.
Hamed Rahimian and Sanjay Mehrotra. Frameworks and results in distributionally robust optimization. Open Journal of Mathematical Optimization, 3:1 85, 2022.
Sidney I. Resnick. Heavy-tail phenomena: probabilistic and statistical modeling. Springer Science & Business Media, 2007.
James A. Roberts, Tjeerd W. Boonstra, and Michael Breakspear. The heavy tail of the human brain. Current Opinion in Neurobiology, 31:164-172, 2015.
R. Tyrrell Rockafellar. Integrals which are convex functionals. Pacific Journal of Mathematics, 24(3): 525-539, 1968.
Tyrrell Rockafellar and Stan Uryasev. The fundamental risk quadrangle in risk management, optimization and statistical estimation. Surveys in Operations Reseorch and Management Science, 18(1-2):33 53, 2013.
R. Tyrrell Rockafellar, Stan Uryasev, and Michael Zabarankin. Risk luning with generalized linear regression. Mathematics of Operations Research, 33(3):712-72) 729, 2008.
Ben-Zion A. Rubshtein, Genady Ya. Grabarnik, Mustafa A. Muratov, and Yulia S. Pashkova. Foundations of symmetric spaces of measurable functions. Springer, 2016.
Alexander Shapiro. Distributionally robust stochastic programming. SIAM Journal on Optimization, 27(4): 2258-2275, 2017.
Yun Shen, Michael J. Tobia, Tobias Sommer, and Klaus Obermayer. Risk-sonsitive reinforcement learning. Neural Computation, 26(7):1298 1328, 2014.
Rahul Singh, Qinsheng Zhang, and Yongxin Chen. Improving robustness via risk averse distributional reinforcement learning. In Learning for Dynamics and Control, volume 120, pp. 958-968. PMLR, 2020.
Tasuku Soma and Yuichi Yoshida. Statistical learning with conditional value at risk. ar preprinl arXiv:2002.0588 2020.
Zhao Song, Ruosong Wang, Lin Yang, Hongyang Zhang, and Peilin Zhong. Efficient symmetric norm regression via linear sketching. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Accé-Buue, E. Fox, and R. Garnett (eds.), Advances in Newral Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Akiko Takeda and Masashi Sugiyama. v-support vector machine as conditional value-at-risk minimization. Tn Proceedings of the 25th international conference on Machine learning, pp. 1066-1063, 2008
Nassim Nicholas Taleb. The black swan: The impact of the highly improbable. Random House, 2007.
Aviv Tamar, Yinlam Chow, Mohammad Ghavamzadeh, and Shie Mannor. Policy gradient for coherent risk measures. In Advances in newral information processing systems, volume 28, 2015.
34

---

Published in Transactions on Machine Learning Research (01/2023)
Matthias C.M. Troffacs and Gert De Cooman. Lower previsions. John Wiley & Sons, 2014.
Núria Armengol Urpi, Sebastian Curi, and Andreas Krause. Risk-averse oflline reinforcement learning. In International Conference on Learning Representations, 2021.
Sara van de Geer and Johannes Lederer. The Bernstein Orlicz norm and deviation inequalities. Probability theory and related fields, (57(1))225-255) 2013.
Aad W. van der Vaart and Jon A. Wellner. Weak convergence and empirical processes. Springer, 1996.
A.E. (Angela) Van Heerwaarden and Rob Kaas. The Dutch premium principle. Insurance: Mathemalics and Economics, 11(2):129 133, 1992.
Roman Vershynin. High-dimensional probability: An introduction with applications in data science. Cam bridge university press, 2018.
Paolo Vicig. Financial risk measurement with imprecise probabilities. International Journal of Approzimate Reasoning, 19(1):159-174, 2008.
Nithia Vijayan and L.A. Prashanth. Risk-sensitive reinforcement learning via distortion risk measures. arXiv preprinl ar Xív: orXic:2107.04485 2022.
Mariia Vladimirova, Jakob Verbeek, Pablo Mesejo, and Julyan Arbel. Understanding priors in Bayesian neural networks at the unit level. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 6458 6467. PMLR, 09 15 Jun 2019.
Mariia Vladimirova, Stéphane Girard, Hien Nguyen, and Julyan Arbel. Sub-Weibull distributions: Gen- eralizing sub-Gan mub-Gaussian and sub- sabb-xponential properties to heavier tailed distributions. Stat, 9(1):e318, 2020.
Peter Walley, Statistical reasoning with imprecise probabilities. Chapman-Han Hall, 1991.
Shaun S. Wang. A class of distortion operators for pricing financial and insurance risks. The Journal of Risk and Insurance, 67(1):15-38 36, 2000.
Jon Wellner. The Gennett-Orlice norm. Sankkya A, 79(2):355 383, 2017.
Robert C. Williamson and Aditya Menon. Fairness risk measures. In International Conference on Machine Learning, volume 97, pp. 6786-6797. PMLR, 2019.
Jingzhao Zhang, Aditya Krishna Menon, Andreas Veit, Srinadh Bhojanapalli, Sanjiv Kumar, and Suvrit, Sra. Coping with label shift via distributionally robust optimisation. In International Conference on Learning Representations, 2021.
A
Appendix
A.1 Orlicz Regret Measures
A.1.1 Proof of Proposition 3.2
Proof. The convex conjugate of a (not necessarily convex) function is always convex. To see that g is finite- valued R due to the supercoercivity of (a well-known statement, (Edgar & Sucheston 1992 Dommel
& Pichler 2021
Föllmer & Schied, 2002 ) assume by contradiction that there exists a y R such that
2 g(v) sup f(x).
z20
35

---

Published in Transactions on Machine Learning Research (01/2023)
Since f is proper, i.c. never takes -00, the supremum cannot be attained for some z E R. Therefore we need to investigate only the limit as + oo:
f(x)
( lim f(x)= = lim x
x+0
But since y fixed, this could only yield co if lim
y f(x)
2 0, which contradicts the supercoercivity
assumplion limg-+e0 [(x]
To see that is increasing (non-decreasing), observe that if y1 S y2: then
g(v1)
sup 21 f() S sup y22 f(2) g #20
#20
A.1.2 Proof of Proposition 3.9
Proof.
V1
Positive homogencity: let A > 0.
AX
(xxX)) = inf t
++ Eg
120
(
i t X
inf EX
+ Eg
1s0
(X).
2008 for subadditivity of an Orlicz norm, which also works in our setting of an Orlicz regret measure. First observe that X 4+1 1 + Bg (X) is convex due to the convexity of We also know that for an arbitrary > 0 there exist some t,, l2 >> 0 so that
V2
For subadditivity, present the proof from Cui et al.
1
+ Eg(tix)) - VX()) +
i
- (1 + Eg(l2 S VV(
1/2
Let t
1/2. 4
Then it holds
Vg(X + 1) < - (1 + Eg(t(X ( +
t1 + t2
t2 4 + t
12 ))
1+ Bg
11/2
1 l2
1/2
1
< (1 + Eg(t1X)
+ (1 + Eg(tz s (X) + Vy(Y) +
t
to
V3
Monotonicity directly follows from monotonicity of g (Lemma 3.2
V4 We want show VX(X) 2 [[X] VX e M. Let be a divergence function, i.e. we have f(1) 0. Consequently, we have that
g sup xy f(x) 2 y f(1)= = Vy E.
But then:
X
X
E X = tE
< inf t + tE
) VX)
X
< inf 1>0
1 Eg
36

---

Published in Transactions on Machine Learning Research (01/2023)
V5
To see that X = 0 (0(0) 0, observe that
(0)
inf t (1 + Eg (0)) lim / (1 + Eg(0)) = 1>0
due to
s(0) sup0 f(z) < xc.
=20
Also note that since f(1) = 0, we have g(0) 2 0.
For the converse direction, assume (X()) for X 2 0. We want to show that it follows that X = 0. By
contradiction, assume X * 0 (P-a.e.). But then since X 0, we have ][X] > Due to aversity V4 we know (X) 2 [[X], therefore (X) 0.
A.1.3
Proof of Proposition 3.10
Proof.
X
Vg.. (X)
inf 130
1 + Ege
8
1
e.
inf t 120
+ Eg
8
t
(= inf te t>o
1
X
+
Eg
E
i
(6 +
X
inf i E>0
Rg
f
A.1.4 Proof of Proposition 3.11
Proof. That ÿ is a valid Young function is obvious since g(0) = and it inherits the properties g (increasing, convex etc.) on the nonnegative domain. Recall that X[1,00)- conjugate is therefore given by
h(y) = sup xy f(x) max sup z20
my, sup - ]((0)
#21
For negalive y, have hyy) 0, since the maximum is attained in the left term with value 0 (here the supremuun is attained at = 0), since the right term will be negative. Thus h(y) = g(v) = 0 for y < 0. It remains to show that for positive we have h(y) = g(y). For this, we show that for positive y, only the second term of suffices:
glv) max sup ay f(x), supay f(x) sup ry f(x), y (oS<<1
x21
a21
Assune by contradiction that y is positive and the maximum is attained in the first term. Note that supx21 zy f(x) 2 y since f(1)= =0. 0. But:
sup zy f(x) y,
05
since ry S y and fxx) 2 0. Hence the maxinum is attained in the second term. Thus we have shown that h and therefore 7 is the conjugate of
37

---

Published in Transactions on Machine Learning Research (01/2023)
A.1.5 Proof of Proposition 3.14
Proof. The first inequality is obvious. Second,
{ (x))= - inf
(X)
22 Eg
X
inft 130
E2 + Eg
E2 +
1>0
E1
8
62 i
81 + Eg
7 ) = 82
X
1 1>0
which is true since 52
A.1.6 Proof of Proposition 3.17
To establish the dual representation
Vx(X) = sup Z c , 2, ([ 1 V C
we follow roughly Kosmol & Müller-Vie Wichard
2011 pp. 238-239). First note that Vg is finite on L
(Proposition 3.12 We begin with dual variables Z C but later show that the restriction to L is
possible. The crucial point is that the functionals X + B[g(X)] and Z + E[f(Z)) stand in conjugate
relationship due to the conjugacy of and f:
B[g(X)]
sup E - E[)(Z)]
(12)
ZEC ,20
This result is from (Rockafellar 1968 It follows from the fact that is lower semicontinuous proper convex function with interior points in its domain and therefore, G(w, x:) g(x) is normal convex integrand [Rockaflllrr 1968 Lemma 2). The conjugacy then follows from [Rockafellar 1968, Corollary in Theorem
2). Also note that the restriction to dual variables with Z 2 0 is possible due to f(z) co for z <<6 0 (this is essentially due to the monotonicity of V J. Then the Orlicz regret measure can be formulated as
)
X I
Vx(X) = inft
1 + Eg
inf 1+
sup ZECI .200
Cs0
2>0
inf (>0
sup 1E2.220
[E[XZ] {EE[ 1))
Now consider the dual representation:
DR(X)= = 7, 11} EX
2
Observe that an interior point of the constraint set due to the fact that f(1) = 0. We therefore get strong Lagrangian duality since Slater's condition holds, as observed for instance by Shapiro (017)
DR(X) inf sup t>0 720
[ (E[[(Z)) 1)) = V,(X),
which establishes the desired equality. Furthermore, it is sufficient to restrict ourselves to dual variables Z L). Assune by contraposition Z & C¹. Hence || = oc. But || < EIJ(2)) holds, see Rubshtein et al. 2016 Proposition 14.2.1). But then E[)(2)) = xc, and since E(J(Z) < B[S(Z) such Z is not feasible.
18eTe following statement well-known under different technical setups: a regret measure is monotone if and only dual variables are nonnegative. See instance Rockafellar & Uryasev
2013) the C²
38

---

Published in Transactions on Machine Learning Research (01/2023)
A.1.7 Proof of Proposition 3.19
Proof. We begin to show that the dual representation holds:
Rg. (X) sup[E[Xz] Z e 0,
AX E L.
(13)
Consider how defined the divergence risk measure in the first place:
Rg.s (X)= =
sup Q:dy(Q.P)Sc
BolX].
Here, the supremum is taken over the set of all valid probability mcasures Q which are absolutely continuous with respect to P. and further satisfy the f-divergence constraint. The necessary shift in perspective is to view Q not as a probability measure, but instead take the following route:
Qz(A)= = / 7(w) dP(w),
(14)
for the Radon-Nikodym derivative Z (which can be more suggestively writlen as 4Q Q is a measure, i.e. a set function, whereas is a random variable. Then observe that
Sc
where the expectation is with respect to the base measure P. Furthermore, imposing the constraint that Z yields a valid probability measure via [4] amounts exactly to requiring that 2 and E|Z] =
Observe that the envelope representation 13 the envelope representation of the corresponding Orlicz regret measure, but intersected with the constraint that E[Z] 1. If we worked on reflexive Banach space (where the bidual coincides with the space itself), together with appropriate closedness condition, we could use the following result. For instance, consider for moment the dual pair of L and L (the exponent 2 has dual exponent 2 since 1/2 + 1/2 1).
Denote the support function of the set Q as 0g:
xo(X)=
sup (X, (),
Q = (x,0) S VX
QEO
and define & = E L [[ = 1}. The following is well-known (see for instance Fröhlich & Williamson (2022) for statement on L²):
Proposition A.1. On L let V = dg be positively homogeneous, subadditive and monotonic functional, i.e. coherent regret measure, and Q be its supported set. Let Q' = QnE + 0. Then R := ao is coherent risk measure and R(X) = infeER V(X - c) +c.
The proof uses reflexivity of L Unfortunately, Orlicz spaces are reflexive and only if they satisfy the A2 condition, which limits the speed of growth of the Young function; see for instance (Kosmol Müller- Wichards 2011 p. 197, p. 234). Since we do not want to assume Ag throughout (this would for instance exclude the KT. divergence risk measure), we cannot rely the statement, but guided by it for intuition.
Our aim is to show that Rg.e is given by the infimal convolution
(X) = inf Vg, (X - 4).
peR
The proof works analogously to A.1.6 and hence Kosmol & Müller-Wicharde 2011 ; confer also Dommel & Pichler
2021 Theorem 5.1). Start with the envelope representation
(x(X)= = sup E[X 2, [,, V.e..
39

---

Published in Transactions on Machine Learning Research (01/2023)
Regarding the restriction Z E Li, the same remark in A.1.6 applies. Again, since Z 1 is an interior point of the constraint set, observe that by Lagrange duality:
(x)
inf PERI120
sup
[E[XZ] [()] 1) A[B[Z) 1)}
ZE ,20
f
X - P
= inf
t
1
+
sup
E
Z E1
ZE ,220
t
inf HER,1>0
1 + 11 +
sup
E
{(+ 11
2 - B[(c(2])
ZEC ,220
(
=
inf #2R,220
/
(1+ Ege
(15)
where we applied 12 the conjugacy relation of and de- We started with X L but the subsequent
Proposition 3.21 shows that Rg is finite C which is its natural domain Pichler 2013). Henceforth we restrict X to L"
Now we explicitly compute the infimal convolution of V
(x)
inf + VX - 11)
HER
X-
inf HER,1>0
-+ +
7 (17)
"=t
inl
ii 11+7
6 Rg
BB
(x7
X - 17
PER,b0
X
inf 7 e Eg AER
where we used the form of Vg, with explicit E (Proposition 3.10
This is the representation obtained by
Dommel & Pichler (2011) But we can also hide in the function ge:
(x) = inf AER,1>0
(s+ + f Eg
11
R=ye
inf
Ee
11+ + - Eg
X
7
μE
iER
i=t
1
X
)
inf
11 1 + 1
Eg
E
11
PER,150
(1 (1+ Rgg
X
inf 1. pER,1>0
which coineides with 15 and therefore have established that. (X) inf jnlEER p Vg, (X- - -p).
The "natural eatension" representation: Finally, we establish that the following envelope representation also holds:
Rg.(() = sup [Eoz Qz e E,E $ V (Y) VY ]
WX C
When one work with support functions, this representation is easy to show However, in our Orlicz setling we have to prove it. For this, consider the representation (the validity of which we show later):
Rg. (X) sup
c 0, vX C3,
(16)
16 Just consider the definition the supported support function and then use standard results obtain constraints Z and E[Z] from monotonicity and translation equivariance, which imply Qz
40

---

Published in Transactions on Machine Learning Research (01/2023)
where
a(Z)== = sup [B| E (X) S0.
which is the envelope representation of convex risk measure. Since in our case, we have coherent risk measure, it is standard that in fact the "penalty function" can be taken to be the convex indicator ig of the envelope 2, where 2 = E 2 2 0, B = 1, (2( S 1}. Then:
Rg (X) = sup E E|XZ| - (2 Z c¹, 2 -
The acceptance set A = (X E C : (X) - 0}. We have:
alZ) - iz(Z) iz = sup [ X E C Rg (X) 50).
Recall that the convex indicator function is iz (Z)
2 and to0 otherwise. Thus we find that
E c 2 2 0. [ 1, sup [ X C Rg (X) s 03}
(17)
/ c¹ 2 0, E| 1, [X 0 AX C with Rg (X) 0).
This follows since Rg (0) = 0, i.c. 0 is always in the acceptance set, hence the inner supremum in 17
never be < O. Further, with similar reasoning in Bellini & Gianin (0088 Proposition 17):
2 2 c z 0, B|Z = 1, B < 0 VX E L with Rg (X) s }
UE cf z 0, E|Z = 1| < Rg. (X) VX C3 =: =
That Zt S Z is obvious from the definition of Rg.e. Conversely, we show C Zt. Lot Z E i.c 2 [[Z] and [| 0 VX E L with Rg < 0. Then consider y= X Ry (X) for arbitrary X L Then Y e _ since an Orlicz space is closed under subtraction of constants (consider the definition of the Luxemburg norm). Due to translation equivariance, Rg. (Y) Rg. (X Rgg (X)) Rg. (X) Rg. (X) 0 0, hence Y is acceptable and therefore E Z] < 0 by assumption that Z E Z. Then we know that
E S 0 + B[(X- - Rg. (X)) < 0 e E[X Rg. (X) s 0 + E X S Rg. (X),
noting that for a constant c, it holds E[(X - c) = E[X c since E[Z] = 1. Since was arbitrary in L3, we conclude that Z E 21, and since was arbitrary, C 2 Having established 2 2t, we continue:
z= zt
ZE c ZZ 2 = E
inf μ Vg 1)) XX E 6O
MER
EE ci .220, E = E| S 4 V (X (X-1) AX L YPER
{ZE cl :220
E|X n V. (X (x-) VX E C3 V
{z dl. E|] [ n) (X---M) VX C" e
Z cl 220, Z E < Vg. (Y) VY e
again noting that E[(( 2| / 0.
E X since B[z] = Also, we may substitute Y= X n, where
We still need to establish that the representation in 16 holds. It follows from Arai 2010 Theorem 1); note that their strictness assumption on the Young function is not needed for this. To apply the result, it remains to show that Ry.s has the "order lover-semicomminoous property (Biagini & Frittelli 2009). But this follows from the fact that we have before established the envelope representation (13), which expresses Ry as pointwise supremum over a family of order Dover-semicontimouss functionals. Thus it is itself order sever-semicontimoous (Biagini & Frittelli 2009. Section 3.1); note that all Orlicz Banach lattices are order separable (Biagini & Frittelli 2009 Remark 16).
41

---

Published in Transactions on Machine Learning Research (01/2023)
A.2 Rearrangement Invariant Banach Norms and Fundamental Functions
A.2.1 Proof of Proposition
Proof. Due lo nonnegativity of indicator functions, the fundamental function coincides with the fundamental function of the Orlicz norm || 111 (due to Proposition 3.12 is known (Kosmol & Müller- -ichhrd- 2011 Lemma 7.4.4) that this fundamental function is (b(1) 1 (1/0); since 1/1 1, the choice of or does not matter; note that we defined (y) = sup f(x) < v}. To intuit the result, we give non-rigorous
"computation". Let E be an arbitrary set of measure t:
(v,(1)= - =Vx(E) =sup { Xxx(()>>())) du (B:BB((Z)))))) = $1}
sup (t- 21 f(2)) + (1- t) f(zo) s 11.
That it is possible to decompose Z into 21 and 2u is intuitive, but requires formal justification. Due to the nonnegativity of f, clearly 2g = 0 the best choice. For t 1-21 to be as large as possible, we then need t. f() (rocall the left-continuity of f), hence 21 = f- Thus Φv, (t) t (t). Noting that -- the fundamental function at different risk aversion levels can be obtained easily.
To see that Φv, concave, note that since f convex and increasing (non-decreasing), J-1 is concave Rubsshtein et al. 2016 172). Let az= sup(# f(x) = 00. Note that f is strictly increasing on [ay. 00) and f-1 can only have jump at 0. Then p p (Lp)) convex, due to the perspective transform of convex function being convex in both arguments, whence it follows that p + p1/p is concave (0, 1j. The next proposition, A.2.2 shows that it is furthermore continuous al 0 and therefore concave on [].
A.2.2 Proof of Proposition 4.12
Proof.
FF1
Consider σv, (1) = 1i (t).
Hence
lim Øv, (1) = lim f 1.0
1
x= 1/t
-x)
210
x x- 00
By the supercoercivity assumption on ʃ (equivalently, on f), we have lim f(x) 00. But then, its inverse f-1 (resp. }-1)) satisfies the mirror property of "antissupercoercivitt""""" lim [()) = 0. For the case when f is finite, see (Kosmol & Müller Wichards 2011 Lemma 6.1.26). If f is not finite everywhere, then its inverse is bounded; therefore lim
0.
It is easy to see that this holds also in the other direction. Assume that lim,jo Φv, (t) = 0. Then f is supercoercive.
It remains to establish that the same property holds for ΦR,- This readily follows from the equivalence of risk and regret: we can apply Theorem 4.22 in Fröhlich & Williamson (2022) and consider the fraction:
ΦR, (1) K" lim 410 Φv, (L)
If Φv, was continuous 0, but OR, was not, then K" oo and then it follows that the norms would not be equivalent.
We need to show that f is finite then (0) lim sel oo and otherwise d'(0) < cc. We begin again by considcring Φv, From the definition of the difference quotient get:
FF2
8y (0) = lim Φv, (t) t
42

---

Published in Transactions on Machine Learning Research (01/2023)
and further
Φv, (1)
lim f-1 (1) lim (z). 110
lim <40
If f is finite, then is and its inverse is unbounded and therefore obtain Vi, (0) 00. If f(x) for some x E Rt, then its inverse is bounded; consequently, Vy, (0) < co.
We also have Vv, (0) xo + dR (0) = oo, that is, the fundamental function of the risk measure shares the same property. This readily follows by applying Fröhlich & Williamson (2022 Corollary 4.27).
A.2.3 Proof of Proposition 4.15
Proof. First, we observe that there exists lg < such that (v, (to) = (1/to) = 1, unless f is the pathological divergence function corresponding the expectation. This is due to f-) > 1 (unless limæ11 f(x) = so). Since the fundamental function is increasing and continuous, it follows that there exists to such that ov, (4) (1/to) 1.
Let E' be arbitrary measurable set of measure t. The fundamental function is then:
Φn, (t) Rg(XE)= = sup {L Xx(W)) dw : Z 2 0, B 11 =
Next, observe that for t < to the following dual variable f (1/t) is feasible:
1/t
E E
Z(w)
1-/00"(100)
$ E.
Since Z 2 0, [||] 1 (by construction) and furthermore, noting the left-continuity of f-1:
1 1- = ((-)))))+(- ( (1/t) $
1-t
This holds since i <0<<<1 and /11(1 1 and (1/1( < (since S (0), we have
0 05 tf (1/t)
1-t 31,
and because f is by assumption a Young divergence, f(*) = 0. Thus we have shown that this Z is a feasible dual variable. With this Z, we obtain that on, (t) 2 t (1/t) for t S to. But this just the fundamental function the corresponding regret measure. Since holds that Rx(X) I VX() due to the infimal convolution (this is also clear from the dual representations), we find that ΦR, (t) = (1/t) for < lo-
Due to translation equivariance it holds that dR, (t) S Vt and ΦR, (1) 1. Thus obtain the statement Φn, (t) = min
Finally, in the special case of the expectation, (x = E[X], where
0 5 x 5
f(x)
x 1
obtain (1/t)) = for <LS S and hence the formula holds and reduces to ΦE(() t.
7
43

---

Published in Transactions on Machine Learning Research (01/2023)
A.2.4 Proof of Proposition 4.19
Proof. The fundamental function of an Orlicz norm || 11 with conjugate Young function & Kosmol Müller-Wicharde 2011 Lemma 7.4.4):
((t) = t (1/t), t Ee (0.1).
Due to properties the perspective transform, it is guaranteed that the resulting Φ-1 is concave and increasing on | (1), observe that
Φ (z)
x x21.
II & is concave, - is convex and then its perspective transform (8,x) + xs-0)(s/x) is convex in both arguments (this is well known). By setting = we see that + - (x) is convex, hence 4-1 is concave.
To see that Φ- is increasing on ( oo), note that t ++ @(t)tt is decreasing due to quasiconcavity, hence <~> (1/t) is decreasing and therefore x + (xx) is increasing.
The condition that continuous at indeed necessary for & to be supercoercive, cf. A.2.2
Also, if (0) is unbounded, + is finite and the behaviour of + is fully specified for all x 2 --) If contrast (0) < oc, we have bounded --1 and can therefore identify the behaviour of & for $ (-)(1) s S bo. where bap sup + < oo = =0 (0). Thus such 4 is infinite for all x bą.
However, for 0 x < Φ (1), we cannot identify the behaviour f. Consequently, any left- cett-contimouss increasing convex "completion" of f this interval will be compatible with the fundamental function. To see that least one such choice of must exist, take & and extend it to the whole nonnegative domain by:
(() 05151
x'(1)
h(t)) t>1,
in such manner that Φt RR+ R+ is concave; such øi obviously exists: for instance choose a linear continuation with slope 4'(1) shifted by ((1,, i.e h(t) 4(1) 4 (1)(t 1). Then using the equality
1 (1/1) o
we can construct an Orlicz norm with the desired properties. Repeating the above arguments then yields that O is legitimate Young function, where we explicitly set +()) O.
A.3 Tail-Specific Marcinkiewicz and Lorentz Norms
A.3.1 Proof of Proposition 5.6
Proof. From (Krenn et al. 1982 Lemma 5.3, Theorem 5.3 we know that M1 to M3 are equivalent for
the case of an infinite mcasure space, i.c. P(2) d0. First assume that we knew that MI equivalent for P())= 1. Then, M3
M3 are also
M4 let. ||t ox (1)/ co. Consequently:
|| ox (1)44| ||Ex(0)|M sup
Ex (1) < 3K EX Ex(!)
octg1 (x)
Also, as a general fact, the decreasing rearrangement. is dominated by the maximal function, iee. Ex(() < Exe (t), E (0, 1]. Conversely, if Ex and Ex are equivalent, M5 holds (Rubshtein et al. 2016 p. 162). Finally, M5 holds, then M3 obviously holds by definition of || "[Mevva
Exlla, & K K ||E|||MMMsess = K sup USE51 (1)
Ex (x oc.
177o prevent confusion, we remark that Krein et al. (1982 use the symbol Ms. to denote what in our notation is Mage that is, we index the Marcinkiewica norm its lundamental Tunction, whereas Krein al. (1882) index it by fundamental function of its associate space.
44

---

Published in Transactions on Machine Learning Research (01/2023)
Observing that (t)/t) (t).). the equivalence of M2 and M3 for the case of both P(.) = 1 or P(Ω) oo, can be obtained from Pick et al. 2013 p. 265). In summary, we have shown that the implications form a circle.
are also equivalent the case of P(22) = Assume without loss of (this corresponds simply rescaling one norm and then using the associate relation of the fundamental functions). Consider the following extension of ox [0, (0, + (0)
We shall now show that M1 to M3
generality that (x) xx
to the whole nonnegative domain:
054sl (t) 1 (1(--1) 1<<<,,
(x
which exlends ex as a straight line and where the slope chosen so as to guarantee concavity (assume without loss of generality that Φx is concave, otherwise first use the least concave majorant to concavify the quasi-concave ox', thereby preserving equivalence of Marcinkiewicz norms). Let @(t) t//x(tt. We now show the following:
Lemma A.2. | las, and ||- Ill,, are equivalent on Ω = [0, if and only if I- IlM, and l1 Illlas, are equivalent Ω a= (0)
The Marcinkiewicz norm in the case of P(tt) = oo is defined in the obvious way, as sup 0<
| Ma
(0(1)x"()))
and X* R+ + R+ and X (0, oo) + R++ are defined in the obvious way.
Proof of Lemma:: First assume that IMg and || I|,; are equivalent on 10, Then is trivial that las, and ||- MM, are equivalent on = [0. 1], by simply extending a random variable X E Ma from Ω = 1] to & = [0, oo) as (1) = for > For the converse direction, assume that I- and || lIM, are equivalent on 9 [0, 1]. Then we want show that I- las, and || Il,, are equivalent by demonstrating that ( + (1)tt 1/õ(1) e M2, which is equivalent to the desired statement due to Pick et al. (2013) p.
265). Then:
||t - xxx sup 200 S (4xx:(4) dw 80. oct<x
By assumption, we have that t + ox (t)/t / Ms and therefore only have to investigate > 1. But:
= sup K<t< 1 + o' (1)(t- t) (L 1/0(w) dw + 1 1/((w) dw)
Since t + ox (t)/t Ms. we have c = fo 1/6(w) dw I supocesı o(t) So 1/0(w) dw < oo; essentially this follows from L being the largest of all r.i spaces. Furthermore:
1
(ct
c+ ["
1+ &x (1) 1) dw dw)
I<< sup 1 (1)(t 11ff-1) 1)
= sup 1ete 1 &x (1)(t-1) (1(f--) 1)) (c (1) - ox (o) - o'x- + log(t)) & cc.
1
To ensure that this is finite, we need to chock only the limit as t - oo:
(x)t 6'x (1) log(t) ox (1) log(t)
lim +00
1 + ox (1) -
Thus we have that +++ ox (t)/t Mj, which completes the proof of the lemma. Finally, we need
show the following:
45

---

Published in Transactions on Machine Learning Research (01/2023)
Lemma A.3. Vs suPocIS1
(x/(//s (x)
if and only if Vs supoercco (x)
(x/(1/a)
Proof of Lemma: That the condition on ox implies the condition of ox is obvious. Conversely, assume the condition holds for 6x on [0, 1]. Then we need show only that also holds for Φx when < t oo, they coincide for 0 S 1. Let 1 be arbitrary. For any fixed 1 t, we obviously have that the fraction is < 1 and thus need to investigate only the limit as t - xc. Then:
1 (1(1//5)
lim 1-00
11) <1.
1 (1)(t
Chaining all equivalences together, we obtain that measure spaces.
MI to M5 are equivalent for both finite and infinite
A.3.2 Fundamental Functions of Lexp and LlogL
1.0
Lexp
0.8
LlogL
0.6
0.4
0.2
0.0
0.0
0.2
0.4
0.6
0.8
1.0
Figure 2: Fundamental functions of Lexp and LlogL.
The fundamental function of Lexps given by Φ (1) 1/(1 log(!)), is only quasiconcave, concave. In contrast, the fundamental function (t) = t + (log(1/t) t/ d/Oee (t) is concave; see Figure 2
A.3.3 Proof of Proposition 5.11
Assume Y is differentiable on 1). Then the condition that éx (t) = (+)+)++= ((1t))==Fx(00 (1- -t incrcasing becomes
d Fx (s)
(1-) >) + dt 1
(s)
= 1-
Fx (s)
ds
We Gronwall-type inequality.
Proposition A.4 (adapted from Bainov & Simeonov
2013
Lemma 1.1)). Let f 10, oo) be continuous and
differentiable and g [0, oc) be continuous; assume f(0) v So and
ds f(8) {s))
200
Then:
exep exp ( (()) (())) ar)
f(8) < Jo
As 2 0.
46

---

Published in Transactions on Machine Learning Research (01/2023)
Applying this statement with f(s) Fx (s), fo = === (0) and g(s) 11(11111111 and constraining the domain yields
() exp 1 dr
E (0.1)
logg[1 - x))
Vs E (0,1)
o (
vs E (0,
+ Y (t) (t) (1)
A4 E (0,1).
A.4 Utility-Based Shortfall Risk
Definition A.5 Föölmee Schied 2002 2016 ). A convex risk measure on some space X is a functional
R: X R u foo), which monotone CS translation equivariant {C4 and convez:
R(aX + (1 a) & aR(X) + (1 -a)
Typically, also R(0)- = 0 is demanded. Positive homogeneity and subadditivity together imply convexity; but convexity on its own weaker condition. Thus a coherent risk measure is also a convex risk measure.
In this section, we define loss function & R- I to be function which is Young function RR and further satisfies {(x) 0 for x S 0. Consider the acceptance set:
A= (X [[((()) mo)-
The intuition is that a decision maker who has a loss function and some decision threshold To considers a loss variable X "acceptable" B[e(X)] S zo- Subsequently, let Io 1. We assumed ((x) for x S 0, which is non-standard, but allows for a cleaner connection to Orlicz space theory. Intuitively, this implies that acceptable random variables are characterized by their losses only, whereas gains (negative values) are neglected in the acceptance set. This also fits with our overall focus on right tails only. However, the trade-off between losses and gains naturally arises by enforcing translation equivariance. A convex risk measure, the UBSR, can be constructed as follows:
UBSR; = inf E R X - m AA.
Hence, UBSR, specifies the smallest amount that must be subtracted from the risky position X so as to make it acceptable. This construction automatically guarantees translation equivariance and since & is increasing and convex, UBSR, is a convex risk mcasure (Föllmer & Schied 2002
To study the envelope representation of UBSRe, one must choose pair of Banach spaces. Fölleer & Schied (2002) studied the UBSR on _ which is an overly conservative choice. In fact, the natural setting is the framework of Orlicz spaces, shown by Biagini & Frittelli (2009) and Arai (2011). We denote the conjugale of l as l". Observe that f is also legilimate Young function 18 I+, which is supercoercive and E (xr) = 00 for x < O. To state the envelope representation of UBSR, on its natural space, we need to define the Orlicz class.
Definition A.6. Given a Young function 4, the Orlicz class po is defined as
pt = =X B[O(IX)) < c0).
By definition of the Luzemburg norm, we obviously have pd C L. We remark that ph need not be linear subspace of the Orlicz space L., bul it does contain a linear subspace itself, called the Orlicz heart a However, if & satisfies the Ag condition (Kosmol & Müller- Wichards, 20011 p. 197, p. 234), then fact M p$ L. Intuitively, such connot grow too fast. For instance, the rapidly growing (xx) (exp(x) 1)x( does not, salisfy A2.
18 For this, assumption that Elg) for y < is important, then e increasing zo). Confer Föllmer Schied 2002 Lemma 11).
Supercoercive means that Tomy 8 (y)/y no, ef. Definition and Föllmer Schied 2002
47

---

Published in Transactions on Machine Learning Research (01/2023)
Indeed, the UBSR is finite on the Orlicz class p (consider =0); here, understand f its restriction to IRt, where it is Young function.
Proposition A.7 (Föllmer & Schied 2002
Biagini & Frittelli (009]). The following envelope represen-
tation holds for the utility-based based shortfall risk measure UBSR; P - R:
UBSR&(X) = sup
[[ sup ar a(Z): e c 2 0, ElZ 1}.
where the penalty function a C + IR is:
a sup([ X E L. X sup[B[X2] X X22,,X2,, s 1}.
(18)
In fact, we observe that (z7)) || 1-- Z 2 0, i.e. the penalty function is an Orlicz norm, and therefore:
UBSR((X) sup [[XZ| inf t(1 E (Z/t)) Z L. Z 0, E =1
(19)
Proof. We adapt Proposition 32 in [Biagini & Frittelli 2009) to loss-based orientation and to the addi-
tional assumplion that ((x) = for S 0. By definition, ar (2 || 9- is finile L but a(Z) is only evaluated for Z 2 0, hence we take C as its domain to emphasize this. The restriction to X 2 0 in (18) is possible since Z 2 0 and ((x) = 0 for x < 0, hence a negative part of X is neglected.
Föllmer & Schied 20002; who established a variant 19 in the - setting, remarked in passing the similarity to the Amemiya formulation of an Orlicz norm, but did not further go into it.
Among the convex risk measures, the utility-based shortfall risk measure has particularly desirable properties such invariance under randomization (Giesecke et al. 2008) and elicitability (Bellini & Bignozzi] 2015). Most important in the present context, however, is that since the penalty function is an Orlicz norm, immediately apply our method of Section G to achieve desired lail sensitivity. In contrast to coherent risk measure, a convex risk measure considers all alternative probability measures Qz from an entire space, but they are considered more or less relevant due to the penalty function; note that Z 2 0 and E|Z] 1, hence the Qz induced via [14 are valid probability measures. If the penalty function only takes on the values 0 and +00, the risk measure coherent. In this sense, a convex risk measure acts a smoother fashion than a coherent risk measure, where only subset of probability measures considered all. By specifying the tail sensitivity of the penalty function || we specify how alternative probability measures, conceptualized density ratios with respect to the base measure, are punished for their reweighting in tail-sensitive way.
In the context of machine learning, the UBSR seems particularly useful to handle tail risk in reinforcement learning applications, as exemplified by Shen et al. (2014). More generally, any context which calls for a risk measure without positive homogeneity is a possible application for the UBSR.
Example A.8. Föllmer & Schied 2002) Let 1 < p xo and f(x) = 12x²) [(0,00)- Then
(r)) 8
y 0
(2
N 0,
and
a(Z) = //1 // (E(Z") 1/
Consider + c0. Then (v) = io.11 and therefore UBSR, = [[ since any Z which has Z(w) > 1 for some w is infinitely punished; this is just the L - associate relationship.
48

---

Published in Transactions on Machine Learning Research (01/2023)
A.5 Plain Language Summary and Diagram
When training machine learning systems with dala, engineers typically try to minimize average prediction error. Yet the justification for why the average is the best summary of those errors is weak: some errors could be very high, some very low. When a prodiction error corresponds to a human individual (for example when we predict who gets a loan), this is problematic, as some individuals then suffer a much higher error than others. In such a situation, we would like to have a summary which also penalizes inequality of prediction errors. To this end, we can minimize a "risk measure" of the prediction errors. A risk measure is like an average, but puts more weight on the higher prediction errors. Many researchers have studied risk measures and proposed many different families of risk measures. But one important aspect, which has not been studied systematically yet, is how sensitive risk measures react to very large errors. This is called the tail scnsitivity and it is the focus of our work. We study risk measures in a unified framework, which was proposed by mathematicians. In this framework, we find that there exists a simple way of characterizing how tail sensitive risk measure is. An outcome of our work is a method generate risk measures which have exactly desired fail sensitivity. This can be used by machine learning engineer to flexibly tune the influence of very high prediction errors.
49

---

Published in Transactions on Machine Learning Research (01/2023)
Figure 3: Summary of key elements in the paper.
x²)()) - 1 ff'xx (w) de cVar
('W = 2 (x(A) & w) = (1 w)
(x(A) = P(ww e a: [X())) > An) = =x >)
Idea: constrain growth of CVar's. s.
[|X] M. 00 < (²) Ex(t)
|| sup x
I Equivalent? Check Krein
Idea: constrain growth of quantiles.
X + 3e: x*(t) < Ex(()
X M, Oct
Quasi Marcinkiewicz Norm
Marcinkiewicz Norm
condition (Prop 5.6)
0cts1
Basic Quantities
(x
Equivalent? Check "dual" to condition Prop 6.7.
f(a) = (z) - +(1)) X1.30)
Ryg (X) sup [ Z Qz e C, P) <
(l = sup (t) 1Z < 1), (1/=)) = (x)
1Xx1 = sus (L Z (w) Z e 220,, 11
Design idea: choose envelope function as backwards quantiles of reference distribution, ie. Ex(t) = (t) For example: choose Y to exponential random variable
(x) = (t)
Envelope Function and Fundamental Function express tail sensitivity:
|| = sup
f-Divergence Risk Measure
Rearrangement Invariant Norm
|| ||
1
Orlicz Norm
(') (w) du 1/ < 1, EX
ZZ (t) 1 < 1}, Qx(()
11
1:1 (up to equivalence)
Ex
t
Rx f. Fx - w) d)x(ww
|| = -f (") dox(ww
Spectral Risk Measure
Ex 4 Ex + Φx 47
Note:
Lorentz Norm
50


Reviews:
[{"review_id": "Cxu2GrC1Fs", "reviewer": "Reviewer_chCF", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- The purpose of the paper is relevant, and the proposed solution is elegant\n- The paper is globally well-written, although it is pretty dense\n- It draws interesting connections with fields that could be of interest to the machine learning community\n- As far as I could check the maths are neat and correct\n\n**Weaknesses**\n- The paper is very dense, and pretty involved to read, due to the introduction of many concepts that most people in the machine learning community will not be aware of. I must say that the latter seems inevitable given the nature of the results provided, and some credit should be given to the authors, who did their best to make it as digest as possible\n- One side effect of the above point is that it is sometimes difficult to identify the proper contribution of this work, with respect to previously introduced notions. Again, this seems inevitable, as a lot of context is required, and the authors are extremely clear about what is new, it is more of a byproduct of the nature of the article\n- I might missed something, but the authors motivate the introduction of regret measures by the need for considering gains and losses differently. However, in the Expected Risk Minimization framework considered, most loss functions have only positive values, such that the latter seems overly sophisticated. Do the authors have precise examples where regret measures are needed? In addition, the norm induced by $V_g$ (which is studied in the rest of the paper) uses the absolute value and looses the asymmetry, so it was not clear to me why introducing an asymmetric measure was needed\n- Another important drawback I see about the proposed risk measures is their computability. In particular, how easy is it to estimate them with finitely many observations? Furthermore, Expected (or Empirical) Risk Minimization is often tackled through gradient descent, which means differentiating through the risk measure: how would this tranpose with the propose risk measures?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose alternatives to expected risk minimization by introducing risk measures that are tail sensitive. First, they introduce _Orlicz regret measures_, a generalization of Orlicz norms that treat gains and losses differently (Definition 3.8). However, regret measures are not translation invariant, so that authors introduce _divergence risk measures_ (Proposition 3.19). Then, Orlicz regret measures and divergence risk measures are analyzed through the lens of rearrangement Banach norms. In particular, their fundamental functions are derived (Propositions 4.11 and 4.15). The latter are in one-to-one relationship with the envelope functions, that define the tail sensitivity by constraining the tail behavior of the dual variable. In Section 6, the authors provide several examples of Orlicz norms obtained by different choices of envelope functions (Examples 6.3 to 6.6). In Section 7 some Orlicz deviation inequalities are presented, while Section 8 focuses on computing some risk measures.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "**Major**: See _weaknesses_ paragraph above\n\n**Minor:**\n- General: I would not number the equation which are not referred to in the text, but it is more a matter of taste\n- p.2, 2nd paragraph: \"coherent\" is not yet defined. At least a pointer to Sec. 3,4 could help\n- Def. 2.3: I would emphasize more on the Amemiya norm (even including it in the definition), as it is the one which is used to generalize to Orlicz regret measures\n- Prop. 3.2: \"and and\"\n- Def. 3.8: I would add a remark, as $g$ is used here instead of $\\Phi$ (an exact analogy would require using $f$). Things are clarified in Prop. 3.11, stating that $g$ is the young function, so the latter could be moved next to the definition for more clarity.\n- Prop. 3.9: V5, should be $V_g$ instead of $V$\n- Prop. 3.10 and 3.14: the link between these propositions and the rest of the section could be improved\n- I find it a bit misleading to introduce risk measures for $g_\\varepsilon$. Why not a for generic $g$ and then specifying that the envelope representation changes when considering $g_\\varepsilon$\n- Eq. (49): $\\phi_\\mathcal{X}(0) = 0$\n- Eq. (104): remove one closing parenthesis in the l.h.s.\n- Eq. (128): $V_g$\n- Appendix A.2.3.: first line, $t_0$ instead of $t$", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "YB8FF9zpfq", "reviewer": "Reviewer_gcQw", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n\na)\tThe paper is written in a clear and understandable way\n\nb)\tThe authors provide an interesting connection between the tail behavior of random variables and their Luxemburg norms.\n\n\n**Weakness**: \n\nThe authors start the paper emphasizing the relevance of their contribution with respect to the field of machine learning. They justify this claim by stating that the Average Loss is an insufficient measure of risk as it does not take into account tail properties of the loss. However, afterwards the authors almost solely emphasize on theoretical properties of risk measures, not considering this connection to ML anymore. This leaves important questions unanswered.\nSome are:\n\n**a.)** How exactly should the risk measures/norms be included into ML algorithms and how does it change the resulting prediction? \n\n**b.)** What is the benefit of employing the risk measures/norms?\n \n**c.)** Can the presented risk measures/ norms be computed efficiently if there are only samples available?\n \n**d.)** How much computational cost does it add to the ML algorithm? \n\n**e.)** What is the interpretation of small loss tails? Just fitting the observed data?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This contribution regards the construction of spaces of random variables which inherit specific tail properties. These spaces are Orlicz spaces which are closely related to risk measures derived from f-divergences. The paper relates their norms with the tail properties of the random variables contained in the space. It further provides a deviation inequality, bounding the excess probability with the Luxemburg norm as well as the envelope function. ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Adress the questions **a.)** - **e.)**", "broader_impact_concerns": "No concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "7lNGOsUO7U", "reviewer": "Reviewer_2ucL", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n--\n- This work provides new risk measure framework which extend the previous risk measures\n- Some of the discussions of this paper is interesting for me, e.g., the section 7 discussion is really interesting.\n\n**Weaknesses**\n--\n- There is *no experiment* in this work to show that their risk measures can improve the some metrics in machine learning problems in comparison with empirical risk minimisation.\n- The *application of risk measures* in machine learning is not clarified.\n- Some related works are not discussed, e.g., [1], [2].\n\n\n\n**References**\n--\n- [1]: Föllmer, Hans, and Thomas Knispel. \"Entropic risk measures: Coherence vs. convexity, model ambiguity and robust large deviations.\" Stochastics and Dynamics 11.02n03 (2011): 333-351.\n- [2]: Li, Tian, et al. \"Tilted empirical risk minimization.\" arXiv preprint arXiv:2007.01162 (2020).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this work the authors propose a general approach to tune tail sensitivity of a risk measure and add this concept to empirical risk minimisation.\nThey also provide a connection between tail sensitivity and fundamental functions. For example, the Orlicz regret measures are proposed and it is shown that its extension is f-divergence risk measures.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "I propose the following changes:\n\n- **(Critical)** The authors must compare the implementation of their risk measure to ERM in a straightforward example or experiment.\n- **(Critical)** I know there are some applications of the proposed risk measures in finance, but I prefer the applications in machine learning or statistical learning. For instance, why should someone care about the tail sensitivity in machine learning? The authors should make clear the applications of their proposed risk measures in machine learning or statistical learning. When should the risk measure be designed depending on our scenario?\n- **(non-critical)** How can we evaluate the robustness of your suggested risk measures in light of label noise or outlier?\n- **(non-critical)** Is it possible to analysis the generalisation error or excess risk under the proposed risk measures for a supervised learning scenario?\n- **(non-critical)** As the paper is too long, I propose to add a summary page or table to summarise your result in that page or table.\n", "broader_impact_concerns": "This submission does not require a Broader Impact Statement.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors have addressed the concerns raised by reviewers, as well as incorporated changes to the presentation, especially towards providing descriptions of the work in closer-to-lay terms. The authors have promised to do another round of proof checking, which I am fine to permit, and so I'm recommending acceptance, with minor revisions, to allow these small changes. I'll add that the opening sentence of the second paragraph (starting "Another concern...") is a garden path sentence for me, and should be revised to have simpler structure.
Decision: Accept (poster)