arXiv:1712.03610v2 [math.PR] 27 Dec 2017

LOGARITHMIC DIVERGENCES FROM OPTIMAL TRANSPORT
AND RÉNYI GEOMETRY
TING-KAM LEONARD WONG
Abstract. Divergences, also known as contrast functions, are distance-like
quantities defined on manifolds of non-negative or probability measures. Using a duality motivated by optimal transport, we introduce and study a parameterized family of L(±α) -divergences which includes the Bregman divergence corresponding to the Euclidean quadratic cost, and the L-divergence
introduced by Pal and Wong [27, 28] in connection with portfolio theory and
a logarithmic cost. They admit natural generalizations of exponential family
that are closely related to the α-family and q-exponential family. In particular,
the L(±α) -divergences of the corresponding potential functions are Rényi divergences. Using this unified framework we prove that the induced geometries
are dually projectively flat with constant curvatures, and satisfy a generalized
Pythagorean theorem. Conversely, we show that if a statistical manifold is
dually projectively flat with constant curvature ±α with α > 0, then it is
locally induced by an L(∓α) -divergence. We define in this context a canonical
divergence which extends the one for dually flat manifolds.

1. Introduction
In this paper we study geometric properties of statistical manifolds defined by
solutions to some optimal transport problems. We show that they characterize
dually projectively flat manifolds with constant curvatures and provide natural
geometries to generalizations of exponential family. Before stating our main results
let us describe the background and motivations of our study.
1.1. Background. Information geometry studies manifolds of probability distributions and measures using geometric ideas. For modern introductions to its theory
and applications we refer the reader to the excellent recent monographs [3] and [9].
Typically the geometry is induced by a divergence which is a distance-like quantity
D [q : p] ≥ 0 defined on the underlying manifold M . Of particular importance is
the Bregman divergence. Given a differentiable concave function ϕ(ξ) defined on a
convex domain Ω in Rd , the Bregman divergence of ϕ is defined for ξ, ξ ′ ∈ Ω by
(1.1)

D(0+) [ξ : ξ ′ ] = Dϕ(ξ ′ ) · (ξ − ξ ′ ) − (ϕ(ξ) − ϕ(ξ ′ )) ,

where D is the Euclidean gradient and a · b is the dot product. (The meaning
of the superscript 0+ will become clear in Section 1.2.) For example, if M is an
exponential family of probability densities where ξ is the natural parameter, the
relative entropy, also known as the Kullback-Leibler divergence, can be expressed
as the Bregman divergence of the cumulant generating function ϕ(ξ) (which is
Date: December 29, 2017.
Key words and phrases. Information geometry, statistical manifold, optimal transport, space of
constant curvature, projective flatness, Rényi divergence, α-divergence.
1

2

TING-KAM LEONARD WONG

convex, so in (1.1) we may consider −ϕ or use D(0−) to be defined in Section 2).
The differential geometry of Bregman divergence was first studied by Nagaoka and
Amari in [23].
Following the general framework established by Eguchi [15, 16], the geometry
induced on M by a given divergence D [· : ·] consists of a Riemannian metric g
and a dual pair (∇, ∇∗ ) of torsion-free affine connections (see Definition 4.1). We
call the quadruplet (M, g, ∇, ∇∗ ) a dualistic structure or a statistical manifold.
The Riemannian metric captures the quadratic approximation of D [q : p] when
q ≈ p, whereas the two affine connections capture higher order local properties
and allow us to define primal and dual geodesics that are in some sense compatible
with the divergence. In contrast with usual Riemannian geometry where the LeviCivita connection is the canonical connection defined by the metric, in information
geometry we usually use a pair of connections (∇, ∇∗ ) to quantify the asymmetry
of the divergence: generally we have D [p : q] 6= D [q : p]. When the divergence is
symmetric (for example when D [q : p] = 12 d(p, q)2 where d is a metric) both ∇ and
∇∗ coincide with the Levi-Civita connection. From this geometric perspective, the
Bregman divergence is fundamental in the sense that it is the canonical divergence
which generates a dually flat geometry, i.e., both the primal and dual connections
∇ and ∇∗ have zero curvature (see [3, Section 6.6] and [9, Section 4.2]; this is
also a limiting case of Theorem 1.3). Moreover, the Bregman divergence satisfies a
generalized Pythagorean theorem which enables explicit computation and geometric
interpretation of projections based on the divergence. By now these results are
classical and numerous applications can be found in the two monographs cited
above.
Motivated by portfolio theory, in a series of papers [26, 27, 35, 28, 36] we introduced and studied the L-divergence (L stands for logarithmic) defined by
(1.2)

D(1) [ξ : ξ ′ ] = log(1 + ∇ϕ(ξ ′ ) · (ξ − ξ ′ )) − (ϕ(ξ) − ϕ(ξ ′ )) ,

where now the potential function ϕ is exponentially concave, i.e, Φ = eϕ is concave.
Being the logarithm of the concave function Φ, ϕ is itself concave and the Ldivergence has a logarithmic correction term that captures the extra concavity.
An important example of L-divergence is the free energy defined for probability
measures ν, ν ′ ≪ µ by
D [ν : ν ′ ] = log

Z

 Z
dν ′
dν ′
dµ − log
dµ,
dν
dν

R
dν
where ϕ(ν) = log dµ
dµ. Exponentially concave functions appear naturally in
many recent applications in analysis, probability and statistics. In particular, let
us mention the paper [17] on Bochner’s inequality on metric measure spaces. Other
examples can be found in the references of [28].
In [28] we established the following results where the underlying manifold is the
open unit simplex:
(i) The dualistic structure induced by the L-divergence (1.2) is dually projectively flat with constant curvature −1, and the generalized Pythagorean
theorem (which has a financial interpretation in terms of optimal trading
frequency) holds true.

LOGARITHMIC DIVERGENCES AND RÉNYI GEOMETRY

3

(ii) The L-divergence can be expressed in terms of the optimal transport map
with respect to a logarithmic cost function, whereas the Bregman divergence corresponds to the classical quadratic cost. Furthermore, displacement interpolations for these transport problems are related to the dual
geodesics of the induced dualistic structure.
It is natural to ask if there is a unified framework that covers both Bregman and Ldivergences. Moreover, can we characterize statistical manifolds that satisfy these
(very strong) geometric properties? On the other hand, Amari and Nagaoka [2] considered α-families of probability distributions and the corresponding α-divergences
that generalize the exponential family and the Kullback-Leibler divergence. We
will see that these questions are intimately related.
The connection between optimal transport and information geometry is a recent
topic that has started to receive serious attention. For example, the recent paper
[4] studies the manifold of entropy-relaxed optimal transport plans and defines
a divergence which interpolates between the Kullback-Leibler divergence and the
Wasserstein metric (optimal transport cost). A dynamic approach is given in [13].
Also see [25] which builds on the ideas in [28] and embeds a large family of optimal
transport problems in a generalized simplex.
1.2. Summary of main results. The key objects of study in this paper are the
L(±α) -divergences where α > 0. The L(α) -divergence is defined by
1
log(1 + α∇ϕ(ξ ′ ) · (ξ − ξ ′ )) − (ϕ(ξ) − ϕ(ξ ′ )) ,
α
where the function ϕ is α-exponentially concave, i.e., eαϕ is concave. To cover
the case of positive curvature, we will also consider a local L(−α) -divergence where
ϕ is α-exponentially convex, i.e., eαϕ is convex. When α = 1 we have the L(1) divergence (1.2), and when α → 0+ it reduces to the Bregman divergence (1.1) of
a concave function (D(0−) is the Bregman divergence of a convex function). Based
on financial arguments, the L(α) -divergence was introduced recently in [36] as the
canonical interpolation between the Bregman and L-divergences. In this paper we
consider also the L(−α) -divergence show that they are of fundamental importance.
While we worked on the open unit simplex in our previous paper [28], here we
consider a general open convex domain Ω in Rd . This answers a question asked in
[28, Section 1] about extending the results to general domains. By a translation
if necessary, we may assume that 0 ∈ Ω (this is used for the normalization of the
function c(α) in (1.4)). Nevertheless, we note that exponential concavity/convexity
impose certain restrictions on the domains of ϕ and the L(−α) -divergence.
In Section 2 we prove the following result which connects the L(±α) -divergence
with optimal transport and generalizes the classical Legendre duality for Bregman
divergence.

(1.3)

D(α) [ξ : ξ ′ ] =

Theorem 1.1. For α > 0, consider the function
(1.4)

c(α) (x, y) =

1
log (1 + αx · y) ,
α

where x · y is the Euclidean dot product. Then the L(α) -divergence (2.5) admits the
self-dual representation
(1.5)

D(α) [ξ : ξ ′ ] = c(α) (ξ, η ′ ) − ϕ(ξ) − ψ(η ′ ),

4

TING-KAM LEONARD WONG

where ψ is the α-conjugate of ϕ and is (locally) α-exponentially concave, and η =
D(α) ϕ(ξ) is the α-gradient which generalizes the Legendre transformation.
Similarly, the L(−α) -divergence has the self-dual representation
(1.6)

D(−α) [ξ : ξ ′ ] = ϕ(ξ) + ψ(η ′ ) − c(α) (ξ, η ′ ),

where now ϕ and ψ are α-exponentially convex.
Precise statements corresponding to Theorem 1.1 are given in Theorems 2.12,
2.13, 2.15 and Corollary 2.23. The self-dual expressions (1.5) and (1.6) are motivated by optimal transport which identifies the dual coordinate system η as the
image of ξ under the optimal transport map. The connection with optimal transport is described in detail in Section 2.4.
Motivated by the close relationships between Bregman divergence and exponential family (see [10]), in Section 3 we consider generalizations of exponential family
that are closely related to the α-family and q-exponential family [2, 24, 5]. We
show that the analogue of the cumulant generating function is α-exponentially concave/convex (Propositions 3.3 and 3.4), and the corresponding L(±α) -divergence
is the Rényi divergence (Theorem 3.6). Moreover, the dual function is the Rényi
entropy. Our results thus provide a new approach to the geometry of Rényi and
α-divergences. When α → 0 we recover the dually flat geometry of exponential
family. The case of finite simplex is discussed in Section 5.4.
In Sections 4 and 5 we study the statistical manifold (M = Ω, g, ∇, ∇∗ ) induced
by the L(±α) -diverence. Our approach, based on the duality in Theorem 1.1, gives
a unified treatment covering both the Bregman and L-divergences and simplifies
the proofs in [28]. The following result summarizes Theorems 5.2, 5.9 and 5.10.
Theorem 1.2. For α > 0, the dualistic structure induced by the L(±α) -divergence
defined on an open convex set Ω ⊂ Rd with d ≥ 2 is dually projectively flat, has
constant curvature ∓α, and the generalized Pythagorean theorem holds. (When
α → 0+ we reduce to the dually flat case.)
By dual projective flatness, we mean that there exist ‘affine’ coordinate systems
(ξ and η) under which the primal and dual geodesics are straight lines up to time
reparameterizations (for the precise technical statement see Definition 5.1). Since
the properties in Theorem 1.2 are very strong and spaces of constant curvatures
play fundamental roles in differential geometry, it is natural to ask if Theorem 1.2
has a converse. Indeed there is one and it will be proved in Section 6 (see Theorems
6.1 and 6.2).
Theorem 1.3. Consider a dualistic structure (M, g, ∇, ∇∗ ) which is dually projectively flat and has constant curvature −α < 0. Then locally there exist affine
coordinate systems ξ and η for ∇ and ∇∗ respectively, and α-exponentially concave
functions ϕ(ξ) and ψ(η) which satisfy the generalized Fenchel identity
ϕ(ξ) + ψ(η) ≡ c(α) (ξ, η).
Moreover, the self-dual representation (1.5) defines locally a canonical divergence
which induces the given dualistic structure. Analogous statements hold when the
curvature is α > 0.
Our results establish the L(±α) -divergence as the canonical divergence for a dually projectively flat statistical manifold with constant curvature, thus generalizing

LOGARITHMIC DIVERGENCES AND RÉNYI GEOMETRY

5

the Bregman divergence for dually flat manifolds (see [19] for another characterization based on affine differential geometry). Recently, Ay and Amari [8] defined
a canonical divergence for an arbitrary statistical manifold and showed that on a
dually flat manifold (i.e., α = 0) it reduces to the Bregman divergence. It is interesting to know if their canonical divergence is consistent with our L(±α) -divergences.
Since the L(±α) -divergences have properties analogous to those of the Bregman
divergence, explicit computations are tractable and a natural question is to study
how they perform in applied problems such as clustering [10] and statistical estimation. Projectively flat connections, statistical manifolds with constant curvatures
and relations with affine differential geometry have been studied in the literature;
see in particular [19, 14, 20, 22, 12]. Using ideas of optimal transport and the
L(±α) -divergence, we consider these properties as a whole and are able to give a
new and elegant characterization of the geometry.
In optimal transport, it is well known that the quadratic cost, which corresponds
in our setting to the Bregman divergence, is intimately related to Brownian motion
and the heat equation (via the Wasserstein gradient flow of entropy, see [18, 7]) as
well as large deviations [1] and the Schrödinger problem [21]. A program suggested
in [28] is to study the analogues for our logarithmic cost functions. We plan to
address some of these questions in future research. Another intriguing problem is
to connect other information-geometric structures with optimal transport.
2. Logarithmic divergences and optimal transport
We begin by defining exponential concavity and convexity as well as the associated L(±α) -divergences D(±α) . They admit generalized self-dual representations
that are motivated by the theory of optimal transport.
2.1. Exponential concavity and convexity.
Definition 2.1. Let Ω ⊂ Rd be an open convex set, ϕ : Ω → R, and fix α > 0.
Write Φ = eαϕ .
(i) We say that ϕ is α-exponentially concave if Φ is concave on Ω. For α = 0+
(that is, the limit as α → 0+ ), we say that ϕ is 0+ -exponentially concave
if ϕ is concave.
(ii) We say that ϕ is α-exponentially convex if Φ is convex on Ω. For α = 0+ ,
we say that ϕ is 0+ -exponentially convex if ϕ is convex.
Assuming ϕ is twice continuously differentiable on Ω and denoting respectively
by Dϕ and D2 ϕ the Euclidean gradient and Hessian, we have

(2.1)
D2 Φ = D2 eαϕ = αeαϕ D2 ϕ + α(Dϕ)(Dϕ)⊤ ,

where (Dϕ)⊤ is the transpose of Dϕ (regarded as a column vector). From this we
have the following elementary lemma which justifies the limiting cases in Definition
2.1.
Lemma 2.2. Suppose ϕ is twice continuously differentiable on Ω and α > 0.
(i) ϕ is α-exponentially concave if and only if
(2.2)

− D2 ϕ − α(Dϕ)(Dϕ)⊤ ≥ 0.

(ii) ϕ is α-exponentially convex if and only if
(2.3)

D2 ϕ + α(Dϕ)(Dϕ)⊤ ≥ 0.

6

TING-KAM LEONARD WONG

Here the inequalities are in the sense of positive semidefinite matrix.
Note that because of the product term in (2.1), we cannot pass between exponential concavity and convexity simply by considering −ϕ. In particular, from
Lemma 2.2 we see that an α-exponentially concave function is always concave, but
an α-exponentially convex function is not necessarily convex. In Section 4 we will
regard (2.2) and (2.3) as the Riemannian matrices of D(α) and D(−α) respectively.
Remark 2.3. Given an α-exponentially concave function ϕ on Ω, we cannot extend
ϕ to Rd simply by letting ϕ = −∞ on Rd \ Ω. This is because e−∞ = 0 but Φ = eαϕ
is required to be non-negative and concave.
2.2. L(±α) -divergences. By convention we always let α > 0. Suppose ϕ is differentiable and α-exponentially concave. By the concavity of Φ = eαϕ , for any
ξ, ξ ′ ∈ Ω we have
′

(2.4) Φ(ξ ′ ) + DΦ(ξ ′ ) · (ξ − ξ ′ ) ≥ Φ(ξ) ⇒ 1 + αDϕ(ξ ′ ) · (ξ − ξ ′ ) ≥ eα(ϕ(ξ)−ϕ(ξ )) .
This motivates the following definition of the L(α) -divergence. It is different from
the Bregman divergence in that our divergence is given by the ratio rather than
the difference in (2.4).
Definition 2.4 (L(α) -divergence). If ϕ is differentiable and α-exponentially concave, we define the L(α) -divergence of ϕ by
1
(2.5) D(α) [ξ : ξ ′ ] = log (1 + αDϕ(ξ ′ ) · (ξ − ξ ′ )) − (ϕ(ξ) − ϕ(ξ ′ )) , ξ, ξ ′ ∈ Ω.
α
For α = 0+ , we define the L(0+) -divergence by the Bregman divergence (1.1).
From (2.4) we see that D(α) [ξ : ξ ′ ] ≥ 0, and if Φ is strictly concave, then
D [ξ : ξ ′ ] = 0 only if ξ = ξ ′ .
For an α-exponentially convex function, the analog of (2.4) is
(α)

(2.6)

′

1 + αDϕ(ξ ′ ) · (ξ − ξ ′ ) ≤ eα(ϕ(ξ)−ϕ(ξ )) .

Unfortunately, now the left hand side of (2.6) may become negative and so its logarithm may not exist. Nevertheless, we can define the L(−α) -divergence locally when
ξ and ξ ′ are close. This is sufficient for defining the dualistic structure (g, ∇, ∇∗ ).
Definition 2.5 (L(−α) -divergence). If ϕ is differentiable and α-exponentially convex, we define the L(−α) -divergence of ϕ locally by
1
(2.7)
D(−α) [ξ : ξ ′ ] = (ϕ(ξ) − ϕ(ξ ′ )) − log (1 + αDϕ(ξ ′ ) · (ξ − ξ ′ )) ,
α
for all pairs (ξ, ξ ′ ) in Ω such that the logarithm in (2.7) exists. For α = 0+ ,
ϕ is convex and we define (globally) the L(0−) -divergence as the usual Bregman
divergence
(2.8)

D(0−) [ξ : ξ ′ ] = (ϕ(ξ) − ϕ(ξ ′ )) − Dϕ(ξ ′ ) · (ξ − ξ ′ ).

2.3. Duality of L(±α) divergence. In this subsection we develop a duality theory
for α-exponentially concave/convex functions. Some of these ideas are implicit in
[28]. In particular, we generalize the Legendre transform and conjugate which will
be used to formulate the self-dual representations of the L(±α) -divergences. In
Section 2.4 we consider an abstract version of this construction terms of of optimal
transport.

LOGARITHMIC DIVERGENCES AND RÉNYI GEOMETRY

7

To ensure that the potential function ϕ is well-behaved and the differential geometric objects associated with the L(±α) -divergences are well-defined, we will impose throughout this paper some regularity conditions.
2.3.1. Duality for α-exponentially concave functions. We first consider the case
where ϕ is α-exponentially concave. By convention we always assume α > 0.
Condition 2.6 (conditions for L(α) -divergence). We assume ϕ is smooth on Ω and
the matrix −D2 ϕ − α(Dϕ)(Dϕ)⊤ is strictly positive definite on Ω. Moreover, by
considering a translation of the variable ξ if necessary, we assume without loss of
generalizty that for all ξ ∈ Ω we have
(2.9)

1 − αDϕ(ξ) · ξ = 1 − α

d
X

ξℓ

ℓ=1

Note that by strict concavity of Φ = e

αφ

∂ϕ
(ξ) > 0.
∂ξ ℓ

and (2.4), (2.9) holds whenever 0 ∈ Ω.

Observe that there does not exist any positive and strictly concave function Φ =
eαϕ on the line (infinite in both directions). Thus, in order that Condition 2.6 holds,
the domain Ω must not contain any line (rays are fine). This is different from the
classical Bregman case where the concave function may be globally defined on Rd .
Examples of feasible domains in our setting include the open simplex (considered
in [28]) and the positive quadrant Rd++ = (0, ∞)d .
Condition (2.9) will be used in the following construction. For each ξ ′ ∈ Ω (which
is open), the supporting tangent hyperplane of Φ = eαϕ at ξ ′ is given by
′

ξ ∈ Ω 7→ Φ(ξ ′ ) + DΦ(ξ ′ ) · (ξ − ξ ′ ) = eαϕ(ξ ) (1 − αDϕ(ξ ′ ) · ξ ′ + αDϕ(ξ ′ ) · ξ) .
Thus (2.9) is equivalent to positivity of the constant coefficient. By concavity of Φ,
for ξ ∈ Ω we have
(2.10)

′

eαϕ(ξ) = min
eαϕ(ξ ) ((1 − αDϕ(ξ ′ ) · ξ ′ ) + αDϕ(ξ ′ ) · ξ) .
′
ξ ∈Ω

Note that the minimum is attained uniquely by ξ ′ = ξ. This and the logarithm
is the basis of Theorem 2.12 which is our duality result. To prepare for it let us
introduce some notations.
Definition 2.7 (α-gradient). We define the α-gradient D(α) ϕ(ξ) at ξ ∈ Ω by
(2.11)

D(α) ϕ(ξ) =

1
Dϕ(ξ),
1 − αDϕ(ξ) · ξ

where Dϕ = D(0) ϕ is the Euclidean gradient.
Definition 2.8 (dual coordinate). We define the dual coordinate η by
(2.12)

η = D(α) ϕ(ξ).

We let Ω′ = (D(α) ϕ)(Ω) be the range of η.
This terminology is justified by the following result.
Proposition 2.9. The α-gradient D(α) ϕ is a diffeomorphism from Ω onto Ω′ which
is an open set in Rd .

8

TING-KAM LEONARD WONG

Proof. It is clear from (2.11) that the map D(α) ϕ is smooth.
To show that D(α) ϕ is injective, suppose towards a contradiction that D(α) ϕ(ξ) =
(α)
D ϕ(ξ ′ ) for distinct ξ, ξ ′ ∈ Ω. By definition, we have
Dϕ(ξ ′ )
Dϕ(ξ)
.
=
1 − αDϕ(ξ) · ξ
1 − αDϕ(ξ ′ ) · ξ ′
It follows that for any ξ ′′ ∈ Ω, we have
1+

αDϕ(ξ) · ξ ′′
αDϕ(ξ ′ ) · ξ ′′
.
= 1+
1 − αDϕ(ξ) · ξ
1 − αDϕ(ξ ′ ) · ξ ′

Rearranging, we have the identity
1 + αDϕ(ξ ′ ) · (ξ ′′ − ξ ′ )
1 + αDϕ(ξ) · (ξ ′′ − ξ)
.
=
1 − αDϕ(ξ) · ξ
1 − αDϕ(ξ ′ ) · ξ ′
Letting ξ ′′ be ξ and ξ ′ respectively, we see after some computation that
(1 + αDϕ(ξ) · (ξ ′ − ξ)) · (1 + αDϕ(ξ ′ ) · (ξ − ξ ′ )) = 1.
Observe that this is equivalent to
D(α) [ξ ′ : ξ] + D(α) [ξ : ξ ′ ] = 0,
which is a contradiction since D(α) [ξ ′ : ξ] , D(α) [ξ : ξ ′ ] > 0 by Condition 2.6. This
proves that the α-gradient is injective.
Finally, from Condition 2.6 again we have −D2 ϕ − α(Dϕ)(Dϕ)⊤ > 0. This can
be used to show that the Jacobian ∂η
∂ξ is invertible. By the inverse function theorem,
(α)
′
D ϕ : Ω → Ω is a diffeomorphism and Ω′ is open. (Essentially the same result is
proved in [28, Theorem 3.2] so we do not provide the details. Also see (4.6) below
which gives an explicit expression of the Jacobian.)

Remark 2.10. Note that the range Ω′ of the α-gradient may not be convex even
when α = 0+ . An explicit example is the concave function ϕ(ξ) = log(1 − kξk2 )
defined on say the open square Ω = (−0.5, 0.5)2. To ensure convexity of Ω′ so that
we have a complete correspondence, we may consider analogues of convex functions
of Legendre type (see [30, p.258]). Nevertheless, as we have shown the dual variable
η is always a global coordinate system of the manifold M = Ω.
Definition 2.11 (α-conjugate). Consider the function c(α) defined for x, y ∈ Rd
with 1 + αx · y > 0 by
1
(2.13)
c(α) (x, y) = log(1 + αx · y),
α
We define the α-conjugate ψ of ϕ on Ω′ = D(α) ϕ(Ω) by

−1
(2.14)
ψ(η) = c(α) (ξ, η) − ϕ(ξ), ξ = D(α) ϕ
(η).
By (2.19) below 1 + αξ · η > 0, hence c(α) (ξ, η) and ψ(η) are well-defined.

Theorem 2.12 (α-duality). For ξ ∈ Ω, η ∈ Ω′ we have


(α)
′
′
c
(ξ,
η
)
−
ψ(η
)
,
(2.15)
ϕ(ξ) = min
′
′
η ∈Ω

(2.16)



(α) ′
′
c
(ξ
,
η)
−
ϕ(ξ
)
.
ψ(η) = min
′
ξ ∈Ω

LOGARITHMIC DIVERGENCES AND RÉNYI GEOMETRY

9

Furthermore, ψ is locally α-exponentially concave on Ω′ in the sense that −D2 ψ(η)−
α(Dψ(η))(Dψ(η))⊤ > 0 for η ∈ Ω′ , and we have 1 − αDψ(η) · η > 0 on Ω′ so that
1
Dψ(η) is well-defined. We have
the α-gradient D(α) ψ(η) = 1−αDψ(η)·η


(2.17)

−1
= D(α) ψ.
D(α) ϕ

Proof. Taking logarithm in (2.10) and rearranging, we have
(2.18)
1
ϕ(ξ) = min
ϕ(ξ ′ ) + log (1 − αDϕ(ξ ′ ) · ξ ′ + αDϕ(ξ ′ ) · ξ)
ξ ′ ∈Ω
α


1
Dϕ(ξ ′ )
1
·
ξ
+ ϕ(ξ ′ ) + log (1 − αDϕ(ξ ′ ) · ξ ′ )
log
1
+
α
= min
′
′
′
ξ ∈Ω α
1 − αDϕ(ξ ) · ξ
α
1
= min
c(α) (ξ, η ′ ) + ϕ(ξ ′ ) + log (1 − αDϕ(ξ ′ ) · ξ ′ ) .
ξ ′ ∈Ω
α
Note that since η ′ = D(α) ϕ(ξ ′ ), from (2.11) we have the useful identity
1 + αξ ′ · η ′ =

(2.19)

1
.
1 − αDϕ(ξ ′ ) · ξ ′

With this observation, we see that
ϕ(ξ ′ ) +



1
log (1 − αϕ(ξ ′ ) · ξ ′ ) = − c(α) (ξ ′ , η ′ ) − ϕ(ξ ′ ) = −ψ(η ′ ).
α

Putting this into (2.18), we obtain (2.15). Note that this derivation explains our
definitions of c(α) and the α-gradient.
From (2.15), for any η, ξ ′ we have
ϕ(ξ ′ ) ≤ c(α) (ξ ′ , η) − ψ(η) ⇒ ψ(η) ≤ c(α) (ξ ′ , η) − ϕ(ξ ′ ).
By definition we have ψ(η) = c(α) (ξ, η)− ϕ(ξ), so (2.16) holds as well. In particular,
for any ξ, ξ ′ ∈ Ω we have
ψ(η) = c(α) (ξ, η) − ϕ(ξ) ≤ c(α) (ξ ′ , η) − ϕ(ξ ′ ).

(2.20)

Using (2.19), we compute
c

(α)

′

(ξ , η) − c

(α)



1
1 + αξ ′ · η
(ξ, η) = log
α
1 + αξ · η


η
1
1
′
+ αξ ·
= log
α
1 + αξ · η
1 + αξ · η
1
′
= log (1 + αDϕ(ξ) · (ξ − ξ)) .
α

It follows that
(2.21)

c(α) (ξ ′ , η) − c(α) (ξ, η) − (ϕ(ξ ′ ) − ϕ(ξ)) = D(α) [ξ ′ : ξ]

is the L(α) -divergence of ϕ.
By (2.15), for any ξ ∈ Ω, η is the unique minimizer (over Ω′ ) of
η ′ 7→

1
log (1 + αξ · η ′ ) − ψ(η ′ ).
α

10

TING-KAM LEONARD WONG

ξ
= Dψ(η). Rearranging,
Since Ω′ is open, we have the first order condition 1+αξ·η
we have
1
1 − αDψ(η) · η =
.
1 + αξ · η
Thus D(α) ψ(η) is well-defined, D(α) ψ(η) = ξ, and we have (2.17).
It remains to show that

− D2 ψ(η) − α(Dψ(η))(Dψ(η))⊤

(2.22)

is strictly positive definite on Ω′ . Expressing (2.21) in terms of ψ using the definition
(2.14) of α-conjugate, we can show that
1
D(α) [ξ : ξ ′ ] = log (1 + αDψ(η) · (η ′ − η)) − (ψ(η ′ ) − ψ(η)) ,
α
so the L(α) divergence of ϕ is the L(α) -divergence of ψ with the arguments interchanged (see Corollary 2.22 for a more general result). It follows that (2.22) is the
Riemannian matrix of D(α) in the dual coordinates η. Since ϕ satisfies Condition
2.6 by assumption, the matrix (2.22) is strictly positive definite as well.

From (2.14) and (2.21), we immediately obtain
Theorem 2.13 (self-dual representation). The L(α) -divergence of ϕ admits the
self-dual representation
D(α) [ξ : ξ ′ ] = c(α) (ξ, η ′ ) − ϕ(ξ) − ψ(η ′ ),

(2.23)

ξ, ξ ′ ∈ Ω.

The self-dual representation (2.25) generalizes [3, Theorem 1.1] which addresses
the case of Bregman divergence (α = 0+ ). This completes the circle of ideas
and the duality theory for α-exponentially concave functions. A general self-dual
representation for c-divergence is proved in Theorem 2.21.
2.3.2. Duality for α-exponentially convex functions. Now we state without proof
the analogous results for the L(−α) -divergence defined by an α-exponentially convex function. The only difference is that because eαϕ is convex, we have to take
maximums instead of minimums and switch the inequalities accordingly. Also, as
the L(−α) -divergence may not be globally defined, here we only consider the local
geometry.
Condition 2.14 (conditions for L(−α) -divergence). We assume ϕ is smooth and
the matrix D2 ϕ + α(Dϕ)(Dϕ)⊤ is strictly positive definite on Ω. By a translation
if necessary, we assume that 0 ∈ Ω. By restricting the domain if necessary, we
assume that
1 + αDϕ(ξ ′ ) · (ξ − ξ ′ ) > 0

(2.24)

for all ξ, ξ ′ ∈ Ω. This implies that the L(−α) -divergence is defined for all ξ, ξ ′ ∈ Ω.
Theorem 2.15 (duality for L(−α) -divergence). Consider the α-gradient and c(α)
defined as in (2.11) and (2.13). Then the map ξ 7→ η = D(α) ϕ(ξ) is a diffeomorphism, and the (−α)-conjugate ψ defined by (2.14) is locally α-exponentially
convex. The functions ϕ and ψ are related by




(α) ′
′
(α)
′
′
c
(ξ
,
η)
−
ϕ(ξ
)
.
c
(ξ,
η
)
−
ψ(η
)
,
ψ(η)
=
max
ϕ(ξ) = max
′
′
′
η ∈Ω

The L
(2.25)

(−α)

ξ ∈Ω

-divergence (2.7) admits the self-dual representation
D(−α) [ξ : ξ ′ ] = ϕ(ξ) + ψ(η ′ ) − c(α) (ξ, η ′ ),

ξ, ξ ′ ∈ Ω.

LOGARITHMIC DIVERGENCES AND RÉNYI GEOMETRY

11

2.4. c-divergence induced by an optimal transport map. The ideas in Section 2.3 can be developed in a rather general framework. In this subsection we
review some basic results of optimal transport and introduce a c-divergence defined by an optimal transport map. This c-divergence was first introduced in [28]
and here we give a more systematic exposition. For introductions to the theory of
optimal transport we refer the reader to standard references such as [6, 31, 33, 34].
2.4.1. Preliminaries in optimal transport. Let X and Y be Polish spaces (i.e., complete and separable metric spaces) interpreted respectively as the source and target
spaces of the optimal transport problem. Let c : X × Y → R be a continuous
cost function. Given Borel probability measures µ ∈ P(X ) and ν ∈ P(Y), the
Monge-Kantorovich optimal transport problem is
Z
c(x, y)dγ(x, y),
(2.26)
inf
γ

X ×Y

where the infimum is taken over joint distributions γ ∈ P(X × Y) whose marginals
are µ and ν respectively. We call γ a coupling of the pair (µ, ν) and write γ ∈
Π(µ, ν). If γ attains the infimum in (2.26), we say that it is an optimal coupling.
For certain cost functions and under suitable conditions on µ and ν, the optimal
coupling has the form
γ = (Id × T )# µ
for some measurable map T : X → Y. Here we use F# µ = µ ◦ F −1 to denote the
pushfoward of a measure. In this case the optimal coupling is deterministic, and
we call T an optimal transport map.
Remark 2.16. Note that if we replace the cost function c by
(2.27)

c̃(x, y) = c(x, y) + h(x) + k(y),

where h and k and real-valued functions, then for any coupling γ ∈ Π(µ, ν) we have
Z
Z
Z
Z
c̃dγ = cdγ + hdµ + kdν.

The last two integrals, if they exist, are determined once µ and ν are fixed and
is independent of the coupling γ. This means that we are free to modify the cost
function in the manner of (2.27) without changing the optimal couplings.
Before discussing some general results about the Monge-Kantorovich optimal
transport problem (2.26), it is helpful to review an important and motivating special
case.
Example 2.17 (quadratic cost). Let X = Y = Rd and let
d

(2.28)

c(x, y) =

1X i
1
(x − y i )2
kx − yk2 =
2
2 i=1

be the quadratic cost (here we write x = (x1 , . . . , xd ) ∈ Rd ). Suppose µ, ν ∈ P(Rd )
have finite second moments, i.e.,
Z
Z
kxk2 dµ(x) < ∞,
kyk2 dν(y) < ∞,

and suppose µ is absolutely continuous with respect to the Lebesgue measure on Rd .
By Brenier’s theorem [11], there exists a convex function ϕ : Rd → R ∪ {+∞} such

12

TING-KAM LEONARD WONG

that its gradient ∇ϕ (which is well-defined µ-a.e.) pushforwards µ to ν. Moreover,
the deterministic coupling
γ = (Id × ∇ϕ)# µ
solves the Monge-Kantorovich problem. In fact, it can be shown that the optimal
transport map T (x) = ∇ϕ(x), which is a Legendre transformation, is unique µ-a.e.
Note that since
1
1
1
kx − yk2 = −x · y + kxk2 + kyk2 ,
2
2
2
by Remark 2.16 we may consider instead the cost function c̃(x, y) = −x · y, or even
c̃(x, y) = x · y by a change of variable y 7→ −y. This gives the limiting case for the
parameterized cost function c(α) (x, y) given in (2.13).
In general, solutions to the optimal transport problem (2.26) can be characterized
using generalized concepts of convex analysis. In the following definition we fix a
cost function c : X × Y → R.
Definition 2.18. Let f : X → R ∪ {−∞} and g : Y → R ∪ {−∞}.
(i) The c-transforms of f and g are defined respectively by
(2.29)

f c (y) = inf (c(x, y) − f (x)) ,

y ∈ Y,

g c (x) = inf (c(x, y) − g(x)) ,

x ∈ X.

x∈X
y∈Y

(ii) We say that f (respectively g) is c-concave if f cc = f (respectively g cc = g).
(iii) If f and g are c-concave, their c-superdifferentials are defined by
(2.30)

∂f = {(x, y) ∈ X × Y : f (x) + f c (y) = c(x, y)},
∂g = {(x, y) ∈ X × Y : g c (x) + g(y) = c(x, y)}.

If f is c-concave on X , then
(2.31)

f (x) + f c (y) ≤ c(x, y)

for all (x, y) ∈ X × Y, and equality holds if and only if (x, y) ∈ ∂f . We call (2.31)
the (generalized) Fenchel inequality (or identity when equality holds).
Theorem 2.19. Suppose the cost function c is continuous and bounded below, and
there exist a ∈ L1 (µ) and b ∈ L1 (ν) such that c ≤ a + b. Then, a coupling γ of
the pair (µ, ν) solves the Monge-Kantorovich problem (2.26) if and only if there
exists a c-concave function f on X such that supp(γ) ⊂ ∂ c f . We call the c-concave
function f a Kantorovich potential.
Proof. See [6, Theorem 1.13] where this result is called the Fundamental Theorem
of Optimal Transport.

By Theorem 2.19, a c-concave function f on X (or, rather, its c-superdifferential)
can be regarded as encoding the solution to an optimal transport problem. If the
cost function c is not bounded below but satisfies suitable convexity properties, we
may modify c in the manner of (2.27) such that c(x, y) + h(x) + k(y) is bounded
below and Theorem 2.19 is still applicable.

LOGARITHMIC DIVERGENCES AND RÉNYI GEOMETRY

13

2.4.2. c-divergence. Let f be a c-concave function on X . Assume that f is cdifferentiable in the following sense: for all x ∈ X there is a unique element y ∈ Y
such that (x, y) ∈ ∂ c f . We call y = Dc f (x) the c-gradient of f . This is a strong
assumption, but is satisfied for the cost function c(α) as seen in the proof of Theorem
2.12. In this case, we may regard Dc f : X → Y as an optimal transport map. For
a detailed discussion of the smoothness properties of optimal transport maps see
[34, Chapter 11].
It is easy to show using the definition of c-concavity that for any x, x′ ∈ X we
have
(2.32)

f (x′ ) + c(x, y ′ ) − c(x′ , y ′ ) ≥ f (x),

y ′ = Dc f (x′ ).

Analogous definitions and results hold for a c-concave function g on Y.
The inequality (2.32) motivates the definition of c-divergence (a more suggestive
name may be transport divergence) which was first introduced in [28]. From (2.21)
we see that the L(±α) -divergences are special cases.
Definition 2.20 (c-divergence). Let f be c-concave and c-differentiable on X . The
c-divergence of f is the functional Df : X × X → [0, ∞) defined by
(2.33) Df [x : x′ ] = c(x, y ′ )−c(x′ , y ′ )−(f (x)−f (x′ )),

x, x′ ∈ X ,

y ′ = Dc f (x′ ).

If g is a c-differentiable c-concave function on Y, we define the dual c-divergence
Dg : Y × Y → [0, ∞) by
(2.34) D∗g [y : y ′ ] = c(x′ , y) − c(x′ , y ′ ) − (g(y) − f (y ′)),

y, y ′ ∈ Y,

x′ = Dc g(y ′ ).

Intuitively, the c-divergence Df [x : x′ ] quantifies the deviation of (x, y ′ ) from
the optimal coupling (x′ , y ′ = Dc f (x′ )) ∈ ∂ c f .
Theorem 2.21 (self-dual representations). Let y ′ = Dc f (x′ ). Then
(2.35)

Df [x : x′ ] = c(x, y ′ ) − f (x) − f c (y ′ ).

Similarly, if x = Dc g(y). Then
(2.36)

D∗g [y : y ′ ] = c(x′ , y) − g(y) − g c (x′ ).

Proof. By the generalized Fenchel identity (2.31), we have
f (x′ ) + f c (y ′ ) = c(x′ , y ′ ) ⇒ f c (y ′ ) = c(x′ , y ′ ) − f (x′ ).
It follows that
Df [x : x′ ] = c(x, y ′ ) − c(x′ , y ′ ) − (f (x) − f (x′ ))
= c(x, y ′ ) − f (x) − f c (y ′ ).
This gives (2.36). The proof of the second identity (2.36) is similar.



Corollary 2.22 (Duality of c-divergence). Suppose f c is c-differentiable on the
range of Dc f . Let y = Dc f (x) and y ′ = Dc f (x′ ). Then
(2.37)

Df [x′ : x] = D∗f c [y : y ′ ] .

Thus, if we identify x with y and x′ with y ′ , then the dual divergence of f c is simply
the divergence of f with the arguments interchanged.
We conclude this section by specializing to the case of L(α) -divergence. Because
of the logarithm, to state the result in the framework of c-divergence we need to
pick a specific domain so that c(α) (x, y) is well-defined. A natural domain for this
purpose is the positive quadrant Rd++ .

14

TING-KAM LEONARD WONG

Corollary 2.23. Consider the space X = Y = Rd++ = (0, ∞)d . (To make it
Pd
Polish, we may use for example the metric d(x, y) = i=1 (log xi − log y i )2 .) On
Rd++ consider the continuous cost function c(α) (x, y) = α1 log(1 + αx · y). If ϕ is
α-exponentially concave on Rd++ satisfying Condition 2.6, then it is c(α) -concave.
The c(α) -gradient of ϕ is the α-gradient, and the c(α) -divergence of ϕ is the L(α) divergence D(α) [· : ·].
Proof. Once the cost function c(α) is identified, the development follows the same
lines as in [28, Section 3] and has been covered using a slightly different language
in Section 2.2.

In the limit α → 0+ , we see that the Bregman divergence D(0+) corresponds to
the c-divergence of c(x, y) = x · y, and D(0−) corresponds to c(x, y) = −x · y or
equivalently the quadratic cost 21 kx − yk2 .
3. Generalizations of exponential family and Rényi divergence
It is well known that Bregman divergences (D(0±) ) arise naturally in applications
involving exponential families. In fact, it was shown by Banerjee et al in [10]
that there is a one-to-one correspondence between what they call regular Bregman
divergences and regular exponential families. Are there parameterized families of
probability distributions that correspond to the L(±α) divergences?
In this section we consider generalizations of exponential family that are closely
related to the α-family [2, Section 2.6] and the q-exponential family [3, Section
4.3]. We show that a suitably defined potential function is α-exponentially concave/convex, and the corresponding L(±α) -divergences give the Rényi divergence.
This gives a natural framework to study the geometry induced by the Rényi divergence.
3.1. Motivations. First let us recall the main idea in [10]. Consider an exponential
family of probability densities of the form
(3.1)

log p(x, ξ) = ξ · h(x) − ϕ(ξ),

where h(x) = (h1 (x), . . . , hd (x)) is a vector of sufficient statistics, and ϕ is the
convex cumulant generating function. Let D = D(0−) be the Bregman divergence
of ϕ. By the self-dual representation (induced by the function c(ξ, h(x)) = −ξ ·h(x);
see Section 2.4), we have
(3.2)

log p(x, ξ) = −D [ξ : ξ ′ ] + ψ(h(x)),

where ψ is the conjugate function and h(x) plays the role of the dual variable η ′ .
Remarkably, the conjugate ψ is the negative Shannon entropy (see [3, Section 2.1]).
Now let us consider an L(α) -divergence D(α) with α > 0 and domain Ω = Rd++ ,
where ϕ is α-exponentially concave. Its self-dual representation (Theorem 2.13) is
D(α) [ξ : ξ ′ ] =

1
log (1 + αξ · η ′ ) − ϕ(ξ) − ψ(η ′ ).
α

Imitating (3.2), let us consider a parameterized density of the form
(3.3)

log p(x, ξ) = −D(α) [ξ : ξ ′ ] − ψ(h(x)) = −

1
log(1 + αξ · h(x)) + ϕ(ξ),
α

LOGARITHMIC DIVERGENCES AND RÉNYI GEOMETRY

15

where h(x) = η ′ ∈ Rd++ as well. Exponentiating both sides, we have
(3.4)

p(x, ξ) = (1 + αξ · h(x))

1
−α

eϕ(ξ) .

If µ is the dominating measure of the family, the normalizing function ϕ(ξ) in (3.4)
is given by
Z
1
(3.5)
ϕ(ξ) = − log (1 + αξ · h(x))− α dµ(x).

It is the analogue of the cumulant generating function. For lack of a better name,
let us call (3.4) an F (α) -family.
Similarly, the L(−α) -divergence motivates the F (−α) -family
(3.6)

1

p(x, ξ) = (1 + αξ · h(x)) α e−ϕ(ξ) ,

where
(3.7)

ϕ(ξ) = log

Z

1

(1 + αξ · h(x)) α dµ(x).

Remark 3.1. For α′ 6= 1, Amari and Nagaoka [2, Section 2.6] defined an α′ -family
of the form
X
1−α′
2
ξ i F i (x), L(α) (u) =
L(α) (p(x, ξ)) = C(x) +
u 2 ,
′
1−α
i
!# 2 ′
"
(3.8)
1−α
X
1 − α′
ξ i F i (x)
C(x) +
or p(x, ξ) =
,
2
i

where C and F i are functions on X . Comparing this with (3.5) and (3.7), we see
that our F (±α) -families are essentially the their α′ -family with a different parameterization and a difference choice of the potential function. We decide to keep our
formulation so as to be consistent with (3.3) and the framework of the paper.
3.2. Exponential concavity/convexity of ϕ. In order that the densities (3.4)
and (3.6) are well-defined, we need to check that the potential functions (3.5) and
(3.7) define α-exponentially concave/convex functions.
We impose the following conditions which formalize the above constructions.
Condition 3.2. Let µ be a probability measure on a measurable space X . Let
h(x) = (h1 (x), . . . , hd (x)) be a vector function defined on X and with values in
Rd++ , such that if ξ · h(x) = ξ ′ · h(x) µ-almost everywhere on X then ξ = ξ ′ .
For the F (α) -case, we assume that the integral in (3.5) exists for all ξ in an open
convex set Ω ⊂ Rd++ and we can differentiate with respect to ξ under the integral
sign. Analogous conditions are imposed for the F (−α) -case.
3.2.1. F (α) -family.
Proposition 3.3. For α > 0 fixed, the potential function ϕ(ξ) of the F (α) -family
defined by (3.5) is α-exponentially concave on Ω.
Proof. We need to check that Φ = eαϕ is concave on Ω. From (3.5), we have
−α
Z
1
,
(3.9)
Φ(ξ) =
(1 + αξ · h(x))− α dµ(x)
which can be regarded as a generalized harmonic average.

16

TING-KAM LEONARD WONG

Let X be the random element X(x) = x, where x ∈ X and X is the sample
space. Given ξ ∈ Ω, let Eξ denote the expectation with respect to p(x, ξ), and let
Zξ be the random vector defined by
(3.10)

Zξ =

h(X)
.
1 + αξ · h(X)

Differentiating (3.9) and simplifying, we have
Z
−α−1 Z

1
1
∂Φ
−α
−1 i
−α
(ξ) = α
(1 + αξ · h(x)) dµ
h (x)dµ
(1 + αξ · h(x))
∂ξ i
−α Z
Z
1
p(x, ξ)zξi dµ(x)
=α
(1 + αξ · h(x))− α dµ(x)
= αΦ(ξ)Eξ Zξi .

In particular, we have
(3.11)

Dϕ(ξ) = Eξ [Zξ ]

which can be interpreted as an expectation parameter. Differentiating one more
time and calculating carefully, we get


∂2Φ
(ξ) = α(1 + α) Eξ [Zξi ]Eξ [Zξj ] − Eξ [Zξi Zξj ]
i
j
∂ξ ∂ξ
= −α(1 + α)Cov(Zξi , Zξj ).

Since D2 Φ is a negative multiple of the covariance matrix of Zξ , Φ is concave and
ϕ is α-exponentially concave.

3.2.2. F (−α) -family. Next we consider the F (−α) -family. Interestingly, the result
is a little different and depends on the value of α.
Proposition 3.4. Consider the potential function ϕ(ξ) of the F (−α) -family defined
by (3.7).
(i) If 0 < α < 1, then ϕ is α-exponentially convex.
(ii) If α > 1, then ϕ is α-exponentially concave.
Proof. Now we heave
Φ(ξ) = eαϕ(ξ) =

Z

1

(1 + αξ · h) α dµ

α

.

A computation similar to that in the proof of Proposition 3.3 shows that


h(X)
Dϕ(ξ) = Eξ Zξ = Eξ
1 + αξ · h(X)
and
∂2Φ
(ξ) = α(1 − α)Covξ (Zξi , Zξj ).
∂ξ i ∂ξ j
Now Φ is convex if 0 < α < 1 and concave if α > 1.


LOGARITHMIC DIVERGENCES AND RÉNYI GEOMETRY

17

3.3. Rényi entropy and divergence. Based on information-theoretic reasonings,
Rényi entropy and divergence were first introduced by Rényi in [29] and they have
numerous applications in information theory and statistical physics among other
areas. See [32] for a survey of their remarkable properties. We show that these
quantities are naturally related to our F (±α) -families.
First let us recall the definitions of Rényi entropy and divergence. Same as above
we fix a measure space (X , µ).
Definition 3.5. Let α̃ ∈ (0, 1) ∪ (1, ∞). Let P , Q be probability measures on X
dQ
that are absolutely continuous with respect to µ. We let p = dP
dµ and q = dµ be
their densities.
(i) The Rényi entropy of P of order α̃ (and with reference measure µ) is
defined by
Z
1
(3.12)
Hα̃ (P ) =
log pα̃ dµ.
1 − α̃

We also denote the above by Hα̃ (p).
(ii) The Rényi divergence or order α̃ between P and Q is defined by
Z
1
log pα̃ q 1−α̃ dµ.
(3.13)
Dα̃ (P ||Q) =
α̃ − 1

Note that Hα̃ (P ) = −Dα̃ (P ||µ). We use a different notation to distinguish the
Rényi divergence from the L(±α) -divergences D(±α) [ξ : ξ ′ ]. It is well known that
as α̃ → 1, the Rényi entropy and divergence converge respectively to the Shannon
entropy and the Kullback-Leibler divergence.
The following is the main result of this section.
Theorem 3.6.
(i) For an F (α) -family with α > 0, the L(α) -divergence of ϕ is the Rényi
entropy of order α̃ := 1 + α:
(3.14)

D(α) [ξ : ξ ′ ] = Dα̃ (p(·, ξ ′ )||p(·, ξ)) .
Moreover, the α-conjugate ψ of ϕ is the Rényi entropy of order α̃:

(3.15)

ψ(η) = Hα̃ (p(·, ξ)),

η = D(α) ϕ(ξ).

(ii) Consider an F (−α) -family. If 0 < α < 1, the L(−α) -divergence of ϕ is the
Rényi entropy of order α̃ := 1 − α:
(3.16)

D(−α) [ξ : ξ ′ ] = Dα̃ (p(·, ξ ′ )||p(·, ξ)) ,
and the (−α)-conjugate is ψ(η) = −Hα̃ (p(·, ξ)) (note the minus sign).
If α > 1, the L(α) -divergence of ϕ is given by

(3.17)

α−1
Dα (p(·, ξ)||p(·, ξ ′ )) ,
α
Note that here α̃ = α and the order of the arguments is switched. The
conjugate is
Z
−1
log p(x, ξ)1−α dµ.
ψ(η) =
α
Note that it is not a Rényi entropy since 1 − α < 0.
D(α) [ξ : ξ ′ ] =

18

TING-KAM LEONARD WONG

Proof. (i) We want to compute
1
D(α) [ξ : ξ ′ ] = log (1 + αDϕ(ξ ′ ) · (ξ − ξ ′ )) − (ϕ(ξ) − ϕ(ξ ′ )).
α
First, using (3.10) and (3.11), we have

(3.18)

1 + αDϕ(ξ ′ ) · (ξ − ξ ′ ) = 1 + αEξ′ Zξ′ · (ξ − ξ ′ )


h(X)
′
= 1 + αEξ′
· (ξ − ξ )
1 + αξ ′ · h(X)
1 + αξ · h(X)
= Eξ′
.
1 + αξ ′ · h(X)

From (3.4), we have
′

1 + αξ ′ · h(X) =

eαϕ(ξ )
.
p(X, ξ ′ )α

It follows from (3.18) that
1
D(α) [ξ : ξ ′ ] = log (1 + αDϕ(ξ ′ ) · (ξ − ξ ′ )) − (ϕ(ξ) − ϕ(ξ ′ ))
α
eαϕ(ξ) /p(X, ξ)α
1
− (ϕ(ξ) − ϕ(ξ ′ ))
= log Eξ′ αϕ(ξ′ )
α
e
/p(X, ξ ′ )α
1
p(X, ξ ′ )α
= log Eξ′
.
α
p(X, ξ)α
Expressing the divergence in terms of the integral, we have
Z
1
D(α) [ξ : ξ ′ ] = log p(x, ξ ′ )1+α p(x, ξ)−α dµ
α
Z
1
log p(x, ξ ′ )α̃ p(x, ξ)1−α̃ dµ,
=
α̃ − 1

where α̃ = 1 + α > 1. This shows that D(α) is the Rényi divergence of order α̃.
Next we consider the α-conjugate
1
ψ(η) = log (1 + αξ · η) − ϕ(ξ).
α
Using the identity (2.19) and following the argument in (3.18), we have
−1

1 + αξ · η = (1 − αDϕ(ξ) · ξ)

−1
= Eξ (1 + αξ · h(X))−1
i−1
 h
.
= Eξ p(X, ξ)α e−αϕ(ξ)

From this we get
Z
Z
1
1
1+α −αϕ(ξ)
ψ(η) = − log p(x, ξ)
e
dµ − ϕ(ξ) =
p(x, ξ)α̃ dµ = Hα̃ (p(·, ξ)).
α
1 − α̃
This proves (i), and the proof of (ii) is similar.

Remark 3.7. It appears from the above results that the F (−α) -family with α > 1
is different from the other cases. Nevertheless, the F (α) -family for α > 0 and the
F (−α) -family for 0 < α < 1 together give all Rényi entropy/divergence of order
α̃ ∈ (0, 1) ∪ (1, ∞) (see Figure 1).

LOGARITHMIC DIVERGENCES AND RÉNYI GEOMETRY

F (−α)
α̃ = 1 − α

19

F (α)
α̃ = 1 + α
α̃

0

1

Figure 1. The order α̃ of the Rényi entropy/divergence for an
F (±α) family.
Example 3.8 (finite simplex). Let X = {0, 1, . . . , d} be a finite set and let µ be the
counting measure on X . Fix α > 0. Let hi (x) = δ i (x) be the indicator function
for point i, i = 1, . . . , d. Then (3.4) gives, for α > 0, the F (α) -family of probability
mass functions on X given by
(3.19)

p(i, ξ) = (1 + αξ i )−1/α eϕ(ξ) ,

i = 1, . . . , d,

p(0, ξ) = eϕ(ξ) ,

where the α-exponentially concave function ϕ is


d
X
(1 + αξ j )−1/α  ,
ϕ(ξ) = log p(0, ξ) = − log 1 +
j=1

and the domain is Ω = {ξ ∈ Rd : 1 + αξ j > 0}. The coordinate ξ is aPglobal
coordinate system of the open simplex Sd = {p = (p0 , . . . , pd ) ∈ (0, 1)1+d : i pi =
1}. From (3.19), we have

α

1
p(0, ξ)
i
ξ =
− 1 , i = 1, . . . , d.
α
p(i, ξ)
By Theorem 3.6, the L(α) -divergence of ϕ is the discrete Rényi divergence of order
α̃ = 1 + α > 0.
Analogously, we may also express the simplex Sd as an F (−α) -family where
1

p(i, ξ) = (1 + αξ i ) α e−ϕ(ξ) ,

i = 1, . . . , d, p(0, ξ) = e−ϕ(ξ) ,


d
X
(1 + αξ j )1/α  ,
ϕ(ξ) = − log p(0, ξ) = log 1 +
j=1

and

ξi =

1
α



p(i, ξ)
p(0, ξ)

α


−1 ,

i = 1, . . . , d.

When 0 < α < 1, the L(−α) -divergence of ϕ is the discrete Rényi divergence of
order α̃ = 1 − α > 0.
The corresponding dualistic structures will be considered in Section 5.4.
4. Dualistic structure of L(±α) -divergence
In this and the next sections we study the dualistic structure induced by a given
L(±α) -divergence. First we state the general definition of dualistic structure. For
preliminaries in differential geometry see [3, 9].
Definition 4.1. Let M be a smooth manifold. A dualistic structure on M is a
quadruplet (M, g, ∇, ∇∗ ) where g (denoted by h·, ·i) is a Riemannian metric, and

20

TING-KAM LEONARD WONG

(∇, ∇∗ ) is a pair of torsion-free affine connections that are dual with respect to g:
for any vector fields X, Y and Z, the covariant derivatives satisfy the identity
(4.1)

ZhX, Y i = h∇Z X, Y i + hX, ∇∗Z Y i.

We also call M equipped with a dualistic structure a statistical manifold.
For a dualistic structure, it is easy to see that the average ∇ = 12 (∇ + ∇∗ ) is the
Levi-Civita connection of the metric g. By the general results of Eguchi [15, 16],
any divergence (see [3, Definition 1.1]) on a manifold M defines a dualistic structure
on M . It is this structure induced by D(±α) that we study in this section.
We will derive explicit coordinate representations of this dualistic structure. The
geometric consequences will be studied in Section 5. Let us stress that while any
divergence induces a geometric structure, the self-dual representations of the L(±α) divergences motivated by optimal transport suggest the appropriate coordinate
systems.
By the results of Section 3, for an F (±α) -family this gives the geometry induced by the Rényi divergence. The Rényi divergence is closely related to the
α-divergence.1 Our framework is more general as it does not depend on the probabilistic representation. We discuss the connection with the α-divergences in Section
5.4.
4.1. Notations. We focus on the case of L(α) -divergence for an α-exponentially
concave function ϕ. The arguments for the L(−α) -divergence are essentially the
same (with some changes of signs) and will be left for the reader. The main results
for the L(−α) -divergence will be stated in Section 4.3.
We will use the self-dual representation derived in Theorem 2.13:
1
(4.2)
D [ξ : ξ ′ ] = D(α) [ξ : ξ ′ ] = log (1 + αξ · η ′ ) − ϕ(ξ) − ψ(η ′ ).
α
Here η = D(α) ϕ(ξ) is the dual coordinate system given by the α-gradient. To
simplify the notations we also use the notation
(4.3)

Π(ξ, η ′ ) = Π(α) (ξ, η ′ ) = 1 + αξ · η ′

for ξ ∈ Ω, η ′ ∈ Ω′ .
We consider the smooth manifold M = Ω where the primal coordinate ξ (identity
map) and the dual coordinate η = D(α) ϕ(ξ) are global coordinate systems. Note
that by symmetry of the cost function c(α) and the associated α-duality, we only
need to consider the primal coordinate system ξ. Consider the coordinate vector
fields ∂ξ∂ 1 , . . . , ∂ξ∂ d . By definition, the coordinate representations of (g, ∇, ∇∗ ) under
ξ are given by


∂ ∂
∂
∂
= − i ′ j D [ξ : ξ ′ ]
,
,
gij (ξ) =
∂ξ i ∂ξ j
∂ξ ∂ξ
ξ=ξ ′


∂2
∂
∂
∂
Γijk (ξ) = ∇ ∂ i j , k = − i j ′ k D [ξ : ξ ′ ]
,
(4.4)
∂ξ ∂ξ
∂ξ
∂ξ ∂ξ ∂ξ
ξ=ξ ′


∂
∂2
∂
∂
∗
∗
Γijk (ξ) = ∇ ∂
=
−
,
D [ξ : ξ ′ ]
.
′
j ∂ξ k
∂ξ i ∂ξ ′ j ∂ξ k
∂ξ i ∂ξ
ξ=ξ ′
1The author thanks Shun-ichi Amari for pointing this out.

LOGARITHMIC DIVERGENCES AND RÉNYI GEOMETRY

21

We also define
(4.5)

Γij k (ξ) = Γijm (ξ)g mk (ξ),

Γ∗ij k (ξ) = Γ∗ijm (ξ)g mk (ξ),

where g ij (ξ) is the inverse of (gij (ξ)) and the Einstein summation convention (see
for example [3, p.21]) is used. This implies that
∂
∂
∇ ∂ i j = Γij k (ξ) k
∂ξ ∂ξ
∂ξ
and similarly for the dual
∇∗ .  

 connection
∂η
∂ξ
∂ξ i
∂η i
be the Jacobian matrices of the
We also let ∂η = ∂ηj and ∂ξ = ∂ξ
j
transition maps. They are inverses of each other.
4.2. Dualistic structure of L(α) -divergence.
4.2.1. The Riemannian metric.
Proposition 4.2. The Riemannian matrix G(ξ) = (gij (ξ)) is given by


−1
α
∂η
(4.6)
G(ξ) = (α)
.
I − (α)
ηξ ⊤
∂ξ
Π (ξ, η)
Π (ξ, η)
Its inverse is given by


∂ξ
I + αηξ ⊤ .
(4.7)
G−1 (ξ) = g ij (ξ) = −Π(α) (ξ, η)
∂η
Proof. Using the self-dual representation (4.2), we have
η ′i
∂ϕ
∂
′
D
[ξ
:
ξ
]
=
− i (ξ),
i
′
∂ξ
Π(ξ, η ) ∂ξ
(4.8)

1
∂2
D [ξ : ξ ′ ] =
i
′j
∂ξ ∂ξ
Π(ξ, η ′ )2

d

X ∂η ′ℓ
∂η ′i
Π(ξ, η ) ′j − αη ′i
ξ ℓ ′j
∂ξ
∂ξ
′

ℓ=1

!

.

Setting ξ = ξ ′ and expressing in matrix form, we obtain (4.6).
We obtain (4.7) by taking the inverse of (4.6). For the middle term, we may use
the Sherman-Morrison formula (see (4.10) below) to get

−1
α
I−
ηξ ⊤
= I + αηξ ⊤ .
Π(ξ, η)

Remark 4.3. In the above proposition the Riemannian matrix is given in terms
of the Jacobian matrix ∂η
∂ξ . This is to emphasize the role of duality (compare
(4.6) with [3, (1.66)]) and to enable explicit expressions of the Christoffel symbols
Γij k = Γijm g mk which appear in the primal geodesic equations.
It is not difficult to show directly using the formula (2.5) of the L(α) -divergence
that
(4.9)

G(ξ) = −D2 ϕ(ξ) − α(Dϕ(ξ))(Dϕ(ξ))⊤ ,

which is exactly the positive definite matrix in (2.2). Using the general ShermanMorrison formula
−1
A−1 uv ⊤ A−1
(4.10)
A + uv ⊤
= A−1 −
,
1 + v ⊤ A−1 u

22

TING-KAM LEONARD WONG

we can invert (4.9) to get
(D2 ϕ)−1 (Dϕ)(Dϕ)⊤ (D2 ϕ)−1
.
1 + α(Dϕ)⊤ (D2 ϕ)−1 (Dϕ)
While this formula is explicit, it is difficult to use in differential-geometric computations.
G−1 (ξ) = −(D2 ϕ)−1 + α

4.2.2. Affine connections.
Proposition 4.4. The Christoffel symbols of the primal connection are given by
(4.11)
d
i
j
X
α
∂η ℓ
α
2α2
j ∂η
i ∂η
i j
Γijk (ξ) = (α)
η
η
η
η
ξℓ k ,
+
−
k
k
2
(α)
2
(α)
3
∂ξ
∂ξ
∂ξ
Π (ξ, η)
Π (ξ, η)
Π (ξ, η)
ℓ=1



∂ϕ
∂ϕ k
−α
(4.12)
δ + j δik .
η i δjk + η j δik = −α
Γij k (ξ) = (α)
∂ξ i j
∂ξ
Π (ξ, η)
where δ·· is Dirac’s delta function.
Proof. From our previous computation (4.8), we have
d

X ∂η ′ℓ
∂ ∂
∂η ′i
1
α
D [ξ : ξ ′ ] =
−
η ′i
ξ ℓ ′k .
i
′k
′
′k
′
2
∂ξ ∂ξ
Π(ξ, η ) ∂ξ
Π(ξ, η )
∂ξ
ℓ=1

′

Differentiating one more time and writing Π = Π(ξ, η ), we get

d

X ∂η ′ℓ
∂
−α
∂η ′i
α
∂η ′j
2α
∂2
D [ξ : ξ ′ ] = 2 η ′j ′k − 2 η ′i ′k + 3 η ′i η ′j
ξ ℓ ′k .
i
j
′k
∂ξ ∂ξ ∂ξ
Π
∂η
Π
∂ξ
Π
∂ξ
ℓ=1

′

Setting ξ = ξ , we obtain (4.11).
To prove (4.12), we first use (4.7) to write

d

g

mk

(ξ) = −Π(ξ, η)

X ∂ξ m
∂ξ m
+α
ηℓ ξ k
k
∂η
∂η ℓ
ℓ=1

It follows that
Γij k (ξ) = Γijm (ξ)g mk (ξ)
=−

α i ∂η j
2α2 i j X ℓ ∂η ℓ
α j ∂η i
+
−
ηη
η
η
ξ
Π ∂ξ m
Π ∂ξ m
Π2
∂ξ m
ℓ

2

=

2

!

!

.

d

X ∂ξ m
∂ξ m
+
α
ηℓ ξ k
∂η k
∂η ℓ
ℓ=1

3

−α j k α i k 2α i j k 2α i j k 2α i j k
η δi − η δj + 2 η η ξ −
ηη ξ + 2ηη ξ
Π
Π
Π
Π
Π

d
X

!

ξ ℓ ηℓ .

ℓ=1

We observe that the last three terms cancel out and the resulting expression gives
the first equality in (4.11). The second equality follows from the definition of η as
the α-gradient (2.11) as well as the identity (2.19).

Remark 4.5. Same as the Riemannian matrix, we may represent Γijk (ξ) using the
function ϕ. For completeness, it is given by

 2
∂ 2 ϕ ∂ϕ
∂ ϕ ∂ϕ
2 ∂ϕ ∂ϕ ∂ϕ
+α
+ j k i .
Γijk (ξ) = 2α
∂ξ i ∂ξ j ∂ξ k
∂ξ i ∂ξ k ∂ξ j
∂ξ ∂ξ ∂ξ
Without using the self-dual representation, it is not easy to see how this formula
leads to (4.12).

LOGARITHMIC DIVERGENCES AND RÉNYI GEOMETRY

23

4.3. Dualistic structure of L(−α) -divergence. Here we state the corresponding
results for the L(−α) -divergence.
Proposition 4.6. For the dualistic structure generated by D(−α) where ϕ is αexponentially convex, we have
(4.13)

(4.14)

gij (ξ) =
k

∂ 2ϕ
∂ϕ
∂ϕ
(ξ) + α i (ξ) j (ξ),
i
j
∂ξ ∂ξ
∂ξ
∂ξ

Γij (ξ) = −α




∂ϕ
∂ϕ
k
k
(ξ)δj + j (ξ)δi .
∂ξ i
∂ξ

Note that (4.12) and (4.14) have the same form (even though here ϕ is αexponentially convex). This is because Γij k = Γijm g mk and the changes of signs
cancel out in the product.
5. Geometric consequences
With all the coefficients available, we derive in this section the geometric properties of the dualistic structure (g, ∇, ∇∗ ). We state the results for any D(±α) with
α > 0, but the proofs will again be given for the case of D(α) . In Section 5.4 we
specialize to the unit simplex and explain the connections with the α-divergence.
5.1. Dual projective flatness. The following definition is taken from [22].
e be torsion-free affine connecDefinition 5.1 (Projective flatness). Let ∇ and ∇
tions on a smooth manifold. They are projectively equivalent if there exists a differential 1-form τ such that
(5.1)

e X Y + τ (X)Y + τ (Y )X
∇X Y = ∇

for any smooth vector fields X and Y . We say that ∇ is projectively flat if ∇ is
projectively equivalent to a flat connection (a connection whose Riemann-Christoffel
curvature tensor vanishes).
If we write τ = ai (ξ)dξ i , say, using the primal coordinate system, then (5.1) is
equivalent to
(5.2)

e ij k (ξ) + ai (ξ)δjk + aj (ξ)δik .
Γij k (ξ) = Γ

Theorem 5.2. For any D(±α) , the primal connection ∇ and the dual connection
∇∗ are projectively flat. Thus we say that the induced dualistic structures are dually
projectively flat.
Proof. We only consider the primal connection. Consider the 1-form defined by
τ (X) = −αXϕ. In primal coordinates, we have
(5.3)

τ =−

d
X
i=1

α

∂ϕ i
dξ .
∂ξ i

e be the flat Euclidean connection with respect to the coordinate system ξ
Let ∇
e ij k (ξ) ≡ 0. From (5.2), we see that ∇ is
such that its Christoffel symbols satisfy Γ
e
projectively equivalent to ∇ and thus ∇ is projectively flat.


24

TING-KAM LEONARD WONG

Projective flatness can be related to the behaviors of the geodesics. Recall that
a (smooth) curve γ : [0, 1] → M is said to be a primal geodesic if ∇γ̇(t) γ̇(t) ≡ 0.
Equivalently, its primal coordinate representation ξ(t) = (ξ 1 (t), . . . , ξ d (t)) satisfies
the primal geodesic equations
(5.4)
ξ̈ k (t) + ξ˙i (t)ξ˙j (t)Γij k (ξ(t)) = 0, k = 1, . . . , d,
where the dots denote derivatives with respect to time. Similarly, the curve γ is a
dual geodesic if ∇∗γ̇(t) γ̇(t) ≡ 0.
The following result is well-known in differential geometry.
e are projectively equivalent, then a ∇-geodesic is a
Proposition 5.3. If ∇ and ∇
e
∇-geodesic up to a time reparameterization, and vice versa.

Corollary 5.4. For the dualistic structure induced by D(±α) , if γ is a primal (dual)
geodesic, then its trace in the primal (dual) coordinate system is a straight line.

Thus the primal geodesics are straight lines under the primal coordinate system
but run at non-constant speeds. Let us consider the time reparameterization. Using
(4.11), we note that the primal geodesic equation (5.4) is equivalent to the single
vector equation
˙ dϕ(ξ(t)) .
(5.5)
ξ̈(t) = 2αξ(t)
dt
From this we immediately see that the trace of the primal geodesic is a straight
line under the coordinate ξ.
Proposition 5.5. Let γ : [0, 1] → M be a primal geodesic. Then, in primal
coordinates, we have
(5.6)

ξ(t) = (1 − h(t))ξ(0) + h(t)ξ(1),

where h : [0, 1] → [0, 1] is given by

Rt

h(t) = R 01

(5.7)

0

e2αϕ(ξ(s)) ds
e2αϕ(ξ(s)) ds

.

Proof. We already know that a primal geodesic can be written in the form (5.6).
Plugging this into the geodesic equation (5.5), we have
d
dϕ(ξ(t))
dϕ(ξ(t))
⇒
log h′ (t) = 2α
.
dt
dt
dt
Integrating and using the fact that h(0) = 0 and h(1) = 1, we obtain the desired
formula (5.7).

h′′ (t) = 2αh′ (t)

A natural question is the relationship between the primal and dual coordinate
vector fields. This is studied in the following proposition.
Proposition 5.6.
(i) For D(α) , the inner product between ∂ξ∂ i and ∂η∂ j is


∂
−1
∂
α
(5.8)
=
,
ξ j ηi .
δij +
∂ξ i ∂η j
Π(ξ, η)
Π(ξ, η)2
(ii) For D(−α) , the inner product between ∂ξ∂ i and ∂η∂ j is


1
∂
α
∂
=
,
ξ j ηi .
δij −
(5.9)
∂ξ i ∂η j
Π(ξ, η)
Π(ξ, η)2

LOGARITHMIC DIVERGENCES AND RÉNYI GEOMETRY

Proof. Consider (i). Using Proposition 4.2, we compute
+

 *

X ∂ξ m  ∂
∂ X ∂ξ m ∂
∂
∂
∂
=
=
,
,
,
∂ξ i ∂η j
∂ξ i m ∂η j ∂ξ m
∂η j ∂ξ i ∂ξ m
m
=

X ∂ξ m
m

∂η j

X ∂η ℓ
−1 ∂η i
α
i
+
η
ξℓ m
Π(ξ, η) ∂ξ m
Π(ξ, η)2
∂ξ
ℓ

We obtain (5.8) under some simplification.

25

!

.


The formulas (5.8) and (5.9) are quite interesting. In the dually flat case (i.e.,
D(0±) , alsoDsee [3, Theorem
6.6]), the two coordinate fields are orthonormal in the
E

sense that ∂ξ∂ i , ∂η∂ j = ±δij . When α > 0, the first term of the inner product is
conformal to ±δij , but there is an extra term related to duality. This formula is
the key ingredient in the proof of the generalized Pythagorean theorem.
5.2. Curvatures. Next we consider the primal Riemann-Christoffel curvature tensor R defined by
(5.10)

R(X, Y )Z = ∇X ∇Y Z − ∇Y ∇X Z − ∇[X,Y ] Z,

where X, Y, Z are vector fields and [X, Y ] = XY − Y X is the Lie bracket. In the
primal coordinate system, we use the notation


∂
∂
∂
∂
,
= Rijk ℓ ℓ .
R
∂ξ i ∂ξ j ∂ξ k
∂ξ

It can be shown (see for example [3, Section 5.8]) that the coefficients are given by
(5.11)

Rijk ℓ = ∂i Γjk ℓ − ∂j Γik ℓ + Γjk m Γim ℓ − Γik m Γjm ℓ .

The dual curvature tensor R∗ is defined analogously.
Proposition 5.7.
(i) For D(α) , the primal curvature tensor is given by

(5.12)
Rijk ℓ (ξ) = α gik (ξ)δjℓ − gjk (ξ)δiℓ .

(ii) For D(−α) , the primal curvature tensor is given by

(5.13)
Rijk ℓ (ξ) = −α gik (ξ)δjℓ − gjk (ξ)δiℓ .

Proof. This is a straightforward but lengthy computation, so we omit the details.

Definition 5.8. When dim M ≥ 2, a torsion-free connection has constant sectional
curvature k ∈ R with respect to the metric g if its curvature tensor R satisfies the
identity
(5.14)

R(X, Y )Z = k (hY, ZiX − hX, ZiY )

for all vector fields X, Y and Z.
Using coordinate vector fields, we see that R has constant curvature k if and
only if in any coordinate system we have

(5.15)
Rijk ℓ = k gjk δiℓ − gik δjℓ .

Comparing this with (5.12) and (5.13), we obtain

26

TING-KAM LEONARD WONG

Theorem 5.9. When dim Ω = d ≥ 2, for D(±α) the primal and dual connections
have constant sectional curvature ∓α with respect to the induced metric.
It is known that for a general dualistic structure (M, g, ∇, ∇∗ ), ∇ has constant
sectional curvature k with respect to g if and only if the same statement holds for
∇∗ . For a proof see [12, Proposition 8.1.4]. Some general properties of dualistic
structures with constant curvatures are given in [12, Theorem 9.7.2]. For example, it
is known that such a manifold is conjugate symmetric, conjugate Ricci-symmetric,
and the connections ∇ and ∇∗ are equiaffine. In Section 6, we show that if we
assume in addition that the manifold is dually projectively flat in the sense of
Theorem 5.2, then the geometry can be characterized elegantly in terms of the
L(±α) -divergence.
5.3. Generalized Pythagorean theorem. The generalized Pythagorean theorem is a fundamental result in information geometry and has numerous applications
in information theory, statistics and machine learning. It was first proved for the
Bregman divergence (D(±α) in our context) which induces a dually flat geometry.
For an exposition of this classical result see [3, Chapter 1]. In [27, 28] we introduced
the L-divergence (L(α) -divergence when α = 1) on the unit simplex and proved the
generalized Pythagorean theorem. In [28] the proof is quite different from that of
the Bregman case and is rather involved because there we expressed the dualistic
structure in terms of the exponential coordinate system. Here we give a unified and
simplified treatment covering all L(±α) -divergences.
Theorem 5.10. Consider any L(±α) -divergence D(±α) [· : ·] and the induced dualistic structure. Let p, q, r ∈ Ω and suppose that the dual geodesic from q to p exists
(this amounts to say that the line segment between ηq and ηp lies in Ω′ . We need
this assumption because Ω′ may not be convex). Then the generalized Pythagorean
relation
(5.16)

D(±α) [q : p] + D(±α) [r : p] = D(±α) [r : p]

holds if and only if the primal geodesic from q to r and the dual geodesic from q to
p meet g-orthogonally at q.
Proof. Again we consider the case of D(α) ; the proof for D(−α) is similar. By the
self-dual representation (Theorem 2.13), we have
1
D(α) [q : p] = log (1 + αξq · ηp ) − ϕ(ξq ) − ψ(ηp ),
α
1
D(α) [r : q] = log (1 + αξr · ηq ) − ϕ(ξr ) − ψ(ηq ),
α
1
D(α) [r : p] = log (1 + αξr · ηp ) − ϕ(ξr ) − ψ(ηp ).
α
Using these expressions and the Fenchel identity, we see that the Pythagorean
relation (5.16) holds if and only if
(1 + αξq · ηp ) (1 + αξr · ηq ) = (1 + αξr · ηp ) (1 + αξq · ηq ) .
Expanding and simplifying, we have
(5.17)

(ξr − ξq ) · (ηp − ηq ) = α(ξq · ηp )(ξr · ηq ) − α(ξr · ηp )(ξq · ηq ).

On the other hand, consider the primal geodesic from q to r. By projective
flatness, in primal coordinates, the initial velocity vector is proportional to ξr − ξq .

LOGARITHMIC DIVERGENCES AND RÉNYI GEOMETRY

27

Similarly, in dual coordinates, the initial velocity of the dual geodesic from q to p
is proportional to ηp − ηq . By Proposition 5.6, we see that the two geodesics are
orthogonal at q if and only if


∂
j
j
i
i ∂
0 = (ξr − ξq ) i , (ηp − ηq ) j
∂ξ
∂η


X
α i j
−1
i
i
j
j
δij + 2 ηq ξq
(ξr − ξq )(ηp − ηq )
=
Π
Π
i,j

!
X
X
−1
α
(ξri − ξqi )ηqi  (ηpj − ηqj )ξqj  ,
=
(ξr − ξq ) · (ηp − ηq ) + 2
Π
Π
j
i
where Π = Π(ξq , ηq ). Rearranging, we have

(1 + αξq · ηq )(ξr · ηp − ξr · ηq − ξq · ηp + ξq · ηq )
= α(ξr · ηq − ξq · ηq )(ξq · ηp − ξq · ηq ).
The proof is completed by verifying that this and (5.17) are equivalent.



5.4. The α-divergence. In this subsection we specialize to the unit simplex
Sd = {p = (p0 , p1 , . . . , pd ) ∈ (0, 1)1+d : p0 + · · · + pd = 1}.
The following definition is taken from [3, (3.39)]. Also see [9, Definition 2.9].
Definition 5.11 (α-divergence). Let α 6= ±1. The α-divergence is defined by
!
d
X
1+α
4
i 1−α
i
, p, q ∈ Sd .
(p ) 2 (q ) 2
1−
(5.18)
Dα [p : q] =
1 − α2
i=0
We also consider the discrete Rényi divergence given by
!
d
X
1
α̃ 1−α̃
.
(5.19)
Dα̃ (p||q) =
pi qi
log
α̃ − 1
i=0

Note the difference in the notations to avoid confusion. The following elementary
lemma (whose proof is omitted) shows that the two divergences are monotone
transformations of each other.
Lemma 5.12. For α̃ ∈ (0, 1) ∪ (1, ∞), let α = 1 − 2α̃ ∈ (−∞, −1) ∪ (−1, 1). Then
for p, q ∈ Sd we have
1
(5.20)
Dα̃ (p||q) =
log (1 + α̃(α̃ − 1)Dα [p : q]) .
α̃ − 1
As in Example 3.8, we may express Dα̃ (p||q) as the L(α̃−1) -divergence of the
F
-potential function when α̃ > 1, and as the L(−(1−α̃)) -divergence of the
−(1−α̃)
F
-potential function when 0 < α̃ < 1.
Let α and α̃ be as in Lemma 5.12. Let (g, ∇, ∇∗ ) be the dualistic structure
˜ ∇
˜ ∗ ) be that induced
induced on Sd by the α-divergence Dα [p : q], and let (g̃, ∇,
by the Rényi divergence Dα̃ (p||q). Since the two divergences are related by a
monotone transformation, from (4.4), (5.20) and the chain rule we have gij = α̃g̃ij ,
Γijk = α̃Γ̃ijk and Γ∗ijk = α̃Γ̃∗ijk . (Here the multiplier α̃ is the derivative of the
transformation (5.20) at 0.) This implies that Γij k = Γ̃ij k and Γ∗ij k = Γ̃∗ij k , so
(α̃−1)

28

TING-KAM LEONARD WONG

the two dualistic structures have the same primal and dual geodesics and the same
primal and dual curvature tensors (see (5.11)).
Theorem 5.13. For α = 1 − 2α̃ ∈ (−∞, −1) ∪ (−1, 1), the statistical manifold
(Sd , g, ∇, ∇∗ ) induced by the α-divergence (5.18) is dually projectively flat with con2
stant curvature 1−α
4 , and a generalized Pythagorean theorem holds.
Proof. We only indicate how the curvature is determined, as the rest follows immediately from other results of this section. We have α̃ = 1−α
2 . Suppose α ∈
′
(−∞, −1) > 0. The Rényi divergence Dα̃ (p||q) is then an L(α ) -divergence where
α̃ = 1 + α′ ∈ (1, ∞). By Theorem 5.9, we have
R̃(X, Y )Z = −α′ (g̃(Y, Z) − g̃(X, Z)Y ) .
But R = R̃ and g̃ = α̃g. So we have
R(X, Y )Z = −α′ α̃ (g(Y, Z) − g(X, Z)) =
This shows that the curvature is

1−α2
4 .

1 − α2
(g(Y, Z) − g(X, Z)) .
4

The case α ∈ (−1, 1) is similar.



In this context, our geometry reduces to the geometry of the α-divergence. See in
2
is determined using affine differential
particular [2, p.179] where the curvature 1−α
4
geometry. Their (8.67) is the product form of our (5.16) for the Rényi divergence
(5.20). Thus, our results provide a new approach to study the geometry (compare
with [2, Section 3.6]). We plan to carry out a deeper study in future papers.
6. Dually projectively flat manifolds with constant curvatures
Using the expressions given in (4.4), any divergence D [· : ·] on a manifold M
defines a dualistic structure (M, g, ∇, ∇∗ ). This operation is not one-to-one; it is
easy to construct examples where two different divergences induce the same dualistic
structure on M . An interesting question is whether we can single out a divergence
which is in some sense most natural for a given dualistic structure. If so, we call it
a canonical divergence. For dually flat manifolds, it is possible to show that there
is a canonical divergence of Bregman type (see [3, Section 6.6]). Thus, a dualistic
structure is dually flat if and only if it is (locally) induced by a Bregman divergence.
In this section we ask and solve the same question where the manifold is dually
projectively flat with constant curvature. We assume that the manifold M has dimension d ≥ 2, and both ∇ and ∇∗ are projectively flat and have constant sectional
curvature ±α where α > 0. (Note that if α = 0 the manifold is dually flat and we
reduce to the classical setting.) Our aim is to characterize the geometry based on
these properties. The key idea is to use these properties to extend in a novel way
the proofs of [3, Theorem 6.2] and [9, Theorem 4.3] which address the dually flat
case. Using tools of affine differential geometry, an alternative characterization is
given in [19, Corollary 1].
6.1. Dual coordinates and potential functions.
Theorem 6.1. Consider a satistical manifold (M, g, ∇, ∇∗ ) such that both ∇ and
∇∗ are projectively flat and have constant non-positive sectional curvature −α where
α > 0. Then, near each point of M , there exist local coordinate systems ξ and η
such that the following statements hold:

LOGARITHMIC DIVERGENCES AND RÉNYI GEOMETRY

29

(i) The primal geodesics are straight lines in the ξ coordinates up to time
reparameterizations, and the dual geodesics are straight lines in the η coordinates up to time reparameterizations.
Pd
(ii) The coordinates satisfy the constraint 1 + α ℓ=1 ξ ℓ η ℓ > 0.
(iii) There exist local α-exponentially concave functions ϕ(ξ) and ψ(η) such
that the generalized Fenchel identity
(6.1)

ϕ(ξ) + ψ(η) ≡ c(α) (ξ, η) =

1
log (1 + αξ · η)
α

holds true.
(iv) Under the coordinate systems ξ and η respectively, the Riemannian metric
is given by
∂ϕ
∂ϕ
∂2ϕ
(ξ) − α i (ξ) j (ξ),
i
j
∂ξ ∂ξ
∂ξ
∂ξ
∂2ψ
∂ψ
∂ψ
gij (η) = − i j (η) − α i (η) j (η).
∂η ∂η
∂η
∂η
gij (ξ) = −

(6.2)

If the constant curvature is +α > 0, analogous statements hold where the functions
ϕ and ψ in (iii) are α-exponentially convex, and in (6.2) the negative signs are
replaced by positive signs.
Proof. Let the curvature be −α < 0 (the proof for positive curvature is similar).
Consider the primal connection ∇ which is projectively flat. By Definition 5.1,
e and a 1-form τ such that (5.1) holds. As ∇
e is
there exists a flat connection ∇
1
d
flat, near each point there exists an affine coordinate system ξ = (ξ , . . . , ξ ) under
e
which the ∇-geodesics
are constant velocity straight lines. By projective equivalence, the primal geodesics are straight lines under the ξ coordinates up to time
reparameterizations. Furthermore, if we write τ = ai (ξ)dξ i where the ai (ξ)’s are
smooth functions of ξ, by (5.2) we have
(6.3)

Γij k (ξ) = ai (ξ)δjk + aj (ξ)δik .

Motivated by our previous result (5.3), we want to show that
(6.4)

ai (ξ) = −α

∂ϕ
(ξ)
∂ξ i

for some α-exponentially concave function ϕ. To this end we will use the assumption
that ∇ has constant sectional curvature −α with respect to g.
Using the representation (6.3) of the primal Christoffel symbols, we compute
(6.5)

Rijk ℓ (ξ) = ∂i Γjk ℓ − ∂j Γik ℓ + Γjk m Γim ℓ − Γik m Γjm ℓ
= (∂i aj − ∂j ai )δkℓ + (∂i ak − ak ai )δjℓ − (∂j ak − ak aj )δiℓ .

On the other hand, since ∇ has constant curvature −α, by (5.15) we have

(6.6)
Rijk ℓ (ξ) = α gik (ξ)δjℓ − gjk (ξ)δiℓ .

Equating (6.5) and (6.6) and using the assumption that dim M = d ≥ 2 (so that
we can pick different indices), we see that
(6.7)

∂ai
∂aj
(ξ) = j (ξ)
∂ξ i
∂ξ

30

TING-KAM LEONARD WONG

and
(6.8)

αgij (ξ) = ∂i aj (ξ) − ai (ξ)aj (ξ).

By (6.7), the 1-form τ = ai (ξ)dξ i is closed. So, locally, it is exact. Thus, there
exists a locally defined function ϕ(ξ) such that our claim (6.4) holds. In particular,
the 1-form τ is locally given by τ (X) = −αXϕ.
Now we may write the metric (6.8) in the form
(6.9)

gij (ξ) = −

∂ϕ
∂ϕ
∂2ϕ
(ξ) − α i (ξ) j (ξ).
i
j
∂ξ ∂ξ
∂ξ
∂ξ

Since the Riemannian matrix (gij (ξ)) is strictly positive definite, from Remark 4.3
we know that D2 eαϕ < 0. In particular, ϕ is locally α-exponentially concave.
Without loss of generality we may assume 0 belongs to the domain of ϕ. This
implies
1 − αDϕ(ξ) · ξ > 0,
and the α-gradient
(6.10)

η = ∇(α) ϕ(ξ) =

1
∇ϕ(ξ)
1 − αDϕ(ξ) · ξ

is well-defined. This implies as in (2.19) that 1 + αξ · η > 0.
With the dual coordinate η defined by (6.10), we define the dual function ψ(η)
by the generalized Fenchel identity:
(6.11)

ψ(η) = c(α) (ξ, η) − ϕ(ξ).

By α-duality ψ is locally α-exponentially concave, and the Fenchel identity (6.1)
holds by construction.
Since the dual connection ∇∗ is uniquely determined given g and ∇ (see [3,
(6.6)]), with the results in Section 4 it is a now routine exercise to verify that η is
an affine coordinate system for the projectively flat dual connection ∇∗ , and the
metric is also given by the second formula in (6.2).

6.2. Canonical divergence. Using Theorem 6.1, we can define locally a canonical
divergence which generalizes the one for a dually flat manifold. It is interesting to
know whether this canonical divergence coincides with the one defined by Ay and
Amari in [8] for a general dualistic structure.
Theorem 6.2 (canonical divergence). Consider the setting of Theorem 6.1. Let ξ,
η, ϕ and ψ be given as in the theorem.
(i) If the curvature is −α < 0, we define the local L(α) -divergence
(6.12)

D(α) [q : p] =

1
log (1 + αξq · ηp ) − ϕ(ξq ) − ψ(ηp ).
α

(ii) If the curvature is α > 0, we define the local L(−α) -divergence
1
log (1 + αξq · ηp ) .
α
Then these divergences are independent of the choices of ξ, η, ϕ and ψ. Furthermore, the canonical divergence induces the given Riemannian metric and the primal
and dual connections.

(6.13)

D(−α) [q : p] = ϕ(ξq ) + ψ(ηp ) −

LOGARITHMIC DIVERGENCES AND RÉNYI GEOMETRY

31

Proof. Note that the primal affine coordinate system ξ is unique up to an affine
transformation. The rest are then determined by (6.10) and (6.11). By a direct
computation, we can verify that (6.12) and (6.13) remain invariant.

Acknowledgment
Some of the results were obtained when the author was teaching a topics course
on optimal transport and information geometry at the University of Southern California. He thanks his students for their interest and feedbacks. He also thanks
John Man-shun Ma, Jeremy Toulisse and Soumik Pal for helpful discussions. He
is grateful to Shun-ichi Amari for his insightful comments and for pointing out the
connection with the α-divergence.
References
[1] S. Adams, N. Dirr, M. A. Peletier, and J. Zimmer. From a large-deviations principle to the
Wasserstein gradient flow: a new micro-macro passage. Communications in Mathematical
Physics, 307(3):791–815, 2011.
[2] S. Amari, H. Nagaoka, and D. Harada. Methods of Information Geometry. Translations of
mathematical monographs. American Mathematical Society, 2007.
[3] S.-I. Amari. Information Geometry and Its Applications. Springer, 2016.
[4] S.-I. Amari, R. Karakida, and M. Oizumi. Information geometry connecting Wasserstein
distance and Kullback-Leibler divergence via the entropy-relaxed transportation problem.
arXiv preprint arXiv:1709.10219, 2017.
[5] S.-I. Amari and A. Ohara. Geometry of q-exponential family of probability distributions.
Entropy, 13(6):1170–1185, 2011.
[6] L. Ambrosio and N. Gigli. A users guide to optimal transport. In Modelling and optimisation
of flows on networks, pages 1–155. Springer, 2013.
[7] L. Ambrosio, N. Gigli, and G. Savaré. Gradient Flows: In Metric Spaces and in the Space of
Probability Measures. Springer, 2008.
[8] N. Ay and S.-i. Amari. A novel approach to canonical divergences within information geometry. Entropy, 17(12):8111–8129, 2015.
[9] N. Ay, J. Jost, H. Lê, and L. Schwachhöfer. Information geometry. Springer, 2017.
[10] A. Banerjee, S. Merugu, I. S. Dhillon, and J. Ghosh. Clustering with bregman divergences.
Journal of machine learning research, 6(Oct):1705–1749, 2005.
[11] Y. Brenier. Polar factorization and monotone rearrangement of vector-valued functions. Communications on pure and applied mathematics, 44(4):375–417, 1991.
[12] O. Calin and C. Udrişte. Geometric modeling in probability and statistics. Springer, 2014.
[13] L. Chizat, G. Peyré, B. Schmitzer, and F.-X. Vialard. Unbalanced optimal transport: geometry and kantorovich formulation. arXiv preprint arXiv:1508.05216, 2015.
[14] F. Dillen, K. Nomizu, and L. Vranken. Conjugate connections and radon’s theorem in affine
differential geometry. Monatshefte für Mathematik, 109(3):221–235, 1990.
[15] S. Eguchi. Second order efficiency of minimum contrast estimators in a curved exponential
family. The Annals of Statistics, 11(3):793–803, 1983.
[16] S. Eguchi et al. Geometry of minimum contrast. Hiroshima Mathematical Journal, 22(3):631–
647, 1992.
[17] M. Erbar, K. Kuwada, and K.-T. Sturm. On the equivalence of the entropic curvaturedimension condition and bochners inequality on metric measure spaces. Inventiones mathematicae, 201(3):993–1071, 2015.
[18] R. Jordan, D. Kinderlehrer, and F. Otto. The variational formulation of the Fokker–Planck
equation. SIAM journal on mathematical analysis, 29(1):1–17, 1998.
[19] T. Kurose. Dual connections and affine geometry. Mathematische Zeitschrift, 203(1):115–121,
1990.
[20] T. Kurose. On the divergences of 1-conformally flat statistical manifolds. Tohoku Mathematical Journal, Second Series, 46(3):427–433, 1994.
[21] C. Léonard. From the Schrödinger problem to the Monge–Kantorovich problem. Journal of
Functional Analysis, 262(4):1879–1920, 2012.

32

TING-KAM LEONARD WONG

[22] H. Matsuzoe. Geometry of contrast functions and conformal geometry. Hiroshima Mathematical Journal, 29(1):175–191, 1999.
[23] H. Nagaoka and S.-i. Amari. Differential geometry of smooth families of probability distributions. Technical Report METR 82–7, University of Tokyo, 1982.
[24] J. Naudts. The q-exponential family in statistical physics. In Journal of Physics: Conference
Series, volume 201, page 012003. IOP Publishing, 2010.
[25] S. Pal. Embedding optimal transports in statistical manifolds. arXiv preprint
arXiv:1708.08148, 2017.
[26] S. Pal and T.-K. L. Wong. Energy, entropy, and arbitrage. arXiv preprint arXiv:1308.5376,
2013.
[27] S. Pal and T.-K. L. Wong. The geometry of relative arbitrage. Mathematics and Financial
Economics, 10(3):263–293, 2016.
[28] S. Pal and T.-K. L. Wong. Exponentially concave functions and a new information geometry.
Annals of Probability, forthcoming.
[29] A. Rényi. On measures of entropy and information. In Proceedings of the Fourth Berkeley
Symposium on Mathematical Statistics and Probability, Volume 1: Contributions to the
Theory of Statistics. The Regents of the University of California, 1961.
[30] R. T. Rockafellar. Convex Analysis. Princeton Landmarks in Mathematics. Princeton University Press, 1997.
[31] F. Santambrogio. Optimal transport for applied mathematicians. Birkäuser, NY, 2015.
[32] T. Van Erven and P. Harremos. Rényi divergence and Kullback-Leibler divergence. IEEE
Transactions on Information Theory, 60(7):3797–3820, 2014.
[33] C. Villani. Topics in Optimal Transportation. American Mathematical Society, 2003.
[34] C. Villani. Optimal Transport: Old and New. Springer, 2008.
[35] T.-K. L. Wong. Optimization of relative arbitrage. Annals of Finance, 11(3-4):345–382, 2015.
[36] T.-K. L. Wong. On portfolios generated by optimal transport. arXiv preprint
arXiv:1709.03169, 2017.
University of Southern California, Los Angeles, CA 90089, United States
E-mail address: tkleonardwong@gmail.com

