Mathematical optimization and statistical theories
using geometric methods
Date: October 20–21, 2022 (Japan Standard Time)
Venue : Academic Extension Center (Osaka Metropolitan University)
Contents : Workshop (Hybrid: physical/virtual)• This workshop is held as a part of OCAMI Joint Usage/Research (JPMXP0619217849)“MEXT Joint Usage/Research Center on Mathematics and Theoretical Physics”• This workshop is also supported by Japan Science and Technology Agency,
CREST“Innovation of Deep Structured Models with Representation of Mathematical Intelligence” in ”Creating information utilization platform by integrating mathematical and information sciences, and development to society”

Organizers: Hideto Nakashima (ISM: hideto (at) ism.ac.jp), Yoshihiko Konno(OMU), Hideyuki Ishi (OMU), Kenji Fukumizu (ISM)
Program• October 20 (Thursday)
13:00–13:50

Shoji Toyota (SOKENDAI)
Invariance Learning based on Label Hierarchy

14:00–14:50

Sho Sonoda (RIKEN AIP)
Ridgelet Transforms for Neural Networks on Manifolds andHilbert Spaces

15:00–15:50

Tomonari Sei (The University of Tokyo)
Ushio Tanaka (Osaka Metropolitan University)
Stein-type distributions on Riemannian manifolds

16:10–17:00

Tomasz Skalski (Wroclaw University of Science and Technology:
LAREMA, University of Angers)
On LASSO and SLOPE estimators and their pattern recovery

17:10–18:00

Carlos Améndola (Technical University of Berlin)
Likelihood geometry of correlation models• October 21 (Friday)
9:00– 9:50

Piotr Zwiernik (University of Toronto)
Mixed convex exponential families and locally associated graphicalmodels

11:00–11:50

Koichi Tojo (RIKEN Center for Advanced Intelligence Project)Classiﬁcation problem of invariant q-exponential families on homogeneous spaces

13:50–14:40

Yoshihiko Konno (Osaka Metropolitan University)
Adaptive shrinkage of singular values for a low-rank matrix mean
when a covariance matrix is unknown

14:50–15:40

Satoshi Kuriki (The Institute of Statistical Mathematics)
Expected Euler characteristic heuristic for smooth Gaussian random ﬁelds with inhomogeneous marginals

16:00–16:50

Piotr Graczyk (LAREMA, University of Angers)
Pattern recovery by SLOPE

2Invariance Learning based on Label Hierarchy
Shoji Toyota
The Graduate University for Advanced Studies (SOKENDAI)
Training data used in machine learning may contain features that are spuriously correlated to the labels of data. Deep Neural Networks (DNNs) often learn such biased
correlations embedded in training data and hence may fail to predict desired labels of test
data generated by a diﬀerent distribution from one to provide training data. To solve theproblem, Invariance Learning (IL) is a rapidly developed approach to overcome the issueof biased correlation, which is caused by some bias in the distribution of a training dataset(e.g., [1]). IL estimates a predictor invariant to the change of distributions, aiming atkeeping good performance in unseen distributions as well as in the training distributions.
While the IL approach has attracted much attention, requiring training data frommultiple distributions may hinder wide applications in practice; preparing training data
in many distributions often involves expensive data annotation.
To mitigate the problem of annotation cost, we propose a novel IL framework for thesituation where the training data of target classiﬁcation is given in only one distribution,
while the task of higher label hierarchy, which needs lower annotation cost, has datafrom multiple distributions. The new IL framework signiﬁcantly reduces the annotationcost in comparison with previous IL methods; we need exhausting annotation of originalclasses only for one distribution and just causaer labels for other distributions. Numericalsimulations and theoretical analysis verify the eﬀectiveness of our framework.

References[1] M. Arjovsky, L. Bottou, I. Gulrajani, and D. Lopez-Paz. Invariant Risk Minimization.
arXiv:1907.02893, 2019.

Ridgelet Transforms for Neural Networks on
Manifolds and Hilbert Spaces
Sho Sonoda
RIKEN AIP, Tokyo 103–0027 Japan
sho.sonoda@riken.jp

Abstract
To investigate how neural network parameters are organized and arranged, it iseasier to study the distribution of parameters than to study the parameters in eachneuron. The ridgelet transform is a pseudo-inverse operator (or an analysis operator)that maps a given function f to the parameter distribution γ so that a network
ZS[γ](x) :=γ(a, b)σ(a · x − b)dadb, x ∈ Rm
Rm ×R

represents f , i.e., S[γ] = f . For depth-2 fully-connected networks on Euclidean space,
the ridgelet transform has been discovered up to the closed-form expression, thus we
could describe how the parameters are organized. However, for a variety of modern
neural network architectures, the closed-form expression has not been known . Recently,
our research group has developed a systematic scheme to derive ridgelet transforms
for fully-connected layers on manifolds (noncompact symmetric spaces G/K) (Sonodaet al., 2022b) and for group convolution layers on abstract Hilbert spaces H (Sonodaet al., 2022a). In this talk, the speaker will explain a natural way to derive those ridgelet
transforms.

References
S. Sonoda, I. Ishikawa, and M. Ikeda. Universality of Group Convolutional Neural Networks Basedon Ridgelet Analysis on Groups. In Advances in Neural Information Processing Systems 35, 2022a.
S. Sonoda, I. Ishikawa, and M. Ikeda. Fully-Connected Network on Noncompact Symmetric Space andRidgelet Transform based on Helgason-Fourier Analysis. In Proceedings of the 39th International
Conference on Machine Learning, volume 162, 2022b.

1

Stein-type distributions on Riemannian manifolds
Tomonari Sei (The University of Tokyo)∗1Ushio Tanaka (Osaka Metropolitan University)∗2

1. Stein-type distributions on the Euclidean space
Let P 2 be the set of probability distributions µ on Rd with mean zero and finite second
moments such that each marginal distribution µi (i = 1, . . . , d) is absolutely continuouswith respect to the Lebesgue measure dxi on R. We say that a probability distributionµ ∈ P 2 is Stein-type if it satisfies
( d)
∫∫
∑
f (xi )
xj dµ = f ′ (xi )dµ, i = 1, . . . , d,j=1

for any absolutely continuous function f : R → R with bounded derivative f ′ .
Let Tcw be the set of coordinate-wise transformations T (x) = (T1 (x1 ), . . . , Td (xd ))such that each Ti is non-decreasing. In [2], it is shown that for any given µ0 ∈ P 2 ,there exists T ∈ Tcw such that T♯ µ0 is Stein-type. The transformation is characterized
by a minimizer of a functional)2∫ (∑d ∫d∑dµi
1
F (µ) =log
dµi +xi dµ,dxi
2 i=1i=1
over a fiber {T♯ µ0 | T ∈ Tcw }. The fiber is totally geodesic in the L2 -Wasserstein spaceand F is convex with respect to displacement interpolation. The optimal map T isapplied to the problem of determining a general index in [2].

2. Generalization to manifolds
We generalize the Stein-type distributions to those∏don Riemannian manifolds. Thedspace R is replaced with a product space M = i=1 Mi , where each Mi is a Riemannian manifold. The space P 2 of distributions is defined as well. Let Tcw be theset of coordinate-wise transformations T (x) = (T1 (x1 ), . . . , Td (xd )) such that each
Ti : Mi → Mi is monotone. Here, Ti is said to be monotone if it is written asTi (xi ) = expxi ∇ϕi (xi ) with a cost convex function ϕi : Mi → R (see [1]). The Steintype distribution is defined by a minimizer of a functional∫d ∫∑dµiF (µ) =logdµi + V (x)dµ,dxi
i=1
over a fiber {T♯ µ0 | T ∈ Tcw }, where V : M → R is a given function.

References[1] McCann, R. J. (2001). Polar factorization of maps on Riemannian manifolds, Geometricand Functional Analysis, 11, 589–608.
[2] Sei, T. (2022). Coordinate-wise transformation of probability distributions to achieve aStein-type identity, Information Geometry, 5, 325–354.
∗1∗2e-mail: sei@mist.i.u-tokyo.ac.jp
e-mail: utanaka@omu.ac.jp

On LASSO and SLOPE estimators and theirpattern recovery
Tomasz Skalski1,2
1Wrocław University of Science and Technology, Poland
2
LAREMA, University of Angers, France

Least Absolute Shrinkage and Selection Operator (LASSO) and Sorted `1 Penalized Estimator (SLOPE) are the regularization methods used for fitting
high-dimensional regression models. They allow to reduce the model dimension by nullifying some of the regression coefficients. Moreover, SLOPE allows the further reduction by equalizing some of nonzero coefficients, whichallows to identify situations where some of true regression coefficients areequal.
We shall introduce the notion of the pattern for LASSO and SLOPE and its
subdifferential-induced generalization to other convex penalized estimators,
which will be illustrated carefully in the case of the orthogonal design matrix.
This talk will present new results on the strong consistency of SLOPE estimators and on the strong consistency of pattern recovery by SLOPE when
the design matrix is orthogonal. We shall also present the relations of LASSO
and SLOPE with root system induced convex hulls.
The research was supported by a French Government Scholarship and by
Centre Henri Lebesgue, program ANR-11-LABX-0020-0.

References[1] M. Bogdan, X. Dupuis, P. Graczyk, B. Kołodziejek, T. Skalski, P. Tardivel, M. Wilczyński. Pattern Recovery by SLOPE. ArXiv 2203.12086.
[2] U. Schneider, P. Tardivel. The geometry of uniqueness, sparsity and clustering in penalized estimation. ArXiv 2004.09106.
[3] T. Skalski, P. Graczyk, B. Kołodziejek, M. Wilczyński. Pattern recoveryand signal denoising by SLOPE when the design matrix is orthogonal.
ArXiv 2202.08573.
[4] P. Tardivel, T. Skalski, P. Graczyk, U. Schneider. The Geometry ofPattern Recovery by Penalized and Structured Estimators. 2021. hal03262087.

Likelihood Geometry of Correlation Models
Carlos Améndola
Technical University of Berlin

We present a problem where algebra appears naturally when estimatingcorrelation matrices, that is, standardized covariance matrices. Concretely,
we study the geometry of maximum likelihood estimation for correlation
matrices, which form an affine space of symmetric matrices defined by setting
the diagonal entries to one.
We study the likelihood geometry for this model and linear submodels
that encode additional symmetries. We also consider the problem of minimizing two closely related functions of the covariance matrix: the Stein’sloss and the symmetrized Stein’s loss. Unlike the Gaussian log-likelihood,these two functions are convex and hence admit a unique positive definiteoptimum.
Studying the critical points in all three settings leads to systems of nonlinear equations, and we compute some of the algebraic degree invariants
that measure the algebraic complexity of each optimization problem.
This is joint work with Piotr Zwiernik (University of Toronto, Canada).

Mixed convex exponential families and locallyassociated graphical models
Piotr Zwiernik (University of Toronto)

Abstract
In statistical exponential families the log-likelihood forms a concave function in the canonical parameters. Therefore, any model given by convex
constraints in these canonical parameters admits a unique maximum likelihood estimator (MLE). Such models are called convex exponential families.
For models that are convex in the mean parameters (e.g. Gaussian covariancegraph models) the maximum likelihood estimation is much more complicatedand the likelihood function typically has many local optima. One solution isto replace the MLE with so called dual likelihood estimator, which is uniquelydeﬁned and asymptotically has the same distribution as the MLE. In this talkI will consider a much more general setting, where the model is given by convex constraints on some canonical parameters and convex constraints on theremaining mean parameters. We call such models mixed convex exponentialfamilies. We propose for these models a 2-step optimization procedure which
relies on solving two convex problems. We show that the resulting estimator
has asymptotically the same distribution as the MLE. Our work was motivated by locally associated Gaussian graphical models that form a suitablerelaxation of Gaussian totally positive distributions.
(Joint work with Steﬀen Lauritzen, University of Copenhagen)Classiﬁcation problem of invariant
q-exponential families on homogeneous spaces
Koichi Tojo
RIKEN Center for Advanced Intelligence ProjectAbstract
Q-exponential family is a natural generalization of exponential family andis an important subject in the ﬁelds of information geometry and statistics.
Widely used q-exponential families such as normal distributions and Cauchy
distributions have a symmetry. More precisely, the sample space can be
regarded as a homogeneous space G/H and the family of distributions on itis G-invariant with respect to the induced G-action by pushforward. Thenthe following problem naturally arises:
Classify G-invariant q-exponential families on G/H.
I would like to talk about a strategy to solve this problem using “qdeformation” of an exponential family. Moreover, we give a new SL(2, R)invariant q-exponential family on the upper half plane.
This is a joint work with Taro Yoshino.

Adaptive shrinkage of singular values for a low-rank matrix mean when a covariancematrix is unknown
Yoshihiko Konno
Department of Mathematics, Osaka Metropolitan University
Assume that
" m,#n, p are positive integers" such# that" min{m,# n} ≥ p and that we observeXXΞ
a matrix
which is modeled as=+ E where Ξ is an m × pYY0n×p
non-random matrix(unknown and its rank may be less than min{p, m}), E is an (m +n) × p error matrix(unobservable) whose rows are identically distributed as Np (0p , Σ), ap-variate real normal distribution with zero mean vector and covariance matrix Σ. We
assume that Σ is a p × p positive-definite and unknown matrix.
We consider the problem of estimating Ξ under a low-rank mean matrix condition, i.e.,rank Ξ = r < p;r is unknown©
ªb Ξ| Σ) = tr (Ξb − Ξ)> (Ξb − Ξ)Σ−1 , where Ξb := Ξ(X,bunder a loss function L(Ξ,Y ) is>an estimator of Ξ. Here A and trA stand for the transpose and the trace of a squareb Ξ| Σ) is given by the expected value of the loss
matrix A. The risk function of R(Ξ,function where the expectation is taken with respect to the joint distribution of (X, Y ).
We give Steins’s unbiased risk estimate for estimators of the formb=ΞµXp¶hj (`j )uj v >j(Y > Y )1/2 .

j=1Here hj : [0, ∞) → [0, ∞), (j = 1, 2, . . . , p) are absolutely continuous functions andU LV > is the singular value decomposition of X(Y > Y )−1/2 where U = (u1 , u2 , . . . , up )is an m × p matrix such that U > U = I p (the p × p identity matrix), V = (v 1 , v 2 , . . . , v p )is a p × p orthogonal matrix, and L is a p × p diagonal matrix whose j-th diagonal
element is given by `j . Note that we may assume that `1 > `2 > · · · > `p > 0 (almosteverywhere) with out loss of generality. Based on SURE formula, we propose an adaptive
soft-theshholding rule to the singular values `1 , `2 , . . . , `p . Furthermore, the results above
are extended to the complex normal distribution setup.

Expected Euler characteristic heuristic for smooth Gaussian
random fields with inhomogeneous marginals
Satoshi Kuriki
The Institute of Statistical Mathematics
10-3 Midoricho, Tachikawa, Tokyo 190-8562, Japankuriki@ism.ac.jp

Abstract
Expected Euler characteristic (EC) heuristic is a method for approximating the tailprobability of the maximum of a Gaussian random field. In this talk, we provide an expected Euler characteristic formula for the approximate tail probability and its relative
approximation error when the index set M is a closed manifold and the mean and varianceof the marginal distribution are not necessarily constant. When the variance is constant,[TTA05] proved that the relative approximation error is exponentially small in a general
setting where the index set M is a stratified manifold. When the variance is not constant,
it is shown that only the subset Msupp of M , referred to as the supporting index set,
contributes to the maximum tail probability. The proposed tail probability formula isan integral of the Euler characteristic density over Msupp , and its relative approximation
error is proven to be exponentially small as in the case of constant variance. These results
are generalizations of [KTT22], who addressed a restricted case of finite Karhunen-Loèveexpansion by the volume-of-tube method. As an example, the tail probability formulafor the largest eigenvalues of noncentral Wishart matrices Wp (ν, Σ; Φ) and its relative approximation error are obtained. Numerical experience supports the high accuracy of the
expected Euler characteristic formulas regardless of whether the marginals are homogeneous or inhomogeneous.
Keywords: Borel’s inequality, Kac-Rice formula, noncentral Wishart distribution, volumeof-tube method, Weyl’s tube formula.

References[KTT22] Satoshi Kuriki, Akimichi Takemura, and Jonathan E. Taylor, The volume-oftube method for gaussian random fields with inhomogeneous variance, Journal ofMultivariate Analysis 188 (2022), 104819.
[TTA05] Jonathan E. Taylor, Akimichi Takemura, and Robert J. Adler, Validity of the
expected Euler characteristic heuristic, Ann. Probab. 33 (2005), no. 4, 1362–1396.

PATTERN RECOVERY BY SLOPEPIOTR GRACZYKAbstract
I will present recent results obtained in [1] jointly with M. Bogdan,
X. Dupuis, B. Kolodziejek, T. Skalski, P. Tardivel and M. Wilczyński.
SLOPE is a popular method for dimensionality reduction in the highdimensional regression. Indeed, some regression coefficient estimatesof SLOPE can be null (sparsity) or can be equal in absolute value(clustering). Consequently, SLOPE may eliminate irrelevant predictors
and may identify groups of predictors having the same influence on thevector of responses.
The notion of SLOPE pattern allows to derive theoretical properties
on sparsity and clustering by SLOPE. Specifically, the SLOPE patternof a vector provides: the sign of its components (positive, negative ornull), the clusters (indices of components equal in absolute value) andclusters ranking.
In this research we give a necessary and sufficient condition for
SLOPE pattern recovery of an unknown vector of regression coefficients.

References[1] M. Bogdan, X. Dupuis, P. Graczyk, B. Kolodziejek, T. Skalski, P. Tardivel,M. Wilczyński, Pattern recovery by SLOPE (2022), arXiv:2203.12086.
Université d’Angers, CNRS, LAREMA, SFR MATHSTIC, F-49000 Angers,
FranceEmail address: graczyk@univ-angers.fr