ISSN2188－1200

IMI Workshop of the Joint Usage Research Projects

Construction of Mathematical Basis for
Realizing Data Rating Service

Editors: Katsuki Fujisawa, Shizuo Kaji, Toru Ishihara, Masaaki Kondo,

Yuji Shinano, Takuji Tanigawa, Naoko Nakayama
九州大学マス・フォア・インダストリ研究所

MI Lecture Note Vol.91 : Kyushu University

IMI Workshop of the Joint Usage Research Projects

Construction of Mathematical Basis for Realizing
Data Rating Service■Editors:
Katsuki Fujisawa, Shizuo Kaji, Toru Ishihara, Masaaki Kondo,
Yuji Shinano, Takuji Tanigawa, Naoko NakayamaAAbout MI Lecture Note Series
The Math-for-Industry (MI) Lecture Note Series is the successor to the COE Lecture
Notes, which were published for the 21st COE Program “Development of Dynamic
Mathematics with High Functionality,” sponsored by Japan’s Ministry of Education,
Culture, Sports, Science and Technology (MEXT) from 2003 to 2007. The MI Lecture Note Series has published the notes of lectures organized under the following two
programs: “Training Program for Ph.D. and New Master’s Degree in Mathematics as
Required by Industry,” adopted as a Support Program for Improving Graduate School
Education by MEXT from 2007 to 2009; and “Education-and-Research Hub for
Mathematics-for-Industry,” adopted as a Global COE Program by MEXT from 2008 to2012.
In accordance with the establishment of the Institute of Mathematics for Industry (IMI)in April 2011 and the authorization of IMI’s Joint Research Center for Advanced andFundamental Mathematics-for-Industry as a MEXT Joint Usage / Research Center in
April 2013, hereafter the MI Lecture Notes Series will publish lecture notes and proceedings by worldwide researchers of MI to contribute to the development of MI.
October 2022
Kenji Kajiwara
Director, Institute of Mathematics for Industry

IMI Workshop of the Joint Usage Research Projects

Construction of Mathematical Basis for Realizing Data Rating Service
MI Lecture Note Vol.91, Institute of Mathematics for Industry, Kyushu University
ISSN 2188-1200
Date of issue: December 27, 2022
Editors: K
 atsuki Fujisawa, Shizuo Kaji, Toru Ishihara, Masaaki Kondo, Yuji Shinano, 
Takuji Tanigawa, Naoko Nakayama
Publisher:
Institute of Mathematics for Industry, Kyushu University
Graduate School of Mathematics, Kyushu University
Motooka 744, Nishi-ku, Fukuoka, 819-0395, JAPANTel +81-(0)92-802-4402, Fax +81-(0)92-802-4405
URL https://www.imi.kyushu-u.ac.jp/

BPreface

In November 2020, the Institute of Mathematics for Industry (IMI) of Kyushu University,
SoftBank Corporation and MAMEZOU Corporation began joint research on the realization ofa "data rating" system that will use mathematical theory to objectively determine the qualityof various types of digital data ("data") accumulated by companies, local governments,
educational and research institutions, etc. The three parties will use the "data rating" system
to clarify the quality of data held by industry, government, and academia. By clarifying thequality of data held by industry, government, and academia through "data rating," the threeparties aim to promote the mutual use of data and revitalize the data distribution market.
This IMI Workshop of the Joint Usage Research Projects “Construction of Mathematical
Basis for Realizing Data Rating Service” held on September 21st and 22, 2022.

And thisworkshop was held jointly with the following international workshop.
The 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop on Advances in Classical andQuantum Algorithms for Optimization and Machine Learning
RIKEN, IMI, The Institute of Statistical Mathematics (ISM), the National University ofSingapore (NUS), the Zuse Institute Berlin (ZIB), and the NHR Center at ZIB hold the sixth
workshop on mathematical optimization and related fields. This workshop was held at the
University of Tokyo from September 16 to 19 and at Kyushu University from September 21 to
22, 2022. The workshop also discussed methodologies for establishing a new mathematicalfoundation (algorithm) for "data rating," building theory, and conducting empiricalexperiments. This lecture note contains the materials of the lectures given at the workshop,
and the Japanese-language lectures were edited in a separate volume. For more information
about this workshop, please refer to the website below 1.
November 2022.
Editors
Katsuki Fujisawa, Shizuo Kaji

(Kyushu University)

Toru Ishihara(Nagoya University)

Masaaki Kondo(Keio University)

Yuji Shinano(Zuse Institute Berlin)

Takuji Tanigawa(SoftBank Corp.)

Naoko Nakayama(MAMEZOU Corp.)i2022年度 九州大学マス・フォア・インダストリ研究所 共同利用・共同研究 女性研究者活躍支援研究-研究集会(I)

データ格付けサービス実現のための数理基盤の構築
9 21 22

2022.

水

木

九州大学 伊都キャンパス ウエスト1号館 C棟
２階教育情報システム室（W1-C-201）
Computer Lab (W1-C-201), West Zone 1,
Ito campus, Kyushu University

Program■9月21日（水）■9月22日（木）

The 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop
on Advances in Classical and Quantum Algorithms for
Optimization and Machine Learning

10:30‒11:10

09:30‒11:30 Session 13
Thorsten Koch（ZIB）
João Doriguello（NUS）
Ralf Borndörfer（ZIB）
11:00‒11:20 Coﬀee Break
11:20‒12:20 Session 14
Pierre-Louis Poirion（RIKEN）
Akifumi Okuno（ISM / RIKEN）
12:20‒13:50 Lunch Break
13:50‒15:10 Session 15
Niels Lindner（ZIB）
Inci Yüksel-Ergün（ZIB）
Jaap Pedersen（ZIB）
15:10‒15:30 Break
15:30‒16:50 Session 16
Uwe Gotzes（Open Grid Europe GmbH）
Ying Chen（NUS）
Osamu Saeki（IMI）
Katsuki Fujisawa（IMI）

株式会社豆蔵 執行役員 C.D.O．
（DX事業推進統括）安井 昌男
11:15‒12:00

株式会社Fixstars Amplify 代表取締役社長 CEO

平岡 卓爾

12:00-13:30 昼休憩
13:30‒14:00
株式会社豆蔵 デジタル戦略支援事業 AI-Techチーム コンサルタント

林 沛萱

14:15‒15:00
立教大学 大学院人工知能科学研究科 特任准教授
株式会社豆蔵
デジタル戦略支援事業 AI-Techチームリーダー チーフ・コンサルタント

石川 真之介15:15‒16:00
横河電機株式会社 常務執行役員（CIO）
デジタル戦略本部長 兼 デジタルソリューション本部 DX－Platformセンター長

舩生 幸宏[研究代表者]

中山 尚子（株式会社豆蔵）

参加
無料
事前申込制

[組織委員]

谷川 拓司（ソフトバンク株式会社）
品野 勇治（Zuse Institute Berlin）
近藤 正章（慶応大学）
石原 亨（名古屋大学）
鍛冶 静雄（九州大学）
藤澤 克樹（九州大学）[共催機関]

理化学研究所（RIKEN）
九州大学マス・フォア・インダストリ研究所（IMI）
統計数理研究所（ISM）
National University of Singapore（NUS）
Zuse Institute Berlin（ZIB）
NHR Center at ZIB

最新情報
最新情報は、下記のホームページをご覧ください
URL：https://sites.google.com/view/
6th-riken-imi-ism-zib-workshop/program

16:15‒16:50
株式会社豆蔵 チーフ・コンサルタント

中山 尚子

参加申込
参加にあたっては、下記の参加申込をお願いします
URL:https://forms.gle/H3rA1D7EmHMGKQtC6 【申込フォーム】

E-mail : imikyoten@jimu.kyushu-u.ac.jp

https://joint.imi.kyushu-u.ac.jp/post-6137/

ii

Program

September 17th
09:20--11:30 Session 01⚫

09:20--09:30 Akiko Takeda (RIKEN / Univ Tokyo) "Opening Remarks"⚫

09:30--10:00 Masashi Sugiyama (RIKEN / Univ Tokyo) "Recent Advances in Machine
Learning from Noisy Labels"⚫

10:00--10:30 Patrick Gelß (ZIB) "Low-rank tensor representations of quantum circuits"⚫

10:30--11:00 Shintaro Momose (NEC) "Aurora Vector Annealing to Solve Social Issues
and Acceleration by NEC's Supercomputer, SX-Aurora TSUBASA"

11:00--11:20 Coffee Break
11:20--12:20 Session 02⚫

11:20--11:50 Kazuma Tsuji (MUFG Bank) "Pairwise Conditional Gradients without⚫

11:50--12:20 Christoph Spiegel (ZIB) "Proofs in Extremal Combinatorics throughSwap Steps and Sparser Kernel Herding"
Optimization"⚫

12:20--13:50 Lunch Break⚫

13:50--15:20 Ice Breaking

15:20--16:20 Session 03⚫

15:20--15:50 Yuji Shinano (ZIB) "The UG framework version 1.0: An update"⚫

15:50--16:20 Junko Hosoda (Hitachi) "A parallel algorithm combining relaxation andheuristic for the integrated long-haul and local vehicle routing problem on an adaptive
transportation network"

16:20--16:40 Break
16:40--17:40 Session 04⚫

16:40--17:10 Koichi Fujii (NTT DATA MSI) "Solving Large Scale QAPs by Massively
Parallel DNN-based Branch-and-bound Method"⚫

17:10--17:40 Elias Wirth (ZIB) "Approximate Vanishing Ideal Computations at Scale"

September 18th
09:30--11:30 Session 05⚫

09:30--10:00 Katsuki Fujisawa (Kyushu Univ) "Mobility Optimization Engine and its
Real-world Applications"⚫

10:00--10:30 Hiroki Ishikura (Kyushu Univ) "Towards an optimal operation of
automated storage and retrieval system with multiple machines"⚫

10:30--11:00 Nozomi Hata (Kyushu Univ) "Theoretical Analysis for Representation
Learning Methods of Graph-Structured Data"

11:00--11:20 Coffee Breakiii

11:20--12:20 Session 06⚫

11:20--11:50 Mark Turner (TU Berlin) "Adaptive Cut Selection in Mixed-Integer
Linear Programming"⚫

11:50--12:20 Ryohei Yokoyama (Osaka Metro Univ) "A Quadratic Programming
Approach for Performance Analysis of Energy Systems"

12:20--13:50 Lunch Break
13:50--15:50 Ice Breaking
15:50--16:50 Session 07⚫

15:50--16:20 Shizuo Kaji (Kyushu Univ) "Geometric Learning of Ranking
Distributions"⚫

16:20--16:50 Akiko Takeda (RIKEN / Univ Tokyo) "Bilevel Optimization for
Machine Learning Problems"

September 19th
09:30--11:30 Session 08⚫

09:30--10:00 Sebastian Pokutta (ZIB) "Convex integer optimization withFrank-Wolfe methods"⚫

10:00--10:30 Shota Takahashi (SOKENDAI / ISM) "Bregman Proximal DC
Algorithms and Their Application to Blind Deconvolution with Nonsmooth
Regularization"⚫

10:30--11:00 Akira Tanaka (NICT) "Port Set Clustering for Internet-Wide Scanner"

11:00--11:20 Coffee Break
11:20--12:20 Session 09⚫

11:20--11:50 Atsushi Miyauchi (Univ Tokyo) "Finding densest k-connected
subgraphs"⚫

11:50--12:20 Antoine Deza (McMaster Univ) "Worst-case constructions for linear
optimization"

12:20--13:50 Lunch Break
13:50--15:20 Session 10⚫

13:50--14:20 Xun Shen (Tokyo Inst Tech) "Approximate Methods for Solving
Chance Constrained Linear Programs in Probability Measure Space"⚫

14:20--14:50 Jun-ya Gotoh (Chuo Univ) "Knot Selection of B-Spline Regression via
Trimmed Regularizer"⚫

14:50--15:20 Keisuke Yano (ISM) "Minimum information dependence modelingand its application"

15:20--15:40 Break
15:40--17:10 Session 11⚫

15:40--16:10 Naoki Marumo (NTT / Univ Tokyo) "A generalized
Levenberg–Marquardt method for large-scale composite minimization"⚫

16:10--16:40 Shunji Umetani (Osaka Univ) "BIPSOL: A metaheuristic solver forlarge-scale binary integer programs"iv

⚫

16:40--17:10 Masahiro Nakao (RIKEN) "Performance of the supercomputer
Fugaku for Graph500 benchmark"September 21st
09:30--11:30 Session 12⚫

09:30--10:00 Thorsten Koch (ZIB) "Notes on Solving QUBOs and Quantum
Computing"⚫

10:00--10:30 João Doriguello (NUS) "Quantum algorithm for stochastic optimal
stopping problems with applications in finance"⚫

10:30--11:00 Ralf Borndörfer (ZIB) "Multicriteria Shortest Path Algorithms"

11:00--11:20 Coffee Break
11:20--12:20 Session 13⚫

11:20--11:50 Pierre-Louis Poirion (RIKEN) "Randomized subspace regularized
Newton method for unconstrained non-convex optimization"⚫

11:50--12:20 Akifumi Okuno (ISM / RIKEN) "Minimax Analysis for Inverse Risk in
Nonparametric Invertible Regression"

12:20--13:50 Lunch Break
13:50--15:10 Session 14⚫

13:50--14:20 Niels Lindner (ZIB) "On the geometry of periodic timetables in public
transport"⚫

14:20--14:50 Inci Yüksel-Ergün (ZIB) "Improving data quality in the presence ofsuperhuman complexity in data errors"⚫

14:50--15:10 Jaap Pedersen (ZIB) "Optimal discrete pipe sizing for tree-shaped
CO2 networks"

15:10--15:30 Break
15:30--16:50 Session 15⚫

15:30--16:00 Uwe Gotzes (OGE) "Spotlights on success stories of public-private
partnership"⚫

16:00--16:30 Ying Chen (NUS) "Deep Switching State Space Model (DS3M) for
Nonlinear Time Series Forecasting with Regime Switching"⚫

16:30--16:40 Osamu Saeki (Kyushu Univ) "Institute of Mathematics for Industry:
its uniqueness, strength and prospects"⚫

16:40--16:50 Katsuki Fujisawa (Kyushu Univ) "Closing Remarks"v

vi

Table of contents
preface ����������������������������������� i
Poster����������������������������������� ii
Program ��������������������������������� iii
Group Photo�������������������������������� vi
Recent Advances in Machine Learning from Noisy Labels

Masashi SUGIYAMA (RIKEN Center for Advanced Intelligence Project/
Graduate School of Frontier Sciences, The University of Tokyo.) ��������� 1

Low-Rank Tensor Representations of Quantum Circuits

Patrick Gelß (Zuse Institute Berlin, Germany)������������������ 19

NEC’s Quantum Computing Technologies

Shintaro MOMOSE (Quantum Computing Business Department Advanced Platform
Division NEC Corporation) �������������������������� 33

Pairwise Conditional Gradients without Swap Steps and Sparser Kernel Herding

Kazuma Tsuji (MUFG Bank, Ltd., Tokyo, Japan)����������������� 43

Proofs in Extremal Combinatorics through Optimization

Christoph SPIEGEL (Zuse Institute Berlin, Germany) �������������� 59

The UG framework version 1.0: An update

Yuji Shinano (Zuse Institute Berlin, Germany)������������������ 69

A Parallel Algorithm Combining Relaxatino and heuristic for the integrated longhaul and local vehicle routing problem

Junko HOSODA (Controls and Robotics Innovation Center, Hitachi Ltd., Japan),
Stephen J. MAHER (University of Exeter, United Kingdom), Yuji SHINANO (ZuseInstitute Berlin, Germany) and Honas Christoffer VILLUMSEN (Hitachi Europe Ltd.,
Denmark)���������������������������������� 91

Solving Large Scale QAPs by Massively Parallel DNN-based Branch-and-boundMethodKoichi FUJII (NTT DATA Mathematical Systems Inc. Japan)���������� 109

Mobility Optimization Engine and its Real-world Applications

Katsuki FUJISAWA (Institute of Mathematics for Industry, Kyushu University, Japan)

137

Towards an optimal operation of automated storage and retrieval system withmultiple machines

Hiroki ISHIKURA (Kyushu University, Japan)����������������� 153

Adaptive Cut Selection in Mixed-Integer Linear Programming

Mark Turner (Chair of Software and Algorithms for Discrete Optimization, Institute ofMathematics, Technische Universität Berlin) ����������������� 173

A Quadratic Programming Approach for Performance Analysis of Energy Systems

Ryohei YOKOYAMA (Department of Mechanical Engineering, Osaka Metropolitan
University, Japan) ����������������������������� 183

vii

Geometric Learning of Ranking Distributions"Shizuo KAJI (Institute of Mathematics for Industry, Kyushu University, Japan)�� 195

Convex integer optimization with Frank-Wolfe methods

Sebastian POKUTTA (Zuse Institute Berlin and TU Berlin) ���������� 209

Bregman Proximal DC Algorithms and Their Application to Blind Deconvolution
with Nonsmooth Regularization

Shota Takahashi (The Graduate University for Advanced Studies, Japan) ���� 241

Theoretical Analysis for Representation Learning Methods of Graph-Structured Data
Nozomi HATA (Institute of Mathematics for Industry, Kyushu University, Japan)

251

Port Set Clustering for Internet-Wide Scanner

Akira TANAKA, Chansu Han, Takeshi Takahashi (National Institute of Information andCommunications Technology, Japan)��������������������� 271

Finding Densest k-Connected Subgraphs

Atsushi MIYAUCHI (Graduate School of Information Science and Technology,
University of Tokyo, Japan)������������������������� 301

Worst-case constructions for linear optimization

Antoine DEZA (McMaster University, Canada) ���������������� 317

Approximate Methods for Solving Chance Constrained Linear Programs in
Probability Measure Space

Xun Shen (Tokyo Institute of Technology, Japan) ��������������� 337

A generalized Levenberg–Marquardt method for large-scale composite minimization

Naoki MARUMO (University of Tokyo / NTT, Japan) ������������� 349

BIPSOL: A metaheuristic solver for large-scale binary integer programs

Shunji UMETANI (Graduate School of Information Science and Technology, Osaka
University, Japan) ����������������������������� 363

Performance of the supercomputer Fugaku for Graph500 benchmark

Masahiro Nakao (RIKEN Center for Computational Science, Japan) ������� 377

Notes on Solving QUBOs and Quantum Computing

Prof. Dr. Thorsten KOCH (Zuse Institute Berlin and Technical University of Berlin,
Germany) ��������������������������������� 391

Quantum algorithm for stochastic optimal stopping problems with applications infinanceJoão F. DORIGUELLO (Centre for Quantum Technologies, National University ofSingapore, Singapore)���������������������������� 421

Multicriteria Shortest Path Algorithms

Ralf BORNDÖRFER (Zuse Institute Berlin & Freie Universität Berlin, Berlin, Germany) ������������������������������������� 439

Randomized subspace regularized Newton method for unconstrained non-convex
optimization

Pierre-Louis POIRION (RIKEN-AIP, Tokyo, Japan)��������������� 463

Minimax Analysis for Inverse Risk in Nonparametric Invertible Regression

Akifumi OKUNO (Institute of Statistical Mathematics / RIKEN AIP, Japan) ��� 481

viii

On the geometry of periodic timetables in public transport

Niels LINDNER (Zuse Institute Berlin, Germany) ��������������� 501

Improving Data Quality in the Presence of Superhuman Complexity in Data Errors

Inci YÜKSEL-ERGÜN ( Applied Algorithmic Intelligence Methods, Zuse Institute Berlin,
Germany) ��������������������������������� 531

Optimal discrete pipe sizing for tree-shaped CO2 networks

Jaap Pedersen, Thi Thai Le, Thorsten Koch, Janina Zittel (Zuse Institute Berlin, Berlin,
Germany Technische Universität Berlin, Berlin, Germany) ����������� 547

Spotlights on success stories of public-private partnership

Uwe GOTZES (Open Grid Europe GmbH, Essen, Germany)����������� 561

Deep Switching State Space Model (DS3M) for Nonlinear Time Series Forecastingwith Regime Switching

Xiuqin Xu (Integrative Sciences and Engineering Programme, NUS Graduate School
Institute of Data Science, National University of Singapore)
Ying Chen (Department of Mathematics, National University of Singapore Asian
Institute of Digital Finance, National University of Singapore, Risk Management
Institute, National University of Singapore, Singapore)������������� 573

Institute of Mathematics for Industry: its uniqueness, strength and prospects

Osamu SAEKI (Institute of Mathematics for Industry, Kyushu University, Japan)� 607

ix

The 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop on
Advances in Classical and Quantum Algorithms for
Optimization and Machine Learning
September 16th - 19th, 2022, Tokyo (The University of Tokyo), Japan,
and September 21st - 22nd, 2022, Fukuoka (Kyushu University), Japan

Recent Advances in Machine Learningfrom Noisy Labels
Masashi SUGIYAMARIKEN Center for Advanced Intelligence Project/
Graduate School of Frontier Sciences, The University of Tokyo.
sugi@k.u-tokyo.ac.jp
Supervised learning from noisy output is one of the classical problems in machinelearning. While this task is relatively straightforward in regression since independent
additive noise cancels with big data, classification from noisy labels is still a challenging
research topic. Recently, it has been shown that when the noise transition matrix which
specifies the label flipping probability is available, the bias caused by label noise can becanceled by appropriately correcting the loss function. However, when the noisetransition matrix is unknown, which is often the case in practice, its estimation only fromnoisy labels is not straightforward due to its non-identifiability. In this talk, I will givean overview of recent advances in classification from noisy labels, including joint
estimation of the noise transition matrix and a classifier, analysis of identifiability
conditions, and extension to instance-dependent noise.

References[1] Patrini, G., Rozza, A., Menon, A. K., Nock, R., & Qu, L. In Proceedings of the IEEE Conference
on Computer Vision and Pattern Recognition (CVPR2017), pp. 1944-1952, 2017.
[2] Zhang, Y., Niu, G., & Sugiyama, M. Learning noise transition matrix from only noisy labels via
total variation regularization. In Proceedings of 38th International Conference on Machine
Learning (ICML2021), pp. 12501-12512, 2021.
[3] Li, X., Liu, T., Han, B., Niu, G., & Sugiyama, M. Provably end-to-end label-noise learning
without anchor points. In Proceedings of 38th International Conference on Machine Learning(ICML2021), pp. 6403-6413, 2021.
[4] Xia, X., Liu, T., Han, B., Wang, N., Gong, M., Liu, H., Niu, G., Tao, D., & Sugiyama, M. Partsdependent label noise: Towards instance-dependent label noise. In Advances in Neural
Information Processing Systems 33, pp. 7597-7610, 2020.
[5] Berthon, A., Han, B., Niu, G., Liu, T., & Sugiyama, M. Confidence scores make instancedependent label-noise learning possible. In Proceedings of 38th International Conference on
Machine Learning (ICML2021), pp. 825-836, 2021.
[6] Cheng, D., Liu, T., Ning, Y., Wang, N., Han, B., Niu, G., Gao, X., & Sugiyama, M. Instancedependent label-noise learning with manifold-regularized transition matrix estimation. In
Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR2022), pp. 16630-16639, 2022.
[7] Han, B., Yao, Q., Yu, X., Niu, G., Xu, M., Hu, W., Tsang, I., & Sugiyama, M. Co-teaching:
Robust training deep neural networks with extremely noisy labels. In Advances in Neural
Information Processing Systems 31, pp. 8527-8537, 2018.
[8] Han, B., Niu, G., Yu, X., Yao, Q., Xu, M., Tsang, I., & Sugiyama, M. SIGUA: Forgetting may
make learning with noisy labels more robust. In Proceedings of 37th International Conference on
Machine Learning (ICML2020), pp. 4006-4016, 2020.
[9] Sugiyama, M., Bao, H., Ishida, T., Lu, N., Sakai, T., & Niu, G. Machine Learning from Weak
Supervision: An Empirical Risk Minimization Approach, MIT Press, 2022.

1

Sep. 17, 2022

The 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop
on Advances in Classical and Quantum Algorithms
for Optimization and Machine Learning

Recent Advances in Machine
Learning from Noisy Labels
Masashi Sugiyama
RIKEN Center for Advanced Intelligence Project/
The University of Tokyo
http://www.ms.k.u-tokyo.ac.jp/sugi/

RIKEN-AIP

2

 MEXT Advanced Intelligence Project (2016-2025):
 130 employed researchers (36% international, 23% female) 200 visiting researchers, 100 domestic students
 140 international interns (total) Missions: Develop new AI technology (ML, Opt, math) Accelerate scientific research (cancer, material, genomics)
 Solve socially critical problems (disaster, elderly healthcare) Study of ELSI in AI (ethical guidelines, personal data) Human resource development (researchers, engineers)
Distributed offices
across Japan

SendaiKana Tsukuba
Shiga gawa
Kyoto
Tokyo
Nara Nagoya
Fukuoka

Main office in theheart of Tokyo

2

Standard Supervised Learning
Goal: Learn a predictor
input-output training data
: Input3

from.

: OutputApproach: Training error minimization(a.k.a. empirical risk minimization or
maximum likelihood estimation): LossWithout Output Noise4Suppose there is no noise in output:
Regression

Classification

OutputClass 1

True function

Boundary

Input

Class 3

Training error minimizationis statistically consistent: WhenClass 2,converges to true .

Big data helps!
3With Output NoiseOutput

5

is often corrupted by noise: Due to sensor errors, human errors, etc.

Regression

Classification
Class 2

Class 1

OutputTrue function

Boundary

Input

Class 3

In the noisy case, does big data still help?

Noisy-Output Regression

6

Standard noise assumptions: Additive:

True function

Output Input-independent: Zero-mean:

Learned function

Input

Noisy training error minimization isstill statistically consistent.

Naïve use of big data still helps!
4Noisy-Output Classification

7

Standard assumptions: Class-conditional noise (input-independent flip):

Noisy training error minimization is
not always statistically consistent:

Need to explicitly remove
the influence of label noise in learning!

Generic Approach (1)8Unsupervised noisy data removal:
 Hotelling’s T2 statistics

LOF scores k-means clustering
 local outlier factor (LOF)
Breunig et al. (SIGMOD2000)

https://en.wikipedia.org/wiki/Harold_Hotellinghttps://en.wikipedia.org/wiki/Local_outlier_factor

Easy to use, but this iscompletely heuristic and no supervision is used.

5

Generic Approach (2)9Regularization: keeping the norm of the model
parameters small for preventing overfitting.
 Tikhonov regularization
Space of model parameters

https://en.wikipedia.org/wiki/
Andrey_Nikolayevich_Tikhonov

Regularized
Non-Regularizedℓ2-norm ball

Nice theory, but smoothing is
not enough to cope with strong label noise.

https://en.wikipedia.org/wiki/Overfitting

Generic Approach (3)10Robust statistics: suppressing the influenceof noisy data by a gentle loss.
 Huber loss

Training with
ramp loss Ramp loss
Squaredhinge loss

https://en.wikipedia.org/wiki/Peter_J._Huber

Ramp
loss

Huber
hinge loss

Noisy data

Classification margin

Nice theory for regression (additive noise), but
not very robust in classification (flipping noise).
6

Goal of Noisy-Label Classification 11
These generic approaches were notspecifically designed for handling label noise.
In this talk, I review recent advancesin noisy-label multi-class classificationthat explicitly handle noisy supervision: Forward/backward loss correction.
 Noise transition estimation.
 Coping with non-identifiability.
 Input-dependent label noise.

Contents
1. Introduction
2. Technical background
3. Single-step approach
4. Beyond anchor points
5. Further challenges

712Formulation

13

Clean training data:
Noisy training data:：Input instance：Clean class label：Noisy class label

Probabilistic classifier in simplex:
 Each element approximatesthe class-posterior probability.

Class 1

Class 2

Boundary

Loss:Class 3Modeling Class-Conditional Noise 14
1

Noise transition matrix:
 Probability of flipping

to

0

0

0.1 0.8 0.1.

0.5 0.50

We may encode human-cognitive bias:
Han, Yao, Niu, Zhou, Tsang,
Zhang & Sugiyama (NeurIPS2018)

Visualization as a simplex:

8

Zhang, Niu & Sugiyama (ICML2021)

Loss Correction

15

Patrini, Rozza, Menon, Nock & Qu (CVPR2017)Forward correction: Add noise byClassifier-consistency

Backward correction: Remove noise byClassifier-consistency

Risk-consistencyIfis given, consistency can be guaranteed!

Identifiability of Noise Transition
In practice, we need to estimate
from noisy training data
However,16.

is non-identifiable in general:can be decomposed as,
where
are some transition matrices.
 Then

Let’s use anchor points (100%-certain samples):9

Estimation of Noise Transition
with Anchor Points

17Given anchor points,
can be naïvely estimated asis a probabilistic classifier learned
from noisy training data.

Even if anchor points are unknown,
as long as they exist in noisy training data,
we may find them as.

Further Improvements

18

We typically use deep learning to obtain:

 Then it is often over-confident and unreliable.
Zhang, Niu & Sugiyama(ICML2021)

Estimatedis revised during classifier training:
Xia, Liu, Wang, Han, Gong, Niu & Sugiyama (NeurIPS2019)

Instead of explicitly finding anchor points,
latent labels are utilized:

Yao, Liu, Han, Gong, Deng, Niu, Sugiyama & Tao (NeurIPS2020)10

Contents

19

1. Introduction
2. Technical background
3. Single-step approach
4. Beyond anchor points
5. Further challenges

Challenge20Current approaches are in two-step:
1.
2.

Estimate transition matrix .
Use estimatedto train a classifierStep 1 is done without regard to Step 2:
 Estimation error ofin Step 1
can be magnified in Step 2.

We want to estimate andsimultaneously in one-step.

11.

Naïve Solution

21

Naively, we may learn the noise transition andclassifier at the same time asHowever, the solution is not unique: With any invertible transition matrix,
are solutions.

anyWe need a certain constraint to obtain
the right solution:

Total Variation Regularization

22

Zhang, Niu & Sugiyama (ICML2021)

Noise transition
in total variation distance:

is contraction

 Cleaner class-posteriors havea larger total variation distance!

Let’s use this knowledge as a regularizer:
 Under the anchor point assumption,

the empirical solution has statistical consistency.

12

Contents

23

1. Introduction
2. Technical background
3. Single-step approach
4. Beyond anchor points
5. Further challenges

Challenges24

To overcome the non-identifiability of: Anchor points are explicitly used.

This condition has been relaxed to: Only the existence of anchor points is assumed

Can we further relax this assumption?

13

Non-identifiability of T

25

can be visualized as a simplex,
containing all training data.
Generally, such a simplex is not unique.
Anchor points are vertices of the true simplex.
 Explicitly using anchor points naively recovers

Non-identifiability of T (cont.)
Only the existence of anchor points still
guarantees the identifiability of .
Even without anchor points, “sufficiently
scattered” training data can guarantee theconsisntency (with the next algorithm).

14

.

26

Volume Minimization

27

Li, Liu, Han, Niu & Sugiyama (ICML2021)

Under the “sufficiently scattered” assumption,
minimizing the volume of the transition matrix
guarantees consistency!

Contents
1. Introduction
2. Technical background
3. Single-step approach
4. Beyond anchor points
5. Further challenges

15

28

Beyond Class-Conditional NoiseInstance-independence
in class-conditional noiseis restrictive.

Class-conditional

29

Instance-dependent

Instance-dependent noise: Extremely challenging problem!

Various heuristic solutions:

Xia, Liu, Han, Wang,
Gong, Liu, Niu, Tao& Sugiyama (NeurIPS2020)

 Parts-based estimation
 Use of additional confidence scores
 Manifold regularization

Berthon, Han,Niu, Liu& Sugiyama (ICML2021)

Cheng, Liu, Ning, Wang, Han, Niu,
Gao & Sugiyama (CVPR2022)

Co-teaching30

 Memorization of neural nets:

Arpit et al. (ICML2017)
Zhang et al. (ICLR2017)

 Stochastic gradient descent fits clean data faster.

 However, naïve early stopping does not work well.

 “Co-teaching” between two neural nets:
 Teach small-loss data each other.

Han, Yao, Yu, Niu, Xu, Hu, Tsang & Sugiyama (NeurIPS2018)

 Teach only disagreed data.
Yu, Han, Yao, Niu, Tsang & Sugiyama (ICML2019)

 Gradient ascent for large-loss data.
Han, Niu, Yu, Yao, Xu, Tsang & Sugiyama (ICML2020) No theory but very robust in experiments:
 Works well even if 50% random label flipping!

16

Summary31

 Classification requires explicit treatment of label noise: Loss correction by noise transition is promising.

 However, noise transition is
generally non-identifiable.
 Recent development allows its consistentestimation under mild assumptions.

 Real-world noise is often instance-dependent: Heuristic solutions have been developed.

 Super-robustness by co-teaching: Heuristic solutions have been developed.

17

The 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop on
Advances in Classical and Quantum Algorithms for
Optimization and Machine Learning
September 16th - 19th, 2022, Tokyo (The university of Tokyo), Japan,
and September 21st - 22nd, 2022, Fukuoka (Kyushu University), Japan

Low-Rank Tensor Representations of
Quantum Circuits
Patrick Gelß
Zuse Institute Berli n, Germ any
gelss@zib.de
Quantum computing is arguably one of the most revolutionary and disruptive
technologies of this century. Due to the ever-increasing number of potential applications
as well as the continuing rise in complexity, the development, simulation, optimization,and physical realization of quantum circuits is of utmost importance for designing novel
algorithms. We show how matrix product states (MPSs) and matrix product operators(MPOs) can be used to express not only the state of the system but also quantum gatesand entire quantum circuits as low-rank tensors. This allows us to analyze and simulate
complex quantum circuits on classical computers and to gain insight into the underlying
structure of the system. We present different examples to demonstrate the advantages of
MPO formulations and provide a new perspective on the construction of quantum
algorithms.

19

Low-rank tensor
decompositions ofquantum circuits
Zuse
Institute
BerlinPatrick Gelß
joint work with Stefan Klus, Zarin Shakibaei,
and Sebastian Pokutta

Universityof Surrey

Freie
Universität
Berlin6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop
Tokyo, Japan
September 17, 2022

Motivation

Technische
Universität
BerlinTensor Decomposition

Quantum Circuits

Results

M OTIVATIONQUANTUM SIMULATION• tensor product representation
already been used

• in particular, MPS/TT format• reduce storage consumption
and computational effort• AIM: construct compact

expressions of quantum
circuits in form of MPOs

• MPS-MPO contraction torepresent wave functions• direct insight into

network structure• reduce CPU time forquantum simulations onclassical computers: towards QOPT, QML ...
RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR // P. Gelß // Low-rank tensor decompositions of quantum circuits // 01

20

Motivation

Tensor Decomposition

Quantum Circuits

Results

T ENSORS

1
0
1
1

1
0
1
1

0
1
1
0

1 0 1 10 1 1 01 1 11 0 00 0 01 1 0
0 1 01 1 11 1 10 1 0
1 0 0 1

1
1
0
1

Z
_
_
_
_
D
_
^ C is the linear• tensor: T œ CD = Cd1 ◊d2 ◊···◊dnspace of tensors with¶ (T + U)x ,...,xn = Tx1 ,...,xn + Ux1 ,...,xn___
modes d1 , . . . , dn
_
_
\

1

¶ (– · T)x ,...,xn = – · Tx1 ,...,xn
1

Z
_
_
_
_
D◊D
_
is the space^ C• tensor operator: L œ CD◊D = Cd1 ◊···◊dn ◊d1 ◊···◊dnÿ¶ (L · T)x ,...,xn =Lx1 ,...xn ,y1 ,...,yn · Ty1 ,...,yn1

of linear maps from___CD to CD__\y1 ,...,yn

RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR // P. Gelß // Low-rank tensor decompositions of quantum circuits // 02

Motivation

Tensor Decomposition

Quantum Circuits

Results

T ENSOR P RODUCTÕÕ• given tensors T œ Cd1 ◊···◊dn and U œ Cd1 ◊···◊dm , the tensor product T ¢ U
is defined by(T ¢ U)x ,...,xn ,y ,...,ym = Tx1 ,...,xc · Uy1 ,...,yd
1

u1
..
.

um

v1

¢

..
.

vn

=

1

u1 · v1···

u1 · vnum · v1···

um · vn

..
.

..
.

v1
M

¢

..
.

vn

=

M · v1RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR // P. Gelß // Low-rank tensor decompositions of quantum circuits // 03

21

M · vnMotivation

Tensor Decomposition

Quantum Circuits

Results

T ENSOR P RODUCT

1
1
1

1 1
1 0
1 0 11 1
0 0 0
1 11 0 1
0 1
1 1

1
1
1

=

w
v

T
S T
1 1 1
1
W
X
W X
U1 0 1V ¢ U0V
1 1 1
1

=

=

u

S

+

S

T
S T
1 0 1
0
W
X
W X
U0 0 0V ¢ U1V
1 0 1
0

S T S T S T
S T S T S T
S T S T S T
1
1
1
1
0
1
1
1
0
W X W X W X
W X W X W X
W X W X W X
U1V ¢ U0V ¢ U0V + U0V ¢ U1V ¢ U0V + U0V ¢ U0V ¢ U1V
1
1
1
1
0
1
1
1
0

S T S T
1
1
W X W XU1V U0V
1
1

}

~

u
w
w
w
w
w
w
w
w
v

¢

S T S T
1
0
W X W XU 0 V U0V
1
0
S T S T
0
1
W X W XU1V U0V
0
1

}








~

u
¢

w
w
w
w
w
w
w
w
v

S T
1
W X
U0V
1
S T
0
W X
U1V
0

}








~

RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR // P. Gelß // Low-rank tensor decompositions of quantum circuits // 04

Motivation

Tensor Decomposition

Quantum Circuits

M ATRIX P RODUCT S TATES AKA T ENSOR T RAINS

chemical reaction networks and catalytic systems

system identification

image classification

RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR // P. Gelß // Low-rank tensor decompositions of quantum circuits // 05

22

Results

Motivation

Tensor Decomposition

Quantum Circuits

Results

M ATRIX P RODUCT S TATES AKA T ENSOR T RAINSD EFINITIONA FFLECK , 1987 / O SELEDETS , 2009

A tensor T œ Cd1 ◊···◊dn is said to be in the MPS/TT format if

T=

r1
r0
ÿ
ÿ

k0 =1 k1 =1

···

rn
ÿ

kd =1

(1)
0

(2)
1

(n)
n≠1 ,:,kn

Tk ,:,k ¢ Tk ,:,k ¢ · · · ¢ Tk
1

2

The tensors T(1) , . . . , T(n) with T(i) œ Cri≠1 ◊di ◊ri are called cores and the numbers ri are bond dimensions or ranks. It holds that r0 = rn = 1 and ri Ø 1 for
i = 1, . . . , n ≠ 1.

Idea: represent high-dimensional systems, e.g., quantum registers,in MPS/TT format in order to mitigate the curse of dimensionality

RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR // P. Gelß // Low-rank tensor decompositions of quantum circuits // 06

Motivation

Tensor Decomposition

Quantum Circuits

M ATRIX P RODUCT S TATES AKA T ENSOR T RAINS• explicit notation
T=

r0
r1
ÿ
ÿ

k0 =1 k1 =1

···

rn
ÿ

kn =1

(1)
0

(2)
1

(n)
n≠1 ,:,kn

Tk ,:,k ¢ Tk ,:,k ¢ · · · ¢ Tk
1

2

≈∆
(n)
Tx1 ,x2 ,...,xn = T(1):,x1 ,: · . . . · T:,xn ,:• core notation
r

(1)

(1)

T1,:,1 · · · T1,:,r1z}u (n≠1)
} u
}
(n≠1)
(n)
(2)
(2)
T1,:,1 · · · T1,:,rn≠1
T1,:,1
T1,:,1 · · · T1,:,r2

w

w

w .
..
..
..
..
 ¢ ··· ¢ w
¢w

¢w
.
.
.
.
~
v
~ v
~
v ..
(n)
(2)
(2)
(n≠1)
(n≠1)
Trn≠1 ,:,1
Tr1 ,:,1 · · · Tr1 ,:,r2
Trn≠2 ,:,1 · · · Trn≠2 ,:,rn≠1u

RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR // P. Gelß // Low-rank tensor decompositions of quantum circuits // 07

23

Results

Motivation

Tensor Decomposition

Quantum Circuits

Results

M ATRIX P RODUCT S TATES AKA T ENSOR T RAINS• graphical notation¶ depict a tensor T œ Rd1 ◊···◊dn as a circle with n arms:···¶ tensor contraction:
matrix · vector :·matrix · matrix :·T T core · T T core :

·

=
=
=

¶ tensor-train representation:

RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR // P. Gelß // Low-rank tensor decompositions of quantum circuits // 08

Motivation

Tensor Decomposition

Quantum Circuits

Results

M ATRIX P RODUCT S TATES AKA T ENSOR T RAINS• conversion into TT format
Initial tensor

Isolate first mode

Apply SVD (M = U

VT)Isolate second mode

Apply SVD..
.
Tensor-train approximation

d1

d2

···

dn≠1

dn

d1

d 2 · . . . · dn

d1

d 2 · . . . · dn

d1
d1

d1

..
.

d2

d 3 · . . . · dnd2

d 3 · . . . · dnd2

RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR // P. Gelß // Low-rank tensor decompositions of quantum circuits // 09

24

dn≠1

dn

Motivation

Tensor Decomposition

Quantum Circuits

Results

M ATRIX P RODUCT S TATES AKA T ENSOR T RAINS• examples¶ |Φ± Í±¶ |Ψ Í

=Ô1 (|00Í ± |11Í)2=

Ô1 (|01Í ± |10Í)2¶ |GHZÍ =

¶ |WÍ

=

Ô1
2

r

|0Í

t |
z
|0Í
±|1Í ¢
|1Í

=

Ô1
2

r

|0Í

t |
z
|1Í
±|1Í ¢|0Í|tÔ1 (|0 . . . 0Í + |1 . . . 1Í)
2

r

t
z
|0Í
|1Í ¢

=

Ô1
2

=

Ô1 (|10 . . . 0Í + · · · + |0 . . . 01Í)
n

=

Ô1
n

|0Í

r

|1Í

t
|0Í
|0Í ¢
|1Í
z

|1Í

¢ ··· ¢

|

|

|0Í

t

|1Í

¢

|

|0Í
¢ ··· ¢
|0Í
|1Í

t

|

t

|

|1Í
|0Í

|0Í
¢
|0Í
|1Í

RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR // P. Gelß // Low-rank tensor decompositions of quantum circuits // 10

Motivation

Tensor Decomposition

Quantum Circuits

Results

M ATRIX P RODUCT S TATES AKA T ENSOR T RAINS• generative sampling

F ERRIS ET AL ., 2012 / H AN ET AL ., 2018◊n

œ C2 , then it is possible to-2
directly sample from the probability distribution given by Px1 ,...,xn = - x1 ,...,xn - /Z

¶ suppose the wave function is given in form of an MPS¶ Px1 ,...,xn =x1 ,...,xn

x1 ,...,xn≈∆

P = diag( )¶ conditional probabilities for sampling: P ik =

q

xi

k+1

,...,xim

PIyi ,...,yi
1

RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR // P. Gelß // Low-rank tensor decompositions of quantum circuits // 11

25

k≠1

, : ,xi

k+1 ,...,xim

Motivation

Tensor Decomposition

Quantum Circuits

Results

Q UANTUM C IRCUITS IN MPO F ORMAT• Q: How to represent quantum gates and circuits?

• A: Express them as matrix product operators.
uT(i) œ Cri≠1 ◊di ◊ri¿
G(i) œ Cri≠1 ◊di ◊di ◊ri

(i)

(i)

G1,:,:,1 · · · G1,:,:,ri

}

w

..
..
w

.
.
v
~
(i)
(i)
Gri≠1 ,:,:,1 · · · Gri≠1 ,:,:,ri

RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR // P. Gelß // Low-rank tensor decompositions of quantum circuits // 12

Motivation

Tensor Decomposition

Quantum Circuits

Results

Q UANTUM C IRCUITS IN MPO F ORMAT• MPO representations of controlled gates¶ controlled NOT gate (CNOT):S
1
W0W
CNOT = W
U0
0

0
1
0
0

0
0
0
1

T
0
C
0X
X≥ 1
X=
1V
0
0

D
C
0
0
¢I +
0
0

D
r
0
¢ ‡x = I ¢2 + C ¢ (‡x ≠ I) = I
1

z
C ¢

I
‡x ≠ I|t|t¶ controlled-controlled NOT gate (CCNOT):
S
1
W0
W
W
W0
W
W0
CCNOT = W
W0
W
W
W0
W
U0
0

0
1
0
0
0
0
0
0

0
0
1
0
0
0
0
0

0
0
0
1
0
0
0
0

0
0
0
0
1
0
0
0

0
0
0
0
0
1
0
0

0
0
0
0
0
0
0
1

T
0
0X
X
X
0X
X
r
0X
X≥
= I ¢3 + C ¢ C ¢ (‡x ≠ I) = I
0X
X
X
0X
X
1V
0

t
z
I
C ¢

RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR // P. Gelß // Low-rank tensor decompositions of quantum circuits // 13

26

C

|

¢

I
‡x ≠ IMotivation

Tensor Decomposition

Quantum Circuits

Results

Q UANTUM C IRCUITS IN MPO F ORMAT• MPO representations for multi-qubit systems¶ single-qubit gates (acting on n-qubit system):¶ controlled gates (e.g., CPHASE, CNOT):¶ controlled-controlled gates (e.g., CCNOT):RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR // P. Gelß // Low-rank tensor decompositions of quantum circuits // 14

Motivation

Tensor Decomposition

Quantum Circuits

Results

Q UANTUM C IRCUITS IN MPO F ORMAT• Simon’s algorithm¶ determine hidden bitstring b with f (x) = f (y) … x = y ü b

r

¶

G =

¶

G|0Í =

t

B

t
z
1r
|0Í
|+Í |≠Í ¢
4

|

A

t
z
I
B ¢

I

|

¢

A

u
I
w‡
w x
¢w
v
B

A

|

|0Í

¢

t

|+Í |≠Í

}

t

A

¢
B
I~
‡x

|+Í

| t |
r
B
I
¢
¢ A
A
‡x

u
|0Í
w|1Í
w
¢w
v
|≠Í
|

z
B ¢

t

I
‡x

|

with A = 12

C

}

D

1
1

and B = 12

t
| t |
t |

r
z
|+Í |≠Í|0Í|0Í¢¢ |+Í |≠Í ¢¢|0Í~|≠Í |+Í
|1Í
|1Í
|1Í

RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR // P. Gelß // Low-rank tensor decompositions of quantum circuits // 15

27

1
1

∆

C

1
≠1

D

≠1
1

b = 1010

Motivation

Tensor Decomposition

Quantum Circuits

Results

Q UANTUM C IRCUITS IN MPO F ORMAT• Quantum full adder¶ adds input qubits |AÍ, |BÍ, |Cin Í, produces sum |SÍ and carry-out qubit |Cout Í

¶ G =

r

‡x C0
C

1
with C0 =
0

I

‡x C1

z

u

C0
w
¢v 0
0

0
C1
C0
D

C1
C0
0

D
C
0
0
and C1 = C =
0
0

0
1

u
}
C1
0
wC
 w 0
0 ~¢w
v0
C1
0

}
0
t |
I
0

¢
‡x
C1 ~
C0

RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR // P. Gelß // Low-rank tensor decompositions of quantum circuits // 16

Motivation

Tensor Decomposition

Quantum Circuits

N UMERICAL R ESULTS• open-source tensor-train library for Python using NumPy and SciPy• simulation and analysis of systems with high-dimensional state spaces• reduce memory consumption and computational costs• includes model building, tensor-based solvers, and data-driven methods• possible application areas: Markovian master equations, nearest-neighborinteractions, nonlinear dynamical systems, quantum simulation, etc.

RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR // P. Gelß // Low-rank tensor decompositions of quantum circuits // 17

28

Results

Motivation

Tensor Decomposition

Quantum Circuits

Results

N UMERICAL R ESULTS• Quantum full adder network• construct network of quantum full adders by
concatenating MPOs:

r
z r
z r
z r
z
(1)
(2)
(3)
(4)
G = GQFA ¢ GQFA ¢ GQFA ¢ GQFA ¢ JIK ¢ · · · ¢ JIK
z r
z r
z r
z
r
(1)
(2)
(3)
(4)
· JIK ¢ JIK ¢ JIK ¢ GQFA ¢ GQFA ¢ GQFA ¢ GQFA ¢ JIK ¢ · · · ¢ JIK
·...

z r
z r
z r
z
r
(1)
(2)
(3)
(4)
· JIK ¢ · · · ¢ JIK ¢ GQFA ¢ GQFA ¢ GQFA ¢ GQFA

• define initial quantum state:=

C D C D C D C D C D C D C D C D C D
Ô1
Ô1
Ô1
Ô1
Ô1
Ô1
1
1
1
¢ 12 ¢ 12 ¢¢ 12 ¢ 12 ¢ . . .
¢ 12 ¢ 12 ¢
Ô
Ô
Ô
Ô
Ô
Ô
0
0
0
2
2
2
2
2
2

• compute final probability distribution of|S1 , . . . , Sn , Cout Í by generative sampling
on MPSs

RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR // P. Gelß // Low-rank tensor decompositions of quantum circuits // 18

Motivation

Tensor Decomposition

Quantum Circuits

N UMERICAL R ESULTS• compute 106 samples of the output
for different numbers of QFAs in
the network• same distributions as obtainedwith Qiskit• CPU time of MPO-based approach
depends only linearly on thenumber of QFAs

• for instance, for nQFA = 100, theMPO simulation needs about 30 s only

RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR // P. Gelß // Low-rank tensor decompositions of quantum circuits // 19

29

Results

Motivation

Tensor Decomposition

Quantum Circuits

Results

N UMERICAL R ESULTS• Quantum Fourier transform¶ map quantum states between computational and Fourier basisz
1 r
Gi = Ô I ¢(i≠1) ¢
2
¢

t

I
0

0
R2

|

tC

D

1
0

1
0

t

I
0

¢ ··· ¢

C

0
1

0
Rn≠i

D|

0
≠1

|

¢

t

I
Rn≠i+1|RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR // P. Gelß // Low-rank tensor decompositions of quantum circuits // 20

Motivation

Tensor Decomposition

Quantum Circuits

Results

N UMERICAL R ESULTS• Shor’s algorithm• given M œ N, choose 1 < a < M (coprime)• initialize input register with 2n qubits and target
register with n qubits, where 2n > M• apply Hadamard gates and modular exponentiation circuit Uf with f (x) = ax mod M• use inverse QFT to calculate period of f andfind factors of M
• consider M = 15, a œ {2, 4, 7, 8, 11, 13, 14}:a

¶ Uf can be constructed as MPO with ranks
bounded by either 2 or 4

2, 7, 8, 13¶ orthonormalize MPO cores between
applications of QFT≠1 gate groups

4, 11, 14

y
0
64
128
192
0
128

RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR // P. Gelß // Low-rank tensor decompositions of quantum circuits // 21

30

p
1
4
2
4
1
2

(M1 , M2 )?
(3, 5)(3, 1)(3, 5)?
(3, 1)

Thanks for your attention
Links
Scikit-TT on GitHub: http://github.com/PGelss/scikit_tt

Collaborations
Dr. Stefan Klus, Department of Mathematics, University of Surrey
Zarin Shakibaei, Telekom Innovation Laboratories, TU Berlin
Prof. Sebastian Pokutta, AI in Society, Science, and Technology, ZIB

Publications
P. Gelß, S. Klus, Z. Shakibaei, S. Pokutta. Low-rank tensor decompositions of quantum circuits, in submission
F. Nüske, P. Gelß, S. Klus, C. Clementi. Tensor-based computation of metastable and coherent sets, Physica D, 2021
P. Gelß, S. Klus, J. Eisert, C. Schütte. Multidimensional approximation of nonlinear dynamical systems, J. Comput. Nonlinear Dynam., 2019
S. Klus, P. Gelß, S. Peitz, C. Schütte. Tensor-based dynamic mode decomposition, Nonlinearity, 2018
P. Gelß, S. Klus, S. Matera, C. Schütte. Nearest-neighbor interaction systems in the tensor-train format, J. Comput. Phys., 201731

The 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop on
Advances in Classical and Quantum Algorithms for
Optimization and Machine Learning
September 16th - 19th, 2022, Tokyo (The university of Tokyo), Japan,
and September 21st - 22nd, 2022, Fukuoka (Kyushu University), Japan

NEC’s Quantum Computing
Technologies
Shintaro MOMOSE
Quantum Computing Business Department
Advanced Platform Division
NEC Corporation
s-momoseak@nec.com
This presentation consists of two parts, discussing SX-Aurora TSUBASA vector
supercomputer and introducing digital annealer working on SX-Aurora TSUBASA
called Aurora Vector Annealer. The first half of the presentation shows the vector
architecture of SX-Aurora TSUBASA, especially its latest vector processors having thehighest-level memory bandwidth. Sustained performance and power efficiency are also
discussed, as well as NEC's future plans and roadmap. The second half of thepresentation shows NEC's quantum computing strategies and their products to provide
higher sustained performance in the annealing/optimization fields. NEC developed the
Aurora Vector Annealer as a digital annealer and has a strong business relationship with
D-Wave providing a quantum annealer. NEC aims at solving various social issues byusing the quantum/digital annealing technologies and by developing a hybrid platform
with supercomputer and quantum/digital annealer to provide much higher sustained
performance.

33

The 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop
September 17th, 2022. Tokyo Japan

NEC’s Quantum Computing Technologies

Shintaro MOMOSE, Ph.D. (Director)
Quantum Computing Business Department
Advanced Platform Division
NEC Corporation© NEC Corporation 2021
2022

Contents
 NEC’s Strategy for Quantum Computing Vector Annealing on SX-Aurora TSUBASA

 Case Study

2© NEC Corporation 2022

34

Architecture of SX-Aurora TSUBASASX-Aurora TSUBASAControlling

Processing

VectorEngineStandard
X86/Linux
Server

X86
CPU

Linux
OS

-Vector Processor-1.5TB/s Memory Access-Low Power 200W

VE
tools

VE
OS
Library

Vector
CPU

APP
Tools

VE
library

Vectorcompiler© NEC Corporation 2022

VE20 Processor
VE20 Specifications
Processor Version

Type 20A

Type 20B

Cores/processor

108Cache Function

Software Controllable

Memory capacity

48GB

Memory bandwidth

1.53TB/s

Power

HBM2

HBM2

~300W (TDP)~200W (Application)© NEC Corporation 2022

35

core

core

core

core

core

core

core

core

HBM2 I/F

3TB/s

LLC 8MB

Cache bandwidth

core

HBM2

HBM2 I/F

16MB

core

HBM2

HBM2 I/F

Cache capacity

HBM2

HBM2 I/F

2.45TF (DP)4.91TF (SP)LLC 8MB

3.07TF (DP)6.14TF (SP)HBM2 I/F

Processor
performance

HBM2 I/F

Core performance

307GF (DP)614GF (SP)HBM2

Why is NEC Focusing on Quantum Computing?
 Both HPC and Quantum technologies will be used for higher sustained performance NEC develops HPC, Simulated Annealing on Aurora, Quantum Annealer and Quantum Gate
For higher sustained performance,
NEC continuously combine HPC andnew cutting edge technologies

HPC

 Step1: Annealing on HPC resource

System Sustained Performance

Tight coupling of HPC and QC
to provide higher sustained performance

QC Vector Annealing on SX-Aurora TSUBASA
 Using Quantum Annealer to accelerate Step2: HPC/QC Hybrid Computing

 Tight coupled HPC/QC hybrid system to reach higher
sustained performance in scientific/industrial fields

 Step3: Introducing “QC Gate” as new era
 NISC type: Around 2030
 FTQC type: Around 2050

NISQ
Noisy Intermediate-Scale QC
FTQC
Fault Tolerant QC

SX-Aurora TSUBASA
2022

6© NEC Corporation 2021

step1

step2

2030

step3

2050

NEC Group Internal Use Only

NEC’s Initiative in Quantum Computing
Since succeeding in the world’s first demonstration of solid -state qubit operation,
NEC has been working towards the social implementation of quantum computing.

2030
1999 World’s 1st
Demonstrated
solid-state qubit
operation

Aiming to provide models for solving
issues using quantum annealing andquantum gate

2020-2021
 Collaboration with D-Wave Aurora Vector Annealing
on SX-Aurora TSUBASA(Published in Nature*1)

2040～
MOONSHOT program to achievepractical application of gate-based
method2023
Aim to obtain extraordinary
long coherence time on
quantum annealing machine※2*1: Y. Nakamura et al., Nature 398, 786 (1999)*2; Based on results obtained from a project commissioned by the New Energy and Industrial Technology Development Organization (NEDO).
7© NEC Corporation 2022

36

NEC is leading the development of quantum annealing
devices to enable practical use of superconducting
quantum annealing machine in 2023
Quantum annealing device that
NEC has been working on

Superconducting quantum annealing machine (mockup)
External
deviceControlled input

Read out(The heart part of the machine)

External
deviceNormal temperature～K
mK～ mK (≒-273℃)* Based on results obtained from a project commissioned by
the New Energy and Industrial Technology Development Organization (NEDO).

8

SupportParticipating organizations

NEDO

NEC (Deputy rep. facilities responsible for operationalization and practical application)
AIST (Rep.), Tokyo Institute of Technology, Waseda Univ.,
Yokohama National Univ., and others© NEC Corporation 2022

In addition to focusing on the quantum annealing method to address
society’s optimization needs, NEC is also promoting research anddevelopment toward practical application of the gate -based method.

Quantum Computing(Broadly defined to include quantum behavior)

Annealing method, etc.

Solves combinatorial optimization problems by means of
the Ising model or other statistical physics model
Quantum Annealing
Superconducting
Circuits

product
9
9

R&D

©
2022
© NEC
NECCorporation
Corporation
2021

OpticalParametric
Oscillator

Digital Circuits

AIST

D-WaveQuantum gate method

Performs calculations by replacingclassical computer bits with qubits

Hitachi Fujitsu Toshiba

RIKEN
UTokyo

NTT

IBM

R&D

product*Based on NEC’s survey. （Due to limited space, not all institutions are covered.)37

GoogleRigettiIntelVector Annealing on SX-Aurora TSUBASA
11© NEC Corporation 2022

NEC Vector Annealing

VA Performance is provided by:Matrix operation acceleration by VE, large and fast memory, and optimized algorithm for VE
Vector operation on VE

Avoiding Redundant Search andOptimized algorithm for VE

Energy calculation is matrix operation

Existing search

Qij
SpinStatex+Effect from neighbor spin

0 or 1

Full connect 100k qbits/VE andhigh memory bandwidth

Problem

Problem
Formalize

Formalize

 48GB memory capacity and
1.5TB/s memory bandwidth

Search Multi card supports larger numberof qbits (100k qbits x n)1/2
13

skip constraint violations

Check constraint© NEC Corporation 2022

38

Violation→try again

Search considering
constraint

Constraintx

VA search

Including constraint violations

betterxfor(i=0; i<numSpins; i++)
DeltaEnergy +=Qij[FlipSpin][i] x SpinState[i]computational complexity
reduction

Case Study
16© NEC Corporation 2022

Solving Social Issues Using Quantum Computing
NEC is trying to apply QC technologies for practical use with partners

Development with Co-creation Partners
SMBC Group/ JRI / NEC Platforms / NEC Fielding etc.
Advertisement
Infrastructure• Matching/Recommendation• Com. base station• Surveillance sensor

Manufacturing• Production plan
• Parts ordering plan

Traffic/Logistics

Financial• Card fraud
detection• Crew shift• Delivery plan
• Load placement

Material/Drug• Screening• Monte Carlosimulation• Experimental• Risk calculation• parameter search※研究中、顧客実証～実用に至るものが含まれています

Leap Quantum Cloud
Service17

NEC Vector Annealing
Service© NEC Corporation 2022

39

Real Time TSUNAMI Disaster Simulation &Real Time Proposing Optimal Evacuation Routes“Next Generation Supercomputing Platform assisted by Quantum Annealing”
R&D with Tohoku University
Tsunami simulation

earthquake

Evacuation route

Disaster simulation

Supercomputing

Annealing

Tohoku University
Information Science

Prof. Kobayashi, Prof. Ozeki

Disaster Science

Prof. Koshimura
Associate Prof. Erick

Riken

Associate Prof. Ota

18© NEC Corporation 2022

Evacuation Route (Annealing)

Use Case: Production Planning Optimization

Optimizing complex planning for multi-product manufacturing lines
Higher versatile processing equipment needs
highly optimized product planning for higher efficiency• Switching processed product makes idling time of equipment• Production planning can reduce the idling time and also reduce humanresources
Independent
processing

Line
1

A

Line
2

D

Processing
using same tool

C

A→C

D→E

E

Overlapping setup time
23

switching

E→F

Line
1

B

C→B

F

F→G

Line
2

G

C & E use same tool

© NEC Corporation 2022

40

A
G

A→C

G→D

D

C
D→E

C→B

E

B
E→F

Not only shorter processing time,
but also reduce human resources

F

Use Case: Delivery Route and Schedule Optimization
for reducing costs, time, energy, CO2, etc.
delivery route

Customer
9:00

Office13:15

Arrive at
13:00

Arrive at 8:50
Arrive at
9:50
10:00

EX.

Delivery of parts anddispatch of Engineers

delivery schedule• Parts are delivered by truck• Engineers move by car/train• Have to consider skills of each engineer
24© NEC Corporation 2022

41

14:30

Arrive at
14:00

Factory

The 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop on
Advances in Classical and Quantum Algorithms for Optimization andMachine Learning
September 16th - 19th, 2022, Tokyo (The university of Tokyo), Japan,
and September 21st - 22nd, 2022, Fukuoka (Kyushu University), Japan

Pairwise Conditional
Gradients without Swap Steps andSparser Kernel Herding
Kazuma Tsuji
MUFG Bank, Ltd., Tokyo, Japan
takahashi.shota@ism.ac.jp
The Pairwise Conditional Gradients (PCG) algorithm [1] is a powerful extension of the FrankWolfe algorithm leading to particularly sparse solutions, which makes PCG very appealing forproblems such as sparse signal recovery, sparse regression, and kernel herding. Unfortunately,
PCG exhibits so-called swap steps that might not provide sufficient primal progress. The
number of these bad steps is bounded by a function in the dimension and as such known
guarantees do not generalize to the infinite-dimensional case, which would be needed forkernel herding. We propose a new variant of PCG, the so-called Blended Pairwise Conditional
Gradients (BPCG) which is a combination of Blended Conditional Gradients [2] and PCG, and
BPCG does not exhibit swap steps. The convergence rate of BPCG is basically that of PCG if
no drop steps would occur and as such is no worse than PCG but improves and provides new
rates in many cases. Moreover, we observe in the numerical experiments that BPCG’s solutions
are much sparser than those of PCG. We apply BPCG to the kernel herding setting, where wederive nice quadrature rules and provide numerical results demonstrating the performance ofour method.

References[1] S. Lacoste-Julien and M. Jaggi. On the global linear convergence of Frank-Wolfe optimization variants. In
C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural
Information Processing Systems 28, pages 496–504. Curran Associates, Inc., 2015. URL
http://papers.nips.cc/paper/ 5925-on-the-global-linear-convergence-of-frank-wolfe-optimi pdf.
[2] G. Braun, S. Pokutta, D. Tu, and S. Wright. Blended conditional gradients: the unconditioning ofconditional gradients. In Proceedings of the 36th International Conference on Machine Learning (PMLR),
volume 97, pages 735–743, 2019.

43

Pairwise Conditional Gradients without Swap Stepsand Sparser Kernel Herding
Kazuma Tsuji (MUFG bank)Coresearchers:
Ken’ichiro Tanaka (The University of Tokyo, PRESTO)
Sebastian Pokutta (AISST, ZIB)

2022/09/17

Outline of today’s talk

The main topic of today’s talk is Conditional Gradients methods.
We propose a new variant of Conditional Gradients which is called
Blended Pairwise Conditional Gradients (BPCG).
BPCG algorithm works well in high dimensional cases and outputs
highly sparse solutions practically.
The contents of today’s talk are written in Tsuji et al. (2022) in detail.

2 / 2844

CG algorithm
Conditional Gradients (Levitin and Polyak, 1966) are in an important classof first-order methods for constrained convex minimization, i.e., solving
min f (x)xœC(f : convex, C µ Rd : convex compact region).

CG algorithm is an iterative first-order method.
The solution of CG algorithm is represented as a convex combinationof the vertices of C:
›t =
.

n
ÿ
i=1

ci vi

A

{vi }ni=1 µ VC , C = conv(VC ),

n
ÿ
i=1

B

ci = 13 / 28
Algorithm
12
3

wi = argmaxvœVC È≠Òf (›i ), vÍ(C = conv(VC ))determine the step-size –i (0 Æ –i Æ 1)

›i+1 = ›i + –i (wi ≠ ›i ) = (1 ≠ –i )›i + –i wi

Figure: CG Algorithm

4 / 2845

Variants of CG algorithm

There are many variants of CG algorithm to achieve better performance(faster convergence, computational efficiency, sparser solutions, etc.).
We explain the following two variants of CG algorithm:
Pairwise CG algorithm (Lacoste-Julien and Jaggi, 2015)
Blended CG algorithm (Braun et al., 2019)5 / 28Pairwise CG method
Pairwise CG method (Lacoste-Julien and Jaggi, 2015) uses the direction
wt ≠ at instead of wt ≠ ›t for the update of current solutions:dt = (wt ≠ ›t ) + (›t ≠ at ) = wt ≠ at (Pairwise direction)
¸

˚˙

FW

˝

›t+1 = ›t + –t dt
A

¸

˚˙

Away

˝

wt = argmaxvœVC Èv, ≠Òf (›t )Í
at = argminvœSt Èv, ≠Òf (›t )Í (›t œ conv(St ) µ VC )St : vertices set that construct the convex combination of ›t

Figure: Pairwise CG
46

B

6 / 28

Blended CG
Blended Conditional Gradients (Braun, Pokutta, Tu, and Wright, 2019)qq›t = ki=1 ci vi ( ki=1 ci = 1, c1 , . . . , ck Ø 0, St = {vi }ki=1 µ VC ).
at Ω argminvœSt È≠Òf (›t ), vÍ , st Ω argmaxvœSt È≠Òf (›t ), vÍ
wt Ω argmaxvœVC È≠Òf (›t ), vÍ
A = ÈÒf (›t ), at ≠ st Í (local pairwise gap) : upper bound of local error f (›t ) ≠ minxœconv(St ) f (x)B = ÈÒf (›t ), ›t ≠ wt Í (dual gap) : upper bound of global error
f (›t ) ≠ minxœC f (x)Algorithm
1. A Ø B
optimize the convex coefficients {ci }ki=1 by SiGD which is an optimization method on a simplex.
2. A < B›t+1 = ›t + –t (wt ≠ ›t ), St+1 Ω St ﬁ {wt } (vanilla CG update).
7 / 28

Algorithm Blended Conditional Gradientsfor t = 0 to T ≠ 1 doat Ω argminvœSt È≠Òf (›t ), vÍst Ω argmaxvœSt È≠Òf (›t ), vÍ
wt Ω argmaxvœVC È≠Òf (›t ), vÍ
if ÈÒf (›t ), at ≠ st Í Ø ÈÒf (›t ), ›t ≠ wt Í then
optimize the convex coefficients {ci }ki=1 .
else›t+1 = ›t + –t (wt ≠ ›t ) {FW step}end ifend forBy the structure of algorithm, new vertices are added only whenconvex coefficients are sufficiently optimized. Therefore, BCG outputs
highly sparse solutions.
8 / 2847

Convergence speed of PCG and BCG
PCG and BCG achieve faster convergence rates than CG algorithm:
Table: Theoretical convergence rates (finite-dimensional cases)

CG
PCG
BCG

L-smoothO( T1 )O( T1 )O( T1 )

Strongly convex and polytopeO( T1 )
exp(≠cP T )exp(≠cB T )However, both algorithms suffer in high-dimensional cases. In particular, wecannot guarantee convergence in infinite-dimensional cases !

9 / 28Bottleneck of BCG and PCG
PCG
Swap Step:

Swap Step means the step in which at is swapped by wt .
Swap steps affect theoretical analysis and a dimension-dependent
constant appears in convergence rates.

BCG
Simplex Gradient Descent (SiGD):

SiGD is the coefficients optimization method in BCG.
SiGD is the optimization method on the simplex
JIk-ÿ
kci = 1, ci Ø 0 .
(c1 , . . . , ck ) œ R i=1

The convergence rate of SiGD includes the dimension of polytope andtherefore BCG includes a dimension-dependent term in convergence rate.

10 / 2848

BPCG algorithm (proposed algorithm)
We propose the following BPCG algorithm. The framework uses that of
BCG and the difference is the local Pairwise step.
Algorithm Blended Pairwise Conditional Gradientsfor t = 0 to T ≠ 1 doat Ω argminvœSt È≠Òf (›t ), vÍst Ω argmaxvœSt È≠Òf (›t ), vÍ
wt Ω argmaxvœVC È≠Òf (›t ), vÍ
if ÈÒf (›t ), at ≠ st Í Ø ÈÒf (›t ), ›t ≠ wt Í then›t+1 = ›t + –t (st ≠ at ) {local pairwise step}else›t+1 = ›t + –t (wt ≠ ›t ) {FW step}end ifend for
The moving direction of BPCG is dt = st ≠ at or dt = wt ≠ ›t .

11 / 28

Local pairwise steps
In local Pairwise steps, the direcionA

dt = st ≠ at

B

at = argmin È≠Òf (›t ), vÍ , st = argmax È≠Òf (›t ), vÍ
vœSt

vœSt

is used.
Properties of local pairwise steps:
By the definition of st and at , local pairwise updates are equivalent to
the implementation of the PCG over St .
Only the two coefficients that correspond to st and at are changed.
We do not mind swap steps in local pairwise steps.
12 / 2849

Analysis of BPCG for L-smooth functions
Roughly speaking, for L-smooth convex functions, we haveÈÒf (›t ), dt Í2,
ht ≠ ht+1 Ø
2LD2
where ht = f (›t ) ≠ f (› ú ) and dt is the moving direction of BPCG.
Case(A) ÈÒf (›t ), at ≠ st Í Ø ÈÒf (›t ), ›t ≠ vt ÍÈÒf (›t ), ›t ≠ vt Í2h2t
ÈÒf (›t ), st ≠ at Í2ØØ
ht ≠ht+1 Ø
2LD2
2LD2
2LD2
Case(B) ÈÒf (›t ), at ≠ st Í Æ ÈÒf (›t ), ›t ≠ vt Í
ht ≠ ht+1 ØÈÒf (›t ), ›t ≠ vt Í2
h2t
Ø
2LD2
2LD2
13 / 28

Theoretical analysis: general smooth case
TheoremP : convex feasible domain with diameter D (dim P can be Œ)
f : convex and L-smooth.
Let {›i }Ti=0 µ P be the sequence given by the BPCG algorithm. Then, it
holds thatf (›T ) ≠ f (› ú ) Æ

4LD2
.
T

Since the constant factor 4LD2 does not depend on the dimension of thedomain, we can apply this result to infinite-dimensional cases!

14 / 2850

Analysis of BPCG for strongly convex functions
Roughly speaking, for L-smooth convex functions, we haveÈÒf (›t ), dt Í2
ht ≠ ht+1 Ø
2LD2. We use the following two Lemmas:

Lemma (Lacoste-Julien and Jaggi (2015), Inequalities (23) and (28) )Assume that f is µ-strongly convex and P is a polytope with pyramidal
width ”. Then,ÈÒf (›t ), at ≠ wt Í2
ht Æ
.
2µ” 2

Lemma
For each step t, an inequality 2ÈÒf (›t ), dt Í Ø ÈÒf (›t ), at ≠ wt Í holds.
15 / 28

Theoretical analysis: polytopes and strongly convex case

TheoremP : finite-dimensional polytope with pyramidal width ” and diameter D
f : µ-strongly convex and L-smoothConsider the sequence {›i }Ti=0 µ P obtained by the BPCG algorithm. Then,it holds thatf (›T ) ≠ f (› ú ) Æ (f (›0 ) ≠ f (› ú )) exp (≠cf,P T ) ,2µ”
where cf,P := 12 min{ 12 , 4LD
2 }.

16 / 2851

Compare the constant factor in convergece rate
The convergence rate for finite-dimensional polytope case:
BPCG
exp(≠cf T )cf =1 µ” 21min{ ,}
2
2 4LD2PCG
exp(≠ck(T ))k(T ) Ø T /(3|VC |! + 1)
BPCG bounds the constant factor better than PCG.

17 / 28

Compare BPCG to other variants
BPCG ensures O( T1 ) convergence in infinite-dimensional cases.

BPCG ensures linear convergence for strongly convex and polytope
cases.
Moreover, BPCG outputs highly sparse solutions since BPCG inherits
the framewrok of BCG.
Table: Theoretical convergence rate

CG
PCG
BCG
BPCG

L-smooth
infinite-dimensional domain
O( T1 )
7
7
O( T1 )

Strongly convex,
finite-dimensional polytopeO( T1 )
exp(≠cP T )exp(≠cB T )exp(≠cBP T )18 / 2852

Lazified Version of BPCGIn BPCG, we need to compute the dual gap
max ÈÒf (›t ), ›t ≠ vÍvœVCin each iteration and we need |VC | times access.
To reduce computational cost, we employ the lazification techique (Braunet al., 2017).
The lazification means the follwoing estimation
Φt ¥ max ÈÒf (›t ), ›t ≠ vÍvœVC

19 / 28

Lazified Version of BPCG
Algorithm Lazified Blended Pairwise Conditional Gradientsfor t = 0 to T ≠ 1 doat Ω argminvœSt È≠Òf (›t ), vÍst Ω argmaxvœSt È≠Òf (›t ), vÍ
wt Ω argmaxvœVC È≠Òf (›t ), vÍ
if ÈÒf (›t ), at ≠ st Í Ø Φt then›t+1 = ›t + –t (st ≠ at ) {local Pairwise step}else
if ÈÒf (›t ), ›t ≠ wt Í Ø Φt /J then›t+1 = ›t + –t (wt ≠ ›t ) {FW step}else
Φt+1 = Φt /2
end ifend ifend for20 / 28
53

Theoretical analysis: general smooth case
TheoremP : convex feasible domain with diameter Df : convex and L-smooth.
{›i }Ti=0 µ P : output of the Lazified BPCG algorithm .

Case (A) If f is µ-strongly convex and P is a polytope with pyramidal
width ” > 0, we havef (›t ) ≠ f (› ú ) = O (exp (≠c T ))(T æ Œ)for a constant c > 0 independent of T .
Case (B) If f is only convex and L-smooth, we have3

1
f (›T ) ≠ f (› ) = O
Tú

4(T æ Œ).
21 / 28

Numerical experiments for finite-dimensional problems
We confirm the effectiveness of BPCG through numerical experiments.
BPCG and Lazified BPCG are compared with CG, ACG(Wolfe, 1970) and
PCG.
Problem 1 : Convex optimization over probability simplex
min Îx ≠ x0 Î22xœRnHere, ∆(n) := {x œ Rn |x0 œ ∆(n).

qn

s.t. x œ ∆(n),i=1 xi = 1, xi Ø 0 (i = 1, . . . , n)} andProblem 2 : ¸p norm ballmin Îx ≠ x0 Î22xœRnÎ · Îp means the p norm.

s.t. ÎxÎp Æ 1

22 / 2854

Problem 1 (n = 200) : iterations and computational time

Figure: Problem 1 (x-axis: left: iterations , right: computational time
y-axis: top: primal gap, bottom: dual gap)

Convergence speed of BPCG
Iterations: Competitive with PCG.
Computational time: BPCG is the fastest.

23 / 28

Problem 1 (n = 500) : sparsity of solutions

Figure: Problem 1: Convergence of the primal gap for the number of vertices thatare the members of convex combination of a solution.

BPCG and Lazified BPCG output sparse solutions.
24 / 2855

Problem 2 (p = 5, n = 1000) : sparsity of solutions

Figure: Problem 2: Convergence of primal gap for the number of vertices that are
the members of convex combination of a solution.

BPCG and Lazified BPCG output much sparser solutions.
25 / 28

Numerical experiments (Kernel Herding)P(Ω) : all probability measures on Ω œdMMD(·, ·) : distance between probability measures measured in a
Reproducing Kernel Hilbert Space (RKHS) on Ω
Kernel Herding solves the following minimization problem overinfinite-dimensional domain P(Ω) using a CG manner:argmin MMD2 (µ, ›)›œP( )(µ œ P(Ω)).

The output of Kernel Herding is a discrete measure
›=

n
ÿ
i=1

Ê i ” xi({Êi }ni=1 µ R, {xi }di=1 µ Rd ).

Using an efficient CG method, we want to derive › that approximates µ withsmall number of nodes n. That is, we want to derive nice sparse soltions.
26 / 2856

BPCG for kernel herding
Domain : Ω = [≠1, 1]2 , Kernel : Matérn kernel with ‹ = 32 , 52 .
5
7
Optimal rates of the convergence of MMD is n≠ 4 , n≠ 4 , respectively.

Figure: Matérn kernel (‹ = 3/2) (left) and Matérn kernel (‹ = 5/2) (right)

27 / 28

Summary

To overcome the difficulties of PCG and BCG, we proposed the BPCG
algorithm.

We showed that for BPCG we can ensure O( T1 ) convergence even if
the dimension of convex constraints is infinite. For strongly convex andpolytope cases, we can guarantee the linear rate.
Through numerical experiments, we showed the practical effectivenessof BPCG. In particular, sparsity of solutions is notable.

28 / 28
57

Reference I
G. Braun, S. Pokutta, and D. Zink. Lazifying conditional gradientalgorithms. In Proceedings of the 34th International Conference on
Machine Learning, pages 566–575, 2017.
G. Braun, S. Pokutta, D. Tu, and S. Wright. Blended conditional gradients:
the unconditioning of conditional gradients. In Proceedings of the 36th
International Conference on Machine Learning (PMLR), volume 97, pages
735–743, 2019.

S. Lacoste-Julien and M. Jaggi. On the global linear convergence ofFrank-Wolfe optimization variants. In C. Cortes, N. D. Lawrence, D. D.
Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural
Information Processing Systems 28, pages 496–504. Curran Associates,
Inc., 2015. URL http://papers.nips.cc/paper/
5925-on-the-global-linear-convergence-of-frank-wolfe-optimization-var
pdf.
29 / 28

Reference II

E. S. Levitin and B. T. Polyak. Constrained minimization methods. USSR
Computational Mathematics and Mathematical Physics, 6(5):1–50, 1966.
K. K. Tsuji, K. Tanaka, and S. Pokutta. Pairwise conditional gradients
without swap steps and sparser kernel herding. In International
Conference on Machine Learning, pages 21864–21883. PMLR, 2022.
P. Wolfe. Convergence theory in nonlinear programming. In Integer andNonlinear Programming, pages 1–36. North-Holland, Amsterdam, 1970.

30 / 28
58

The 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop on
Advances in Classical and Quantum Algorithms for
Optimization and Machine Learning
September 16th - 19th, 2022, Tokyo (The university of Tokyo), Japan,
and September 21st - 22nd, 2022, Fukuoka (Kyushu University), Japan

Proofs in Extremal Combinatorics
through Optimization
Christoph SPIEGEL
Zuse Institute Berlin, Germany
spiegel@zib.de
We present a fully computer-assisted proof system for solving a particular family ofproblems in Extremal Combinatorics. Existing techniques using Flag Algebras have
proven powerful in the past, but have so far lacked a computational counterpart to derivematching constructive bounds. We demonstrate that common search heuristics arecapable of finding constructions far beyond the reach of human intuition. Additionally,
the most obvious downside of such heuristics, namely a missing guarantee of global
optimality, can often be fully eliminated in this case through lower bounds and stability
results coming from the Flag Algebra approach.
To illustrate the potential of this approach, we study two related and well-known
problems in Extremal Graph Theory that go back to questions of Erdős from the 60s.
Most notably, we present the first major improvement in the upper bound of the Ramsey
multiplicity of K₄ in 25 years, precisely determine the first off-diagonal Ramsey
multiplicity number, and settle the minimum number of independent sets of size four ingraphs with clique number strictly less than five.

59

Proofs in Extremal
Combinatorics throughOptimization
6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop
Christoph Spiegel
17th of September 2022

Results are joint work with...

Olaf Parczyk

Freie Universität Berlin

Sebastian Pokutta
Zuse Institute Berlin

Tibor Szabó

Freie Universität Berlin

Research partially funded through Math+ project EF1-12

60

Proofs in Combinatorics through Optimization

1. The Ramsey Multiplicity Problem
2. Search Heuristics for Upper Bounds
3. Flag Algebras for Lower Bounds
4. A Related Problem

1. The Ramsey Multiplicity Problem

The Ramsey Multiplicity of triangles
Theorem (Ramsey 1930)

For any t œ N there exists R(t) œ N such that any 2-edge-coloring of the completegraph of order at least R(t) contains a monochromatic clique of size t.
A well-known question: Can we determine R(t)?
A related question: How many cliques do we need to have? That means, letting
kt (G) denote the fraction of all possible t-cliques in G, what is
ct = lim min{kt (G) + kt (G) : G graph of order n}?
næŒ

Theorem (Goodman 1959)

c3 = 1/4.

æSame as Erdős-Rényi
random graph!

61

æConjecture (Erdős 1962)

ct = 21≠(2) .
t

1. The Ramsey Multiplicity Problem

Ramsey Multiplicity beyond triangles
Theorem (Thomason 1989 / 1997)c4

Æ 0.970 · 2≠5 and c

5

Theorem (Even-Zohar and Linial ’15)

Æ 0.881 · 2≠9 .

c4 Æ 0.969 · 2≠5 .

Erdős conjecture was false! But what about lower bounds?
Theorem (Giraud 1976)c4

Ø 0.695 · 2≠5 .

Theorem (Sperfeld / Nieß’11)c4

Ø 0.914 · 2≠5 .

Theorem (Grzesik et al. ’20)

c4 Ø 0.947 · 2≠5 .

Both the best upper and lower bounds heavily rely on computer-assistence!
Theorem (Parczyk, Pokutta, S., and Szabó 2022+)

c4 Æ 0.964 · 2≠5 and 0.780 · 2≠9 Æ c5 Æ 0.874 · 2≠9 .
How can we use Optimization to formulate mathematical proofs?

Proofs in Combinatorics through Optimization

1. The Ramsey Multiplicity Problem
2. Search Heuristics for Upper Bounds
3. Flag Algebras for Lower Bounds
4. A Related Problem

62

2. Search Heuristics for Upper Bounds

Graph blow-ups

We want constructive bounds that are ‘finitely describable’. Random graphs are one
source for such constructions. Another natural deterministic one are graph blow-ups.
Definition

The m-fold blow-up C [m] of a graph C is given by replacing each vertex in C with anindependent set of size m. Two vertices are adjacent if the originals were.
Using blow-ups, we can derive an upper bounds for ct from any graph C through
!

"

!

"

ct Æ lim kt C [m] + kt C [m] .
mæŒThis is in fact efficiently computable since
lim kt (C [m]) = nt kt (C ) /nt

mæŒ

and

lim kt (C [m]) =

mæŒ

t
ÿ
j=1

(1)

S(t, j)nj kj (C )/nt . (2)2. Search Heuristics for Upper Bounds

Constructing graphs through search heuristics
1

2

)
*
n
For fixed n and s œ {0, 1}(2) let Cs = [n], ij : i < j, s(j≠1)+i = 1 and consider
s
ÿ

min
n
sœ{0,1}(2) j=1

2

S(t, j)nj kj (Cs ) nt kt (Cs )
+
.
nt
nt

So we have found our optimization problem! How to solve it?
For n / 40 we can use Search Heuristics.
Unfortunately even n = 40 is much too small for c4 and c5 , barely disproving Erdős’original conjecture. Can we use combinatorial insights to bias the search space?

63

2. Search Heuristics for Upper Bounds

Constructing Cayley graphs through search heuristics
Thomason’s constructions are based on computing the values of XOR-graph-products.
The results are in fact Cayley graphs in C3◊2 ◊ C2◊5 and C3 ◊ C2◊6 .
Definition

Given an abelian group G and set S ™ G ı satisfying S ≠1 = S, the associated Cayley
graph has vertex set G and g1 , g2 œ G are adjacent if and only if g1≠1 g2 œ S.
Idea. Why not directly search Cayley graph constructions?
The binary vector s now represents the generating set S. Since |G|/2 < |S| < |G| thenumber of variables is therefore linear (instead of quadratic) in the number of vertices!
The groups C3 ◊ C2◊8 and C3 ◊ C2◊6 give the improved upper bounds for c4 and c5 .

Proofs in Combinatorics through Optimization

1. The Ramsey Multiplicity Problem
2. Search Heuristics for Upper Bounds
3. Flag Algebras for Lower Bounds
4. A Related Problem

64

3. Flag Algebras for Lower Bounds

A trivial computational lower bound
The Flag Algebra SDP approach can be seen as (i) a formalized Cauchy-Schwarz-typeargument and (ii) an improvement over a trivial computational lower bound.
Let dH (G) denote the probability that v (H) vertices chosen uniformly at random in G
induce a copy of H. Writing ct (G) = kt (G) + kt (G), basic double counting gives us
ct (G) =ÿH graphv (H)=NdH (G) ct (H)(3)

for t Æ N Æ v (G). For any N Ø t this implies a trivial lower bound ofct Ø min ct (H).
H graphv (H)=N(4)

3. Flag Algebras for Lower Bounds

The Flag Algebras SDP approach
Razborov (2007) introduced Flag Algebras in order to study this type of problem. One
important observation is that for any Q ≤ 0 the coefficients aH = ÈQ, DH Í satisfyÿ

H graphv (H)=NdH (G) aH Æ O(1/v (G))(5)

for any graph G. Through (3) this implies the (hopefully improved) boundct Ø min ct (H) ≠ aH .
H graphv (H)=N

This approach gives the best current lower bounds for c4 and c5 . The biggest
bottleneck for further improvements consists of finding Q for larger N.

65

(6)

Proofs in Combinatorics through Optimization

1. The Ramsey Multiplicity Problem
2. Search Heuristics for Upper Bounds
3. Flag Algebras for Lower Bounds
4. A Related Problem

4. A Related Problem

Off-diagonal Ramsey Multiplicity
Question. Determining c3 is easy, but even c4 has been unresolved for over 60 years,
so can we say more when studying the off-diagonal variant
cs,t = lim min{ks (G) + kt (G) : |G| = n}?
næŒA famous result of Reiher from 2016 implies that c2,t = 1/(t ≠ 1).
Theorem (Parczyk, Pokutta, S., and Szabó 2022+)

c3,4 = 689 · 3≠8 and any large enough graph G admits a strong homomorphism into
the Schläfli graph after changing at most O(k3 (G) + k4 (G) ≠ c3,4 ) v (G)2 edges.
The fact that we can show stability proves that the search heuristic founda unique global optimum over all graphs of order 27!

66

Thank you for your attention!

67

The 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop on
Advances in Classical and Quantum Algorithms for
Optimization and Machine Learning
September 16th - 19th, 2022, Tokyo (The university of Tokyo), Japan,
and September 21st - 22nd, 2022, Fukuoka (Kyushu University), JapanThe UG framework version 1.0:
An update
Yuji Shinano
Zuse Institute Berlin, Germany
shinano@zib.de
The Ubiquity Generator Framework (UG) version 1.0 was released last year. It was
designed to parallelize powerful state-of-the-art branch-and-bound based solvers
externally in order to exploit their powerful performance. We call the underlying solvers``base solvers''; originally, a base solver is a branch-and-bound based solver, but in thecurrent release, it is redefined as any solver that is being parallelized by UG, since, inversion 1.0, it was generalized to be a software framework for high-level task
parallelization. In this talk, we present the concept of high-level task parallelization andits flexibility. We will show a few recent success stories of the instantiated parallel
solvers by UG version 1.0.

References[1] Y. Shinano, T. Achterberg, T. Berthold, S. Heinz, T. Koch and M. Winkler, "Solving Open MIP
Instances with ParaSCIP on Supercomputers Using up to 80,000 Cores," 2016 IEEE International
Parallel and Distributed Processing Symposium (IPDPS), 2016, pp. 770-779, doi:
10.1109/IPDPS.2016.56.
[2] G. Gamrath, T. Koch, S. Maher, D. Rehfeldt, and Y. Shinano, “SCIP-Jack—a solver for STP andvariants with parallelization extensions,”Mathematical Programming Computation, vol. 9, no. 2,
pp. 231–296,2017.
[3] Y. Shinano, D. Rehfeldt, and T. Koch, “Building optimal steiner trees on supercomputers byusing up to 43,000 cores,” in Integration of Constraint Programming, Artificial Intelligence,and Operations Research.CPAIOR 2019, vol. 11494, 2019, pp. 529–539.
[4] Y. Shinano, D. Rehfeldt, and T. Gally, “An easy way to build parallel state-of-the-art
combinatorial optimization problem solvers: A com-putational study on solving steiner tree
problems and mixed integer semidefinite programs by using ug[scip-*,*]-libraries,” in2019
IEEE International Parallel and Distributed Processing Symposium Workshops(IPDPSW),
2019, pp. 530–541.
[5] K. Fujii, N. Ito, S. Kim, M. Kojima, Y. Shinano, AND K.-C. Toh, Solving challenging large
scale QAPs, Tech. Rep. 21-02, ZIB, Takustr. 7, 14195 Berlin,2021.
[6] N.Tateiwa, Y.Shinano, S.Nakamura, A.Yoshida, S.Kaji, M.Yasuda, and K.Fujisawa. Massive
parallelization for finding shortest lattice vectors based on ubiquity generator framework. In
SC20: International Conference for High Performance Computing, Networking, Storage andAnalysis, pages 1-15. IEEE, 2020.

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

The 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop on
Advances in Classical and Quantum Algorithms for
Optimization and Machine Learning
September 16th - 19th, 2022, Tokyo (The university of Tokyo), Japan,
and September 21st - 22nd, 2022, Fukuoka (Kyushu University), JapanA Parallel Algorithm Combining
Relaxatino and heuristic for the
integrated long-haul and local vehicle
routing problem
Junko HOSODA*1, Stephen J. MAHER*2,
Yuji SHINANO*3 and Honas Christoffer
VILLUMSEN*4*1 Controls and Robotics Innovation Center, H itachi Ltd.,Japan
junko.hosoda.dp@hitachi.com*2 University of Exeter, United Kingdom *3 Zuse Institute
Berlin, Germany *4 Hi tachi Europe Ltd., Denmark
Commodity consolidation and vehicle route coordination are fundamental features ofthe logistics problem. This problem is called the supply chain service network design
problem (SCSNDP); the SCSNDP includes three problems: the warehouseconsolidation problem (WCP), the service network design problem (SNDP), and the
pickup and delivery problem (PDP). To obtain high-quality solutions, a combined
relaxation and heuristic algorithm is proposed[1]. The relaxation solver sets theboundaries of the solution space by considering the trend of the solution space. The
heuristic solver finds a high-quality solution that satisfies all the constraints within thebounded solution space; using the UG framework, the relaxation and heuristic solvers
are executed in parallel. The results show that the parallel execution of the relaxationand heuristic influences the quality of the SCSNDP solution.

References[1] Junko Hosoda, Stepehn J. Maher, Yuji Shinano, and Jonas Christoffer Villumsen, “Location,transshipment and routing: An adaptive transportation network integrating long-haul and local
vehicle routing”, EURO Journal on Transportation and Logistics, 100091, 2022,
https://doi.org/10.1016/j.ejtl.2022.100091

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

The 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop on
Advances in Classical and Quantum Algorithms for
Optimization and Machine Learning
September 16th - 19th, 2022, Tokyo (The university of Tokyo), Japan,
and September 21st - 22nd, 2022, Fukuoka (Kyushu University), Japan

Solving Large Scale QAPs
by Massively Parallel DNN-based
Branch-and-bound Method
Koichi FUJIINTT DATA Mathematical Systems Inc. Japanfujii@msi.co.jp
We report our progress on the project for solving large scale quadratic assignment
problems (QAPs). Our main approach to solve QAPs is a parallel branch-and-boundmethod efficiently implemented on a powerful computer system, using the Ubiquity
Generator Framework (UG) which can utilize more than 100,000 cores ([1]). Newtonbracketing method, the method we utilize to solve Lagrangian doubly nonnegative(DNN) relaxation of subproblems of QAPs, gives strong lower bounds, but it requires
more computational time ([2]) which makes difficult to scale in parallelization. We haveadded some new features to UG such as Enhanced Checkpoint or Huge Checkpoint File
Split to overcome these obstacles. In this talk, we describe the details of new features of
UG for solving QAPs and present some preliminary numerical results of solving large
QAPs on supercomputers ([3]).

References[1] Shinano, Yuji. "The ubiquity generator framework: 7 years of progress in parallelizing branchand-bound." Operations Research Proceedings 2017. Springer, Cham, 2018. 143-149.
[2] Kim, Sunyoung, Masakazu Kojima, and Kim-Chuan Toh. "A Newton-bracketing method for a
simple conic optimization problem." Optimization Methods and Software 36.2-3 (2021): 371388.
[3] Fujii, Koichi, Naoki Ito, Sunyoung Kim, Masakazu Kojima, Yuji Shinano, and Kim-Chuan Toh.
"Solving challenging large scale qaps." arXiv preprint arXiv:2101.09629 (2021).

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

The 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop on
Advances in Classical and Quantum Algorithms for
Optimization and Machine Learning
September 16th - 19th, 2022, Tokyo (The university of Tokyo), Japan,
and September 21st - 22nd, 2022, Fukuoka (Kyushu University), Japan

Mobility Optimization Engine andits Real-world Applications
Katsuki FUJISAWA
Institute of Mathematics for Industry, Kyushu University, Japanfujisawa@imi.kyushu-u.ac.jp
Various efforts have been made to realize a so-called super-smart society recently.
Our project team builds services to create new industries and other services throughcorporate collaboration [1,2,3]. We have utilized large-scale computing infrastructuresand developed the Cyber-Physical System Mobility Optimization Engine (CPS-MOE)
that provides various functions, including creating new industries. It can reduce cost andindustrial waste and constructing services to calculate the optimum control schedule oftransportation agencies. The latest research results and industry-academia collaborative
projects using CPS-MOE will be presented in this talk.

References[1] Akihiro Yoshida, Tatsuru Higurashi, Masaki Maruishi, Nariaki Tateiwa, Nozomi Hata, Akira
Tanaka, Takashi Wakamatsu, Kenichi Nagamatsu, Akira Tajima, and Katsuki Fujisawa, “NewPerformance Index “Attractiveness Factor” for Evaluating Websites via Obtaining Transition ofUsers’ Interests”, Data Science and Engineering, Volume 5, Issue 1, pp. 48-64, March 2020,
https://doi.org/10.1007/s41019-019-00112-1[2] Akihiro Yoshida, Yosuke Yatsushiro, Nozomi Hata, Tatsuru Higurashi, Nariaki Tateiwa, Takashi
Wakamatsu, Akira Tanaka, Kenichi Nagamatsu, and Katsuki Fujisawa, “Practical End-to-End
Repositioning Algorithm for Managing Bike-Sharing System”, The proceedings of the IEEE
BigData2019, 2019, https://doi.org/10.1109/BigData47090.2019.9005986[3] Nozomi Hata, Takashi Nakayama, Akira Tanaka, Takashi Wakamatsu, Akihiro Yoshida, Nariaki
Tateiwa, Yuri Nishikawa, Jun Ozawa, and Katsuki Fujisawa, “Mobility Optimization on Cyber
Physical System via Multiple Object Tracking and Mathematical Programming”, the Fifth
International Workshop on High Performance Big Graph Data Management, Analysis, andMining (BigGraphs 2018), to be held in conjunction with the 2018 IEEE International
Conference on Big Data (IEEE BigData 2018), 2018,https://doi.org/10.1109/BigData.2018.8622146

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

The 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop on
Advances in Classical and Quantum Algorithms for
Optimization and Machine Learning
September 16th - 19th, 2022, Tokyo (The university of Tokyo), Japan,
and September 21st - 22nd, 2022, Fukuoka (Kyushu University), Japan

Towards an optimal operation of
automated storage and retrieval systemwith multiple machines
Hiroki ISHIKURA
Kyushu University, Japan
tomomi_ishikura@kyudai.jp
We aim to improve the efficiency of a new type of automated storage and retrieval systems(AS/RSs) called multi-control automated storage and retrieval systems (MC-AS/RSs). MC-AS/RSs
have multiple storage/retrieval (S/R) machines that operate independently according to storage andretrieval requests. Consequently, MC-AS/RSs can transport loads farther without using human labor,
thereby requiring fewer human resources than conventional AS/RSs. However, the structure andcontrol method of AS/RSs are complex because multiple S/R machines must be controlled
simultaneously. Therefore, when operating an MC-AS/RS, many factors must be considered, such as
the sequence and transport timing. We propose an optimization method using a time-expanded
network (TEN) to solve these problems and generate optimal operational methods. First, our methodmodels an AS/RS with a TEN to calculate the optimal sequence and conveyance timing whileconsidering the movements of multiple S/R machines. Second, we formulate the operational efficiencyof the MC-AS/RS as a problem of minimizing the sum of execution times of requests on the TEN.
Finally, we generate the request order necessary for practical use based on the results. The mechanisms
implemented to achieve include a generator, optimizer, and scheduler. Our experiments confirm that
this method reduces the total execution time of requests compared with other rule-based methods. Thismethod enables us to propose an efficient operation method for AS/RSs with a complex structure ofmultiple carriers.

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

The 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop on
Advances in Classical and Quantum Algorithms for
Optimization and Machine Learning
September 16th - 19th, 2022, Tokyo (The university of Tokyo), Japan,
and September 21st - 22nd, 2022, Fukuoka (Kyushu University), Japan

Adaptive Cut Selection in Mixed-Integer
Linear Programming
Mark Turner
Chair of Software and Algorithms for Discrete Optimization,
Institute of Mathematics, Technische Universität Berlin,
S traße des 17. Juni 135, 10623 Berlin, Germ any
turner@zib.de
Cut selection is a subroutine used in all modern mixed-integer linear programming
solvers with the goal of selecting a subset of generated cuts that induce optimal solver
performance. These solvers have millions of parameter combinations, and so are
excellent candidates for parameter tuning. Cut selection scoring rules are usuallyweighted sums of different measurements, where the weights are parameters. We
present a parametric family of mixed-integer linear programs together with infinitely
many family-wide valid cuts. Some of these cuts can induce integer optimal solutions
directly after being applied, while others fail to do so even if an infinite amount are
applied. We show for a specific cut selection rule, that any finite grid search of theparameter space will always miss all parameter values, which select integer optimal
inducing cuts in an infinite amount of our problems. We propose a variation on thedesign of existing graph convolutional neural networks, adapting them to learn cut
selection rule parameters. We present a reinforcement learning framework for selecting
cuts, and train our design using said framework over MIPLIB 2017. Our framework anddesign show that adaptive cut selection does substantially improve performance over adiverse set of instances, but that finding a single function describing such a rule isdifficult.

173

Adaptive Cut Selection in Mixed-Integer Linear
Programming
M. Turner
T. Koch, F. Serrano, M. Winkler
18.09.22
6th RIKEN-IMI-ISM-NUS-ZIBMODAL-NHR Workshop – Tokyo

The definition of a MILP
This is a Mixed-Integer Linear Program (MILP):
argmin{c| x | Ax  b, l  x  u, x 2 Z|J | ⇥ Rn−|J | }x

I c 2 Rn - Objective coefficient vector
I A 2 Rm⇥n - Constraint matrix
I b 2 Rm - RHS constraint vector
I l, u 2 {R, −1, 1}n - Lower and uppervariable bound vectors
I J ✓ {1, . . . , n} - Set of indices ofinteger variables
M. Turner

Adaptive Cut Selection in Mixed-Integer Linear Programming

174

1

The definition of a cut
A cut is a constraint that does not remove any feasible solutions when added.
We restrict ourselves to linear cuts.
A cut ↵ = (↵0 , · · · , ↵n ) 2 Rn+1 is valid, where the set of feasible solutions is IX and:nX↵i xi  ↵0 , 8x 2 IX , where x = (x1 , · · · , xn )i=1

xLP
M. Turner

Adaptive Cut Selection in Mixed-Integer Linear Programming2

Cut selection
The purpose of cuts is to tighten the linear programming (LP) relaxation.
Cuts sometimes called separators, as they’re often generated by separating specificpoints (xLP ).
n
X↵i xiLP > ↵0 , where xLP = (x1LP , . . . , xnLP )i=1

Given the set of generated cuts S 0 = {↵1 , · · · , ↵|S 0 | }, find a subset S ✓ S 0 to add tothe formulation. That is the cut selection subproblem.
Con of adding all cuts: Large computational burden when solving larger LPs at each node
Con of adding no cuts: Substantially more nodes needed to solve to optimality
M. Turner

Adaptive Cut Selection in Mixed-Integer Linear Programming

175

3

Existing literature for cut selection
Older comprehensive computational experiments on cut selection:
I Constraint integer programming - Achterberg, 2007, PhD Thesis
I Embedding {0, 1/2}-Cuts in a Branch-and-Cut Framework: A Computational Study Giuseppe et al, 2007, doi 10.1287/ijoc.1050.0162

I Implementing cutting plane management and selection techniques - Wesselmann et al, 2011
Summary: Cut selection best as cheap heuristic. Filtering parallel cuts is most important.
Recent machine learning experiments on cut selection:
I Reinforcement learning for integer programming: Learning to cut - Y. Tang et al, 2020, MLR Press
I Learning to Select Cuts for Efficient Mixed-Integer Programming - Z. Huang et al, 2021,
https://doi.org/10.1016/j.patcog.2021.108353

I Adaptive Cut Selection in Mixed-Integer Linear Programming - M. Turner et al, 2022
Summary: Improvement is possible, but non-trivial and diﬃcult to generalise.
M. Turner

Adaptive Cut Selection in Mixed-Integer Linear Programming

4

Cut selection rule in SCIP
Cuts are scored as a weighted sum of measurements.
Eﬃcacy - Distance between cut and xLP .
Directed cuto↵ distance - Distance between cut and xLP in the direction of some primal
solution x̂.
Integer support - Ratio of non-zero coeﬃcients that are for integer valued variables.
Objective parallelism - Absolute cosine similarity measure between cut and objective.
λ1 ⇤ eff + λ2 ⇤ dcd + λ3 ⇤ isp + λ4 ⇤ obp, λ1 + λ2 + λ3 + λ4 = 1,
λi ≥ 0 8i 2 {1, 2, 3, 4}, λ = [λ1 , λ2 , λ3 , λ4 ]Algorithmic Idea: Add highest scoring cut. Filter all parallel cuts. Repeat until no cuts.
M. Turner

Adaptive Cut Selection in Mixed-Integer Linear Programming

176

5

Example of worst-case performance
Questions:
I Can a grid-search of the parameter space miss all cut selector parameter values thatwould quickly solve an instance?
I Can this be extended to an infinite family of instances, all of which only solve
quickly for values outside the grid-search?
I Can this infinite family be made to, no matter the grid-search, have an infinite
subset of instances that do not solve quickly?
Answer: Yes. By choosing a simplified cut selection rule, and disabling all other solver
settings (e.g. branching), we manage to prove this.
Is this useful: It’s incredibly cool. It motivates the need for adaptive cut selection if thesefringe cases are to be handled. Modern solvers are so interconnected however, thatproving this for practical solving processes is impossible.
M. Turner

Adaptive Cut Selection in Mixed-Integer Linear Programming

6

Learn mapping from MILP instances to λThat is, can we learn a function that outputs values {λ1 , λ2 , λ3 , λ4 }, which induce
optimal solver performance for the input instance.
Computational setup:
I Force 50 rounds of cuts
I Select exactly 10 cuts per round
I Stop solve process after the 50 rounds of cuts
I Enable all cut generators
I Test-set MIPLIB 2017. Filter out all numerically troubling instances. Root solve
must take at most 20s and be non-optimal with standard parameters.
I Use MIPLIB solution as primal
Default constant SCIP 8.0 parameters: [1.0, 0.0, 0.1, 0.1] (eff, dcd, isp, obp)
M. Turner

Adaptive Cut Selection in Mixed-Integer Linear Programming

177

7

Parameter sweep (Experiment)
Improvement measure: Relative improvement of gap closed compared to run using
standard cut selector parameters.
Experiment: Get the best improvement per instance with parameters from the following
grid-search:
4X

λi = 1, where λi =

i=1

βi
,
10

βi 2 N,8i 2 {1, 2, 3, 4}

Aim: Provide a lower bound on the potential improvement gain. In a perfect world, ourlearnt function will be at-least as good as a grid-search approach.

M. Turner

Adaptive Cut Selection in Mixed-Integer Linear Programming

8

Parameter sweep (Results)

M. Turner

Adaptive Cut Selection in Mixed-Integer Linear Programming

178

9

Phrase problem using reinforcement learning
We formulate our problem as a single step Markov decision process.
Initial state: MILP instance post-presolve, but before any cuts are added.
Terminal state: MILP instance after 50 rounds of cuts.
Action: Decision of cut selector parameter values followed by applying 50 round of cuts.
Reward: Relative gap improvement compared to standard cut selector parameter values.
Additional Info:I Actions drawn from a normal distribution, modelled as a graph neural network.
I Only use static features. All information is available before first LP solver.
I Approximate instance distribution using sample average approximation on MIPLIB.
I Evaluate trained network using mean of distribution.
M. Turner

Adaptive Cut Selection in Mixed-Integer Linear Programming

10

One model per instance (Overfitting results)

M. Turner

Adaptive Cut Selection in Mixed-Integer Linear Programming

179

11

Single model for all instances (Results)

M. Turner

Adaptive Cut Selection in Mixed-Integer Linear Programming

12

Thanks for listening!

If interested and would like more details, please feel free to email me at turner@zib.de orread the paper this presentation was based on:
Adaptive Cut Selection in Mixed-Integer Linear Programming, 2022
M. Turner, T. Koch, F. Serrano, M. Winkler
Preprint: https://arxiv.org/abs/2202.10962

M. Turner

Adaptive Cut Selection in Mixed-Integer Linear Programming

180

13

Outlook / Future Research

I Explore non-linear cut selection rules
I Explore di↵erent set of cut measurements
I Directly rank cuts with learned model
I Learn additional parameters in combination with cut selector parameters
I Define better standards of improved solver performance

M. Turner

Adaptive Cut Selection in Mixed-Integer Linear Programming

181

14

The 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop on
Advances in Classical and Quantum Algorithms for
Optimization and Machine Learning
September 16th - 19th, 2022, Tokyo (The university of Tokyo), Japan,
and September 21st - 22nd, 2022, Fukuoka (Kyushu University), JapanA Quadratic Programming Approach for
Performance Analysis of Energy Systems
Ryohei YOKOYAMA
Department of Mechanical Engineering, Osaka Metropolitan
University, Japanryokoyama@omu.ac.jp
The optimization of energy systems for their design and operation necessitates theanalysis of their performances under many different conditions. To analyze static(steady) and dynamic (unsteady) performances, it is necessary to solve nonlinear
algebraic equations and nonlinear differential algebraic equations, respectively [1, 2].
Nonlinear equations have been solved conventionally by the Newton-Raphson method,where the solution of linearized equations is repeated until convergence. On one hand,
however, the Jacobian matrices may not be regular because of network structures andoperating conditions of systems. On the other hand, they may not be calculated becauseof violated restrictions on variables used for equations. It is a burden for analysts to takeaccount of avoiding these situations in modeling systems. Thus, an alternative approachis necessary to reduce the burden. The singular value decomposition approach, which
derives least squares and minimum norm solutions, can resolve the former situation, butcannot resolve the latter situation. In this work, a quadratic programming approach will
be proposed to derive least squares and minimum norm solutions under restrictions on
variables. Some examples will be presented to show the effectiveness of the proposed
approach.

References[1] Ryohei Yokoyama, Shinsuke Takeuchi, and Koichi Ito, “Thermoeconomic Analysis andOptimization of a Gas Turbine Cogeneration Unit by a Systems Approach,” Proceedings of the
ASME Turbo Expo 2005, Paper No. GT2005-68392, pp. 1-7, 2005,
https://doi.org/10.1115/GT2005-68392[2] Ryohei Yokoyama, “Performance Analysis and Optimization of a CO2 Heat Pump Water
Heating System,” Xin-Rong Zhang and Hiroshi Yamaguchi (Eds.), Transcritical CO2 Heat Pump:
Fundamentals and Applications, Chapter 9, pp. 249-282, 2021, John Wiley & Sons,
https://doi.org/10.1002/9781118380055.ch9

183

184

185

186

187

188

189

190

191

192

193

194

The 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop on
Advances in Classical and Quantum Algorithms for
Optimization and Machine Learning
September 16th - 19th, 2022, Tokyo (The university of Tokyo), Japan,
and September 21st - 22nd, 2022, Fukuoka (Kyushu University), Japan

Geometric Learning of Ranking
Distributions"Shizuo KAJI
Institute of Mathematics for Industry, Kyushu University, Japanskaji@imi.kyushu-u.ac.j
Given a finite set X of n items, a complete order (permutation) of the items is calleda ranking of X. A ranking distribution over X is a collection of rankings of X. Most
existing models are classified into two groups; distance based and utility based. The
former assumes the probability of a ranking depends on the distance from the central
ranking, while the latter assumes the existence of the global utility value for each item
which is independent of raters. We introduce a high-fidelity model of a ranking
distribution utilising a novel geometric idea based on the hyperplane arrangement. We
will also discuss efficient learning and sampling algorithms ([1]).

References[1] Shizuo Kaji, Akira Horiguchi, Takuro Abe, Yohsuke Watanabe, “A Hyper-surface Arrangement Model ofRanking Distributions”, KDD '21: Proceedings of the 27th ACM SIGKDD Conference on Knowledge
Discovery & Data Mining, 796--804, 2021, https://doi.org/10.1145/3447548.3467253

195Modelling Preference
withHyperplane Arrangement
Shizuo Kaji
joint with A. Horiguchi, T. Abe, Y. Watanabe

Summary
Objective: construct a geometric representation of ranking dataaRanking data>>

metric space
(M,d)

>

,

c : items

Idea
a>c>b

partial orders on items

judges

b

c>b>a

b>c>a

c

b

>

Decision boundarya vs b

a>b>c
a

c>a>b

b>a>c

b vs cc vs a
Uniform sampling from the distribution= Uniform sampling from M

An embedding of items into a metric space M defines equidistant hyper-surfaces, which divide M into cells.
We find such an embedding, where the volume of each cell is proportional to the probability of the corresponding ranking.

Application to real world data

Challenge• The distribution over complete rankings on n items is n! dimensional• Each judge may provide only a partial ranking (not a full ranking)Representation capacity of our model
Histogram of ranked positions
Fatty tunadata

Sea urchin

model

SUSHI3-2016 dataset consistsof rankings by 5000 judges on
10 sushi items.

Complex, multi-modal
distribution of thedata is captured wellby the model
Many people rank Fatty tuna first or high.

Our model visualises somecharacteristics of the
population’s preference.

People love or hate Sea urchin.

196

Preference
preference of items

judge

>

,

>

>

>

◦ Each judge has his or her own preference◦ A judge may not know (or reveal) preference among some items

More formally,
B: a set of items
salmonegg

A complete ranking of B
salmon>tuna>> cucumber

egg

tuna
A partial ranking of B

cucumbersalmon

>

egg

,

tuna

> cucumber◦ Preference among the set of items can be modelled by an order on B
◦ An order can be partial;e.g., in the second case above, we do not know if egg > tuna or tuna < egg◦ Preference depends on judges(A judge is not necessarily a human but a criterion such as price and nutrition)◦ Ranking data consists of rankings by many judges

197

Definition of Order (Ranking)◦A partial order on a set B satisfies◦a < b and b > a do not occur at the same time◦a < b and b < c imply a < c◦A total order on B further satisfies◦a > b or b > a or a = b for any a,b ∈ B◦An order is described by a set of pairwise comparisonsex. { a>b, b>c, a>d }◦ remark: a partial order = transitive closure of an acyclic directed graphIn this talk:complete (or full) ranking = total orderincomplete (or partial) ranking = partial order

Ranking data given bya collection of pairwise comparisonsjudgehigher

lower

0

a

b

0

b

d

0

a

c

1

d

b

1

d

a

2

c

a

2

b

c

198

Ranking data=
rankings by many judges=
Pairwise comparisonsassociated with judges

Remark: Various problems occur as
special instances of ranking inference
Classification
Input

OutputPenguin > Puffin > Pigeon > … > Cat
Probability/confidence

Classifier + Data give rise to a ranking data

MODELLINGRANKING
DISTRIBUTION

199

Goal Today
Constructing a geometric model for ranking data (a set of partial rankings)
Assumptions:◦ Judges are indistinguishable (i.e., the data is just a set of partial rankings)◦ Each incomplete ranking can be completed(i.e., each judge has an unknown complete ranking)
These mean that the data is represented by a probability distribution overthe permutation group Sn of n items (|B|=n).
We call such a distribution a ranking distribution on n items
Two main difficulties:(1)A probability distribution over Sn is (n!-1) dimensional!
(2)How can we complete an incomplete ranking?

Existing models for ranking distribution
Two popular and basic models are
It has the mode at !" and theprobability decrease as the
distance from !" increases.

◦ Mallows’ φ model◦ ! " ∝ exp −()! ", """ ∈ ,#◦ Parameters: ( > 0, "" ∈ ,#◦ )! is a distance on Sn (e.g. the Kendal tau distance)◦ Plackett-Luce’s model%('! )"#!$% %('" )◦ ! " ∝ ∏$!"# ∑&◦ Parameters: /: 1 → 3$"" ∈ &$

Each item has a utility value
and selected one by one
according to the value

Reference: Marden, Analyzing and Modeling Rank Data, 1996

200

Geometric model in general◦ Representation learning is a field of research to find arepresentation of (implicitly) structured objects.
◦ The representation space often admits geometric structures.
(e.g., Euclidean space and other metric spaces)
objects

representation

image

vectorword
Points in a metric spacegraph

Ex. Graph representation translates a combinatorial structure (adjacency)
to a geometric one (metric)

Equidistant Line Arrangement on planeaDecision boundarya>b>ca vs b

R2
b>a>c

a>c>b

b vs cc vs a

b
c>a>b

b>c>a
c>b>a
c

201

• Each compartment corresponds to a
total ordering• They are subdivision of Voronoi cells• A partial ordering corresponds to theunion of compartments(e.g., the Voronoi of a = {a>b and a>c})

Equidistant Hyper-surface Arrangement
Decision boundary
a>c>b
Riemannian
manifold Ma>b>ca vs b

a
c>b>a

b>c>a

c
c>a>b

b

b vs cc vs a

Uniform sampling from the distribution=
Uniform sampling from M

b>a>c

Our model is an equidistant hyper-surface arrangement such thatthe volume of each cell is proportional to the probability of the corresponding ranking.

Learning Algorithm
Optimising coordinates in M ofboth judges and items by minimising
Triplet loss"% = $ $ max() ** , , − ) *+ , , , −.)&∈( )! $")#

Coulomb’s potential
1", = $/*-. ) ,* , ,+Target function: " = "% + 2 3 ",First, place judges in the right cells by "%Later, adjust volume by ",202

90 judges prefer item 0, 10 judges item 1
M = disk

Dimension constraints
Not all arrangements can be realised as equidistant arrangement.
(e.g., three equidistant lines of a triangle meet at the circumcentre)

Theorem
An equidistant arrangement in Dm of (generic)n points, we have n! cells if and only if n-2 < m.
◦ This gives a clue on the choice of the embedding dimension m.
◦ Note that the dimension of the representation space is mn.

EXPERIMENTS
WITH SUSHI DATASET

203

The SUSHI3-2016 dataset◦ Consists of complete rankings of 10 items by 5000 judges◦ Collected by T. Kamishima et al.
◦ Available at http://www.kamishima.net/sushi/

Learning the model◦ We chose 4 = 50 as the metric space.
◦ The theorem suggests dim =|B|-1.
◦ The results was almost same for ! = #$%◦ It took about two hours on Ryzen 2990WX.
◦ Visualisation was interpretable:◦ Fatty tuna has a large Voronoi region◦ Egg and Cucumber are similar and not
popular◦ Sea urchin and Salmon roe are similar anddistinctive

PCA projection of D9

204

Representation capacity: ranking position
data
model

Sea urchin

Fatty tuna

Many people rank Fatty tuna first or high.

People love or hate Sea urchin.

Complex, multi-modal distribution is captured well by the model

Representation capacity: conditional probability
data
model

for judges with egg > tunarank of eel

for judges with tuna > egg

The ranked position of eel (non-raw) depends heavily on
judges’ preference between tuna and egg,
which the model successfully captures.

205Generalisation capacity
Experimental setting

Not necessarily top-4

Input: Each judge reveals his or her ranking of, say, four items.
Output: The distribution for all ten items
input

The probability of a>b>c>d>e>f>g>h>i>j is 0.01

Judge u: a > c > b > dJudge v: e > g > d > b

output

The probability of b>f>e>i>j>a>g>h>d>c is 0.02…

…

This can be used for prediction:
For example, “what is the conditional probability ofg being ranked in the first by a judge with a>c>b?”

successfully captures the characteristics of the data that fatty
tuna is generally preferred while sea urchin is controversial.
Our model

Evaluation metrics for top-4 rankings

Comparison target: α = 10

model
Uniform
Plackett-Luce
↵
2
4
6
8
10
n-pointsOurs
B = {b1 , .
by scaling. Consider
Corr 0.025(±0.058)
0.436
Corr 0.165 0.795 0.840 0.868 0.886
correspond to labels. For 1 0.886
i < j  n,sKL
0.388(±0.001)
sKL 0.370 0.154 0.150 0.142 0.121
mid-hyper-plane 0.283
of bi and bj0.121
defined asW
2.51(±0.110)
2.03 of bi and
0.457
W
2.502 0.832 0.770 0.666 0.457
taining the mid-point
bj and w
model
Uniform
Plackett-LuceOurs
is bi bj . The setof hyper-planes{Hij }1i
↵
2
4
6
8
10
Corr 6 items
0.012(±0.041)
0.779
0.960
Am,n
Corr
0.893 0.922
0.940ranking
0.960among
Each 0.546
judge revealeda partial
out. of 10.
sKL
0.166(±0.001)
0.073
0.011
A chamber of Am,n is a connectedcsKL
0.106 for0.023
0.019ranking
0.016with
0.011
Evaluated
the top-4
W
1.894(±0.023)
0.891
0.157
[H2Am,n H, and the set of all chambers
is dW
0.965 0.305 0.287 0.247 0.157
How to interpret:◦ Correlation of probability (higher is better)
Note
that
p(w)
>
0 of
for$ w
2 Stheand only
2: inThe
fidelityof thefittedmodelsn if SUSHI
Table 3: Model performance for learning Table
partialIf youaregivenranking= to4◦
symmetrised
KL-divergence(lowerisbetter)dataset.
(top)
for
top-4
ranking(bottom)forsubbercorresponding
tow.
Hence,
wehave
random
items
from
each
judge
formation. (top) for top-4 ranking (bottom) for sub(thechoice
of
four
items
variesforrankingforfivelabels
{0,
4,
5,
7,
9}.
The
metrics
areo
We
have
to
show
that
a
≥
n!
if
and
m,n
ranking forfive labels
0, 4, 5, 7,(lower
and 9.is better)◦ Wasserstein
distance
judge),
youIf
can
the
introduced in each§3.
B isgeneric.
B tellis generic,then am,n =population’s top-4 ranking
follows immediately from Corollary in p
distribution fairly well (corr = 0.795).
the following, we show that am,n < n!
Recall that am,n coincides with the tota
the complexified complement U := (Rm \Am,n (see, for example, [24]). In other w
206

am,n =mX
i=0dimC H i (U, C) = dimC

Conclusion
A model for ranking data which◦has high representation capacity◦utilises a geometric structure as regularisation◦is relatively low dimensional◦is easy to sample from◦is mathematically interesting◦comes with an implementation:
https://github.com/shizuo-kaji/rankLearningOpen problem

2◦ Parametrisation of arrangements

213◦ Efficient sampling under a condition◦ Stricter bounds for embedding dimension

231

1◦ Choice of the ambient manifold◦ Combinatorics of arrangement as a topological invariant(c.f. the homotopy type of the configuration spacecan sometimes distinguish homeo types)◦ Optimal transport on Sn(e.g., how the combinatorics helps computation?)

321

123

132

312

◦ Generalisation to partial rankings
Toy example of S1◦ Evaluation by myself by eating a lot of sushi

Always antipodal

207

3

The 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop on
Advances in Classical and Quantum Algorithms for
Optimization and Machine Learning
September 16th - 19th, 2022, Tokyo (The university of Tokyo), Japan,
and September 21st - 22nd, 2022, Fukuoka (Kyushu University), Japan

Convex integer optimization withFrank-Wolfe methods
Sebastian POKUTTA
Zuse Institute Berlin and TU Berlin
pokutta@zib.de
Mixed-integer nonlinear optimization is a broad class of problems that feature
combinatorial structures and nonlinearities. Typical exact methods combine a branchand-bound scheme with relaxation and separation subroutines. We investigate theproperties and advantages of error-adaptive first-order methods based on the FrankWolfe algorithm for this setting, requiring only a gradient oracle for the objectivefunction and linear optimization over the feasible set. In particular, we will study thealgorithmic consequences of optimizing with a branch-and-bound approach where thesubproblem is solved over the convex hull of the mixed-integer feasible set thanks tolinear oracle calls, compared to solving the subproblems over the continuous relaxationof the same set. This novel approach computes feasible solutions while working on asingle representation of the polyhedral constraints, leveraging the full extent of MixedInteger Programming (MIP) solvers without an outer approximation scheme.
(joint work with Deborah Hendrych, Hannah Troppens, Mathieu Besançon)

References[1] Hendrych, D., Troppens, H., Besançon, M., and Pokutta, S. (2022). Convex integer optimization
with Frank-Wolfe methods. Preprint. https://arxiv.org/abs/2208.11010

209

Convex integer optimization with Frank-Wolfe methods
Sebastian Pokutta
joint work with: Mathieu Besançon, Deborah Hendrych, and Hannah Troppens
Technische Universität Berlin
andZuse Institute Berlin
pokutta@math.tu-berlin.de@spokutta

6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop on
Advances in Classical and Quantum Algorithms for Optimizationand Machine Learning
September 19th, 2022 · Tokyo, Japan

MATH
Berlin Mathematics Research Center

What is this talk about?
Introduction

A mixed-integer convex optimization method
based on conditional gradients.

Sebastian Pokutta · Boscia: Mixed-Integer Conditional Gradients210

1 / 23

What is this talk about?
Introduction

A mixed-integer convex optimization method
based on conditional gradients.
Why? Conditional gradients generate sparse iterates, leading to lower fractionality, and hence
less branching.

Sebastian Pokutta · Boscia: Mixed-Integer Conditional Gradients

1 / 23What is this talk about?
Introduction

A mixed-integer convex optimization method
based on conditional gradients.
Why? Conditional gradients generate sparse iterates, leading to lower fractionality, and hence
less branching.
Today: A brief overview of approach and solver.

Sebastian Pokutta · Boscia: Mixed-Integer Conditional Gradients211

1 / 23

What is this talk about?
Introduction

A mixed-integer convex optimization method
based on conditional gradients.
Why? Conditional gradients generate sparse iterates, leading to lower fractionality, and hence
less branching.
Today: A brief overview of approach and solver.
Outline• Recap: Conditional Gradients a.k.a. the Frank-Wolfe algorithm• Mixed-Integer Conditional Gradients• Julia Package Boscia.jl(Hyperlinked) References are not exhaustive; check references contained therein.

Sebastian Pokutta · Boscia: Mixed-Integer Conditional Gradients

1 / 23

Conditional Gradientsa.k.a. the Frank-Wolfe algorithm—The Basics—

Sebastian Pokutta · Boscia: Mixed-Integer Conditional Gradients212

2 / 23

The basic problem
Conditional Gradients a.k.a. the Frank-Wolfe algorithm

Given a smooth and convex function f and a polytope
P, solve optimization problem:

Sebastian Pokutta · Boscia: Mixed-Integer Conditional Gradients3 / 23The basic problem
Conditional Gradients a.k.a. the Frank-Wolfe algorithm

Given a smooth and convex function f and a polytope
P, solve optimization problem:

min f (x )x 2P(baseProblem)

Source: [Jaggi, 2013]

Sebastian Pokutta · Boscia: Mixed-Integer Conditional Gradients213

3 / 23

The basic problem
Conditional Gradients a.k.a. the Frank-Wolfe algorithm

Given a smooth and convex function f and a polytope
P, solve optimization problem:

min f (x )x 2P(baseProblem)

Source: [Jaggi, 2013]

1. Very versatile model
2. Can use various types of information about both f and P
3. Works very well in (continuous) real-world applications
4. At the core of many (all?) learning algorithms (albeit mostly non-convex case)

Sebastian Pokutta · Boscia: Mixed-Integer Conditional Gradients3 / 23The basic problem
Conditional Gradients a.k.a. the Frank-Wolfe algorithm

Given a smooth and convex function f and a polytope
P, solve optimization problem:

min f (x )x 2P(baseProblem)

Source: [Jaggi, 2013]

Our setup.
1. Access to P. Linear Minimization Oracle (LMO): Given linear objective c returnxarg min c T v .
v 2PSebastian Pokutta · Boscia: Mixed-Integer Conditional Gradients214

3 / 23

The basic problem
Conditional Gradients a.k.a. the Frank-Wolfe algorithm

Given a smooth and convex function f and a polytope
P, solve optimization problem:

min f (x )x 2P(baseProblem)

Source: [Jaggi, 2013]

Our setup.
1. Access to P. Linear Minimization Oracle (LMO): Given linear objective c returnxarg min c T v .
v 2P

2. Access to f . First-Order Oracle (FO): Given x returnr f (x )andf (x ).

Sebastian Pokutta · Boscia: Mixed-Integer Conditional Gradients3 / 23The basic problem
Conditional Gradients a.k.a. the Frank-Wolfe algorithm

Given a smooth and convex function f and a polytope
P, solve optimization problem:

min f (x )x 2P(baseProblem)

Source: [Jaggi, 2013]

Our setup.
1. Access to P. Linear Minimization Oracle (LMO): Given linear objective c returnxarg min c T v .
v 2P

2. Access to f . First-Order Oracle (FO): Given x returnr f (x )andf (x ).

) Complexity of convex optimization relative to LO/FO oracle

Sebastian Pokutta · Boscia: Mixed-Integer Conditional Gradients215

3 / 23

Interlude: why LMOs?
Conditional Gradients a.k.a. the Frank-Wolfe algorithm

LMO model has many advantages.
1. Includes explicit formulation via constraints
2. Some problems do not posess ‘small’ formulations but have efficient LMOs.
Example: Matching Polytope[Rothvoss, 2014, Braun and Pokutta, 2015a,b, Braun et al., 2015, 2017a]
3. Allows modeling of compact convex constraints as long as we have an LMO.
Example: SDP cone
4. Often much faster than projection.
Example: nuclear norm. Largest singular vector (Lanczos method) vs. full SVD
5. LMO is a black box for the algorithms
6. For many LMOs of interest close form solutions available.
Example: ✓ 1 -ball for LASSO regression.
For an overview see: [Combettes and Pokutta, 2021]

Sebastian Pokutta · Boscia: Mixed-Integer Conditional Gradients

4 / 23The basic problem
Conditional Gradients a.k.a. the Frank-Wolfe algorithm

Basic notions. Let f : Rn ! R be a differentiable function.

Sebastian Pokutta · Boscia: Mixed-Integer Conditional Gradients216

5 / 23

The basic problem
Conditional Gradients a.k.a. the Frank-Wolfe algorithm

Basic notions. Let f : Rn ! R be a differentiable function.
Definition (Convexity)
For all x , y it holds:f (y ) − f (x ) ≥ hrf (x ), y − x i .
In particular, all local minima are global minima.

Sebastian Pokutta · Boscia: Mixed-Integer Conditional Gradients

5 / 23The basic problem
Conditional Gradients a.k.a. the Frank-Wolfe algorithm

Basic notions. Let f : Rn ! R be a differentiable function.
Definition (Convexity)
For all x , y it holds:f (y ) − f (x ) ≥ hrf (x ), y − x i .
In particular, all local minima are global minima.
Definition (L-Smoothness)
For all x , y it holds:f (y ) − f (x )  hrf (x ), y − x i +Lky − x k2 .
2

Sebastian Pokutta · Boscia: Mixed-Integer Conditional Gradients217

5 / 23

The basic problem
Conditional Gradients a.k.a. the Frank-Wolfe algorithm

Basic notions. Let f : Rn ! R be a differentiable function.
Definition (Convexity)
For all x , y it holds:f (y ) − f (x ) ≥ hrf (x ), y − x i .
In particular, all local minima are global minima.
Definition (L-Smoothness)
For all x , y it holds:f (y ) − f (x )  hrf (x ), y − x i +Lky − x k2 .
2

Sebastian Pokutta · Boscia: Mixed-Integer Conditional Gradients

5 / 23

The Frank-Wolfe Algorithm
Conditional Gradients a.k.a. the Frank-Wolfe algorithm

Algorithm Frank-Wolfe Algorithm (FW)
1: x0 2 P
2: for t = 0 to T − 1 do
3:vt
arg min hrf (xt ), v i

xt +1
5: end for
4:

x⇤

v 2P

xt + ✏t (vt − xt )[Frank and Wolfe, 1956, Levitin and Polyak, 1966]

Sebastian Pokutta · Boscia: Mixed-Integer Conditional Gradients218

6 / 23

The Frank-Wolfe Algorithm
Conditional Gradients a.k.a. the Frank-Wolfe algorithm

xt

Algorithm Frank-Wolfe Algorithm (FW)
1: x0 2 P
2: for t = 0 to T − 1 do
3:vt
arg min hrf (xt ), v i

xt +1
5: end for
4:

x⇤

v 2P

xt + ✏t (vt − xt )[Frank and Wolfe, 1956, Levitin and Polyak, 1966]

Sebastian Pokutta · Boscia: Mixed-Integer Conditional Gradients

6 / 23

The Frank-Wolfe Algorithm
Conditional Gradients a.k.a. the Frank-Wolfe algorithm

xt−rf (xt )Algorithm Frank-Wolfe Algorithm (FW)
1: x0 2 P
2: for t = 0 to T − 1 do
3:vt
arg min hrf (xt ), v i

xt +1
5: end for
4:

x⇤

v 2P

xt + ✏t (vt − xt )[Frank and Wolfe, 1956, Levitin and Polyak, 1966]

Sebastian Pokutta · Boscia: Mixed-Integer Conditional Gradients219

6 / 23

The Frank-Wolfe Algorithm
Conditional Gradients a.k.a. the Frank-Wolfe algorithm

xt−rf (xt )Algorithm Frank-Wolfe Algorithm (FW)
1: x0 2 P
2: for t = 0 to T − 1 do
3:vt
arg min hrf (xt ), v i

xt +1
5: end for
4:

x⇤

v 2P

xt + ✏t (vt − xt )vt
[Frank and Wolfe, 1956, Levitin and Polyak, 1966]

Sebastian Pokutta · Boscia: Mixed-Integer Conditional Gradients

6 / 23

The Frank-Wolfe Algorithm
Conditional Gradients a.k.a. the Frank-Wolfe algorithm

xt

Algorithm Frank-Wolfe Algorithm (FW)
1: x0 2 P
2: for t = 0 to T − 1 do
3:vt
arg min hrf (xt ), v i

xt +1
5: end for
4:

x⇤

v 2P

xt + ✏t (vt − xt )vt
[Frank and Wolfe, 1956, Levitin and Polyak, 1966]

Sebastian Pokutta · Boscia: Mixed-Integer Conditional Gradients220

6 / 23

The Frank-Wolfe Algorithm
Conditional Gradients a.k.a. the Frank-Wolfe algorithm

xt

Algorithm Frank-Wolfe Algorithm (FW)
1: x0 2 P
2: for t = 0 to T − 1 do
3:vt
arg min hrf (xt ), v i

xt +1
5: end for
4:

x⇤

xt +1

v 2P

xt + ✏t (vt − xt )vt
[Frank and Wolfe, 1956, Levitin and Polyak, 1966]

Sebastian Pokutta · Boscia: Mixed-Integer Conditional Gradients

6 / 23

The Frank-Wolfe Algorithm
Conditional Gradients a.k.a. the Frank-Wolfe algorithm

xt

Algorithm Frank-Wolfe Algorithm (FW)
1: x0 2 P
2: for t = 0 to T − 1 do
3:vt
arg min hrf (xt ), v i

xt +1
5: end for
4:

x⇤

xt +1

v 2P

xt + ✏t (vt − xt )vt
[Frank and Wolfe, 1956, Levitin and Polyak, 1966]

Sebastian Pokutta · Boscia: Mixed-Integer Conditional Gradients

221

6 / 23

The Frank-Wolfe Algorithm
Conditional Gradients a.k.a. the Frank-Wolfe algorithm

xt

Algorithm Frank-Wolfe Algorithm (FW)
1: x0 2 P
2: for t = 0 to T − 1 do
3:vt
arg min hrf (xt ), v i

xt +1
5: end for
4:

x⇤

xt +1

v 2P

xt + ✏t (vt − xt )vt
[Frank and Wolfe, 1956, Levitin and Polyak, 1966]

Advantages:• Extremely simple and robust: no complicated data structures to maintain• Easy to implement: requires only the two oracles• Projection-free: feasibility convex combination and LO oracle.

• Sparsity: optimal solution is a convex combination of (usually) vertices.

Sebastian Pokutta · Boscia: Mixed-Integer Conditional Gradients

6 / 23

The Frank-Wolfe Algorithm
Conditional Gradients a.k.a. the Frank-Wolfe algorithm

xt

Algorithm Frank-Wolfe Algorithm (FW)
1: x0 2 P
2: for t = 0 to T − 1 do
3:vt
arg min hrf (xt ), v i

xt +1
5: end for
4:

x⇤

xt +1

v 2P

xt + ✏t (vt − xt )vt
[Frank and Wolfe, 1956, Levitin and Polyak, 1966]

Advantages:• Extremely simple and robust: no complicated data structures to maintain• Easy to implement: requires only the two oracles• Projection-free: feasibility convex combination and LO oracle.

• Sparsity: optimal solution is a convex combination of (usually) vertices.
Disadvantages:• Suboptimal convergence rate of O(1/T )Sebastian Pokutta · Boscia: Mixed-Integer Conditional Gradients222

6 / 23

The Frank-Wolfe Algorithm
Conditional Gradients a.k.a. the Frank-Wolfe algorithm

xt

Algorithm Frank-Wolfe Algorithm (FW)
1: x0 2 P
2: for t = 0 to T − 1 do
3:vt
arg min hrf (xt ), v i

xt +1
5: end for
4:

x⇤

xt +1

v 2P

xt + ✏t (vt − xt )vt
[Frank and Wolfe, 1956, Levitin and Polyak, 1966]

Advantages:• Extremely simple and robust: no complicated data structures to maintain• Easy to implement: requires only the two oracles• Projection-free: feasibility convex combination and LO oracle.

• Sparsity: optimal solution is a convex combination of (usually) vertices.
Disadvantages:• Suboptimal convergence rate of O(1/T )) Despite (theoretically) suboptimal rate heavily used in applications due to simplicity.
Sebastian Pokutta · Boscia: Mixed-Integer Conditional Gradients

6 / 23

Simple Convergence Proof
Conditional Gradients a.k.a. the Frank-Wolfe algorithm

Theorem (Convergence rate of the vanilla Frank-Wolfe Algorithm)
Let f be L-smooth convex, P be polytope with diameter D. With choice ✏t ⌘ t +23 :f (xt ) − f (x ⇤ ) 

2LD2
.
t+3

Sebastian Pokutta · Boscia: Mixed-Integer Conditional Gradients223

7 / 23

Simple Convergence Proof
Conditional Gradients a.k.a. the Frank-Wolfe algorithm

Theorem (Convergence rate of the vanilla Frank-Wolfe Algorithm)
Let f be L-smooth convex, P be polytope with diameter D. With choice ✏t ⌘ t +23 :f (xt ) − f (x ⇤ ) 

2LD2
.
t+3

Proof Sketch.
By smoothness:f (xt +1 ) − f (xt )  hrf (xt ), xt +1 − xt i +

2

L ✏t
L
k xt +1 − xt k 2 = ✏t hrf (xt ), vt − xt i +k vt − xt k 2 .
2
2⇤

Sebastian Pokutta · Boscia: Mixed-Integer Conditional Gradients

7 / 23

Simple Convergence Proof
Conditional Gradients a.k.a. the Frank-Wolfe algorithm

Theorem (Convergence rate of the vanilla Frank-Wolfe Algorithm)
Let f be L-smooth convex, P be polytope with diameter D. With choice ✏t ⌘ t +23 :f (xt ) − f (x ⇤ ) 

2LD2
.
t+3

Proof Sketch.
By smoothness:f (xt +1 ) − f (xt )  hrf (xt ), xt +1 − xt i +

2

L ✏t
L
k xt +1 − xt k 2 = ✏t hrf (xt ), vt − xt i +k vt − xt k 2 .
2
2

LP maximality and convexity: hrf (xt ), vt − xt i  hrf (xt ), x ⇤ − xt i  f (x ⇤ ) − f (xt ). Moreover, k vt − xt k  D.

Sebastian Pokutta · Boscia: Mixed-Integer Conditional Gradients224⇤

7 / 23

Simple Convergence Proof
Conditional Gradients a.k.a. the Frank-Wolfe algorithm

Theorem (Convergence rate of the vanilla Frank-Wolfe Algorithm)
Let f be L-smooth convex, P be polytope with diameter D. With choice ✏t ⌘ t +23 :f (xt ) − f (x ⇤ ) 

2LD2
.
t+3

Proof Sketch.
By smoothness:f (xt +1 ) − f (xt )  hrf (xt ), xt +1 − xt i +

2

L ✏t
L
k xt +1 − xt k 2 = ✏t hrf (xt ), vt − xt i +k vt − xt k 2 .
2
2

LP maximality and convexity: hrf (xt ), vt − xt i  hrf (xt ), x ⇤ − xt i  f (x ⇤ ) − f (xt ). Moreover, k vt − xt k  D.

Thus:

f (xt +1 ) − f (x ⇤ )  (1 − ✏t )(f (xt ) − f (x ⇤ )) + ✏t2

LD2
.
2

⇤

Sebastian Pokutta · Boscia: Mixed-Integer Conditional Gradients

7 / 23

Simple Convergence Proof
Conditional Gradients a.k.a. the Frank-Wolfe algorithm

Theorem (Convergence rate of the vanilla Frank-Wolfe Algorithm)
Let f be L-smooth convex, P be polytope with diameter D. With choice ✏t ⌘ t +23 :f (xt ) − f (x ⇤ ) 

2LD2
.
t+3

Proof Sketch.
By smoothness:f (xt +1 ) − f (xt )  hrf (xt ), xt +1 − xt i +

2

L ✏t
L
k xt +1 − xt k 2 = ✏t hrf (xt ), vt − xt i +k vt − xt k 2 .
2
2

LP maximality and convexity: hrf (xt ), vt − xt i  hrf (xt ), x ⇤ − xt i  f (x ⇤ ) − f (xt ). Moreover, k vt − xt k  D.

Thus:

f (xt +1 ) − f (x ⇤ )  (1 − ✏t )(f (xt ) − f (x ⇤ )) + ✏t2

LD2
.
2

By Induction (plugging in the guarantee + definition of ✏t ):✓f (xt +1 ) − f (x ⇤ )  1 −by (t + 2)(t + 4)  (t + 3)2 .

2
t+3

◆

2LD2
4
LD2
2LD2 (t + 2)
2LD2
+
·
=

,
t+3
2
t+4
(t + 3)2(t + 3)2

Sebastian Pokutta · Boscia: Mixed-Integer Conditional Gradients225⇤

7 / 23

Significant progress over the recent years (incomplete list)
Conditional Gradients a.k.a. the Frank-Wolfe algorithm

1. Strongly convex case

[Garber and Hazan, 2013, Lacoste-Julien and Jaggi, 2015, Lan and Zhou, 2016, Garber and Meshi, 2016]

2. Non-convex case[Lacoste-Julien, 2016]

3. Online case

[Hazan and Kale, 2012]

4. Stochastic variants and adaptive gradients[Hazan and Luo, 2016, Reddi et al., 2016, Combettes et al., 2020]

5. Sharp functions and sharp regions[Kerdreux et al., 2019, 2021a,b]

6. Acceleration[Diakonikolas et al., 2020, Bach, 2020, Carderera et al., 2021]

7. Specialized variants[Freund et al., 2017, Braun et al., 2017b, 2019b,a]Conditional Gradients very competitive: simple, robust, real-world performance.

For more background etc see our survey!

[Braun et al., 2022]

Sebastian Pokutta · Boscia: Mixed-Integer Conditional Gradients

8 / 23Mixed-Integer Conditional Gradients—The Framework—

Sebastian Pokutta · Boscia: Mixed-Integer Conditional Gradients

226

9 / 23

Problem setting
Mixed-Integer Conditional Gradients

Basically. Smooth convex objective over MIPs.

Sebastian Pokutta · Boscia: Mixed-Integer Conditional Gradients

10 / 23

Problem setting
Mixed-Integer Conditional Gradients

Basically. Smooth convex objective over MIPs.
Slightly more general:min f (x , y )x ,ys.t. x 2 X

xj 2 Z 8j 2 J
y2Ywith LMO over (X \ bounds) ⇥ Y .

Sebastian Pokutta · Boscia: Mixed-Integer Conditional Gradients227

10 / 23

Three main algorithmic frameworks for MINLPs
Mixed-Integer Conditional Gradients

Diamond blocks represent nodal relaxations in the given framework.

BnB
bound

node

NLP
Standard BnB framework on top
of NLP relaxations.

Sebastian Pokutta · Boscia: Mixed-Integer Conditional Gradients

11 / 23

Three main algorithmic frameworks for MINLPs
Mixed-Integer Conditional Gradients

Diamond blocks represent nodal relaxations in the given framework.

BnB

BnB

fixings

bound

node

node
bound

NLP
Standard BnB framework on top
of NLP relaxations.

solution

NLP

LP

LP-based MINLP frameworks
and outer approximations

Sebastian Pokutta · Boscia: Mixed-Integer Conditional Gradients

228

11 / 23

Three main algorithmic frameworks for MINLPs
Mixed-Integer Conditional Gradients

Diamond blocks represent nodal relaxations in the given framework.

BnB

BnB

BnB

bound

node

fixings
node

bound

node

bound

NLP

LP

NLP

NLP

solution

direction

gradient

MIP

Standard BnB framework on top
of NLP relaxations.

LP-based MINLP frameworks
and outer approximations
Our approach. Linearized models solved as MIPs
within the Frank-Wolfe algorithm on top of which
we branch

Tree of trees or forest ! Boscia (Corsican) = Forest.

Sebastian Pokutta · Boscia: Mixed-Integer Conditional Gradients

11 / 23

Branching: continuous relaxation (usual approach)
Mixed-Integer Conditional Gradientsx̃Sebastian Pokutta · Boscia: Mixed-Integer Conditional Gradients229

12 / 23

Branching: continuous relaxation (usual approach)
Mixed-Integer Conditional Gradientsx̃Sebastian Pokutta · Boscia: Mixed-Integer Conditional Gradients

12 / 23

Branching: mixed-integer hull (our approach)
Mixed-Integer Conditional Gradientsx̃Sebastian Pokutta · Boscia: Mixed-Integer Conditional Gradients230

13 / 23

Branching: mixed-integer hull (our approach)
Mixed-Integer Conditional Gradients

x̃l

x̃

x̃r

Sebastian Pokutta · Boscia: Mixed-Integer Conditional Gradients

13 / 23

Branching: mixed-integer hull (our approach)
Mixed-Integer Conditional Gradients

x̃l

x̃

x̃r

Open question.
Can we define adaptive criteria to choose relaxation?
(E.g., geometry of the feasible set, conditioning of the function)

Sebastian Pokutta · Boscia: Mixed-Integer Conditional Gradients

231

13 / 23

Reducing number of MIP oracle calls
Mixed-Integer Conditional Gradients

We use Blended Pairwise Conditional Gradients (BPCG)

Sebastian Pokutta · Boscia: Mixed-Integer Conditional Gradients[Tsuji et al., 2022]

14 / 23

Reducing number of MIP oracle calls
Mixed-Integer Conditional Gradients

We use Blended Pairwise Conditional Gradients (BPCG)[Tsuji et al., 2022]• Lazification. aggressively reuse old solutions

Sebastian Pokutta · Boscia: Mixed-Integer Conditional Gradients232

14 / 23

Reducing number of MIP oracle calls
Mixed-Integer Conditional Gradients

We use Blended Pairwise Conditional Gradients (BPCG)[Tsuji et al., 2022]• Lazification. aggressively reuse old solutions• Blending. perform local steps for sparsity ! low fractionality

Sebastian Pokutta · Boscia: Mixed-Integer Conditional Gradients

14 / 23

Reducing number of MIP oracle calls
Mixed-Integer Conditional Gradients

We use Blended Pairwise Conditional Gradients (BPCG)[Tsuji et al., 2022]• Lazification. aggressively reuse old solutions• Blending. perform local steps for sparsity ! low fractionality• Active set. branching means simply splitting convex combination

Sebastian Pokutta · Boscia: Mixed-Integer Conditional Gradients233

14 / 23

Reducing number of MIP oracle calls
Mixed-Integer Conditional Gradients

We use Blended Pairwise Conditional Gradients (BPCG)[Tsuji et al., 2022]• Lazification. aggressively reuse old solutions• Blending. perform local steps for sparsity ! low fractionality• Active set. branching means simply splitting convex combination• Discarded set. reuse solutions from previous nodes

Sebastian Pokutta · Boscia: Mixed-Integer Conditional Gradients

14 / 23

Reducing number of MIP oracle calls
Mixed-Integer Conditional Gradients

We use Blended Pairwise Conditional Gradients (BPCG)[Tsuji et al., 2022]• Lazification. aggressively reuse old solutions• Blending. perform local steps for sparsity ! low fractionality• Active set. branching means simply splitting convex combination• Discarded set. reuse solutions from previous nodes• Incomplete resolution and warmstarts. Less work per node
Does it help?

Sebastian Pokutta · Boscia: Mixed-Integer Conditional Gradients234

14 / 23

Reducing number of MIP oracle calls
Mixed-Integer Conditional Gradients

We use Blended Pairwise Conditional Gradients (BPCG)[Tsuji et al., 2022]• Lazification. aggressively reuse old solutions• Blending. perform local steps for sparsity ! low fractionality• Active set. branching means simply splitting convex combination• Discarded set. reuse solutions from previous nodes• Incomplete resolution and warmstarts. Less work per node
Does it help?
8

25

6

20

LMO calls

Avg set size

7

5

15

4
3

1

6
Active set

11

Node depth

Discarded set

16

10

LMO calls

On average something like 7 to 10 sub-MIPs per node.
Sebastian Pokutta · Boscia: Mixed-Integer Conditional Gradients

14 / 23

Reducing cost for each MIP
Mixed-Integer Conditional Gradients

Subproblems are MIPs. Leverage MIP advances:• Cutting-planes• Domain propagation• Presolving• Primal heuristics• etc.

Sebastian Pokutta · Boscia: Mixed-Integer Conditional Gradients235

15 / 23

Reducing cost for each MIP
Mixed-Integer Conditional Gradients

Subproblems are MIPs. Leverage MIP advances:• Cutting-planes• Domain propagation• Presolving• Primal heuristics• etc.

Moreover, we can reuse information across solves heavily:• MIP solver called with different objectives within node

• Identical polyhedron with updated bounds solved across nodes• All found primal solutions are valid for main problem

Sebastian Pokutta · Boscia: Mixed-Integer Conditional Gradients

15 / 23

Reducing cost for each MIP
Mixed-Integer Conditional Gradients

Subproblems are MIPs. Leverage MIP advances:• Cutting-planes• Domain propagation• Presolving• Primal heuristics• etc.

Moreover, we can reuse information across solves heavily:• MIP solver called with different objectives within node

• Identical polyhedron with updated bounds solved across nodes• All found primal solutions are valid for main problem
Question of MIP reoptimization:[Gamrath et al., 2015]

Which information should be (conditionally) transferred across instances?
Sebastian Pokutta · Boscia: Mixed-Integer Conditional Gradients236

15 / 23

Boscia.jl—The Code—Sebastian Pokutta · Boscia: Mixed-Integer Conditional Gradients

16 / 23

The package
Boscia.jl• Julia package• Based on Bonobo.jl (BnB package) and FrankWolfe.jl (our FW package)• Via MOI can use basically any MIP solver; some features specific to SCIP

• Includes other features such as hybrid branching• Available under MIT license

Sebastian Pokutta · Boscia: Mixed-Integer Conditional Gradients237

17 / 23

Example: CodeBoscia.jl⌥⌅

using Boscia
using FrankWolfe
using Random
using SCIP
using LinearAlgebra
import MathOptInterface
const MOI = MathOptInterfacen = 6
const diffw = 0.5 * ones(n)o = SCIP.Optimizer()
MOI.set(o, MOI.Silent(), true)x = MOI.add_variables(o, n)
for xi in x
MOI.add_constraint(o, xi, MOI.GreaterThan(0.0))
MOI.add_constraint(o, xi, MOI.LessThan(1.0))
MOI.add_constraint(o, xi, MOI.ZeroOne())
end
lmo = FrankWolfe.MathOptLMO(o)function f(x)return sum(0.5*(x.-diffw).ˆ2)end
function grad!(storage, x)@. storage = x-diffw
end⌃

x, _, result = Boscia.solve(f, grad!, lmo, verbose = true)

Sebastian Pokutta · Boscia: Mixed-Integer Conditional Gradients⇧

18 / 23

Example: planted solution in high-dimensional space
Boscia.jl

julia> include("examples/low_dim_in_high_dim.jl")
Boscia Algorithm.
Parameter settings.
Tree traversal strategy: Move best boundBranching strategy: Most infeasible
Absolute dual gap tolerance: 1.000000e-06
Relative dual gap tolerance: 1.000000e-02
Frank-Wolfe subproblem tolerance: 1.000000e-05
Total number of varibales: 12
Number of integer variables: 0
Number of binary variables: 12--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------Iteration
Open
Bound
Incumbent
Gap (abs)
Gap (rel)Time (s)Nodes/sec
FW (ms)LMO (ms) LMO (calls c)
FW (Its)#ActiveSet Discarded--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------1
2 -2.297891e+03 -1.977958e+03
3.199328e+02
1.617490e-01
9.150000e-01
3.278689e+00
758
3
11
10001
1
0
*
2
3 -2.297891e+03 -2.238338e+03
5.955322e+01
2.660600e-02
1.442000e+00
3.467406e+00
526
11
46
10001
17
0
*
4
5 -2.292074e+03 -2.239976e+03
5.209869e+01
2.325860e-02
2.500000e+00
3.600000e+00
531
6
101
10001
11
0
*
5
6 -2.292074e+03 -2.242301e+03
4.977302e+01
2.219729e-02
3.024000e+00
3.637566e+00
523
10
133
10001
12
0
*
6
7 -2.292074e+03 -2.242301e+03
4.977302e+01
2.219729e-02
3.544000e+00
3.668172e+00
519
11
164
10001
4
0
*
16
17 -2.282246e+03 -2.243023e+03
3.922226e+01
1.748634e-02
8.726000e+00
3.781802e+00
524
8
439
10001
6
2
*
21
22 -2.280549e+03 -2.243325e+03
3.722381e+01
1.659314e-02
1.131100e+01
3.801609e+00
514
5
564
10001
7
1
*
29
30 -2.279719e+03 -2.244814e+03
3.490433e+01
1.554887e-02
1.544900e+01
3.819017e+00
522
8
761
10001
3
1
*
66
67 -2.271953e+03 -2.245231e+03
2.672273e+01
1.190200e-02
3.451800e+01
3.853062e+00
517
8
1616
10001
1
2
*
100
101 -2.268603e+03 -2.245231e+03
2.337210e+01
1.040967e-02
5.204200e+01
3.862265e+00
516
6
2357
10001
2
1
119
120 -2.267387e+03 -2.245231e+03
2.215595e+01
9.868008e-03
6.184900e+01
3.864250e+00
530
6
2778
10001
4
0
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------Solution Statistics.
Solution Status: Optimal (tolerance reached)Primal Objective: -2245.23067557406
Dual Bound: -2267.3866295644566
Dual Gap (relative): 0.009868007876175864
Search Statistics.
Total number of nodes processed: 239
Total number of lmo calls: 2782
Total time (s): 61.853
LMO calls / sec: 44.9776082000873
Nodes / sec: 3.8640001293389163
LMO calls / node: 11.640167364016737

Sebastian Pokutta · Boscia: Mixed-Integer Conditional Gradients238

19 / 23

Example: computational results: Sparse Regression

2.20

10000
8000

2.15

6000
2.10

4000

Upper boundLower boundTotal lmo calls

2000

2.05

250

500

750

1000

1250

17500

3.20

15000
12500

3.15

10000
3.10

7500

3.05

5000

3.00

0
0

3.25

Upper boundLower boundTotal lmo calls

2500
0

2.95

1500

Lmo calls

12000

Dual gap

2.25

Lmo calls

Dual gap

Boscia.jl

0

Number of nodes

500

1000

1500

2000

Number of nodes

High-dimensional sparse regression problem
with ✓ 0 -constraints

High-dimensional sparse regression problem
over mixed-integer feasible region.

Sebastian Pokutta · Boscia: Mixed-Integer Conditional Gradients

20 / 23

Thank you!

Preprint: arxiv.org/abs/2208.11010,
Convex integer optimization with FW methods
Package available at github.com/ZIB-IOL/Boscia.jl[Hendrych et al., 2022]

In a nutshell. Minimize smooth convex objective over any MIP. Applications in engineering, sparse prediction models,statistics, and relaxation of combinatorial problems.

Sebastian Pokutta · Boscia: Mixed-Integer Conditional Gradients239

21 / 23

References I
F. Bach. On the effectiveness of Richardson extrapolation in machine learning. arXiv preprint 2002.02835v3, July 2020.
G. Braun and S. Pokutta. The matching polytope does not admit fully-polynomial size relaxation schemes. Proceeedings of SODA, 2015a.
G. Braun and S. Pokutta. The matching polytope does not admit fully-polynomial size relaxation schemes. IEEE Transactions on Information Theory, 61(10):1–11, 2015b.
G. Braun, S. Pokutta, and D. Zink. Inapproximability of combinatorial problems via small LPs and SDPs. Proceeedings of STOC, 2015.
G. Braun, R. Jain, T. Lee, and S. Pokutta. Information-theoretic approximations of the nonnegative rank. Computational Complexity, 26(1):147–197, 2017a.
G. Braun, S. Pokutta, and D. Zink. Lazifying Conditional Gradient Algorithms. Proceedings of the International Conference on Machine Learning (ICML), 2017b.
G. Braun, S. Pokutta, D. Tu, and S. Wright. Blended Conditional Gradients: the unconditioning of conditional gradients. Proceedings of ICML, 2019a.
G. Braun, S. Pokutta, and D. Zink. Lazifying Conditional Gradient Algorithms. Journal of Machine Learning Research (JMLR), 20(71):1–42, 2019b.
G. Braun, A. Carderera, C. W. Combettes, H. Hassani, A. Karbasi, A. Mokthari, and S. Pokutta. Conditional gradient methods. preprint, 8 2022.
A. Carderera, J. Diakonikolas, C. Y. Lin, and S. Pokutta. Parameter-free Locally Accelerated Conditional Gradients. to appear in Proceedings of ICML, 2 2021.
C. W. Combettes and S. Pokutta. Complexity of Linear Minimization and Projection on Some Sets. to appear in Operations Research Letters, 1 2021.
C. W. Combettes, C. Spiegel, and S. Pokutta. Projection-Free Adaptive Gradients for Large-Scale Optimization. preprint, 10 2020.
J. Diakonikolas, A. Carderera, and S. Pokutta. Locally Accelerated Conditional Gradients. Proceedings of AISTATS, 2020.
M. Frank and P. Wolfe. An algorithm for quadratic programming. Naval Research Logistics Quarterly, 3(1-2):95–110, 1956.
R. M. Freund, P. Grigas, and R. Mazumder. An extended Frank-Wolfe method with “in-face” directions, and its application to low-rank matrix completion. SIAM Journal on
Optimization, 27(1):319–346, 2017.
G. Gamrath, B. Hiller, and J. Witzig. Reoptimization techniques for mip solvers. In International Symposium on Experimental Algorithms, pages 181–192. Springer, 2015.
D. Garber and E. Hazan. Playing non-linear games with linear oracles. In 2013 IEEE 54th Annual Symposium on Foundations of Computer Science, pages 420–428.
IEEE, 2013.
D. Garber and O. Meshi. Linear-memory and decomposition-invariant linearly convergent conditional gradient algorithm for structured polytopes. In D. D. Lee,
M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems 29, pages 1001–1009. Curran Associates, Inc., 2016. URL http://papers.nips.cc/paper/

6115-linear-memory- and- decomposition- invariant- linearly- convergent- conditional- gradient- algorithm- for- structured- polytopes.
pdf.
E. Hazan and S. Kale. Projection-free online learning. In Proceedings of the 29th International Conference on Machine Learning, 2012.
E. Hazan and H. Luo. Variance-reduced and projection-free stochastic optimization. In International Conference on Machine Learning, pages 1263–1271, 2016.
D. Hendrych, H. Troppens, M. Besançon, and S. Pokutta. Convex integer optimization with frank-wolfe methods. preprint, 8 2022.
Sebastian Pokutta · Boscia: Mixed-Integer Conditional Gradients

22 / 23

References II

M. Jaggi. Revisiting Frank-Wolfe: projection-free sparse convex optimization. In Proceedings of the 30th International Conference on Machine Learning, pages 427–435,
2013.
T. Kerdreux, A. d’Aspremont, and S. Pokutta. Restarting Frank-Wolfe. Proceedings of AISTATS, 2019.
T. Kerdreux, A. d’Aspremont, and S. Pokutta. Projection-Free Optimization on Uniformly Convex Sets. to appear in Proceedings of AISTATS, 1 2021a.
T. Kerdreux, A. d’Aspremont, and S. Pokutta. Local and Global Uniform Convexity Conditions. preprint, 2 2021b.
S. Lacoste-Julien. Convergence rate of Frank-Wolfe for non-convex objectives. arXiv preprint arXiv:1607.00345, 2016.
S. Lacoste-Julien and M. Jaggi. On the global linear convergence of Frank-Wolfe optimization variants. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and
R. Garnett, editors, Advances in Neural Information Processing Systems 28, pages 496–504. Curran Associates, Inc., 2015. URL
http://papers.nips.cc/paper/5925- on- the- global- linear- convergence- of- frank-wolfe- optimization- variants.pdf.
G. Lan and Y. Zhou. Conditional gradient sliding for convex optimization. SIAM Journal on Optimization, 26(2):1379–1409, 2016.
E. S. Levitin and B. T. Polyak. Constrained minimization methods. USSR Computational Mathematics and Mathematical Physics, 6(5):1–50, 1966.
S. J. Reddi, S. Sra, B. Póczos, and A. Smola. Stochastic Frank-Wolfe methods for nonconvex optimization. In 2016 54th Annual Allerton Conference on Communication,
Control, and Computing (Allerton), pages 1244–1251. IEEE, 2016.
T. Rothvoss. The matching polytope has exponential extension complexity. In Symposium on Theory of Computing, pages 263–272, 2014.
K. Tsuji, K. Tanaka, and S. Pokutta. Pairwise Conditional Gradients without Swap Steps and Sparser Kernel Herding. To Appear in Proceedings of ICML, 5 2022.

Sebastian Pokutta · Boscia: Mixed-Integer Conditional Gradients240

23 / 23

The 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop on
Advances in Classical and Quantum Algorithms for
Optimization and Machine Learning
September 16th - 19th, 2022, Tokyo (The university of Tokyo), Japan,
and September 21st - 22nd, 2022, Fukuoka (Kyushu University), Japan

Bregman Proximal DC Algorithms andTheir Application to
Blind Deconvolution with
Nonsmooth Regularization
Shota Takahashi
The Graduate University for Advanced Studies, Japan
takahashi.shota@ism.ac.jp

Blind deconvolution is a technique to recover an original signal without knowing a
convolving filter. It is naturally formulated as a minimization of a quartic objectivefunction under some assumption. Because its differentiable part does not have a
Lipschitz continuous gradient, existing first-order methods are not theoreticallysupported. In this presentation, we reformulate the objective function as a difference of
convex (DC) functions and add nonsmooth regularization. Then, we apply the Bregman
proximal DC algorithm (BPDCA) and the BPDCA with extrapolation (BPDCAe),
whose convergences are theoretically guaranteed under the L-smooth adaptable (Lsmad) property. BPDCAe outperformed other existing algorithms in image deburring
applications. This talk is based on [1] and [2].

References[1] Shota Takahashi, Mituhiro Fukuda, and Mirai Tanaka, “New Bregman proximal type algorithms
for solving DC optimization problems”, Computational Optimization and Applications, online,
September 2022, https://doi.org/10.1007/s10589-022-00411-w[2] Shota Takahashi, Mituhiro Fukuda, and Shiro Ikeda, “Blind deconvolution with non-smooth
regularization via Bregman proximal DCAs”, Signal Processing, Volume 202, 108734, January2023, https://doi.org/10.1016/j.sigpro.2022.108734

241

Bregman Proximal DC Algorithms and Their Application to
Blind Deconvolution with Nonsmooth Regularization
The 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop on Advances inClassical and Quantum Algorithms for Optimization and Machine Learning
Shota Takahashi1
September 19th, 20221

Department of Statistical Science, The Graduate University for Advanced Studies (SOKENDAI)

Takahashi, S., Fukuda, M., and Tanaka, M. (2022).
New Bregman proximal type algorithms for solving DC optimization problems.
Computational Optimization and Applications, online.
Takahashi, S., Tanaka, M., and Ikeda, S. (2023).
Blind deconvolution with non-smooth regularization via Bregman proximal DCAs.
Signal Processing, 202:108734.

Di↵erence of convex functions (DC) optimization
Definition (DC optimization problem with a regularization term)
Given convex functions F1 , F2 , G : Rd ! (−1, +1], consider the following DC optimizaion problem:minz2cl CF1 (z) − F2 (z) + G (z),(1)

dwhere F1 is C , and C ⇢ R is a nonempty open convex set.
1

Exisiting algorithms (⇠k 2 @F2 (z k ) is a subgradient of F2 at z k ):• DC algorithm (DCA): z k+1 2 argminz2cl C {F1 (z) − h⇠ k , zi + G (z)}.

• Its subproblem is computationally demanding unless F1 and G have simple structures.

1• Proximal DCA: z k+1 = argminz2cl C {hrF1 (z k ) − ⇠ k , zi + G (z) + 2λkz − z k k22 }, λ 2 (0, L1 ).

• To guarantee its global convergence, it requires F1 is L-smooth (rF1 is Lipschitz continuous).
• When F1 is not L-smooth (L cannot be defined), it is not practical.

• Bregman proximal gradient (BPG): z k+1 = argminz2cl C {hrF (z k ), zi + G (z) + λ1 DH (z, z k )}.
• The Bregman distance DH (z, w ) := H(z) − H(w ) − hrH(w ), z − w i, where H is C 1 and convex.
• Does not require L-smoothness of F = F1 − F2 (when F2 is also C 1 ).

1 / 14242

Today’s contents
Definition (DC optimization problem)
Given convex functions F1 , F2 , G : Rd ! (−1, +1], consider the following DC optimizaion problem:minz2cl CF1 (z) − F2 (z) + G (z),(1)

where F1 is C 1 , and C ⇢ Rd is a nonempty open convex set.
Overview• Introduce proximal DCA using the Bregman distance.
• The Bregman distance DH (z, w ) := H(z) − H(w ) − hrH(w ), z − w i, where H is C 1 and convex.
• The proposed methods converge to a stationary point of (1) under the L-smooth adaptableproperty instead of L-smoothness.

• Application to blind deconvolution.
• For BPG, finding an appropriate H for F is difficult. Using DC decoposition, it is easier.

2 / 14

Proposed method: Bregman proximal DC algorithm
Bregman proximal DC algorithm (BPDCA) [Takahashi et al., 2022]
Input: z 0 2 C , λ > 0, and a convex and C 1 function H.
for k = 0, 1, 2, . . . , :Compute ⇠ k 2 @F2 (z k ) and�
⇢1
z k+1 = argmin hrF1 (z k ) − ⇠ k , zi + G (z) + DH (z, z k ) .
λz2cl C(2)

BPDCA is a method combined with BPG [Bolte et al., 2018] and proximal DCA [Wen et al., 2017].
• Minimizes a first-order approximation of the objective function,
F1 (z) − F2 (z) + G (z) ' hrF1 (z k ) − ⇠ k , z − z k i + F1 (z k ) − F2 (z k ) +G (z),|{z}const.
��
with the Bregman proximality λ1 DH (z, z k ) = λ1 H(z) − H(z k ) − hrH(z k ), z − z k i as (2).

(3)

• Does not require the di↵erentiability of G .

• When G has a suﬃciently simple structure, such as G = k · k1 , (2) is solved in a closed form.
3 / 14243

Proposed method: BPDCA with extraporation
The accelerated version of BPDCA.
BPDCA with extrapolation (BPDCAe) [Takahashi et al., 2022]
Input: z −1 = z 0 2 C , λ > 0, t−1 = t0 = 1, and a convex and C 1 function H.
for k = 0, 1, 2, . . . , :p
1+

1+4t 2k
Set w k = z k + tk t1k−1 (z k − z k−1 ) and tk+1 =.
2kkCompute ⇠ 2 @F2 (z ) and�
⇢1
z k+1 = argmin hrF1 (w k ) − ⇠ k , zi + G (z) + DH (z, w k ) .
λz2cl C(4)

Reset tk−1 = tk = 1 and w k = z k if either of the following conditions holds:• k ⌘ 0 (mod N), N 2 N is given.

• DH (z k , w k ) > ⇢DH (z k−1 , z k ), where ⇢ 2 [0, 1) is given.

/ C.
• wk 24 / 14

Convergence analysis: Decreasing property
Definition (L-smooth adaptable (L-smad) [Bolte et al., 2018])
Let F , H : Rd ! (−1, +1] be C 1 and H be convex. The pair (F , H) is called L-smooth adaptable(L-smad) if there exists L > 0 such that LH − F and LH + F are convex.
The L-smad property is a generalization of L-smoothness.
When H = 12 k · k22 , the L-smad property corresponds to L-smoothness.
If the pair (F1 , H) is L-smad, the sequence of the objective function value
generated by BPDCA is decreasing.

:= F1 − F2 + G

Lemma (Decreasing property [Takahashi et al., 2022])
Let {z k }k be a sequence generated by BPDCA. Then, it holds thatλ (z k+1 )  λ (z k ) − (1 − λL)DH (z k+1 , z k ).

In particular, the decreasing property in the objective function value(5)

is ensured with 0 < λL < 1.
5 / 14244

Convergence analysis: Global convergence
When the objective function is a Kurdyka-Lojasiewicz (KL) function, the following theorems holds.
Theorem (Global convergence [Takahashi et al., 2022])
Let {z k }k be a sequence generated by BPDCA. Assume that the objective function is a KL function.
Then, {z k }k converges to a stationary point z̃.
Theorem (Rate of convergence [Takahashi et al., 2022])
Let {z k }k be a sequence generated by BPDCA and assume that {z k }k converges to a stationary
point z̃. Assume that the objective function is a KL function with φ(s) = cs 1−✓ for some ✓ 2 [0, 1)
and c > 0. Then, the following statements hold:• If ✓ = 0, there exists k0 > 0 such that z k is constant for k > k0 (finite);• If ✓ 2 (0, 12 ], there exist c1 , k1 > 0, and ⌘ 2 (0, 1) such that kz k − z̃k2 < c1 ⌘ k for k > k1 (linear);1✓• If ✓ 2 ( 12 , 1), there exist c2 > 0 and k2 > 0 such that kz k − z̃k2 < c2 k − 2✓ 1 for k > k2 (sublinear).

For BPDCAe, similar convergence results hold.
6 / 14Blind deconvolution
Definition (Blind deconvolution)
Consider the convolution of a filter f 2 Rm and a signal g 2 Rm , given byỹ = f ⇤ g ,(6)where ⇤ denotes the convolution. Our goal is to recover g from ỹ without knowing f .
• Non-blind deconvolution: Recover g with known f .
• Blind deconvolution: Recover g and f simultaneously. It is a charenging problem.
Application: Astronomy, communication engineering, and image processing.

Figure 1: Recover the original image g from the blurred image ỹ by removing the blurring kernel f .
7 / 14245

Optimization problem for blind deconvolution
For the linear operators B̃ and Ã, assume that there exist the true (h ◦ , x ◦ ) 2 Rd1 ⇥ Rd2 such thatf = B̃h ◦ , g = Ãx ◦ .
p
Applying the discrete Fourier transform mF 2 Cm⇥m to both sides of ỹ = f ⇤ g ,y = Bh ◦ � Ax ◦ ,

(7)
(8)

where y := p1m F ỹ , B := F B̃, and A := F Ã, and � denotes the Hadamard (elementwise) product,and ¯· denotes the complex conjugate.
Definition (Optimization problem for blind deconvolution)
1
kBh � Ax − y k22 +G (h, x).
min(h,x)2cl C
|2
{z
}

(9)

=:F (h,x)• F is nonconvex and not L-smooth.

• G : Rd1 ⇥ Rd2 ! (−1, +1] is convex and non-smooth (not di↵erentiable) as a sparse regularizer.
• Use sparse regularization (for example, `1 norm) when h or x have sparse structures.
• [Li et al., 2019] used a smooth G using `2 norm.

8 / 14

DC decomposition for blind deconvolution
Because F has a bilinear term 2 RehBh � Ax, y i, it is diﬃcult to find H satisfying the L-smad property.
! BPG [Bolte et al., 2018] is not applicable.
DC decomposition for blind deconvolution
Our optimization problem for blind deconvolution:
1min
kBh � Ax − y k22 +G (h, x),(h,x)2cl C
|2
{z
}

(9)

F1 (h, x) − F2 (h, x) + G (h, x).

(10)=:F (h,x)

F has a DC decomposition F = F1 − F2 for two convex functions F1 and F2 :1
1
1
F1 (h, x) = kBhk44 + kAxk44 + (kBh � Axk22 + ky � Bhk22 + kAxk22 + ky k22 ),
4
4
2
1
1
1
F2 (h, x) = kBhk44 + kAxk44 + kȳ � Bh + Axk22 .
4
42
Problem (9) is equivalent to the following DC optimization problem:min(h,x)2cl C

9 / 14246

L-smad property for blind deconvolution
Consider the L-smad property of (F1 , H) to apply BPDCA(e).
Theorem (L-smad property [Takahashi et al., 2023])
Let H be defined by�2 1 �
�1�
khk22 + kxk22 +khk22 + kxk22 .
4
2
By denoting bj and aj be the jth column vectors of B H and AH , respectively, then for any L
satisfyingmX��
3kbj k42 + 3kaj k42 + kbj k22 kaj k22 + |yj |2 kbj k22 + kaj k22 ,L≥H(h, x) =

(11)

(12)

j=1

the pair (F1 , H) is L-smad.

BPDCA(e) converges to a stationary point of (9).
Adjuste L in our numerical experiments.
Backtracking can be applied to BPDCA(e) but its calculations are sometimes expensive.
10 / 14

Numerical experiments: Blind deconvolution
Setting• The blurring kernel h 2 Rd1 has its elements in• The wavelet coefficients x 2 Rd2 (d2 = 2562 ).

pp
p
d1 ⇥ d1 pixels ( d1 = 48).

• B̃ is an operator reshaping h, and Ã is an inverse discrete wavelet transform operator.
• Therefore, f = B̃h ◦ , g = Ãx ◦ . The pixcels of the original image is 512 ⇥ 512.

• The regularizer G (h, x) = ✓khk1 .

• The feasible region cl C = {(h, x) 2 Rd1 ⇥ Rd2 | h ≥ 0, x ≥ 0}.

Figure 2: Recover (h ◦ , x ◦ ) from y = p1m F ỹ .
11 / 14247

Comparisons of algorithms
Algorithms• BPDCAe [Takahashi et al., 2022]: The accelerated version of BPDCA.
• BPDCA [Takahashi et al., 2022]• FISTA [Beck and Teboulle, 2009]: Adjusts the step size by backtracking.
• Alternating minimization (AM): Minimizes with respect to h and x alternately (itssubproblems are solved by FISTA (10 iterations); the number of the maximum iteration is 3,000).

•(h, x) = 12 kBh � Ax − y k22 + ✓khk1 .

• (h ◦ , x ◦ ) is the ground truth,◦=(h ◦ , x ◦ ).

• ✓ = 0.01.

12 / 14Comparisons of the recovered images(a) Ground truth(b) Intial point(c) BPDCAe(d) BPDCA(e) FISTA(f) AM

Figure 3: The upper row: the recovered hs. The lower row: the recovered xs.
13 / 14248

Summary and future work
Summary• Introduce BPDCA(e) for solving a DC optimization problem without L-smoothness.
• Reformulate blind deconvolution as a DC optimization problem with non-smooth regularization
and apply it to BPDCA(e).
Future work
• How to choose the Bregman distance DH .
• Application to self-calibration in radio interferometric imaging.
Takahashi, S., Fukuda, M., and Tanaka, M. (2022).
New Bregman proximal type algorithms for solving DC optimization problems.
Computational Optimization and Applications, online.
Takahashi, S., Tanaka, M., and Ikeda, S. (2023).
Blind deconvolution with non-smooth regularization via Bregman proximal DCAs.
Signal Processing, 202:108734.

References249

14 / 14References i[Beck and Teboulle, 2009] Beck, A. and Teboulle, M. (2009).
A fast iterative shrinkage-thresholding algorithm for linear inverse problems.
SIAM Journal on Imaging Sciences, 2(1):183–202.
[Bolte et al., 2018] Bolte, J., Sabach, S., Teboulle, M., and Vaisbourd, Y. (2018).
First order methods beyond convexity and Lipschitz gradient continuity with applications
to quadratic inverse problems.
SIAM Journal on Optimization, 28(3):2131–2151.
[Li et al., 2019] Li, X., Ling, S., Strohmer, T., and Wei, K. (2019).
Rapid, robust, and reliable blind deconvolution via nonconvex optimization.
Applied and Computational Harmonic Analysis, 47(3):893–934.
[Takahashi et al., 2022] Takahashi, S., Fukuda, M., and Tanaka, M. (2022).
New Bregman proximal type algorithms for solving DC optimization problems.
Computational Optimization and Applications, online.

References ii[Takahashi et al., 2023] Takahashi, S., Tanaka, M., and Ikeda, S. (2023).
Blind deconvolution with non-smooth regularization via Bregman proximal DCAs.
Signal Processing, 202:108734.
[Wen et al., 2017] Wen, B., Chen, X., and Pong, T. K. (2017).
A proximal di↵erence-of-convex algorithm with extrapolation.
Computational Optimization and Applications, 69(2):297–324.

250

The 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop on
Advances in Classical and Quantum Algorithms for
Optimization and Machine Learning
September 16th - 19th, 2022, Tokyo (The university of Tokyo), Japan,
and September 21st - 22nd, 2022, Fukuoka (Kyushu University), Japan

Theoretical Analysis
for Representation Learning Methodsof Graph-Structured Data
Nozomi HATA
Institute of Mathematics for Industry, Kyushu University, Japann.hata@kyudai.jp
Graph-structured data is one of the representative discrete data structures, while
continuous data is usually represented as vectors. Continuous representation of graphstructured data refers to the assignment of vectors to nodes in the graph-structured dataso that the relationship between two nodes can be recovered using these vectors. By
representing graph-structured data in continuous space, we can apply various algorithms
in continuous space to real-world applications such as link prediction, attributeprediction, information retrieval, and question answering while preserving
combinatorial characteristics of graph-structured data. In this presentation, we focus on
the theoretical representational power of representation methods. Some representation
methods, such as [1] or [2], can represent any inputs accurately. Such a property is calledfull expressiveness. We theoretically proved that some representation methods which
are not fully expressive are, in fact, almost fully expressive. This presentation introduces
almost fully expressive models and shows numerical results for link prediction tasks.

References[1] Ivana Balažević, Carl Allen, and Timothy M. Hospedales, “TuckER: Tensor Factorization forKnowledge Graph Completion”, Proceedings of the 2019 Conference on Empirical Methods in
Natural Language Processing and the 9th International Joint Conference on Natural Language
Processing (EMNLP-IJCNLP). 2019.
[2] Ralph Abboud, Ismail Ceylan, Thomas Lukasiewicz, and Tommaso Salvatori, "BoxE: A Box
Embedding Model for Knowledge Base Completion." Advances in Neural Information
Processing Systems 33 (2020): 9649-9661.

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

The 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop on
Advances in Classical and Quantum Algorithms for
Optimization and Machine Learning
September 16th - 19th, 2022, Tokyo (The university of Tokyo), Japan,
and September 21st - 22nd, 2022, Fukuoka (Kyushu University), JapanPort Set Clustering for Internet-Wide
Scanner
Akira TANAKA, Chansu Han, Takeshi
Takahashi
National Institute of Information and Communications
Technology, Japantanaka.akira@nict.go.jp
Indiscriminate IoT attacks have increased in recent years. Adversaries confirm if
vulnerable destination ports are open as a preliminary step of the attack, and thisprocedure is called port scan. The darknet, also known as a network telescope, is used
for observing such port scan activities. It passively monitors network traffic with an
unreachable dark IP address block; thus, it receives not regular network traffic but
Internet-wide scans for attack or investigation. Our goal is to specify scan activities for
attack purposes by focusing on the destination ports of packets collected from a darknet.
We treat each source IP address as a multiset made from the pairs of the destination port
and the number of packets. We create a distance on the multisets and perform clustering
using the distance. Multisets contribute to more fine clustering than clustering using port
sets or the number of packets. We also propose the speedup technique for clustering
based on the property of the distance.

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

The 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop on
Advances in Classical and Quantum Algorithms for
Optimization and Machine Learning
September 16th - 19th, 2022, Tokyo (The university of Tokyo), Japan,
and September 21st - 22nd, 2022, Fukuoka (Kyushu University), Japan

Finding Densest k-Connected
Subgraphs
Atsushi MIYAUCHI
Graduate School of Information Science and Technology,
University of Tokyo, Japanmiyauchi@mist.i.u-tokyo.ac.jp
Dense subgraph discovery is an important graph-mining primitive with a variety of realworld applications. One of the most well-studied optimization problems for dense
subgraph discovery is the densest subgraph problem, where given an edge-weighted
undirected graph, we are asked to find a subgraph that maximizes the average degree.
Although this problem can be solved exactly in polynomial time and well-approximately
in almost linear time, a densest subgraph has a structural drawback, namely, thesubgraph may be disconnected by removing only a few vertices/edges within it. In thistalk, we propose an algorithmic framework to find a dense subgraph that is wellconnected in terms of vertex/edge connectivity. This talk is based on joint work [1] withFrancesco Bonchi (CENTAI), David García-Soriano (ISI Foundation), andCharalampos E. Tsourakakis (Boston University).

References[1] Francesco Bonchi, David García-Soriano, Atsushi Miyauchi, and Charalampos E. Tsourakakis,“Finding Densest k-Connected Subgraphs,” Discrete Applied Mathematics 305, pp. 34–47, 2021,
https://doi.org/10.1016/j.dam.2021.08.032.

301

Finding Densest k-Connected Subgraphs
Discrete Applied Mathematics 305, pp. 34–47, 2021

Francesco Bonchi1
David Garcı́a-Soriano2
Atsushi Miyauchi3
Charalampos E. Tsourakakis2,4
1

CENTAI2

ISI Foundation

3

University of Tokyo

4

Boston University

Sep. 19, 2022
1 / 26Introduction

2 / 26302

Dense subgraph discovery

Dense subgraphs in real-world:• social groups
in friendship networks• communities & spam link farms
in Web graphs• molecular complexesin protein interaction networks

Dense subgraph discovery is a fundamental task in graph mining

3 / 26

How to detect dense subgraphs

The most common way is to utilize optimization theory:
Step 1: Introduce a quality function
Step 2: Define an optimization model (with or without constraints)
Step 3: Solve the model exactly or approximately
Many optimization models and algorithms have been developed

4 / 26303

Densest subgraph problem
Let G = (V , E, w) be an edge-weighted graph (w : E → Q>0 )Problem (Densest subgraph)
Input: G = (V , E, w)
Output: S ⊆ V that maximizes d(S) :=w(S)(density)|S|

w(S): sum of edge weights in the induced subgraph G[S]• Polynomial-time exact algorithms• LP-based algorithm [Charikar ’00]
• flow-based algorithm [Goldberg ’84]

• Almost-linear-time 1/2-approximation algorithm [Charikar ’00]
An optimal solution is referred to as a densest subgraph
5 / 26Drawback of densest subgraphs
Densest subgraphs are not necessarily well-connected

Barbell graph• The entire graph is the densest subgraph• Removing only one edge or two vertices separates it
Densest subgraphs may not be robust to vertex/edge failure

6 / 26304

Not only in theory...
Densest subgraphs in real-world Web graphs:

web-BerkStan

web-Google

web-NotreDame

web-Stanford

7 / 26

Terminology
Let G = (V , E) be a graph• S ⊂ V is called a vertex separator if its removal divides G into atleast two nonempty subgraphs between which there are no edges• The vertex connectivity κ(G) is the smallest cardinality of avertex separator of G if G is not a clique and |V | − 1 otherwise• G is said to be k-vertex-connected if κ(G) ≥ k

Table: Statistics of the four densest subgraphs S DS ⊆ V
Graph
web-BerkStan
web-Google
web-NotreDame
web-Stanford|S DS |

392
123
1,367
597

|E(S DS )|40,535
3,449
107,526
35,456d(S)κ(G[S DS ])min-deg(G[S DS ])103.41
28.04
78.66
59.39

12
30
1
60

201
30
155
60

Note: κ(G[S DS ]) ≤ min-deg(G[S DS ])8 / 26305

Our contribution
Problem (Densest k-vertex-connected subgraph)
Input: G = (V , E, w) and k ∈ Z>0
Output: S ⊆ V that maximizes d(S) =w(S)under κ(G[S]) ≥ k|S|• Generalization of Mader’s theorem• Algorithm for finding a Mader subgraph• Bicriteria approximation algorithm• Approximation algorithm

The edge-connectivity counterparts are obtained but omitted

9 / 26

Mader’s theorem & Mader subgraph

10 / 26306

Mader’s theorem

Theorem [Mader ’72]
Let G = (V , E) be a graph and let d ∈ Z>0

If d(V ) ≥ d, then G has a (d/2 + 1)-vertex-connected subgraphwhose minimum degree of vertices is greater than dThis theorem says that• dense graphs contain well-connected subgraphs• (although dense graphs are not necessarily well-connected)

11 / 26

Generalization to edge-weighted graphs

Theorem [This work]
Let G = (V , E, w) be an edge-weighted graph and let d ∈ R>0

If d(V ) ≥ d, then G has a (d/wmax /2 + 1)-vertex-connected
subgraph whose minimum weighted degree of vertices is greaterthan dProof strategy is the same as that of the original theorem
A subgraph whose existence is guaranteed by our theorem isreferred to as a Mader subgraph

12 / 26307

Algorithm for finding Mader subgraph
Let G∗ be a Mader subgraph of G = (V , E, w)
Strategy• Our algorithm keeps a family H of subgraphs of G

• Exactly one subgraph in H contains G∗ as its subgraph• In each iteration, our algorithm tests whether a subgraph in H isa Mader subgraph or not
YES: Output the subgraph
NO: Divide the subgraph into strictly smaller pieces

This is based on the algorithm for finding the most highly connected
subgraph in terms of vertex connectivity [Matula ’78]

13 / 26

Algorithm for finding Mader subgraphAn important subprocedure:Peel(G, d)Input: G = (V , E, w) and d ∈ R>0
Output: G[S] or Null
S ← V;
while S = ∅vmin ← argminv ∈S degS (v );// degS (v ) is the weighted degree of v in G[S]if degS (vmin ) > d then return G[S];S ← S \ {vmin };return Null;• Often used for dense subgraph discovery (e.g., [Charikar ’00])• This algorithm runs in O(|E| + |V | log |V |) time

14 / 26

308

Algorithm for finding Mader subgraph
Mader subgraph(G)Input: G = (V , E, w)
Output: G[S]H ← Peel(G, d(V
 ));

max 
τ ← d(V )/w+ 1; // vertex connectivity guaranteed by our theorem
2
H ← family of the connected components of H that have at least τ + 1 vertices;
if there exists a clique K in H then return K ;15 / 26

Algorithm for finding Mader subgraph
Mader subgraph(G)Input: G = (V , E, w)
Output: G[S]H ← Peel(G, d(V
 ));
max 
τ ← d(V )/w+ 1; // vertex connectivity guaranteed by our theorem
2
H ← family of the connected components of H that have at least τ + 1 vertices;
if there exists a clique K in H then return K ;while True
H  ← an arbitrary element of H;
C ← the minimum vertex separator of H  ;if |C| ≥ τ then return H  ;S ← family of the connected components of G[V (H  ) \ C];// V (H  ) denotes the vertex set of H H ← ∅;
for each S ∈ S
if T := Peel(G[S ∪ C], d(V )) has at least τ + 1 vertices then H ← H ∪ {T };if there exists a clique K in H then return K ;H ← (H \ {H  }) ∪ H ;15 / 26309

Analysis

Theorem
Mader subgraph(G) outputs a Mader subgraph of G in poly time
Proof (sketch):• It suffices to show the while-loop terminates in polynomial time

• The time complexity of each iteration is dominated by computing
the minimum vertex separator (i.e., polynomial)• The number of iterations is bounded by |V |Note: The actual time complexity is O(|V |19/4 )16 / 26

Bicriteria approximation algorithm

17 / 26

310

Algorithm (with parameter γ ∈ [1, 2])
Input: G = (V , E, w) and k ∈ Z>0
Output: S ⊆ V or INFEASIBLE

Find the family of maximal k-vertex-connected subgraphs{G[S1 ], . . . , G[Sp ]}; // Use the algorithm by [Makino ’88]
if there is no k-vertex-connected subgraph found thenreturn INFEASIBLE;
for i = 1, . . . , pSi∗ ← Si ;Find a densest subgraph SiDS (without any constraint) in G[Si ];
d(SiDS )/wmax 
if k ≤ γ+ 1 then2Si∗ ← The vertex set of Mader subgraph(G[SiDS ]);return S ∈ argmaxS∈{S ∗ ,...,Sp∗ } d(S);1

18 / 26

Analysis
Theoremmin
Our algorithm is a polynomial-time γ4 · wwmax, 1/γ -bicriteria
approximation algorithm (∀γ ∈ [1, 2])
Let S ⊆ V be the output• d(S) ≥ γ4 · wwmin · OPT (OPT: optimal value of the original problem)max• G[S] is (k/γ)-vertex-connected

Note:


• We can get 14 · wwmin -approximation by setting γ = 1
max• The time complexity is O(|V |(|V |19/4 + TDS ))(TDS : time complexity of computing a densest subgraph in G)
19 / 26311

Analysis
Theoremmin
Our algorithm is a polynomial-time γ4 · wwmax, 1/γ -bicriteria
approximation algorithm (∀γ ∈ [1, 2])
Proof (sketch): Let S ⊆ V be the output• G[S] is (k/γ)-vertex-connected• It suffices to show G[Si∗ ] is (k/γ)-vertex-connected (∀i = 1, . . . , p)



DS
• If k ≤ γ d(Si 2)/wmax  + 1 does not hold, Si∗ is given by Si ; so OK• Otherwise Si∗ is the vertex set of Mader subgraph(G[SiDS ])• Apply the generalized Mader’s theorem:d(SiDS )/wmax +1-vertex-connected; so γk -vertex-connectedSi∗ is2
20 / 26

Analysis
Theoremmin
Our algorithm is a polynomial-time γ4 · wwmax, 1/γ -bicriteria
approximation algorithm (∀γ ∈ [1, 2])• d(S) ≥ γ4 · wwmin · OPT
max• Let OPTi be the optimal value of the original problem on G[Si ]• It suffices to show d(Si∗ ) ≥ γ4 · wwmin · OPTi
max



DS
• If k ≤ γ d(Si 2)/wmax  + 1 does not hold, Si∗ is given by Si ;so every vertex in G[Si∗ ] has weighted degree of at leastminminwmin k > · · · > γ2 · wwmax· OPTi , implying d(G[Si∗ ]) ≥ γ4 · wwmax· OPTi• Otherwise Si∗ is the vertex set of Mader subgraph(G[SiDS ])• Apply the generalized Mader’s theorem:Si∗ has the minimum weighted degree of at least d(SiDS );so d(Si∗ ) ≥ d(SiDS )/2 ≥ OPTi /2 (irrespective of edge weights)

312

20 / 26

Approximation algorithm

21 / 26

Algorithm

Input: G = (V , E, w) and k ∈ Z>0
Output: S ⊆ V or INFEASIBLE

S ← argmax{κ(G[S]) | S ⊆ V }; // Use the algorithm by [Matula ’78]
if κ(G[S]) ≥ k then return S;
return INFEASIBLE;
This runs in O(|V |2 (κ(G)2 · min{|V |3/4 , κ(G)3/2 } + κ(G)|V |)) time

22 / 26313

Analysis
Theorem
Our algorithm is a polynomial-timealgorithm



wmin
6
19 · wmax

Note: This is better than the previous-approximation

1 wmin4 · wmax-approximation

Proof (sketch):• Theorem by [Bernshteyn & Kostochka ’16]:

Let G = (V , E) be a graph and t ∈ Z with t ≥ 2

If G satisfies |V | ≥ 52 t and |E| > 19
12 t(|V | − t), then G has a(t + 1)-vertex-connected subgraph• Use the theorem as in the analysis of the bicriteria approximation
23 / 26

Conclusion

24 / 26314

Summary
Problem (Densest k-vertex-connected subgraph)
Input: G = (V , E, w)
Output: S ⊆ V that maximizes d(S) under κ(G[S]) ≥ k• Generalization of Mader’s theorem• Algorithm for finding a Mader subgraph• γ4 · wwmin , 1/γ -bicriteria approximation algorithm (γ ∈ [1, 2])
max


6
min
• 19
-approximation algorithm· wwmax
The edge-connectivity counterparts are obtained but omitted

25 / 26

Future work

• Design better (bicriteria or ordinary) approximation algorithms• Conduct experiments to investigate practical performance• Analyze the computational complexity (NP-hardness etc.)

Thank you!

26 / 26

315

The 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop on
Advances in Classical and Quantum Algorithms for
Optimization and Machine Learning
September 16th - 19th, 2022, Tokyo (The university of Tokyo), Japan,
and September 21st - 22nd, 2022, Fukuoka (Kyushu University), Japan

Worst-case constructions for linear
optimization
Antoine DEZA
McMaster University, Canada
deza@mcmaster.ca

Worst-case constructions have helped providing a deeper understanding of how thestructural properties of the input affect the computational performance of linear
optimization. Recent examples include the construction of Allamigeon et al. for which
the interior point method performs an exponential number of iterations, and thus is notstrongly polynomial. In a similar spirit, recent lower bounds on the number of simplex
pivots required in the worst-case to perform linear optimization over a lattice polytope
will be presented, as well as the first worst-case instances for geometric scaling methods
that solve integer optimization problems by primal augmentation steps.

References[1] Antoine Deza, Mingfei Hao, and Lionel Pournin: Sizing the White Whale. Fields InstituteCommunications Series on Data Science and Optimization (to appear).
[2] Antoine Deza, Sebastian Pokutta, and Lionel Pournin: The complexity of geometric scaling.
arXiv:2205.04063 (2022).
[3] Antoine Deza and Lionel Pournin: Primitive point packing. Mathematika 68 (2022) 979 – 1007.
[4] Antoine Deza, Lionel Pournin, and Noriyoshi Sukegawa: The diameter of lattice zonotopes.
Proceedings of the American Mathematical Society 148 (2020) 3507 – 3516.
[5] Anna Deza, Antoine Deza, Zhongyan Guan, and Lionel Pournin: Distances between vertices oflattice polytopes. Optimization Letters 14 (2020) 309 – 326.
[6] Antoine Deza, Asaf Levin, Syed Meesum, and Shmuel Onn: Optimization over degree
sequences. SIAM Journal on Discrete Mathematics 32 (2018) 2067 – 2079.
[7] Antoine Deza, George Manoussakis, and Shmuel Onn: Primitive zonotopes. Discrete andComputational Geometry 60 (2018) 27 – 39.

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

The 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop on
Advances in Classical and Quantum Algorithms for
Optimization and Machine Learning
September 16th - 19th, 2022, Tokyo (The university of Tokyo), Japan,
and September 21st - 22nd, 2022, Fukuoka (Kyushu University), Japan

Approximate Methods for Solving
Chance Constrained Linear Programsin Probability Measure Space
Xun ShenTokyo Institute of Technology, Japan
shen.x.ad@m.titech.ac.jp
Many risk-aware decision-making problems can be formulated as a chanceconstrained linear program in probability measure space, which is NP-hard andunsolvable directly. In this talk, we introduce approximate methods to address this NPhard problem. In the proposed methods, the original problem is approximated by two
kinds of solvable optimization problems in finite-dimension space. We show theconvergence of the approximations and give numerical experiments including astochastic control problem for validation. Two numerical examples are presented toshow the effectiveness of the proposed method.

337

338

339

340

341

342

343

344

345

346

347

The 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop on
Advances in Classical and Quantum Algorithms for
Optimization and Machine Learning
September 16th - 19th, 2022, Tokyo (The university of Tokyo), Japan,
and September 21st - 22nd, 2022, Fukuoka (Kyushu University), Japan

A generalized Levenberg–Marquardt
method for large-scale composite
minimization
Naoki MARUMO
University of Tokyo / NTT, Japan
marumo-naoki@g.ecc.u-tokyo.ac.jp

We propose a new generalized Levenberg–Marquardt method for minimizing the sumof a smooth composite function and a convex function. The method enjoys threetheoretical guarantees: iteration complexity bound, oracle complexity bound, and local
convergence under an error bound condition [1]. Numerical results show that the
proposed method performs well for some large-scale problems.

References[1] N. Yamashita and M. Fukushima. On the rate of convergence of the Levenberg–Marquardt
method. In G. Alefeld and X. Chen, editors, Topics in Numerical Analysis, pages 239–249, Vienna,
2001. Springer Vienna.

349

Sep 19, 2022

6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop

A generalized Levenberg–Marquardt method
for large-scale composite minimization
Naoki Marumo (NTT / UTokyo, Japan)
joint work with Takayuki Okuno and Akiko Takeda

Outline: Problem setting & contribution

2 / 18Composite minimization
min g(x) + h(c(x))

x∈Rd• g : Rd → R ∪ {+∞}: convex
c : Rd → Rn :h : Rn → R:smooth (Lipschitz Jacobian)convex & smooth (Lipschitz gradient)• Nonconvex optimization problem with many applications• Levenberg–Marquardt (LM) method: efficient and widely used
(e.g., implemented in MATLAB and SciPy)• We propose a new LM method with both an oracle complexity boundand a local quadratic convergence guarantee 
350

Application: Empirical risk minimization in ML

3 / 18N

1 
(φx (ai ), bi )
N

min

x∈Rd

i=1

• (ai , bi ) ∈ Rp × Rq : training data (i = 1, . . . , N )• φx : Rp → Rq :•  : Rq × Rq → R:

machine learning model with parameter x
loss function• c(x) = (φx (a1 ), . . . , φx (aN )) ∈ Rq×N
• h(y) = h(y1 , . . . , yN ) =N
1 (yi , bi )N i=1

min h(c(x))

x∈RdLevenberg–Marquardt (LM) method

4 / 18• Iterative method that uses the composite structure of the problem• Construct a subproblem for the k-th iterate xk .
Set xk+1 to be an (approximate) solution to the subproblem
Subproblem for LM

linear approx. of c(x) damping term (µ > 0)
 µ
k
k
k
min g(x) + h c(x ) + ∇c(x )(x − x ) + x − xk 22
2
x∈Rd


cf. original problem
min g(x) + h(c(x))

x∈Rd• ∇c(xk ) ∈ Rn×d : Jacobian matrix• The subproblem is strongly convex and much easier than the original351

Levenberg–Marquardt (LM) method

5 / 18• Introduced for least-squares problems (g(·) = 0, h(·) = 12 ·22 )[Levenberg, 1944, Marquardt, 1963]• Extended to general g, h

[Nesterov, 2007, Lewis and Wright, 2016, Drusvyatskiy and Lewis, 2018]• Many other LM methods[Osborne, 1976, Yamashita and Fukushima, 2001, Dan et al., 2002, Kanzow
et al., 2004, Ueda and Yamashita, 2010, Behling and Fischer, 2012,
Drusvyatskiy and Paquette, 2019, Bergou et al., 2020, Marumo et al., 2020]...

Main differences between LMs

6 / 18Assumptions on g, h

How to set the damping parameter µ

Algorithm for subproblems

Theoretical guarantees

352

Features of our LM

6 / 18Assumptions on g, h
→ g: convex, h: smooth convex. Not restricted to least squares
How to set the damping parameter µ→ Adaptively
Algorithm for subproblems→ Accelerated proximal gradient with a particular termination cond.
Theoretical guarantees→ Iteration complexity, oracle complexity,
local quadratic convergence (under additional assumptions)First LM to achieve both oracle complexity and local quadratic conv.

cf.: iteration and oracle complexity
Iteration complexity
The number of iterations required to find an ε-stationary point• Used when we, at each iteration,
1

compute the Jacobian ∇c(xk ) ∈ Rn×d and2

solve the subproblem using it,

and the cost of 1 is dominant• Then,(Tocal cost)  (Iteration complexity) × (Cost for ∇c(xk ))• Mainly for small- or medium-scale problems (d, n  103 )353

7 / 18

cf.: iteration and oracle complexity

8 / 18

Oracle complexity
The number of oracle calls required to find an ε-stationary point
Oracles assumed in this work
• c(x), ∇c(x)u, ∇c(x) v(x, u ∈ Rd , v ∈ Rn )(y ∈ Rn )• h(y), ∇h(y)• proxηg (x) := argmin ηg(z) + 12 x − z2(x ∈ Rd , η > 0)z∈Rd

• Used when we, at each iteration,• do not compute ∇c(xk ) ∈ Rn×d explicitly and• solve the subproblem using ∇c(xk )u, ∇c(xk ) v
• Mainly for medium- or large-scale problems (d, n  103 )Comparison of theoretical guarantees
General g, h

9 / 18#Iteration#Oracle

Existing LM 1

[Yamashita and Fukushima, 2001]

2

O[Ueda and Yamashita, 2010]

3

O[Marumo et al., 2020]

O[Marumo et al., 2020]

5

Our LM

cf.: Prox. grad. 1



2



√
√



∆κ



∆κ


∆κ
 
O Kh
√ 
O ∆


O Kh κ 
√ 
O ∆κ

4

[Drusvyatskiy and Paquette, 2019]√• κ, κ ≥ 1: constants like a condition number

O

√

y∈R

354



∆κ

√ 
Õ Kh κ√ √ Õ ∆ κ
O K h κ
√ 
O ∆κ


• Kh , Lh : Lipschitz constants of h, ∇h. Normalized to Lh = 1• ∆ := g(x0 ) + h(c(x0 )) − min g(x) + min h(y)
n
x∈Rd

Local





Numerical example: Rosenbrock functionmin(x,y)∈R2

10 / 18(x − 1)2 + 100(x2 − y)2

1.2
10°1

Objective function value

1.0
0.8

y

0.6
0.4
0.2

Ours
DL
GD

0.0
°0.2
°0.2

0.0

0.2

0.4

0.6

0.8

1.0

1.2

10°4
10°7
10°10
Ours
DL
GD

10°13
10°16

0

100

200

300

400

500

Iterationx

Our LM converges faster than existing LM and Gradient Descent Previous parts of this talk:• Problem setting• LM method• Our contribution & comparison with existing work
Remaining parts:• Details of the proposed LM and theoretical guarantees• Numerical experiments with large-scale problems

355

Proposed LM method

12 / 18

k-th iteration of our LM
1 µk := ρ F (xk ) − (g ∗ + h∗ )(ρ: constant)



µk

k+1
d
k
2 x
∈ x ∈ R  ω̄k,µk (x) ≤x − x 2
2• F (x) := g(x) + h(c(x)),

g ∗ := min g(x),x∈Rdh∗ := minn h(y)y∈R• ω̄k,µk (x): (sub)gradient norm for the subproblem•

2

is computed by accelerated proximal gradient

Key point of the proposed LM method

13 / 18

k-th iteration of our LM
1 µk := ρ F (xk ) − (g ∗ + h∗ )(ρ: constant)



µk

2 xk+1 ∈ x ∈ Rd  ω̄k,µk (x) ≤x − xk 2
2
Difficulty on µ:• small µ → worsen cond. number of subproblem → increase #oracle• large µ → fail quadratic convergence
Difficulty on the accuracy of subproblem’s solution:• accurate→ increase costs per iteration → increase #oracle• inaccurate → fail quadratic convergence & increase #iteration1

and 2 strike a good balance!
356

Theoretical guarantee: Complexity (Main result 1)

14 / 18

Complexity for an ε-stationary point √• Iteration complexity: O Lc εL2 h ∆ (F (x0 ) − F ∗ )√• Oracle per iteration: O( κ log κ) √√
Lc Lh ∆∗• Oracle complexity:O(F (x0 ) − F ) κ log κε2• ε-stationary point: a point x ∈ Rd s.t. ω(x) ≤ εω(x) := min p + ∇c(x) ∇h(c(x))2p∈∂g(x)• Lc , Lh : Lipschitz constants of ∇c, ∇h,• ∆ := F (x0 ) − (g ∗ + h∗ ),√Lhκ := 1 + L √sup∇c(xk )2op
∆
c

k∈N

F ∗ := min F (x)x∈Rd

Theoretical guarantee: Local quad. conv. (Main result 2) 15 / 18
Assume:• Zero-residual: F (x∗ ) = g ∗ + h∗ for some x∗ ∈ Rd ,• x0 is sufficiently close to x∗ ,• Error bound condition.
Then, (F (xk )) converges to F∞ := g ∗ + h∗ . Futhermore, for some C > 0,2
F (xk+1 ) − F∞ ≤ C F (xk ) − F∞ , ∀k ≥ 0.

Def.: Error bound conditionγ
dist(xk , X ∗ )2 ≤ F (xk ) − (g ∗ + h∗ ),For some γ > 0,2

∀k ≥ 0.

• dist(xk , X ∗ ): distance between xk and the optimal solution set X ∗• Weaker than strong convexity
357

Numerical result 1: Classification with NN (MNIST)

Ours(10°3 )DP(10°2 , 100 )DP(10°2 , 100 )Test error rate (%)

Objective function value8

Ours(10°3 )100

16 / 18

PG(10°1 )

10°2

10°4

PG(10°1 )

7

6

5

10°6
0

5000

10000

15000

20000

25000

30000

0

5000

10000

Oracle cost

15000

20000

25000

30000

Oracle cost

N

1 
min −
bi , LogSoftmax(φx (ai ))
N i=1
x∈Rd
• (ai , bi ) ∈ [0, 1]784 × {0, 1}10 : training data. Pair of image and label• φx : R784 → R10 :• d = 104,938,

neural network with parameter x

n = 10N = 60,000

Numerical result 2: Nonneg. Matrix Factr. (MovieLens)
1.10

Ours(10°4 )DP(10°3 , 10°1 )

1.05

10°1

10°3

Test RMSE

Objective function value

101

17 / 18

Ours(10°4 )DP(10°3 , 10°1 )10

DP(10°1 , 10°1 )PG(10°3 )

1.00

0.95

DP(10°1 , 10°1 )°5

PG(10°3 )
0

1000

0.90
2000

3000

4000

5000

0

1000

Oracle cost

min
p×r

U ∈R+

, V ∈Rq×r
+

1
|Ω|

2000

3000

4000

Oracle cost (i,j,s)∈Ω

ui , vj  − s2
+ λ U 2F + V 2F• Ω: training dataset. (i, j, s) ∈ Ω: user i rated movie j as s ∈ {1, . . . , 5}

• λ = 10−10 : regularization parameter• d = pr + qr = 1,312,500,

n = |Ω| = 80,000
358

5000

Conclusion

18 / 18

min g(x) + h(c(x))

x∈Rd

Our LM has iteration and oracle complexity bounds and a local
quadratic convergence guarantee
k-th iteration of our LM
1 µk := ρ F (xk ) − (g ∗ + h∗ )(ρ: constant)



µk

k+1
d
2 x
x − xk 2∈ x ∈ R  ω̄k,µk (x) ≤2• Subproblems are solved by an accelerated proximal gradient method• Parameter µ and the accuracy of subproblem’s solution are carefully set• Our LM is practical for large-scale problems (d  105 –106 )Appendix: Lemmas for complexity bounds

19 / 18

Iteration complexity is derived from Lemma 1, andoracle complexity is derived from Lemmas 1 and 2
Lemma 1 (this work)
F (xk+1 ) ≤ F (xk ) −µk k+1x− xk 22 ,4

∀k ≥ 0

Lemma 2 (this work)
Oracle complexity for 2 is√O( κk log κk ),κk := 1 +359

Lh
∇c(xk )2opµk

Appendix: Example of zero-residual problem

20 / 18• f : Rd → Rn : given smooth & nonlinear function, n < d• x∗ ∈ R d :unknown signal, sparse• y := f (x∗ ):observation

Goal: Recover the sparse signal x∗ from the low-dimensional observation y
Nonlinear compressed sensing
min y − f (x)22 ,x∈Rds.t. x1 ≤ r• The constraint x1 ≤ r enhances the sparsity of x
• This problem is composite minimization and zero-residual

References I

21 / 18

R. Behling and A. Fischer. A unified local convergence analysis of inexact constrained
Levenberg–Marquardt methods. Optimization Letters, 6(5):927–940, 2012.
E. H. Bergou, Y. Diouane, and V. Kungurtsev. Convergence and complexity analysis ofa Levenberg–Marquardt algorithm for inverse problems. Journal of Optimization
Theory and Applications, 185(3):927–944, 2020.
H. Dan, N. Yamashita, and M. Fukushima. Convergence properties of the inexact
Levenberg–Marquardt method under local error bound conditions. Optimization
Methods and Software, 17(4):605–626, 2002.
D. Drusvyatskiy and A. S. Lewis. Error bounds, quadratic growth, and linear convergenceof proximal methods. Mathematics of Operations Research, 43(3):919–948, 2018.
D. Drusvyatskiy and C. Paquette. Efficiency of minimizing compositions of convex
functions and smooth maps. Mathematical Programming, 178(1-2):503–558, 2019.
C. Kanzow, N. Yamashita, and M. Fukushima. Levenberg–Marquardt methods withstrong local convergence properties for solving nonlinear equations with convex
constraints. Journal of Computational and Applied Mathematics, 172(2):375–397,
2004.
K. Levenberg. A method for the solution of certain non-linear problems in least squares.
Quarterly of Applied Aathematics, 2(2):164–168, 1944.
360

References II

22 / 18

A. S. Lewis and S. J. Wright. A proximal method for composite minimization.
Mathematical Programming, 158(1):501–546, 2016.
D. W. Marquardt. An algorithm for least-squares estimation of nonlinear parameters.
Journal of the Society for Industrial and Applied Mathematics, 11(2):431–441, 1963.
N. Marumo, T. Okuno, and A. Takeda. Majorization-minimization-based
Levenberg–Marquardt method for constrained nonlinear least squares. arXiv preprint
arXiv:2004.08259, 2020.
Y. Nesterov. Modified Gauss–Newton scheme with worst case guarantees for global
performance. Optimisation Methods and Software, 22(3):469–483, 2007.
M. R. Osborne. Nonlinear least squares—the Levenberg algorithm revisited. The Journalof the Australian Mathematical Society. Series B. Applied Mathematics, 19(3):343–357, 1976.
K. Ueda and N. Yamashita. On a global complexity bound of the Levenberg–Marquardt
method. Journal of Optimization Theory and Applications, 147(3):443–453, 2010.
N. Yamashita and M. Fukushima. On the rate of convergence of theLevenberg–Marquardt method. In G. Alefeld and X. Chen, editors, Topics inNumerical Analysis, pages 239–249, Vienna, 2001. Springer Vienna.

361

The 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop on
Advances in Classical and Quantum Algorithms for
Optimization and Machine Learning
September 16th - 19th, 2022, Tokyo (The university of Tokyo), Japan,
and September 21st - 22nd, 2022, Fukuoka (Kyushu University), JapanBIPSOL: A metaheuristic solver forlarge-scale binary integer programs
Shunji UMETANI
Graduate School of Information Science and Technology,
Osaka University, Japanumetani@ist.osaka-u.ac.jp
Metaheuristics have proven to a comprehensive approach to attain good solutions forhard combinatorial optimization problems. However, they are usually based on specificcharacteristics of the problem to be solved, which makes them hard to develop efficient
general purpose solvers for such as the mixed integer programs (MIPs) and the constraint
satisfaction problems (CSPs). In designing metaheuristics for combinatorial optimization
problems, the quality of solutions typically improves if larger and sophisticatedneighborhoods are used, while computation time of searching the neighborhood also
increases rapidly. BIPSOL is a metaheuristic solver for large-scale binary integer
programs (BIPs) that introduces a generalized technique of the neighbor-list used fortraveling salesman problem (TSP) to generate smaller and structured neighborhoods
automatically [1,2]. We incorporate an efficient incremental evaluation of solutions anda dynamic control mechanisms of penalty weights into BIPSOL. In this talk, we show
some progress of development in BIPSOL and future directions.

References[1] Shunji Umetani, Exploiting variable associations to configure efficient local search algorithm inlarge-scale binary integer programs, European Journal of Operational Research, 263 (2017), 7281. https://doi.org/10.1016/j.ejor.2017.05.025[2] Shunji Umetani, Exploiting variable associations to configure efficient local search algorithm inlarge-scale set partitioning problems, Proceedings of 29th AAAI Conference on Artificial
Intelligence (AAAI-15), 1226-1232.

363

364

365

366

367

368

369

370

371

372

373

374

375

376

The 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop on
Advances in Classical and Quantum Algorithms for
Optimization and Machine Learning
September 16th - 19th, 2022, Tokyo (The university of Tokyo), Japan,
and September 21st - 22nd, 2022, Fukuoka (Kyushu University), Japan

Performance of the supercomputer
Fugaku for Graph500 benchmark
Masahiro Nakao
RIKEN Center for Computational Science, Japanmasahiro.nakao@riken.jp
We present the performance of the supercomputer Fugaku for breadth-first search(BFS) in the Graph500 benchmark, which is known as a ranking benchmark used toevaluate large-scale graph processing performance on supercomputer systems. Fugaku
is a huge-scale Japanese exascale supercomputer that consists of 158,976 nodes. We
evaluate the BFS performance for a large-scale graph consisting of about 2.2 trillion
vertices and 35.2 trillion edges using the whole Fugaku system, and achieve 102,955
giga-traversed edges per second, resulting in the first position of Graph500 BFS
ranking[1, 2].

References[1] Masahiro Nakao, Koji Ueno, Katsuki Fujisawa, Yuetsu Kodama, Mitsuhisa Sato. ``Performanceof the Supercomputer Fugaku for Breadth-First Search in Graph500 Benchmark.'', ISC 2021, Jun.
2021, https://doi.org/10.1007/978-3-030-78713-4_20[2] https://graph500.org

377

Performance of the supercomputer Fugaku forGraph500 benchmark
Joint work with‡†
Masahiro Nakao†*
Koji Ueno, Katsuki Fujisawa, Yuetsu Kodama, †Mitsuhisa Sato
† RIKEN Center for Computational Science‡ Fixstars Corporation* Institute of Mathematics for Industry, Kyushu University
The 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop

Backgroundin supercomputers

http://opt.imi.kyushu-u.ac.jp/lab/jp/activities.html

2378

Parallelize BFS and SSSP algorithms on supercomputers
Parallelize fundamental graph algorithms on supercomputers

Breadth First Search (BFS) in cooperation with IMISingle-Source Shortest Path (SSSP) in cooperation with ZIB and IMILarge scale supercomputers
Graph algorithm that runs efficiently on such systems is a challenging research
The K computer : 82,944 nodes[1]Supercomputer Fugaku : 158,976 nodes[2][1] Koji Ueno et al: Efficient breadth-firstsearch on massively parallel anddistributed-memory machines.
Data Science and Engineering, 20162012- 20192020 -[2] Masahiro Nakao et al: Performance of theSupercomputer Fugaku for Breadth-First
Search in Graph500 Benchmark.'' ISC 2021

3

Graph500

https://graph500.org

Graph500 is a competition for evaluating performance of large-scale graph processing
The performance unit is a traversed edges per second (TEPS)
1GTEPS : Search 1 billion edges per second
Graph500 list is updated twice a year (June and November in ISC and SC)
BFS : The K computer ranked first 10 times from 2014 to 2019
BFS : Supercomputer Fugaku ranks first from 2020 to now
SSSP : Supercomputer Fugaku ranks 2nd in June 2022
In graph500, an artificial graph called the “Kronecker graph” is used
Some vertices are connected to many other vertices while numerous others areconnected to only a few vertices
Social network is known to have a similar property

4379

Objective
This presentation describes the performance tuning of BFS for the Graph500
submission and experimental evaluation results conducted on Fugaku

Summary
Use a large-graph with 2.2 trillion vertices and 35.2 trillion edges (SCALE=41)
Archive 102,955 GTEPSThe performance of Fugaku is 3.3 times better than that of the K computer

5

Outline
BFS in Graph500 Benchmark
The supercomputer Fugaku
Tuning BFS on the supercomputer Fugaku
Full node evaluation

6

380

Overview of BFS in Graph500

BFS

Input : Kronecker graph and root vertex

Output : BFS tree

Repeat BFS 64 times from different root vertex
The harmonic mean of the performance in 64 trials is used as the final performance7

Hybrid-BFS [Beamer, 2012] Scott Beamer et al. Direction-optimizing breadth-first searchHybrid-BFS runs while switching between Top-down and Bottom-up

Search for unvisited vertices
from visited vertices

Search for visited vertices
from unvisited vertices

In the middle of BFS, the number of vertices being visited increases explosively,
so it is inefficient in only Top-down
Hybrid-BFS switches between Top-down and Bottom-up on the situation

8381

Number of vertex checks
SCALE = 26

Graph with 2^{SCALE} vertices and 2^{SCALE+4} edges
67.1 million vertices and 1.1 billion edges
Top-down

Bottom-up

0

2

2,103,840,895

Hybrid-BFS
2

1

66,206

1,766,587,029

66,2062

346,918,235

52,667,691

52,667,691

3

1,727,195,615

12,820,854

12,820,854

4

29,557,400

103,184

103,1845

82,357

21,467

21,467

6

221

21,240221

Total

2,103,820,036

3,936,062,360

65,679,625

Rate

100.00%187.09%

3.12%

Hybrid-BFS decides when toswitch between Top-down and
Bottom-up from informationsuch as the number ofvertices being searched.

9

2D Hybrid-BFS [Beamer, 2013] Scott Beamer, et. al. Distributed Memory Breadth-First
Search Revisited: Enabling Bottom-Up Search. IPDPSW '13.

Adjacency matrix is distributed to a 2D process grid (R x C)

Communication only within the column processes and row processes

Allgatherv, Alltoallv, point-to-point (isend/irecv/wait)
The closer the R and C values are, the smaller the total communication size
Based on this 2D Hybrid-BFS, we implemented BFS with various ideas toimprove performance[1][1] Koji Ueno et al: Efficient breadth-first search on massively parallel anddistributed-memory machines. Data Science and Engineering, (2016)10382

Outline
BFS in Graph500 Benchmark
The supercomputer Fugaku
Tuning BFS on the supercomputer Fugaku
Full node evaluation

11

The supercomputer Fugaku
The supercomputer Fugaku, which is developed jointly by RIKEN and Fujitsu Limited
based on Arm technology
Located in RIKEN Center for Computational Science in Kobe, Hyogo, Japan
158,976 compute nodes
Start sharing in March 2021

12

383

Specification of Computer NodeEach node has a single CPU
Each CPU has 48 compute cores and 2/4 assistantcores. The assistant cores handle the interrupts OS andcommunication
2.0 GHz or 2.2 GHz for each job
Each CPU consists of 4 CMGs
Each CMG consists of 12 + 1 cores and 8GiB HBM2
It is recommended that the number of processes
per CPU is a divisor of 4
Each CPU has 10 network cables

CPU (A64FX)

L2 Cache Coherent control between CMGs

CMG : Core Memory GroupNOC : Network on Chip
TNI: Tofu Network Interface
13

Network topology of Fugaku
6D mesh/torus : XYZabc-axis
The size of abc is (2,3,2)
The size of XYZ is (24,23,24)
so it has 24*23*24*2*3*2 = 158,976 nodes
NodeProcess Mapping
Discrete assignment
1D torus or mesh
2D torus or mesh
3D torus or mesh

12 nodesR

C
https://pr.fujitsu.com/jp/news/2020/04/28.html
14384

Outline
BFS in Graph500 Benchmark
The supercomputer Fugaku
Tuning BFS on the supercomputer Fugaku
Full node evaluation

15

Number of processes per node (1/2)

Performance

1.75

4ppn

16,00012,000
8,000

1ppn

4,000
512

Number of nodes

The result of 16,384 nodes for 2ppn could not
be measured due to a system malfunction.

1.50

4ppn

1.25

2ppn
1.00

1,024 2,048 4,096 8,192 16,384

Performance Ratio with 1ppn

512

1,024 2,048 4,096 8,192 16,384

Better2ppn

Ratio with 1ppn

20,0000

In the cases of 1ppn and 2ppn, the cache hit
rate decreases because the memory accessesby threads cross the CMG.

BetterPerformance (GTEPS)

Process per node (ppn)
1 process 48 threads per node (1ppn)
2 processes 24threads per node (2ppn)
4 processes 12threads per node (4ppn)

1ppn

Number of nodes

The larger the number of nodes, the smaller the performance difference
16385

Number of processes per node (2/2)
1ppn

4ppn

imbalance
point-to-point
alltoallv

Comm.

allgatherv
calculation
512

1,024 2,048 4,096 8,192 16,384

512

1,024 2,048 4,096 8,192 16,384

Number of nodes

Number of nodes

As the number of nodes increases, the rate of communication increases
1ppn has a smaller rate of communication than 4ppn
If the number of nodes is increased further, the communication ratio will increase.
Thus, we select 1ppn, which can bring out the full communication performance17

Use of Eager method (1/2)
In the point-to-point communication of most MPI implementations, the Eager and

Rendezvous methods are implemented
Although most MPI implementations switch the Eager and Rendezvous methods
automatically depending on message size, optimal message size depends on application
Eager

Rendezvous

Asynchronous communication thatcan start/end the message sending
process regardless of the state of
the sending/receiving processes

Synchronous communication to
send/receive messages after both MPI
processes are ready to communicate
18386

Use of Eager method (2/2)
In the default setting, Rendezvous was selected for all send/recv communication of BFS

0.5

imbalance
point-to-point

0.4
0.3

Eager

0.1
0

alltoallv

Rendezvous

0.2

allgather

BetterElapsed time (sec.)

Fujitsu MPI library on Fugaku can set the threshold for switching between Eager and
Rendezvous methods
We change the threshold to 512 Kbytes from default value to use Eager method
Since Fugaku's compute node has 32 Gbytes memory, the buffer size is relatively small

calculation

512 1,024 2,048 4,096 8,192 16,384
Number of nodes

16,384 nodes

Achieved a performance
improvement of 2.3%
19

Power management (1/2)
User can specify CPU frequency for each job
Normal mode : 2.0 GHz
Boost mode : 2.2 GHz

Eco mode : Two floating-point arithmetic pipelines of A64FX are limited to one, andpower control is performed according to the maximum power
Since BFS does not perform floating-point arithmetic, the use of Eco mode can be
expected to reduce power consumption without affecting performance
Normal :2.0 GHz, two floating-point arithmetic pipelines (in previous evaluations)

2.2 GHz, two floating-point arithmetic pipelines
Boost :
Normal Eco : 2.0 GHz, one floating-point arithmetic pipeline
Boost Eco : 2.2 GHz, one floating-point arithmetic pipeline

20

387

Power management (2/2)

1.06

Boost

1.04

Boost Eco

1.02
1.00
0.98

Normal512

1,024 2,048 4,096 8,192 16,384

Number of nodes

Eco

1.40
1.30

Boost Eco

1.20
1.10

Normal

1.00
0.90

Boost
512

BetterEcoPower Efficiency Ratio

Power Efficiency (TEPS/W) ratio

BetterPerformance Ratio

Performance (TEPS) ratio
1.08

1,024 2,048 4,096 8,192 16,384

Number of nodes

Boost Eco mode has a good balance between performance and power efficiency
The performance in Boost Eco mode is 3.6 % better than that in Normal mode
The power efficiency in Boost Eco mode is 27.2 % better than that in Normal mode
The results of Boost Eco mode for 16,384 nodes are 18,607 GTEPS and 1,408 kW

21

Outline
BFS in Graph500 Benchmark
The supercomputer Fugaku
Tuning BFS on the supercomputer Fugaku
Full node evaluation

22

388

Six-dimensional process mapping (1/2)
The size of six axes in Fugaku network is (X, Y, Z, a, b, c) = (23, 24, 23, 2, 3, 2)
It is desirable that the values of R and C process grid of BFS are close
We assign the processes to (R, C) = (XY, Zabc) = (552, 288)
Since neighborhood communication occurs in BFS, we assign the processes
physically next to each other in row/column dimension

Node

12 nodesR

C
23

Six-dimensional process mapping (2/2)
We evaluate the BFS performance for a large-scale graph consisting of about 2.2

trillion vertices and 35.2 trillion edges using the whole Fugaku system (158,976 nodes)
Boost Eco mode

Performance: 102,956 GTEPS, Power: 14,961 kW, Efficiency: 6.9 MTEPS/W

nodes), and power efficiency
is 1.9 times that of IBM
Sequoia (Blue Gene/Q)

The K computer did not measure
power. At IBM Sequoia, the graphis the same size as Fugaku.

10^6

1.2

10^5

0.8

0.4

10^4

10^3
512

2,048

8,192

32,768 131,072

Number of nodes

0

Parallel Efficiencyof the K computer (82,944

Performance (GTEPS)

Performance is 3.3 times that

14.9 imbalance
11.6 send/recv
33.2 alltoallv
14.7 allgather
25.6 calculation
158,976 nodes
24389

Summary
Tune performance of BFS in Graph500 on Fugaku

We evaluate the BFS performance for a large-scalegraph consisting of about 2.2 trillion vertices and 35.2
trillion edges using the whole Fugaku system

Achieve 102,955 GTEPS, resulting in the first positionof Graph500 lists in from 2020 to now

Future works

Develop SSSP in cooperation with ZIB and IMISome ideas to improve performance of BFS

25

390

The 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop on
Advances in Classical and Quantum Algorithms for
Optimization and Machine Learning
September 16th - 19th, 2022, Tokyo (The university of Tokyo), Japan,
and September 21st - 22nd, 2022, Fukuoka (Kyushu University), Japan

Notes on Solving QUBOs
and Quantum Computing
Prof. Dr. Thorsten KOCH
Zuse Institute Berlin and Technical University of Berlin,
Germany
koch@zib.de
It is regularly claimed that quantum computers will bring breakthrough progress insolving challenging combinatorial optimization problems relevant in practice. In particular,
Quadratic Unconstraint Binary Optimization (QUBO) problems are said to be the model ofchoice for use in (adiabatic) quantum systems. Combinatorial Optimization searches for an
optimum object in a finite but usually vast collection of objects. This approach can be used
for many practical purposes, like efficient allocation of limited resources, network planning,and hundreds of other applications in almost all fields, e.g., finance, production, scheduling,
and inventory control. However, many combinatorial optimization problems are known to
be NP-hard. This theoretical statement about worst-case runtime complexity is oftentranslated simplistically as "intractable"; however, the practical side looks different. In manycases, it is possible to solve such problems to proven global optimality. We explain some of
the meaning and implications, review the state of affairs, the potential of quantum
computing, and give new computational results regarding solving OUBOs.

References[1] Kamil Bradler, Hugo Wallner, “Certain properties and applications of shallow bosoniccircuits”, 2021, https://arxiv.org/abs/2112.09766[2] Catherine C. McGeoch, “Adiabatic Quantum Computation and Quantum Annealing”,

Synthesis Lectures on Quantum Computing, Springer, 2014,
https://link.springer.com/book/10.1007/978-3-031-02518-1[3] Feng Pan, Keyang Chen, Pan Zhang, “Solving the Sampling Problem of the Sycamore
Quantum Circuits”, Physical Review Letters, Vol. 129, Issue 9, 2022
https://journals.aps.org/prl/abstract/10.1103/PhysRevLett.129.090502[4] Richard M. Karp, “Reducibility among Combinatorial Problems”, Complexity of Computer
Computations, The IBM Research Symposia Series, pp. 85-103, 1972
[5] Leo Liberti, “Undecidability and hardness in mixed-integer nonlinear programming”,
RAIRO Operations Research, Volume 53, pp. 81-109, 2019,https://hal.archives-ouvertes.fr/hal-02104836[6] Robin Brown, David E. Bernal Neira, Davide Venturelle, Marco Pavone, “Copositive
programming for mixed-binary quadratic optimization via Ising Solvers”, 2022
https://arxiv.org/abs/2207.13630[7] Fred Glover, Gary Kochenberger, Yu Du, „Quantum Bridge Analytics I: A Tutorial on
Formulating and Using QUBO Models”, 2022, https://arxiv.org/abs/1811.11538[8] Daniel Bienstock, Oktay Günlük, “Computational experience with a difficult mixed-integer
multicommodity flow problem”, Mathematical Programming, Volume 68, pp. 213-237,
1995, https://doi.org/10.1007/BF01585766[9] Daniel Rehfeldt, Thorsten Koch, Yuji Shinano, “Faster exact solution of sparse MaxCut and
QUBO problems”, 2022, https://arxiv.org/abs/2202.02305[10] Michael Jünger, Elisabeth Lobe, Petra Mutzel, Gerhard Reinelt, Franz Rendl, Giovanni

391

Rinaldi, „Quantum Annealing versus Digital Computing: An Experimental Comparison“,
ACM Journal of Experimental Algorithmics, Volume 26, pp. 1-30, 2021,
https://doi.org/10.1145/3459606[11] Scott Aaronson, “NP-complete Problems and Physical Reality”, 2005
https://arxiv.org/abs/quant-ph/0502072[12] Giacomo Nannicini, “An Introduction to Quantum Computing, Without the Physics”, 2017,https://arxiv.org/abs/1708.03684[13] Pradeep Niroula, “Conquering the challenge of quantum optimization”, Physics World,
2022, https://physicsworld.com/a/conquering-the-challenge-of-quantum-optimization[14] Lennart Bittel, Martin Kliesch, “Training Variational Quantum Algorithms Is NP-Hard”,
Physical Review Letters, Volume 127, 2021,
https://journals.aps.org/prl/abstract/10.1103/PhysRevLett.127.120502[15] “Twenty Questions for Donald Knuth”, InformIT, 2014
https://www.informit.com/articles/article.aspx?p=2213858&WT.mc_id=Author_Knuth_20Q
uestions

392

News: The Ministry of Defence has procured the government’s first quantum computer.
https://quantumzeitgeist.com/british-ministry-of-defence-procures-governments-first-quantum-computer

They’ll work with Orca Computing, a British company, to investigate andapply quantum technology in military defense.
Quantum computer manufacturers claim their devices can solve complex
problems that classical computers cannot solve.
About Orca I find there: British Quantum Computing Firm ORCA, Claims
Breakthrough in Quantum Computers

https://quantumzeitgeist.com/british-quantum-computing-firm-orca-claims-breakthrough-in-quantum-computers

The major challenge identified by one of the UK’s leading quantum
computing experts Prof Morton, was the ability of the Orca computer toscale up quickly.

“Scaling up is very important for the computer to be able to serve its purposeby performing complex, highly scientific and experimental tasks such ascombating climate changes, accelerating artificial intelligence, shipnavigation or even drug development.
These tasks require millions of qubits to be successful when in reality, theOrca computer has just four qubits which is very far from ideal. “However, the company assured that they will scale up in the next two years.
Looking at the Orca Webpage I found then:https://www.orcacomputing.com/news/orca-computing-provides-uk-mod-with-first-quantum-computer-j6pns-rzbhh

ORCA discovers new algorithm for solving QUBO problems with near-term,‘shallow’ quantum computers
On the webpage Orca claims to solve binary knapsack problems with 70
variables faster than any other QC and classical methods.
Notes on Solving QUBOs and Quantum Computing

As a reference to the conclusions a scientific article published by the Orca
scientists was given: https://arxiv.org/abs/2112.09766
Unfortunately, there is nothing about knapsacks, but:
2.8. Scalability and other challenges.
How plausible is it to scale the presented variational method?
The first problem the reader can point out is the fact that despite the parity
map effectively coarse graining the measurement results of an M-mode
circuit it is not a sustainable strategy since the size of an M-qubit Hilbert
space grows exponentially. […]But here we tacitly assume that the measurement outcomes are uniform
which is not the case. We observed that randomly chosen parameters of thestudied shallow circuits typically result in a small set of measurement
patterns to have high probability enabling it to be sampled with confidencewith a bounded number of repeated measurements.
The caveat is, of course, that even if this trend continues as M grows it
inevitably means that the ratio of such reliably estimated measurementswith respect to all possible patterns decreases exponentially.
On the one hand, this still enables us to use the proposed variationalalgorithm. However, the odds of getting stuck in a local minimum most likely
increase. How exactly it affects the ability to reach a global solution is amatter of a more detailed study.

Thorsten Koch

TU Berlin / Zuse Institute Berlin (ZIB)

Notes on Solving QUBOs and Quantum Computing
Thorsten Koch
Daniel Rehfeldt
Yuji Shinano
Department:
Applied AlgorithmicIntelligence Methods

Chair of
Software and Algorithms
for Discrete Optimization

393

2

“In theory, there is no difference between theory and practice;
but in practice, there is.”― unknown (not Einstein, or Feynman)

Notes on Solving QUBOs and Quantum Computing

Thorsten Koch

TU Berlin / Zuse Institute Berlin (ZIB)

7

News: The Ministry of Defence has procured the government’s first quantum computer.
https://quantumzeitgeist.com/british-ministry-of-defence-procures-governments-first-quantum-computer

They’ll work with Orca Computing, a British company, to investigate andapply quantum technology in military defense.
Quantum computer manufacturers claim their devices can solve complex
problems that classical computers cannot solve.
About Orca I find there: British Quantum Computing Firm ORCA, Claims
Breakthrough in Quantum Computers

https://quantumzeitgeist.com/british-quantum-computing-firm-orca-claims-breakthrough-in-quantum-computers

The major challenge identified by one of the UK’s leading quantum
computing experts Prof Morton, was the ability of the Orca computer toscale up quickly.

“Scaling up is very important for the computer to be able to serve its purposeby performing complex, highly scientific and experimental tasks such ascombating climate changes, accelerating artificial intelligence, shipnavigation or even drug development.
These tasks require millions of qubits to be successful when in reality, theOrca computer has just four qubits which is very far from ideal. “However, the company assured that they will scale up in the next two years.
Looking at the Orca Webpage I found then:https://www.orcacomputing.com/news/orca-computing-provides-uk-mod-with-first-quantum-computer-j6pns-rzbhh

ORCA discovers new algorithm for solving QUBO problems with near-term,‘shallow’ quantum computers
On the webpage Orca claims to solve binary knapsack problems with 70
variables faster than any other QC and classical methods.
Notes on Solving QUBOs and Quantum Computing

As a reference to the conclusions a scientific article published by the Orca
scientists was given: https://arxiv.org/abs/2112.09766
Unfortunately, there is nothing about knapsacks, but:
2.8. Scalability and other challenges.
How plausible is it to scale the presented variational method?
The first problem the reader can point out is the fact that despite the parity
map effectively coarse graining the measurement results of an M-mode
circuit it is not a sustainable strategy since the size of an M-qubit Hilbert
space grows exponentially. […]But here we tacitly assume that the measurement outcomes are uniform
which is not the case. We observed that randomly chosen parameters of thestudied shallow circuits typically result in a small set of measurement
patterns to have high probability enabling it to be sampled with confidencewith a bounded number of repeated measurements.
The caveat is, of course, that even if this trend continues as M grows it
inevitably means that the ratio of such reliably estimated measurementswith respect to all possible patterns decreases exponentially.
On the one hand, this still enables us to use the proposed variationalalgorithm. However, the odds of getting stuck in a local minimum most likely
increase. How exactly it affects the ability to reach a global solution is amatter of a more detailed study.

Thorsten Koch

TU Berlin / Zuse Institute Berlin (ZIB)3943

https://xkcd.com/221

Being Deterministic“Insanity Is Doing the Same Thing Over and Over Again
and Expecting Different Results.”― unknown
An algorithm is deterministic, if given a particular input, it will always produce the same output, withthe underlying machine always passing through the same sequence of states. https://en.wikipedia.org/wiki/Deterministic_algorithm
A deterministic algorithm computes a mathematical function; a function has a unique value for any input
in its domain, and the algorithm is a process that produces this particular value as output.
Non-determinism can result, for example, from:▷ use of an external state other than the input, such as a hardware timer value.
▷ if multiple processors writing to the same data at the same time. In this case, the precise order inwhich each processor writes its data will affect the result.
While digital computers are thought of being deterministic,
Quantum computers a probabilistic, i.e., non-deterministic by definition.
Notes on Solving QUBOs and Quantum Computing

Thorsten Koch

TU Berlin / Zuse Institute Berlin (ZIB)

9What does “solving” an NP-hard problem typically mean?

Being able to …
TheoreticalComputer Scientist… compute proven optimal solutions to every instance of this problem
class with at most this effort

Applied Discrete
Mathematician… practically compute within numerical tolerances proven optimal
solutions to these particular (relevant) instances of the problem class inreasonable time

Physicist, Quantum
Computing Researcher… compute reasonably good solutions to these (selected) particularinstances of the problem class in very short time

Notes on Solving QUBOs and Quantum Computing

Thorsten Koch

TU Berlin / Zuse Institute Berlin (ZIB)

395

8

Setting the stage“Combinatorial Optimization searches for an optimum object in a finite collection of objects. Typically,
the collection has a concise representation, while the number of objects is huge --- more precisely, grows
exponentially in the size of the representation. So scanning all objects one by one and selecting the best
one is not an option.”― Alexander Schrijver, Combinatorial Optimization, 2003, Page 1.

min $(&) with ( = &, +, ,, - ∈ ℤ$ ∶ 1 & ≤ +, , ≤ & ≤ 3!∈#

For the rest of the talk, we assume: !: # → ℤ is a linear or quadratic function, i.e.,! & = ( ! & + & ! *&, ( ∈ ℤ" , * ∈ ℤ"×" , and -: # → ℤ" is a linear function, i.e., g & = .&, . ∈ ℤ"×" .
Note: argmin !(&) = argmax −!(&) and - & + : = ;, : ≥ 0 ⟺ - & ≤ 0, and similar for ≥.

We defined everything using integer numbers. If we would use rational numbers, we could then scale
them by the least common multiple of all denominators to make everything integer.
Notes on Solving QUBOs and Quantum Computing

Thorsten Koch

TU Berlin / Zuse Institute Berlin (ZIB)

11

Disclaimer

For the rest of this talk,unless noted otherwise,
everything I say,will refer (only) to
Combinatorial Optimization Problems.
Notes on Solving QUBOs and Quantum Computing

Thorsten Koch

TU Berlin / Zuse Institute Berlin (ZIB)

396

10

The Bigger (complexity) Picture
From Catherine C. McGeoch: Adiabatic Quantum Computation and Quantum Annealing

BQP is bounded-error quantum
polynomial time. A quantum modelof computation uses anarrangement of quantum circuits (inthe gate model) operating on qubits.
These problems can be solved by aquantum computer in polynomial
time with an error probability of at
most 1/3.
QMA stands for Quantum MerlinArthur. This is the set of decision
problems for which yes-instances
can be verified by a quantum
computer in polynomial time, andno-instances rejected in polynomial
time, with error probability at most
1/3. This is the quantum analog ofNP, related to BQP as NP is related
to P. QMAC is the set of complete
problems for this class.
QCMA stands for Quantum-classical
Merlin-Arthur. This class is similar to
QMA but the witness for a yesinstance must be a classical string.

Notes on Solving QUBOs and Quantum Computing

Thorsten Koch

TU Berlin / Zuse Institute Berlin (ZIB)

13P“easy”

By Behnam Esfahbod, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=3532181

Complexity classes
Solvable in polynomial time. A running time of10!" # $#""" for input size $ is poly time.

NP“Yes” answer can be checked in poly time. It does
not matter where the answer comes from.

NP-Complete

Hardest problems in NP. If you could solve one ofthem in poly time, then you can solve them all.

NP-Hard“intractable”

As least as hard as the hardest problems in NP.
Need not to be in NP.

Integer factorization is in NP.
Current assumption: it is not in P and not NP-hard => not NP-complete.
Integer Programming/Optimization is in NP.
Proven to be NP-hard and therefore NP-complete
A proof of P=NP might not be constructive and include a huge constant.
Notes on Solving QUBOs and Quantum Computing

Thorsten Koch

TU Berlin / Zuse Institute Berlin (ZIB)

397

Open
12

Quantum Supremacy
https://www.quantamagazine.org/john-preskill-explains-quantum-supremacy-20191002
In 2012, I proposed the term “quantum supremacy” to describe the point where quantum computers
can do things that classical computers can’t, regardless of whether those tasks are useful.
https://scottaaronson.blog/?p=4317
Scott’s Supreme Quantum Supremacy FAQ!
Q1. What is quantum computational supremacy?
Often abbreviated to just “quantum supremacy,” the term refers to the use of a quantum computer tosolve some well-defined set of problems that would take orders of magnitude longer to solve with any
currently known algorithms running on existing classical computers—and not for incidental reasons, but
for reasons of asymptotic quantum complexity. The emphasis here is on being as sure as possible thatthe problem really was solved quantumly and really is classically intractable, and ideally achieving the
speedup soon (with the noisy, non-universal QCs of the present or very near future). If the problem isalso useful for something, then so much the better, but that’s not at all necessary. The Wright Flyer andthe Fermi pile weren’t useful in themselves.
Notes on Solving QUBOs and Quantum Computing

Thorsten Koch

TU Berlin / Zuse Institute Berlin (ZIB)

Slide by Scott Aaronson: https://www.scottaaronson.com/talks/speedup.ppt

An Inconvenient Truth

If we set aside NP-complete problems, there just aren’t thatmany compelling candidates left for exponential quantum
speedups! (And for many of those, we do have exponential speedups,
and for many of the rest we have polynomial ones)
3SAT

NP-completeL
Pr atti
ob ce
lem
s

NP

BQP

(Quantum P)

P

Qu
an
Sim tum

Iso

Fa
cto
rin
g

h
ap
Gr

NP-hard398

P≠BQP, NPËBQP:Plausible conjectures,
which we have nohope of proving given
the current state ofcomplexity theory

15

Integer Linear Program (ILP)

3 , $ & = 5% &min $(&) with ( = & ∈ ℤ$ ∶ 4& ≤ +, , ≤ & ≤ !∈#Solving this problem is in general NP-hard, even if have & ∈ 0,1 . However, is important to note thatwithout the integrality requirement, i.e., for & ∈ ℝ" the problem can be solved in polynomial time.
What does NP-hard mean:(1) If we get some %, we can check in polynomial time whether it belongs to & and compute ' % .
(2) Finding the minimum % might, in the worst case, be as difficult to solve as any other problem in NP.
(3) As of today, no algorithm is known that has better than exponential worst case runtime complexity to do so.
And there is little hope to find one. Note: Integer factorization is not NP-hard!
What does NP-hard not necessarily mean:(1) Finding some % ∈ & is difficult. Might be difficult, but doesn’t have to.
(2) Finding the minimum % ∈ & is difficult.
Finding it might actually be easy, proving it is the minimum is the hard part.
(3) Large size instances of NP-hard problems are intractable or unsolvable in principle.
Notes on Solving QUBOs and Quantum Computing

Thorsten Koch

TU Berlin / Zuse Institute Berlin (ZIB)17

https://arxiv.org/abs/2111.03011
Solving the sampling problem of the Sycamore quantum supremacy circuits
Feng Pan, Keyang Chen, Pan Zhang
We study the problem of generating independent samples from the output distribution of Google's
Sycamore quantum circuits with a target fidelity, which is believed to be beyond the reach of classicalsupercomputers and has been used to demonstrate quantum supremacy. We propose a new method toclassically solve this problem by contracting the corresponding tensor network just once, and ismassively more efficient than existing methods in obtaining a large number of uncorrelated samples witha target fidelity. For the Sycamore quantum supremacy circuit with 53 qubits and 20 cycles, we havegenerated one million uncorrelated bitstrings {s} which are sampled from a distribution Pˆ(s)=|ψˆ(s)|2,
where the approximate state ψˆ has fidelity F≈0.0037. The whole computation has cost about 15 hours
on a computational cluster with 512 GPUs. The obtained one million samples, the contraction code andcontraction order are made public. If our algorithm could be implemented with high efficiency on amodern supercomputer with ExaFLOPS performance, we estimate that ideally, the simulation would costa few dozens of seconds, which is faster than Google's quantum hardware.
https://journals.aps.org/prl/abstract/10.1103/PhysRevLett.129.090502

Notes on Solving QUBOs and Quantum Computing

Thorsten Koch

TU Berlin / Zuse Institute Berlin (ZIB)

399

16

Notes on NP-hardSubset-sum: Given & ∈ ℤ" and ; ∈ ℤ exists a subset B ⊆ {1, … , F} such that ∑$∈& &$ = ;,Subset-sum is NP-hard (even with &$ ∈ ℕ )However: Subset-sum bounded by a constant J > &$ ∈ ℕ is in P
On a Computer &$ is always bounded if we restrict ourselves to, say 64-bit integers.

For a mathematician nearly all numbers are larger than 2'(. (there are only finite many exceptions)

But in real-life even the US dept in pennies is just 100 M 30 M 10)*2'(
= 18.446.744.073.709.551.616)+
3.000.000.000.000.000
3 M 10 =Adding numbers up to this is O(1)NP-hard problems are extremely difficult as a class and in theory. In practice, it depends.
Notes on Solving QUBOs and Quantum Computing

Thorsten Koch

TU Berlin / Zuse Institute Berlin (ZIB)

19

Problem classes

QUBO = ILP

CIP
SAT

Satisfiability

ILP

SAT is NP-complete, so is ILP andeverything that encompasses it.
Constraint Integer
Programming

That a class of problems is NP-complete should
not stop you from solving instancesof this class to proven optimality

MILP

Mixed Integer
Linear Programming

CP

Linear
Programming

Integer Linear
Programming

LP

Constraint
Programming

Notes on Solving QUBOs and Quantum Computing

doi:10.1145/800157.805047

Richard M. Karp (1972)
Reducibility Among Combinatorial Problems
doi:10.1007/978-1-4684-2001-2_9

MINLP

Symboldarstellung
Räume sind nicht größenmäßig
korrekt dargestellt.

Stephen Cook (1971)
The Complexity of Theorem Proving Procedures

Leo Liberti (2019)
Undecidability and Hardness in MINLP

https://doi.org/10.1051/ro/2018036

Mixed Integer Non-linear
Programming

Matthias Köppe (2010)
On the complexity of nonlinear mixed-integer optimization

Thorsten Koch

TU Berlin / Zuse Institute Berlin (ZIB)

400

arXiv:1006.4895v1

18

Examples for finding some ! ∈ # is difficult. Might be, but doesn’t have to.
We can write our problem as a decision problem (and minimize by binary search):3 ∧ 5% & = :(& ≠ ∅ ? with (& = & ∈ ℤ$ : 4& ≤ +, , ≤ & ≤ -In this case finding some ! is equivalent to solving the problem.

Or, using some suitable big constant $, we can move the constraints into the objective:

min $(&) with ( = & ∈ ℤ$ : , ≤ & ≤ 3 , $ & = 5%& + <(+ − %&)!∈#now it is obviously trivial to find some ! ∈ #.

This difference in difficultyis one reason why people
believe ^ ≠ `^Note: To solve an ILP, i.e., to optimality two things must be done:(1) ∃ :Find the minimum & ∗ ∈ #.
(2) ∀ :Prove there exists no & ∗ ∈ # with ! & < !(& ∗ ) .
Y ∧ \(U) < \(U∗ ) = ∅This is equivalent to showing: U ∈ ℤ-: VU ≤ W, X ≤ U ≤ Z
Notes on Solving QUBOs and Quantum Computing

Thorsten Koch

TU Berlin / Zuse Institute Berlin (ZIB)

21

What O = Order of means: both pictures are equally correct

Notes on Solving QUBOs and Quantum Computing

Thorsten Koch

TU Berlin / Zuse Institute Berlin (ZIB)

401

20

What are QUBOs?
QUBO : Quadratic Unconstraint Binary Optimization
UBQP : Unconstrained Binary Quadratic Program(BIQ : Binary Integer Quadratic problem)

▷
▷
▷
▷
▷

min $ , %$(∈ ),+ )& is a vector of binary variables, * is a square F×F matrix of constants
Since QUBOs are unconstraint, any 0/1 vector is a feasible solution
All QUBOs can be brought to the form where * is symmetric or upper triangular
Solving QUBO (in general) is NP-hardSince & is binary, &! = &!" holds ⟹ The coefficients of the linear terms of the objective function
correspond to the diagonal entries of *Notes on Solving QUBOs and Quantum Computing

Thorsten Koch

TU Berlin / Zuse Institute Berlin (ZIB)23

https://arxiv.org/abs/2207.13630
Copositive programming for mixed-binary quadratic optimization via Ising solvers
Robin Brown, David E. Bernal Neira, Davide Venturelli, Marco Pavone
Recent years have seen significant advances in quantum/quantum-inspired technologies capable of approximately
searching for the ground state of Ising spin Hamiltonians.
The promise of leveraging such technologies to accelerate the solution of difficult optimization problems has spurred
an increased interest in exploring methods to integrate Ising problems as part of their solution process, with existing
approaches ranging from direct transcription to hybrid quantum-classical approaches rooted in existing optimization
algorithms. Due to the heuristic and black-box nature of the underlying Ising solvers, many such approaches have
limited optimality guarantees.
While some hybrid algorithms may converge to global optima, their underlying classical algorithms typically rely on
exhaustive search, making it unclear if such algorithmic scaffolds are primed to take advantage of speed-ups that the
Ising solver may offer.
In this paper, we propose a framework for solving mixed-binary quadratic programs (MBQP) to global optimality
using black-box and heuristic Ising solvers. We show the exactness of a convex copositive reformulation of MBQPs,
which we propose to solve via a hybrid quantum-classical cutting-plane algorithm. The classical portion of this hybrid
framework is guaranteed to be polynomial time, suggesting that when applied to NP-hard problems, the complexity ofthe solution is shifted onto the subroutine handled by the Ising solver.
Notes on Solving QUBOs and Quantum Computing

Thorsten Koch

TU Berlin / Zuse Institute Berlin (ZIB)

402

22

BIP to QUBO

BIP
min

'∈ (,* #BIPs can be reformulated as QUBOs by
putting the constraints into the objectivewith a penalty term *. The penalty should bezero if and only if the constraint is fulfilled.

>+?

s.t. C? ≤ D

QUBO

Glover, Kochenberger, Du (2019):
A Tutorial on Formulating and Using QUBO Models
arXiv:1811.11538

min > + ? , + E C? + FG − D + (C? + FG − D)'∈ (,* #=>min ? + H?

-∈ (,* #where * ∈ ℝ$×$ and symmetricNotes on Solving QUBOs and Quantum Computing

Thorsten Koch

TU Berlin / Zuse Institute Berlin (ZIB)

25

From QUBO transformation, E. Rodríguez-Heck, Gurobi Days Digital May 03,2022

Ising Hamiltonian• Quantum annealing is a special-purpose device that finds the minimum energy of an
Ising Hamiltonian heuristically
h0
h2
J02

i2V

hi s i +

X

J1

Spins

si 2 {−1, +1}

Jij si sj ,{i,j}2E

s2

s1

J 13

h1

• A Quadratic Unconstrained Binary Optimization (QUBO) problem can be transformed
into an Ising Hamiltonian with a simple algebraic manipulation:T

x Qx
Notes on Solving QUBOs and Quantum Computing

xi =si + 12

si 2 {−1, 1}
Thorsten Kochs 3 h3
J 34

HIsing =X

Quadratic terms(couplers)

J 23

Linear terms (biases,
local fields)2s0

h4 s 4

HIsing34
TU Berlin / Zuse Institute Berlin (ZIB)© 2022 Gurobi Optimization, LLC. Confidential, All Rights Reserved | 24403

Max-Cut
Graph formulation c = d, e, t

max j t!4 with B ⊂ d, w ⊂ d, B ∩ w = ∅, B ∪ w = d&,6!∈&,4∈6

Ising formulation$ $1
max j j t!4 1 − &! &4 , &5 ∈ 1, −1 for { ∈ {1, … , F}
4
!3- 43-Binary Linear Programing formulation:$

Binary Quadratic Programing formulation:$

max j j t!4 u!4!3- 43-u!4 ≤ &! + &4
u!4 ≤ 2 − (&! + &4 )&5 ∈ 0,1
u!4 ∈ {0,1}Notes on Solving QUBOs and Quantum Computing$$

max j j t!4 (&! + &4 − 2&!4 ))∈ +,- &&! = &4 = 0 ⇒ u!4 = 0!3- 43-Can be written as:&! = &4 = 1 ⇒ u!4 = 0

Thorsten Kochmin ! & 0!

!∈ #,% !

TU Berlin / Zuse Institute Berlin (ZIB)

27

Independent (stable) Set
Given a graph c = (d, e), find the maximum size independent set of nodes
max |B| with g, h ∈ B ⟹ (g, h) ∉ e&⊆(

Binary Linear Programing formulation:
max j &. subject to &/ + &. ≤ 1 for all (g, h) ∈ e)∈ +,- |%|.∈(Unconstraint Quadratic Binary Programing formulation:min − j &." + ^ M j &/ M &.

)∈ +,- |%|Notes on Solving QUBOs and Quantum Computing.∈((/,.)∈2

Thorsten Koch

=0

404

TU Berlin / Zuse Institute Berlin (ZIB)

26

Some more notes on QUBOs
▷
▷
▷
▷
▷
▷
▷
▷
▷
▷
▷

In theory possible to model all ILP and SAT, but not MILP
Number of constraints is not important,
only the number of Variables!
Constraint with large support result in dense Q,
cardinality constraint is worst
While most available software works on dense instances
this limits the problem size dramatically
General ^ .& − ; 6 (.& − ;). Beware numerical trouble!
QUBO is unconstraint and pure binary, nearly all heuristic
ideas work nicely
Glover, Kochenberger, Du (2019):
A Tutorial on Formulating and Using QUBO Models
Preprocessing is limited compared to ILP/SAT
arXiv:1811.11538
Impossible to distinguish between feasibility and optimization. When to stop?
Best for problems where there is some “natural” quadratic formulation. (But, e.g., QAP is dense)On the primal side hard to win against problem specific heuristic approaches
How to get good lower bounds? (LP/SDP/Newton-Bracket)

Notes on Solving QUBOs and Quantum Computing

Thorsten Koch

TU Berlin / Zuse Institute Berlin (ZIB)

31

Transformations
Modelling a linear relationship quadratic:& = & ", & ∈ 0,1

Modelling multiplication of binary variables } = &- M &" = &- ∧ &", for &-, &" ∈ 0,1 :} ≤ &} ≤ &"} ≥ &- + &" − 1} ∈ {0,1}
Modelling general Integer variables from Binary variables:$7-

Alternatively:

j 2! &! ≤ `, &! ∈ 0,1 ⟺ u ∈ {0, … , min(2$ − 1, `)}8!3+

j ~ M &! = u and!3-

Notes on Solving QUBOs and Quantum Computing

8

j &! = 1 for &! ∈ 0,1 , u ∈ 1, … , `!3-

Thorsten Koch

TU Berlin / Zuse Institute Berlin (ZIB)

405

29

Once upon a time

Notes on Solving QUBOs and Quantum Computing

Thorsten Koch

TU Berlin / Zuse Institute Berlin (ZIB)

34

ReminderSolving an optimization problem meansYmin \(U) with Ä = U, W, X, Z ∈ ℤ; ∶ Ç U ≤ W, X ≤ U ≤ Z
9∈:(1) finding a feasible solution (often trivial, but not always)(2) Proving it is the best one
Solving a decision (feasibility) Problem meansYÄ ≠ ∅ ⟺ min É with Ä = U, W, X, Z ∈ ℤ; ∶ Ç U ≤ W, X ≤ U ≤ Z(1) Finding a solution

9∈:

A QUBO by definition is an unconstraint optimization problem
min ' U> ÑU
9∈ <,=

i.e., finding a solution is absolutely trivial (e.g., & = 0)
We can transform between the 3, so be careful what is done exactly and how a solution translates back.

Notes on Solving QUBOs and Quantum Computing

Thorsten Koch

TU Berlin / Zuse Institute Berlin (ZIB)

406

33

Best solutions so far

Notes on Solving QUBOs and Quantum Computing

Thorsten Koch

TU Berlin / Zuse Institute Berlin (ZIB)

36

Notes on Solving QUBOs and Quantum Computing

Thorsten Koch

TU Berlin / Zuse Institute Berlin (ZIB)

35

407

Best know solution

Notes on Solving QUBOs and Quantum Computing

Thorsten Koch

TU Berlin / Zuse Institute Berlin (ZIB)38

ICONG(2)

s.t.

min u
+ %*(
()*

+ %(*
()*

'+*(

+ '+*( − + '+*(
()*

()*

+ '+*(
+

'+*(
%*(

=

2

=

2

≤
=

≤
≥
∈

0
# %*(
6*+

1, ., 4 ∈ {1, … , 24}
for all .
for all .
for all 1, . ≠ 4
for all i, 1

for all . ≠ 4{0,1}

for all . ≠ 4

Notes on Solving QUBOs and Quantum Computing(1)

Best known solution:

665.5714

(2)

8

0

Root relaxation objective: 30.28571
dano3mip root relaxation : 576.2316

for all 1, . ≠ 4! = ###

(3)
(4)
(5)
(6)
(7)

Thorsten Koch

30.28571
350.63706
425.04748
436.46971
Gurobi 9.5.1 root cuts 439.96568

95.4%
47.3%
36.1%
34.4%
33.9%

original model 577.82468
with some cuts removed 577.88492__

TU Berlin / Zuse Institute Berlin (ZIB)

408

37

Make it all integer
min u

s.t.

+ %*(
()*

+ %(*
()*

'+*(

+ '+*( − + '+*(
()*

()*

+ '+*(+'+*(%*(

Scale by 1000 for 3 decimal digits
=

2

=

2

for all .
for all .

≤0 # %*(

for all 1, . ≠ 4≤

8

for all . ≠ 4

∈

{0,1}

= >??? # 6*+

Root relaxation objective 3028.571

for all i, 1

∈ {?, … , >???} for all 1, . ≠ 4

Notes on Solving QUBOs and Quantum Computing

for all . ≠ 4

Thorsten Koch

TU Berlin / Zuse Institute Berlin (ZIB)

40

Transforming ICONG(2) into a QUBU (so we can solve on a QC)

Todo:
1. Make it all integer
2. Make it equality (C? = D)
3. Make it binary
4. Put the constraints into the objective

Notes on Solving QUBOs and Quantum Computing

Thorsten Koch

TU Berlin / Zuse Institute Berlin (ZIB)

409

39

Making it binary
Modelling general Integer variables u ∈ 0, … , ` using Binary variables &! ∈ 0,1 :
8

j &!
!3-

8

representation is not unique

Q(S)

8

we need an extra constraint, which quadratic
looks like ∑*,(∈ #,…4 ,*)( %* # %( , and is densej ~ M &! ∧ j &! ≤ 1

Q(S)j ~ M &! +

Q( S)

!3-

!3-

$≔ 8

$7-

!3-

!3-,!∈A5$ ≔ CDE6 8j!3+

2! M &! +j ~ M &$B!

$7-j!3-,"7∈A6

2! M &$B! `&≔with "! ≔ $ ∈ 1, … , ) − 1 ∑"∈$ $ = - − ∑"%!

Notes on Solving QUBOs and Quantum Computing(

Q(log ! S)

representation is not uniqueswitching from 28 − 1 to 28 changes all involved
variables, only unique for powers of 2&≔ *+,! ($ ∧ " minimal}, ") ≔ $ ∈ 1, … , ) − 1 ∑"∈$ $ = - − ∑"%!
Thorsten Koch

TU Berlin / Zuse Institute Berlin (ZIB)

2" ∧ " minimal}

42

Transforming an ILP to a system of equationsmin −2%# − %! subject to
,∈ℤ

80%#
%#
−5%#
7%#
−2%#
3%#
3%#

−
+
−
+
+
+
+
%
%

10%!
20%!
20%!
2%!
2%!
4%!
4%!

s.t.

5! ≤ 7l≤!≤9! ∈ ℤ'

min 1 & !

≤
≤
≤
≤
≤
≥
≤
≤
≥

−7
120
−32
48
−7
5
5
O
P

Notes on Solving QUBOs and Quantum Computing

min −2%# − %! subject to
,∈ℤ

80%#
%#
5%#
7%#
2%#
3%#

⟺

−
+
+
+
−
+

10%!
20%!
20%!
2%!
2%!
4%!
O

+
+
−
+
−

≤

6#
6!
6/
60
61
%
6

min 1 & !
s.t.

⟺

. ∈ ℝ?×$ , ( ∈ ℝ$ , ;, Ö, g ∈ ℝ?
Thorsten Koch

−7
120
2
48
7
5
P
0

5! + <' = = 7l≤!≤9! ∈ ℤ'=≥0= ∈ ℝ'TU Berlin / Zuse Institute Berlin (ZIB)

410

=
=
=
=
=
=
≤
≥

Converting toequation form
adds m variables.

41

Faster than a Quantum computer

MAX-Reaper:An exact solver for QUBO and Max-Cut
Faster exact solution of sparse MaxCut and QUBO problems
Daniel Rehfeldt, Thorsten Koch, Yuji Shinano
doi: 10.48550/arXiv.2202.02305

Notes on Solving QUBOs and Quantum Computing

Thorsten Koch

TU Berlin / Zuse Institute Berlin (ZIB)

45

Putting the constraints into the objective
min I

s.t.

6 7"-."

6 7-"-."

8/"- + :/"- − ; < 7"-

6 8/"- − 6 8/"- − ; < :"/
-."

-."

=

2

=

2

=

0

=

for all >, $ ≠ @0

for all $ ≠ @

=

8/"-

∈

{0, … , ;}∈{0, … , ;}7"-:/":"-∈∈

Notes on Solving QUBOs and Quantum Computing

for all $0

6 8/"- + :"- − A/

for all $for all i, >{0,1}

for all $ ≠ @{0, … , ;}for all >, $ ≠ @for all >, $ ≠ @for all $ ≠ @min X : % ! + * Y% + Z6 − [ : (Y% + Z6 − [)9∈ ",# 0

Nodes, commodities: 1, ., 4 ∈ {1, … , 24}

Integer to binary: 0 = 666 → log ! 666 + 4 = 13

for all . ≠ 4 → 24×23 = 552

for all 1, . ≠ 4 → 24×552 = 13248

Binary variables total:

552 + 13× 13248 + 13248 + 552 = 352,176⟹ ` ∈ ℤ/1!#;<×/1!#;<

i.e., we need at least 352,176 qubits
The range of the coefficients in ` is at least up to
666/ = 295,408,296

And ` will not be particular sparse!

Thorsten Koch

TU Berlin / Zuse Institute Berlin (ZIB)

411

43

Overall Architecture
UG Framework for massive parallelization
SCIP Constraint Integer Programming Framework
Max-Cut Specialization
Cutting planes

Preprocessing
GPU basedprimal heuristic

Heuristics

LP
Solver

Conflict analysis

The final code will be able to completely utilize existing HPC resources:
Run multicore on many nodes employing GPU accelerators in parallel.
Notes on Solving QUBOs and Quantum Computing

Thorsten Koch

TU Berlin / Zuse Institute Berlin (ZIB)

47

Branch-and-cut algorithm for sparse QUBO and Max-Cut problems
Main components:▷

Presolving▷ Simple Domain Propagation▷ Problem-specific cutting planes (optimized implementation)▷ Primal heuristics▷ Parallel branch-and-bound search via UG framework(still experimental).
About 15,000 lines of code + 1M lines for SCIP and LP solver.
Already faster than existing QUBO/Max-Cut solvers on sparse
QUBO and Max-Cut benchmarks in many cases.
Notes on Solving QUBOs and Quantum Computing

Thorsten Koch

TU Berlin / Zuse Institute Berlin (ZIB)

412

46

Preprocessing
Whenever one tries to solve real-world (and other) problem instances, the first thing to do is toimplement some preprocessing. We found some new techniques to improve performance.
As one can see, the IsingChain and the K64-chimera instances practically vanish.

Notes on Solving QUBOs and Quantum Computing

Thorsten Koch

TU Berlin / Zuse Institute Berlin (ZIB)

51

Selected Benchmarks (there are more)
Name

#

|V|

|E|

Description

DIMACS4

512-3375

1,536-10,125

Instances from 7th DIMACS Challenge

IsingChain30

100-300

4,950-44,850

Max-Cut instances from physics applications

QBLIB

22

120-1,225

602-34,876

QUBOs from QPLIB instances

Mannino4

48-487

1,128-8,511

Frequency assignment problems

l64-dwave

80

2,049

8,064

Paintshop30

10-1,000

22-2,498

QUBO Instances modelling the binary paintshop probnlem

Torus

18

100-343

200-1,029

Max-Cut instances from physics applications

Kernel14

33-2,888

91-2,981

Instances from various sources

GKAa-d35

20-125204-7,788

Randomly generated

Max-Cut instances from D-Wave Chimera graphs|V|: Number of vertices (Max-Cut), or F of matrix * ∈ ℝ $×$ (QUBO).
|E|: Number of edges (Max-Cut), or number of non-zero entries in * (QUBO)
Notes on Solving QUBOs and Quantum Computing

Thorsten Koch

TU Berlin / Zuse Institute Berlin (ZIB)

413

50

Quantum Annealing versus Digital Computing: An Experimental Comparison
M. Jünger, E. Lobe, P. Mutzel, G. Reinelt, F. Rendl, G., T. Stollenwerk. 2021.
ACM J. Exp. Algorithmics 26, Article 1.9, doi: 10.1145/3459606
This is paper makes a very detailed and precise comparison with the following conclusion:“However, we should stress the fact that exact optimization requires a lot of time toprove optimality, and thus it is not fair to compare their times with the heuristic
times, but even with this additional burden, the exact algorithms are faster than
D-Wave on a large portion of the sample.
[…]
It may well be (and we hope) that the exciting new quantum computer technology will
make leaps in the future, but in our experiments, we have certainly not observed
superior performance of quantum annealing in comparison to “classical” methods.”

Notes on Solving QUBOs and Quantum Computing

Thorsten Koch

TU Berlin / Zuse Institute Berlin (ZIB)

53

Comparison of the new solver MAX-Reaper (new) and Gurobi 9.5 MIQP (Grb)Time limit: 1 h , single-threaded, Intel Xeon Gold 5122 3.60 GHz, 96 GB, Sep. 2022
Notes on Solving QUBOs and Quantum Computing

Thorsten Koch

TU Berlin / Zuse Institute Berlin (ZIB)

414

52

Running Parallel
Results with non-negatedweights for torusg3/pm3
88 core E7-8880v4@2.20GHz

Notes on Solving QUBOs and Quantum Computing

Thorsten Koch

TU Berlin / Zuse Institute Berlin (ZIB)

55

Comparing to Mc-Sparse

MC was used in thecomparison with the
Quantum Annealers
On the previous slide.
We see still substantial
room for performance
improvement on solving
QUBOs on digitalcomputers.
Experience shows that thisimprovement will happen
esp. on those instances
who are now difficult.

MS data from: Charfreitag, Jünger, Mallach, Mutzel, ALENEX 2022, doi:10.1137/1.9781611977042.5
Mc-Sparse: Exact solutions of sparse maximum cut and sparse unconstrained binary quadratic optimization problems.
Notes on Solving QUBOs and Quantum Computing

Thorsten Koch

TU Berlin / Zuse Institute Berlin (ZIB)

415

54

https://blogs.nvidia.com/blog/2021/11/09/cuquantum-world-record/
Theoreticalcomputerscientist
NVIDIA Sets World
Record for
Quantum
Computing Simulation With cuQuantum Runningon DGX
SuperPOD
Not theoptimalsolution.
mean something different by
We need 0.2 s on a workstation to“efficiently”
find this quality solution.
by thisdefinition
just finding
Driving toward +that
future,NVIDIA
created the largest ever simulation of a quantum algorithm for
2 days to find the optimal solution,a solution is not ”solving” it.
solving the MaxCut problem using cuQuantum, our SDK for accelerating quantum
circuitsimulations
3 more days
to prove
optimality.on aGPU.
In the math world, MaxCut is often cited as an example of an optimization problem no known computercan solve efficiently. MaxCut algorithms are used to design large computer networks, find the optimallayout of chips with billions of silicon pathways and explore the field of statistical physics.
[...]
We used the cuTensorNet library in cuQuantum running on NVIDIA’s in-house supercomputer, Selene, tosimulate a quantum algorithm to solve the MaxCut problem. Using 896 GPUs to simulate 1,688 qubits,
we were able to solve a graph with a whopping 3,375 vertices. That’s 8x more qubits than the previous
largest quantum simulation.
Our solution was also highly accurate, reaching 96% of the best-known answer. We set this new record
with an algorithm developed by NVIDIA researchers and an open-source framework.
Notes on Solving QUBOs and Quantum Computing

Thorsten Koch

TU Berlin / Zuse Institute Berlin (ZIB)57

https://blogs.nvidia.com/blog/2021/11/09/cuquantum-world-record/
NVIDIA Sets World Record for Quantum Computing Simulation With cuQuantum Running on DGX SuperPOD

Driving toward that future, NVIDIA created the largest ever simulation of a quantum algorithm forsolving the MaxCut problem using cuQuantum, our SDK for accelerating quantum circuit simulations on aGPU.
In the math world, MaxCut is often cited as an example of an optimization problem no known computercan solve efficiently. MaxCut algorithms are used to design large computer networks, find the optimallayout of chips with billions of silicon pathways and explore the field of statistical physics.
[...]
We used the cuTensorNet library in cuQuantum running on NVIDIA’s in-house supercomputer, Selene, tosimulate a quantum algorithm to solve the MaxCut problem. Using 896 GPUs to simulate 1,688 qubits,
we were able to solve a graph with a whopping 3,375 vertices. That’s 8x more qubits than the previous
largest quantum simulation.
Our solution was also highly accurate, reaching 96% of the best-known answer. We set this new record
with an algorithm developed by NVIDIA researchers and an open-source framework.
Notes on Solving QUBOs and Quantum Computing

Thorsten Koch

TU Berlin / Zuse Institute Berlin (ZIB)

416

56

Looking backWhat people through about computers in the early 1960s
The Thinking machines (MIT 1961)
https://www.youtube.com/watch?v=aygSMgK3BEM
https://www.youtube.com/watch?v=5YBIrc-6G-0
If you check the movie, you will find, that many of the ideas mentioned became true.
Only in the movie they mention “within 10 years” while in practice its now 60 years later.

Current state in Quantum Computing▷ Still trying to figure out how to build a QC
▷ Doing gate level algorithms
This is comparable to digital computers in the 50s▷ QC is about where the Wright bros were with planes.
Getting a 747 took 67 more years.
▷ There is much to be done in Quantum computing.
It will take time and effort.
Notes on Solving QUBOs and Quantum Computing

Thorsten Koch

TU Berlin / Zuse Institute Berlin (ZIB)

63

There is still a lot missing (or waiting to come)SCIP-Jack is faster than problem specific state-of-the-art solvers for several well-known Steiner problem
variants. Example: The Rooted Prize-Collecting Steiner-Tree-Problem (RPC-STP is NP-hard)
Given an undirected graph c = (d, e), a root Ü ∈ d, edge-weights ( ∶ e → ℚJ+ , andnode-weights à ∶ d → ℚJ+ , a tree B = (d& , e_B) in c is required such that Ü ∈ B andis minimized.

^ B ≔ j (K +
K∈2>

j à.

.∈(\M>

DIMACS fiber network instances, hard instances (>20,000 edges):▷ first publication (Ljubic ’04): > 4,000 s*▷ SCIP-Jack at DIMACS (1st )> 100 sRehfeldt, Koch (2021)▷ best other solver:> 300 s
Implications, conflicts, and reductions for Steiner trees▷ current SCIP-Jack:<1s
doi: 10.1007/978-3-030-73879-2_33
Notes on Solving QUBOs and Quantum Computing

Thorsten Koch

TU Berlin / Zuse Institute Berlin (ZIB)

417

58

https://physicsworld.com/a/conquering-the-challenge-of-quantum-optimization
Quantum computers are often touted as the solution to all our problems. They are expected to curedisease, alleviate world hunger and even help mitigate the effects of climate change. Fuelled by thisenthusiasm, a number of quantum computing firms have started joining established markets. However,despite this interest, there is still a lot of uncertainty around the near-term uses of quantum computers.
A crucial question facing quantum researchers today, in both academia and industry, is a pretty
fundamental one: what problems are best solved with these devices?
[…]
There is, nevertheless, one point on which everyone seems to agree: it is very likely that some problems
exist where quantum optimization is provably superior to classical methods, but these problems will
likely occur in the realm of physics and not in finance or industrial operations. “Nature is quantum. If
nature can solve a problem, so should quantum computers,” says França, who is confident aboutproblems involving molecules or quantum materials like superconductors. “The strongest case forvariational algorithms,” Aaronson says, “seems to be on problems that are themselves quantum.”
Training Variational Quantum Algorithms Is NP-Hard
https://journals.aps.org/prl/abstract/10.1103/PhysRevLett.127.120502
Notes on Solving QUBOs and Quantum Computing

Thorsten Koch

TU Berlin / Zuse Institute Berlin (ZIB)

65

Looking forward“We tend to be too optimistic about the short run, too pessimistic about the long run.”― J. Preskill▷ Practical MILP solving on digital computers got arguably faster at least 42% every year

(combined hard+software) during the last 40 years. This is an exponential speed-up.
Progress in Mathematical Programming Solvers from 2001 to 2020, K., Berthold, Pedersen, Vanaret, ZR-21-20▷ Regarding our QUBO solver, there are still plenty algorithmic improvements possible. Additionally, we will add GPUbased heuristics and distributed memory parallelization to able to run up to 1 million cores.
▷ QC likely will evolve for some very specific applications, first likely around Quantum Simulation. This is the original
application of QC. It has some strong inherit advantage compared to classical computers.

Recommended further Reading:
NP-complete Problems and Physical Reality, Scott Aaronson, https://arxiv.org/abs/quant-ph/0502072
An Introduction to Quantum Computing, without the Physics, Giacomo Nannicini, https://arxiv.org/abs/1708.03684

https://www.scientificamerican.com/article/will-quantum-computing-ever-live-up-to-its-hype
https://www.linkedin.com/pulse/quantum-computing-hype-bad-science-victor-galitski-1c?trk=public_post-content_share-article_title
https://www.scottaaronson.com/talks/speedup.ppt
https://scottaaronson.blog/?p=5387 (and the rest of his Quantum blog entries)
https://physicsworld.com/a/conquering-the-challenge-of-quantum-optimization
https://m-malinowski.github.io/2022/03/11/forecasting-future-of-qc.html
https://www.technologyreview.com/2022/03/28/1048355/quantum-computing-has-a-hype-problem
Notes on Solving QUBOs and Quantum Computing

Thorsten Koch

TU Berlin / Zuse Institute Berlin (ZIB)

418

64

3 reasons why a question in industrial optimization might be difficult to answer
1. The question is not well defined,i.e., the modeling is intricate. Very often, in industry, problems are involved and multi-layered.
Determining a precise definition of the problem, the input and output data, and mapping this to amathematically well-defined computable optimization problem can be challenging.
2. The data needed to solve the problem is not fully available.
Many companies struggle hard to consolidate their IT. Getting out precise numbers is often
surprisingly hard. One fundamental reason is decomposition, which has been necessary, at least in
the past, to counter complexity. As a result, everyone only sees either a very little or very simplified
part of the whole picture, and it is very hard to impossible to collect and the data into a coherent set.
3. The resulting problem is computationally hard to solve.
Since the complexity class of discrete optimization problems often is NP-hard, this is not surprising.
However, experience shows, that solving particular instances works surprisingly well and that usually,
the main reason for the inability to solve a problem is its size. For example, the likes of SAP, Amazon,
Google, Huawei all have extremely large-scale supply-chain-type problems at hand. But not so manyothers. And there are surprisingly few small challenging real-world problems unless the time allowed
for solving is very short.
Notes on Solving QUBOs and Quantum Computing

Thorsten Koch

TU Berlin / Zuse Institute Berlin (ZIB)

67

More Links
Twenty Questions for Donald Knuth
17. Andrew Binstock, Dr. Dobb's: At the ACM Turing Centennial in 2012, you stated that you were
becoming convinced that P = N P. Would you be kind enough to explain your current thinking on thisquestion, how you came to it, and whether this growing conviction came as a surprise to you?
https://www.informit.com/articles/article.aspx?p=2213858&WT.mc_id=Author_Knuth_20Questions

Notes on Solving QUBOs and Quantum Computing

Thorsten Koch

TU Berlin / Zuse Institute Berlin (ZIB)

419

66

The 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop on
Advances in Classical and Quantum Algorithms for
Optimization and Machine Learning
September 16th - 19th, 2022, Tokyo (The university of Tokyo), Japan,
and September 21st - 22nd, 2022, Fukuoka (Kyushu University), Japan

Quantum algorithm for stochastic
optimal stopping problems withapplications in financeJoão F. DORIGUELLO
Centre for Quantum Technologies, National University ofS ingapore, S ingapore
joaofd@nus.edu.sg
The famous least squares Monte Carlo (LSM) algorithm [1,2,3] combines linear leastsquare regression with Monte Carlo simulation to approximately solve problems instochastic optimal stopping theory. In this work, we propose a quantum LSM based on
quantum access to a stochastic process, on quantum circuits for computing the optimal
stopping times, and on quantum techniques for Monte Carlo. For this algorithm, we
elucidate the intricate interplay of function approximation and quantum algorithms forMonte Carlo. Our algorithm achieves a nearly quadratic speedup in the runtime
compared to the LSM algorithm under some mild assumptions. Specifically, ourquantum algorithm can be applied to American option pricing and we analyze a casestudy for the common situation of Brownian motion and geometric Brownian motion
processes.

References[1] F. A. Longstaff and E. S. Schwartz. “Valuing American options by simulation: a simple leastsquares approach”. In: The review of financial studies 14.1 (2001), pp. 113–147.
https://doi.org/10.1093/rfs/14.1.113[2] E. Clément, D. Lamberton, and P. Protter. “An analysis of a least squares regression method forAmerican option pricing”. In: Finance and Stochastics 6.4 (2002), pp. 449–471.
https://doi.org/10.1007/s007800200071[3] D. Z. Zanger. “Convergence of a least-squares Monte Carlo algorithm for bounded
approximating sets”. In: Applied Mathematical Finance 16.2 (2009), pp. 123–150.
https://doi.org/10.1080/13504860802516881

421

422

423

424

425

426

max{S − K , }

K

427

S

(⌦, F, P)(Xt )Tt=(E , E) E ✓ Rd

X =x

zt 2 L (E , ⇢t )(Xt )Tt=(Zt )Tt=
Zt = zt (Xt )(Zt )Tt=428

⌧ : ⌦ ! { , ,...,T}Z⌧ (!) := Z⌧ (!) (!)

⌧

">
sup
⌧

E[Z⌧ ]"

sup⌧E[Z⌧ ] = E[Z⌧ ] = max{Z , E[Z⌧ ]}.

⇢⌧T = T ,⌧t = t {Zt ≥ E[Z⌧t+ |Xt ]} + ⌧t+ {Zt < E[Z⌧t+ |Xt ]},E[Z⌧t+ |Xt ]429

tT− .

⇢⌧T = T ,⌧t = t {Zt ≥ E[Z⌧t+ |Xt ]} + ⌧t+ {Zt < E[Z⌧t+ |Xt ]},E[Z⌧t+ |Xt ]430

tT− .

Ht ✓ L (E , ⇢t ) t 2 { , . . . , T }ft
Ht =E[Z⌧t+ |Xt ]ft 2 Ht{ek : E ! R}mk={ek : E ! R}mk=m

E[Z⌧t+ |Xt ] ⇡ ↵t · e~(Xt ) =nX(↵t )j ej (Xt ),j=e~(·) := (e (·), . . . , em (·))>⇥⇤↵t = arg minm E (Z⌧t+ − a · e~(Xt )) .
a2R

{ek : E ! R}mk=↵t↵ t = A−t bt

bt = E[Z⌧t+ e~(Xt )]At 2 Rm⇥m(At )k,l = E[ek (Xt )el (Xt )].

431

ft
( )

(N) T
)t=

(Xt )Tt= , . . . , (Xt

N
(n)

Zt

(n)

= zt (Xt )
N
(n)
⌧t

E[Z⌧t+ |Xt ] ⇡ ↵t · e~(Xt )↵t = arg minm
a2RN

↵ t = A−
t bt

N
X
�
n=

(n)
(n) �
Z (n) − a · e~(Xt ) .
⌧t+

bt =

N

N
X
n=

(n)

(n)

Z (n) e~(Xt )⌧t+

At 2 Rm⇥m(At )k,l =

N

N
X

(n)

(n)

ek (Xt )el (Xt ).

n=

(n)

{⌧ }N
n=
sup⌧ E[Z⌧ ] = max{Z , E[Z⌧ ]}
(
)
N
X
(n)
U = max Z ,Z (n) .
⌧
N
n=

432

( )

N
( )

(N)

(N)

(Xt , . . . , Xt )T
t=
( )
(N) T
(Zt , . . . , Zt )t=(ek (Xt ), . . . , ek (Xt(n)⌧T = Tt=T−))t2[T ],k2[m]−{At }Tt=n 2 [N]↵ t = A−
t N

T−
{A−
t }t=

PN

n=

(n)

(n)

Z (n) e~(Xt )⌧t+n 2 [N]
(n)
(n)
(n)
(n)
(n)
(n)
⌧t = t {Zt ≥ ↵t · e~(Xt )} + ⌧t+ {Zt < ↵t · e~(Xt )}
n
o
P
(n)
U := max Z , N N
Z
(n)
n=
⌧

433

bt = E[Z⌧t+ e~(Xt )]"2( , )
Var(X )  σeO(σ/")At = E[~e (Xt )~e (Xt )> ]µ = E[X ]X|eµ − µ|  "µe(Xt )Tt=
UP |~ i =p(x) = P[X = x ]X pp(x)|xi

x2E T

QT −
t=

P[Xt+ = xt+ |Xt = xt ]h:E !RVh |xi|~ i = |xi|h(x)i.

434

bt = E[Z⌧t+ e~(Xt )]At = E[~e (Xt )~e (Xt )> ]↵ t = A−
t bt

m
⌧t

⌧t
⌧u

t+ uTt+ uT
⌧t

⌧u

⌧t
(k)

Ct

t 2 [T ]k 2 [m](k)
Ct |xi|~ i = |xi|z⌧t (x) ek (xt− )i.

↵ t Wt(

Wt |xi|⌧t+ (x)i|~ i = |xi|⌧t+ (x)i|⌧t (x)i
Wt |xi|~ i = |xi|T i

t 6= T ,t = T.

(k)Vt(k)
Vt |xi|⌧t (x)i|~ i = |xi|⌧t (x)i|z⌧t (x) ek (xt− )i(k)Ct⌧t (x) = t {zt (xt )(k)†:= WT† . . . Wt+
Wt† Vt Wt Wt+ . . . WT .

↵t · ~e (xt )} + ⌧t+ (x) {zt (xt ) < ↵t · ~e (xt )}435

−{At }Tt=− T−{At }t=

WT : |xi| i 7! |xi|T i

t=Tt 6= T

Wt : |xi|⌧t+ (x)i| i 7! |xi|⌧t+ (x)i|⌧t (x)i(k)

Vtk 2 [m]: |xi|⌧t (x)i| i 7! |xi|⌧t (x)i|z⌧t (x) ek (xt− )i(k)bt−WT† . . . Wt† Vt Wt . . . WT
bt− = E[Z⌧t e~(Xt− )]↵t− = A−t− bt−

k 2 [m]W : |xi|⌧ (x)i| i 7! |xi|⌧ (x)i|⌧ (x)i
V : |xi|⌧ (x)i| i 7! |xi|⌧ (x)i|z⌧ (x) iE[Z⌧ �
]WT† . . . W † V W . . . WT
U := max Z , E[Z⌧ ]"2( , )

�N = ⇥ " mσσmin  mint2[T − ] σmin (At )U|U − E[Z⌧ ]| 

e
O

T

⇣

✓

min

" + max minm ka · e~(Xt ) − E[Z⌧t+ |Xt ]kL (⇢t )Tm" σmin⌘t<T a2R

e
O

436

⇣

T m
"σmin

⌘

◆

�

|U − E[Z⌧ ]| T✓

" + max minm ka · e~(Xt ) − E[Z⌧t+ |Xt ]kL (⇢t )t<T a2R"
max t<T mina2Rm ka · e~(Xt ) − E[Z⌧t+ |Xt ]kL (⇢t )
e
O

⇣

Tm
" σmin

⌘

e
O

⇣

T m
"σmin

σmin
Ht =

q

437

⌘

◆

T

438

The 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop on
Advances in Classical and Quantum Algorithms for
Optimization and Machine Learning
September 16th - 19th, 2022, Tokyo (The university of Tokyo), Japan,
and September 21st - 22nd, 2022, Fukuoka (Kyushu University), Japan

Multicriteria Shortest Path Algorithms
Ralf BORNDÖRFER
Zuse Institute Berlin & Freie Universität Berlin, Berlin, Germany
borndoerfer@zib.de
The optimization of paths subject to different criteria such as length, duration, cost,etc. comes up in all kinds of route planning applications; they lead to the Multiobjective
Shortest Path Problem (MOSP) of computing the Pareto front of efficient solutions. We
propose a new “Multiobjective Dijkstra” label-setting algorithm [1,2] that computes aminimum complete set of Pareto optimal paths; it is based on a lexicographicorganization of the label exploration process. In this way, the main data structure, apriority queue, can be kept small, holding at most one label per node of the underlying
graph, and all extracted labels are guaranteed to be efficient. The resulting algorithm
improves the best know complexity bounds in this area. It gives rise to an FPTAS
approximation variant [3], it can be generalized to a time dependent setting (in the FIFOcase), it is parallelizable, and it works in practical implementations for more than two
objectives.

References[1] Pedro Maristany de las Casas, Ralf Borndörfer, Luitgard Kraus, Antonio Sedeñ-Noda, “AnImproved Multiobjective Shortest Path Algorithm”, Computers & Operations Research, Volume135,105424,November 2021, https://doi.org/10.1016/j.cor.2021.105424 2021
[2] Pedro Maristany de las Casas, Luitgard Kraus, Ralf Borndörfer, Antonio Sedeñ-Noda, “Targeted
Multiobjective Dijkstra Algorithm”, ZIB Report, 2021, arXiv:2110.10978,
https://arxiv.org/abs/2110.10978[3] Pedro Maristany de las Casas, Ralf Borndörfer, Luitgard Kraus, Antonio Sedeñ-Noda, “AnFPTAS for Dynamic Multiobjective Shortest Path Problems”, Algorithms, Volume 14, Number
2, pages 1-22, 2021, https://doi.org/https://doi.org/10.3390/a14020043

439

Multiobjective Shortest Path Algorithms
Ralf Borndörfer Pedro Maristany Luitgard Kraus Antonio Sedeño-Noda
Freie Universität Berlin – Zuse Institute Berlin – MODAL MobilityLab
6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop on Advances in Classicaland Quantum Algorithms for Optimization and Machine Learning

21.09.2022

The Airway Network (2D)

Multiobjective Shortest Path Problems | 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop | Sep 21, 2021

440

2

The Time-dependent 2D Flight Planning Problem
Definition (Time-dependent 2D Flight Planning Problem).
Input:▪ Directed graph 𝐷𝐷 = (𝑉𝑉, 𝐴𝐴) (embedded on the Earth)▪ Source and target nodes 𝑠𝑠, 𝑡𝑡 ∈ 𝑉𝑉
▪ Departure time 𝜏𝜏0▪ Travel time functions (TTFs) 𝑡𝑡𝑎𝑎 : ℝ≥0 → ℝ≥0 for each arc 𝑎𝑎 ∈ 𝐴𝐴,
mapping starting time 𝜏𝜏 to traversal time 𝑡𝑡𝑎𝑎 (𝜏𝜏)

Output:▪ 𝑠𝑠𝑠𝑠-path path 𝑝𝑝 = 𝑣𝑣0 , … , 𝑣𝑣𝑛𝑛 minimizing
s.t. 𝜏𝜏𝑖𝑖 = 𝜏𝜏𝑖𝑖−1 + 𝑡𝑡(𝑣𝑣𝑖𝑖−1,𝑣𝑣𝑖𝑖) (𝜏𝜏𝑖𝑖−1 )𝑡𝑡(𝑝𝑝) ≔ σ𝑛𝑛−1
𝑖𝑖=0 𝑡𝑡(𝑣𝑣𝑖𝑖 ,𝑣𝑣𝑖𝑖 +1) 𝜏𝜏𝑖𝑖
Multiobjective Shortest Path Problems | 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop | Sep 21, 2021

3

FRA-SFO: Min. Fuel vs Min. Distance Track

Multiobjective Shortest Path Problems | 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop | Sep 21, 2021

441

5

FRA-MRS: MFT vs. Min. Overflight Cost Track

Skyvector.com
Multiobjective Shortest Path Problems | 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop | Sep 21, 2021

6

The Static 2D Flight Planning Problem
Definition (Static 2D Flight Planning Problem).
Input:▪ Directed graph 𝐷𝐷 = (𝑉𝑉, 𝐴𝐴) (embedded on the Earth)▪ Source and target nodes 𝑠𝑠, 𝑡𝑡 ∈ 𝑉𝑉
▪ Departure time 𝜏𝜏0▪ Cost 𝑐𝑐𝑎𝑎 ∈ ℕ0 for each arc 𝑎𝑎 ∈ 𝐴𝐴
Geben Sie hier eine Formel ein.

Output:▪ 𝑠𝑠𝑠𝑠-path path 𝑝𝑝 = 𝑣𝑣0 , … , 𝑣𝑣𝑛𝑛 minimizing𝑛𝑛−1

𝑐𝑐(𝑝𝑝) ≔ ෍ 𝑐𝑐(𝑣𝑣𝑖𝑖,𝑣𝑣𝑖𝑖 +1)
𝑖𝑖=0

Multiobjective Shortest Path Problems | 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop | Sep 21, 2021

442

7

The Multiobjective Shortest Path Problem
𝑢𝑢

𝑠𝑠

𝑡𝑡
(2,2)

(1,1)

(1,2)

𝑐𝑐2

(2,2)

𝑣𝑣

𝑐𝑐1

Multiobjective Shortest Path Problems | 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop | Sep 21, 2021

8

The Multiobjective Shortest Path Problem

𝑝𝑝

(1,2)

𝑠𝑠

𝑡𝑡
(2,2)

(1,1)

𝑢𝑢

𝑐𝑐2

(2,2)

3

𝑣𝑣

𝑐𝑐(𝑝𝑝)

2

Multiobjective Shortest Path Problems | 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop | Sep 21, 2021

443

𝑐𝑐1

9

The Multiobjective Shortest Path Problem
𝑡𝑡

(1,2)

𝑠𝑠

4

(2,2)

(1,1)

𝑢𝑢

𝑐𝑐2

𝑟𝑟

(2,2)

𝑐𝑐(𝑟𝑟)

𝑐𝑐(𝑝𝑝)

3

𝑣𝑣

4

2

𝑐𝑐1

Multiobjective Shortest Path Problems | 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop | Sep 21, 2021

10

The Multiobjective Shortest Path Problem

𝑠𝑠

𝑡𝑡

𝑞𝑞
(2,2)

4

(2,2)

(1,2)

(1,1)

𝑢𝑢

𝑐𝑐2

3

2

𝑣𝑣

𝑐𝑐(𝑝𝑝)

2

𝑐𝑐(𝑟𝑟)

𝑐𝑐(𝑞𝑞)

3

4

Multiobjective Shortest Path Problems | 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop | Sep 21, 2021

444

𝑐𝑐1

11

The Multiobjective Shortest Path Problem

𝑠𝑠

𝑞𝑞
(2,2)

4

(2,2)

𝑝𝑝

𝑡𝑡

(1,2)

(1,1)

𝑢𝑢

𝑐𝑐2

𝑟𝑟

𝑐𝑐(𝑝𝑝)

3

2

𝑣𝑣

2

𝑐𝑐(𝑟𝑟)

𝑐𝑐(𝑞𝑞)

3

4

𝑐𝑐1

Definition (Dominance). Let 𝑝𝑝, 𝑟𝑟 be 𝑠𝑠𝑠𝑠-paths in 𝐷𝐷 = (𝑉𝑉, 𝐴𝐴), 𝑐𝑐 ∈
𝐴𝐴×𝑑𝑑
ℝ≥0
. Then 𝑝𝑝 dominates 𝑟𝑟 ∶⟺ 𝑝𝑝 ≼ 𝑟𝑟: ⟺ 𝑐𝑐 𝑝𝑝 ≨ 𝑐𝑐 𝑟𝑟 .
Definition (Efficiency). An undominated 𝑠𝑠𝑠𝑠-path is efficient.
Multiobjective Shortest Path Problems | 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop | Sep 21, 2021

12

The Multiobjective Shortest Path Problem
(1,2)

𝑠𝑠

𝑡𝑡

4

(2,2)

(1,1)

𝑢𝑢

𝑐𝑐2

(2,2)

3

2

𝑣𝑣

𝑐𝑐(𝑝𝑝)

2

𝑐𝑐(𝑟𝑟)

𝑐𝑐(𝑞𝑞)

3

4

Definition (Multiobjective Shortest Path Problem (MOSP)).

𝑐𝑐1

Input: Digraph 𝐷𝐷 = 𝑉𝑉, 𝐴𝐴, 𝑐𝑐 , 𝑐𝑐 ∈ ℕ0𝐴𝐴×𝑑𝑑 , 𝑠𝑠 ∈ 𝑉𝑉.
Output: "Minimally complete" set of efficient 𝑠𝑠𝑠𝑠-paths for all 𝑣𝑣 ∈ 𝑉𝑉.
Note: Only one path for each multidimensional objective.
Multiobjective Shortest Path Problems | 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop | Sep 21, 2021

445

13

The Intractability of MOSP
(1,0)

𝑣𝑣1

(0,1)

𝑣𝑣2

(2,0)

(0,0)

𝑣𝑣3

(0,2)

𝑣𝑣4

(0,0)

𝑣𝑣5

…

(2

𝑣𝑣𝑛𝑛−2

(0,2

𝑛𝑛−2
2 )

𝑛𝑛−2
2 , 0)

𝑣𝑣𝑛𝑛−1

(0,0)

𝑣𝑣𝑛𝑛

Example & Observation (Intractability of MOSP, Hansen[1979]). Even for only two objectives, a MOSP can have an
exponential number of efficient paths.

Multiobjective Shortest Path Problems | 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop | Sep 21, 2021

14

Multiobjective Shortest Path Algorithms
(1,2)

𝑠𝑠

𝑡𝑡
(2,2)

(1,1)

𝑢𝑢

(2,2)

𝑣𝑣

Definition (Explored & Permanent Paths). Let 𝑝𝑝 be an 𝑠𝑠𝑠𝑠-path in adigraph 𝐷𝐷 𝑉𝑉, 𝐴𝐴, 𝑐𝑐 , 𝑐𝑐 ∈ ℕ𝑑𝑑0 , s, 𝑣𝑣 ∈ 𝑉𝑉.
a) 𝑝𝑝 explored ∶⟺ 𝑝𝑝 has been seen but is not known to be efficientb) 𝑝𝑝 permanent ∶⟺ 𝑝𝑝 is guaranteed to be (and remain) efficient
Explored 𝑠𝑠𝑠𝑠-paths are stored in a priority queue 𝑄𝑄, and permanent𝑠𝑠𝑠𝑠-paths in a set 𝑃𝑃𝑣𝑣 .
Multiobjective Shortest Path Problems | 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop | Sep 21, 2021

446

15

Multiobjective Shortest Path Algorithms
𝑐𝑐2

𝑝𝑝 ⋠ 𝑞𝑞and𝑝𝑝 ≺𝑙𝑙𝑙𝑙𝑙𝑙 𝑞𝑞

3𝑞𝑞 ⋠ 𝑝𝑝2

𝑐𝑐(𝑝𝑝)

2

𝑐𝑐(𝑞𝑞)

3

𝑐𝑐1

Definition (Lexicographic Order). Let 𝑝𝑝, 𝑞𝑞 be 𝑠𝑠𝑠𝑠-paths in 𝐷𝐷 = (𝑉𝑉, 𝐴𝐴),𝑐𝑐 ∈ ℕ0𝐴𝐴×𝑑𝑑 . Then𝑝𝑝 ≺𝑙𝑙𝑙𝑙𝑙𝑙 𝑞𝑞 ∶⟺ 𝑐𝑐𝑖𝑖 𝑝𝑝 < 𝑐𝑐𝑖𝑖 (𝑞𝑞) for the first 𝑖𝑖 ∈ [𝑑𝑑] s.t. 𝑐𝑐𝑖𝑖 𝑝𝑝 ≠ 𝑐𝑐𝑖𝑖 (𝑞𝑞).
Multiobjective Shortest Path Problems | 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop | Sep 21, 2021

16

Martins's Algorithm (Martins [1984])
Input: MOSP 𝐷𝐷, 𝑐𝑐, 𝑠𝑠
Output: 𝑃𝑃𝑣𝑣 , 𝑣𝑣 ∈ 𝑉𝑉
1. forall 𝑣𝑣 ∈ 𝑉𝑉 do 𝑃𝑃𝑣𝑣 ← ∅ endforall
2. 𝑝𝑝init ← 𝑠𝑠 , 𝑄𝑄 ← {𝑝𝑝init }3. while 𝑄𝑄 ≠ ∅ do4.
𝑝𝑝 ← 𝑄𝑄. extract_min(), 𝑣𝑣 ← head 𝑝𝑝
5.
𝑃𝑃𝑣𝑣 ← 𝑃𝑃𝑣𝑣 ∪ 𝑝𝑝
6.
forall 𝑤𝑤 ∈ 𝛿𝛿 + (𝑣𝑣) do
7.
if 𝑃𝑃𝑤𝑤 ⋠ (𝑝𝑝, 𝑤𝑤) then8.
𝑄𝑄 ← 𝑄𝑄 ∪ 𝑝𝑝, 𝑤𝑤
9.
𝑄𝑄 ← clean_heap
10.
endif
11. endwhile
12. return 𝑃𝑃𝑣𝑣 , 𝑣𝑣 ∈ 𝑉𝑉

Multiobjective Shortest Path Problems | 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop | Sep 21, 2021

447

17

Martins's Algorithm

𝑠𝑠

𝑣𝑣5

(1,1)

𝑣𝑣1

(3,3)

𝑣𝑣2

(3,6)

(4,4)

𝑣𝑣4

𝑠𝑠

𝑣𝑣1

(0,0)

priority queue 𝑄𝑄
𝑣𝑣2

𝑣𝑣3

𝑣𝑣4

𝑣𝑣5

𝑣𝑣3

Multiobjective Shortest Path Problems | 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop | Sep 21, 2021

18

Martins's Algorithm

𝑠𝑠

𝑣𝑣5

(1,1)

𝑣𝑣1

(3,3)

𝑣𝑣2

(3,6)

(4,4)

𝑣𝑣4

𝑠𝑠

𝑣𝑣1

priority queue 𝑄𝑄

(4,1)

𝑣𝑣2

(2,3)

𝑣𝑣3

(3,3)

𝑣𝑣4

(4,4)

𝑣𝑣5

𝑣𝑣3

Multiobjective Shortest Path Problems | 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop | Sep 21, 2021

448

19

Martins's Algorithm

𝑠𝑠

𝑣𝑣5

(1,1)

𝑣𝑣1

(3,3)

𝑣𝑣2

(3,6)

(4,4)

𝑣𝑣4

𝑠𝑠

𝑣𝑣1

priority queue 𝑄𝑄

(4,1)

𝑣𝑣2

𝑣𝑣3

(3,3)

𝑣𝑣4

(4,4)

𝑣𝑣5

(7,8)

𝑣𝑣3

Multiobjective Shortest Path Problems | 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop | Sep 21, 2021

20

Martins's Algorithm

𝑠𝑠

𝑣𝑣5

(1,1)

𝑣𝑣1

(3,3)

𝑣𝑣2

(3,6)

(4,4)

𝑣𝑣4

𝑠𝑠

𝑣𝑣1

priority queue 𝑄𝑄

(4,1)

𝑣𝑣2

𝑣𝑣3

𝑣𝑣4

(4,4)

𝑣𝑣3

Multiobjective Shortest Path Problems | 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop | Sep 21, 2021

449

𝑣𝑣5

7,8
(6,9)

21

Martins's Algorithm

𝑠𝑠

𝑣𝑣5

(1,1)

𝑣𝑣1

(3,3)

𝑣𝑣2

(3,6)

(4,4)

𝑣𝑣4

𝑠𝑠

𝑣𝑣1

priority queue 𝑄𝑄
𝑣𝑣2

𝑣𝑣3

𝑣𝑣4

(4,4)

𝑣𝑣3

𝑣𝑣5

7,8
(6,9)
(5,10)

Multiobjective Shortest Path Problems | 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop | Sep 21, 2021

22

Martins's Algorithm

𝑠𝑠

𝑣𝑣5

(1,1)

𝑣𝑣1

(3,3)

𝑣𝑣2

(3,6)

(4,4)

𝑣𝑣4

𝑠𝑠

𝑣𝑣1

priority queue 𝑄𝑄
𝑣𝑣2

𝑣𝑣3

𝑣𝑣4

𝑣𝑣5

(5,5)

𝑣𝑣3

Multiobjective Shortest Path Problems | 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop | Sep 21, 2021

450

23

Martins's Algorithm
Input: MOSP 𝐷𝐷, 𝑐𝑐, 𝑠𝑠
Output: 𝑃𝑃𝑣𝑣 , 𝑣𝑣 ∈ 𝑉𝑉
1. forall 𝑣𝑣 ∈ 𝑉𝑉 do 𝑃𝑃𝑣𝑣 ← ∅ endforall
2. 𝑝𝑝init ← 𝑠𝑠 , 𝑄𝑄 ← {𝑝𝑝init }3. while 𝑄𝑄 ≠ ∅ do
4.
𝑝𝑝𝑣𝑣∗ ← 𝑄𝑄. extract_min(), 𝑣𝑣 ← head 𝑝𝑝𝑣𝑣∗
5.
𝑃𝑃𝑣𝑣 ← 𝑃𝑃𝑣𝑣 ∪ 𝑝𝑝𝑣𝑣∗
6.
forall 𝑤𝑤 ∈ 𝛿𝛿 + (𝑣𝑣) doa) 𝑄𝑄 can contain exponentially7.
if 𝑃𝑃𝑤𝑤 ⋠ (𝑝𝑝𝑣𝑣∗ , 𝑤𝑤) thenmany 𝑠𝑠𝑠𝑠-paths for any 𝑣𝑣 ∈ 𝑉𝑉.
8.
𝑄𝑄 ← 𝑄𝑄 ∪ 𝑝𝑝𝑣𝑣∗ , 𝑤𝑤b) clean_heap must access9.
𝑄𝑄 ← clean_heap
them all and remove thedominated ones.
10.
endif
c) Heap properties have to be
11. endwhile
restored after deletions.
12. return 𝑃𝑃 , 𝑣𝑣 ∈ 𝑉𝑉
𝑣𝑣

Multiobjective Shortest Path Problems | 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop | Sep 21, 2021

24

Multiobjective Dijkstra Algorithm (BKMS [2021])
Input: MOSP 𝐷𝐷, 𝑐𝑐, 𝑠𝑠a) Paths extracted from 𝑄𝑄 are
efficient.
Output: 𝑃𝑃𝑣𝑣 , 𝑣𝑣 ∈ 𝑉𝑉
1. forall 𝑣𝑣 ∈ 𝑉𝑉 do 𝑃𝑃𝑣𝑣 ← ∅ endforallb) 𝑄𝑄 contains at most one path
2. 𝑝𝑝init ← 𝑠𝑠 , 𝑄𝑄 ← {𝑝𝑝init }per node: the lex-smallest
undominated path at 𝑃𝑃𝑣𝑣 .
3. while 𝑄𝑄 ≠ ∅ do∗∗
4. 𝑝𝑝𝑣𝑣 ← 𝑄𝑄. extract_min(), 𝑣𝑣 ← head 𝑝𝑝𝑣𝑣
5. 𝑃𝑃𝑣𝑣 ← 𝑃𝑃𝑣𝑣 ∪ 𝑝𝑝𝑣𝑣∗
6. 𝑝𝑝𝑣𝑣new ← arglexmin𝑝𝑝𝑢𝑢 ∈𝑃𝑃𝑢𝑢 :𝑢𝑢∈𝛿𝛿 − (𝑣𝑣) 𝑝𝑝𝑢𝑢 , 𝑣𝑣 : 𝑃𝑃𝑣𝑣 ⋠ 𝑝𝑝𝑢𝑢 , 𝑣𝑣 // next cand. label
7.
8.
9.

if 𝑝𝑝𝑣𝑣new ≠ nil then 𝑄𝑄 ← 𝑄𝑄 ∪ {𝑝𝑝𝑣𝑣new } endif
forall 𝑤𝑤 ∈ 𝛿𝛿 + (𝑣𝑣) do

if 𝑃𝑃𝑤𝑤 ⋠ (𝑝𝑝𝑣𝑣∗ , 𝑤𝑤) and 𝑝𝑝𝑣𝑣∗ , 𝑤𝑤 ≺lex 𝑄𝑄. 𝑤𝑤 then 𝑄𝑄. decrease_key 𝑄𝑄, 𝑤𝑤, 𝑝𝑝𝑣𝑣∗ , 𝑤𝑤

endif

10.
endforall
11. endwhile
12. return 𝑃𝑃𝑣𝑣 , 𝑣𝑣 ∈ 𝑉𝑉

Multiobjective Shortest Path Problems | 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop | Sep 21, 2021

451

25

Multiobjective Dijkstra Algorithm

𝑠𝑠

𝑣𝑣5

(1,1)

𝑣𝑣1

(3,3)

𝑣𝑣2

(3,6)

(4,4)

𝑣𝑣4

𝑠𝑠

𝑣𝑣1

(0,0)

priority queue 𝑄𝑄
𝑣𝑣2

𝑣𝑣3

𝑣𝑣4

𝑣𝑣5

𝑣𝑣3

Multiobjective Shortest Path Problems | 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop | Sep 21, 2021

26

Multiobjective Dijkstra Algorithm

𝑠𝑠

𝑣𝑣5

(1,1)

𝑣𝑣1

(3,3)

𝑣𝑣2

(3,6)

(4,4)

𝑣𝑣4

𝑠𝑠

𝑣𝑣1

priority queue 𝑄𝑄

(4,1)

𝑣𝑣2

(2,3)

𝑣𝑣3

(3,3)

𝑣𝑣4

(4,4)

𝑣𝑣5

𝑣𝑣3

Multiobjective Shortest Path Problems | 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop | Sep 21, 2021

452

27

Multiobjective Dijkstra Algorithm

𝑠𝑠

𝑣𝑣5

(1,1)

𝑣𝑣1

(3,3)

𝑣𝑣2

(3,6)

(4,4)

𝑣𝑣4

𝑠𝑠

𝑣𝑣1

priority queue 𝑄𝑄

(4,1)

𝑣𝑣2

𝑣𝑣3

(3,3)

𝑣𝑣4

(4,4)

𝑣𝑣5

(7,8)

𝑣𝑣3

Multiobjective Shortest Path Problems | 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop | Sep 21, 2021

28

Multiobjective Dijkstra Algorithm

𝑠𝑠

𝑣𝑣5

(1,1)

𝑣𝑣1

(3,3)

𝑣𝑣2

(3,6)

(4,4)

𝑣𝑣4

𝑠𝑠

𝑣𝑣1

priority queue 𝑄𝑄

(4,1)

𝑣𝑣2

𝑣𝑣3

𝑣𝑣4

(4,4)

𝑣𝑣5

(6,9)

𝑣𝑣3

Multiobjective Shortest Path Problems | 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop | Sep 21, 2021

453

29

Multiobjective Dijkstra Algorithm

𝑠𝑠

𝑣𝑣5

(1,1)

𝑣𝑣1

(3,3)

𝑣𝑣2

(3,6)

(4,4)

𝑣𝑣4

𝑠𝑠

𝑣𝑣1

priority queue 𝑄𝑄
𝑣𝑣2

𝑣𝑣3

𝑣𝑣4

(4,4)

𝑣𝑣5

(5,10)

𝑣𝑣3

Multiobjective Shortest Path Problems | 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop | Sep 21, 2021

30

Multiobjective Dijkstra Algorithm

𝑠𝑠

𝑣𝑣5

(1,1)

𝑣𝑣1

(3,3)

𝑣𝑣2

(3,6)

(4,4)

𝑣𝑣4

𝑠𝑠

𝑣𝑣1

priority queue 𝑄𝑄
𝑣𝑣2

𝑣𝑣3

𝑣𝑣4

𝑣𝑣5

(5,5)

𝑣𝑣3

Multiobjective Shortest Path Problems | 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop | Sep 21, 2021

454

31

Multiobjective Dijkstra Algorithm
Theorem (Complexity of the MDA). Let (𝐷𝐷, 𝑐𝑐, 𝑠𝑠) be a MOSP and let▪ 𝑛𝑛 number of nodes▪ 𝑚𝑚 number of arcs
▪ 𝑁𝑁 total number of efficient paths▪ 𝑁𝑁max maximal number of efficient paths at a single node.
Then the complexities of Martins's Algorithm and the MDA are
Algorithm

Martins's Algorithm

MDA

Run time

𝑂𝑂(𝑑𝑑𝑁𝑁 2 𝑛𝑛)

2
𝑂𝑂 𝑑𝑑𝑑𝑑 log 𝑛𝑛 + 𝑑𝑑𝑁𝑁𝑚𝑚𝑚𝑚𝑚𝑚
𝑚𝑚 .

Breugem et. al. [2017]

Proof (Sketch):a) 𝑁𝑁 log 𝑛𝑛 = # iterations × complexity of extract_min.
b)2
𝑚𝑚 = complexity of building paths and checking dominance
𝑁𝑁𝑚𝑚𝑚𝑚𝑚𝑚

Note: 𝑃𝑃𝑣𝑣 ≼ 𝑝𝑝 takes 𝑂𝑂 𝑑𝑑 𝑃𝑃𝑣𝑣

≤ 𝑂𝑂 𝑑𝑑 𝑁𝑁max .

Multiobjective Shortest Path Problems | 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop | Sep 21, 2021

32

Many efficient flight paths exist (time vs fuel).

Multiobjective Shortest Path Problems | 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop | Sep 21, 2021

455

33

Approximating the Set of Efficient Solutions
𝑐𝑐2

𝑐𝑐1

Multiobjective Shortest Path Problems | 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop | Sep 21, 2021

34

Approximating the Set of Efficient Solutions
𝑐𝑐2
𝑐𝑐(𝑝𝑝)

𝑐𝑐(𝑞𝑞)

1
𝑐𝑐(𝑝𝑝)
1 + 𝜖𝜖𝑐𝑐1
Definition (𝝐𝝐-Dominance). Let 𝑝𝑝, 𝑞𝑞 be 𝑠𝑠𝑠𝑠-paths, 𝜖𝜖 > 0. Then𝑝𝑝 𝝐𝝐-dominates 𝑞𝑞 ∶⟺ 𝑝𝑝 ≼𝜖𝜖 𝑞𝑞 ∶⟺ 𝑐𝑐 𝑝𝑝 ≤ 1 + 𝜖𝜖 𝑐𝑐 𝑞𝑞 .
Definition (𝝐𝝐-Cover). 𝑃𝑃𝜖𝜖 is an 𝝐𝝐-cover for a MOSP (𝐷𝐷, 𝑐𝑐, 𝑠𝑠) if forevery 𝑠𝑠𝑠𝑠-path 𝑝𝑝 there is an 𝑠𝑠𝑠𝑠-path 𝑝𝑝′ ∈ 𝑃𝑃𝜖𝜖 s.t. 𝑝𝑝𝑝 ≼𝜖𝜖 𝑝𝑝.
Definition (FPTAS). An FTPAS computes for any 𝜖𝜖 > 0 and any
MOSP (𝐷𝐷, 𝑐𝑐, 𝑠𝑠) an 𝜖𝜖-cover in time polynomial in ⟨𝐷𝐷, 𝑐𝑐, 𝑠𝑠⟩ and 1Τ𝜖𝜖 .
Multiobjective Shortest Path Problems | 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop | Sep 21, 2021

456

35

Approximating the Set of Efficient Solutions
𝑐𝑐2 (log scale)𝑐𝑐1 (log scale)Definition (𝐩𝐩𝐩𝐩𝐩𝐩-function, Tsaggouris & Zaroliagis [2006]). The
lower left corner of the cell of path 𝑝𝑝 is assigned grid coordinates0,𝑐𝑐𝑖𝑖 𝑝𝑝 = 0,
1
log
𝑐𝑐
𝑝𝑝
𝑖𝑖 ∈ 𝑑𝑑 , 𝑟𝑟 = 1 + 𝜖𝜖 𝑛𝑛−1 .
pos𝑖𝑖 𝑝𝑝 ≔ ൞𝑖𝑖
1+, 𝑐𝑐𝑖𝑖 𝑝𝑝 ≠ 0,
log 𝑟𝑟Multiobjective Shortest Path Problems | 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop | Sep 21, 2021

36

Approximating the Set of Efficient Solutions
𝑐𝑐2 (log scale)𝑐𝑐1 (log scale)Observation (Cells and Coverage). Let 𝑝𝑝, 𝑞𝑞 be 𝑠𝑠𝑠𝑠-paths. Thenpos 𝑝𝑝 ≤ pos 𝑞𝑞 ⟹ 𝑝𝑝 ≼𝜖𝜖 𝑞𝑞.
Multiobjective Shortest Path Problems | 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop | Sep 21, 2021

457

37

Approx. Multiobjective Dijkstra Alg. (BKMS [2021])
Input: MOSP 𝐷𝐷, 𝑐𝑐, 𝑠𝑠
Output: 𝑃𝑃𝑣𝑣 , 𝑣𝑣 ∈ 𝑉𝑉
1. forall 𝑣𝑣 ∈ 𝑉𝑉 do 𝑃𝑃𝑣𝑣 ← ∅ endforall
2. 𝑝𝑝init ← 𝑠𝑠 , 𝑄𝑄 ← {𝑝𝑝init }3. while 𝑄𝑄 ≠ ∅ do
4. 𝑝𝑝𝑣𝑣∗ ← 𝑄𝑄. extract_min(), 𝑣𝑣 ← head 𝑝𝑝𝑣𝑣∗
5. 𝑃𝑃𝑣𝑣 ← 𝑃𝑃𝑣𝑣 ∪ 𝑝𝑝𝑣𝑣∗
6. 𝑝𝑝𝑣𝑣new ← arglexmi𝑛𝑛𝑝𝑝𝑢𝑢 ∈𝑃𝑃𝑢𝑢 :𝑢𝑢∈𝛿𝛿 − (𝑣𝑣) 𝑝𝑝𝑢𝑢 , 𝑣𝑣 : 𝑃𝑃𝑣𝑣 ⋠𝜖𝜖 𝑝𝑝𝑢𝑢 , 𝑣𝑣
7.
8.
9.

if 𝑝𝑝𝑣𝑣new ≠ nil then 𝑄𝑄 ← 𝑄𝑄 ∪ {𝑝𝑝𝑣𝑣new } endif
forall 𝑤𝑤 ∈ 𝛿𝛿 + (𝑣𝑣) do

if 𝑃𝑃𝑤𝑤 ⋠𝜖𝜖 (𝑝𝑝𝑣𝑣∗ , 𝑤𝑤) and 𝑝𝑝𝑣𝑣∗ , 𝑤𝑤 ≺lex 𝑄𝑄. 𝑤𝑤 then 𝑄𝑄. decrease_key 𝑄𝑄, 𝑤𝑤, 𝑝𝑝𝑣𝑣∗ , 𝑤𝑤

endif

10.
endforall
11. endwhile
12. return 𝑃𝑃𝑣𝑣 , 𝑣𝑣 ∈ 𝑉𝑉

Multiobjective Shortest Path Problems | 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop | Sep 21, 2021

38

Approximative Multiobjective Dijkstra Algorithm
Lemma (Correctness of the Approximate MDA). Let 𝑝𝑝 be an
efficient 𝑠𝑠𝑠𝑠-path with 𝑘𝑘 arcs. Then the output of the Approximate
MDA contains an 𝑠𝑠𝑠𝑠-path 𝑝𝑝𝑝 s.t.
𝑐𝑐 𝑝𝑝′ ≤ 𝑟𝑟 𝑘𝑘 𝑐𝑐 𝑝𝑝 .

Corollary (Correctness of the Approximate MDA). Under theconditions of the above Lemma,
𝑐𝑐 𝑝𝑝′ ≤ (1 + 𝜖𝜖)𝑐𝑐 𝑝𝑝 .
1

Proof. 𝑘𝑘 ≤ 𝑛𝑛 − 1 and 𝑟𝑟 = 1 + 𝜖𝜖 𝑛𝑛−1 .

Corollary (𝝐𝝐-Cover). The Approximate MDA computes an 𝜖𝜖-cover.

Multiobjective Shortest Path Problems | 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop | Sep 21, 2021

458

39

Approximative Multiobjective Dijkstra Algorithm
Theorem (Complexity of the Approximate MDA). The run timeof the Approximate MDA is
where 𝐶𝐶 ≔𝑂𝑂(𝑑𝑑𝒞𝒞𝑛𝑛 log 𝑛𝑛 + 𝑑𝑑𝒞𝒞 2 𝑚𝑚),max 𝑐𝑐𝑖𝑖 𝑎𝑎 .

𝑖𝑖∈ 𝑑𝑑 ,𝑎𝑎∈𝐴𝐴

Proof. The pos-function takes at most
𝑑𝑑
𝑛𝑛
𝒞𝒞 ≔
log 𝑛𝑛𝑛𝑛
𝜖𝜖values. Hence, every node can have a path in at most 𝒞𝒞 cells, and thenumber of output paths is at most 𝑛𝑛𝑛𝑛. As
2
𝑂𝑂 𝑑𝑑𝑑𝑑 log 𝑛𝑛 + 𝑑𝑑𝑁𝑁𝑚𝑚𝑚𝑚𝑚𝑚
𝑚𝑚𝑁𝑁≤𝑛𝑛𝑁𝑁max≤

the claim follows from 𝑁𝑁max = 𝒞𝒞.

2
𝑂𝑂 𝑑𝑑𝑁𝑁max 𝑛𝑛 log 𝑛𝑛 + 𝑑𝑑𝑁𝑁𝑚𝑚𝑚𝑚𝑚𝑚
𝑚𝑚 ,Corollary (FPATS). The Approximate MDA is an FPTAS for MOSP.
Multiobjective Shortest Path Problems | 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop | Sep 21, 2021

40

Computational Results on Exponential Graphs

Orange: MDA, Blue: Martins
Multiobjective Shortest Path Problems | 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop | Sep 21, 2021

459

41

Computational Results on Grid Graphs

Orange: MDA, Blue: Martins
Multiobjective Shortest Path Problems | 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop | Sep 21, 2021

42

Back to the 2D Flight Planning Problem
Definition (Time-dependent 2D Flight Planning Problem).
Input:▪ Directed graph 𝐷𝐷 = (𝑉𝑉, 𝐴𝐴) (embedded on the Earth)▪ Source and target nodes 𝑠𝑠, 𝑡𝑡 ∈ 𝑉𝑉
▪ Travel time functions (TTFs) 𝑡𝑡𝑎𝑎 : ℝ≥0 → ℝ≥0 for each arc 𝑎𝑎 ∈ 𝐴𝐴,
mapping starting time 𝜏𝜏 to traversal time 𝑡𝑡𝑎𝑎 (𝜏𝜏)▪ Departure time 𝜏𝜏0

Output:▪ 𝑠𝑠𝑠𝑠-path path 𝑝𝑝 = 𝑣𝑣0 , … , 𝑣𝑣𝑛𝑛 minimizing𝑡𝑡(𝑝𝑝) ≔ σ𝑛𝑛−1s.t. 𝜏𝜏𝑖𝑖 = 𝜏𝜏𝑖𝑖−1 + 𝑡𝑡(𝑣𝑣𝑖𝑖−1,𝑣𝑣𝑖𝑖) (𝜏𝜏𝑖𝑖−1 )
𝑖𝑖=0 𝑡𝑡(𝑣𝑣𝑖𝑖 ,𝑣𝑣𝑖𝑖 +1) 𝜏𝜏𝑖𝑖
Multiobjective Shortest Path Problems | 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop | Sep 21, 2021

460

43

Back to the 2D Flight Planning Problem
Observation (Dynamic Arc Costs). Let 𝑐𝑐: 𝐴𝐴 × ℝ𝑑𝑑≥0 → ℝ𝑑𝑑≥0 be a
dynamic arc cost function. Then the cost of an 𝑠𝑠𝑠𝑠-path (𝑝𝑝, 𝑎𝑎) isdefined recursively as
𝑐𝑐 𝑠𝑠 ≔ 0,

𝑐𝑐 𝑝𝑝, 𝑎𝑎 ≔ 𝑐𝑐 𝑝𝑝 + 𝑐𝑐 𝑎𝑎, 𝑐𝑐 𝑝𝑝 .

Proposition & Definition (Dynamic MDA & FIFO Property). If a
dynamic arc cost function satisfies the FIFO property𝑥𝑥 ≤ 𝑦𝑦 ⟹ 𝑥𝑥 + 𝑐𝑐 𝑎𝑎, 𝑥𝑥 ≤ 𝑦𝑦 + 𝑐𝑐 𝑎𝑎, 𝑦𝑦

MDA s.t. dynamic arc costs is correct.

∀𝑥𝑥, 𝑦𝑦, 𝑎𝑎,

Proposition (Dynamic FPTAS). If a dynamic arc cost function ispiecewise linear with positive intercepts, the Approximate MDA iscorrect.

Multiobjective Shortest Path Problems | 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop | Sep 21, 2021

44

Back to the 2D Flight Planning Problem

Labels saved by Dynamic FPTAS on 2D flight planning instances.
Multiobjective Shortest Path Problems | 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop | Sep 21, 2021

461

45

Back to the 2D Flight Planning Problem

Time saved by Dynamic FPTAS on 2D flight planning instances.
Multiobjective Shortest Path Problems | 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop | Sep 21, 2021

46

Summary and 注目してくれてありがとう▪ Dynamic and Static 2D Flight Planning Problem▪ Multiobjective Shortest Path Problem (MOSP)▪ Martins's Algorithm▪ Multiobjective Dijkstra Algorithm (MDA)▪ Approximate MDA (FPTAS)▪ Dynamic Arc Cost Versions▪ A* Versions▪ Papers @ https://www.zib.de/projects/flight-trajectory-optimization-airway-networks

Multiobjective Shortest Path Problems | 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop | Sep 21, 2021

462

47

The 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop on
Advances in Classical and Quantum Algorithms for
Optimization and Machine Learning
September 16th - 19th, 2022, Tokyo (The university of Tokyo), Japan,
and September 21st - 22nd, 2022, Fukuoka (Kyushu University), Japan

Randomized subspace regularized
Newton method for unconstrained
non-convex optimization
Pierre-Louis POIRIONRIKEN-AIP, Tokyo, Japanpierre-louis.poirion@riken.jp
In this talk we present a randomized subspace regularized Newton method for a nonconvex function. We show that our method has global convergence under appropriate
assumptions and its convergence rate is the same as that of the full regularized Newton
method. Furthermore, we can obtain a local linear convergence rate, under some
additional assumptions, and prove that this rate is the best we can hope when using
random subspace.

463

Randomized Subspace Newton Method for
Unconstrained Non-Convex Optimization
The 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop
Pierre-Louis Poirion (RIKEN-AIP)
joint work with Terunari Fuji and Akiko Takeda

November 2, 2022

November 2, 2022

1 / 23

November 2, 2022

2 / 23

Overview

1
2
3
4
5

Introduction
Global convergence
Local convergence
Numerical experiments
Summary464

The gist
Non-convex unconstrained minimization
min f (x),x2Rnwhere f : Rn ! R is twice di↵erentiable

November 2, 2022

3 / 23

November 2, 2022

3 / 23

The gistNon-convex unconstrained minimization
min f (x),x2Rnwhere f : Rn ! R is twice di↵erentiable

Subspace optimization
min f (x + P > u),u2Rs

where P 2 Rs⇥n is a random matrix.

465

The gist
Non-convex unconstrained minimization
min f (x),x2Rnwhere f : Rn ! R is twice di↵erentiable

Subspace optimizationmins f (x + P > u),u2R

where P 2 Rs⇥n is a random matrix.

Can we speed up the computation time?
Global and local convergence properties?

November 2, 2022

3 / 23

Previous works
Random Subspace Newton (RSN) [Gower et al., 2019](f is convex)
By computing the Newton direction on the function u 7! f (xk + Pk> uk ),we obtain uk = −(Pk r2 f (xk )Pk> )−1 Pk rf (xk ), hence
xk+1 = xk − tk Pk> (Pk r2 f (xk )Pk> )−1 Pk rf (xk ).
They prove global sub-linear convergence and local linear convergence if f
is strongly convex.

November 2, 2022

466

4 / 23

Previous works
Random Subspace Newton (RSN) [Gower et al., 2019](f is convex)
By computing the Newton direction on the function u 7! f (xk + Pk> uk ),we obtain uk = −(Pk r2 f (xk )Pk> )−1 Pk rf (xk ), hence
xk+1 = xk − tk Pk> (Pk r2 f (xk )Pk> )−1 Pk rf (xk ).
They prove global sub-linear convergence and local linear convergence if f
is strongly convex.
[Hanzely et al., 2020]: Cubically-regularized subspace Newton
method.
[Kovalev et al., 2020]: random subspace version of the BFGS method.
[Roberts and Royer, 2022]: probabilistic direct-search method in
reduced random spaces (non-convex problems). The authors provesub-linear convergence.
November 2, 2022

4 / 23

Our work
Based on regularized Newton method (RNM) for the unconstrained
non-convex optimization [Ueda and Yamashita, 2010], we propose therandomized subspace regularized Newton method (RS-RNM):
dk = −Pk> (Pk r2 f (xk )Pk> + ⌘k Is )−1 Pk rf (xk ),xk+1 = xk + tk dk ,where ⌘k is defined to ensure Pk r2 f (xk )Pk> + ⌘k Is � 0 and tk satisfies
Armijo’s rule.

November 2, 2022

467

5 / 23Our work
Based on regularized Newton method (RNM) for the unconstrained
non-convex optimization [Ueda and Yamashita, 2010], we propose therandomized subspace regularized Newton method (RS-RNM):
dk = −Pk> (Pk r2 f (xk )Pk> + ⌘k Is )−1 Pk rf (xk ),xk+1 = xk + tk dk ,where ⌘k is defined to ensure Pk r2 f (xk )Pk> + ⌘k Is � 0 and tk satisfies
Armijo’s rule.
In [Ueda and Yamashita, 2010] the authors prove global sub-linear
convergence and local quadratic convergence under local-error boundcondition.
Can we extend these results to the random subspace setting ?

November 2, 2022

5 / 23What is Random Projection

n-dimensionals-dimensional
X
s=O

⇣

log |X |"2

November 2, 2022

468⌘

6 / 23

Random Projection
Lemma JLL
Let P 2 Rd⇥n , Pij ⇠ N(0, 1/s), i.i.d..
Then for any x 2 Rn and " 2 (0, 1), we have
Prob [(1 − ")kxk22  kPxk22  (1 + ")kxk22 ] ≥ 1 − 2 exp(−C"2 s),
where C is an absolute constant.
n-dimensionals-dimensional
X
s=O

⇣

log |X |"2

November 2, 2022⌘

7 / 23

Algorithm 1 Randomized subspace regularized Newton method (RS-RNM)
input: x0 2 Rn , γ ≥ 0, c1 > 1, c2 > 0, ↵, β 2 (0, 1)
1: k
0
2: repeat3:
sample a random matrix: Pk ⇠ Gaussian matrix N (0, 1/s)s⇥n
4:compute the regularized sketched hessian:Mk = Pk r2 f (xk )Pk> + c1 ⇤k Is + c2 krf (xk )kγ Is , where ⇤k =max(0, −λmin (Pk r2 f (xk )Pk> ))5:compute the search direction: dk = −Pk> Mk−1 Pk rf (xk )6:apply the backtracking line search with Armijo’s to compute lk ≥ 0such that (1) holds. Set tk = β lk , xk+1 = xk + tk dk and k
k +1
7: until the stopping criteria is satisfied
8: return the last iterate xk

f (xk ) − f (xk + β lk dk ) ≥ −↵β lk gk> dk .

(1)
November 2, 2022

469

8 / 23

Global convergence
Assumption (1)The level set of f at the initial point x0 is compact, i.e.,⌦ := {Rn : f (x)  f (x0 )} is compact.

November 2, 2022

9 / 23Global convergence
Assumption (1)The level set of f at the initial point x0 is compact, i.e.,⌦ := {Rn : f (x)  f (x0 )} is compact.

Assumption (2)123γ  1/2,↵  1/2,

There exists LH > 0 such thatkr2 f (x) − r2 f (y )k  LH kx − y k,where r1 :=8x, y 2 ⌦ + B(0, r1 ),CUg1−γ n, and krf (xk )k  Ug .
c2 sNovember 2, 2022

470

9 / 23

Global convergenceLettmin = min 1,

βc22 s 2
C 2 LH Ug1−2γ n2!

TheoremSuppose that Assumptions (1) and (2) hold. Let↵tmin.
2C(1 + c1 ) ns UH + 2c2 Ugγ⌘⇣
Then, with probability at least 1 − 2m exp(− C40 s) − exp(−s) , we havep=sf (x0 ) − f ⇤≥minkrf (xk )k.
k=0,1,...,m−1
mpThis global O("−2 ) complexity is the same as that obtained in
November 2, 2022
[Ueda and Yamashita, 2010] for the regularized Newton method.

10 / 23

Local convergence

Assume that {xk } converge to a strict local minima x̄. We show thatthe sequence {f (xk )} converges locally linearly to f (x̄)

when f is strongly convex, we cannot aim at local super-linear
convergence using random subspace.

November 2, 2022

471

11 / 23

Local convergence: assumptions
Assumption (2’)
In a neighborhood of x̄, we have
kr2 f (x) − r2 f (y )k  LH kx − y k.

Assumption (3)s= 0.
n!+1 nWe have that s = o(n), that is, lim

November 2, 2022

12 / 23

Local convergence: assumptions
Assumption (2’)
In a neighborhood of x̄, we have
kr2 f (x) − r2 f (y )k  LH kx − y k.

Assumption (3)s= 0.
n!+1 nWe have that s = o(n), that is, lim

Assumption (4)We assume that
1
2

There exists σ 2 (0, 1) such that r = rank(r2 f (x̄)) ≥ σnThere exists ⇢ 2 (0, 3) and C̃ such that in a neighborhood of x̄,f (xk ) − f (x̄) ≥ C̃ kxk − x̄k⇢ holds.
November 2, 2022

472

12 / 23

Proposition 1
Let 0 < "0 < 1. Then under Assumptions (3) and (4.1) there exists
n0 2 N (which depends only on "0 and σ) and a neighborhood B ⇤ ✓ B̄such that if n ≥ n0 , for any x 2 B ⇤ ,Pr2 f (x)P > ⌫(1 − "0 )2 n 2
σ λ̄Is2sholds with probability at least 1 − 6 exp(−s).

November 2, 2022

13 / 23

Proposition 1
Let 0 < "0 < 1. Then under Assumptions (3) and (4.1) there exists
n0 2 N (which depends only on "0 and σ) and a neighborhood B ⇤ ✓ B̄such that if n ≥ n0 , for any x 2 B ⇤ ,Pr2 f (x)P > ⌫(1 − "0 )2 n 2
σ λ̄Is2sholds with probability at least 1 − 6 exp(−s).

Proposition 2
Under Assumptions (1),(2’) and (4). there exists 0 <  < 1, k0 2 N,
n0 2 N, and C̄ > 0 such that if n ≥ n0 , k ≥ k0 , we have with probability
1 − 6(exp(−s) + exp(− C40 s)):f (xk ) − min f (xk + Pk> u) ≥ C̄ (f (xk ) − f (x̄)).
u

November 2, 2022

473

13 / 23

Local convergence: Theorem 1

Theorem
Under Assumptions (1),(2’),(3) and (4), there exists 0 <  < 1, k0 2 N,
and n0 2 N such that if n ≥ n0 , k ≥ k0 , thenf (xk+1 ) − f (x̄)  (f (xk ) − f (x̄)).
holds with probability at least 1 − 6(exp(−s) + exp(− C40 s)).

November 2, 2022

14 / 23

November 2, 2022

15 / 23

Super-linear convergence?
Assumption (5)We assume that(C + 2)2 s < n.

474

Super-linear convergence?
Assumption (5)We assume that(C + 2)2 s < n.

Theorem
Under Assumptions (2’) and (5), if f is locally strongly convex around x̄.
There exists a constant c > 0 such that for k large enough,kxk+1 − x̄k ≥ ckxk − x̄kholds with probability at least 1 − 2 exp(− C40 ) − 2 exp(−s).

November 2, 2022

15 / 23

Super-linear convergence?
Assumption (5)We assume that(C + 2)2 s < n.

Theorem
Under Assumptions (2’) and (5), if f is locally strongly convex around x̄.
There exists a constant c > 0 such that for k large enough,kxk+1 − x̄k ≥ ckxk − x̄kholds with probability at least 1 − 2 exp(− C40 ) − 2 exp(−s).
We deduce from the theorem and the assumptions that there exists a
constant c 0 such thatf (xk+1 ) − f (x̄) ≥ c 0 (f (xk ) − f (x̄)),with probability at least 1 − 2 exp(− C40 ) − 2 exp(−s).

475November 2, 2022

15 / 23

Numerical experiments: Support vector regression
Data: 8i  m, (xi , yi ) 2 Rn ⇥ {0, 1}, we aim minimizing sum of a loss
function and a regularizerm

1 X`(yi − xi> w ) + λkw k2 .
f (w ) =m
i=1Internet advertisements dataset from UCI
repository[Dua and Gra↵, 2017] processed so that the number of
instances is m = 600 and and n = 1500.
Comparison with Gradient Descent (GD) and Regularized Newton
Method (RNM)
Step sizes are all determined by Armijo backtracking line search
The parameters are fixed as follows:
c1 = 2, c2 = 1, γ = 0.5, ↵ = 0.3, β = 0.5, s 2 {100, 200, 400}.
November 2, 2022

16 / 23

November 2, 2022

17 / 23

Loss function`(t) =

2t 2
t2 + 4
`(t)
1
t

−1

O 1

Figure: The robust loss functions.

476

Figure: iterations versus f (w ) (log10 -scale)November 2, 2022

18 / 23

Figure: iterations versus krf (w )k (log10 -scale)).

November 2, 2022

477

19 / 23

Figure: computation time versus krf (w )k (log10 -scale).

November 2, 2022

20 / 23

Future work

Can we find a second order subspace algorithm with local superlinear
convergence ? Full paper: ”T. Fuji, P.L. Poirion, A. Takeda, Randomized

subspace regularized Newton method for unconstrained non-convex
optimization. arXiv:2209.04170, (2022)”

November 2, 2022

478

21 / 23

Reference I
Dua, D. and Gra↵, C. (2017).
UCI machine learning repository.
Gower, R., Kovalev, D., Lieder, F., and Richtárik, P. (2019).
RSN: Randomized Subspace Newton.
Adv. Neural Inf. Process. Syst., 32:616–625.
Hanzely, F., Doikov, N., Richtárik, P., and Nesterov, Y. (2020).
Stochastic subspace cubic Newton method.
In III, H. D. and Singh, A., editors, Proceedings of the 37th International
Conference on Machine Learning, volume 119 of Proceedings of Machine Learning
Research, pages 4027–4038. PMLR.
Kovalev, D., Gower, R. M., Richtárik, P., and Rogozin, A. (2020).
Fast linear convergence of randomized bfgs.
arXiv preprint arXiv:2002.11337.
Roberts, L. and Royer, C. W. (2022).
Direct search based on probabilistic descent in reduced spaces.
arXiv preprint arXiv:2204.01275.
November 2, 2022

22 / 23

Reference II

Ueda, K. and Yamashita, N. (2010).
Convergence properties of the regularized newton method for the unconstrained
nonconvex optimization.
Appl. Math. Optim., 62(1):27–46.

November 2, 2022

479

23 / 23

The 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop on
Advances in Classical and Quantum Algorithms for
Optimization and Machine Learning
September 16th - 19th, 2022, Tokyo (The university of Tokyo), Japan,
and September 21st - 22nd, 2022, Fukuoka (Kyushu University), Japan

Minimax Analysis for Inverse Risk in
Nonparametric Invertible Regression
Akifumi OKUNO
Institute of Statistical Mathematics / RIKEN AIP, Japanokuno@ism.ac.jp

Learning invertibility from data and exploiting an invertible estimator are used in many
domains, such as statistics, econometrics, and machine learning. Although theconsistency and universality of invertible estimators have been well investigated,
analysis on the efficiency of these methods is still under development. In this study, we
study a minimax risk for estimating invertible functions. We first introduce an inverse
L2-risk to evaluate an estimator which preserves invertibility. Then, we derive lower
and upper rates for a minimax inverse risk by exploiting a representation of invertible
functions using level-sets. To obtain an upper bound, we develop an estimator
asymptotically almost everywhere invertible, whose risk attains the derived minimax
lower rate up to logarithmic factors. This work is a joint work with M. Imaizumi (U.
Tokyo), and is based on a preprint of ours [1].

References[1] Akifumi Okuno, Masaaki Imaizumi, “Minimax Analysis for Inverse Risk in NonparametricPlaner Invertible Regression”, CoRR, arXiv preprint https://arxiv.org/abs/2112.00213

481

482

483

484

485

486

487

488

489

490

491

492

493

494

495

496

497

498

499

The 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop on
Advances in Classical and Quantum Algorithms for
Optimization and Machine Learning
September 16th - 19th, 2022, Tokyo (The university of Tokyo), Japan,
and September 21st - 22nd, 2022, Fukuoka (Kyushu University), JapanOn the geometry of periodic timetables
in public transport
Niels LINDNER
Zuse Institute Berlin, Germany
lindner@zib.de
What rhythm is to music, is the timetable to a public transportation system. Many public
transportation networks are operated periodically, and therefore the computation andoptimization of periodic timetables is a frequent and important task. The mathematicalfoundation of periodic timetabling is the Periodic Event Scheduling Problem [1], which iseasy to formulate, has a rich theory, but is notoriously hard to solve. In order to obtain abetter understanding of how to solve periodic timetabling problems, we analyze thegeometry of periodic timetables, and discover surprising connections to tropical and discretegeometry that are beyond the scope of the standard toolbox of combinatorial optimization[2]. We outline how tropical neighborhood search, a new heuristic developed from thesegeometric insights, helped to compute new incumbent solutions for instances of thetimetabling benchmarking library PESPlib [3].

References[1] P. Serafini and W. Ukovich. A mathematical model for periodic scheduling problems. SIAM
Journal on Discrete Mathematics, 2(4):550–581, 1989. https://doi.org/10.1137/0402049[2] E. Bortoletto, N. Lindner, and B. Masing. The tropical and zonotopal geometry of periodic
timetables, 2022. https://doi.org/10.48550/ARXIV.2204.13501[3] E. Bortoletto, N. Lindner, and B. Masing. Tropical Neighbourhood Search: A New Heuristic for
Periodic Timetabling. 22nd Symposium on Algorithmic Approaches for Transportation
Modelling, Optimization, and Systems (ATMOS 2022), 3:1-3:19, 2022.
https://doi.org/10.4230/OASIcs.ATMOS.2022.3

501

On the geometry of periodic timetables
in public transport
Niels Lindner, Enrico Bortoletto, Berenike Masing
MobilityLab
Department Network Optimization
Zuse Institute Berlin
6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop
September 21, 2022

Public Transport...

... is often operated periodically
Images: MaedaAkihiko and Trouper3000, CC-BY-SA 4.0; Rolf Heinrich, Köln, CC-BY 3.0; all via commons.wikimedia.org
Niels Lindner: On the geometry of periodic timetables in public transport

6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop

502

2 / 26

Public Transport...

... is often operated periodically! Periodic Timetable Optimization

Images: MaedaAkihiko and Trouper3000, CC-BY-SA 4.0; Rolf Heinrich, Köln, CC-BY 3.0; all via commons.wikimedia.org
Niels Lindner: On the geometry of periodic timetables in public transport

6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop

3 / 26

Public Transport Planning Cycle
Network Design

strategic
long-term
many decision makers

Line Planning
Periodic Timetable
Optimization

Timetabling
Vehicle Scheduling
Duty Scheduling
Crew Schedulingoperational
short-term
few decision makers

Real-Time Management
Bussieck et al.: Discrete optimization in public rail transport, 1997
Liebchen: Periodic timetable optimization in public transport, 2006

Niels Lindner: On the geometry of periodic timetables in public transport

6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop

503

4 / 26

A Line Network: Tokyo Subway

Tokyo Metro Co., Ltd.
Niels Lindner: On the geometry of periodic timetables in public transport

6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop

5 / 26

From Line Networks to Event-Activity Networks

Line Plan: 3 bidirectional lines, same frequency

Niels Lindner: On the geometry of periodic timetables in public transport

6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop504

6 / 26

From Line Networks to Event-Activity Networks

Event-Activity Network: directed graph G
Niels Lindner: On the geometry of periodic timetables in public transport

6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop

6 / 26

Periodic Timetable Optimization

Events:• arrival◦ departure
Activities:! drive, dwell, turn! transfer...

Event-Activity Network: directed graph G
Niels Lindner: On the geometry of periodic timetables in public transport

6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop

505

7 / 26

Periodic Timetable Optimization[23, 28], 753

Bounds [`, u]
I driving times
I minimum transfer times
I maximum dwell times
I minimum headway times
I ...
Weights w:
I passenger load
I turnaround penalties
I ...
Period time T:
I e.g., T = 60 for 1 hour,
resolution of 1 minute(G, T, `, u, w)
Niels Lindner: On the geometry of periodic timetables in public transport

6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop

8 / 26

Periodic Event Scheduling Problem (PESP)Given
G = (V , A)
T2N
` 2 RA
u 2 RA
w 2 RA≥0find⇡ 2 RVx 2 RAevent-activity network,
period time,
lower bounds,
upper bounds,
weights,

periodic timetable,
periodic tension

such that(1) ⇡j − ⇡i ⌘ xij mod T for all ij 2 A,(2) `  x  u,(3) w> x is minimum,or decide that no such (⇡, x) exists.
(Serafini and Ukovich, 1989)
Niels Lindner: On the geometry of periodic timetables in public transport

6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop

506

9 / 26

Periodic Event Scheduling Problem (PESP)Given
G = (V , A)
T2N
` 2 RA
u 2 RA
w 2 RA≥0find⇡ 2 RVx 2 RAevent-activity network,
period time,
lower bounds,
upper bounds,
weights,

periodic timetable,
periodic tension

Incidence-based MIP formulation:
Minimizes.t.

w> x⇡j − ⇡i = xij − Tpij ,`ij  xij  uij ,⇡i 2 R,pij 2 Z,

p 2 ZAij 2 A,ij 2 A,i 2 V,
ij 2 A.

periodic offsets

such that(1) ⇡j − ⇡i ⌘ xij mod T for all ij 2 A,(2) `  x  u,(3) w> x is minimum,or decide that no such (⇡, x) exists.
(Serafini and Ukovich, 1989)

Niels Lindner: On the geometry of periodic timetables in public transport

6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop

9 / 26

Periodic Event Scheduling Problem (PESP)Given
G = (V , A)
T2N
` 2 RA
u 2 RA
w 2 RA≥0find⇡ 2 RVx 2 RAevent-activity network,
period time,
lower bounds,
upper bounds,
weights,

periodic timetable,
periodic tension

such that(1) ⇡j − ⇡i ⌘ xij mod T for all ij 2 A,(2) `  x  u,(3) w> x is minimum,or decide that no such (⇡, x) exists.

Incidence-based MIP formulation:
Minimizes.t.

w> x⇡j − ⇡i = xij − Tpij ,`ij  xij  uij ,⇡i 2 R,pij 2 Z,

p 2 ZAij 2 A,ij 2 A,i 2 V,
ij 2 A.

periodic offsets

Assumptions after preprocessing:I G is weakly (2-)connected
I G has no arc a 2 A with `a = ua
I 0  ` < T and 0  u − ` < T(Serafini and Ukovich, 1989)
Niels Lindner: On the geometry of periodic timetables in public transport

6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop

507

9 / 26

Periodic Event Scheduling Problem (PESP)Given
G = (V , A)
T2N
` 2 RA
u 2 RA
w 2 RA≥0find⇡ 2 RVx 2 RAevent-activity network,
period time,
lower bounds,
upper bounds,
weights,

Incidence-based MIP formulation:
Minimizes.t.

`ij  xij  uij ,⇡i 2 R,pij 2 Z,

periodic timetable,
periodic tension

such that(1) ⇡j − ⇡i ⌘ xij mod T for all ij 2 A,(2) `  x  u,(3) w> x is minimum,or decide that no such (⇡, x) exists.

w> x⇡j − ⇡i = xij − Tpij ,p 2 ZAij 2 A,ij 2 A,i 2 V,
ij 2 A.

periodic offsets

Symmetry breaking:I could impose 0  ⇡i < T andpij 2 {0, 1, 2}
Redundancy among periodic offsets p:
I could set pij = 0 along spanning tree

(Serafini and Ukovich, 1989)
Niels Lindner: On the geometry of periodic timetables in public transport

6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR
Workshop(Nachtigall, 1994, Liebchen,2006)

9 / 26

Hardness of PESP
TheoryI NP-hard for fixed T ≥ 3
I NP-hard for G with fixed treewidth ≥ 2(in particular for planar G)
I NP-hard cutting plane separation(cycle, change-cycle, flip)
I LP relaxation has trivial solution⇡ ⇤ = 0, x ⇤ = `, p⇤ = `/T

Niels Lindner: On the geometry of periodic timetables in public transport(Odijk, 1994, Nachtigall, 1996)(L. and Reisch, 2022)(Borndörfer et al., 2020, L. and Liebchen, 2020)

6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop

508

10 / 26

Hardness of PESP
TheoryI NP-hard for fixed T ≥ 3(Odijk, 1994, Nachtigall, 1996)

I NP-hard for G with fixed treewidth ≥ 2(in particular for planar G)
I NP-hard cutting plane separation(cycle, change-cycle, flip)
I LP relaxation has trivial solution⇡ ⇤ = 0, x ⇤ = `, p⇤ = `/T(L. and Reisch, 2022)(Borndörfer et al., 2020, L. and Liebchen, 2020)

Practice
I rich literature on algorithms:
MIP techniques, CP, SAT (also MaxSAT and SAT+ML), modulo network simplex,
matching, merging, maximum cuts, graph partitioning, . . .
I several success stories (Berlin, Copenhagen, Netherlands, Switzerland, . . . )Niels Lindner: On the geometry of periodic timetables in public transport

6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop

10 / 26

Hardness of PESP
TheoryI NP-hard for fixed T ≥ 3
I NP-hard for G with fixed treewidth ≥ 2(in particular for planar G)
I NP-hard cutting plane separation(cycle, change-cycle, flip)
I LP relaxation has trivial solution⇡ ⇤ = 0, x ⇤ = `, p⇤ = `/T(Odijk, 1994, Nachtigall, 1996)(L. and Reisch, 2022)(Borndörfer et al., 2020, L. and Liebchen, 2020)

Practice
I rich literature on algorithms:
MIP techniques, CP, SAT (also MaxSAT and SAT+ML), modulo network simplex,
matching, merging, maximum cuts, graph partitioning, . . .
I several success stories (Berlin, Copenhagen, Netherlands, Switzerland, . . . )Summary:

primal:dual:

Niels Lindner: On the geometry of periodic timetables in public transport

6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop

509

10 / 26

PESPlib

http://num.math.uni-goettingen.de/ m.goerigk/pesplib/

I benchmarking library, est. 2012 by Goerigk
I 22 hard to extremely hard PESP instances
I smallest instance ! MIP with 2722 (general) integer variables
I no instance has been solved to optimality so far
Niels Lindner: On the geometry of periodic timetables in public transport

6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop

11 / 26

ConcurrentPESP
Concurrent Framework for Periodic Timetable Optimization
Exact Methods
Branch & Cut

Network Strategies
Preprocessing
Shrinking Heuristics
Graph Partitioning

Gurobi/CPLEX/SCIP interface
several MIP formulations
dynamic cutting planes

for each network

Solution Pool

Starting Heuristics
SAT Solver

Improving Heuristics
Modulo Network Simplex
Multi-Node Cuts
Maximum Cuts...trades off by far more than just concurrency...holds primal and dual records for all 22 PESPlib instances(Borndörfer, L., Roth, 2020)
Niels Lindner: On the geometry of periodic timetables in public transport

6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop

510

12 / 26

Periodic Timetabling Spaces

Question
Can we get more insight by studying the geometry of periodic timetables?

Niels Lindner: On the geometry of periodic timetables in public transport

6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop

13 / 26

Periodic Timetabling Spaces

Question
Can we get more insight by studying the geometry of periodic timetables?

Timetabling Spaces
mixed-integer set of feasible solutions{(⇡, x, p) 2 RV ⇥ RA ⇥ ZA | 8ij 2 A : ⇡j − ⇡i = xij − Tpij , `ij  xij  uij }Niels Lindner: On the geometry of periodic timetables in public transport

6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop

511

13 / 26

Periodic Timetabling Spaces
Question
Can we get more insight by studying the geometry of periodic timetables?

Timetabling Spaces
mixed-integer set of feasible solutions{(⇡, x, p) 2 RV ⇥ RA ⇥ ZA | 8ij 2 A : ⇡j − ⇡i = xij − Tpij , `ij  xij  uij }project

projectx-space X⇡-space ⇧Niels Lindner: On the geometry of periodic timetables in public transport

projectp-space P

6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop

13 / 26

Periodic Timetabling Spaces
Question
Can we get more insight by studying the geometry of periodic timetables?

Timetabling Spaces
mixed-integer set of feasible solutions{(⇡, x, p) 2 R ⇥ RA ⇥ ZA | 8ij 2 A : ⇡j − ⇡i = xij − Tpij , `ij  xij  uij }Vproject

projectx-space X

project⇡-space ⇧p-space P

this talk

Benders decomposition

convex hull
full MIP toolbox
Niels Lindner: On the geometry of periodic timetables in public transport

6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop

512

13 / 26

Gallery of Timetabling Spaces

]

[3,

13

12

[4,

]

v1
[2, 10]v0

v2

PESP instance with n = 3, m = 3, T = 10

Niels Lindner: On the geometry of periodic timetables in public transport

6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop

14 / 26

Gallery of Timetabling Spaces

]

[3,

13

12

[4,

]

v1
[2, 10]v0

v2

PESP instance with n = 3, m = 3, T = 10
(12, 10, 8)(6, 10, 4)(3, 10, 7)(12, 6, 4)(7, 10, 13)(12, 5, 13)(3, 7, 4)(12, 2, 10)(3, 6, 13)(8, 2, 4)(9, 2, 13)(3, 2, 9)conv(X)
Niels Lindner: On the geometry of periodic timetables in public transport

6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop

513

14 / 26

Gallery of Timetabling Spaces
⇡2
13

]

[3,

13

12

[4,

]

v1
[2, 10]v0

1 -1 00 -1 00 -1 1-1 -1 1

12

v2

11
10

PESP instance with n = 3, m = 3, T = 10

9

100

000

-1 0 1

8(12, 10, 8)

7
001
6

(6, 10, 4)(3, 10, 7)(12, 6, 4)

5(7, 10, 13)

4
3(12, 5, 13)(3, 7, 4)

101

002

-1 0 22
1
0(12, 2, 10)(3, 6, 13)(8, 2, 4)

111

011

012

-1 1 2

1
0(9, 2, 13)(3, 2, 9)

1

2

3

4

5

6

7

8

9

10

11

12

13

14

⇡1

⇧/R1 is periodically tiled by polyt(r)opes

conv(X)
Niels Lindner: On the geometry of periodic timetables in public transport

6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop

14 / 26

Decomposing the Space of Periodic Timetables
Decomposition
The space of feasible periodic timetables is⇧ := {⇡ 2 RV | 9p 2 ZA : 8ij 2 A : `ij  ⇡j − ⇡i + Tpij  uij }.

The space ⇧ decomposes into polyhedral regions:[⇧=
R(p), where R(p) := {⇡ 2 RV | 8ij 2 A : `ij − Tpij  ⇡j − ⇡i  uij − Tpij }.
p2ZA

Due to the preprocessing assumption 0  u − ` < T, the union is disjoint.

Niels Lindner: On the geometry of periodic timetables in public transport

6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop

514

15 / 26

Decomposing the Space of Periodic Timetables
Decomposition
The space of feasible periodic timetables is⇧ := {⇡ 2 RV | 9p 2 ZA : 8ij 2 A : `ij  ⇡j − ⇡i + Tpij  uij }.

The space ⇧ decomposes into polyhedral regions:[⇧=
R(p), where R(p) := {⇡ 2 RV | 8ij 2 A : `ij − Tpij  ⇡j − ⇡i  uij − Tpij }.
p2ZA

Due to the preprocessing assumption 0  u − ` < T, the union is disjoint.

Weighted Digraph Polyhedra
Add a reverse copy a of each arc a. This produces a new graph G = (V , A) with V = V .
If we set (p)a := ua − Tpa and (p)a := −`a + Tpa , thenR(p) = {⇡ 2 RV | ⇡j − ⇡i  (p)ij for all ij 2 A}.

This means that R(p) is the weighted digraph polyhedron (Joswig, Loho, 2016)
associated to (G, (p)). In combinatorial optimization terms, R(p) is the polyhedronof feasible potentials in G w.r.t. the arc costs (p).
Niels Lindner: On the geometry of periodic timetables in public transport

6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop

15 / 26

Decomposing the Space of Periodic Timetables
A First Symmetry
If G is weakly connected, then G is strongly connected and by (Joswig, Loho, 2016):
I The recession cone of R(p) is R1.
I The quotient R(p)/R1 is a polytope.
Choosing coordinates on R(p)/R1 amounts to the periodic timetabler’s wisdom thata timetable ⇡ can be fixed at one event v0 2 V to ⇡v0 := 0 without affecting feasiblity
or optimality.

Niels Lindner: On the geometry of periodic timetables in public transport

6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop

515

16 / 26

Decomposing the Space of Periodic Timetables
A First Symmetry
If G is weakly connected, then G is strongly connected and by (Joswig, Loho, 2016):
I The recession cone of R(p) is R1.
I The quotient R(p)/R1 is a polytope.

Choosing coordinates on R(p)/R1 amounts to the periodic timetabler’s wisdom thata timetable ⇡ can be fixed at one event v0 2 V to ⇡v0 := 0 without affecting feasiblity
or optimality.

Polytropes
A polytrope is the convex hull of finitely many points, both in the ordinary and thetropical sense:�()
tconv(x1 , . . . , xn ) :=

n
M
i=1

�
�
i � xi ��

1, . . . ,

n 2 R

=

⇢

�
�
min( i + xi ) ��i=1
n1, . . . ,

�

n 2 R

.

Polytropes are exactly the quotients of weighted digraph polyhedra of strongly
connected digraphs by R1 (Joswig, Kulas, 2010).
Niels Lindner: On the geometry of periodic timetables in public transport

6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop

16 / 26

Decomposing the Space of Periodic Timetables
A First Symmetry
If G is weakly connected, then G is strongly connected and by (Joswig, Loho, 2016):
I The recession cone of R(p) is R1.
I The quotient R(p)/R1 is a polytope.
Choosing coordinates on R(p)/R1 amounts to the periodic timetabler’s wisdom thata timetable ⇡ can be fixed at one event v0 2 V to ⇡v0 := 0 without affecting feasiblity
or optimality.

Polytropes
A polytrope is the convex hull of finitely many points, both in the ordinary and thetropical sense:�()
tconv(x1 , . . . , xn ) :=

n
M
i=1

�
�
i � xi ��

1, . . . ,

n 2 R

=

⇢

�
�
min( i + xi ) ��i=1
n1, . . . ,

n 2 R

�

.

Polytropes are exactly the quotients of weighted digraph polyhedra of strongly
connected digraphs by R1 (Joswig, Kulas, 2010).
Corollary: ⇧/R1 decomposes into the disjoint union of the polytropes R(p)/R1.
Niels Lindner: On the geometry of periodic timetables in public transport

6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop

516

16 / 26

The Periodic Timetabling Torus
Periodicity: If ⇡ 2 ⇧, then ⇡ + Tq 2 ⇧ for all
q 2 ZV .
Consider the space of timetables
inside the (|V | − 1)-dimensional torus
VV

T := (R /(TZ) )/R1.

⇡2
13

1 -1 00 -1 00 -1 1-1 -1 1

12
11
10
9

100

000

-1 0 1

8
7
001
6
5
4
3

101

002

-1 0 2

2
1
0
111

011

012

-1 1 2

1
0

Niels Lindner: On the geometry of periodic timetables in public transport

1

2

3

4

5

6

7

8

9

10

11

12

13

14

⇡1

6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop

17 / 26

The Periodic Timetabling Torus
Periodicity: If ⇡ 2 ⇧, then ⇡ + Tq 2 ⇧ for all
Consider the space of timetables
q 2 ZV .
inside the (|V | − 1)-dimensional torus
T := (RV /(TZ)V )/R1.

13

1 -1 00 -1 00 -1 1-1 -1 1

12
11
10
9

100

000

-1 0 18
7001

Redundancy of periodic offsets: Let Γ be thecycle matrix of an integral cycle basis B of G.
Then R(p) ⌘ R(p0 ) on T iff Γp = Γp0 . We canhence denote R(p) modulo T by R(z), wherez := Γp 2 ZB .

Niels Lindner: On the geometry of periodic timetables in public transport

⇡2

6
5
4
3

101

002

1
0
111

011

012

-1 1 2

1
0

1

2

3

4

5

6

7

6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop

517-1 0 2

2

8

9

10

11

12

13

14

⇡1

17 / 26

The Periodic Timetabling Torus
Periodicity: If ⇡ 2 ⇧, then ⇡ + Tq 2 ⇧ for all
q 2 ZV .
Consider the space of timetables
inside the (|V | − 1)-dimensional torus
VV

T := (R /(TZ) )/R1.

⇡2
13

1 -1 00 -1 00 -1 1-1 -1 1

12
11
10
9

100

000

-1 0 18
7001

Redundancy of periodic offsets: Let Γ be thecycle matrix of an integral cycle basis B of G.
Then R(p) ⌘ R(p0 ) on T iff Γp = Γp0 . We canhence denote R(p) modulo T by R(z), wherez := Γp 2 ZB .

6
5
4
3

101

002

-1 0 2

2
1
0
111

011

012

-1 1 2

1
0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

⇡1

Running example:�⌫Γx12 − 2 + 13
z=

= 2,
T
10
⇠
⇡
Γx
3 − 10 + 4
≥
= 0,
z=
T
10
at most R(0), R(1), R(2) are in T .

Niels Lindner: On the geometry of periodic timetables in public transport

6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop

17 / 26

The Periodic Timetabling Torus
Periodicity: If ⇡ 2 ⇧, then ⇡ + Tq 2 ⇧ for all
Consider the space of timetables
q 2 ZV .
inside the (|V | − 1)-dimensional torus
T := (RV /(TZ)V )/R1.

⇡2
13

1 -1 00 -1 00 -1 1-1 -1 1

12
11
10
9

100

000

-1 0 18
7001

Redundancy of periodic offsets: Let Γ be thecycle matrix of an integral cycle basis B of G.
Then R(p) ⌘ R(p0 ) on T iff Γp = Γp0 . We canhence denote R(p) modulo T by R(z), wherez := Γp 2 ZB .

6
5
4
3

101

002

-1 0 2

2
1
0
111

011

012

-1 1 2

1
0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

⇡1

Running example:�⌫12 − 2 + 13
Γx

= 2,
z=
T
10
⇠
⇡
Γx
3 − 10 + 4
z=
≥
= 0,
T
10
at most R(0), R(1), R(2) are in T .

Niels Lindner: On the geometry of periodic timetables in public transport

6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop

518

17 / 26

More on Timetabling Polytropes
Dimension
I R(p) = ; if and only if G contains a negative weight directed cycle w.r.t. (p).

I The dimension of R(p)/R1 is the number of connected components of theequality graph of (G, (p)) minus 1 (Joswig, Loho, 2016).

Niels Lindner: On the geometry of periodic timetables in public transport

6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop

18 / 26

More on Timetabling Polytropes
Dimension
I R(p) = ; if and only if G contains a negative weight directed cycle w.r.t. (p).

I The dimension of R(p)/R1 is the number of connected components of theequality graph of (G, (p)) minus 1 (Joswig, Loho, 2016).

Vertices
I Every vertex of R(p)/R1 corresponds to a unique spanning subgraph of G.
I For each i 2 V , the i-th tropical vertex of R(p)/R1 corresponds to a shortest
path tree of (G, (p)) rooted at i (Joswig, Kulas, 2010).

Niels Lindner: On the geometry of periodic timetables in public transport

6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop

519

18 / 26

More on Timetabling Polytropes
Dimension
I R(p) = ; if and only if G contains a negative weight directed cycle w.r.t. (p).

I The dimension of R(p)/R1 is the number of connected components of theequality graph of (G, (p)) minus 1 (Joswig, Loho, 2016).

Vertices
I Every vertex of R(p)/R1 corresponds to a unique spanning subgraph of G.
I For each i 2 V , the i-th tropical vertex of R(p)/R1 corresponds to a shortest
path tree of (G, (p)) rooted at i (Joswig, Kulas, 2010).

Relation to the Periodic Tension Polytope (aka conv(X))
I The map mp : ⇡ 7! (⇡j − ⇡i + Tpij )ij2A embeds R(p)/R1 into conv(X).
I conv(X) = conv{im(mp ) | p 2 ZA }.

I im(mp ) is the intersection of the affine space im(B> ) + Tp with the LP relaxationQ
polytope XLP = a2A [`a , ua ], where B denotes the incidence matrix of G.

Niels Lindner: On the geometry of periodic timetables in public transport

6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop

18 / 26

Tropical Neighborhood Search
Polytropes in the Limit InstanceLet R(p)/R1 be a polytrope . The offse t p also defines a polytrope R0 (p)/R1 a of the“limit” instance where u := ` + T. The union of the polytropes is then no longer
disjoint and covers all of RV /R1.
⇡2
13

1 -1 00 -1 00 -1 1-1 -1 1

12
11
10
9

100

000

-1 0 1

8
7
001
6
5
4
3

101

002

-1 0 2

2
1
0
111

011

012

-1 1 2

1
0

1

2

3

4

Niels Lindner: On the geometry of periodic timetables in public transport

5

6

7

8

9

10

11

12

13

14

⇡1

6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop

520

19 / 26

Tropical Neighborhood Search
Polytropes in the Limit InstanceLet R(p)/R1 be a polytrope . The offse t p also defines a polytrope R0 (p)/R1 a of the“limit” instance where u := ` + T. The union of the polytropes is then no longer
disjoint and covers all of RV /R1.

Observation
The R0 (p) induce a polyt(r)opal subdivision of RV /R1.

Niels Lindner: On the geometry of periodic timetables in public transport

6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop

19 / 26

Tropical Neighborhood Search
Polytropes in the Limit InstanceLet R(p)/R1 be a polytrope . The offse t p also defines a polytrope R0 (p)/R1 a of the“limit” instance where u := ` + T. The union of the polytropes is then no longer
disjoint and covers all of RV /R1.

Observation
The R0 (p) induce a polyt(r)opal subdivision of RV /R1.

Neighbors
We call R(p)/R1 and R(p0 )/R1 neighbors if R0 (p)/R1 and R0 (p0 )/R1 intersect in acommon facet.
If R(p)/R1 and R(p0 )/R1 are both neighbors, then p = p0 ± ea for some arc a 2 A.

Niels Lindner: On the geometry of periodic timetables in public transport

6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop

521

19 / 26

Tropical Neighborhood Search
Polytropes in the Limit InstanceLet R(p)/R1 be a polytrope . The offse t p also defines a polytrope R0 (p)/R1 a of the“limit” instance where u := ` + T. The union of the polytropes is then no longer
disjoint and covers all of RV /R1.

Observation
The R0 (p) induce a polyt(r)opal subdivision of RV /R1.

Neighbors
We call R(p)/R1 and R(p0 )/R1 neighbors if R0 (p)/R1 and R0 (p0 )/R1 intersect in acommon facet.
If R(p)/R1 and R(p0 )/R1 are both neighbors, then p = p0 ± ea for some arc a 2 A.

Tropical Neighborhood Search (Baseline)
Given a non-empty polytrope R(p)/R1, solve PESP on R(p)/R1 (this is a linear
program, and dual to uncapacitated min cost flow). While there is an improving
neighbor of R(p)/R1: Go to the best neighboring polytrope, and repeat.
Niels Lindner: On the geometry of periodic timetables in public transport

6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop

19 / 26

Tropical Neighborhood Search

modulo network simplex search spacecolored by objective valuesquares are local non-global optima
Niels Lindner: On the geometry of periodic timetables in public transport

tropical neighborhood search spacecolored by objective value
6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop

522

20 / 26

Algorithm Tuning
I work on the torus T :use R(z) instead of R(p)/R1

Niels Lindner: On the geometry of periodic timetables in public transport

6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop

21 / 26

Algorithm Tuning
I work on the torus T :use R(z) instead of R(p)/R1
I LP formulation:
empirically: impact of up to a factor 14, the behavior seems to depend only on
the instance and not on z

Niels Lindner: On the geometry of periodic timetables in public transport

6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop

523

21 / 26

Algorithm Tuning
I work on the torus T :use R(z) instead of R(p)/R1
I LP formulation:
empirically: impact of up to a factor 14, the behavior seems to depend only on
the instance and not on z
I selection of neighbors:
do not explore all possible neighbors, only those the computed optimal vertexof R(z) is neighboring(best case: 2|A| vs. |V | − 1 neighbors, but we trade speed for quality)

Niels Lindner: On the geometry of periodic timetables in public transport

6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop

21 / 26

Algorithm Tuning
I work on the torus T :use R(z) instead of R(p)/R1
I LP formulation:
empirically: impact of up to a factor 14, the behavior seems to depend only on
the instance and not on z
I selection of neighbors:
do not explore all possible neighbors, only those the computed optimal vertexof R(z) is neighboring(best case: 2|A| vs. |V | − 1 neighbors, but we trade speed for quality)
I sorting of neighbors:changing the order does affect the outcome, but unpredictably(we tried several strategies including “pseudocost branching”)

Niels Lindner: On the geometry of periodic timetables in public transport

6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop

524

21 / 26

Algorithm Tuning
I work on the torus T :use R(z) instead of R(p)/R1
I LP formulation:
empirically: impact of up to a factor 14, the behavior seems to depend only on
the instance and not on z
I selection of neighbors:
do not explore all possible neighbors, only those the computed optimal vertexof R(z) is neighboring(best case: 2|A| vs. |V | − 1 neighbors, but we trade speed for quality)
I sorting of neighbors:changing the order does affect the outcome, but unpredictably(we tried several strategies including “pseudocost branching”)
I stopping criterion:
quality-first rule measured by relative improvement of the objective value(again trading speed for quality)

Niels Lindner: On the geometry of periodic timetables in public transport

6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop

21 / 26

Algorithm Tuning
I work on the torus T :use R(z) instead of R(p)/R1
I LP formulation:
empirically: impact of up to a factor 14, the behavior seems to depend only on
the instance and not on z
I selection of neighbors:
do not explore all possible neighbors, only those the computed optimal vertexof R(z) is neighboring(best case: 2|A| vs. |V | − 1 neighbors, but we trade speed for quality)
I sorting of neighbors:changing the order does affect the outcome, but unpredictably(we tried several strategies including “pseudocost branching”)
I stopping criterion:
quality-first rule measured by relative improvement of the objective value(again trading speed for quality)
I prevent cycling:hashing visited R(z) showed only negligible effects
Niels Lindner: On the geometry of periodic timetables in public transport

6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop

525

21 / 26

Performance of Tropical Neighborhood SearchSet-up
I 8 PESPlib instances
I 32 parameter configurations per instanceI 3 concurrency configurations for ConcurrentPESP
I 1 hour wall time, Intel i7-9700K CPU, 64 GB RAM

Niels Lindner: On the geometry of periodic timetables in public transport

6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop

22 / 26

Performance of Tropical Neighborhood SearchSet-up
I 8 PESPlib instances
I 32 parameter configurations per instanceI 3 concurrency configurations for ConcurrentPESP
I 1 hour wall time, Intel i7-9700K CPU, 64 GB RAM

General Results

I Tropical Neighborhood Search can escape local optima
I slow in the beginning, but becomes important in the late gameNiels Lindner: On the geometry of periodic timetables in public transport

6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop

526

22 / 26

Performance of Tropical Neighborhood SearchSet-up
I 8 PESPlib instances
I 32 parameter configurations per instanceI 3 concurrency configurations for ConcurrentPESP
I 1 hour wall time, Intel i7-9700K CPU, 64 GB RAM

General Results

I Tropical Neighborhood Search can escape local optima
I slow in the beginning, but becomes important in the late game

New PESPlib Incumbents
Instance
BL3
R1L1v
R3L3
R4L4
R4L4v

New Value
6 675 098
42 591 141
40 483 617
36 703 391
61 968 380

Old Value
6 999 313
42 667 746
40 849 585
36 728 402
64 327 217

Niels Lindner: On the geometry of periodic timetables in public transport

Time (s)25 7329 110
3 547
11 122
3 625

found new incumbents for 5
out of 8 instances!

6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop

22 / 26

Tropical Neighborhood Search vs. Other Heuristics

contribution of algorithms in ConcurrentPESP to overall improvement:
tropical neighborhood search, modulo network simplex, maximum cut, reflow, MIP(8 parameter choices on R1L1)

Niels Lindner: On the geometry of periodic timetables in public transport

6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop

527

23 / 26

Selection of Neighbors: All vs. Vertex-Tight

selecting only polytropes neighboring at an optimal vertex is an advantage in the
beginning
Niels Lindner: On the geometry of periodic timetables in public transport

6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop

24 / 26

Final Slide
Conclusion
Tropical Neighborhood Search is a simple yet powerful geometry-inspired method
for that adds new value to the zoo of periodic timetabling heuristics.

Niels Lindner: On the geometry of periodic timetables in public transport

6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop

528

25 / 26

Final Slide
Conclusion
Tropical Neighborhood Search is a simple yet powerful geometry-inspired method
for that adds new value to the zoo of periodic timetabling heuristics.

Further Geometric Questions
I Can we devise more heuristics from the polytropal
decomposition of the timetable space?
I Can we extract dual information from the periodic
timetabling torus?
I Can we exploit the duality relations between the ⇧and P-spaces? ( cycle offset zonotopes)

Niels Lindner: On the geometry of periodic timetables in public transport

6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop

25 / 26

Final Slide
Conclusion
Tropical Neighborhood Search is a simple yet powerful geometry-inspired method
for that adds new value to the zoo of periodic timetabling heuristics.

Further Geometric Questions
I Can we devise more heuristics from the polytropal
decomposition of the timetable space?
I Can we extract dual information from the periodic
timetabling torus?
I Can we exploit the duality relations between the ⇧and P-spaces? ( cycle offset zonotopes)

References
I E. Bortoletto, N. Lindner, B. Masing. The Tropical and Zonotopal Geometry of Periodic Timetables.
arXiv:2204.13501
I E. Bortoletto, N. Lindner, B. Masing. Tropical Neighbourhood Search: A New Heuristic for Periodic
Timetabling. ATMOS 2022 Best Paper Award.
Niels Lindner: On the geometry of periodic timetables in public transport

6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop

529

25 / 26

On the geometry of periodic timetables
in public transport
Niels Lindner, Enrico Bortoletto, Berenike Masing
MobilityLab
Department Network Optimization
Zuse Institute Berlin
6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop
September 21, 2022

Niels Lindner: On the geometry of periodic timetables in public transport

6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop

530

26 / 26

The 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop on
Advances in Classical and Quantum Algorithms for
Optimization and Machine Learning
September 16th - 19th, 2022, Tokyo (The university of Tokyo), Japan,
and September 21st - 22nd, 2022, Fukuoka (Kyushu University), Japan

Improving Data Quality in the Presenceof Superhuman Complexity in Data
ErrorsInci YÜKSEL-ERGÜN
Applied Algorithmic Intelligence Methods, Zuse Institute
Berlin, Germany
yueksel-erguen@zib.de
In our studies to analyze gas network systems, we study building public research data
sets from incomplete data scattered around various data sources. These data sources may
not be consistent with each other or accurate. Thus, during these studies, we use ourdomain-specific mathematical modeling know-how to eliminate the data errors byfilling missing data, or fixing inconsistencies. However, when working with theresulting highly-connected data, we encountered several cases where our analysis
detected data errors that were too complex for humans to understand. Examples are
irreducible infeasible subsystems (IIS) of large mixed-integer programs (MIP) or
bottlenecks in the pressure-coupled pipeline network that is non-linear. While detecting
these errors is a significant achievement, removing such errors is extremely difficult.
Hence, quantifying the data quality is also a key enabler in this study to tell whether thedata is of sufficient quality for the aimed analysis. We present our studies on data quality
improvement in the presence of superhuman complexity in data errors, and explain thechallenges. We report our results on the German high-pressure gas transport network
data set.

References[1] ENTSOG. Transmission Capacity Map. Retrieved fromhttps://www.entsog.eu/maps#transmission-capacity-map-2021. Accessed on 31.10.2022[2] Inci Yueksel-Erguen, Dieter Most, Lothar Wyrwoll, Carlo Schmitt, Janina Zittel. Modeling thetransition of the multimodal pan-European energy system including an integrated analysis ofelectricity and gas transport. Technical Report 22-17. Zuse Institute Berlin, Takustr. 7, 14195
Berlin, 2022. Available online < https://opus4.kobv.de/opus4zib/frontdoor/index/index/docId/8777>[3] Friedrich Kunz, Jens Weibezahn, Philip Hauser, Sina Heidari, Wolf-Peter Schill, Björn Felten,
Mario Kendziorski, Matthias Zech, Jan Zepter, Christian von Hirschhausen, Dominik Möst,
Christoph Weber. Reference Data Set: Electricity, Heat, and Gas Sector Data for Modeling theGerman System (Version 1.0.0), 2017. https://doi.org/10.5281/zenodo.1044463.
[4] Nicola Askham, Denise Cook, Martin Doyle, Helen Fereday, Mike Gibson, Ulrich Landbeck,
Rob Lee, Chris Maynard, Gary Palmer, Julian Schwarzenbach. The six primary dimensions fordata quality assessment. Technical Report, 2013, DAMA United Kingdom.
[5] Inci Yueksel-Erguen, Janina Zittel, Ying Wang, Felix Hennings, Thorsten Koch. Lessons learned
from gas network data preprocessing. Technical Report 20-13. Zuse Institute Berlin, Takustr. 7,14195 Berlin, 2020. Available online https://opus4.kobv.de/opus4zib/frontdoor/index/index/docId/7826[6] Thorsten Koch, Tobias Achterberg, Erling Andersen, Oliver Bastert, Timo Berthold, Robert E.
Bixby, Emilie Danna, Gerald Gamrath, Ambros M. Gleixner, Stefan Heinz, Andrea Lodi, Hans

531

Mittelmann, Ted Ralphs, Domenico Salvagnin, Daniel E. Steffy, Kati Wolter MIPLIB 2010.
Math. Prog. Comp. 3, 103 (2011). https://doi.org/10.1007/s12532-011-0025-9

532

Improving data quality in the presence of superhuman complexity
in data errors
Inci Yüksel-Ergün, Thorsten Koch, Felix Hennings, Janina Zittel
21.09.2022
The 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop
on Advances in Classical and Quantum Algorithms for
Optimization and Machine Learning
Chair of
Software and Algorithms for
Discrete Optimization

Department:
Applied AlgorithmicIntelligence Methods

A2IM Dept.| ZIB | 21.09.2022

1

Introduction• Highly-connected data in industrial applications- provided by industrial partners- consolidated from public data sources- generated using mathematical models• Data errors too complex for humans to understand detected by analysis tools, i.e.,- irreducible infeasible subsystems (IIS) of large mixed-integer programs- bottlenecks in the pressure-coupled pipeline network that is non-linear• Detection and correction of such errors are extremely difficult

2

I. Yüksel Ergün| ZIB | 21.09.2022

533

Supply Infrastructures• Transport power, gas, water, data, goods from suppliers to customers• Consist of complex networks• Digitization of planning and operation of such networks is essential for vital problems, i.e.,- Security of supply of energy, supply chain management, energy transition, decarbonization, etc.
High quality data à Reliable analysis results
High quality data is incredibly costly to obtain both in commercial and public applications, since
supply infrastructures• have complex network structures• were mostly built before digitization age• consist of layered and connected structures that may be operated by different parties• include complex facilities with intricate structures that can be handled by detailed mathematicalmodelsI. Yüksel Ergün| ZIB | 21.09.2022

3

Example: The European Gas Transport Network
European gas transport network in numbers:• 42 member, 10 associated partner, 2 observer
TSOs• > 200 interconnection points, > 170 storages• ≈ 200,000 km transmission pipelines (EU+UK)[1] ENTSOG. Transmission Capacity Map. Retrieved
from https://www.entsog.eu/maps#transmissioncapacity-map-2021. Accessed on 31.10.2022
I. Yüksel Ergün| ZIB | 21.09.2022

4534

Example: The European and Germany Gas Transport
Network
German gas transport network in numbers:• 16TSOs• 44interconnection points, 60storages• ≈ 31K km transmission pipelines (in total >500K
km)[1] ENTSOG. Transmission Capacity Map. Retrieved
from https://www.entsog.eu/maps#transmissioncapacity-map-2021. Accessed on 31.10.2022
I. Yüksel Ergün| ZIB | 21.09.2022

5

Example: Gas Transport Network Data
European Gas Transport
Network: Interconnections
•
•
•
•
•
[1] ENTSOG. Transmission Capacity Map. Retrieved fromhttps://www.entsog.eu/maps#transmission-capacity-map-2021.
Accessed on 31.10.2022

Germany Gas Transport Network:
Topology & Complex facilities

Gas Data Files: Technical properties

Network topology data: .net file (>34K lines; >28K data attributes)≈70 entry & ≈900 exit points≈1650 inner nodes≈1770 pipes≈95 control valves
58 compressor stations• 129 compressors & drivers•

200 valves
Compressor station data: .cs file (>54K lines)
Compressor station characteristic diagrams

Alternative compressor machine configurations[1] ENTSOG. Transmission Capacity Map. Retrieved fromhttps://www.entsog.eu/maps#transmission-capacity-map-2021.
Accessed on 31.10.2022[2] Yueksel-Erguen et al. Modeling the transition of the multimodal
pan-European energy system including an integrated analysis ofelectricity and gas transport. Technical Report 22-17. Zuse Institute
Berlin, Takustr. 7, 14195 Berlin, 2022.
[3] Kunz et al. Reference Data Set: Electricity, Heat, and Gas Sector
Data for Modeling the German System (Version 1.0.0), 2017.
https://doi.org/10.5281/zenodo.1044463.

I. Yüksel Ergün| ZIB | 21.09.2022

M1

M2

M1

M2
6

535

Gas Transport Network – Data Error Example 1

Missing node[2] Yueksel-Erguen et al. Modeling the transition of the multimodal pan-European
energy system including an integrated analysis of electricity and gas transport.
Technical Report 22-17. Zuse Institute Berlin, Takustr. 7, 14195 Berlin, 2022.
[3] Kunz et al. Reference Data Set: Electricity, Heat, and Gas Sector Data for Modelingthe German System (Version 1.0.0), 2017. https://doi.org/10.5281/zenodo.1044463.

.net file should be corrected by:• adding the missing node
• updating end node of existing pipes• adding two pipes between the end nodes of the existing pipes and the added node
I. Yüksel Ergün| ZIB | 21.09.2022

7

Gas Transport Network – Data Error Example 2
Coupled pipes[2] Yueksel-Erguen et al. Modeling the transition of the multimodal pan-European
energy system including an integrated analysis of electricity and gas transport.
Technical Report 22-17. Zuse Institute Berlin, Takustr. 7, 14195 Berlin, 2022.
[3] Kunz et al. Reference Data Set: Electricity, Heat, and Gas Sector Data for Modelingthe German System (Version 1.0.0), 2017. https://doi.org/10.5281/zenodo.1044463.

.net file should be corrected by:• deleting the node
• deleting two of the existing pipes• updating end nodes of the two pipes
I. Yüksel Ergün| ZIB | 21.09.2022

8536

Gas Transport Network – Data Error Example 3

Missing pipe[2] Yueksel-Erguen et al. Modeling the transition of the multimodal pan-European
energy system including an integrated analysis of electricity and gas transport.
Technical Report 22-17. Zuse Institute Berlin, Takustr. 7, 14195 Berlin, 2022.
[3] Kunz et al. Reference Data Set: Electricity, Heat, and Gas Sector Data for Modelingthe German System (Version 1.0.0), 2017. https://doi.org/10.5281/zenodo.1044463.

.net file should be corrected:• adding missing pipeor,
• deleting the disconnected partI. Yüksel Ergün| ZIB | 21.09.2022

9

Gas Transport Network – Data Error Example 4
Compressor Station Configuration Definition

All probable configurations of compressors in a compressor machine with a twocompressor machines:
Conf. 1:

[2] Yueksel-Erguen et al. Modeling the transition of the multimodal pan-European
energy system including an integrated analysis of electricity and gas transport.
Technical Report 22-17. Zuse Institute Berlin, Takustr. 7, 14195 Berlin, 2022.

Conf. 3:
M1

M2

M1

M2

Conf. 2:

M1

M2

M1

M2

Conf. 4:[3] Kunz et al. Reference Data Set: Electricity, Heat, and Gas Sector Data for Modelingthe German System (Version 1.0.0), 2017. https://doi.org/10.5281/zenodo.1044463.

.cs file should be corrected by:• updating the probable configuration(s)I. Yüksel Ergün| ZIB | 21.09.2022

10

537

Gas Transport Network – Data Error Example 5
Compressor Station Gas Flow Direction

Probable gas directions through a bi-directional compressor in two-pipeline connection case:• It is operationally possible for the gas flow from pipe 1 to pipe 2 and vice versa.
• The direction is controlled by valves.
[2] Yueksel-Erguen et al. Modeling the transition of the multimodal
pan-European energy system including an integrated analysis ofelectricity and gas transport. Technical Report 22-17. Zuse Institute
Berlin, Takustr. 7, 14195 Berlin, 2022.

Gas direction: pipeline 1 to pipeline 2
P1

P2

CS →

Gas direction: pipeline 2 to pipeline 2
P1

CS →

P2

[3] Kunz et al. Reference Data Set: Electricity, Heat, and Gas Sector
Data for Modeling the German System (Version 1.0.0), 2017.
https://doi.org/10.5281/zenodo.1044463.

.net file should be corrected by:• updating valve(s)• updating end nodes of the pipe(s)I. Yüksel Ergün| ZIB | 21.09.2022

11

Modeling Infrastructure Networks in Industrial
Applications
Infrastructure network data consists of• Tabular data: technical bounds, physical properties, etc.
• Spatial data: network topology• Complex components: often induce non-linear relations of data elements
It is challenging to detect and correct data errors:• Highly connected• Requires subject matter expertise to explain errors• Optimization is actively exploiting errors

I. Yüksel Ergün| ZIB | 21.09.2022

12

538

Dimensions of Data Quality
Completeness is existence of all required
data in the data set, comprehensivenessAccuracy is the ability to reflect reality, i.e., to beable to correctly describe real world objects

Accuracy

Validity is conformance topredetermined type, range andformatValidityCompleteness

Data
Quality

Consistency means conflict-free data, i.e.,

Consistency attributes of data entities addressing thesame physical attribute should be consistent

Timeliness

Uniqueness
Uniqueness means representing eachreal world object in the data set
exactly once

Timeliness shows whether data is up-to-date
data depending on the analysis context[4] Askham et al. The six primary dimensions for data quality assessment. Technical Report, 2013, DAMA UK.

I. Yüksel Ergün| ZIB | 21.09.2022

13

Data Errors: Non-conformances to data quality
Completeness is existence of all required
data in the data set, comprehensivenessAccuracy is the ability to reflect reality, i.e., to beable to correctly describe real world objects•••

Insufficient detailà cumulative capacities
Wrong modeling assumptions
à flow capacity computation
Data preprocessing/integration issues
à erroneous node coupling

Validity is conformance to predetermined
type, range and format•

Typos in names/non-conformance
to bounds (mostly data entry
related issues)••

Accuracy

ValidityCompleteness

Data
Quality

Consistency means conflict-free data, i.e.,

Consistency attributes of data entities addressing thesame physical attribute should be consistent•

Uniqueness

Uniqueness means representing eachreal world object in the data set
exactly once•

Missing attributes à missing pipes, valves, etc.
Missing entities à missing pipe maximum
pressure values, node height values, etc.

Repeated use of object names

Timeliness

Conflicts in the data setsà node pressure vs. pipe pressure bounds(may be induced by data set integration or data preprocessing
assumptions)

Timeliness shows whether data is up-to-date
data depending on the analysis context•

Untimely data à wrongly forecasted infrastructure related
data for future extensions due to limited information[4] Askham et al. The six primary dimensions for data quality assessment. Technical Report, 2013, DAMA UK.

I. Yüksel Ergün| ZIB | 21.09.2022

14

539

Data Error Detection and Correction
UniquenessValidity

Easier to detect and correct non-conformance using
automated systems like schema validation, i.e., GasLib
format (www.gaslib.zib.de)

Accuracy
CompletenessConsistency
Timeliness

Manual detection or correction is often not possible
Correction of non-conformance often requires subject matter
expertise and human interaction

I. Yüksel Ergün| ZIB | 21.09.2022

15

Eliminating Data Errors
Data Improvement Method

Related Data Quality Dimension

Error

Schema validation

UniquenessValidity

Format mismatch
Nonconformance to bounds

Data augmentation

CompletenessAccuracy

Insufficient detail
Missing entries/attributes

Data generation

CompletenessAccuracy

Insufficient detail
Missing entries/attributes

Consistency
checkheuristics

Consistency
Accuracy

Conflicts in the data set
Wrong modeling assumptions
Data preprocessing errors

Extensive
scenario
analysis

Accuracy
Consistency
Timeliness

Conflicts in the data set
Wrong modeling assumptions
Data preprocessing errors
Untimely data

Error diagnosis
and correction

Examples for consistency check heuristics: [5] Inci Yueksel-Erguen, J. Zittel, Y. Wang, F. Hennings, T. Koch. Lessons learned from gas network data preprocessing. Technical
Report 20-13. Zuse Institute Berlin, Takustr. 7, 14195 Berlin: ZIB, 2020.
I. Yüksel Ergün| ZIB | 21.09.2022

16

540

Extensive Scenario Analysis

Generate practicallyvalid scenarios using
historical flow data

Generate
Flow
Scenario

Utilize mathematicalmodeling methods
to detect infeasibility

Analyze
Infeasibility

Improve Network
Topology Data

Can we relate
infeasibility to data(errors)?

I. Yüksel Ergün| ZIB | 21.09.2022

17

An Example Analysis Set-up – Gas Network Data
Related Data Sets
ENTSOG IP data set
GIE Storage data set

Related Data Sets
DE network topology data set
Node pressure data from TSOs

M1: High-level Pan-European
Gas Network Model
Cumulative gas in-/out- flowof Germany based on IPs

M2: German Gas Transport
Network Model (Linear)Gas in-/out- flow of Germany
based on physical entry/exit
nodes

Resulting data
DE gas network
CS data

M3: German Gas Transport
Network Model (Nonlinear)

Details for the models: [2] I. Yueksel-Erguen, D. Most, L. Wyrwoll, C. Schmitt, J. Zittel. Modeling the transition of the multimodal
pan-European energy system including an integrated analysis of electricity and gas transport. Technical Report 22-17. Zuse
Institute Berlin, Takustr. 7, 14195 Berlin, 2022.

I. Yüksel Ergün| ZIB | 21.09.2022

541

Historicalflow data
Source:ENTSOG Security ofSupply Report Scenarios• Estimated pipeline
capacities• ENTSOG IP- DE network
associations• Parameters related togas properties

18

Mathematical Modeling Methods for Infeasibility Analysis
Slack Formulations•

Different aspects of the formulations can be relaxed with slack variables••

The objective is to minimize the deviation from the original model à zero objective function
The smallest distance from the feasibility(Minimum) Irreducible Infeasible Subsystems (IIS)•

Isolates the infeasibility by variables and constraints••

Not an explicit reason why a IIS is infeasible
Long computation times for large-scale MIPs•

A trivial IIS is not informative
ns1158817 fromMIPLIB 2010*

Constraints

Variables

NZ

Problem

68,455

1,804,022

2,842,044

IIS

2,003

6,002

12,002*[6] Koch et al. MIPLIB 2010. Math. Prog. Comp. 3, 103 (2011). https://doi.org/10.1007/s12532-011-0025-9

I. Yüksel Ergün| ZIB | 21.09.2022

19

Scenario Analysis – Infeasibility Analysis Method
S1. Initiate scenario analysis:
relax all non-linear constraints in the slack formulation
if there is at least one scenario in the scenario set S, select a scenario s from the scenario set and go to S2, else go to S5
S2. Solve the mathematical model
If feasible save the solution, delete the scenario s from the scenario set S and go to S1; else go to S3S3. Solve the slack formulation
if infeasible go to S4
if feasible with non-zero slack: correct the scenario and turn to S3
if feasible with a zero objective function value, tighten one set of the nonlinear constraints in the slack formulation if available and turnto S3, else save the solution and scenario correction, delete the scenario s from the scenario set S and go to S1;
S4. Find the minIIS using the LP minIIS model
if feasible, save the minIIS solution for the scenario, rescale the scenario by 0.95, and go to S4
if infeasible go to S1.
S5. Analyze the saved minIISs
to detect the frequency of existence of network components and constraint types in the minIIS and correct data set, and go to S1.

I. Yüksel Ergün| ZIB | 21.09.2022

20

542

Automated Infeasibility Analysis Method
S0. Initiate data set:
restore the scenario set: S:=S0; reset the data rating DR:=0
S1. Initiate scenario analysis:
relax all non-linear constraints in the slack formulation
if there is at least one scenario in the scenario set S, select a scenario s from the scenario set and go to S2, else go to S5
S2. Solve the mathematical model
If feasible save the solution and the scenario neighborhood scale association with the scenario, delete the scenario s from the scenario
set S and go to S1; else go to S3S3. Solve the slack formulation
if infeasible go to S4
if feasible with non-zero slack: correct the scenario and turn to S3
if feasible with a zero objective function value, tighten one set of the nonlinear constraints in the slack formulation if available and turnto S3, else update the scenario neighborhood scale and go to S2
S4. Find the minIIS using the LP minIIS model
if feasible, save the minIIS solution for the scenario, rescale the scenario by 0.95, and go to S4
if infeasible go to S1.
S5. Analyze the saved minIISs
to detect the frequency of existence of network components and constraint types in the minIIS, correct data set,
S6. Measure the data quality
If data quality is not sufficient, got to S1, else terminate.

I. Yüksel Ergün| ZIB | 21.09.2022

21

Data Quality Rating
Quality of data can be measured by its ability to represent the addressed entities

We need a data rating measure to facilitate the automated improvement by enabling us to• understand whether the improved data set is of sufficient quality level for the aimed analysis•

compare the performance of alternative improvements.. and also lead us/the search to the potential errors and error sources…

Example: Number of scenarios generated from the historical data that the data set finds a feasible routing••

Not informative enoughEspecially in the very beginning of the improvement process – we may get 0 for all scenario sets

I. Yüksel Ergün| ZIB | 21.09.2022

22

543

Data Quality Rating of Infrastructure Network Data - 1
An infrastructure network is an engineered system that is operational:•

Designed to meet certain quality of service requirements given operational requirements•

Robust against uncontrollable factors in the working environment

So, what does it mean from data perspective?
•

The network is designed to find feasible routings to scenarios within the operational concept••

The network is designed to eliminate systematic errors
The network has random errors that cannot be totally eliminated by the design process, so the network
should provide alternative routing solutions for changes in the scenarios
The amount of the change depends on (robustness of) the network•

If a scenario gives a feasible result with the data set,
then a set of scenarios in its neighborhood is expected to be feasible.
I. Yüksel Ergün| ZIB | 21.09.2022

23

Data Quality Rating of Infrastructure Network Data - 2
Operational scenarios of an infrastructure network have two main measurable characteristics:• The amount of commodity that can be routed
Small changes in demand and supply without changing the• The distribution of the commodity in the network
total flow and the main flow direction:

Commodity
distribution 1

Changecommodity
distribution

Source nodeMadir in fl
ec ow
tio
n

Demand and supply are equallydistributed among sink and source
nodes, resp. (node sizes are proportional
to the amount of flow):

low
in f n
Ma ectiodirCommodity
distribution 2

Sink node
I. Yüksel Ergün| ZIB | 21.09.2022

544

Future Outlook: Automated Data Improvement
Using Extensive Scenario Analysis 1
Traditional analysis
Raw tabulardata
Raw spatial
data (ie from
OSM)
Data
wranglingRunmathemati
cal model

I. Yüksel Ergün| ZIB | 21.09.2022

Raw tabular
data

25

Future Outlook: Automated Data Improvement
Using Extensive Scenario Analysis 2
Envisioned Analysis• …

Raw spatial
data (ie fromOSM)

For each scenario
DatawranglingRunmathemati
cal model

Initial network
data set

Feasible?

Yes

UpdateperformancemeasureNo
Measureand classify
infeasibility

Historicalcommodity
flow data(supply anddemand)Datamodifications

Generate
scenarios

Updatenetworkdata set

Updateimprovement
suggestions

Evaluate
Results

Data quality;
Data
modifications

Is
data
quality
OK?

No

Feasible
supply/demand scenarios

Yes

Updatednetworkdata set

Updated network
data set

I. Yüksel Ergün| ZIB | 21.09.2022

26

545

References[1] ENTSOG. Transmission Capacity Map. Retrieved from https://www.entsog.eu/maps#transmission-capacity-map-2021. Accessed on
31.10.2022[2] Inci Yueksel-Erguen, Dieter Most, Lothar Wyrwoll, Carlo Schmitt, Janina Zittel. Modeling the transition of the multimodal pan-European
energy system including an integrated analysis of electricity and gas transport. Technical Report 22-17. Zuse Institute Berlin, Takustr. 7,14195 Berlin, 2022. Available online < https://opus4.kobv.de/opus4-zib/frontdoor/index/index/docId/8777>[3] Friedrich Kunz, Jens Weibezahn, Philip Hauser, Sina Heidari, Wolf-Peter Schill, Björn Felten, Mario Kendziorski, Matthias Zech, Jan Zepter,
Christian von Hirschhausen, Dominik Möst, Christoph Weber. Reference Data Set: Electricity, Heat, and Gas Sector Data for Modeling theGerman System (Version 1.0.0), 2017. https://doi.org/10.5281/zenodo.1044463.
[4] Nicola Askham, Denise Cook, Martin Doyle, Helen Fereday, Mike Gibson, Ulrich Landbeck, Rob Lee, Chris Maynard, Gary Palmer, Julian
Schwarzenbach. The six primary dimensions for data quality assessment. Technical Report, 2013, DAMA United Kingdom.
[5] Inci Yueksel-Erguen, Janina Zittel, Ying Wang, Felix Hennings, Thorsten Koch. Lessons learned from gas network data preprocessing.
Technical Report 20-13. Zuse Institute Berlin, Takustr. 7, 14195 Berlin, 2020. Available online https://opus4.kobv.de/opus4zib/frontdoor/index/index/docId/7826[6] Thorsten Koch, Tobias Achterberg, Erling Andersen, Oliver Bastert, Timo Berthold, Robert E. Bixby, Emilie Danna, Gerald Gamrath,
Ambros M. Gleixner, Stefan Heinz, Andrea Lodi, Hans Mittelmann, Ted Ralphs, Domenico Salvagnin, Daniel E. Steffy, Kati Wolter MIPLIB
2010. Math. Prog. Comp. 3, 103 (2011). https://doi.org/10.1007/s12532-011-0025-9

A2IM Dept.| ZIB | 21.09.2022

27

Thank you for listening!
yueksel-erguen@zib.de

I. Yüksel Ergün| ZIB | 21.09.2022

28

546

The 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop on
Advances in Classical and Quantum Algorithms for
Optimization and Machine Learning
September 16th - 19th, 2022, Tokyo (The university of Tokyo), Japan,
and September 21st - 22nd, 2022, Fukuoka (Kyushu University), Japan

Optimal discrete pipe sizing for treeshaped CO2 networks
Jaap Pedersen, Thi Thai Le, Thorsten Koch, Janina Zittel
Zuse Institute Berlin, Berlin, Germany
Technische Universität Berlin, Berlin, Germany
pedersen@zib.de

Many energy-intensive industries, like the steel industry, plan to switch to renewableenergy sources. Other industries, such as the cement industry, have to rely on carbon
capture utilization and storage (CCUS) technologies to reduce their production
processes' inevitable carbon dioxide (CO2) emissions. However, a new transport
infrastructure needs to be established to connect the point of capture and the point ofstorage or utilization. Given a tree-shaped network transporting captured CO2 frommultiple sources to a single sink, we investigate how to select optimal pipeline diametersfrom a discrete set of diameters. The general problem of optimizing arc capacities inpotential-based fluid networks is already a challenging mixed-integer nonlinear
optimization problem. Adding the highly sensitive and nonlinear behavior of CO2
regarding temperature and pressure changes the problem becomes even harder. We
propose an iterative algorithm that splits the problem into two parts: a) the pipe-sizing
problem under a fixed supply scenario and temperature distribution and b) thethermophysical modeling, including mixing effects, the Joule-Thomson effect, and theheat exchange with the surrounding environment. We show the effectiveness of ourapproach by applying our algorithm to a real-world network planning problem for a CO2
network in Germany.

547

Optimal discrete pipe sizing for tree-shaped CO2 networks
Jaap Pedersen,
T. Le, J. Zittel and T. Koch
21st of September 2022
6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop on
Advances in Classical and Quantum Algorithms for
Optimization and Machine Learning
Acknowledgement: This work has been conducted in the Research Campus MODAL funded by the German Federal Ministry of Education and Research (BMBF) (fund numbers
05M14ZAM, 05M20ZBM).

Motivation - Net-zero by 2050

I Decarbonization of industry, e.g, by shifting
towards renewables
I 8 % of global GHG from cement industry
I CO2 emissions from chemical process itself
I Carbon Capture and Storage (CCS) or
Utilization (CCU)
I New infrastructure to transport CO2 frompoint of capture to storage/utilization

Global greenhouse gas emissions (GHG) in 2018 [IEA, 2020]
Jaap Pedersen, pedersen@zib.de

Optimal discrete pipe sizing for tree-shaped CO2 networks

548

1How to transport CO2 ? - Goal
I CO2 is transported in liquid or supercritical state
I Supercritical: State in which the liquid and gaseous phases cannot be distinguished
I Pipeline networks are most cost efficient
I Network planning involves finding the cost-optimal pipeline diameters

Goal
Determine the cost-optimal pipeline diametersfrom a discrete set of diameters
I in a tree-shaped CO2 network
I with multiple sources and a single sink
I for a given supply scenario
I transport CO2 in liquid or supercritical state=) no transition into gaseous phase
Own figure, data from [Bell et al., 2014]
Jaap Pedersen, pedersen@zib.de

Optimal discrete pipe sizing for tree-shaped CO2 networks

2

Background

I Problem of finding optimal pipeline diameters is acommon problem in real-world applications, e.g.,
in water, natural gas, and hydrogen networks[D’Ambrosio et al., 2015, Lenz and Becker, 2022, Robinius et al., 2019]

I Discrete decision variables w.r.t. diameter size ofpipelines
I Non-convex physics describing flow in pipelines
I Physical properties of CO2 are sensitive againstchanges in pressure and temperature=) Mixed Integer NonLinear Program; hard to solve
Own figure, data from [Bell et al., 2014]

Jaap Pedersen, pedersen@zib.de

Optimal discrete pipe sizing for tree-shaped CO2 networks

549

3

Solution approach

I Iterative algorithm
I Decouple finding optimal pipe
diameters and thermophysicalmodeling
I Exploit the benefits of tree
structure to reduce complexity
I In each iteration updateparameters w.r.t to pressure p andtemperature T
I Stop algorithm if diameters andpressure levels converge

Jaap Pedersen, pedersen@zib.de

Initialization
k=1
k +1

Determine optimal pipe diametersand pressure levels

Update parameters w.r.t to p and T

Compute temperature distribution

No

Diameters andpressure converged?

Yes

Optimal discrete pipe sizing for tree-shaped CO2 networks

END

4

Determine optimal pipe diameters - Notation
Initialization
k=1

Determine optimal pipe diametersand pressure levels

k +1

G = (V , A)

Update parameters w.r.t to p and T

Compute temperature distribution

No

Diameters andpressure converged?

Yes

END

V +, V −, V 0
Api , Apu
Da

Directed tree with set of vertices V
and arcs A; arcs orientated fromleafs to root
Set of entry, exit and inner nodes
Set of pipes and pumps
Set of pipe diameters for a 2 Api

bv 2 R
Inflow/Outflow of vertex v 2 V
pv 2 [p v , p v ] Pressure of vertex v 2 V
Flow over arc a 2 A
qa 2 R≥0
Diameter of pipe a 2 Apida 2 Daxa,d 2 {0, 1} Choice of da 2 Da for a 2 Api

Jaap Pedersen, pedersen@zib.de

Optimal discrete pipe sizing for tree-shaped CO2 networks

550

5Determine optimal pipe diameters - Notation
Incoming and outgoing arcs:
δ − (v ) = {a 2 A | a = (u, v )}
δ + (v ) = {a 2 A | a = (v , u)}Intermediate nodes:
V m := {v 2 V | |δ − (v )| = |δ + (v )| = 1 ^ δ − (v ) [ δ + (v ) 2 Api ^|Da=(u,v ) | > 1 ^ |Da=(v ,w ) | > 1}.
Note: Node v 2/ V m if junction node, tail or head node of pump, tail or head node of pipeline witha fixed diameter

Jaap Pedersen, pedersen@zib.de

Optimal discrete pipe sizing for tree-shaped CO2 networks

6

Determine optimal pipe diameters - Network elements
I Pipelines are main type of element to transport CO2
I In general, fluid flows from higher pressure to lower pressure
I Pressure change in a pipeline
pv − pu = φa (da , qa ) qa |qa | − (Hv0 − Hu0 )⇢g
|
{z
} |
{z
}
friction term

8a = (v , u) 2 Api

elevation di↵erence

I Friction loss coefficient φa (da , qa ) is highly nonlinear
I Pumps can be used to increase pressure of fluid

8a = (v , u) 2 Apu

pu ≥ pv

Hv0 : height of v ; ⇢: density of fluid, g : gravitational constantJaap Pedersen, pedersen@zib.de

Optimal discrete pipe sizing for tree-shaped CO2 networks

551

7

Determine optimal pipe diameters - Model
min

x,q,p

s.t.

X X

ca,d xa,da2Api d2DaX

a2 + (v )qa −pv − pu =

X

a2

X

d2Da

xa,d φa (d, qa ) qa |qa | − (Hv0 − Hu0 )⇢g

p u ≥ pv
p v  pv  p v
Xxa,d = 1d2DaX

8v 2 V ,q a = bv(v )x(u,v ),d d =d2D(u,v )8a = (v , u) 2 Apu ,8v 2 V ,8a 2 Api ,X

x(v ,w ),d dd2D(v ,w )8v 2 V m ,8a 2 Api , 8d 2 Da .

xa,d 2 {0, 1}
Jaap Pedersen, pedersen@zib.de

8a = (v , u) 2 Api ,Optimal discrete pipe sizing for tree-shaped CO2 networks

8

Initialization - Reduce complexity

Initialization
k=1

k +1

Determine optimal pipe diametersand pressure levels

Update parameters w.r.t to p and T

Compute temperature distribution

No

Diameters andpressure converged?

Yes

I Reduce complexity by precomputing unique flow values
and friction loss coefficients
I We have an in-tree graph rooted at a single exit node
and a given inflow scenario, i.e., all inflow are known
I Recursively determine arc flow values starting at leaf
nodes towards the exit node
I For each pipeline a 2 A and each possible diameterd 2 Da determine friction loss coefficient φa (d, qa )END

Assume constant physical parameters, e.g., ⇢ = const, the model to determine optimal pipe
diameters becomes a tractable mixed integer linear program (MILP)
Jaap Pedersen, pedersen@zib.de

Optimal discrete pipe sizing for tree-shaped CO2 networks

552

9

Thermophysical modeling

I Physical properties are nonlinear functions ofpressure and temperature, e.g.:
Density
Viscosity
Heat capacity⇢ = ⇢(p, T )⌫ = ⌫(p, T )cp = cp (p, T )Thermal conductivityf = f (p, T )Jaap Pedersen, pedersen@zib.de

Optimal discrete pipe sizing for tree-shaped CO2 networks

10

Thermophysical modelingE↵ects on the temperature in the network
I Temperature Tv at a junction node v due to mixingPout
a2 (v ) cp,a qa Ta
Tv = Pa2 (v ) cp,a qa

Initialization
k=1

k +1

Determine optimal pipe diametersand pressure levels

I Temperature change along a pipeline

Update parameters w.r.t to p and T

Compute temperature
distribution

No

Diameters andpressure converged?

1kL∆TlnT out = T in + µJT (p out − p in ) −|{z}
q a cp|{z}
Joule-Thomson e↵ectHeat exchange withsurrounding

Yes

END

I For given flow values and pressure distribution,
calculate temperature distribution similar tocomputing flow values

T out : Outlet temperature of pipe; T in : Inlet temperature of pipe; qa : Mass flow over pipe; cp : Heat capacity of fluid; µJT :Joule-Thomson coeﬃcient; k: Heat transmission rate; ∆Tln : Logarithmic temperature di↵erence with surrounding
Jaap Pedersen, pedersen@zib.de

Optimal discrete pipe sizing for tree-shaped CO2 networks

553

11

Algorithm
Initialization:
Flow and Friction Factor
k=1
Determine optimal pipe diametersand pressure levels

k +1

Update parameters w.r.t to p and T

Compute temperature distribution

Diameters andpressure converged?

No

Jaap Pedersen, pedersen@zib.de

Yes

END

Optimal discrete pipe sizing for tree-shaped CO2 networks

12

Case study - Network data
Connect four cement plants to harbour in North-Western Germany by tree-shaped network|V − |
1|V + |
4|V 0 |
1661|V m |
1652

Inflow in kg/s
Temperature in ◦C
Max Pressure in bar

D
16
50
95

P
8
60
105

Set
N

|Api |
1662
S
8
20
88

|Apu |
3

|Da |
12

B
8
30
88

Minimal pressure at exit: 80 bar
Network consists of: ⇠830 km pipelines
Length of each pipe segment: 500 m
Sections 1 and 17: Fixed to 300 mm
Surrounding temperature: 10 ◦C
Jaap Pedersen, pedersen@zib.de

Optimal discrete pipe sizing for tree-shaped CO2 networks

554

13

Case study - Results

Pressure

Jaap Pedersen, pedersen@zib.de

TemperaturePhase state

Optimal discrete pipe sizing for tree-shaped CO2 networks

14

Optimal discrete pipe sizing for tree-shaped CO2 networks

15

Case study - Results

Jaap Pedersen, pedersen@zib.de555

Conclusion

I We presented an iterative algorithm for finding optimal pipeline diameters in a tree-shaped
multi-source single-sink network for CO2 transport
I To account for the complex thermophysical behavior of CO2 , we split the problem intoa) Finding optimal pipe diameters by solving a tractable MILPb) modeling the thermophysical e↵ects in the network, including mixing, heat exchange with thesurrounding and the Joule-Thomson e↵ect

I In each iteration we update physical parameters w.r.t. pressure and temperature levels
I We showed Proof-of-Concept by applying our method to a real-world planning instance

Jaap Pedersen, pedersen@zib.de

Optimal discrete pipe sizing for tree-shaped CO2 networks

16

Outlook

I Verify coarse solution of optimization with results by a simulator
I Robustness of method by applying method to multiple instances and scenarios
I Extend method to handling CO2 -rich fluids =) change of phase envelope, i.e., more complex
pressure bounds
I Extend method to demand-based components, e.g., hydrogen or ammonia as energy carrier=) single-source multiple-sink network

Jaap Pedersen, pedersen@zib.de

Optimal discrete pipe sizing for tree-shaped CO2 networks

556

17

Thank You For Your Attention!
Any Questions?

Jaap Pedersen, pedersen@zib.de

Optimal discrete pipe sizing for tree-shaped CO2 networks

18

References

Bell, I. H., Wronski, J., Quoilin, S., and Lemort, V. (2014).
Pure and Pseudo-pure Fluid Thermophysical Property Evaluation and the Open-Source Thermophysical Property Library CoolProp.
Industrial & Engineering Chemistry Research, 53(6):2498–2508.
D’Ambrosio, C., Lodi, A., Wiese, S., and Bragalli, C. (2015).
Mathematical programming techniques in water network optimization.
European Journal of Operational Research, 243(3):774–788.
IEA (2020).
The role of CCUS in low-carbon power systems.
Technical report, IEA, Paris.
Lenz, R. and Becker, K. H. (2022).
Optimization of capacity expansion in potential-driven networks including multiple looping: A comparison of modelling approaches.
OR Spectrum, 44(1):179–224.
Robinius, M., Schewe, L., Schmidt, M., Stolten, D., Thürauf, J., and Welder, L. (2019).
Robust optimal discrete arc sizing for tree-shaped potential networks.
Computational Optimization and Applications, 73(3):791–819.

Jaap Pedersen, pedersen@zib.de

Optimal discrete pipe sizing for tree-shaped CO2 networks

557

19

Additional Slides

Jaap Pedersen, pedersen@zib.de

Optimal discrete pipe sizing for tree-shaped CO2 networks

20

Determine optimal pipe diameters - Pipeline
Pressure change in a pipeline
pv − pu = φa (da , qa ) qa |qa | − (Hv0 − Hu0 )⇢g
|
{z
} |
{z
}
friction term

8a = (v , u) 2 Api

elevation di↵erence

Friction loss coefficient φa using Weymouth equation
φa (da , qa ) =8La
λa (da , qa )⇡ 2 ⇢da5

Darcy friction factor λa using implicit Colebrook-White equation
✓
◆
2.51
1
"
p = −2 log10
p
with
+
3.7da
λa
Re λa

Re =

da q aAa ⌫Hv0 : height of v ; ⇢: density of fluid, g : gravitational constant; La : length of pipe; ": roughness of pipe; Re: Reynold’s
number; ⌫: dynamic viscosity of fluid; Aa : cross section of pipe
Jaap Pedersen, pedersen@zib.de

Optimal discrete pipe sizing for tree-shaped CO2 networks

558

21

Heat exchange with surrounding in buried pipe1
Determine heat transmission factor ↵Heat exchange

⌫
cp ⌫
=
f /cpfdq
Re = Re(⌫, d, q) =A⌫
Pr = Pr (⌫, f , cp ) =

Nu =

↵d
f

()

(⇣/8) Re Prp
Nu =1 + 12.7 ⇣/8(Pr 2/3⇣ = (1.8 log10 Re↵=∆Q = LQc = kL∆Tln∆Tln =k=Nufd✓ ◆2/3 #
d
1+
L
1)
"

T out

in
s
log TTout TT s2⇡2
+ 1p log ddo + 1s log 4s↵ddStart with initial guess T out,0 , using mean
temperature Tm = (T in + T out,i 1 )/2 andpressure pm = (p in + p out )/2

1.5) 2

1 VDI Wärmeatlas, berlin, Heidelberg: Springer, 2013.
Jaap Pedersen, pedersen@zib.de

T in

cp (pm , Tm ), ⌫(pm , Tm ), f (pm , Tm ), µJT (pm , Tm )1
T out,i = T in + µJT (p out p in )kL∆Tlni 1
qcp

Optimal discrete pipe sizing for tree-shaped CO2 networks

22

Compute flow values

I Let G = (V , A) be directed in-tree graph rooted at a single exit node
I Let V (u) := {v 2 V : (u, v ) 2 A _ (v , u) 2 A} be the set of nodes adjacent to node u
I Let L := {v 2 V : |V (v )| = 1 ^ v 2/ V − } be the set of leafs in the tree G

I For each v 2 L set the flow of the unique outgoing arc a 2 δ + (v ) to qa = bv and update
bu := bu + qa for each neighbor u 2 V (v )S
I Let V = V \L and A = A\ v 2L δ + (v ), update L
I Iterate until A is empty

Jaap Pedersen, pedersen@zib.de

Optimal discrete pipe sizing for tree-shaped CO2 networks

559

23

Compute temperature distribution
I Let G = (V , A) be a directed in-tree graph with given flow value qa for each a 2 A andpressure level pv for each v 2 V .
I Let L := {v 2 V : |V (v )| = 1 ^ v 2/ V − } be the set of leafs in the tree G
I Let V (u) := {v 2 V : (u, v ) 2 A _ (v , u) 2 A} be the set of nodes adjacent to node u
I For each v 2 V + , an inflow temperature Tv = T in is given.
I For each v 2 L, set the inlet temperature of the unique outgoing arc a 2 δ + (v ) to Tain = Tv .
I If a 2 Api , the outlet temperature Taout is determined by solving correlations for buried pipelines
I If a 2 Apu , set Taout = Tain .
S
I Then, for each node u 2 v 2L V (v ), compute its mixing temperature Tu
S
I Let V = V \L and A = A\ v 2L δ + (v )I Update L, and iterate until A is empty

Jaap Pedersen, pedersen@zib.de

Optimal discrete pipe sizing for tree-shaped CO2 networks

24

Optimal discrete pipe sizing for tree-shaped CO2 networks

25

Results

Jaap Pedersen, pedersen@zib.de

560

The 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop on
Advances in Classical and Quantum Algorithms for
Optimization and Machine Learning
September 16th - 19th, 2022, Tokyo (The university of Tokyo), Japan,
and September 21st - 22nd, 2022, Fukuoka (Kyushu University), Japan

Spotlights on success stories ofpublic-private partnership
Uwe GOTZESOpen Grid Europe GmbH, Essen, Germany
uwe.gotzes@oge.net
The Zuse Institute Berlin [1] and OGE, Germany's largest natural gas transmission
system operator [2], have been cooperating for more than a decade in a variety of
application-oriented research projects [3]. In my talk I will briefly go through the historyof the projects and the collaboration. A more recent project addressing telecommunication network design by the application of ZIB’s SCIP-Jack and SCIP [4][5] will
be presented in detail.

References[1] https://www.zib.de/[2] https://oge.net/en[3] https://www.zib.de/features/research-campus-modal[4] https://scipjack.zib.de/[5] https://www.scipopt.org/561

2022/11/15

Spotlights on success stories ofpublic-private partnership
Uwe GOTZES
6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop – 21 September 2022

Shortened and annotated version of the presentation slides
1

Started in 2010
Originated from E.ON Ruhrgas having
a history going back more than 80
years
Headquartered in Essen
1450 employees
Plans, constructs, operates andmonitors one of Europe’s largest
natural gas transmission systems
Total length of about 12,000 km,
around 30 compressor stations with
about 100 units
Hundreds of metering and pressure
regulating plants
450 energy suppliers obtain gas over1000 Exit points
Annual offtake:~ 300 TWh (DSO and industry)~ 880 TWh total offtake incl. other TSO

2

1
562

2022/11/15

We offer…
¡ the transmission of natural gas through our network¡ the technical and commercial services to go with it
Various forecasting
projects

Project Forne

Capacity
calculation

Processing
nominations andofftake behavior

Capacitymarketing

Project OAD

Technical andnon-technicalnetwork operation

Quantity
determination andbilling

Related research projects with ZIB/MODAL
and we provide…

Project Carp (Detailed presentation on the following slides)¡ commercial, technical and IT services for other companies
3

A cross section through a pipeline route

The specially protected right of way strip contains cableducts with fiber optic cables for highspeed internet.
A subsidiary of OGE offers the capacities of this data
network to its customers. Another subsidiary of OGEplans new routes to connect new customers according
to their enquiries. The project Carp delivers new
research driven software tools to support the planning.

4

2
563

2022/11/15

How does a customer enquiry look like?
Mathematically spoken, we are given an undirected weighted graph G.
The subset I consists of the green vertices that form our backbone
network in the cable ducts right next to the gas pipelines.
The k disjoint groups of n_k nodes are all hidden in the dark blue,
densely packed area in the lower left corner of the graph(next slide is a zoom into this area).

5

Backbone network

B

bo
ack

ne n

e tw

o rk

dark blue edges = potential new line constructions

6

3
564

2022/11/15

7

Approach to solve Task A•

Almost a Steiner Tree Problem…but not exactly•

It just so happens that Daniel Rehfeldt from ZIB has developed (one of) the best Steiner treesolvers in the world…so why not talk with him first?

Algorithm:
1.

Replace all edges by an antiparallel pair of directed edges

2.

Connect all vertices from I unidirectionally witch a ‘supernode’ i* at cost 0

3.

Solve as many Steiner Aborescence Problems (SAP) as there are groups and consider
vertices of other groups as already connected to the backbone network

After all SAP for all groups are solved, it might happen, that vertices of several groups areconnected in a tree structure, but not yet to the backbone network. To overcome this flaw:
1.

Set all edge costs of tree edges to 0

2.

Solve ordinary Steiner Problem with all group nodes as terminals

8

4
565

2022/11/15

Example graph with euclidean
distance as edge weights
Backbone network (green) andthree vertex groups (magenta,
turquois and yellow)

9

Solution of the first SAP in the modified
graph according to the algorithm.
Important: choose a group node andnot the supernode as root node (green
circle) for the SAP to make sure thatthe group is connected in a tree
structure in the original graph.

10

5
566

2022/11/15

Solution edges of all three SAP in theoriginal graph.
All groups are connected by a tree
structure, but turqouis and magentagroups are not yet connected to the
backbone network. To achieve this:Solve ordinary SPG with magenta,
turqouis, yellow groups and supernodeas terminals.

11

Hooray, nice feasible solution found!
However, the method is a heuristic
1.

How good is the heuristic?

2.

How can the method be improved?

3.

Example where the heuristic does
not yield the optimal solution?

12

6
567

2022/11/15

Real world example Task A)

13

No

td

is c

us

se

d

in

de

ta

il h

er

e

14

7
568

2022/11/15

Nightly E-Mail ping-pong-session
with Thorsten Koch to solve task D
as follows:

T0

T1

1.

Modify original graph as shown
here (now directed and newvertex T_2)

2.

Set up three shortest path
problems

3.

Bundle all three shortest path
problems in one huge Threeways round trip problem
containing three copies of themodified graph4.

Add additional constraints toforbid shortcuts via T_2

T2
15

16

8
569

2022/11/15

T0

T1
Solution of theThree ways round trip problem.
This solution has a flaw, because it took
a shortcut via T_2 and hence is
not a circle in the original graph.

T2
17

Modifications of the model in red. Effect: Shortcuts via
T_2 are now forbidden. ¥overline{E} denotes theedges of the original graph. ¥delta is Kronecker‘sdelta.

189570

2022/11/15

T0

T1

T2
19

Shortest node disjoint circle
Real world example Task D)

Thank you!
20

10
571

The 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop on
Advances in Classical and Quantum Algorithms for
Optimization and Machine Learning
September 16th - 19th, 2022, Tokyo (The university of Tokyo), Japan,
and September 21st - 22nd, 2022, Fukuoka (Kyushu University), Japan

Deep Switching State Space Model(DS3M) for Nonlinear Time Series
Forecasting with Regime Switching
Xiuqin Xu*1 and Ying Chen*12*1 Integrative Sciences and Engineering Programme, NUS
Graduate School Institute of Data Science, National University ofSingapore
xiuqin.xu@u.nus.edu*2 Department of Mathematics, National University of Singapore
Asian Institute of Digital Finance, National University ofSingapore,
Risk Management Institute, National University of Singapore,
Singaporematcheny@nus.edu.sg

We propose a deep switching state space model (DS3M) for efficient inference andforecasting of nonlinear time series with irregularly switching among various regimes.
The switching among regimes is captured by both discrete and continuous latentvariables with recurrent neural networks. The model is estimated with variational
inference using a reparameterization trick. We test the approach on a variety ofsimulated and real datasets. In all cases, DS3M achieves competitive performancecompared to several state-of-the-art methods (e.g. GRU, SRNN, DSARF, SNLDS), withsuperior forecasting accuracy, convincing interpretability of the discrete latent variables,and powerful representation of the continuous latent variables for different kinds of timeseries. Specifically, the MAPE values increase by 0.09\% to 15.71\% against the secondbest performing alternative models.

573

Deep Switching State Space Model (DS3 M) for
Nonlinear Time Series Forecasting with Regime
Switching
Xiuqin XU
Ying CHEN

https://arxiv.org/abs/2106.02329
Institute of Data Science & Integrative Sciences and Engineering Programme
Department of Mathematics & Risk Management Institute & Asian Institute of
Digital Finance
National University of Singapore

Motivation

Nonlinear time series with regime switching

1-1

Figure: The U.S. monthly unemployment rate 2002 – 2021.

 The U.S. unemployment rate depends on (discrete)unobservable economic status of booming or recession. It isalso influenced by some latent (continuous) variables, e.g.
elasticity of regional wage level and others, that in turn varywith the discrete status.
 Deciphering these discrete and continuous latent variables cangain insights.
DS3 M

574

Motivation

1-2

Technical challenges

� In many studies, researchers need to model and infer similar
type of time series that disobey traditional assumptions of e.g.
linearity, normality and stationarity in the statistical modeling,
but exhibit nonlinearity with stochastic regime switching
behaviours.
� Examples: health care (sleep apnea), economics(unemployment rate), traffic and transportation (metro
passengers volume), meteorology (sea surface temperature),
energy (electricity demand), to name just a few.
� Two challenges:

I a severe modeling misspecification
I lack of interpretation on the stochastic regimes

DS3 M

Motivation

Switching state space models (SSSM)

1-3

� The evolution of time series is assumed to be driven by hidden
factors switching among discrete regimes, see Bae et al.,2014; Fox et al., 2009; François et al., 2014; Ghahramani
et al., 2000.

� The SSSM is a generalization of the Hidden Markov Models(HMMs) and State Space Models (SSMs).

Figure: State Space Model
DS3 M

575

Motivation

1-4

Linear Gaussian State Space Model (LGSSM)

The dynamics in each regime are usually represented by simple
models that could be efficiently estimated even with a small sample
size (Durbin et al., 2012), and the switching among regimes iscontrolled by hidden transition probabilities of a Markov chain.
 Transition function for the latent continuous variable zt :(zt |zt≠1 , xt ) ≥ N(µt , Σz )(1)

where µt = Wz zt≠1 + Wx xt + bz and Σz is covariance.
 Emission function for the observation yt :yt |zt ≥ N(mt , Σy )(2)where mt = Wy zt + by and Σy is covariance.
DS3 M

Motivation

SSSMs and misspecification

1-5

 Hidden Markov Model (HMM): Baum et al. (1966), Linear
Gaussian State Space Model (LGSSM): Durbin et al. (2012),
 Non-linear and non-Gaussian SSM: Doucet et al. (2009),
Julier et al. (1997), and Smith et al. (1962)
 Non-stationary SSM: Ackerson et al. (1970), Chang et al.
(1978), Fox et al. (2009), Ghahramani et al. (2000), Hamilton(1990), and Murphy (1998)- By extending the local linear models with different regimes, theresulting model approximates a globally nonlinear behaviour and is
expected to retain interpretation.
- The existing nonlinear models rely on pre-specified local parametricforms that usually have simple structures, either linear or nonlinear,which may not be comprehensive enough to describe the actual
patterns in the modern nonlinear time series, and thus easily lead tomodel misspecification.
3

DS M

576

Motivation

1-6

Deep learning

Recurrent neural networks (RNN) have emerged as the new
benchmark to model nonlinear time series with highly complex
dependence.

 Gate structure alleviates gradient vanishing:
Long-Short Term Memory (LSTM,Hochreiter et al., 1997),
Gated recurrent unit (GRU,Chung et al., 2014), Transformers(Li et al., 2019) and temporal convolution networks (Senet al., 2019)
DS3 M

Motivation

Recurrent Neural Networks (RNN)

1-7

 Transition function for hidden states ht to encode past input
x1:t with a deterministic nonlinear function
ht = f (ht≠1 , xt )(3)

f is commonly chosen as LSTM or GRU.
 Emission function for yt
yt |ht ≥ ﬁ(yt ; „)(4)

„ = g(ht )(5)g is often chosen to be a nonlinear function.
DS3 M

577

Motivation

Overparametrization and interpretation

1-8

 The classic DL models are deterministic and ignore the presence ofunobserved stochastic signals.
 The only randomness allowed appears in the conditional output
probability models, with either a simple unimodal distribution, e.g.
Gaussian (Salinas et al., 2020), or a mixture of simple unimodal
distributions, e.g. Gaussian Mixture models (Graves, 2013).
 It has to require a large number of parameters to ensure areasonable modeling accuracy (Zhang et al., 2005). This in turnrequires a large sample size to ensure estimation efficiency and to
avoid overfitting.
The relatively small sample size of real data, and more importantly, thestochastic behaviors of regime switching make standard deep learning
approaches computationally infeasible and lack of interpretation on thefitted models.
DS3 M

Motivation

1-9

SSM and RNN

Figure: State Space Model

Figure: Recurrent Neural Network

 SSM only allows for simple state structure or linear transitions;
RNN enables to represent complex dependence with richer
internal states and nonlinear transitions
 The latent variables in SSM are random;
The hidden states in RNNs are deterministic.
DS3 M

578

Motivation

Deep state space models (DSSM)

1-10

DSSM introduce continuous Gaussian latent variables at eachtime-step and combine SSM and RNN/MLP.
 SRNNs (Fraccaro et al., 2016) incorporate the deterministic
dynamics of RNNs by interlacing them with SSM. The hidden states
do not depend on zt and transition is linear.
 STORNs (Bayer et al., 2014), VRNN (Chung et al., 2015) make thetransition equation non-linear via cutting the ties between the latentstates and connect them through the deterministic state of RNNs.

zt continuous.
Uneasy to interpret
DS3 M

Motivation

DSSM with discrete latent variable

1-11

While continuous latent variables has more expressive power,
discrete latent variables representing regime switching are natural.
 Johnson et al. (2016): emission function is a neural network.
 Dong et al. (2020): both the emission and transition functions
are nonlinear neural networks.
 Farnoosh et al. (2021): approximates high-dimensional timeseries with a multiplication of latent factors and latent
weights, where the latent weights are modeled by a nonlinear
autoregressive model, switched by a Markov chain of discretelatent variables.
The evolution of time series is purely driven by the discrete latentvariable only.
DS3 M

579

Motivation

1-12

Deep Switching State Space Model (DS3 M)

� Incorporate both continuous and discrete latent into RNN andprove consistency and stability:
I RNN + nonlinear SSSM: emission and transition governed bya Markov chain of dt and parameterized by MLPs.

I discrete latent dt represents unknown regimes and influences
both Yt and continuous latent zt
I zt in the SSSM could use the long-term information embeddedin the RNN; the RNN is skip-connected to the observations tofurther improve the forecasting.

� Develop efficient estimation based on an approximate
variational inference algorithm that can scale to large data sets� DS3 M can leverage the interpretability of discrete latentvariables, the powerful representation ability of continuous
latent variables, and the nonlinearity of deep learning models
compared to SOTA.

DS3 M

Outline1. Motivation

X

2. SOTA
3. Model4. Experiments
5. Conclusion

DS3 M

580

SOTA

2-13

Settings

� A time series of T observations as y1:T = {y1 , y2 , · · · , yT },yt œ RD .

� A sequence of inputs as x1:T = {x1 , x2 , · · · , xT }, xt œ RU .

I In time series forecasting, xt can be one or multiple lagged
values of the time series, e.g. yt≠1 and higher ordersyt≠2 , yt≠3 , · · · .

I The inputs xt could also contain exogenous variables.

We are interested in modeling p(y1:T |x1:T ) and inferring thepredictive distributions for the one-step-ahead to · -step-aheadobservations {yT +1 , · · · , yT +· } and the discrete latent states{dT +1 , · · · , dT +· }.
DS3 M

SOTA

Switching linear dynamical system (SLDS)

2-14

� The dynamics of each regime is explained by a linear state
space model.
zt = Wz(dt ) zt≠1 + Wx(dt ) xt + bz(dt ) + et , et ≥ N(0, Σz(dt ) )yt = Wy(dt ) zt + by(dt ) + ‘t , ‘t ≥ N(0, Σy(dt ) )

(6)
(7)

� The discrete latent variables dt œ {1, 2, · · · , K } at each time
step t = 1, 2, · · · , T , follows a Markov chain, dt |dt≠1 follows atransition matrix Γ œ R K ◊K , where Γi,j = p(dt = j|dt≠1 = i).
� The discrete latent variables dt have impact on both thecontinuous latent variables zt œ RZ and yt

When K = 1, the model is also termed as the Linear GaussianState Space Model (LGSSM).
DS3 M

581

SOTA

2-15

SOTA

 GRU, SRNN (Fraccaro et al., 2016), DSARF (Farnoosh et al.,2021) and SNLDS (Dong et al., 2020)(a) DSARF(b) SNLDS Most of the extensions assumed that the discrete latent dt
only influences the transition of the continuous latent zt .

 SRNN can be viewed as DS3 M model without discrete latentvariables.
3

DS M

SOTA

Other alternative models

2-16 Recurrent SLDS (rSLDS) by Becker-Ehmck et al. (2019) and
Linderman et al. (2017) which extends the open-loop Markov
dynamics and makes dt depending on the hidden state zt≠1 .
Dong et al. (2020) extended the open-loop Markov dynamics
by making dt depends on last observations.
 Tree structure prior on the switching variables of rSLDS(Nassar et al., 2018), deep Rao-Blackwellised Particle Filter(Kurle et al., 2020) Sometimes can improve accuracy, but can also lead to
unnecessarily frequent state shifts in the estimated discretelatent variables, making interpretations difficult.
DSARF has been shown to outperform several models such asrSLDS, SLDS for time series forecasting (Farnoosh et al., 2021).
DS3 M

582

Model3

3-17

Deep Switching State Space Model (DS M)

 stack an RNN below the switching state space model design a direct connection of the RNN hidden state ht to thetime series yt inspired by the skip connection in ResNet,
Transformers and SRNN.
DS3 M

Model

3-18

Formulation

1. Recurrent step:ht = fh (ht≠1 , xt )fh is chosen as an LSTM or GRU
2. Switching step: p(dt |dt≠1 ) follows a Markovian transition
matrix Γ œ R K ◊K
Γi,j = p(dt = j|dt≠1 = i)

3. Transition step:
(k)

(k)

µt
(k)

(k)

f1 ,f2(k)(zt |zt≠1 , ht , dt = k) ≥ N(zt ; µt , Σt )
(k)

= f1

(zt≠1 , ht ) ,(k)log Σt(k)= f2(zt≠1 , ht )are parameterized by neural network models (MLP)

DS3 M

583

Model3-19

Formulation (Con’t)
4. Output step:(k)

yt |zt , ht , dt = k ≥ ﬁ(Φt )(k)Φt= fo(k) (zt , ht )

(8)
(9)

(k)

fo is parameterized by neural network models (MLP)ﬁ can be chosen according to the stochastic nature of thetime series, e.g. Gaussian for bell-shaped data, Log-Gaussian
for data with asymmetry etc.
The DS3 M includes all the parameters that parameterize thefollowing functions:
(k)

(k)

K
(k) K
◊ = {fh , Γ, fo , {f1 }K
k=1 , {f2 }k=1 , {fo }k=1 }

DS3 M

Model

3-20

Estimation
The loglikelihood is
L(◊) = log p◊ (y1:T |x1:T )p◊ (y1:T , z1:T , d1:T |x1:T ) =rT

t=1 p◊ (yt |zt , ht , dt ) p◊ (zt |zt≠1 , ht , dt ) p◊ (dt |dt≠1 ) L(◊) can be obtained by averaging out z1:T and d1:T in above
joint probability. Intractable!
 Maximum likelihood method is not applicable!
 Use variational inference instead. Specifically, we design aninference network with parameter „, i.e. using anapproximated posterior q„ (z1:T , d1:T |y1:T , x1:T ) for the true
posterior p◊ (z1:T , d1:T |x1:T , y1:T ) and then optimize anevidence lower bound ELBO(◊, „).
DS3 M

584

Model

3-21

Variational Inference For any approximated posterior q„ (z1:T , d1:T |y1:T , x1:T )L(◊) Ø ELBO(◊, „)⁄⁄p◊ (y1:T , z1:T , d1:T |x1:T )=q„ (z1:T , d1:T |y1:T , x1:T ) log
dz1:T dd1:Tq„ (z1:T , d1:T |y1:T , x1:T )= Eq„ [log p◊ (y1:T |z1:T , d1:T , x1:T )]≠ KL (q„ (z1:T , d1:T |y1:T , x1:T ) Îp◊ (z1:T , d1:T |x1:T )) When q„ (z1:T , d1:T |y1:T , x1:T ) = p◊ (z1:T , d1:T |y1:T , x1:T )(the true posterior), we have L(◊) = ELBO(◊, „).

DS3 M

Model

How to choose q„ (z1:T , d1:T |y1:T , x1:T )?

3-22

 To achieve a tight ELBO, consider true posterior factorization
derived from the d-separation (Geiger et al., 1990):p◊ (z1:T , d1:T |y1:T , x1:T ) =Ÿtp◊ (zt |zt≠1 , dt , yt:T , ht:T ) p◊ (dt |dt≠1 , yt:T , ht:T ) We design the approximated posterior with the same form offactorization:Ÿq„ (z1:T , d1:T |y1:T , x1:T ) =q„z (zt |zt≠1 , dt , At ) q„d (dt |dt≠1 , At )twhere At = g„A (At+1 , [yt , ht ]), „ = {„z , „d , „A }, and g„A isparameterized as a backward RNN.
DS3 M

585

Model

3-23

Parameterization

 q„z (zt |zt≠1 , dt , At ) is parameterized to be a Gaussian density:
(k)

(k)

zt |zt≠1 , At , dt = k ≥ N(zt ; µt , Σt )(k)

= g1 (zt≠1 , At ) ,(k)is parameterized by neural network model (MLP)µt(k)

g1 , g2(k)(k)

log Σt(k)= g2 (zt≠1 , At )

 q„d (dt |At , dt≠1 ) is parameterized to be a Categorical
distribution:
dt |At ,dt≠1 =k ≥ Cat(softmax(W (k) At ))
DS3 M

Model

3-24

Inference Network

Figure: The graphical model which represents q„ (z1:T , d1:T |y1:T , x1:T ).
DS3 M

586

ModelHow to estimate the parameters?

3-25

 Instead of maximizing L(◊) w.r.t ◊, we maximize a variational
evidence lower bound ELBO(◊, „) w.r.t ◊, „and ˆELBO(◊,„)
 ˆELBO(◊,„)
ˆ◊ˆ„

 Method: Stochastic gradient descent

DS3 M

Model

3-26

Factorization of ELBO

 With the defined approximate posterior, the ELBO can be
rewritten as
ELBO(◊, „) = Eq„ [log p◊ (y1:T |z1:T , d1:T , h1:T )]≠ KL (q„ (z1:T , d1:T |y1:T , h1:T ) Îp◊ (z1:T , d1:T |h1:T )) The factorization of the approximated posterior:q„ (z1:T , d1:T |y1:T , x1:T ) =Ÿtq„z (zt |zt≠1 , dt , At ) q„d (dt |dt≠1 , At )

 The factorization for the prior:p◊ (z1:T , d1:T |x1:T ) = p◊ (z1:T , d1:T |h1:T ) = p(d1:T )p (z1:T |d1:T , h1:T )Ÿp(dt |dt≠1 )p (zt |zt≠1 , dt , ht )
=
t

DS3 M

587

Model3-27

Factorization of ELBO (Con’t)
ELBO(◊, „)
=

q

t

;

Eq„ú (zt≠1 ,dt≠1 )Eq„ú (zt≠1 ,dt≠1 )≠ Eq„ú (dt≠2 )q

dt q„d (dt )Eq„z (zt ) [log p◊ (yt |zt , dt , ht )] ≠dt q„d (dt )KL [q„z (zt |zt≠1 , dt , At ) Îp◊ (zt |zt≠1 , dt , ht )]<q

dt≠1 q„d (dt≠1 )KL [q„d (dt |dt≠1 , At ) Îp◊ (dt |dt≠1 )] ,

 q„ú (zt , dt ) = q„ú (dt ) =

q

s

s

q„ (z1:t , d1:t |y1:T , x1:T ) dz1:t≠1 dd1:t≠1q„d (d1:t |y1:T , x1:T ) dd1:t≠1 .

DS3 M

Model

3-28

Approximate ELBO

We approximate the ELBO using a Monte Carlo method. We(s)(s)
sample (zt , dt ) for t = 1 · · · , T from q„ú (zt , dt ) using ancestral
sampling.
ELBO(◊, „)
;
1
2
q q
(s)
¥ t
dt q„d (dt ) log p◊ yt |zt , dt , ht
Ë
1
2
1
2È
q
(s)
(s)
≠ dt q„d (dt )KL q„z zt |zt≠1 , dt , At Îp◊ zt |zt≠1 , dt , ht<q
≠ dt≠1 q„d (dt≠1 )KL [q„d (dt |dt≠1 , At ) Îp◊ (dt |dt≠1 )] .
DS3 M

588

ModelGradient:

3-29

ˆELBO(◊,„)
ˆ◊

The derivative of the ELBO(◊, „) with respect to ◊ can becalculated as:
ˆELBO(◊, „)ˆ◊= Eq„ (z1:T ,d1:T |y1:T ,x1:T )Tÿˆ log p◊ (y1:T , z1:T , d1:T |x1:T )ˆ◊ˆ log p◊ (yt |zt , dt , ht ) ˆ log p◊ (zt |zt≠1 , dt , ht ) ˆ log p◊ (dt |dt≠1 )
+
+
}
ˆ◊
ˆ◊
ˆ◊
t=1
1
2
1
2
1
2
(s)
(s)
(s) (s)(s)(s) (s)Tˆ log p◊ zt |zt≠1 , dt , ht
ˆ log p◊ dt |dt≠1
ˆ log p◊ yt |zt , dt , ht
ÿ
¥
+
+
}
Eq„ {
ˆ◊
ˆ◊
ˆ◊
t=1

=

Eq„ {

(10)

DS3 M

Model

Gradient:

3-30

ˆELBO(◊,„)
ˆ„

The derivative of the ELBO(◊, „) with respect to „ is more tricky
as „ appears in the expectation in ELBO(◊, „).
� Score function gradient estimator (Williams, 1992) can beused but suffer from high variance.
� Reparameterization trick (Kingma et al., 2014; Rezende et al.,2014) is often used instead, low variance gradient estimator
I z = g(‘; „)ˆEq(z|x ;„) f (z)ˆE‘ f (g(‘; „))ˆf (g(‘; „))== E‘ [
]
ˆ„
ˆ„
ˆ„

(11)

I e.g.
zt ≥ N(µt , Σt ), ‘t ≥ N(0, 1), zt = g(‘; µt , Σt ) = µt + ‘t Σt
DS3 M

589

ModelPredictive distributions using Monte Carlo3-31

 First make inference on the posterior distributions of(s)(s) (s) T +·{zt , dt }T
t=1 and then generate samples of {zt , dt , yt }t=T ,s = 1, · · · , S. S represents the number of Monte Carlosamples.
 The predictive distributions for the one-step-ahead to· -step-ahead observations {yT +1 , · · · , yT +· } and the discretelatent variables {dT +1 , · · · , dT +· } are then approximatedwith empirical distribution functions of the generated samples.

DS3 M

Model

3-32

Stability in mean square

Theorem(k)(k)

Under regular conditions, for the neural networks f1 and f2 that(k)(k)
parameterize the mean µt and diagonal covariance matrix Σt of(k)(k)
the latent state dynamics zt ≥ N(µt , Σt ) with arbitrary
activation function a, there exists an equivalent pointwise affine
map which ensures that the latent variable zt is globally stable inmean-square.

DS3 M

590

Experiments

4-33

Experiments
� Simulations

I Toy example
I Lorenz attractor

� Real data analysis

I Six datasets in Econ, Medicine, Traffic, Meteorology, Energy
I Sleep apnea, Hangzhou metro flow, Seattle traffic flow, Pacifictemperature, Unemployment rate, French electricity
I First four datasets are analyzed in Farnoosh et al. (2021); The
French electricity is analyzed in Xu et al. (2021), theunemployment rate is selected to represent Econ data.

DS3 M

Experiments

4-34

Toy example
d0 ≥ Bernouli(0.5), z0 = 0CD

0.95 0.05, dt œ {0, 1}
dt |dt≠1 ≥ Γ =0.05 0.95
xt = yt≠1(0)(0)

zt|dt =0 = 0.6zt≠1 + 0.4 ◊ tanh(xt + zt≠1 ) + wt , wt
(1)

(1)

zt|dt =1 = 0.1zt≠1 + 0.2 ◊ sin(xt + zt≠1 ) + wt , wt
(0)

(0)

yt|dt =0 = 1.5zt + tanh(zt ) + vt , vt
(1)

(1)

yt|dt =1 = 0.5zt + sin(zt ) + vt , vt
DS3 M

591

≥ N(0, 5)≥ N(0, 0.5)

≥ N(0, 10)≥ N(0, 1)

Experiments

Toy example (1- step-ahead forecasting)

4-35

Figure: Prediction for the toy example. The red color represents dt = 0
and the blue color for dt = 1
DS3 M

Experiments

4-36

Toy example
Table: Summary of the simulation results (mean ± standard deviation)
Toy
Forecasting

RMSE
Duration for dt=1
Duration for dt=0
Accuracy (%)
F1 score

Inference

Accuracy (%)
F1 score

DS3 M

SNLDS

DSARF

14.572 ± 0.352
7.509 ± 1.579
7.634 ± 1.667
0.788 ± 0.033
0.778 ± 0.023

16.541 ± 0.024
1.282 ± 0.001
1.667 ± 0.012
0.543 ± 0.001
0.549 ± 0.001

15.244 ± 0.136
3.946 ± 0.426
3.274 ± 0.985
0.765 ± 0.047
0.757 ± 0.035

0.849 ± 0.004
0.831 ± 0.005

DS3 M

592

0.692 ± 0.003
0.544 ± 0.002

0.819 ± 0.044
0.808 ± 0.039

Experiments

4-37

Lorenz attractor

Lorenz attractor is a canoinal nolinear dynamical system with the
dynamics:
yt = Wzt + vt , where W œ R 10◊3 , vt ≥ N(0, 0.5I10 ).
ST– (z2 ≠ z1 )dz W
X= U z1 (— ≠ z1 ) ≠ z2 Vdt
z1 z2 ≠ “z3 The latent variable zt = [zt,1 , zt,2 , zt,3 ]T . yt œ R 10 isobservable.
 Simulated the time series with a length of 3 000 and transformthe time series into subsequences with a length of 5.
 Training: Validation: Testing = 1:1:1
DS3 M

Experiments

4-38

Lorenz attractor forecasted switching variable

Figure: DS3 M: The forecasted switching variable against the true zt .
DS3 M

593

Experiments

4-39

Lorenz attractor
Table: Summary of the simulation results (mean ± standard deviation)
LorenzForecasting

RMSE
Accuracy (%)
F1 score

Inference

Accuracy (%)
F1 score

DS3 M

SNLDS

DSARF

0.168 ± 0.017
0.882 ± 0.079
0.837 ± 0.127

0.226 ± 0.065
0.616 ± 0.065
0.600 ± 0.100

0.030 ± 0.000
0.788 ± 0.143
0.775 ± 0.124

0.911 ± 0.068
0.883 ± 0.103

0.744 ± 0.174
0.680 ± 0.244

0.789 ± 0.146
0.761 ± 0.113

DS3 M

Experiments

4-40

Real data
Table: Description of the datasets
DatasetFrequency

Dimension

T _train+T _valid

Sleep
Unemployment
HangzhouSeattlePacific
Electricity

half a second
month
10 mins
5 minsmonthhalf a hour

1
1
80
323
2520
48

1000
639
2160
6624
336
2601

T _test (time)
1000
240
540
1440
60
320

(500 seconds)(20 years)(5 days)(5 days)(5 years)(1 year) Short-term: 1-step-ahead forecast
 Long-term: make multiple forecasts simultaneously for all
T _test, standing at the end of the T _train+T _valid
DS3 M

594

Experiments

4-41

Real data analysis

Table: Summary of forecastes on testing data. The best models are in
bold. “-" indicates the model forecasts diverge to unreasonable values
and are omitted.

Short-termLong-term

Datasets
Sleep
Unemployment
HangzhouSeattlePacific
Electricity
HangzhouSeattlePacific

DS3 M
1201
0.75
32.53
4.16
0.57
2971
47.50
4.17
0.72

SNLDS
2789
1.59
36.67
4.18
15.78
5133
42.83
4.19
-

RMSE
DSARF
1557
1.06
34.81
4.44
0.53
8805
42.28
0.73

SRNN
1806
2.01
33.80
4.17
0.58
3642
60.89
4.17
0.98

GRU
1264
1.05
38.42
4.18
0.56
4784
73.18
16.93
0.76

DS3 M
15.46
4.53
24.04
5.81
1.69
4.58
38.20
5.81
2.15

MAPE (%)
SNLDS DSARF SRNN
88.06
39.25
50.8
16.13
8.11
23.15
23.90
29.73
25.40
5.85
7.27
6.00
58.01
1.57
1.74
7.79
18.64
5.34
50.6
43.65
82.81
5.86
5.81
2.29
2.99

GRU
31.17
5.13
30.48
6.89
1.68
6.60
86.61
27.95
2.22

 Long-term forecasting is only achievable for time series withregular patterns (Farnoosh et al., 2021). Thus, we exclude
some datasets for long-term forecasting.
DS3 M

Experiments

4-42

Short-term forecasts(a) Sleep apnea (measured at 2 Hz)(b) US unemployment rate
DS3 M

595

Experiments

4-42

Short-term forecasts II(c) Hangzhou metro station 0(d) Seattle traffic loop 0(e) Pacific location 0(f) French Electricity 0:00

DS3 M

Conclusion

5-43

Conclusion Proposed a deep switching state space model (DS3 M) for
forecasting nonlinear time series with regime switching.

 The switching among regimes is modeled by both discrete andcontinuous latent variables with recurrent neural networks.
 Developed an efficient scalable inference and learning
method The DS3 M achieved competitive performance against several
state-of-the-art methods for a variety of simulated and real
datasets.
 Code and data are available at https://github.com/Sherry-Xu.
DS3 M

596

Deep Switching State Space Model (DS3 M) for
Nonlinear Time Series Forecasting with Regime
Switching
Xiuqin XU
Ying CHEN

https://arxiv.org/abs/2106.02329
Institute of Data Science & Integrative Sciences and Engineering Programme
Department of Mathematics & Risk Management Institute & Asian Institute of
Digital Finance
National University of Singapore

References

7-44

References
Ackerson, G and K Fu (1970)
On state estimation in switching environments
IEEE Transactions on Automatic Control 15 (1),pp. 10–17.
Bae, G. I., W. C. Kim, and J. M. Mulvey (2014)
Dynamic asset allocation for varied financial markets
under regime switching framework
European Journal of Operational Research 234 (2),pp. 450–458.

DS3 M

597

References

7-44

References II

Baum, L. E. and T. Petrie (1966)
Statistical inference for probabilistic functions of finite
state Markov chains
The annals of mathematical statistics 37 (6),pp. 1554–1563.
Bayer, J. and C. Osendorfer (2014)‘Learning Stochastic Recurrent Networks’
NIPS 2014 Workshop on Advances in Variational
Inference, pp. 1–9.

DS3 M

References

7-44

References III
Becker-Ehmck, P., J. Peters, and P. Van Der Smagt(2019)‘Switching Linear Dynamics for Variational Bayes
Filtering’
International Conference on Machine Learning,pp. 553–562.
Chang, C.-B. and M. Athans (1978)
State estimation for discrete systems with switchingparameters
IEEE Transactions on Aerospace and Electronic
Systems AES-14 (3), pp. 418–425.
DS3 M

598

References

7-44

References IV

Chung, J., C. Gulcehre, K. Cho, and Y. Bengio (2014)‘Empirical evaluation of gated recurrent neuralnetworks on sequence modeling’
NIPS 2014 Workshop on Deep Learning, pp. 1–9.
Chung, J., K. Kastner, L. Dinh, K. Goel,
A. C. Courville, and Y. Bengio (2015)‘A recurrent latent variable model for sequential data’
Advances in Neural Information Processing Systems,
pp. 2980–2988.

DS3 M

References

7-44

References V
Dong, Z., B. Seybold, K. Murphy, and H. Bui (2020)‘Collapsed amortized variational inference forswitching nonlinear dynamical systems’
International Conference on Machine Learning,pp. 2638–2647.
Doucet, A. and A. M. Johansen (2009)
A tutorial on particle filtering and smoothing: Fifteen
years later
Handbook of nonlinear filtering 12 (656-704), p. 3.
Durbin, J. and S. J. Koopman (2012)
Time series analysis by state space methods
Vol. 38
Oxford University Press.
DS3 M

599

References

7-44

References VI
Farnoosh, A., B. Azari, and S. Ostadabbas (2021)‘Deep Switching Auto-Regressive Factorization:
Application to Time Series Forecasting’Proceedings of the AAAI Conference on Artificial
Intelligence
8, pp. 7394–7403.
Fox, E., E. B. Sudderth, M. I. Jordan, and
A. S. Willsky (2009)‘Nonparametric Bayesian learning of switching linear
dynamical systems’
Advances in Neural Information Processing Systems,
pp. 457–464.
DS3 M

References

7-44

References VII
Fraccaro, M., S. K. Sønderby, U. Paquet, and
O. Winther (2016)‘Sequential neural models with stochastic layers’
Advances in Neural Information Processing Systems,
pp. 2199–2207.
François, P., G. Gauthier, and F. Godin (2014)
Optimal hedging when the underlying asset follows aregime-switching Markov process
European Journal of Operational Research 237 (1),
pp. 312–322.
Geiger, D., T. Verma, and J. Pearl (1990)
Identifying independence in Bayesian networks
Networks 20 (5), pp. 507–534.
DS3 M

600

References

7-44

References VIII
Ghahramani, Z. and G. E. Hinton (2000)
Variational learning for switching state-space models
Neural Computation 12 (4), pp. 831–864.
Graves, A. (2013)
Generating Sequences With Recurrent Neural
Networks
CoRR abs/1308.0850
arXiv: 1308.0850.
Hamilton, J. D. (1990)
Analysis of time series subject to changes in regime
Journal of Econometrics 45 (1-2), pp. 39–70.
DS3 M

References

7-44

References IX
Hochreiter, S. and J. Schmidhuber (1997)
Long short-term memory
Neural Computation 9 (8), pp. 1735–1780.

Johnson, M., D. K. Duvenaud, A. Wiltschko,
R. P. Adams, and S. R. Datta (2016)‘Composing graphical models with neural networks forstructured representations and fast inference’
Advances in Neural Information Processing Systems,
pp. 2946–2954.

DS3 M

601

References

7-44

References X
Julier, S. J. and J. K. Uhlmann (1997)‘New extension of the Kalman filter to nonlinear
systems’
Signal processing, sensor fusion, and target
recognition VI
Vol. 3068
International Society for Optics and Photonics,
pp. 182–194.

Kingma, D. P. and M. Welling (2014)‘Auto-Encoding Variational Bayes’
International Conference on Learning Representations,pp. 1–9.
DS3 M

References

7-44

References XI
Kurle, R., S. S. Rangapuram, E. de Bézenac,
S. Günnemann, and J. Gasthaus (2020)‘Deep Rao-Blackwellised Particle Filters for Time
Series Forecasting’
Advances in Neural Information Processing Systems,
pp. 15371–15382.
Li, S., X. Jin, Y. Xuan, X. Zhou, W. Chen,
Y.-X. Wang, and X. Yan (2019)‘Enhancing the locality and breaking the memory
bottleneck of transformer on time series forecasting’
Advances in Neural Information Processing Systems,
pp. 5243–5253.
DS3 M

602

References

7-44

References XII
Linderman, S., M. Johnson, A. Miller, R. Adams,
D. Blei, and L. Paninski (2017)‘Bayesian learning and inference in recurrent switching
linear dynamical systems’
Artificial Intelligence and Statistics, pp. 914–922.
Murphy, K. P. (1998)
Switching kalman filters
.

DS3 M

References

7-44

References XIII

Nassar, J., S. Linderman, M. Bugallo, and I. M. Park(2018)‘Tree-Structured Recurrent Switching Linear
Dynamical Systems for Multi-Scale Modeling’
International Conference on Learning Representations,pp. 1–9.
Rezende, D. J., S. Mohamed, and D. Wierstra (2014)‘Stochastic Backpropagation and Approximate
Inference in Deep Generative Models’International Conference on Machine Learning,pp. 1278–1286.
DS3 M

603

References

7-44

References XIV
Salinas, D., V. Flunkert, J. Gasthaus, andT. Januschowski (2020)
DeepAR: Probabilistic forecasting with autoregressive
recurrent networks
International Journal of Forecasting 36 (3),pp. 1181–1191.
Sen, R., H.-F. Yu, and I. S. Dhillon (2019)‘Think globally, act locally: A deep neural network
approach to high-dimensional time series forecasting’
Advances in Neural Information Processing Systems,
pp. 4837–4846.
DS3 M

References

7-44

References XVSmith, G. L., S. F. Schmidt, and L. A. McGee (1962)
Application of statistical filter theory to the optimal
estimation of position and velocity on board a
circumlunar vehicle.
Williams, R. J. (1992)
Simple statistical gradient-following algorithms forconnectionist reinforcement learning
Machine Learning 8 (3-4), pp. 229–256.
Xu, X., Y. Chen, Y. Goude, and Q. Yao (2021)
Probabilistic Forecasting for Daily Electricity Loads
and Quantiles for Curve-to-Curve Regression
Applied Energy 301, pp. 1–14.
DS3 M

604

References

7-44

References XVIZhang, G. P. and M. Qi (2005)
Neural network forecasting for seasonal and trend
time series
European Journal of Operational Research 160 (2),pp. 501–514.

DS3 M

605

The 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop on
Advances in Classical and Quantum Algorithms for
Optimization and Machine Learning
September 16th - 19th, 2022, Tokyo (The university of Tokyo), Japan,
and September 21st - 22nd, 2022, Fukuoka (Kyushu University), Japan

Institute of Mathematics for Industry:
its uniqueness, strength and prospects
Osamu SAEKI
Institute of Mathematics for Industry, Kyushu University, Japansaeki@imi.kyushu-u.ac.jp
As director of the Institute of Mathematics for Industry (IMI), Kyushu University,
which is a unique institute of industrial mathematics in Japan, the speaker will present
various activities of IMI which characterize its strength; researches in fundamental
mathematics, mathematics applied to other disciplines, joint projects with industry,
together with educational activities, including the WISE program funded by theJapanese government. Some prospects of IMI for the future will also be presented.

References[1] Institute of Mathematics for Industry, Kyushu University, An Unparalleled Research Institute forIndustrial Mathematics Based on Diverse Fields of Mathematical Study, official homepage,https://www.imi.kyushu-u.ac.jp/en/

607

Institute of Mathematics for Industry:
its uniqueness, strength and prospects!"#$%&'#()*+*,(-./,0&12".*.%.(&/3&4#.5($#.*-"&3/,&126%".,7&!"#IMI$
Kyushu University, Fukuoka, JapanThe 6th RIKEN-IMI-ISM-NUS-ZIB-MODAL-NHR Workshop on Advances inClassical and Quantum Algorithms for Optimization and Machine Learning
September 21, 2022

MEXT Joint Use / Research Center
Joint Research Center for Advanced andFundamental Mathematics-for-Industry

Since 2011

Unparalleled industrial mathematics research institute witha foundation in diverse mathematical research

Joint Use
CenterIndustrial
Collaboration

Education
Research
International
Collaboration

Math-for-Industry: A new area of research in mathematics that is valuable as mathematics
itself as well, responding to the demands of society and industry, by reorganizing pure andapplied mathematics.
Joint Use / Research Center・ Workshops (I),(II)・ Short-term joint
research・ Short-term
visiting researcher・ General Research・ Project Research・ International
× Project Research・ Young Researcher
・ Female Researcher

Joint researches with Industry
and social implementations
Joint
Research

Commissioned
research

25

15

Research Achievements
First in the world ranking in the Graph500
benchmark test using the supercomputer Fugaku(total of 15 terms, 5 consecutive victories).

2021

Participation from industry is essential.

International Activities
IMI Australia BranchAsia-Pacific Consortiumof Mathematics for
Industry

Publications

International Journal of
Mathematics for Industry

Springer Series Mathematics
for Industry

Forum “Math-for-Industry”

12 volumes
published608

31 volumes
published

Education
Graduate School of Mathematics

Graduate Program of
Mathematics for Innovation
Long-term research internship（2006--）
Study Group Workshops （2010--）

International Activities of IMIForum “Math-for-Industry” 2008－

Since October 2014

IMI Australia Branch at Melbourne

Graduate Program of Mathematics for Innovation
RIKEN, ISM,
NUS, ZIB also
participate inthis WISE
Program!

The most important graduate program of Kyushu University

609

WISE Program supported by
MEXT, JapanMI レクチャーノートシリーズ刊行にあたり
本レクチャーノートシリーズは、文部科学省 21 世紀 COE プログラム「機能数理学の構築と展開」（H15-19 年度）において作成した COE Lecture Notesの続刊であり、文部科学省大学院教育改革支援プログラム「産業界が求める数学博士と新修士養成」（H19-21 年度）および、同グローバル COE プログラ
ム「マス・フォア・インダストリ教育研究拠点」（H20-24 年度）において行われた講義の講義録として出版されてきた。平成 23 年 4 月のマス・フォア・イ
ンダストリ研究所（IMI）設立と平成 25 年 4 月の IMI の文部科学省共同利用・共同研究拠点として「産業数学の先進的・基礎的共同研究拠点」の認定を受け、今後、レクチャーノートは、マス・フォア・インダストリに関わる国内外の研究者による講義の講義録、会議録等として出版し、マス・フォア・インダ
ストリの本格的な展開に資するものとする。
2022 年 10 月
マス・フォア・インダストリ研究所
所長

梶原 健司

IMI Workshop of the Joint Usage Research Projects

Construction of Mathematical Basis for
Realizing Data Rating Service
発
編

行
集

発

行

印

刷

2022年12月27日
Katsuki Fujisawa, Shizuo Kaji, Toru Ishihara, Masaaki Kondo, Yuji Shinano,Takuji Tanigawa, Naoko Nakayama
九州大学マス・フォア・インダストリ研究所
九州大学大学院数理学府〒819-0395 福岡市西区元岡744
九州大学数理・IMI 事務室
TEL 092-802-4402 FAX 092-802-4405
URL https://www.imi.kyushu-u.ac.jp/
城島印刷株式会社〒810-0012 福岡市中央区白金 2 丁目 9 番 6 号
TEL 092-531-7102 FAX 092-524-4411

シリーズ既刊
Issue

Author／Editor

Title

Published

COE Lecture Note

Mitsuhiro T. NAKAO
Kazuhiro YOKOYAMA

Computer Assisted Proofs - Numeric and Symbolic Approaches 199pages

August 22, 2006

COE Lecture Note

M.J.Shai HARAN

Arithmetical Investigations - Representation theory, Orthogonal
polynomials and Quantum interpolations- 174pages

August 22, 2006

COE Lecture Note Vol.3

Michal BENES
Masato KIMURA
Tatsuyuki NAKAKI

Proceedings of Czech-Japanese Seminar in Applied Mathematics 2005
155pages

COE Lecture Note Vol.4

宮田

辺要素有限要素法による磁界解析 - 機能数理学特別講義 21pages May 15, 2007

COE Lecture Note Vol.5

Francois APERY

Univariate Elimination Subresultants - Bezout formula, Laurent series
and vanishing conditions - 89pages

September 25, 2007

COE Lecture Note Vol.6

Michal BENES
Masato KIMURA
Tatsuyuki NAKAKI

Proceedings of Czech-Japanese Seminar in Applied Mathematics 2006
209pages

October 12, 2007

COE Lecture Note Vol.7

若山 正人
中尾 充宏

九州大学産業技術数理研究センター キックオフミーティング
138pages

October 15, 2007

COE Lecture Note Vol.8

Alberto PARMEGGIANIIntroduction to the Spectral Theory of Non-Commutative Harmonic
Oscillators 233pages

January 31, 2008

COE Lecture Note Vol.9

Michael I.TRIBELSKY

Introduction to Mathematical modeling 23pages

February 15, 2008

COE Lecture Note Vol.10

Jacques FARAUT

Infinite Dimensional Spherical Analysis 74pages

March 14, 2008

COE Lecture Note Vol.11

Gerrit van DIJK

Gelfand Pairs And Beyond 60pages

August 25, 2008

COE Lecture Note Vol.12

Faculty of Mathematics,
Kyushu University

Consortium “MATH for INDUSTRY” First Forum 87pages

September 16, 2008

COE Lecture Note Vol.13

九州大学大学院
数理学研究院

プロシーディング「損保数理に現れる確率モデル」― 日新火災・九州大学 共同研究2008年11月 研究会 ― 82pages

February 6, 2009

健治

October 13, 2006

シリーズ既刊
Issue

Author／Editor

Title

Published

COE Lecture Note Vol.14

Michal Beneš,
Tohru Tsujikawa
Shigetoshi Yazaki

Proceedings of Czech-Japanese Seminar in Applied Mathematics 2008
77pages

February 12, 2009

COE Lecture Note Vol.15

Faculty of Mathematics,
Kyushu University

International Workshop on Verified Computations and Related Topics
129pages

February 23, 2009

COE Lecture Note Vol.16

Alexander Samokhin

Volume Integral Equation Method in Problems of Mathematical Physics February 24, 2009
50pages

COE Lecture Note Vol.17

矢嶋
及川
梶原
辻
福本

非線形波動の数理と物理 66pages

February 27, 2009

COE Lecture Note Vol.18

Tim Hoffmann

Discrete Differential Geometry of Curves and Surfaces 75pages

April 21, 2009

COE Lecture Note Vol.19

Ichiro Suzuki

The Pattern Formation Problem for Autonomous Mobile Robots―Special Lecture in Functional Mathematics― 23pages

April 30, 2009

COE Lecture Note Vol.20

Yasuhide Fukumoto
Yasunori Maekawa

Math-for-Industry Tutorial: Spectral theories of non-Hermitian
operators and their application 184pages

June 19, 2009

COE Lecture Note Vol.21

Faculty of Mathematics,
Kyushu University

Forum “Math-for-Industry”
Casimir Force, Casimir Operators and the Riemann Hypothesis
95pagesNovember 9, 2009

COE Lecture Note Vol.22

Masakazu Suzuki
Hoon Hong
Hirokazu Anai
Chee Yap
Yousuke Sato
Hiroshi Yoshida

The Joint Conference of ASCM 2009 and MACIS 2009:
Asian Symposium on Computer Mathematics Mathematical Aspects ofComputer and Information Sciences 436pages

December 14, 2009

COE Lecture Note Vol.23

荒川 恒男
金子 昌信

多重ゼータ値入門

February 15, 2010

COE Lecture Note Vol.24

Fulton B.Gonzalez

Notes on Integral Geometry and Harmonic Analysis

COE Lecture Note Vol.25

Wayne Rossman

Discrete Constant Mean Curvature Surfaces via Conserved Quantities
130pages

May 31, 2010

COE Lecture Note Vol.26

Mihai Ciucu

Perfect Matchings and Applications

July 2, 2010

徹
正行
健司
英一
康秀

111pages

66pages

125pages

March 12, 2010

シリーズ既刊
Issue

Author／Editor

Title

Published

COE Lecture Note Vol.27

九州大学大学院
数理学研究院

Forum “Math-for-Industry” and Study Group Workshop
Information security, visualization, and inverse problems, on the basisof optimization techniques 100pages

October 21, 2010

COE Lecture Note Vol.28

ANDREAS LANGERMODULAR FORMS, ELLIPTIC AND MODULAR CURVESLECTURES AT KYUSHU UNIVERSITY 2010 62pages

November 26, 2010

COE Lecture Note Vol.29

木田 雅成
原田 昌晃
横山 俊一

Magma で広がる数学の世界

December 27, 2010

COE Lecture Note Vol.30

原
松井
廣島

Mathematical Quantum Field Theory and Renormalization Theory
201pages

January 31, 2011

COE Lecture Note Vol.31

若山 正人
福本 康秀
高木
剛
山本 昌宏

Study Group Workshop 2010 Lecture & ReportFebruary 8, 2011

COE Lecture Note Vol.32

Institute of Mathematics
for Industry,
Kyushu University

Forum “Math-for-Industry” 2011
“TSUNAMI-Mathematical Modelling”
Using Mathematics for Natural Disaster Prediction, Recovery andProvision for the Future 90pages

September 30, 2011

COE Lecture Note Vol.33

若山 正人
福本 康秀
高木
剛
山本 昌宏

Study Group Workshop 2011 Lecture & Report

October 27, 2011

COE Lecture Note Vol.34

Adrian Muntean
Vladimír Chalupecký

Homogenization Method and Multiscale Modeling

COE Lecture Note Vol.35

横山夫
林

計算機代数システムの進展

COE Lecture Note Vol.36

Michal Beneš
Masato Kimura
Shigetoshi Yazaki

Proceedings of Czech-Japanese Seminar in Applied Mathematics 2010
107pages

COE Lecture Note Vol.37

若山 正人
高木
剛
Kirill Morozov
平岡 裕章
木村 正人
白井 朋之
西井 龍映
栄 伸一郎
穴井 宏和
福本 康秀

平成23年度 数学・数理科学と諸科学・産業との連携研究ワーク February 20, 2012
ショップ 拡がっていく数学 ～期待される “見えない力” ～154pages

隆
卓
文生

俊一
紀恵
卓也

157pages

128pages

140pages

72pages

210pages

October 28, 2011

November 30, 2011

January 27, 2012

シリーズ既刊
Issue

Author／Editor

Title

Published

COE Lecture Note Vol.38

Fumio Hiroshima
Itaru Sasaki
Herbert Spohn
Akito Suzuki

Enhanced Binding in Quantum Field Theory

COE Lecture Note Vol.39

Institute of Mathematics
for Industry,
Kyushu University

Multiscale Mathematics: Hierarchy of collective phenomena andinter­relations between hierarchical structures 180pages

March 13, 2012

COE Lecture Note Vol.40

井ノ口順一
太田 泰広
筧
三郎
梶原 健司
松浦
望

離散可積分系・離散微分幾何チュートリアル2012

March 15, 2012

COE Lecture Note Vol.41

Institute of Mathematics
for Industry,
Kyushu University

Forum “Math-for-Industry” 2012
“Information Recovery and Discovery”

COE Lecture Note Vol.42

佐伯
若山
山本

Study Group Workshop 2012 Abstract, Lecture & Report

178pages

November 19, 2012

COE Lecture Note Vol.43

Institute of Mathematics
for Industry,
Kyushu University

Combinatorics and Numerical Analysis Joint Workshop

103pages

December 27, 2012

COE Lecture Note Vol.44

萩原学

モダン符号理論からポストモダン符号理論への展望

107pages

January 30, 2013

COE Lecture Note Vol.45

金山

寛

Joint Research Workshop of Institute of Mathematics for Industry(IMI), Kyushu University“Propagation of Ultra-large-scale Computation by the Domaindecomposition-method for Industrial Problems (PUCDIP 2012)”
121pages

February 19, 2013

COE Lecture Note Vol.46

西井 龍映
栄 伸一郎
岡田 勘三
落合 啓之
小磯 深幸
斎藤 新悟
白井 朋之

科学・技術の研究課題への数学アプローチ―数学モデリングの基礎と展開― 325pages

February 28, 2013

COE Lecture Note Vol.47

SOO TECK LEE

BRANCHING RULES AND BRANCHING ALGEBRAS FOR THE
COMPLEX CLASSICAL GROUPS 40pages

March 8, 2013

COE Lecture Note Vol.48

溝口
脇
平坂
谷口
島袋

博多ワークショップ「組み合わせとその応用」 124pages

March 28, 2013

修
正人
昌宏

佳寛
隼人
貢
哲至
修

204pages

March 12, 2012

152pages

October 22, 2012

91pagesシリーズ既刊
Issue

Author／Editor

COE Lecture Note Vol.49

照井
小原
濱田
横山
穴井
横田

MI Lecture Note Vol.50

章
功任
龍義
俊一
宏和
博史

Title

Published

マス・フォア・インダストリ研究所 共同利用研究集会 II
数式処理研究と産学連携の新たな発展 137pages

August 9, 2013

Ken Anjyo
Hiroyuki Ochiai
Yoshinori Dobashi
Yoshihiro Mizoguchi
Shizuo Kaji

Symposium MEIS2013:
Mathematical Progress in Expressive Image Synthesis

October 21, 2013

MI Lecture Note Vol.51

Institute of Mathematics
for Industry, Kyushu
University

Forum “Math-for-Industry” 2013
“The Impact of Applications on Mathematics”

MI Lecture Note Vol.52

佐伯
岡田
髙木
若山
山本

Study Group Workshop 2013 Abstract, Lecture & Report
142pages

November 15, 2013

MI Lecture Note Vol.53

四方 義啓
櫻井 幸一
安田 貴徳
Xavier Dahan

平成25年度 九州大学マス・フォア・インダストリ研究所
共同利用研究集会 安全・安心社会基盤構築のための代数構造～サイバー社会の信頼性確保のための数理学～ 158pages

December 26, 2013

MI Lecture Note Vol.54

Takashi Takiguchi
Hiroshi Fujiwara

Inverse problems for practice, the present and the future

January 30, 2014

MI Lecture Note Vol.55

栄 伸一郎
溝口 佳寛
脇
隼人
渋田 敬史

Study Group Workshop 2013 数学協働プログラム Lecture & Report
98pages

February 10, 2014

MI Lecture Note Vol.56

Yoshihiro Mizoguchi
Hayato Waki
Takafumi Shibuta
Tetsuji Taniguchi
Osamu Shimabukuro
Makoto Tagami
Hirotake Kurihara
Shuya Chiba

Hakata Workshop 2014
~ Discrete Mathematics and its Applications ~March 28, 2014

MI Lecture Note Vol.57

Institute of Mathematics
for Industry, Kyushu
University

Forum “Math-for-Industry” 2014:“Applications + Practical Conceptualization + Mathematics = fruitful
Innovation” 93pages

October 23, 2014

MI Lecture Note Vol.58

安生健一
落合啓之

Symposium MEIS2014:
Mathematical Progress in Expressive Image Synthesis

November 12, 2014

修
勘三
剛
正人
昌宏

154pages

October 30, 2013

97pages

93pages

141pages

135pagesシリーズ既刊
Issue

Author／Editor

MI Lecture Note Vol.59

西井
岡田
梶原
髙木
若山
脇
山本

龍映
勘三
健司
剛
正人
隼人
昌宏

MI Lecture Note Vol.60

西浦博

MI Lecture Note Vol.61

Title

Published

Study Group Workshop 2014 数学協働プログラム
Abstract, Lecture & Report 196pages

November 14, 2014

平成26年度九州大学 IMI 共同利用研究・研究集会（I）
感染症数理モデルの実用化と産業及び政策での活用のための新たな展開 120pages

November 28, 2014

溝口 佳寛
Jacques Garrigue
萩原
学
Reynald Affeldt

研究集会
高信頼な理論と実装のための定理証明および定理証明器
Theorem proving and provers for reliable theory and implementations(TPP2014) 138pages

February 26, 2015

MI Lecture Note Vol.62

白井朋之

Workshop on “β-transformation and related topics”

March 10, 2015

MI Lecture Note Vol.63

白井朋之

Workshop on “Probabilistic models with determinantal structure”
107pages

August 20, 2015

MI Lecture Note Vol.64

落合
土橋

啓之
宜典

Symposium MEIS2015:
Mathematical Progress in Expressive Image Synthesis

September 18, 2015

MI Lecture Note Vol.65

Institute of Mathematics
for Industry, Kyushu
University

Forum “Math-for-Industry” 2015
“The Role and Importance of Mathematics in Innovation”

MI Lecture Note Vol.66

岡田 勘三
藤澤 克己
白井 朋之
若山 正人
脇
隼人
Philip Broadbridge
山本 昌宏

Study Group Workshop 2015 Abstract, Lecture & Report
156pages

MI Lecture Note Vol.67

Institute of Mathematics
for Industry, Kyushu
University

IMI-La Trobe Joint Conference“Mathematics for Materials Science and Processing”
66pages

February 5, 2016

MI Lecture Note Vol.68

古庄
小谷
新甫

英和
久寿
洋史

結び目と Grothendieck-Teichmüller 群
116pages

February 22, 2016

MI Lecture Note Vol.69

土橋
鍛治

宜典
静雄

Symposium MEIS2016:
Mathematical Progress in Expressive Image Synthesis

MI Lecture Note Vol.70

Institute of Mathematics
for Industry,
Kyushu University

Forum “Math-for-Industry” 2016
“Agriculture as a metaphor for creativity in all human endeavors”
98pagesNovember 2, 2016

MI Lecture Note Vol.71

小磯
二宮
山本

Study Group Workshop 2016 Abstract, Lecture & Report

November 21, 2016

深幸
嘉行
昌宏

59pages

124pages

74pages

October 23, 2015

November 5, 2015

82pages

143pages

October 24, 2016

シリーズ既刊
Issue

Author／Editor
朝雄
泉
文生

Title

Published

MI Lecture Note Vol.72

新井
小嶋
廣島

MI Lecture Note Vol.73

穴田 啓晃
Kirill Morozov
須賀 祐治
奥村 伸也
櫻井 幸一

Secret Sharing for Dependability, Usability and Security of Network
Storage and Its Mathematical Modeling 211pages

March 15, 2017

MI Lecture Note Vol.74

QUISPEL, G. Reinout W.
BADER, Philipp
MCLAREN, David I.
TAGAMI, Daisuke

IMI-La Trobe Joint Conference
Geometric Numerical Integration and its Applications

March 31, 2017

MI Lecture Note Vol.75

手塚
田上
山本

Study Group Workshop 2017 Abstract, Lecture & Report

MI Lecture Note Vol.76

宇田川誠一

Tzitzéica 方程式の有限間隙解に付随した極小曲面の構成理論―Tzitzéica 方程式の楕円関数解を出発点として― 68pages

August 4, 2017

MI Lecture Note Vol.77

松谷 茂樹
佐伯
修
中川 淳一
田上 大助
上坂 正晃
Pierluigi Cesana
濵田 裕康

平成29年度 九州大学マス・フォア・インダストリ研究所
共同利用研究集会（I）
結晶の界面，転位，構造の数理 148pages

December 20, 2017

MI Lecture Note Vol.78

瀧澤 重志
小林 和博
佐藤憲一郎
斎藤
努
清水 正明
間瀬 正啓
藤澤 克樹
神山 直之

平成29年度 九州大学マス・フォア・インダストリ研究所
プロジェクト研究 研究集会（I）
防災・避難計画の数理モデルの高度化と社会実装へ向けて
136pages

February 26, 2018

MI Lecture Note Vol.79

神山
畔上

平成29年度 AIMaP チュートリアル
最適化理論の基礎と応用 96pages

February 28, 2018

MI Lecture Note Vol.80

Kirill Morozov
Hiroaki Anada
Yuji Suga

IMI Workshop of the Joint Research Projects
Cryptographic Technologies for Securing Network Storage
and Their Mathematical Modeling 116pages

March 30, 2018

MI Lecture Note Vol.81

Tsuyoshi Takagi
Masato Wakayama
Keisuke Tanaka
Noboru Kunihiro
Kazufumi Kimoto
Yasuhiko Ikematsu

IMI Workshop of the Joint Research Projects
International Symposium on Mathematics, Quantum Theory,
and Cryptography 246pages

September 25, 2019

MI Lecture Note Vol.82

池森

令和 2 年度 AIMaP チュートリアル
新型コロナウイルス感染症にかかわる諸問題の数理
145pages

March 22, 2021

集
大助
昌宏

直之
秀幸

俊文

Mathematical quantum field theory and related topics

133pages

71pages

118pages

January 27, 2017

October 20, 2017

シリーズ既刊
Issue

Author／Editor

Title

Published

MI Lecture Note Vol.83

早川健太郎
軸丸 芳揮
横須賀洋平
可香谷 隆
林
和希
堺
雄亮

シェル理論・膜理論への微分幾何学からのアプローチとその建築曲面設計への応用 49pages

July 28, 2021

MI Lecture Note Vol.84

Taketoshi Kawabe
Yoshihiro Mizoguchi
Junichi Kako
Masakazu Mukai
Yuji Yasui

SICE-JSAE-AIMaP Tutorial
Advanced Automotive Control and Mathematics

December 27, 2021

MI Lecture Note Vol.85

Hiroaki Anada
Yasuhiko Ikematsu
Koji Nuida
Satsuya Ohata
Yuntao Wang

IMI Workshop of the Joint Usage Research Projects
February 9, 2022
Exploring Mathematical and Practical Principles of Secure Computationand Secret Sharing 114pages

MI Lecture Note Vol.86

濱田 直希
穴井 宏和
梅田 裕平
千葉 一永
佐藤 寛之
能島 裕介
加葉田雄太朗
一木 俊助
早野 健太
佐伯
修

2020年度採択分 九州大学マス・フォア・インダストリ研究所
共同利用研究集会
進化計算の数理 135pages

February 22, 2022

MI Lecture Note Vol.87

Osamu Saeki,
Ho Tu Bao,
Shizuo Kaji,
Kenji Kajiwara,
Nguyen Ha Nam,
Ta Hai Tung,
Melanie Roberts,
Masato Wakayama,
Le Minh Ha,
Philip Broadbridge

Proceedings of Forum “Math-for-Industry” 2021
-Mathematics for Digital Economy- 122pages

March 28, 2022

MI Lecture Note Vol.88

Daniel PACKWOOD
Pierluigi CESANA,
Shigenori FUJIKAWA,
Yasuhide FUKUMOTO,
Petros SOFRONIS,Alex STAYKOVPerspectives on Artificial Intelligence and Machine Learning inMaterials Science, February 4-6, 2022 74pages

November 8, 2022

110pagesMI Lecture Note Vol.89

松谷 茂樹
落合 啓之
井上 和俊
小磯 深幸
佐伯
修
白井 朋之
垂水 竜一
内藤 久資
中川 淳一
濵田 裕康
松江
要
加葉田雄太朗2020年度採択分 九州大学マス・フォア・インダストリ研究所
共同利用研究集会
材料科学における幾何と代数 III 356pages

December 7, 2022

MI Lecture Note Vol.90

中山
谷川
品野
近藤
石原
鍛冶
藤澤

2022年度採択分 九州大学マス・フォア・インダストリ研究所
共同利用研究集会
データ格付けサービス実現のための数理基盤の構築 58pages

December 12, 2022

尚子
拓司
勇治
正章
亨
静雄
克樹

九州大学マス・フォア・インダストリ研究所

九州大学大学院 数理学府
URL http://www.imi.kyushu-u.ac.jp/