# An introduction to pyMarmote and pyMarmoteMDP for Markovian modeling - A tutorial

Alain Jean-Marie, Emmanuel Hyon

► To cite this version:

Alain Jean-Marie, Emmanuel Hyon. An introduction to pyMarmote and pyMarmoteMDP for Markovian modeling - A tutorial. Doctoral. Venezia, Italy. 2024. [https://hal-04747584](https://hal-04747584)

HAL Id: hal-04747584

https://inria.hal.science/hal-04747584v1

Submitted on 22 Oct 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
An introduction to pyMarmote and pyMarmoteMDP
for Markovian modeling
A tutorial
2024 ACM SIGMETRICS / IFIP Performance

Lecturers: Alain Jean-Marie and Emmanuel Hyon
Inria
Univ. Paris Nanterre

Venezia, 10 June 2024
# In a nutshell

## Objective of this tutorial :

* present the capabilities of the Marmote library via its Python interface : `pyMarmote/ pyMarmoteMDP`

* `pyMarmote` : constructing Markov chains on discrete state spaces, determining their structure, computing various distributions

* `pyMarmoteMDP` : constructing Markov Decision Processes with discrete state and action spaces, computing the optimal policy for various criteria.
# Tutorial outline

Tutorial files (and more) are available at :

https:

//marmote.gitlabpages.inria.fr/marmote/python_downloads.html

Their contents is as follows :

<table><thead><tr><th>File</th><th>Topic</th></tr></thead><tbody><tr><td>Lesson0.pdf</td><td>Introduction to the world of Marmote</td></tr><tr><td colspan="2">pyMarmote<br>Markov Chains and their manipulation (A. Jean-Marie)</td></tr><tr><td>Lesson1.ipynb</td><td>Making Markov Chains</td></tr><tr><td>Lesson2.ipynb</td><td>Solving Markov Chains</td></tr><tr><td>Lesson3.ipynb</td><td>Working with state spaces</td></tr><tr><td>Lesson4.ipynb</td><td>Working with distributions</td></tr><tr><td>Lesson5.ipynb</td><td>Predefined Markov Chains</td></tr></tbody></table>
## Tutorial outline (ctd)

<table><thead><tr><th>File</th><th>Topic</th></tr></thead><tbody><tr><td>pyMarmoteMDP<br>Markov Decision Processes and their manipulation (E. Hyon)</td><td></td></tr><tr><td>MDP_Lesson1.ipynb</td><td>Discounted MDP</td></tr><tr><td>MDP_Lesson2.ipynb</td><td>Additional methods</td></tr><tr><td>MDP_Lesson3.ipynb</td><td>Average MDP and policy handling (*)</td></tr><tr><td>MDP_Lesson4.ipynb</td><td>Total Reward MDP with two-D state space (*)</td></tr><tr><td>An application (A. Jean-Marie and E. Hyon)</td><td></td></tr><tr><td>App_Lesson1.ipynb</td><td>The hysteresis queue / performance</td></tr><tr><td>App_Lesson2.ipynb</td><td>A tandem multi server system / optimal control</td></tr></tbody></table>
# If you want to test

## By installing pyMarmote

If you want to test while we talk (or after)

https://marmote.gitlabpages.inria.fr/marmote/python_downloads.html

Two possibilities :

* by installing pyMarmote and before it by installing conda;

* by using Google Colab. [Preferred way]

By installing marmote, you can

* Run the notebooks. download the all_notbooks.zip file on your own computer, extract and run the notebooks with jupyter.

* Run the commented python files issued from the notebooks, download the alls.zip file on your own computer and run them with python.
## If you want to test
Without installing pyMarmote

If you want to test while we talk (or after) **and** you have a Google account : you can use Google Colab.

* go to the web page above (Python downloads)

* download the all_notebooks_collab.zip file on your own computer

* open Google colab https://colab.google/ with your google account

* upload the notebook that you want to use (or all notebooks at once)

* execute the downloaded notebook. Notice that running the first cells (which installs the software) can take up to 4 minutes.

Instructions, examples (C++ and Python) and documentation :

https://marmote.gitlabpages.inria.fr/marmote
# Why Marmote? What can you expect from it?

At the root of Marmote, the observation that :

* Markov modelers (especially students!) often develop their Markov software from scratch: modeling and solution algorithms (including simulation)

* Even basic from-the-book solution algorithms are not readily available

* Solving (numerically) the models is not even the hardest part: setting up the model (matrix) without bugs is often a headache

* Not even the solution to standard models is available.

We hope Marmote will fill these gaps and explain how in various lessons.
## Pedagogical objectives

The focus is not just on numerical solution techniques.

We also want to address the difficulties encountered when passing from the theoretical model to computer code.

We would like to highlight Marmote's ability to address these issues.

Namely

* abstraction / object hierarchy

* modeling Markov chains with (complex) sets, transitions

* the MarmoteMDP package for MDP modeling
# Caveats

## Words of caution :

* The Python interfaces `pyMarmote` and `pyMarmoteMDP` are recent developments : bugs and crashes are still possible !

* Bug reports welcome !

* Suggestions for improvements welcome !

* Test cases/Challenges welcome !