<figure><img src="image_1.png" /></figure>

HAL
open science

# The Morumotto / Sicaqo project

Olivier Geber, Constanza Pardo, Jean-Marie Saurel, Arnaud Lemarchand, Philippe Kowalski, Nicolas Leroy, El-Madani Aissaoui, Claudio Satriano

<figure><img src="image_2.png" /></figure>

## To cite this version:

Olivier Geber, Constanza Pardo, Jean-Marie Saurel, Arnaud Lemarchand, Philippe Kowalski, et al. The Morumotto / Sicaqo project. Rencontres scientifiques et techniques RESIF 2019, Nov 2019, Biarritz, France. hal-02446026

HAL Id: hal-02446026

https://hal.science/hal-02446026

Submitted on 20 Jan 2020

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_3.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_4.png" /></figure>

<figure><img src="image_5.png" /></figure>

## Beta version available

* Source code available on GitHub (https://github.com/IPGP/morumotto)

<figure><img src="image_6.png" /></figure>

## Work flow

### Data update

* Read final archive, create gaps index

* Get source inventories

* Create requests from a pool of sources

* When stack is ready, execute it as multithreaded background task

* Clean data, remove overlaps, then merge it to final archive (see below : request algorithm)

* Update final archive statistics

### Stack management

* Each source has a priority, data coming from higher priority source will be favoured

* Limitations in local or distant CPU and bandwidth can be modified to fit networks specificities

* Tasks are handled either manually or running automatically in a crontab

### Request algorithm

1. Morumotto read gaps in final archive

FOR EACH GAP:

2. Morumotto calls plugins

2.1 plugins fetches data corresponding to the gap (patch) from several sources and copy them into the working archive. In a temporary directory linked to the request

3. Morumotto calls set patch(s)

3.1 set patch(s) copies files concerned with the gap from the final archive to the working archive, then sorts them to remove potential data between gap start and end (in case several gaps have been merged into a bigger one)

3.2 set patch(s) merges merged data, given more importance to higher priority source data, into a final patch that fits on sample level (the gap)

3.3 Merge final patch to temporary file(s) In case of failure: delete temporary data (unless debug mode is ON), request status is set to **Sabado** or **Fondo**

4. Morumotto calls merge, set, merge

4.1 merge to final components temporary file(s) with the corresponding file(s) in final archive and copy the new gap nor data compression problem has been introduced

4.3 merge to final overwrites final archive file(s) with the temporary file(s)

In case of failure : delete temporary data (unless debug mode is ON), request status is set to **Sabado**

5. Morumotto clears requests

<figure><img src="image_7.png" /></figure>

### Quality control

* Check metadata

* Check data quality

* Check waveforms vs metadata

* Check station location

* Plot instrument response

* Plot data completion

<figure><img src="image_8.png" /></figure>

## Interface

### Web interface

* Morumotto is running on a server, users can access it with a classical web browser

* Administration interface for configuration

* Different user levels : normal, advanced and admin

* Normal user can monitor network gaps, overlaps and perform quality control (not affecting data)

* Advanced user can create data requests, change update settings and statistics

* Admin can configure, add & delete requests, statistics, users, etc.

<figure><img src="image_9.png" /></figure>

Monitoring interface : daily gaps per component view

Update Archive

Stack

Crontab

Configuration

<figure><img src="image_10.png" /></figure>

Administration panel : source plugin configuration

<figure><img src="image_11.png" /></figure>

QC interface : station location view

## Software

### Design

* Object oriented

* Modular:

* New plugin integration is easy

* New data & metadata formats can be added

* Works indifferently from data format & structure

* Currently supports miniSEED, dataless SEED and stationXML (FDSN)

* For evenly sampled generic timeseries including-but not only-seismic data

### Implementation

* Built with Djang framework, which includes a template system for the frontend and an ORM for the backend and database management

* Supported RDBMS: MySQL, PostgreSQL

* Algorithms for request creation and stack management are in Python, data quality control and data processing are done using the Obpy library

* Frontend includes several Javascript libraries (Cal-heatmap, Datatables, Leaflet...)

* Multithreading and background tasks is done with Celery, which has a specific djang plugin

* Stack monitoring is done with Flower, a Celery monitoring tool

* Source plugin scripts are written in Shell

### Daemon, crontab, configuration

* The software is deamonized, so that it runs on the startup of the system (using supervisor)

* It currently uses the crontab to execute according to a frequency defined by the user

* install.sh script provided, tested in Debian 9 and Ubuntu > 16.04

* First use of the software comes with an initialisation wizard

### License

* This program is free and open source, under the GNU General Public License version 3

## Command line interface

All data processing & configuration can be done by command line. Few examples :

* Create a request manually :

```
$ python manage.py create_request starttime 2018-04-22T06:00:00Z endtime
2018-04-22T07:00:00Z nslc_list PF.PRO.00.HH PF.RER.90.HHZ --source_list FDSNMIPG
LOCAL_DIR_HOME
```

* Execute stack manually

```
$ python manage.py exec_stack
```

* Update data for a specific window of time

```
$ python manage.py window_update --window_starttime 2018-04-22T00:00:00Z window_endtime
2018-04-28T00:00:00Z
```

## Plugins

* New plugins can be easily integrated into the software

* FDSN Webservice plugin code :

<figure><img src="image_12.png" /></figure>

* On the left : I/O (always the same),

* On the right : the part that needs to be modified to fetch data from a new source

* Then create a new class in source.py :

<figure><img src="image_13.png" /></figure>

## Development

### Calendar

* Functional specification & design dossier shared with RESIF community in April & September 2018

* Available on github :

* github.com/IPGP/sigaco/blob/master/CDC_ValidationDonnees.pdf

* github.com/IPGP/sigaco/blob/master/SiQaCo-Dossier-de-conception.pdf

* Development from Sep. 2018, involving RESIF, volcano observatories from IPGP community & Geoscope

### Perspectives

* (*) Release candidate version will be pushed to the RESIF GitHub

* Source plugins currently under development : Centaur, Q330, NAQS

* Quality Control currently under development

* Installation & testing in observatories & RESIF

* API & Documentation in progress