Package: desy-mento
MENTOis a Python (3.7+ only) package for automatically triggering online processing jobs likeremote data analysis, andlocal visualization tasks,right from the beamline.The package contains modules and utility functions that automate the boring bits likegetting access to remote HPC resourcessubmitting slurm jobs to run online analysisThe package is designed to work on all PETRA III beamlines at DESY.Here is a short YouTube videoabout the main uses ofMENTO,
from a talk given at NOBUGS 2022.UsageIn its simplest form,MENTOcan be used at a PETRA-III beamline to launch a data processing job on a remote
HPC (Maxwell) node, without needing to know how to connect to the HPC cluster Maxwell, or how to authenticate
without having a scientific account for Maxwell, or indeed how job submissions work.A remote analysis program can be started on data acquired at the beamline, by a Python script importing aMENTOmodule, creating aMENTOTriggerobject and callingrun()on it.Here is a minimum working example:from mento import Trigger

my_trigger = Trigger()  # default trigger will submit Slurm jobs on an HPC node

my_analysis = 'maxwell_program.exe'  # path to analysis program available on Maxwell

my_args = ['analysis', 'parameters', 'list', 'inputdatafile']  # arguments required by analysis program, including input data

my_trigger.run([my_analysis, my_args])  # connect to a remote machine with appropriate credentials and start the analysis job thereNote that paths to input data files and output directories can be specified as you see them at
the beamline, and they will be automagically found in the remote core filesystem on Maxwell.Paths to the analysis program on Maxwell, will of course need to refer to the ‘remote’ paths
as seen from a Maxwell node, sinceMENTOcannot know about them from the beamline.A dummy example of runningMENTOto do live data analysis during data acquisition is shown inexample.py.Prerequisites for usingMENTOfor online data analysisWhen using Maxwell to perform the data analysis,MENTOuses Maxwell’s Slurm system that manages access to
the compute nodes and schedules processing jobs according to the permissions and resources granted to the
user requesting the processing.
Depending on the resources allocated to the user runningMENTO, processing jobs may be queued on Maxwell,
and run at a later time (or never).Therefore, to have the data processing jobs run live at the same time as data acquisition during a beamtime,
the recommendation is to request ‘online’ resources on Maxwell at the time of starting the beamtime.
This is explained in detail in the DESY-IT/ASAP3 documentation
for starting beamtimeshere.Here is the short version: when starting your beamtime, use the following command to request an ‘online’ Maxwell
compute node whichMENTOwill then use to run remote data processing jobs:startBeamtime --beamtimeId 999999999 --beamline P9999 --online(Replace the beamline and beamtime ID with your beamline and beamtime ID, of course).If your data processing program needs a GPU, please also add--featuregputo the startBeamtime command.To request ‘online’ Maxwell resources for commissioning runs instead of user beamtimes, please usestartComissioningwith the same arguments as forstartBeamtime(--onlineand--feature).CustomizationsTheMENTOtrigger can be customized based on the beamline’s needs. An example:Online results visualizationFor online visualization of results, we need to automatically run plotting programslocally.This can be quickly achieved by created a customized trigger usingmy_trigger = Trigger(TriggerMethod.LOCAL_BASH_SCRIPT).Callingrun()on this trigger would then launch a visualization program (either directly or
wrapped inside a Bash script) locally.Of course, the input arguments and the parameter list would also need to be appropriately modified.InstallationAt PETRA III beamlines, the easiest way to install theMENTOpackage would be to usepip.
The package can be downloaded as an artifact from the Gitlab repository page, or used directly
in yourpipcommand as follows:python3 -m pip install desy-mentoAPI documentationAPI documentation can be found indocs.The documentation is auto-generated, so it should be
as up-to-date as the comments in the source code. ;)ContributingContributions in all forms are welcome - be it issues, bug reports, or code!CitingIf you usedMENTOand found it relevant to the work you are publishing, please consider citing theMENTOintroduction article that appears in theSRI 2021 proceedings.You can choose to either use the BibTeX citation given below,
or use theCITATION.cfffile to export the citation to your favourite format (BibTeX/APA/RIS/EndNote/CodeMeta).@article{Vijay_Kartik_2022,
doi = {10.1088/1742-6596/2380/1/012104},
url = {https://dx.doi.org/10.1088/1742-6596/2380/1/012104},
year = {2022},
month = {dec},
publisher = {IOP Publishing},
volume = {2380},
number = {1},
pages = {012104},
author = {S Vijay Kartik and Michael Sprung and Fabian Westermeier and Anton Barty},
title = {MENTO: Automated near real-time data analysis at PETRA III},
journal = {Journal of Physics: Conference Series},
}LicencesAll code is licensed under GPL-3.0-or-laterAll documentation is licensed under CC-BY-4.0All other files are licensed under CC0-1.0Full texts of the licences can be found inLICENSES.This project aims to beREUSEcompliant.ContactFor questions and critiques, please contact S. Vijay Kartik <vijay.kartik@desy.de>.