Package: openPMD-api
C++ & Python API for Scientific I/O with openPMDopenPMD is an open meta-data schema that provides meaning and self-description for data sets in science and engineering.
Seethe openPMD standardfor details of this schema.This library provides a reference API for openPMD data handling.
Since openPMD is a schema (or markup) on top of portable, hierarchical file formats, this library implements various backends such as HDF5, ADIOS1, ADIOS2 and JSON.
Writing & reading through those backends and their associated files are supported for serial andMPI-parallelworkflows.UsageC++#include<openPMD/openPMD.hpp>#include<iostream>// ...autos=openPMD::Series("samples/git-sample/data%T.h5",openPMD::Access::READ_ONLY);for(autoconst&[step,it]:s.iterations){std::cout<<"Iteration: "<<step<<"\n";for(autoconst&[name,mesh]:it.meshes){std::cout<<"  Mesh '"<<name<<"' attributes:\n";for(autoconst&val:mesh.attributes())std::cout<<"    "<<val<<'\n';}for(autoconst&[name,species]:it.particles){std::cout<<"  Particle species '"<<name<<"' attributes:\n";for(autoconst&val:species.attributes())std::cout<<"    "<<val<<'\n';}}Pythonimportopenpmd_apiasio# ...series=io.Series("samples/git-sample/data%T.h5",io.Access.read_only)fork_i,iinseries.iterations.items():print("Iteration:{0}".format(k_i))fork_m,mini.meshes.items():print("  Mesh '{0}' attributes:".format(k_m))forainm.attributes:print("{0}".format(a))fork_p,pini.particles.items():print("  Particle species '{0}' attributes:".format(k_p))forainp.attributes:print("{0}".format(a))More!Curious?
Our manual shows fullread & write examples, both serial and MPI-parallel!DependenciesRequired:CMake 3.15.0+C++17 capable compiler, e.g., g++ 7+, clang 7+, MSVC 19.15+, icpc 19+, icpxShipped internally inshare/openPMD/thirdParty/:Catch22.13.10+ (BSL-1.0)pybind112.10.1+ (new BSD)NLohmann-JSON3.9.1+ (MIT)toml113.7.1+ (MIT)I/O backends:JSONHDF51.8.13+ (optional)ADIOS11.13.1+ (optional, deprecated)ADIOS22.7.0+ (optional)while those can be built either with or without:MPI 2.1+, e.g. OpenMPI 1.6.5+ or MPICH2Optional language bindings:Python:Python 3.8 - 3.11pybind11 2.10.1+numpy 1.15+mpi4py 2.1+ (optional, for MPI)pandas 1.0+ (optional, for dataframes)dask 2021+ (optional, for dask dataframes)CUDA C++ (optional, currently used only in tests)InstallationOur community loves to help each other.
Pleasereport installation problemsin case you should get stuck.Chooseoneof the install methods below to get started:Spack# optional:               +python +adios1 -adios2 -hdf5 -mpispackinstallopenpmd-api
spackloadopenpmd-apiConda# optional:                      OpenMPI support  =*=mpi_openmpi*# optional:                        MPICH support  =*=mpi_mpich*condacreate-nopenpmd-cconda-forgeopenpmd-api
condaactivateopenpmdBrewbrewtapopenpmd/openpmd
brewinstallopenpmd-apiPyPIOn very old macOS versions (<10.9) or on exotic processor architectures, this install methodcompiles from sourceagainst the found installations of HDF5, ADIOS1, ADIOS2, and/or MPI (in system paths, from other package managers, or loaded via a module system, ...).# we need pip 19 or newer# optional:                   --userpython3-mpipinstall-Upip# optional:                        --userpython3-mpipinstallopenpmd-apiIf MPI-support shall be enabled, we always have to recompile:# optional:                                    --userpython3-mpipinstall-Upipsetuptoolswheel
python3-mpipinstall-Ucmake# optional:                                                                   --useropenPMD_USE_MPI=ONpython3-mpipinstallopenpmd-api--no-binaryopenpmd-apiFor some exotic architectures and compilers, you might need to disable a compiler feature calledlink-time/interprocedural optimizationif you encounter linking problems:exportCMAKE_INTERPROCEDURAL_OPTIMIZATION=OFF# optional:                                                --userpython3-mpipinstallopenpmd-api--no-binaryopenpmd-apiAdditional CMake options can be passed via individual environment variables, which need to be prefixed withopenPMD_CMAKE_.From SourceopenPMD-api can also be built and installed from source usingCMake:gitclonehttps://github.com/openPMD/openPMD-api.git

mkdiropenPMD-api-buildcdopenPMD-api-build# optional: for full tests, with unzip../openPMD-api/share/openPMD/download_samples.sh# for own install prefix append:#   -DCMAKE_INSTALL_PREFIX=$HOME/somepath# for options append:#   -DopenPMD_USE_...=...# e.g. for python support add:#   -DopenPMD_USE_PYTHON=ON -DPython_EXECUTABLE=$(which python3)cmake../openPMD-api

cmake--build.# optionalctest# sudo might be required for system pathscmake--build.--targetinstallThe following options can be added to thecmakecall to control features.
CMake controls options with prefixed-D, e.g.-DopenPMD_USE_MPI=OFF:CMake OptionValuesDescriptionopenPMD_USE_MPIAUTO/ON/OFFParallel, Multi-Node I/O for clustersopenPMD_USE_HDF5AUTO/ON/OFFHDF5 backend (.h5files)openPMD_USE_ADIOS1AUTO/ON/OFFADIOS1 backend (.bpfiles up to version BP3) - deprecatedopenPMD_USE_ADIOS2AUTO/ON/OFFADIOS2 backend (.bpfiles in BP3, BP4 or higher)openPMD_USE_PYTHONAUTO/ON/OFFEnable Python bindingsopenPMD_USE_INVASIVE_TESTSON/OFFEnable unit tests that modify source code1openPMD_USE_VERIFYON/OFFEnable internal VERIFY (assert) macro independent of build type2openPMD_INSTALLON/OFFAdd installation targetsopenPMD_INSTALL_RPATHON/OFFAdd RPATHs to installed binariesPython_EXECUTABLE(newest found)Path to Python executable1e.g. changes C++ visibility keywords, breaks MSVC2this includes most pre-/post-condition checks, disabling without specific cause is highly discouragedAdditionally, the following libraries are shipped internally.
The following options allow to switch to external installs:CMake OptionValuesLibraryVersionopenPMD_USE_INTERNAL_CATCHON/OFFCatch22.13.10+openPMD_USE_INTERNAL_PYBIND11ON/OFFpybind112.10.1+openPMD_USE_INTERNAL_JSONON/OFFNLohmann-JSON3.9.1+openPMD_USE_INTERNAL_TOML11ON/OFFtoml113.7.1+By default, this will build as a shared library (libopenPMD.[so|dylib|dll]) and installs also its headers.
In order to build a static library, append-DBUILD_SHARED_LIBS=OFFto thecmakecommand.
You can only build a static or a shared library at a time.By default, theReleaseversion is built.
In order to build with debug symbols, pass-DCMAKE_BUILD_TYPE=Debugto yourcmakecommand.By default, tests, examples and command line tools are built.
In order to skip building those, passOFFto thesecmakeoptions:CMake OptionValuesDescriptionopenPMD_BUILD_TESTINGON/OFFBuild testsopenPMD_BUILD_EXAMPLESON/OFFBuild examplesopenPMD_BUILD_CLI_TOOLSON/OFFBuild command-line toolsopenPMD_USE_CUDA_EXAMPLESON/OFFUse CUDA in examplesLinking to your projectThe install will contain header files and libraries in the path set with-DCMAKE_INSTALL_PREFIX.CMakeIf your project is using CMake for its build, one can conveniently use our providedopenPMDConfig.cmakepackage, which is installed alongside the library.First set the following environment hint if openPMD-api wasnotinstalled in a system path:# optional: only needed if installed outside of system pathsexportCMAKE_PREFIX_PATH=$HOME/somepath:$CMAKE_PREFIX_PATHUse the following lines in your project'sCMakeLists.txt:# supports:                       COMPONENTS MPI NOMPI HDF5 ADIOS1 ADIOS2find_package(openPMD0.15.0CONFIG)if(openPMD_FOUND)target_link_libraries(YourTargetPRIVATEopenPMD::openPMD)endif()Alternatively, add the openPMD-api repository source directly to your project and use it via:add_subdirectory("path/to/source/of/openPMD-api")target_link_libraries(YourTargetPRIVATEopenPMD::openPMD)For development workflows, you can even automatically download and build openPMD-api from within a depending CMake project.
Just replace theadd_subdirectorycall with:include(FetchContent)set(CMAKE_POLICY_DEFAULT_CMP0077NEW)set(openPMD_BUILD_CLI_TOOLSOFF)set(openPMD_BUILD_EXAMPLESOFF)set(openPMD_BUILD_TESTINGOFF)set(openPMD_BUILD_SHARED_LIBSOFF)# precedence over BUILD_SHARED_LIBS if neededset(openPMD_INSTALLOFF)# or instead use:# set(openPMD_INSTALL ${BUILD_SHARED_LIBS})  # only install if used as a shared libraryset(openPMD_USE_PYTHONOFF)FetchContent_Declare(openPMDGIT_REPOSITORY"https://github.com/openPMD/openPMD-api.git"GIT_TAG"0.15.0")FetchContent_MakeAvailable(openPMD)ManuallyIf your (Linux/OSX) project is build by calling the compiler directly or uses a manually writtenMakefile, consider using ouropenPMD.pchelper file forpkg-config, which are installed alongside the library.First set the following environment hint if openPMD-api wasnotinstalled in a system path:# optional: only needed if installed outside of system pathsexportPKG_CONFIG_PATH=$HOME/somepath/lib/pkgconfig:$PKG_CONFIG_PATHAdditional linker and compiler flags for your project are available via:# switch to check if openPMD-api was build as static library# (via BUILD_SHARED_LIBS=OFF) or as shared library (default)if["$(pkg-config--variable=staticopenPMD)"=="true"]thenpkg-config--libs--staticopenPMD# -L/usr/local/lib -L/usr/lib/x86_64-linux-gnu/openmpi/lib -lopenPMD -pthread /usr/lib/libmpi.so -pthread /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi_cxx.so /usr/lib/libmpi.so /usr/lib/x86_64-linux-gnu/hdf5/openmpi/libhdf5.so /usr/lib/x86_64-linux-gnu/libsz.so /usr/lib/x86_64-linux-gnu/libz.so /usr/lib/x86_64-linux-gnu/libdl.so /usr/lib/x86_64-linux-gnu/libm.so -pthread /usr/lib/libmpi.so -pthread /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi_cxx.so /usr/lib/libmpi.soelsepkg-config--libsopenPMD# -L${HOME}/somepath/lib -lopenPMDfipkg-config--cflagsopenPMD# -I${HOME}/somepath/includeAuthor ContributionsopenPMD-api is developed by many people.
It was initially started by theComputational Radiation Physics GroupatHZDRas successor tolibSplash, generalizing thesuccessful HDF5 & ADIOS1 implementationsinPIConGPU.
The following people and institutionscontributedto openPMD-api:Axel Huebl (HZDR, now LBNL):
project lead, releases, documentation, automated CI/CD, Python bindings, Dask, installation & packaging, prior reference implementationsFranz Poeschel (CASUS):
JSON & ADIOS2 backend, data staging/streaming, reworked class designFabian Koller (HZDR):
initial library design and implementation with HDF5 & ADIOS1 backendJunmin Gu (LBNL):
non-collective parallel I/O fixes, ADIOS improvements, benchmarksFurther thanks go to improvements and contributions from:Carsten Fortmann-Grote (EU XFEL GmbH, now MPI-EvolBio):
draft of our Python unit testsDominik Stańczak (Warsaw University of Technology):
documentation improvementsRay Donnelly (Anaconda, Inc.):
support on conda packaging and libc++ quirksJames Amundson (FNAL):
compile fix for newer compilersRené Widera (HZDR):
design improvements for initial API designErik Zenker (HZDR):
design improvements for initial API designSergei Bastrakov (HZDR):
documentation improvements (windows)Rémi Lehe (LBNL):
package integration testing on macOS and LinuxLígia Diana Amorim (LBNL):
package integration testing on macOSKseniia Bastrakova (HZDR):
compatibility testingRichard Pausch (HZDR):
compatibility testing, documentation improvementsPaweł Ordyna (HZDR):
report on NVCC warningsDmitry Ganyushin (ORNL):
Dask prototyping & ADIOS2 benchmarkingJohn Kirkham (NVIDIA):
Dask guidance & reviewsErik Schnetter (PITP):
C++ API bug fixesJean Luca Bez (LBNL):
HDF5 performance tuningBernhard Manfred Gruber (CERN):
CMake fix for parallel HDF5Nils Schild (IPP):
CMake improvements for subprojectsGrantsThe openPMD-api authors acknowledge support via the following programs.
Supported by the CAMPA collaboration, a project of the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research and Office of High Energy Physics, Scientific Discovery through Advanced Computing (SciDAC) program.
Previously supported by the Consortium for Advanced Modeling of Particles Accelerators (CAMPA), funded by the U.S. DOE Office of Science under Contract No. DE-AC02-05CH11231.
Supported by the Exascale Computing Project (17-SC-20-SC), a collaborative effort of two U.S. Department of Energy organizations (Office of Science and the National Nuclear Security Administration).
This project has received funding from the European Unions Horizon 2020 research and innovation programme under grant agreement No 654220.
This work was partially funded by the Center of Advanced Systems Understanding (CASUS), which is financed by Germany's Federal Ministry of Education and Research (BMBF) and by the Saxon Ministry for Science, Culture and Tourism (SMWK) with tax funds on the basis of the budget approved by the Saxon State Parliament.Transitive ContributionsopenPMD-api stands on the shoulders of giants and we are grateful for the following projects included as direct dependencies:ADIOS1andADIOS2byS. Klasky (ORNL), team, collaboratorsandcontributorsCatch2byPhil Nash,Martin HořeňovskýandcontributorsHDF5 bythe HDF groupand communityjsonbyNiels Lohmannandcontributorstoml11byToru Niinaandcontributorspybind11byWenzel Jakob (EPFL)andcontributorsall contributors to the evolution of modern C++ and early library preview developers, e.g.Michael Park (Facebook)theCMake build systemandcontributorspackaging support by theconda-forge,PyPIandSpackcommunities, among otherstheopenPMD-standardbyAxel Huebl (HZDR, now LBNL)andcontributors