Managing the Scientific Software Ecosystem with Spack Todd Gamblin*, Gregory Becker*, Kiel Friedt*, Alicia Klinvex†, Gregory L. Lee*, Matthew LeGendre*, Peter Scheibel*, Barry Smith‡, James Willenbring† *Lawrence Livermore National Laboratory †Sandia National Laboratories ‡Argonne National Laboratory Combinatorial software complexity Large scientific codes can be extremely complex ARES tcl tk scipy python cmake hpdf opclient boost zlib numpy bzip2 LAPACK gsl HDF5 gperftools papi GA bdivxml sgeos_xml Scallop rng perﬂib memusage timers Silo SAMRAI HYPRE matprop overlink qd LEOS MSlib Laser CRETIN tdf Cheetah DSD Teton Nuclear ASCLaser MPI ncurses sqlite readline openssl BLAS Physics Utility Math External Types of Packages Developers build 4 versions of ARES 36 different ways. Linux BG/Q Cray XE6 MVAPICH MVAPICH2 OpenMPI BG/Q MPI Cray MPI GCC C P L D C P L D Intel 14 C P L D Intel 15 C P L D D PGI D C P L D C L D Clang C P L D C L D XL C P L D Dependency graph for one configuration of ARES, an LLNL multi-physics code. (L)ite (D)evelopment spack/opt/ linux-x86_64/ gcc-4.7.2/ mpileaks-1.1-0f54bf34cadk/ intel-14.1/ hdf5-1.8.15-lkf14aq3nqiz/ bgq/ xl-12.1/ hdf5-1-8.16-fqb3a15abrwx/ ... mpileaks mpi callpath dyninst libdwarf libelf Hash Spack can install arbitrarily many configurations of each package https://spack.io github/LLNL/spack @spackpm Spack has a rapidly growing open source community Spec syntax lets users build on demand –Build scientific software from source for best performance & portability across exotic platforms –Automate deployment for app developers, facility staff, and users –Provide integration testing harness for large software stacks. –No more manual builds: spack install <package> Done. –Build a community; leverage build automation work across facilities & teams Spack is a flexible, build-from-source package manager for scientific software that supports multiple versions, compilers, and configurations. Overview and Goals $ spack install mpileaks unconstrained $ spack install mpileaks@3.3 @ custom version $ spack install mpileaks@3.3 %gcc@4.7.3 % custom compiler $ spack install mpileaks@3.3 %gcc@4.7.3 +threads +/- build option $ spack install mpileaks@3.3 cppflags=‘-O3’ setting compiler flags $ spack install mpileaks@3.3 os=CNL10 target=haswell setting target for cross-compile $ spack install mpileaks@3.3 ^mpich@3.2 %gcc@4.9.3 ^ constraints on dependencies `spack fiind` shows what is installed. $ spack find ==> 103 installed packages. -- linux-x86_64 / gcc@4.4.7 -------------------------------- ImageMagick@6.8.9-10 glib@2.42.1 libtiff@4.0.3 pango@1.36.8 qt@4.8.6 SAMRAI@3.9.1 graphlib@2.0.0 libtool@2.4.2 parmetis@4.0.3 qt@5.4.0 adept-utils@1.0 gtkplus@2.24.25 libxcb@1.11 pixman@0.32.6 ravel@1.0.0 atk@2.14.0 harfbuzz@0.9.37 libxml2@2.9.2 py-dateutil@2.4.0 readline@6.3 boost@1.55.0 hdf5@1.8.13 llvm@3.0 py-ipython@2.3.1 scotch@6.0.3 callpath@1.0.2 jpeg@9a mpich@3.0.4 py-numpy@1.9.1 stat@2.1.0 dyninst@8.1.2 libdwarf@20130729 ncurses@5.9 py-pytz@2014.10 xz@5.2.0 -- linux-x86_64 / gcc@4.8.2 -------------------------------- adept-utils@1.0.1 boost@1.55.0 cmake@5.6-special libdwarf@20130729 mpich@3.0.4 adept-utils@1.0.1 cmake@5.6 dyninst@8.1.2 libelf@0.8.13 openmpi@1.8.2 -- linux-x86_64 / intel@14.0.2 ----------------------------- hwloc@1.9 mpich@3.0.4 starpu@1.1.4 -- linux-x86_64 / intel@15.0.0 ----------------------------- adept-utils@1.0.1 boost@1.55.0 libdwarf@20130729 libelf@0.8.13 mpich@3.0.4 -- linux-x86_64 / intel@15.0.1 ----------------------------- adept-utils@1.0.1 callpath@1.0.2 libdwarf@20130729 mpich@3.0.4 boost@1.55.0 hwloc@1.9 libelf@0.8.13 starpu@1.1.4 $ spack find callpath ==> 2 installed packages. -- linux-x86_64 / clang@3.4 ———————— -- linux-x86_64 / gcc@4.9.2 ------------- callpath@1.0.2 callpath@1.0.2 $ spack find -dl callpath ==> 2 installed packages. -- linux-x86_64 / clang@3.4 ----------- -- linux-x86_64 / gcc@4.9.2 --------- -- xv2clz2 callpath@1.0.2 udltshs callpath@1.0.2 ckjazss ^adept-utils@1.0