The	
  Spack	
  Package	  Manager:	  Bringing	
  Order	
  to	
  HPC	  So@ware	
  Chaos	  Supercompu)ng	
  2015	
  (SC15)	
  Aus)n,	
  Texas	
  November 18, 2015

LLNL-PRES-803375
This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory
under contract DE-AC52-07NA27344. Lawrence Livermore National Security, LLC

Todd	
  Gamblin,	
  Ma@hew	
  LeGendre,	
  Michael	
  R.	
  Colle@e,	
  Gregory	
  L.	
  Lee,	
  Adam	  Moody,	
  Bronis	
  R.	
  de	
  Supinski,	
  and	
  Sco@	  Futral	  h"p://bit.ly/spack-­‐git	
  What	  is	
  the	
  producDon	
  environment	
  for	
  HPC?	
  § Someone’s	
  home	  directory?	
  § LLNL?	
  LANL?	
  Sandia?	
  ANL?	
  LBL?	
  TACC?	
  — Environments	
  at	
  large-­‐scale	
  sites	  are	  very	  diﬀerent.	
  § Which	
  MPI	
  implementa)on?	
  § Which	
  compiler?	
  § Which	
  dependencies?	
  § Which	
  versions	  of	
  dependencies?	
  — Many	  applica)ons	
  require	
  speciﬁc	
  dependency	
  versions.	
  Real answer: there isn’t a single production environment or a standard way to build.
LLNL-PRES-803375

h"p://bit.ly/spack-­‐git	
  

2	
  

HPC	
  so@ware	
  is	
  becoming	
  increasingly	
  complex	  § Not	  much	  standardiza)on	
  in	
  HPC:	
  every	
  machine/app	
  has	  a	
  diﬀerent	
  so^ware	
  stack	  § Sites	
  share	
  unique	
  hardware	
  among	
  teams	
  with	  very	  diﬀerent	
  requirements	  — Users	
  want	  to	
  experiment	
  with	  many	  exo)c	
  architectures,	
  compilers,	
  MPI	  versions	  — All	  of	
  this	  is	
  necessary	
  to	
  get	  the	  best	  performance	  § Example	
  environment	
  for	
  some	  LLNL	
  codes:	  x

48 third party packagesx

Up to 7 compilers
Intel GCC XLC ClangPGI Cray Pathscalex

mvapich

3 MPI versions
mvapich2 OpenMPI

Oh, and 2-3 versions ofeach packagex

3-ish Platforms
Linux BlueGene Cray

= ~7,500 combinations

We want an easy way to quickly sample the space, to build configurations on demand!
LLNL-PRES-803375

h"p://bit.ly/spack-­‐git	
  

3	
  

Most	
  exisDng	
  tools	
  do	
  not	
  support	
  combinatorial	
  versioning	  § Tradi)onal	
  binary	
  package	
  managers	
  — RPM,	
  yum,	
  APT,	
  yast,	
  etc.	
  — Designed	
  to	
  manage	
  a	
  single	
  stack.	
  — Install	
  one	  version	
  of	
  each	  package	
  in	
  a	
  single	
  preﬁx	  (/usr).	
  — Seamless	
  upgrades	
  to	
  a	
  stable,	
  well	  tested	
  stack	  § Port	  systems	  — BSD	  Ports,	
  portage,	
  Macports,	
  Homebrew,	
  Gentoo,	
  etc.	
  — Minimal	
  support	
  for	
  builds	
  parameterized	
  by	
  compilers,	
  dependency	
  versions.	
  § Virtual	
  Machines	
  and	
  Linux	
  Containers	
  (Docker)	
  — Containers	
  allow	  users	
  to	
  build	
  environments	
  for	
  diﬀerent	
  applica)ons.	
  — Does	  not	
  solve	
  the	
  build	
  problem	
  (someone	  has	  to	
  build	
  the	
  image)	
  — Performance,	
  security,	
  and	
  upgrade	
  issues	
  prevent	
  widespread	
  HPC	
  deployment.	
  LLNL-PRES-803375

h"p://bit.ly/spack-­‐git	
  

4	
  

How	
  do	
  HPC	
  sites	
  deal	  with	  combinatorial	
  builds?	
  § HPC	  so^ware	
  is	
  typically	
  installed	
  manually	
  in	
  a	
  directory	
  hierarchy.	
  — Hierarchy	
  o^en	  doesn’t	
  give	  all	
  needed	
  informa)on	
  about	
  a	
  build.	
  — Sites	
  can	
  run	
  out	  of	
  unique	
  directory	
  names	
  quickly.	
  

Site

Naming Convention

LLNL/ usr / global / tools / $arch / $package / $version/ usr / local / tools / $package-$compiler-$build-$version

Oak Ridge/ $arch / $package / $version / $build

TACC/ $compiler-$comp_version / $mpi / $mpi_version / $package / $version

LLNL-PRES-803375

h"p://bit.ly/spack-­‐git	
  

5	
  

Environment	
  modules	
  can	
  help,	
  but	  are	
  hard	  to	
  get	
  right.	
  $ module avail--------------------------- /opt/modules/modulefiles ---------------------------acml-gnu/4.4
intel/12.0
mvapich2-pgi-ofa/1.7
acml-gnu_mp/4.4
intel/13.0
mvapich2-pgi-psm/1.7
acml-intel/4.4
intel/14.0(default)
mvapich2-pgi-shmem/1.7...
$ module load intel/13.0$ module load mvapich2-pgi-shmem/1.7§ Advantages:	  — Swap	  diﬀerent	
  library	  versions	
  dynamically,	
  in	
  a	
  shell.	
  — Abstracts	
  a	
  lot	  of	
  environment	
  complexity	
  from	  the	
  user.	
  § Disadvantages:	  	
  — Users	
  must	  typically	
  remember	
  to	
  load	  the	
  same	  module	  that	  they	  built	
  with.	
  • Easy	  to	
  load	  wrong	
  module	
  and	
  break	  code.	
  — Many	  sites	
  and	
  vendors	
  deploy	
  extremely	
  bri@le,	
  inconsistent	
  modules.	
  — Module	
  systems	
  do	
  not	
  build	
  so^ware;	
  they	  only	  change	
  the	
  environment.	
  LLNL-PRES-803375

h"p://bit.ly/spack-­‐git	
  

6	
  

Spack	  handles	
  combinatorial	
  so@ware	
  complexity.	
  § Each	  unique	
  dependency	
  graph	  is	  a	
  unique	  Dependency	
  DAG	
  conﬁgura.on.	
  mpi
mpileaks

libdwarf
callpathdyninstHash	
  InstallaDon	
  Layout	
  libelfspack/opt/!
linux-x86_64/!
gcc-4.7.2/!
mpileaks-1.1-0f54bf34cadk/!
intel-14.1/!
hdf5-1.8.15-lkf14aq3nqiz/!
bgq/!
xl-12.1/!
hdf5-1-8.16-fqb3a15abrwx/!
...!

LLNL-PRES-803375§ Each	  conﬁgura)on	
  installed	
  in	
  a	
  unique	
  directory.	
  — Conﬁgura)ons	
  of	
  the	
  same	  package	
  can	
  coexist.	
  § Hash	  of	  en)re	
  directed	
  acyclic	
  graph	
  (DAG)	
  is	
  appended	  to	
  each	  preﬁx.	
  	
  § Installed	
  packages	
  automa)cally	
  ﬁnd	
  dependencies	  — Spack	
  embeds	
  RPATHs	
  in	
  binaries.	
  — No	
  need	  to	
  use	  modules	
  or	
  set	
  LD_LIBRARY_PATH	  — Things	
  work	  the	
  way	
  you	
  built	  them	  h"p://bit.ly/spack-­‐git	
  

7	
  

`spack	
  list`	
  shows	
  what	  packages	
  are	  available	  $ spack list==> 243 packages.
activeharmony coreutils
adept-utils
cppcheck
apex
cram
arpackcscopeasciidoc
cube
atk
czmq
atlas
dbus
autoconf
docbook-xml
automaded
doxygenautomake
dri2proto
bear
dtcmp
bib2xhtml
dyninstbinutils
elfutils
bison
extraeboost
exuberant-ctags
bowtie2fish
boxlib
flex
bzip2
flux
cairo
fontconfig
callpath
freetypecblas
gasnet
cgm
gcc
clang
gdk-pixbuf
cloog
geos
cmake
gflagsLLNL-PRES-803375

ghostscript
git
glib
glm
globalglog
gmp
gnutlsgperf
gperftools
graphlib
graphvizgtkplusharfbuzz
hdf5
hwloc
hypre
icu
icu4c
ImageMagick
isl
jdk
jpeg
launchmon
lcms

leveldb
libarchive
libcircle
libdrmlibdwarf
libelflibevent
libffilibgcrypt
libgpg-error
libjpeg-turbo
libjson-c
libmnglibmonitor
libNBClibpciaccesslibpnglibsodium
libtiff
libtoollibunwindlibuuidlibxcblibxml2
libxshmfence

libxslt
llvm
llvm-lld
lmdb
lua
lwgrp
lwm2
matio
memaxes
mesa
metis
Mitos
mpc
mpe2
mpfr
mpibashmpich
mpileaks
mrnet
munge
mustermvapich2
nasm
ncdu
ncurses

netcdfnetgauge
netlib-blas
netlib-lapack
nettleompss
ompt-openmp
opari2
openmpi
openssl
otf
otf2
pango
papi
paraverparaview
parmetis
parpack
pcre
petsc
pidx
pixmanpkg-config
pmgr_collective
postgresql

ppl
protobuf
py-basemap
py-biopython
py-cffipy-cythonpy-dateutil
py-epydoc
py-genders
py-gnuplot
py-h5py
py-ipython
py-libxml2
py-lockfile
py-makopy-matplotlib
py-mock
py-mpi4py
py-mx
py-nosepy-numpy
py-pandas
py-pexpect
py-pilpy-pmwpy-pychecker
py-pycparser
py-pyelftools
py-pygments
py-pylint
py-pyparpy-pyparsing
py-pyqtpy-pysidepy-python-daemon
py-pytzpy-rpy2py-scientificpython
py-scikit-learn
py-scipy
py-setuptools
py-shiboken
py-sippy-sixpy-sphinx
py-sympy
py-virtualenv
py-yapf
pythonqhullh"p://bit.ly/spack-­‐git	
  qtqthreadsR
ravel
readlinerose
ruby
SAMRAI
samtools
scalascascorepscotchscrsilosnappyspindlesqlitestatsundials
swig
task
taskd
tau
tcl
the_silver_searcher

thrifttk
tmux
tmuxinator
trilinos
uncrustify
util-linux
vim
vtk
wget
wx
wxpropgrid
xcb-protoxz
yasm
zeromq
zlib
zsh

8	
  

Spack	  provides	
  a	
  spec	  syntax	
  to	
  describe	
  customized	
  DAG	
  conﬁguraDons	  $ spack install mpileaks

unconstrained!

!
$ spack install mpileaks@3.3@ custom version!

!
$ spack install mpileaks@3.3 %gcc@4.7.3% custom compiler!

!
$ spack install mpileaks@3.3 %gcc@4.7.3 +threads+/- build option!

!
$ spack install mpileaks@3.3 =bgq

= cross-compile!

§ Each	
  expression	
  is	
  a	
  spec	  for	  a	  par)cular	
  conﬁgura)on	  — Each	  clause	
  adds	  a	
  constraint	
  to	
  the	
  spec	  — Constraints	
  are	  op)onal	
  –	
  specify	
  only	  what	  you	
  need.	
  — Customize	
  install	
  on	
  the	
  command	
  line!	
  § Syntax	
  abstracts	
  details	
  in	
  the	
  common	
  case	  — Makes	  parameteriza)on	  by	
  version,	
  compiler,	
  and	  op)ons	
  easy	  when	  necessary	  LLNL-PRES-803375

h"p://bit.ly/spack-­‐git	
  

9	
  

Spack	  Specs	
  can	  constrain	
  versions	  of	
  dependencies	
  mpi
mpileaks

libdwarf
callpathdyninstlibelf$ spack install mpileaks %intel@12.1 ^libelf@0.8.12!

§ Spack	
  ensures	  one	  conﬁgura)on	
  of	
  each	  library	
  per	  DAG	  — Ensures	
  ABI	  consistency.	
  — User	  does	  not	
  need	  to	
  know	
  DAG	
  structure;	
  only	  the	
  dependency	
  names.	
  § Spack	
  can	
  ensure	
  that	  builds	
  use	  the	
  same	  compiler,	
  or	
  you	
  can	
  mix	  — Working	
  on	
  ensuring	
  ABI	
  compa)bility	
  when	  compilers	
  are	
  mixed.	
  LLNL-PRES-803375

h"p://bit.ly/spack-­‐git	
  

10	
  

Spack	  handles	
  ABI-­‐incompaDble,	
  versioned	
  interfaces	
  like	  MPI	
  mpi
mpileaks

libdwarf
callpathdyninstlibelf§ mpi	  is	
  a	
  virtual	
  dependency	  § Install	
  the	
  same	  package	
  built	
  with	  two	
  diﬀerent	
  MPI	
  implementa)ons:	  $ spack install mpileaks ^mvapich@1.9!

$ spack install mpileaks ^openmpi@1.4:!

§ Let	  Spack	
  choose	
  MPI	  version,	
  as	
  long	  as	
  it	
  provides	
  MPI	  2	  interface:	  $ spack install mpileaks ^mpi@2!

LLNL-PRES-803375

h"p://bit.ly/spack-­‐git	
  

11	
  

Spack	  packages	
  are	
  simple	
  Python	
  scripts.	
  from spack import *class Dyninst(Package):"""API for dynamic binary instrumentation.""”Metadata

homepage = "https://paradyn.org"
version('8.2.1', 'abf60b7faabe7a2e’, url="http://www.paradyn.org/release8.2/DyninstAPI-8.2.1.tgz")
version('8.1.2', 'bf03b33375afa66f’, url="http://www.paradyn.org/release8.1.2/DyninstAPI-8.1.2.tgz")
version('8.1.1', 'd1a04e995b7aa709’, url="http://www.paradyn.org/release8.1/DyninstAPI-8.1.1.tgz")

Versions and URLs

depends_on("libelf")
depends_on("libdwarf")
depends_on("boost@1.42:")

Dependencies
Patches, variants (not shown)

def install(self, spec, prefix):
libelf = spec['libelf'].prefix
libdwarf = spec['libdwarf'].prefixwith working_dir('spack-build', create=True):
cmake('..','-DBoost_INCLUDE_DIR=%s' % spec['boost'].prefix.include,'-DBoost_LIBRARY_DIR=%s' % spec['boost'].prefix.lib,'-DBoost_NO_SYSTEM_PATHS=TRUE’*std_cmake_args)
make()
make("install")Commands for installation
Access build config throughthe spec parameter.

@when('@:8.1')
def install(self, spec, prefix):
configure("--prefix=" + prefix)
make()
make("install")LLNL-PRES-803375

h"p://bit.ly/spack-­‐git	
  

12	
  

Dependencies	
  in	
  Spack	
  may	  be	
  opDonal.	
  § The	  user	  can	
  deﬁne	
  named	
  variants:	  variant("python", default=False, “Build with python support”)
depends_on("python", when="+python")§ And	  use	  them	  to	
  install:	  $ spack install vim +python$ spack install vim –python	  § Dependencies	
  may	  be	  op)onal	
  according	  to	
  other	
  condi)ons:	
  e.g.,	
  gcc	
  dependency	
  on	
  mpc	
  from	  4.5	
  on:	  	
  depends_on("mpc", when="@4.5:")§ DAG	
  is	
  not	
  always	  complete	
  before	
  concre)za)on!	
  LLNL-PRES-803375

h"p://bit.ly/spack-­‐git	
  

13	
  

ConcreDzaDon	
  ﬁlls	
  in	
  missing	
  conﬁguraDon	
  details	
  when	
  the	
  user	  is	
  not	
  explicit.	
  mpileaks ^callpath@1.0+debug ^libelf@0.8.11!

User input: abstract spec with some constraints
spec.yaml

Normalize

mpileaks@2.3%gcc@4.7.3=linux-ppc64

mpileaks

callpath@1.0+debugmpi

dyninstcallpath@1.0%gcc@a4.7.3+debug=linux-ppc64

Concretize

libdwarf

libelf@0.8.11

Abstract, normalized specwith some dependencies.
LLNL-PRES-803375

mpich@3.0.4%gcc@4.7.3=linux-ppc64

dyninst@8.1.2%gcc@4.7.3=linux-ppc64

Store

libdwarf@20130729%gcc@4.7.3=linux-ppc64

libelf@0.8.11%gcc@4.7.3=linux-ppc64

Concrete spec is fully constrained
and can be passed to install.

h"p://bit.ly/spack-­‐git	
  

spec:
- mpileaks:
arch: linux-x86_64
compiler:
name: gcc
version: 4.9.2
dependencies:
adept-utils: kszrtkpbzac3ss2ixcjkcorlaybnptp4
callpath: bah5f4h4d2n47mgycej2mtrnrivvxy77
mpich: aa4ar6ifj23yijqmdabeakpejcli72t3
hash: 33hjjhxi7p6gyzn5ptgyes7sghyprujh
variants: {}version: '1.0'- adept-utils:
arch: linux-x86_64
compiler:
name: gcc
version: 4.9.2
dependencies:
boost: teesjv7ehpe5ksspjim5dk43a7qnowlq
mpich: aa4ar6ifj23yijqmdabeakpejcli72t3
hash: kszrtkpbzac3ss2ixcjkcorlaybnptp4
variants: {}version: 1.0.1- boost:arch: linux-x86_64
compiler:
name: gcc
version: 4.9.2
dependencies: {}hash: teesjv7ehpe5ksspjim5dk43a7qnowlq
variants: {}version: 1.59.0...

Detailed provenance is storedwith the installed package14	  ConcreDzaDon	
  algorithm	
  iterates	
  unDl	  the	  DAG	
  does	  not	
  change.	
  Abstract Specs$ spack install
mpileaks@2.3

IntersectConstraints

Resolve
Virtual Deps

Concretize
Parameters
install()Command Line
Package
Files

Site
ConfigUser
ConfigConcrete Spec§ When	
  underspeciﬁed,	
  concre)za)on	
  chooses	
  a	
  value	  based	  on	
  user/site	
  preferences.	
  § Concre)za)on	
  must	  add	  new	
  dependencies	
  in	
  response	
  to	
  constraint	
  updates.	
  § Current	
  algorithm	
  is	
  greedy,	
  will	  not	
  backtrack	
  once	  a	
  decision	
  is	
  made.	
  — Can	
  fail	  to	
  ﬁnd	  a	
  build	
  that	  sa)sﬁes	
  a	
  query,	
  but	
  has	  not	
  happened	
  for	
  current	
  packages.	
  — Really	
  needs	
  a	
  full	  constraint	
  solver	
  (coming	  soon!)	
  LLNL-PRES-803375

h"p://bit.ly/spack-­‐git	
  

15	
  

Spack	  builds	
  each	  package	  in	
  its	
  own	
  compilaDon	
  environment	  Spack	
  Process	  § Forking	
  build	
  process	
  isolates	
  environment	
  for	
  each	  build.	
  do_install()Install dep1

Install dep2…

§ Compiler	
  wrappers	
  add	  include,	
  lib,	
  and	
  RPATH	
  ﬂags	  — Ensure	
  that	  dependencies	
  are	
  found	
  automa)cally	
  Install package

icc	
  

Fork

Build	  	
  Process	
  

icpc	
  

ifort	
  

Set	
  up	
  environment	  CC = spack/env/spack-cc
CXX = spack/env/spack-c++
F77 = spack/env/spack-f77
FC = spack/env/spack-f90

SPACK_CC = /opt/ic-15.1/bin/icc
SPACK_CXX = /opt/ic-15.1/bin/icpc
SPACK_F77 = /opt/ic-15.1/bin/ifort
SPACK_FC = /opt/ic-15.1/bin/ifort

PKG_CONFIG_PATH= ...
CMAKE_PREFIX_PATH = ...
LIBRARY_PATH= ...

PATH = spack/env:$PATH

install()

LLNL-PRES-803375

Compiler	
  wrappers	
  (cc, c++, f77, f90)	
  -I /dep1-prefix/include-L /dep1-prefix/lib-Wl,-rpath=/dep1-prefix/lib

conﬁgure	
  h"p://bit.ly/spack-­‐git	
  

make	
  

make	
  install	
  

16	
  

Build	  automaDon	
  allows	
  tedious	
  work	  to	
  be	
  leveraged.	
  § Spack	
  enables	
  teams	
  to	
  share	
  work.	
  — Archives	
  common	
  library	
  build	
  recipes.	
  — Prevents	
  duplica)on	
  of	
  build	
  eﬀort.	
  — We	
  can	
  share	
  builds	
  among	
  LC,	  code	  teams,	
  and	
  users	  § Patches	
  allow	  rapid	
  deployment	
  of	
  bug	
  ﬁxes	  	
  — App	
  team	  por)ng	
  a	
  library	
  may	  not	
  own	
  its	
  repo.	
  — Library	
  teams	  may	  not	  have	  )me	  to	
  ﬁx	  issues	
  quickly.	
  — Code	  teams	
  can	  ﬁx	  quickly,	
  then	  feed	  back	  changes.	
  § Python	  allowed	
  quick	
  adop)on	
  by	
  code	  teams.	
  — Many	  app	
  developers	
  already	
  know	
  Python	  — Spec	  syntax	
  provides	
  extra	
  expressiveness.	
  LLNL-PRES-803375

Library
Teams

Livermore
ComputingCode Teams

Spack
Users

h"p://bit.ly/spack-­‐git	
  

Spack
Contributors

17	
  

Use	
  Case	  1:	
  Managing	  combinatorial	
  installaDons	  $ spack find==> 103 installed packages.
-- linux-x86_64 / gcc@4.4.7 -------------------------------ImageMagick@6.8.9-10 glib@2.42.1
libtiff@4.0.3
pango@1.36.8
SAMRAI@3.9.1
graphlib@2.0.0
libtool@2.4.2
parmetis@4.0.3
adept-utils@1.0
gtkplus@2.24.25
libxcb@1.11
pixman@0.32.6
atk@2.14.0
harfbuzz@0.9.37
libxml2@2.9.2
py-dateutil@2.4.0
boost@1.55.0
hdf5@1.8.13
llvm@3.0
py-ipython@2.3.1
cairo@1.14.0
icu@54.1
metis@5.1.0
py-nose@1.3.4
callpath@1.0.2
jpeg@9ampich@3.0.4
py-numpy@1.9.1
dyninst@8.1.2
libdwarf@20130729 ncurses@5.9
py-pytz@2014.10
dyninst@8.1.2
libelf@0.8.13
ocr@2015-02-16
py-setuptools@11.3.1
fontconfig@2.11.1
libffi@3.1
openssl@1.0.1h
py-six@1.9.0
freetype@2.5.3
libmng@2.0.2
otf@1.12.5salmon python@2.7.8
gdk-pixbuf@2.31.2
libpng@1.6.16
otf2@1.4
qhull@1.0-- linux-x86_64 / gcc@4.8.2 -------------------------------adept-utils@1.0.1 boost@1.55.0 cmake@5.6-special libdwarf@20130729
adept-utils@1.0.1 cmake@5.6
dyninst@8.1.2
libelf@0.8.13

qt@4.8.6
qt@5.4.0
ravel@1.0.0
readline@6.3
scotch@6.0.3
starpu@1.1.4
stat@2.1.0
xz@5.2.0
zlib@1.2.8

mpich@3.0.4
openmpi@1.8.2-- linux-x86_64 / intel@14.0.2 ----------------------------hwloc@1.9 mpich@3.0.4 starpu@1.1.4-- linux-x86_64 / intel@15.0.0 ----------------------------adept-utils@1.0.1 boost@1.55.0 libdwarf@20130729 libelf@0.8.13§ spack find	
  shows	
  all	  installed	
  conﬁgura)ons	
  — Mul)ple	  versions	
  of	
  

same	
  package	
  are	
  ok.	  § Packages	
  are	  divided	
  by	
  

architecture/compiler.	
  § Spack	
  also	
  generates	
  module	  ﬁles.	
  mpich@3.0.4— Don’t	
  have	  to	
  use	  them.	
  -- linux-x86_64 / intel@15.0.1 ----------------------------adept-utils@1.0.1 callpath@1.0.2 libdwarf@20130729 mpich@3.0.4
boost@1.55.0
hwloc@1.9
libelf@0.8.13
starpu@1.1.4

LLNL-PRES-803375

h"p://bit.ly/spack-­‐git	
  

18	
  

Using	  the	  Spec	  syntax,	
  Spack	
  can	  restrict	
  queries	  $ spack find mpich==> 5 installed packages.
-- linux-x86_64 / gcc@4.4.7 -------------------------------mpich@3.0.4-- linux-x86_64 / gcc@4.8.2 -------------------------------mpich@3.0.4-- linux-x86_64 / intel@14.0.2 ----------------------------mpich@3.0.4§ Querying	
  by	
  package	
  name	  retrieves	
  a	
  subset	  -- linux-x86_64 / intel@15.0.0 ----------------------------mpich@3.0.4-- linux-x86_64 / intel@15.0.1 ----------------------------mpich@3.0.4

LLNL-PRES-803375

h"p://bit.ly/spack-­‐git	
  

19	
  

The	
  Spec	  syntax	
  doubles	
  as	
  a	
  query	
  language	
  to	
  allow	
  reﬁnement	
  of	
  searches.	
  $ spack find libelf==> 5 installed packages.
-- linux-x86_64 / gcc@4.4.7 --------libelf@0.8.12 libelf@0.8.13-- linux-x86_64 / gcc@4.8.2 --------libelf@0.8.13-- linux-x86_64 / intel@15.0.0 -----libelf@0.8.13

Query versions of libelf package
List only those built with Intel compiler.
$ spack find libelf %intel-- linux-x86_64 / intel@15.0.0 -----libelf@0.8.13-- linux-x86_64 / intel@15.0.1 -----libelf@0.8.13-- linux-x86_64 / intel@15.0.1 -----libelf@0.8.13$ spack find libelf %intel@15.0.1-- linux-x86_64 / intel@15.0.1 -----libelf@0.8.13

Restrict to specific compiler version
LLNL-PRES-803375

h"p://bit.ly/spack-­‐git	
  

20	
  

Users	  can	
  query	
  the	
  full	  dependency	
  conﬁguraDon	
  	
  
of	
  installed	
  packages.	
  $ spack find callpath==> 2 installed packages.
-- linux-x86_64 / clang@3.4 ————————
callpath@1.0.2

Expand dependencies
with spack find -d-- linux-x86_64 / gcc@4.9.2 ------------callpath@1.0.2$ spack find -dl callpath==> 2 installed packages.
-- linux-x86_64 / clang@3.4 ----------xv2clz2
callpath@1.0.2
ckjazss^adept-utils@1.0.1
3ws43m4^boost@1.59.0
ft7znm6^mpich@3.1.4
qqnuet3^dyninst@8.2.1
3ws43m4^boost@1.59.0
g65rdud^libdwarf@20130729
cj5p5fk^libelf@0.8.13
cj5p5fk^libelf@0.8.13
g65rdud^libdwarf@20130729
cj5p5fk^libelf@0.8.13
cj5p5fk^libelf@0.8.13
ft7znm6^mpich@3.1.4-- linux-x86_64 / gcc@4.9.2 ----------udltshs
callpath@1.0.2
rfsu7fb^adept-utils@1.0.1
ybet64y^boost@1.55.0
aa4ar6i^mpich@3.1.4
tmnnge5^dyninst@8.2.1
ybet64y^boost@1.55.0
g2mxrl2^libdwarf@20130729
ynpai3j^libelf@0.8.13
ynpai3j^libelf@0.8.13
g2mxrl2^libdwarf@20130729
ynpai3j^libelf@0.8.13
ynpai3j^libelf@0.8.13
aa4ar6i^mpich@3.1.4§ Architecture,	
  compiler,	
  and	
  dependency	
  versions	
  may	
  diﬀer	  between	
  builds.	
  LLNL-PRES-803375

h"p://bit.ly/spack-­‐git	
  

21	
  

Use	
  Case	
  2:	
  Package	  Views	
  for	
  HPC	  Center	
  Installs	
  spack/opt/!
linux-x86_64/!
gcc-4.7.2/!
mpileaks-1.1-0f54bf34cadk/!
intel-14.1/!
hdf5-1.8.15-lkf14aq3nqiz/!
bgq/!
xl-12.1/!
hdf5-1-8.16-fqb3a15abrwx/!
...!

/software/!
linux-x86_64/!
gcc-4.7.2/!
mvapich-1.9/!
mpileaks-1.1/!
intel-14.1/!
mvapich-1.9/!
hdf5-1.8.15/!
bgq/!
xl-12.1/!
ibm-mpi/!
hdf5-1-8.16/!
...!

§ Many	  users	
  like	  to	
  navigate	
  a	
  readable	
  directory	
  hierarchy	  — Spack’s	
  combinatorial	
  package	
  space	  is	
  large	
  and	
  can	  be	
  hard	  to	
  navigate	  § Spack	
  can	
  generate	  a	
  coarser	
  tree	  view	
  of	
  symbolic	
  links	  — View	
  is	
  a	
  projec)on	
  from	  the	
  higher-­‐dimensional	  Spack	  space	  — Some	  names	
  may	  conﬂict,	
  but	
  spec	  syntax	
  allows	
  us	
  to	
  express	
  preferences	
  to	
  guide	
  view	
  crea)on.	
  LLNL-PRES-803375

h"p://bit.ly/spack-­‐git	
  

22	
  

Use	
  case	
  3:	
  Python	
  and	
  other	
  interpreted	
  languages	  $ spack install python@2.7.10==> Building python.
==> Successfully installed python.
Fetch: 5.01s. Build: 97.16s. Total: 103.17s.
[+] /home/gamblin2/spack/opt/spack/linux-x86_64/gcc-4.9.2/python-2.7.10-y2zr767$ spack extensions python@2.7.10==> python@2.7.10%gcc@4.9.2=linux-x86_64-y2zr767==> 49 extensions:geos
py-h5py
py-numpy
py-pyparlibxml2py-ipython
py-pandas
py-pyparsing
py-basemap
py-libxml2
py-pexpect
py-pyqt
py-biopython py-lockfile
py-pilpy-pysidepy-cffipy-makopy-pmwpy-python-daemon
py-cythonpy-matplotlib py-pychecker
py-pytz
py-dateutil
py-mockpy-pycparser
py-rpy2py-epydoc
py-mpi4py
py-pyelftools py-scientificpython
py-genders
py-mx
py-pygments
py-scikit-learn
py-gnuplot
py-nosepy-pylint
py-scipy==> 3 installed:-- linux-x86_64 / gcc@4.9.2 -----------------------------------py-nose@1.3.6 py-numpy@1.9.2 py-setuptools@18.1==> None currently activated.
$ spack activate py-numpy==> Activated extension py-setuptools-18.1-gcc-4.9.2-ru7w3lx==> Activated extension py-nose-1.3.6-gcc-4.9.2-vudjpwc==> Activated extension py-numpy-1.9.2-gcc@4.9.2-45hjazt$ spack deactivate -a py-numpy==> Deactivated extension py-numpy-1.9.2-gcc@4.9.2-45hjazt==> Deactivated extension py-nose-1.3.6-gcc-4.9.2-vudjpwc==> Deactivated extension py-setuptools-18.1-gcc-4.9.2-ru7w3lx

LLNL-PRES-803375

py-setuptools
py-shiboken
py-sippy-sixpy-sphinx
py-sympy
py-virtualenv
py-yapf
thrift§ Many	  interpreted	
  languages	
  have	  their	  own	
  mechanisms	
  for	  modules,	
  e.g.:	
  — Require	
  installa)on	
  into	  interpreter	  preﬁx	  — Breaks	
  combinatorial	
  versioning	
  § Spack	
  installs	
  each	  Python	
  package	  in	
  its	  own	
  preﬁx	  § “Ac)va)ng”	
  links	
  an	
  extension	
  into	  the	  interpreter	
  directory	
  on	
  demand	
  — Supports	
  .egg,	
  merging	
  .pth	  ﬁles	  — Mechanism	
  is	
  extensible	
  to	
  other	  languages	
  — Similar	
  to	
  virtualenv,	
  but	  Spack	
  allows	
  much	
  more	  build	
  customiza)on.	
  h"p://bit.ly/spack-­‐git	
  

23	
  

Use	
  case	
  4:	
  Spack	
  is	
  being	
  adopted	
  by	
  LLNL	
  code	  teams	
  
ARES

scipy

numpy

pythongsl Laser gperftools papi

CRETIN Scallop GA

LAPACKsqlite readline openssl bzip2 BLAS

overlink

SAMRAIrng perflib memusage timers

Silo

boost

Cheetah LEOS

HDF5

hpdf

opclient

MSlib

matprop

Teton HYPRE

qd

tdf

sgeos_xml

DSD Nuclear

bdivxml

ASCLaser

tk

tcl

MPI

Types of Packages
ncursescmake

Physics Utility Math Externalzlib§ ARES	
  is	
  a	
  1,	
  2,	
  and	
  3-­‐D	  radia)on	
  hydrodynamics	
  code	  § Spack	
  automates	
  the	
  build	
  of	
  ARES	  and	  all	  of	
  its	
  dependencies	  — The	
  ARES	  conﬁgura)on	
  shown	
  above	  has	  47	  dependencies	  LLNL-PRES-803375

h"p://bit.ly/spack-­‐git	
  

24	
  

ARES	
  has	
  used	  Spack	
  to	
  test	
  36	
  diﬀerent	
  conﬁguraDons	
  

Linux
BG/Q
Cray XE6§ Nightlyof ARESaree link.
To keepbuilds
packageinstallations
MVAPICHMVAPICH2
OpenMPI
BG/Q MPI
Cray MPI
at right.
Spack shown
has a well-definedmechanismGCC
C P L DC P L D
4 codeversions: of internal
ks; it uses
a combination
Intel 14
C P L D
• (C)urrent
Production
site-definedpolicies
to define an orderIntel 15
C P L DDparameters.
By default,
Spack prefers• (P)revious
Production
PGI
D
C P L D
C L D
ompiled• with(L)itenewer compilers to older
Clang
C P L DC L D
XL
C P L D
ompilers.
It has well-defined, but not• (D)evelopment
er of preference for deciding between
Table 3: Configurations of ARES built with Spack:§ compilers.
Learning Spackand porting
took asingledeveloper
2 months,
half-time.
fferent
The default
policies all libraries(C)urrent
and(P)revious
production,(L)ite, and(D)evelopment).
uration files, by either users or by sites.
§ typicallyPreviously,
teamcompiler,
was only able to automate its development Linux builds.
s may
use thethe Intel• Spack
enabledthorough testing
configurations
system’sdefault
gcc@4.4.7.
Theseof many moreon 11
LLNL physics packages, 4 LLNL math/meshing libraries,• Testing with Spack helped find compilationand
issueswhen using
Clang
compiler. The utility libraries handle tasks inby adding:
8 LLNLutilitylibraries.
cluding logging, I/O, and performance measurement. ARES also
gcc@4.4.7§ Spack is helping the team port to LANL’s new Trinity (Cray XC-40) machineuses 23 external software packages, including MPI, BLAS, Python,and many other libraries. Together, these packages are written in ale, which would cause the ambiguous25	
  diverse set h"p://bit.ly/spack-­‐git	
  of languages including C, C++, Fortran, Python and tcl
LLNL-PRES-803375

Related	
  work	  § OS	
  package	
  managers	
  — Don’t	
  handle	  combinatorial	
  builds	  — Single	
  compiler;	
  single	
  stable	  version	
  of	
  pkg.	
  — Allow	
  smooth	
  upgrades	
  and	  predictable	
  user	  experience.	
  § Gentoo	
  Preﬁx	  — Based	
  on	
  Gentoo	  Linux:	
  builds	
  from	  source,	
  installs	
  into	  common	
  preﬁx	  — Common	
  preﬁx	
  limits	
  mul)-­‐compiler	
  and	  mul)-­‐version	
  support.	
  § Nix	  (from	
  NixOS)	
  — Allows	
  many	  separate	
  conﬁgura)ons	  — Packages	
  are	
  cryptographically	
  hashed.	
  — Mul)-­‐compiler,	
  version	
  support	
  is	
  limited	  — No	
  virtual	
  dependencies	  — No	  syntax	
  for	  parameteriza)on.	
  LLNL-PRES-803375§ EasyBuild	
  (HPC	
  U.	
  Ghent)	
  — Requires	
  a	
  ﬁle	  per	  conﬁgura)on	
  of	
  so^ware	
  • 3300	  conﬁg	
  ﬁles	  for	
  600	
  packages	
  (!)	
  

— Limited	
  command	
  line	  interface	  — Limited	
  DAG	
  and	
  dependency	
  analysis	  § Hashdist	
  — No	
  spec	  syntax,	
  more	  package	
  ﬁle	
  and	
  proﬁle	  edi)ng	  required,	
  less	  composable.	
  — Compiler/architecture	
  support	
  is	
  limited	
  § Smithy	
  (ORNL),	
  Maali	  (Pawsey)	
  

• No	
  dependency	
  management;	
  only	  install	
  automa)on	
  h"p://bit.ly/spack-­‐git	
  

26	
  

Many	  new	
  feature	
  developments	
  are	  in	
  progress	  § Current:	
  — Lmod	  hierarchy	
  integra)on	  — External	
  dependencies	  • Autodetect	
  system	
  MPI	
  and	
  other	
  packages	
  — Custom	
  compiler	
  ﬂag	
  injec)on	  — XML	
  Test	  output	
  (JUnit)	
  • Each	  dependency	
  exposed	
  as	
  test	  case	  — Be@er	
  Cray	  environment	
  integra)on	
  http://bit.ly/spack-git§ Planned:	  — Use	
  compiler	
  wrappers	
  to	
  apply	
  tools	  to	
  large	  codes	  • Klocwork,	
  thread	
  sani)zers,	
  etc.	
  — Dependencies	
  on	
  compiler	
  features	
  (C++11,	
  lambdas,	
  OpenMP	
  versions)	
  — Automa)c	
  ABI	
  checking	
  &	
  upgrading	  LLNL-PRES-803375

h"p://bit.ly/spack-­‐git	
  

27	
  

The	
  Spack	  project	
  is	
  growing	
  rapidly.	
  § Spack	
  is	
  ﬂexible	
  enough	
  for	
  HPC	
  needs	
  — From	  single	
  users	
  of	
  small	
  clusters,	
  to	
  large	  code	  teams	
  on	
  top-­‐10	
  supercomputers.	
  § Spack	
  is	
  star)ng	
  to	
  be	
  used	  in	
  produc)on	  at	
  LLNL	  — Build,	
  test,	  and	
  deployment	
  by	
  code	  teams.	
  — Tools,	
  libraries,	
  and	
  Python	
  at	
  Livermore	
  Compu)ng.	
  — Build	  research	
  projects	
  for	
  students,	
  postdocs.	
  § Spack	
  has	  a	
  rapidly	
  growing	
  external	
  community.	
  — NERSC	
  is	
  working	
  with	
  LLNL	
  on	
  Cray	  support	
  for	
  Cori.	
  — Argonne/IIT	
  cluster	
  challenge	
  project.	
  — Kitware	
  contribu)ng	  ParaView	  builds	
  &	
  features.	
  — INRIA	
  using	  Spack	
  to	
  package	
  MORSE	
  numerical	
  so^ware	
  — Users	
  and	  contributors	
  at	
  EPFL,	
  U.	
  Oregon,	  Sandia,	
  LANL.	
  

LLNL-PRES-803375

h"p://bit.ly/spack-­‐git	
  Get	
  Spack!	
  h"ps://bit.ly/spack-­‐git	
  

28	
  

Builds	
  share	  as	
  many	  dependencies	
  as	
  possible	
  openmpi
mpileaks2
libdwarf

callpath2
dyninstlibelfcallpath
mpileaksmpich§ May	  add	
  space	
  overhead	
  compared	
  to	
  an	
  LD_LIBRARY_PATH	
  based	  system	  § Safer	
  than	  modules	
  or	
  LD_LIBRARY_PATH	
  since	
  the	
  user	  cannot	  get	
  deps	
  wrong	  — Installa)ons	
  always	
  run	  they	
  way	  they	  are	
  built.	
  § Above	
  shows	
  mpileaks	
  built	
  with	  mpich,	
  then	  openmpi	
  — Do@ed	  packages	
  must	  be	
  rebuilt.	
  LLNL-PRES-803375

h"p://bit.ly/spack-­‐git	
  

30	
  

Time (seconds)

ConcreDzaDon	
  Dme	  is	
  reasonable	
  even	
  for	
  large	
  packages.	
  
9
8
7
6
5
4
3
2
1
0

Linux, IBM Power7, 3.6Ghz
Linux, Intel Sandy Bridge, 2.6Ghz
Linux, Intel Haswell, 2.3Ghz

0

10

20

30

40

50

Package DAG size (nodes)§ Fixed-­‐point	
  concre)za)on	
  algorithm	
  scales	  quadra)cally	  § Spack	
  graphs	
  are	
  small,	
  even	
  for	  the	
  largest	
  packages	  — Thousands	
  of	
  dependencies	
  are	
  unlikely,	
  even	  in	
  mul)-­‐million	
  line	  code	  bases.	
  — Using	  a	
  proper	
  constraint	
  solver	
  will	  speed	  this	
  up.	  LLNL-PRES-803375

h"p://bit.ly/spack-­‐git	
  

31	
  

400
350
300
250
200
150
100
50
0

Overhead (% time)

Build time (seconds)

Compiler	
  wrappers	
  incur	
  some	  overhead	
  Wrappers, NFS
Wrappers, Temp FS
No Wrappers, Temp FS

libelflibpngmpileaks

libdwarf

pythondyninstLAPACK

80
70
60
50
40
30
20
10
0

62.7
48.0

46.4
35.6

9.5

libelfLibraries

Wrappers, NFS
Wrappers

12.3

9.4

libpngmpileaks

17.7
6.6

libdwarf

10.2

python

16.6
4.9

-0.4

dyninst

6.0

LAPACK

Libraries§ Extra	
  script	
  layer	  requires	
  some	  overhead	  § Spack’s	
  decision	
  to	
  build	
  in	
  tmp	
  ﬁlesystem	
  improves	
  more	  than	  script	
  overhead	
  hurts.	
  LLNL-PRES-803375

h"p://bit.ly/spack-­‐git	
  

32	
  

Future	
  direcDon:	
  Dependencies	
  on	
  compiler	
  features	  § Profusion	
  of	
  new	
  compiler	
  features	
  frequently	
  causes	
  build	  confusion:	
  — C++11	  feature	
  support	  — OpenMP	
  language	
  levels	  — CUDA	  compute	
  capabili)es	  § Spack	
  could	
  allow	
  packages	
  to	
  request	
  compiler	
  features	
  like	  dependencies:	
  

	
  

require(‘cxx11-lambda’)!
require(‘openmp@4:’)!

§ Spack	
  could:	
  1.
Ensure	  that	  a	
  compiler	
  with	  these	  features	
  is	
  used	  2.
Ensure	  consistency	
  among	  compiler	
  run)mes	
  in	
  the	
  same	  DAG.	
  LLNL-PRES-803375

h"p://bit.ly/spack-­‐git	
  

33	
  

Future	
  direcDon:	
  Compiler	
  wrappers	
  for	
  tools	
  § AutomaDcally	
  adding	  source	
  instrumentaDon	
  to	
  large	  codes	
  is	
  diﬃcult	  — Usually	
  requires	
  a	
  lot	  of	
  eﬀort,	
  especially	
  if	
  libraries	
  need	  to	  be	
  instrumented	
  as	
  well.	
  § Spack	
  could	
  expose	
  Klocwork,	
  Scalasca,	
  TAU,	
  etc.	
  as	
  “secondary”	
  compiler	
  wrappers.	
  — Allow	  user	  to	
  build	
  many	  instrumented	
  versions	  of	
  large	  codes,	
  with	  many	  diﬀerent	
  compilers:	
  
	
  
	
  
spack install application@3.3 %gcc@4.7.3 +tau!
	  § Spack	
  packages	
  provide	  a	
  general	
  interface	  to	
  build	
  details.	
  § LLNL	
  PRUNER	
  debugging	
  tool	  is	
  looking	
  into	  this.	
  — Uses	  LLVM	  for	
  instrumenta)on;	
  needs	
  to	
  cover	
  all	
  libraries.	
  LLNL-PRES-803375

h"p://bit.ly/spack-­‐git	
  

34	
  

Future	
  direcDon:	
  AutomaDc	
  ABI	
  checking	  § We’re	
  starDng	
  to	
  add	  the	
  ability	
  to	
  link	  to	
  external	
  packages	  — Vendor	
  MPI	  — OS-­‐provided	
  packages	
  that	  are	
  costly	
  to	
  rebuild	
  § External	
  packages	
  are	
  already	
  built,	
  so:	  — Can’t	
  always	
  match	  compiler	
  exactly	
  — Can’t	
  always	
  match	
  dependency	
  versions	
  exactly	
  § Need	  to	
  guarantee	
  that	  the	
  RPATH’d	
  version	
  of	
  a	
  library	
  is	
  compaDble	
  with	  one	
  

that	
  an	
  external	
  package	
  was	
  built	
  with	  — Allows	
  more	  builds	
  to	
  succeed	  — Poten)ally	
  violates	
  ABI	
  compa)bility	
  § Looking	
  into	  using	  libabigail	
  from	  RedHat	
  to	
  do	
  some	  checking	  at	
  install	
  Dme.	
  LLNL-PRES-803375

h"p://bit.ly/spack-­‐git	
  35