Package: libsncompress
Summarylibsncompress– efficient and reproducible Python utility for
compressing supernova cosmological data.IntroductionThe Python packagelibsncompressimplements the linear compression
method described in the paper “Application of Bayesian graphs to SN Ia
data analysis and compression” (C. Ma, P.-S. Corasaniti, &
B. A. Bassett, 2016,MNRAS, submitted, “M16”; accepted version: 2016
MNRAS, 463, 1651, DOI:☞10.1093/mnras/stw2069, BibCode:☞2016MNRAS.463.1651M).
It is designed for use with theJLAdataset, but can be easily
extended for other similar datasets.It also includes a Python executable script,☞jlacompress,
that serves as a command-line user interface.The intended usage includes the following tasks:To obtain a compressed sample of the JLA dataset, obtaining the
location (mean) vector and scatter (covariance) matrix of the
luminosity distance modulus, possibly for subsequent cosmological
analysis.To perform cross-validation on portions of the dataset.To study the posterior distribution of SN standardization parametersalpha,beta, anddelta_M.To replicate the results in the above-cited research paper that is
built on the preceding tasks.To help developing new methods of data analysis and compression with
current and future data based on the current work.The programs work with both Python 2.7 and 3.6/3.7.InstallationThe most convenient to install the package depends on your intended
usage.To simply use the package and compression script, just install the
latest version fro the PyPI withpip:pip install -U libsncompressAdditional Python packages are required at runtime (please refer to the
section “Dependencies” for the list of dependencies of the current
version). The recommended installation method is to use the above
command, which will make sure that the supporting packages are installed
automatically.To contribute to the package development, run the tests with real data,
or verify the reproducibility of the research, it is necessary to clone
the repository usinggit:git clone https://gitlab.com/congma/libsncompress.gitThe full repository includes also the necessary files for testing and
verification. Please refer to the section “Testing and Development”
for further details.It is also possible to use the library packagelibsncompresswithout
installation, for example, by including them directly in your own
project.Using the JLA Compression ScriptThis utility comes with an executable scriptjlacompressthat is
tailored to the compression of theJLAdataset, as done in ourM16paper.Synopsisjlacompress [-h] [-d DIR] [-t FILE] [-p PREFIX] [-c z1 z2 [...]] [-n] [-v]The script requires JLA data files to run. See the section “Data
Files” for details.Command-Line Options-h,--help: show help message and exit-dDIR,--covdirDIR: path to directory of FITS covariance
files (default:./covmat)-tFILE,--tableFILE: path to JLA data table file (default:./jla_lcparams.txt)-pPREFIX,--output-prefixPREFIX: prefix to output file
names-cz1 z2[...],--controlsz1 z2[...]: locations of control
points (as redshift). At least two control points are required. If
unspecified, use the default control points in theJLA paper.-n,--no-logdet: don’t use the correct conditional
probability (default: off;warning:use at your own risk)-v,--verbose: turn on verbose output (default: off)OutputThe script writes three output files when it solves the optimization
problem successfully:Mean (approximate, actually posterior-maximizing) compression
parameters in the order of (alpha,beta,delta_M,mu1,mu2, …), wheremuNis the value of distance modulus
at theN-th control point. It is therefore a list ofN + 3numbersCovariance matrix (symmetric, and with all elements filled) of the
parameters. It is of shapeN + 3×N + 3, and the first 3
rows and columns correspond to the three standardization parametersalpha,beta, anddelta_M.Redshifts of theNcontrol points.The default output file names aremean.txt,cov.txtandredshift.txtrespectively, but they can be prefixed by arbitrary
strings specified by the user with the-p/--prefixoption.The slash (/) character in the prefix will be understood as
directory separators. If the directory part of a resulting path does not
exist, it will be created if possible, and nested directories may be
created by this process.The output files will have suffix-no-logdetappended to the path
but before the.txtextension, if-nor--no-logdetis
specified.When verbose output is enabled by-vor--verbose, additional
text will be written to the standard error.Exit StatusThe script exits with0for success. Any other value indicates
error.Example UsageAssuming the data files are in their default locations, the following
command reproduces the default compression results in theJLA paper.jlacompress -nData FilesThe JLA data files arerequiredfor using the package. However, we
cannot distribute them with the source package. Please read theJLA
readmepage for details about the data files.The followingtwofiles must be downloaded:The file☞jla_likelihood_v6.tgz,
compressed archive containing the filedata/jla_lcparams.txt.
This file contains the supernova sample catalogue. The other files in
this archive are not necessary.The FITS files containing the components of data covariance, in the
compressed archive☞covmat_v6.tgz.
The non-FITS files in this archive are not necessary.If the JLA data archives are already downloaded, you simply need to
extract the required files and specify their locations when using thejlacompressscript, as describedabove.The Git repository includes a shell script to download and extract these
files:☞download_jla.sh.
This script is meant to be run manually, and it is not distributed with
the wheel distribution on PyPI. However, it is included in the source
distribution, even if it’s skipped during installation of the sdist.To use the download script, simply invoking the script in the repository
(or extracted sdist tarball) directory./download_jla.shwill suffice – this will populate thetestdatadirectory with the
necessary files and check the file integrity. Doing so also ensures that
the tests can run.Testing and DevelopmentUsinglibsncompressin Your ProjectTo use the package directly in your own Python project, simplyimportlibsncompressThis will import three classes from its sub-modules into thelibsncompressnamespace:BinnedSN: data-file loader and pre-processorBinCollection: redshift binning and sanitizer; not very useful on
its ownCovEvaluator: the actual compressorThe first thing you need to do is to specify a list (ornumpyarray)
of control points, by theirbase-10 logarithmvalues. Currently, due
to development legacy, the “binning” class and methods are not
particularly efficient. This is usually not a problem because it will be
used only once.This list or array of control points must be encapsulate inanothercontainer (list, array, or tuple, etc.) before passing to the
initializer oflibsncompress.BinnedSNclass. The instance can be
initialized bybinned_sn=libsncompress.BinnedSN(basedirpath,tablepath,logbins=control_points)Herebasedirpathis the path to the directory containing the FITS
covariance data files,tablepaththe path to the text file
containing the JLA dataset table, andlogbinsis the nested list of
control points just obtained.After this, we can initialize the evaluatorlibsncompress.CovEvaluatorclass, which implements the evaluation of
probability log-density functions and their first 2 derivatives, like
this:ev=libsncompress.CovEvaluator(binned_sn,withlogdet=True)The optional argumentwithlogdetcontrols whether the full effect of
parameter-dependent covariance matrix is taken into account. It is so
named due to the ubiquitous presence of “ln det Cov” term. It defaults
toTruebut can be set toFalse, which will evaluate the
functions as if the customary chi-squared method were used.TheCovEvaluatorinstance,ev, provides a methodminimize,
which is a wrapper ofscipy.optimize.minimize. Additional positional
and keyword arguments are passed over to that function. The recommended
optimization algorithm istrust-ncgwhich fully utilizes the Hessian
matrix. This is the default minimization algorithm if left unspecified,
and other algorithms supported by☞scipy.optimize.minimizecan be passed as the optional keyword parametermethod.The return value ofCovEvaluator.minimizemethod is simply that of
the underlyingscipyfunction, but with results suitably scaled.The Hessian of log-PDF function can be obtained, then, at the minimizing
point in the parameter space. This can be used for constructing the
approximate covariance of compression parameters.Please notice that this implementation here is not a general, abstract
implementation of the linear compression method detailed inour
paper. It specifically implements the sawtooth-basis compression,
which is compatible with the originalJLA one. The implementation
details, as well as the exposed API, are likely to see significant
revisions in the future.Setting Up the Testing EnvironmentTo run the tests (including the reproducibility tests), it is necessary
to set up the environment with supporting packages and data.As described in thepreceding section, “Data Files”, the recommended
way is to clone the Git repository and populate thetestdatadirectory in the repository with the necessary files, which can be done
using thedownload_jla.shscript.After obtaining the data files, it is recommended to use the recent
version of☞toxto manage the
testing environments.pip install 'tox >= 2.8.0'Although not strictly necessary for running the tests themselvesper
se, it is recommended to install the☞pandocprogram (please consult your
operating system documentation) and the☞pypandocPython package.Running the TestsIf you have both Python 2.7 and 3.6/3.7 installed, simply invokingtoxwill create the source distribution and run the tests under both Python
variants. The default configuration will pull the latest supporting
packages from PyPI specified in the filedevel-requirements.txt.If you have only one working variant of Python, for example Python 2.7,
you can runtox -e py2,coverage-reportand skip the unavailable test environment setting.Reproducibility TestsOne important goal of the test suits in this repository is to ensure
that the results of JLA SNIa compression are always reproducible.First, as we have shown inM16, theJLAcompression results (their
Tables F.1 and F.2), especially the covariance matrix, are “very close”
to the ones obtained using this program on theJLA data release, but
with the (highly discouraged)withlogdet=Falseoption enabled forlibsncompress.CovEvaluator.Second, the compression results produced by this program on the released
JLA data must match those presented inM16, Tables A1 and A2.The reproducibility tests check that these constraints are satisfied by
all revisions to the codebase. These tests are included in thetests/test_reprod.pyscript and are run bytoxby default.Dependencies☞six(unknown version), for
Python 2 and 3 compatibility;☞numpy(>= 1.6.0), for array
data structure and basic operations;☞scipy(>= 0.11.0), for linear
algebra and numerical optimization;☞astropy(unknown version), for
loading FITS files with theastropy.io.fitsmodule, which
replaces the dependence on☞pyfitsin earlier
versions;☞cachetools(unknown
version), for caching partial evaluation results, which is essential
for compression speed.Performance NotesPerformance is mostly determined by the following two conditions:Underlying BLAS/LAPACK libraries used bynumpy/scipy,
especially the “linear solver by Cholesky decomposition”,(D)POTRSfunction of LAPACK. ForNetLib LAPACK, this in turn
is largely determined by the speed of the level-3 BLAS triangular
solver,(D)TRSM. The NetLib reference implementation is rather
naive, and an optimized implementation of BLAS is likely to boost the
performance.Choice of initial value and scaling for numerical optimization. If
they are suitably chosen, the number of iterations required to
achieve convergence is reduced.The script☞jlacompressattempts to automatically create acceptable initial value and scaling
that is optimized for thedefaultcompression used in theJLA
paper. The automatic initial value and scaling are not optimized for
any other usage cases.Reporting BugsPlease report problems via theissue tracker.BibliographyIf you use this program in your research, we would like to suggest you
cite the following paper (“M16”):Ma, C., Corasaniti, P.-S., & Bassett, B. A. 2016, MNRAS, 463, 1651,☞doi: 10.1093/mnras/stw2069The following BibTeX entry could be useful in a LaTeX document:@ARTICLE{2016MNRAS.463.1651M,
   author = {{Ma}, C. and {Corasaniti}, P.-S. and {Bassett}, B.~A.},
    title = "{Application of Bayesian graphs to SN Ia data analysis and compression}",
  journal = {MNRAS},
archivePrefix = "arXiv",
   eprint = {1603.08519},
     year = 2016,
    month = dec,
   volume = 463,
    pages = {1651-1665},
      doi = {10.1093/mnras/stw2069}
}