Package: pySpecData
To learn more about pyspecdata, you can head over to thedocumentation.If you already know that you want to install,
and you are using Anaconda, you should seeconda_upgrade.md.Please note this package is heavily utilized by three other packages that our lab manages on github:ODNP processing scriptsClasses for communicating with instrumentsSpinCore ExtensionWe have somewhat recently added fast compiled Fortran functions for things like
2D ILT (Tikhonov regularization with basis set compression) for NMR (Nuclear Magnetic Resonance),
so please read the install instructions
carefully!pySpecDataObject-oriented Python package for processing spectral data – or in general,n-dimensional data with labeled axes (i.e.N-dimensional gridded data or “nddata”).
It depends on numpy, which provides very fast manipulations ofN-dimensional gridded arrays (“ndarray”).
This package has some overlap with xarray,
but it doesn’t attempt to mimic pandas notation,
shooting instead for very compact notation for natural slicing, etc.
It mainly focuses on making it easier toquickly write good code
for processing spectroscopy data.
In particular, it takes care of various features related to fourier
transformation, error propagation, and direct products in multidimensional data with
little to no interaction from the user.If you are working in a lab developing new spectroscopic methodologies, then this package is definitely for you.
If you deal with multi-dimensional data of some other form, then it’s likely for you.
Features include:FeaturesLabeled axes allow one to manipulate datasets (potentially with different dimensions) without having to explicitly keep track of what the different dimensions correspond to.  Code becomes more legible.  Also, tiling, direct product, and gridding functions become obsolete.Fourier transformation with automatic manipulation of axes.Automatic error propagation.Commands likeplot(data)will generate a plot with automatically labeled
axes, errors, and units.
All of this information is also written to HDF5 when the data is saved.Simplified curve fitting that takes advantage of labeled axes and Python’s symbolic algebra package (sympy).The code is written so that it can be integrated into a nicely formatted PDF lab notebook.The same code can be run on the command line (to generate pop-up plot windows) and embedded into a LaTeX document.Extension to other output formats, such as HTML or markdown, should be relatively straightforward.In a multimedia environment like jupyter, you don’t need a separate plot
command.  The code can automatically choose a plotting style appropriate to
the code (eventually, the general preferences for this can just be configured
at the beginning of the jupyter notebook).More detailed web documentation will be coming soon.NMR/ESR specificBecause it was written primarily for NMR and ESR data, it also includes:Routines for reading commercial raw data (e.g.Bruker, Kea) into nddata
objects with all relevant information.The object-oriented features make it much easier to process raw phase-cycled
data and to simultaneously view multiple (potentially interfering) coherence
pathways.Contains functions for baseline correction, peak integration,etc.(Not yet in packaged version) A basic compiled routine for propagating
density matrices that can be used to predict the response to shaped pulses.Version NotesNote that the current version is intended just for collaborators,etc.(Though, if you do really want to use it for interesting science,
we are happy to work with you to make it work for your purposes.)
A public-use version 1.0.0, to be accompanied by useful demonstrations, is planned within a year.(Note that the email currently linked to the PyPI account is infrequently checked –if you have interest in this software, please find J. Franck’s website and contact by that email.)History/Roadmap(Current version in bold)0.9.5First version distributed on pypi.python.org.0.9.5.10.9.5.1.1
Some important debugging, and also addedpyspecdata.ipy→ executing the following at the top of a jupyter notebook:%pylabinline%load_extpyspecdata.ipywill cause nddata to “display” as labeled plots.0.9.5.1.2
added ability to load power saturation 2D data from Bruker0.9.5.1.3
XEpr data loaded with dBm units rather than W unitsaddedto_ppmfunction for Bruker files0.9.5.1.4
Improved internal logging, and started to remove gratuitous dependencies,%load_ext pyspecdata.ipyincludes%pylab inline, so that only%load_extpyspecdata.ipyis required for jupyter.0.9.5.1.6Removed several legacy modules, and added docstrings for the remaining modules.Begin phasing out earlierCustomErrorclass.Makenumpypretty printing available from thegeneral_functionsmodule.Add xelatex support to the notebook wrapper.Start to move file search routines away from demanding a single “data directory.”Improved support for 2D Bruker XEPRMade it possible to call standard trig functions withnddataas an argument.0.9.5.1.7ILT (Tikhonov regularization) with SVD Kernel compression
(1 and 2 dimensions)smooshandchunkdeal with axes properly0.9.5.3Current Versionupgrade to Python 3 and begin to flesh out documentation0.9.5.40.9.5.4.1
-to_ppmshould only be a method of inherited class
- 1.5 and 2.5 D ILT1.0We are working on four major upgrades relative to the 0.9 sequence:Axes as objects rather than a set of separate attributes of nddata.Remove dependence on pytables in favor of h5py.Replace figure lists with “plotting contexts,” which will still
enable PDF vs. GUI plotting, but will better integrated with Qt and
more object-oriented in natureComma-separated indexing to work correctly with all indexing types.
(0.9.5 requires sequential brackets rather than comma-separated
indexing for some combined range selections.)1.0.2GUI for setting configuration directories.Means for dealing with non-linearly spaced data in image plots
(0.9.5 auto-detects log spacing in 1D plots,
but pretends that image plots are linear – we will implement linear spline
interpolation algorithm)1.0.3Bruker DSP phase correction for raw data from newer versions of Topspin that is in sync with the code from nmrglue.1.0.4Package a make-less copy of lapack to allow a cross-platform build of density matrix propagation routines.1.1.0Integrate with ACERT NLSL Python package for simulation and fitting of ESR spectra.1.2.0Implement a version of figure list that can be interfaced with Qt.Installation NotesHighly Recommended:Install the following packages using a good package-management system (conda or linux package manager), rather than relying onpiporsetuptoolsto install them:numpyscipysympypyqtpytables (in future work, we hope to eliminate dependence on this package)matplotlibh5pyThe python libraries, and a Fortran compiler.  Under anaconda, these are supplied bylibpythonandmingw, respectively.For example, on Windows withAnaconda 2.7.
– just runconda install-canaconda numpy scipy sympy pyqt pytables matplotlib h5py libpython mingw.On CentOS7, we’ve testedyum installpython-matplotlibpython-matplotlib-qt4python-develsympy h5pypython-tablesscipy(after runningyum installepel-releaseto install the EPEL distribution)(If you don’t install these packages with your systempipwill try to install them, and there is a good chance it will fail – it’s known not to work great with several of these;setuptoolsshould error out and tell you to install the packages.)mayavi: Mayavi can be used (and gives very nice graphics), but frequently lags behind common Python distros.
Therefore, this package was written so that it doesn’t depend on mayavi.
Rather, you can just importmayavi.mlaband pass it to any figure list that you initialize:figlist_var(mlab = mayavi.mlab)Installation for developers(Once these are installed,
to install from github, justgit clonehttps://github.com/jmfranck/pyspecdata.gitthen move to the directory where setup.py lives,
and dopython setup_paramset.py installfollowed bypython setup.py develop)Important note for conda on Windows 10:For reasons that we don’t understand, the Fortran compiler can give odd errors, depending on which terminal you are using to install.
This appears to be Windows’ fault, rather than conda’s (?).
We highly recommend trying both the Anaconda prompt, as well as the standard dos prompt (press start: typecmd) if you experience errors related to compilation.For compiled extensionsAll compiled extensions are currently stripped out, but will be slowlyadded back in.If you are on windows, you will need some additional packages to enable compilation:libpythonmingwThe last one is specific to Windows, and provide things like thegccandgfortrancompiler.Quick-StartTo get started with this code:Install a good Python 2.7 distributionOn Windows or MacOS:Anaconda 2.7.  When installing select “install for all users.”Install libraries that pyspecdata depends on. (If you’re interested in why you need to do this first, see installation notes below.)On Windows or MacOS: in the Anaconda Prompt, runconda install numpy scipy sympy pyqt pytables matplotlib h5py libpython mingw.For Mac, you can also use homebrew.
Note that, in the current version python is renamed topython2,
andpiptopip2.
Most packages can just be installed withpip2under homebrew.
If you want HDF5 functionality, you will need to runbrew tap homebrew/sciencefollowed bybrew install hdf5.On Linux, just use your package manager (aptitude,yum,etc.) to install these libraries.Installparamset_pyspecdata:pip install paramset_pyspecdata,
thenpyspecdata:pip install pyspecdataor follow the “Installation for developers” section above.If you have difficulties with the install, check that you have a gfortran
compiler installed (in conda windows, this comes from mingw) and that, if
you are using windows, you are trying to install from a standard dos
prompt (we like to use git bash, but anaconda and related compilers can
misbehave from git bash sometimes).Set up directories.
You can run the commandpyspecdata_dataconfigto assist with this.It creates a file in your home directory
called_pyspecdata(Windows  – note the underscore)
or.pyspecdata(Mac or Linux).Here is an example – you can copy and paste it as a starting point:[General]
data_directory = c:/Users/yourusername/exp_data
notebook_directory = c:/Users/yourusername/notebookNote that any backslashes are substituted with forward slashes.
Also note that you will
need to change the directories to refer to real directories that already
exist or that you create on your hard drive (see below).
Note that on Windows, you can use notebook,etc.to create this file,
but it cannot have a .txt,etc.extension.Where is my “home directory”? (Where do I put the_pyspecdatafile?)On Windows, your home directory is likely something likeC:\Users\yourusername.
You can access your home directory by opening any file folder window, and
starting to type your name in the address bar – it’s the first folder that shows up
underneath.On MacOS and Linux, it’s the directory indicated by~.  On Linux,
this typically expands to/home/yourusername.On any OS, you can always find your home directory in Python usingimport os;printos.path.expanduser('~')What are these directories? → You can either create them or point to existing directories.data_directorymust be set.  It is a directory, anywhere on the
hard drive, where you store all your raw experimental data.  It must
contain at least one subdirectory – each subdirectory stores
different “experiment types,” typically acquired on different instruments
(e.g.you might have subdirectories named400MHz_NMR,500MHz_NMR,95GHz_ESR, andXband_ESR).The library now supports having datasets packed into.zipor.tgzfiles.
For example, Bruker NMR files typically comprise a directory with several subdirectories for the numbered experiments.
We routinely pack these up as zip files on the spectrometer, and directly read the data from the zip files.If you’re setting up a lab, you might want to separately sync each different
experiment type folders usingseafile.Or you can sync the whole data directory with dropbox.If set, thenotebook_directoryis intended to contain latex
files with embedded python code, as well as some processed
output.Do notuse quotes to surround the directory name.  Even if it contains
spaces, do not use quotes, and do not escape spaces with backslashes.Note that on Windows, your desktop folder is typically inC:\Users\yourusername\DesktopWhy do I need to do this?Setting this configuration allows you to move code between different
computers (e.g.a spectrometer computer, a desktop, and a laptop),
and re-use the same code, even though the locations of the files are
changing.  This should work even across different operating systems.It specifically enables functions likefind_file(...),get_datadir(...),etc.that can search the data directory for a
file name matching some basic criteria.
You should always use these to load your data,
andneveruse the absolute path.The GUI tool that will allow you to set up_pyspecdataby pointing
and clicking has not yet been set up.Notes on compilation of NNLSWe recently added a compiled extension that performs non-negative least-squares for regularization (DOSY/Relaxometry/etc.)Under linux or mac, you should have a gcc and gfortran compiler installed, and should make sure you have libpython for this to work.Under anaconda on windows, we have run into some trouble sometimes where it gives you an error 127.
We recommend using the normal dos command prompt (cmd) to install pyspecdata, and make sure that your path is set such thatwhere gccyields a gcc.exe (NOT .bat) file andwhere pythonyields the anaconda python executable.
(Recent versions of mingw appear to put .bat files in a preferential location
in the path, and these .bat files seem to mess everything up, including
compatibility with the git bash prompt.)