Package: nirwals
nirwals_reduce - instrumental detrending pipeline for SALT NIRWALSComponentsThe installation includes both the actual python package, as well as links to a small number of
standalone tasks to run different functions of the code. These includenirwals_reduce.py-- run the science pipeline on one or more frames, taking raw data from individual reads to
science-ready data, calibrated in counts per second, and including both uncertainties and several quality flags.nirwals_fit_nonlinearity.py-- generate non-linearity correction coefficients from a sequence of flat-field imagesnirwals_stalker.py-- On-the-fly reduction tool that monitors a specified directory, performs a limited but fast
data reduction, and displays the resulting results in a ds9 viewer in near-real time.Installation and dependenciesThe nirwals package requires the following standalone python packages to run, most of which are likely already installed
on any common astronomy system:astropymatplotlibnumpyscipypandasmultiparlog -- parallel processing safe loggingpyvo -- VO capabilities, including the SAMP protocol used only for the on-the-fly reduction toolAll dependencies should be installed automatically as part of any installation (see below), but can also be installed
individually.Note:depending on your local python environment setup, the installation may require admin privileges to install the
standalone tools.installation via pip (recommended)Nirwals pipeline is available via pip. To install, simply runpip install nirwalsnew pip releases are handled automatically via GitHub, so this provides access to the latest stable version.Installation directly from GitHub sourceAlternatively, you can install the pipeline directly from the source on github.git clone https://github.com/SALT-NIRWALS/nirwals.git
cd nirwals
python3 setup.py installUnlike the pip method above, this provides access to the bleeding edge of the code.How to run:nirwals_reducedata reduction pipelinenirwals_reduce [options] file.fitsAvailable options--maxfiles=Nspecifies the maximum number of files to open for a given
up-the-ramp group. This is mostly to limit RAM usage. Default is no limit.--nonlinearity=file.fitsApply non-linearity corrections to the reference-pixel/first-read subtracted
dataset. The reference file should be a file generated via the
rssnir_fit_nonlinearity tool to contain the pixel-level corrections in the
correct format--flat=flat.fitsSpecify a flatfield frame. Not implemented yet.--dark=dark.fitsSubtract a dark-current correction from the entire input data cube. Userssnir_makedark.pyto generate the dark calibration frame.--output=_suffix_When generating the output filename, the specifiedsuffixis inserted into the
input filename. Example: for input filerss_test.fitsthe output filename would
be _rss_test.suffix.fits. Default is "reduced".--refpixelUse the reference pixel in the first & last 4 rows and columns to
subtraced an instrumental pedestal level off all the input data. If not specified
the first read is considered to contain this zero-exposure offset.--dumpsMostly used for debugging. When provided the tool also writes a number
of intermediate data products to disk that allow testing and verification.Example call:/work/rss/rss_reduce.py  --refpixel --maxfiles=70 SALT_data_RN_20220606/20220606_RN_URG_2reads_9dB.540.1.20.fitsoutput:rkotulla@legion:/work/rss/salt> ../rss_reduce/rss_reduce.py --refpixel \
    --maxfiles=70 SALT_data_RN_20220606/20220606_RN_URG_2reads_9dB.540.1.20.fits
/work/rss/salt/SALT_data_RN_20220606/20220606_RN_URG_2reads_9dB.540.1.20.fits
/work/rss/salt/SALT_data_RN_20220606/20220606_RN_URG_2reads_9dB.540.1.1.fits
 -- /work/rss/salt/SALT_data_RN_20220606/20220606_RN_URG_2reads_9dB.540.1.2.fits
 -- /work/rss/salt/SALT_data_RN_20220606/20220606_RN_URG_2reads_9dB.540.1.3.fits
 -- /work/rss/salt/SALT_data_RN_20220606/20220606_RN_URG_2reads_9dB.540.1.4.fits
...
 -- /work/rss/salt/SALT_data_RN_20220606/20220606_RN_URG_2reads_9dB.540.1.247.fits
 -- /work/rss/salt/SALT_data_RN_20220606/20220606_RN_URG_2reads_9dB.540.1.248.fits
 -- /work/rss/salt/SALT_data_RN_20220606/20220606_RN_URG_2reads_9dB.540.1.249.fits
 -- /work/rss/salt/SALT_data_RN_20220606/20220606_RN_URG_2reads_9dB.540.1.250.fits
Limiting filelist to 70 frames
(70, 2048, 2048)
Applying non-linearity corrections
No nonlinearity corrections loaded, skipping
No linearized data found, using raw data instead
No dark correction requested, skipping
diff stack: (70, 2048, 2048)
Identifying bad pixels
Cleaning image cube
calculating final image from stack
Writing reduced results to 20220606_RN_URG_2reads_9dB.540.1.reduced.fits
all done!Caveats and limitationsNot yet supported are fowler-reads of any kind, in particular when combined with
up the ramp sampling.Watch out when running on large numbers of up-the-ramp samples to avoid running out
of memory (RAM). At this time the tool is optimized towards computing time at the
expense of memory demand. If in doubt or to begin use the--maxfilesoption to limit the number
the number of open files and thus the memory footprint.How to run: On-the-fly reduction and monitoringThe basic function of thenirwals_stalker.pytool is to provide a on-the-fly data reduction and inspection in near
real-time for use at the telescope. Towards this goal, this tool includes:reference pixel correctionnon-linearity correctionconversion of raw reads to observed count rates in counts per secondproper handling of saturated pixelsautomatic display of the resulting data file in ds9, with communication via the SAMP protocol.Note that this tool does NOT include a stand-alone SAMP server (at least not yet; i.e. for the automatic ds9
functionality to work you need an active SAMP Hub, e.g. by running topcat).To run, one needs to specify a directory to monitor, a staging dir, and some optional options that modify some of the
internal algorithm.nirwals_stalker.py incoming_data_dir --nonlinearity=/some/where/nonlinpoly.fits --stage=/some/other/dirAvailable optionsOptions mirror the naming convention and functionality ofnirwals_reducewherever possible.--stagestaging directory--nonlinearityspecify the non-linearity correction parameter file--refpixelreference pixels mode (use blockyslope2 for best performance)--testRun a test, simulating the arrival of new files in the monitored directory. Syntax is--test=delay:@filelist, with delay giving a delay time between successive frames in seconds, and filelists specifying
a file containing a list of "newly arrived" files, with one file per line (lines starting twith # are ignored)Run-times were tested on a modern laptop (i7 CPU, 32 GB RAM). Using a "full" reduction mode, including non-linearity
takes approx 0.4 to 0.5 seconds per frame, from finding the newly arrived frame to end of writing the final result file.
Given the minimum read time of the NIRWALS instrument of ~0.7 seconds this should allow monitoring incoming data in
effectively real time (i.e. the previous frame is displayed before the next read is fully read out).