Doc 6.13, ver. 1.0 | Oct 2018

ALMA Science Pipeline User’s
Guide for CASA 5.4.
Interferometric and Single-Dish Processing

ALMA, an international astronomy facility, is a partnership of ESO (representing its member states), NSF (USA) and NINS
(Japan), together with NRC (Canada), NSC and ASIAA (Taiwan), and KASI (Republic of Korea), in cooperation with theRepublic of Chile. The Joint ALMA Observatory is operated by ESO, AUI/NRAO and NAOJ.

For further information or to comment on this document, please contact your regional Helpdesk through the ALMA User
Portal at www.almascience.org. Helpdesk tickets will be directed to the appropriate ALMA Regional Center at ESO,NAOJ or NRAO.

Version

Date

Editors

3.13v1.0 CASA 4.5.1

January 2016

Pipeline Team

4.13v1.0 CASA 4.7.0

October 2016

Pipeline Team

4.13v2.0 CASA 4.7.2

July 2017

Pipeline Team

5.13v1.0 CASA 5.1.1

November 2017

Pipeline Team

6.13v1.0 CASA 5.4.0

October 2018

Pipeline TeamIn publications, please refer to this document as:
ALMA Pipeline Team, 2017, ALMA Science Pipeline User’s Guide, ALMA Doc 6.13

Table of Contents:
Table of Contents: ...................................................................................................................... 1
1

2
3
4

5

6

7

8

9

The ALMA Science Pipeline ................................................................................................. 3

1.1
1.2

Purpose of this document ...................................................................................................... 3
Pipeline Overview and nomenclature .................................................................................... 3

Quick Start .......................................................................................................................... 4
What’s New in Cycle 6 ........................................................................................................ 4

3.1

Current Known Limitations of the Cycle 6 Pipeline ................................................................. 9

Pipeline Versions & Documentation ................................................................................. 11

4.1
4.2
4.3
4.4

Obtaining the Pipeline .......................................................................................................... 11
Pipeline-related Documentation .......................................................................................... 11
Pipeline & CASA Versions ..................................................................................................... 11
Pipeline and CASA tasks ....................................................................................................... 12

Data Processing Files ........................................................................................................ 13

5.1
5.2
5.3
5.4
5.5

Archived scripts .................................................................................................................... 13
Pipeline “Helper” text files ................................................................................................... 13
The Pipeline script to restore calibrated MSs: casa_piperestorescript.py ............................ 14
The Pipeline processing script: casa_pipescript.py ............................................................... 15
CASA equivalent commands file: casa_commands.log ......................................................... 19

Modifying a Pipeline Run using casa_pipescript.py .......................................................... 19

6.1
6.2
6.3
6.4
6.5
6.6

Pipeline re-processing considerations .................................................................................. 19
Preparing to run casa_pipescript.py ..................................................................................... 20
Modifying Calibration Commands ........................................................................................ 21
Modifying IF Pipeline Imaging Commands............................................................................ 21
Manual imaging after running casa_pipescript.py................................................................ 22
Manipulating the Pipeline Context ....................................................................................... 22

Description of Pipeline “Helper” Text Files ....................................................................... 23

7.1
7.2
7.3
7.4
7.5
7.6

IF Pipeline: flux.csv ............................................................................................................... 23
SD Pipeline: jyperk.csv.......................................................................................................... 25
IF Pipeline: antennapos.csv .................................................................................................. 25
Both IF & SD Pipeline: uid*flagtemplate.txt and uid*flagtsystemplate.txt........................... 26
IF Imaging Pipeline: uid*flagtargetstemplate.txt.................................................................. 27
IF Imaging Pipeline: cont.dat ................................................................................................ 27

The Pipeline WebLog ........................................................................................................ 29

8.1
8.2
8.3
8.4
8.5
8.6
8.7

Overview .............................................................................................................................. 29
Navigation ............................................................................................................................ 30
Home Page ........................................................................................................................... 30
By Topic Summary Page ....................................................................................................... 32
By Task Summary Page ......................................................................................................... 32
Task Pages ............................................................................................................................ 33
WebLog Quality Assessment (QA) Scoring ............................................................................ 37

The “By task” WebLog for Interferometric Data............................................................... 42

9.1
9.2
9.3
9.4

hifa_importdata ................................................................................................................... 42
hifa_flagdata ........................................................................................................................ 42
hifa_fluxcalflag ..................................................................................................................... 42
hif_rawflagchans .................................................................................................................. 43

9.5
9.6
9.7
9.8
9.9
9.10
9.11
9.12
9.13
9.14
9.15
9.16
9.17
9.18
9.19
9.20
9.21
9.22
9.23
9.24
9.25
9.26
9.27
9.28
9.29
9.30
9.31
9.32
9.33
9.34
9.35
9.36
9.37
9.38

hif_refant ............................................................................................................................. 45
h_tsyscal............................................................................................................................... 45
hifa_tsysflag ......................................................................................................................... 45
hifa_antpos .......................................................................................................................... 47
hifa_wvrgcalflag ................................................................................................................... 47
hif_lowgainflag..................................................................................................................... 47
hif_setmodel ........................................................................................................................ 48
hifa_bandpassflag ................................................................................................................ 48
hifa_spwphaseup ................................................................................................................. 49
hifa_gfluxscaleflag ................................................................................................................ 49
hifa_gfluxscale...................................................................................................................... 49
hifa_timegaincal ................................................................................................................... 50
hif_applycal .......................................................................................................................... 50
hif_makeimlist: Set-up parameters for calibrator images .................................................... 50
hif_makeimages: calibrator images ...................................................................................... 51
hif_makeimlist: check source imaging .................................................................................. 51
hif_makeimages: check source imaging and QA information ............................................... 51
hifa_imageprecheck ............................................................................................................. 52
hif_checkproductsize: Mitigation to avoid overly long runs ................................................. 53
hif_exportdata...................................................................................................................... 56
hif_mstransform................................................................................................................... 56
hifa_flagtargets .................................................................................................................... 56
hif_makeimlist: Set-up parameters for target per-spw continuum imaging ......................... 56
hif_findcont .......................................................................................................................... 56
hif_uvcontfit ......................................................................................................................... 57
hif_uvcontsub....................................................................................................................... 57
hif_makeimages: Make target per-spw continuum images .................................................. 57
hif_makeimlist: Set-up parameters for target aggregate continuum images ....................... 60
hif_makeimages: Make target aggregate continuum images ............................................... 60
hif_makeimlist: Set-up image parameters for target cube imaging ...................................... 60
hif_makeimages: Make target cubes .................................................................................... 60
hif_makeimlist: Set-up image parameters for representative bandwidth target cube imaging
62
hif_makeimages: Make representative bandwidth target cube ........................................... 62
hif_exportdata...................................................................................................................... 62

The “By task” WebLog for Single-Dish Data ............................................................................. 63
9.39
9.40
9.41
9.42
9.43
9.44
9.45
9.46
9.47
9.48

hsd_importdata.................................................................................................................... 63
hsd_flagdata......................................................................................................................... 63
h_tsyscal............................................................................................................................... 63
hsd_tsysflag.......................................................................................................................... 63
hsd_skycal ............................................................................................................................ 64
hsd_k2jycal........................................................................................................................... 64
hsd_applycal......................................................................................................................... 64
hsd_baseline ........................................................................................................................ 65
hsd_blflag ............................................................................................................................. 69
hsd_imaging ......................................................................................................................... 69

2

1

The ALMA Science Pipeline

1.1 Purpose of this document
This document describes how to obtain the ALMA Pipeline, how to use it to calibrate andimage ALMA interferometric (IF) and single-dish (SD) data, and a description of the
Pipeline WebLog (collection of web pages with diagnostic information describing thepipeline run). Since interferometric and single-dish data are calibrated and imaged using
different procedures and diagnostics, their recalibration procedures and WebLogs aredescribed separately.
This document is applicable for the Cycle 6 version of the ALMA Pipeline that is included
in CASA 5.4.1, deployed for use in ALMA Operations in October 2018. This version islabeled as Pipeline Version 42030 (Pipeline-CASA54-P1-B) CASA Version 5.4.0-68 in
the WebLog.

1.2 Pipeline Overview and nomenclature
The ALMA Science Pipeline is used for the automated calibration and imaging of ALMA
interferometric and single-dish data. ALMA Interferometric data refers to observations
obtained with either the ALMA 12-m Array or 7-m Array, while single-dish data refers to
observations obtained with the 12-m dishes of the ALMA Total Power Array.
The Pipeline consists of modular calibration and imaging tasks within the Common
Astronomy Software Applications (CASA) data reduction package that are selected andput together in a specific order based on standard prescriptions or recipes. The ALMA
pipeline recipes cover the processing requirements of ALMA “standard” interferometricand single-dish observing modes. Datasets resulting from ALMA “non-standard”
observing modes are, as a rule, processed outside the pipeline, using manually modified
CASA scripts. The standard and non-standard observing modes are defined in the
Proposer’s Guide for each ALMA proposal cycle. The science pipeline is not yet
commissioned for the combination of datasets obtained from different array components(separate IF array observations, or IF plus SD combinations).
The pipeline operates on a completed dataset that is comprised of all of the quality
assured individual executions that result from completing a Scheduling Block (SB). An
individual SB execution results in a dataset referred to as an ASDM (for ALMA Science
Data Model), or EB (Execution block) and the collection of ASDMs (EBs) from a single
SB are collected into a data structure called a Member Observing Unit Set (MOUS), whichis the data unit that the pipeline operates on. The pipeline produces the following:
calibration products for each ASDM (including calibration and flagging files and tables);imaging products (FITS images) made from all ASDMs (although not necessarily for all
science targets); informative logs and scripts; and a WebLog consisting of a collection of
webpages with diagnostic messages, tables, figures, and “Quality Assurance" (QA)
scores. These products are reviewed as part of the ALMA Quality Assurance process,and, if satisfactory, are stored into the ALMA Science Archive. See the ALMA TechnicalHandbook for details on the ALMA data structures, quality assurance criteria, andarchiving system.
The Pipeline is data-driven: i.e. the characteristics of each dataset drive the calibrationand imaging strategy (the Pipeline Heuristics). During the Pipeline run, criticalinformation (for example, which calibration tables are used) are stored in the pipeline
Context. Both the Heuristics and the Context are implemented as python classes.
3In order to determine if the Pipeline was used in the processing of an ALMA dataset,please look at the WebLog or consult the README file in the data delivery package.
Some projects may contain a mix of both manually and Pipeline-calibrated data.

2

Quick Start
•
•
•
•

3

If you want to understand what data is in a downloaded package, and the steps
and quality of how it was processed see: section 8: The Pipeline WebLog.
If you want to restore the calibrated MS, run scriptForPI.py or
casa_piperestorescript.py, section 5.1.
If you want to see, edit, or rerun the pipeline task commands that were run, youwant casa_pipescript.py described in section 6.
If you want to see the CASA task calls that were used, either look at the casa log
linked at the bottom of each pipeline processing stage of the “By Task” section of
the weblog, or see the full casa_commands.log (Sec. 5.5)

What’s New in Cycle 6

Major new capabilities include:•••

Parallel (multi-core) imaging, for significantly faster (factors of 2-5) imaging
runtimes especially for MOUS with large cubes. To take advantage of this, simply
run CASA in MPI mode, using the mpicasa command included with CASA.
Heterogeneous array data (e.g. 7m ACA data with 12m antennas included insome executions to improve calibration) can now be fully calibrated and correctly
flagged.
Ephemeris sources are now correctly imaged: multiple executions are registeredin position using tclean’s. “trackfield” parameter, and spectral cubes are correctly
created in the source’s rest frame, including across multiple executions.

Additional new features listed by weblog section and processing stage include:••

Display of the processing environment on the weblog Home page (section 8.3)
Plots of uv coverage on the Sky Setup details of the weblog Home page, and again
on the applycal page after potential flagging during calibration:•

The concept of a virtual spw ID has been added to deal with the situation where
spw IDs change between executions within the MOUS (as can happen inparticular for heterogeneous array data). The value is added as a new column on
4

•

•
•
•

•

•

the Spectral Setup Details page, even if the values are all identical to the “real”
spw ID
The algorithm for computing the spectral index of calibrators (inau.getALMAFluxcsv) has been improved to set separationWindow=3 days
instead of 0, meaning that more recent measurements will often be used tocompute spectral index than in Cycle 5.
The weblog “By Task” page now includes the runtime of each stage
hifa_importdata: The age of nearest monitor point (in days) has been added tothe table of Imported Flux Densities; negative values were observed after scienceEBhifa_tsysflag:o a warning is printed if spectra were fully flagged before running tsysflag
and their percentages highlighted in flag summary table “flagged before”
column. This addition is to aid in understanding the rare cases when an
entire antenna or spw is flagged by online TelCal due to bad Tsys results.
o The parameter fnm_byfield has been set to True to prevent large elevationdifferences between targets from causing unnecessary flags (mostly
affects high frequencies)o The ATMOSPHERE intent has been removed from the Flagged datasummary tableo A new flagtsystemplate.txt file allows manual Tsys channel flagging, e.g. ininstances of Milky Way line emission contaminating Tsys spectra.
hifa_bandpassflag and hifa_gfluxscaleflag
o Anchors (table of contents) added to top of page for navigationo Fully flagged antennas are removed from refant list, and those flagged in
one or more spw moved to end of refant list before final bandpass solves
o New heuristic for “ultra-high” flags, which could sometimes be missed by
the other heuristics (too few baselines or timestamps to be picked up)o Iteration added: if any flags are found, it applies them and then re-runs theheuristics again (once), this helps in cases where something egregious
initially elevates the statistics or when there are two levels of problems
o For heterogeneous arrays, 12m-12m, 12m-7m and 7m-7m baselines areevaluated separately.
o The apriori plots were removed as offering little diagnostic benefit for thetime-cost to make themo Bandpass solutions will have minimum of 8 channels; this limit wasn’t fully
enforced in prior cycles. A warning is shown if recommended SNR (>50)
would require fewer channels, but this is acceptable if no solutions fail.
hifa_gfluxscale:o Anchors (table of contents) added to top of pageo Antennas Used for Flux Scaling Table added. Blank entries mean that all
UV ranges and/or all antennas were used, which will be the case forquasars.

5

o Derived flux density vs catalog flux density plots for non-amplitude
calibrators:••

hif_applycal:o several rarely-used plots removed to speed up processing timeo added per-ms representative spw / representative science target uvcoverage plot (flagged visibilities shown in red) at bottom of pagehif_makeimages (check source imaging) - new stageo summary of check source fitting results per spw, and per EB:
o Check source position offset from catalogue position in mas andsynthesized beams, warning level > 0.15*synthesizedBeamo Fitted [Peak Intensity / Flux Density] Ratio can help to assess
decorrelation AND presence of resolved emission, warning level < 0.8
o [Fitted / gfluxscale] Flux Density Ratio, warning level < 0.8
o Warning also given if S/N of fitted or gfluxscale flux densities are low < 20
o The images themselves which are per EB / spw are located below thetable, and except for being per EB are unchanged

6

•

•

•

hifa_imageprecheck:o “Goals from OT” section now lists Bandwidth for Sensitivity, PI Goal
sensitivity, and Single Continuum = T/Fo Sensitivity calculation is improved: It never uses a single channel
anymore, Spw overlap is taken into account, if repBW < repSPW
bandwidth (spw containing representative frequency) it calculatessensitivity of repSPW and then adjusts to repBW, and if repBW >repSPW, it calculates the aggregate BW sensitivity and then adjusts torepBWo Automatic choice of robust: If a non-standard robust results in a beamwith both axes within the PI’s desired range, then that robust is used, witha warning printed. If not, then the robust that has a beam area closest to
the mean of the PI’s range is chosen. If that is still outside the PI’s range,
an error is printed. See Figure 23 for an example table with non-standard
robust chosen.
hif_restoredata - new in image recipeo Previous processing environmento Rep. Target tableo Flagging summary for Target
hif_findcont:Now generates and evaluates the mean spectrum of a masked region of the dirty
line+cont image constructed from moment0 and moment8 (peak) images. In
addition, the heuristics that assess the mean spectrum have been improved – ina great majority of cases, equal or more continuum channels are found than
before.

7•hif_makeimages (cubes)
o “psf fit failed” warning is now printed if the per channel major axis changes
from the median by more than a factor of +/- 2 as this usually indicatessome channels are bad and likely need to be flagged.
o Both the center frequency AND rest frequency (i.e. zero velocity)
frequency are shown in the weblog for cube imaging•

hsd_importdatao Changed default value for createmms to 'false' to avoid creating MMS. A
new empty flagging template file is now generated in the working directorywith the default filename "[msname].flagtsystemplate.txt". This file can beused to provide manual Tsys flags used by the "hsd_tsysflag" task.

•

hsd_tsysflago Added a new parameter "filetemplate" (default:"[msname].flagtsystemplate.txt") to specify the name of the file containing
manual Tsys flags to apply. Manual Tsys flags are applied prior toevaluation of Tsys flagging heuristics.
o A warning is now shown if any Tsys spectra were found to already be fully
flagged (for any polarisation) prior to the evaluation of the Tsys flagging
heuristics.
o Updated 'nmedian' metric: if no data is found in tsys caltable for given spw,
field(s), and pol, then a warning is raised and no flagging view is created.

•

hsd_skycalo Elevation difference of ON_SOURCE data with respect to preceding andsubsequent OFF_SOURCE data are calculated.

•

hsd_baselineo Improved line detection. Selection of the algorithm used in the clustering
analysis to check the validity of detected line features include ’kmean’algorithm and hierarchical clustering algorithm ’hierarchy’. The default
algorithm is ’kmean’.
o Support flexible manual line window (mask) that allows to set line windows
to multiple spws simultaneously.

•

hsd_blflago Fixed a bug in estimation of expected rms.
8•

hsd_imagingo Added restfreq parameter to customize rest frequency value for each spw.
Default is to take the value from the MS.
o Direction reference frame for output images is taken from the MS.

•

hsd_exportdatao Report maximum elevation difference between ON and OFF topipeline_aquareport.
o The flagging template files (including the new manual Tsys template file) are
now included in the exported "auxproducts.tgz" file.

3.1 Current Known Limitations of the Cycle 6 Pipeline
The current Known limitations of the Cycle 6 pipeline include:•

The pipeline is commissioned only for ALMA “standard mode” observations, as
defined in the Proposers Guide for the latest cycle, subject to the additional
restrictions listed below.

•

All raw data (ASDMs) run through the pipeline must have complete and properly
formatted binary and metadata. This is not always the case for ASDMs from earlier
ALMA cycles. In particular:§ The SD pipeline can only be run on data from Cycle 3 or later.
§ The IF pipeline will not work with ALMA Cycle 0 data, or with some Cycle 1 – 2
data.

•

Manually calibrated data from Cycles 1 – 3 are likely to have problems if run
through the pipeline.

•

The raw data (ASDMs) run through the pipeline should have a “quality assurance
level 0” (QA0) assessment of “QA0 Pass”. Running the pipeline on non-quality
assured data (“QA0 SemiPass” or “QA0 Fail”) is not expected to give sound results
and may fail.

•The pipeline assumes that it has access to all of the available RAM on the node
where it is run. If other processes use significant amounts of this RAM, the pipeline
may fail.

•If the number of files opened by the pipeline (which increases with the number of
ASDMs being processed) exceeds the server specification, the pipeline will fail.
This limit can generally be increased using a command like “ulimit –Sn 4096”•

The hsd_baseline task has a minor bug that may affect a few datasets.
It aborts when a dataset meets all of the following conditions:(1) The project includes two or more target sources to be observed,(2) multiple MSs are processed,(3) at least one of the MSs does not contain all the sources to be observed,(4) such MS is by chance selected as a "first" MS to be processed by thepipeline.
9Additional limitations of the Interferometric Pipeline are:•

While the IF pipeline calibration and flagging tasks do include low signal-to-noise
heuristics, they will produce poor results if the calibrators are too weak.

•

The IF pipeline does not perform automated science target flagging. Templateflagging files (names like uid*_flagtargetstemplate.txt, one per ASDM)
are provided for users to add their own flags; these will be applied during thehifa_flagtargets task if the pipeline imaging script is re-run.

•

In order to increase delivery rates of data to PIs, the archived imaging products
may be binned in frequency, limited in the imaged field of view, and/or restricted toa subset of sources. Users can make the missing products by making small
modifications to the scripts that are archived with the data.

•

The frequency ranges for interferometric continuum identification and subtraction
are done in an automated manner that works well over a very broad range of
observing modes and source properties. In some cases (e.g. hot core line
emission, noisy broadband continuum), it is expected that better results can beobtained by more careful examination of individual sources and/or spectral
windows. If the data are heavily binned in frequency before this task is run, theresults may be compromised. The user can edit cont.dat (sec. 7.6) and rerun
sections of the imaging pipeline to obtain their own continuum subtracted visibilities
and new line images.

•

The IF PL imaging steps use the “effective channel bandwidth” from the raw data
file to calculate the theoretical image sensitivity and hence clean thresholds. This
information is not correctly entered for ALMA data from Cycles 2 and earlier; as aresult, the clean thresholds will be higher than intended when such data is run
through the imaging pipeline.

•

The pipeline does not include science target self-calibration. Therefore, thepipeline imaging products of bright sources will be dynamic range limited.

•

The interferometric imaging pipeline commands should work with measurement
sets calibrated outside the pipeline, but this has not been tested extensively andmay have as-yet undetermined failure modes.

Additional limitations of the Single-dish Pipeline are:•

The SD pipeline does not work properly with ephemeris targets•

The frequency ranges for single dish line identification and spectral baseline
subtraction are done in an automated manner that has been optimized to detectmoderate channel width (wider than 100 channels) emission lines at the center ofa spectral window. It is expected that better results can be obtained by more careful
examination of individual sources and/or spectral windows. The following cases
are most strongly affected:§ Narrow emission lines (less than 100 channels), especially in TDM mode.
§ Emission at the edge of a spectral window.
§ Cubes with a “forest” of emission lines.

•

The SD pipeline imaging results may be unusable if there is emission in the “off”
position and/or if the atmospheric line features still remain in the calibrated data.
10

•
•

The number and total size of all ASDMs run through the SD pipeline cannot exceed
limits set by the server specification (e.g. 50GB raw data for 64GB RAM).
If hsd_baseline is run manually to subtract the baseline in an individual spw, blflag
and hsd_imaging should be run before proceeding to subtract the baseline fromthe next spw. Otherwise, hsd_baseline will overwrite the baseline solutions for theprevious spws.

A list of pipeline “known issues” that arise after the publication date of this document ismaintained on the ALMA Science Portal at http://almascience.org/processing/sciencepipeline#KI. This list will be updated as issues are discovered during the cycle.

4

Pipeline Versions & Documentation

4.1 Obtaining the Pipeline
A link to the version of CASA 5.4.0 that includes the ALMA pipeline is available, alongwith installation instructions and supporting documentation, from the Science Pipeline
section of the ALMA Science Portal at http://www.almascience.org (under the“Processing” tab, or directly at http://almascience.org/processing/science-pipeline). If any
issues are encountered with CASA 5.4.0 installation, please contact the ALMA Helpdesk
via the link on the ALMA Science Portal.
The pipeline tasks become available by starting up CASA using the command:% casa --pipeline
Or to run CASA with pipeline tasks using MPI (multi-core parallelization):% mpicasa -n 8 casa --pipeline

4.2 Pipeline-related Documentation
The User documentation currently relating to the Pipeline is also available from the
Science Pipeline section of the Science Portal referenced above. This includes:•

ALMA Science Pipeline User’s Guide: This document.

•

ALMA Science Pipeline Reference Manual: Description of individual Pipelinetasks.

Examples of common re-imaging modifications to the IF pipeline script are given at:https://casaguides.nrao.edu/index.php/ALMA_Imaging_Pipeline_Reprocessing.
In addition, Chapters 10, 11, and 13 of the ALMA Technical Handbook provide more
information in calibration in general, how Quality Assurance is performed, and how datais archived.

4.3 Pipeline & CASA Versions
The pipeline heuristic tasks have a specific version number, and are bundled with a
specific version of CASA. These versions are reported in the README file that is archived
with the pipeline data products, and are also reported on the Home page of the WebLog
for each pipeline-processed dataset (see Figure 9 for an example).
11

In general, the version of Pipeline+CASA used in ALMA Operations to calibrate and image
the archived data will be the same as the publicly posted pipeline version available fromthe “Obtaining CASA” page, but they can be slightly different (e.g. for bugs that have an
operational workaround but which are fixed in the posted public version). There is a
“Pipeline
VersionTracker”
available
fromthe
Science
Portalat
http://almascience.org/processing/science-pipeline#version, which lists the versions of
CASA+Pipeline used in ALMA Operations as well as the versions which should be used
for any restoring or reprocessing of the data from the same cycle (as described elsewherein this document).

4.4 Pipeline and CASA tasks
The pipeline heuristics are written as special CASA tasks, where they appear with a hif_or hifa_ (for interferometric) or hsd_ (for single-dish) prefix. They can be viewed andexecuted within CASA in exactly the same way as other CASA tasks (if one has launched
CASA with “--pipeline”). For example, one can view the possible inputs for the task
hifa_importdata by typing inp hifa_importdata. To see all the tasks available in
CASA, type tasklist. The pipeline heuristics use CASA tasks wherever possible toperform the data reduction or imaging. E.g. the pipeline bandpass calibration task
hifa_bandpass calls the CASA bandpass task, and the interferometric imaging task
hif_makeimages calls the CASA imaging task tclean.
The standard pipeline processing recipes are deterministic and should always give thesame result for the same data. However, the CASA pipeline tasks are designed to behighly flexible, so that they can have the default inputs over-ridden with user-specified
values, or be added, subtracted, or rearranged to produce alternative processing recipes.
This enables a manual “mix and match” mode for data reduction and imaging thatcombines standard CASA pipeline tasks with other CASA commands or python code to
produce scripts that are better tuned to the idiosyncrasies of a specific dataset. The exact
pipeline commands that will reproduce the standard recipe are delivered with eachdataset, in a script called member.<mous_uid>.<recipe>.casa_pipescript.py (see section
4.3 below). One could edit and add to that script to implement “mixed mode” processing.
Some common “manual mode” modifications are presented in the Sec. 7 below. A
complete list of the variables for each pipeline task is given in the ALMA Science Pipeline
Reference Manual.
CASA pipeline tasks operate like other CASA tasks. In particular, the scope of variables
follow CASA rules. This means that when CASA pipeline tasks are called with noarguments, they will assume any previously defined variables used by the task, whereas
calling the same task with at least one argument will not. For example, typing thecommands “refant=’DA45’; hifa_gfluxscale();” will use the antenna named‘DA45’ as the reference antenna, whereas typing “refant=’DA45’;
hifa_gfluxscale(pipelinemode=’interactive’);” will result in the pipeline
picking a reference antenna according to its default heuristics.
This document, along with the Pipeline Reference Manual, describe key aspects of the
CASA pipeline tasks. Important changes to other CASA tasks are documented in the
Release Notes for the corresponding CASA release, available from the CASA page athttps://casa.nrao.edu/current_casa.shtml.

12

5Data Processing Files

5.1 Archived scripts
There are several scripts that are archived with ALMA data deliveries. These are fully
described in the document ALMA QA2 Data Products for Cycle 6 available from ALMA
Science
Portalunderthe“Processing”
tab(or
directlyathttps://almascience.nrao.edu/processing/qa2-data-products). The particular scripts for a
specific dataset should also be described in the QA2 report archived with the data
products. This report will vary based on how the data were processed (pipeline calibrated+ imaged; pipeline calibrated & manually imaged; manually calibrated + pipeline imaged,
manually calibrated + manually imaged).
The scripts produced by the pipeline are returned to the user called
member.<mous_uid>.<recipe>.casa_pipescript.py and,member.<mous_uid>.<recipe>.casa_piperestorescript.py. The former includes all
pipeline processing commands that were run on the data, and is more fully described
below. The latter “restores” the data, which means that rather than re-running thepipeline calibration commands, it uses previously derived calibration and flagging tables
and applies them directly to the raw data, producing a calibrated measurement set. Thisis much quicker and requires less computing resources than re-running the pipeline
calibration commands. "However, expert users should be aware that if the latter, fastermethod it used, then the state of the measurement sets are not exactly the same as in acomplete run (e.g. the model of the calibrators will not be set)."Every delivery package includes a master script called
member.<mous_uid>.scriptForPI.py that will reproduce the calibrated data regardlessof how it was processed. This script is not created by the pipeline, but instead by thedata packaging software so that it is produced for both pipeline and manually reduced
data. For pipeline calibrated data, it will simply invoke the pipeline-produced
casa_piperestorescript.py or casa_pipescript.py.
Using scriptForPI.py is the recommended and fastest method of obtaining
calibrated ALMA data from the delivery. However, one can also run the pipeline
casa_piperestorescript.py using the steps in Sec. 5.3. To change the calibration results,
one would re-run the commands in casa_pipescript.py after making modifications, as
described in Sec. 6.

5.2 Pipeline “Helper” text files
Both the IF and SD pipeline use a number of text files that, if present, will affect thepipeline results (e.g. by applying manually identified flags or by updating calibrator fluxes
or antenna positions before calculating the calibration tables). These files are particularly
useful for users to over-ride the default pipeline behavior when re-running the pipeline athome, as more fully described in Sec. 7 below. They include the following:•

flux.csv: This file is used by the IF pipeline to update the flux of calibrators. The
flux of the calibrator with the “AMPLITUDE” intent will affect the overall flux scaleof the data. If this file is not present where the pipeline is run, the pipeline willattempt to contact the ALMA source catalog for previously recorded flux densities,
and if that doesn’t succeed, the fluxes in the ASDM(s) will be used, representing
the best flux estimate at the time the SB was executed. If no flux value appears ineither the flux.csv file or the ASDM, a flux of 1.0 Jy is adopted.
13

•

jyperk.csv: This file is used by the SD pipeline to set the “Kelvin to Jansky”
calibration factors which set the overall fluxscale of the data. If it is not present
where the pipeline is run, then a conversion factor of unity is assumed.

•

antennapos.csv: This file is used by the IF pipeline to update the positions of theantenna elements. If it is not present where the pipeline is run, the positions in theASDM(s) will be used.

•

uid*flagtemplate.txt: This file is used to add additional CASA flagging commands
that will be applied to the data before the calibration tables are calculated.

•

uid*flagtsystemplate.txt: This file is used to add additional CASA flagging
commands that will be applied to the tsys spws before the calibration tables arecalculated.

•

uid*flagtargetstemplate.txt: This file is used to add additional CASA flagging
commands that will be applied to the data after the calibration tables are calculated,but before science target imaging is performed.

•

cont.dat: This file is used to specify the continuum frequency ranges used forconstructing the continuum images and creating the continuum-subtracted cubes.
This particular file is described in more detail below and in the reimaging casaguide
https://casaguides.nrao.edu/index.php/ALMA_Imaging_Pipeline_Reprocessing

The format of each of these files is given in Sec. 7.

5.3 The Pipeline script to restore calibrated MSs:
casa_piperestorescript.py
To restore data calibrated by the pipeline, one can either run scriptForPI.py as
described in ALMA QA2 Data Products for Cycle 6 available from ALMA Science
Portal under the “Processing” tab (or directly at
https://almascience.nrao.edu/processing/qa2-data-products), or one can run thepipeline-provided casa_piperestorescript.py :• Create rawdata/, working/, and products/ subdirectories•

Download the raw ASDMs from the archive and put them in rawdata/. Make surethe naming of the raw ALMA data is consistent with those provided in the script(e.g. if the data ends in .asdm.sdm then move to names which do not have thissuffix).

•

Copy or move *manifest.xml, *caltables.tgz, *flagversions.tgz, and*calapply.txt to products/•

Copy uid*casa_piperetorescript.py to casa_piperestorescript.py to working/•In working/, start casa –pipeline, and execfile(“casa_piperestorescript.py”)

5.3.1 Results from running the SD casa_piperestorescript.py• A calibrated MS for each ASDM with a name like "*.ms". Note that the baseline
subtraction is not done for the restored calibrated MS.
Running the script through hsd_baseline command will additionally create:• A calibrated, baseline subtracted MS for each ASDM with a name like "*ms_bl".
The pipeline "automatic" mode reproduces the baseline subtraction. If instead theuser may want to set the mask ranges to be used for baseline subtraction, CASA
14

task sdbaseline() is recommended to use. In such case, please be aware of that
WebLog is not generated for CASA tasks. If the baseline subtraction is done with
the CASA task sdbaseline(), any further Pipeline tasks cannot be used.
Running the script through hsd_blflag command will result in:• flagging based on the baseline rms for each ASDM. The hsd_blflag commandhas to be run after hsd_baseline at least once. In the standard operation,
hsd_baseline and hsd_blflag are repeated twice to improve the quality of baseline
detection.
Running the script through hsd_imaging command will additionally create:• native resolution images per spectral window, antenna, and source.

5.4 The Pipeline processing script: casa_pipescript.py
5.4.1 Format of casa_pipescript.py
The complete set of pipeline commands are given in the script casa_pipescript.py. Thisis a python script that includes all tasks and parameter values, in the correct sequence,that were used for the pipeline run. A typical casa_pipescript.py script for a SD Pipelinerun (including both calibration+imaging steps) is shown in Figure 1, while a typical IF
pipeline script that includes both pipeline calibration and imaging steps is shown in Figure2.
For data that were both calibrated and imaged in the pipeline (including all SD data run
through the pipeline), the casa_pipescript.py file will include both the calibration andimaging pipeline commands. For IF data that were calibrated in the pipeline but imaged
outside of the pipeline, the casa_pipescript.py file will only include the IF calibration
pipeline commands (up to the line “# Start of pipeline imaging commands” line), and thearchived data will include a separate scriptForImaging.py script containing the manual(CASA) imaging commands. If instead the IF data were manually calibrated and pipeline
imaged, the scriptForPI.py would include the manual (CASA) calibration commands, andthe IF pipeline imaging commands (those following the line “# Start of pipeline imaging
commands” line in Figure 5) would be included in a separate scriptForImaging.py script.

15

__rethrow_casa_exceptions=Trueh_init()hsd_importdata(vis = ['uid___A002_X877e41_X452'])
hsd_flagdata(pipelinemode=’automatic’) ## Uses
*flagtemplate.txt
h_tsyscal(pipelinemode=’automatic’)
hsd_tsysflag(pipelinemode=’automatic’)
hsd_skycal(pipelinemode=’automatic’)
hsd_k2jycal(pipelinemode=’automatic’) ## Uses jyperk.csv
hsd_applycal(pipelinemode=’automatic’)
hsd_baseline(pipelinemode=’automatic’)
hsd_blflag(pipelinemode=’automatic’)
hsd_baseline(pipelinemode=’automatic’)
hsd_blflag(pipelinemode=’automatic’)
hsd_imaging(pipelinemode=’automatic’)h_save()Figure 1: Example of the Single Dish Pipeline calibration + imaging script casa_pipescript.py. The “##”comment line identifies the pipeline command that uses one of the pipeline “helper” text files described in Sec.
5.2.

16

__rethrow_casa_exceptions = Trueh_init()try:
hifa_importdata(dbservice=False,
vis=['uid___A002_X877e41_X452'], session=['session_1'])## Uses flux.csv
hifa_flagdata(pipelinemode="automatic")##Uses *flagtemplate.txt
hifa_fluxcalflag(pipelinemode="automatic")
hif_rawflagchans(pipelinemode="automatic")
hif_refant(pipelinemode="automatic")
h_tsyscal(pipelinemode="automatic")
hifa_tsysflag(pipelinemode="automatic")
hifa_antpos(pipelinemode="automatic") ## Uses antennapos.csv
hifa_wvrgcalflag(pipelinemode="automatic")
hif_lowgainflag(pipelinemode="automatic")
hif_setmodels(pipelinemode="automatic")
hifa_bandpassflag(pipelinemode="automatic")
hifa_spwphaseup(pipelinemode="automatic")
hifa_gfluxscaleflag(pipelinemode="automatic")
hifa_gfluxscale(pipelinemode="automatic")
hifa_timegaincal(pipelinemode="automatic")
hif_applycal(pipelinemode="automatic")
hif_makeimlist(intent='PHASE,BANDPASS')
hif_makeimages(pipelinemode="automatic")
hif_makeimlist(per_eb=True, intent='CHECK')
hif_makeimages(pipelinemode="automatic")
hifa_imageprecheck(pipelinemode="automatic")
hif_checkproductsize(maxproductsize=350.0, maxcubesize=40.0,
maxcubelimit=60.0)
hifa_exportdata(pipelinemode="automatic")# Start of pipeline imaging commands
hif_mstransform(pipelinemode="automatic")
hifa_flagtargets(pipelinemode="automatic")## Uses *flagtargetstemplate.txt
hif_makeimlist(specmode='mfs')## Uses cont.dat
hif_findcont(pipelinemode="automatic") ## Modifies cont.dat
hif_uvcontfit(pipelinemode="automatic") ## Uses cont.dat
hif_uvcontsub(pipelinemode="automatic")
hif_makeimages(pipelinemode="automatic")## Uses cont.dat
hif_makeimlist(specmode='cont')## Uses cont.dat
hif_makeimages(pipelinemode="automatic")## Uses cont.dat
hif_makeimlist(specmode=’cube’)## Uses cont.dat
hif_makeimages(pipelinemode="automatic")## Uses cont.dat
hif_makeimlist(specmode=’refBW’)## Uses cont.dat
hif_makeimages(pipelinemode="automatic")## Uses cont.dat
finally:h_save()Figure 2: Example of an IF Pipeline casa_pipescript.py script for a dataset that was run through the
Pipeline for both calibration and imaging. The “##” comment lines identify the pipeline commands that uses
one of the pipeline “helper” text files described in Sec. 5.2

17

The tasks names, order, and parameter values in the casa_pipescript.py script reflect
the processing recipe used for each individual delivery. Additionally, the pipelinemodeparameter is set to “automatic” for each task. In this mode, the task takes the default
settings for each tasks and only a limited number of parameters are exposed for editing
by a user. Setting the pipeline mode to “interactive” will usually enable the values of alarger number of parameters to be changed. To see the variables available in the pipeline“automatic” mode, type “pipelinemode=’automatic’; inp <task_name>” at the
CASA command line. To see the variables available in the pipeline “interactive” mode,
type “pipelinemode=’interactive’; inp <task_name>”. See the ALMA
Science Pipeline Reference Manual for more details, and Sec. 6 below for examples ofmodified pipeline re-runs.

5.4.2 Results from running the single dish casa_pipescript.py
Running the script will create:•

A calibrated, baseline subtracted MS for each ASDM with a name like
uid___A00X_XXXX_XXX.ms_bl.

•

Baseline subtracted image cubes of the the science targets in *.image format (1per spectral window, all antennas combined, at the native correlator frequency
spacing).

•

A pipeline-*/html directory containingo The Pipeline WebLog (see Sec. 0).
o The casa_commands.log file (see Sec. 5.5).

5.4.3 Results from running the interferometric casa_pipescript.py
Running the script through the first hif_makeimages command (calibrator imaging) will
create:•A calibrated MS for each ASDM with a name like uid___A00X_XXXX_XXX.ms.
This ms includes both calibrator and science data and all spectral windows, withthe raw data in the DATA column, and the calibrated continuum + line data in the
CORRECTED column.

•

Continuum images of the bandpass, phase, and (if present) check source
calibrators (1 per spectral window, in *.image format). To view a *.image file e.g.
use casaviewer image_file_name.

•

A pipeline-*/html directory containing:o The Pipeline WebLog (see Sec. 0).
o The casa_commands.log file (see Sec. 5.5).

Running the script through hif_mstransform command will additionally create:•

A calibrated MS for each ASDM containing only science target data (only sciencetargetsandspectralwindows),
with
a
name
like
uid___A00X_XXXX_XXX_target.ms. This ms will have the raw data in the
DATA column, and the calibrated continuum + line data in the CORRECTED
column.
18

Running the script through hif_uvcontsub command will result in:•

The science-target only MS (uid___A00X_XXXX_XXX_target.ms), now withthe calibrated continuum + line data in the DATA column, and the calibrated
continuum subtracted data in the CORRECTED column.

Running the script through the final hif_makeimages command (science target spectralline imaging) will additionally create:•

Per-spw continuum images, aggregate continuum images, and continuum
subtracted image cubes of at least some science targets (the number of targets
may be reduced either automatically – see Sec. 9.23 – or manually).

5.5 CASA equivalent commands file: casa_commands.log
The casa_commands.log file is written by the pipeline to provide a list of the equivalent
CASA task commands (as opposed to Pipeline tasks) used by the Pipeline to process adataset. While this log cannot be used to create a CASA reduction script that is identicalto the Pipeline processing, it provides executable CASA commands with the parametersettings used by the pipeline. The log is commented to indicate which Pipeline stage thetasks were called from and why. The imaging commands given in this file can be easily
modified to produce new imaging products with more finely tuned inputs (e.g. interactive
masks and deeper cleaning thresholds).

6

Modifying a Pipeline Run using casa_pipescript.py

6.1 Pipeline re-processing considerations
As a rule, it does not make sense to rerun the casa_pipescript.py exactly as delivered,since this will merely reproduce the calibrated measurement set (which for IF Pipeline
calibrated data is much more easily generated using scriptForPI.py or
casa_piperestorescript.py to “restore” the calibration, as described in Sec. 5.1 above)
and/or already-delivered products. Instead, it is likely that the user may want to redo thecalibration after some modifications or produced modified imaging products. This section
describes a few of the more common calibration and imaging changes for both the IF and
SD Pipeline tasks. See the ALMA Science Pipeline Reference Manual for more
complete details on the pipeline tasks and their inputs.
Re-running the pipeline can be very resource-intensive, both from a compute-time anddisk-space perspective. For the compute time, an idea of how long the pipeline took whencan be inferred from the WebLog (using the Execution Duration shown on the top of the“Home” page of the WebLog – see Figure 9, or the Task Execution Statistics that arelisted for each task in the “By Task” part of the WebLog – see e.g. Figure 14). Those
times, however, reflect the run times using the ALMA Operations processing clusters,
which have 64 – 256 GB RAM, and likely use parallel processing (multi-core) for imaging.
Concerning disk space, to re-run SD or IF pipeline calibration, it is advisable to have asystem with at least 8 GB RAM, and 50 – 75 GB free disk space per ASDM. To re-run the
IF imaging pipeline, it is advisable to have a system with ≥64 GB RAM, and the available
disk space needs to be 10 – 100 times the expected size of the final imaging products.

19

The above resource requirements for the IF imaging pipeline are rather daunting.
However, in practice, it is unlikely that the imaging pipeline commands would need to be
rerun in their entirety. It would be much quicker and demand much less computing
resources to only image the sources and or spectral windows (spw) or channels ofinterest, at an appropriate spectral resolution. This can be done by finding thecorresponding tclean() command in the provided casa_commands.log file, modifying
it as desired, and running it in CASA. These commands work on the measurement set
created by the pipeline hif_mstransform() command, so that part of the imaging
script would need to be run first.
Please contact ALMA via the Helpdesk if assistance is needed with data reprocessing.

6.2 Preparing to run casa_pipescript.py
The following steps describe how to modify and re-run the Pipeline, starting from theproducts and directory structure created after downloading the data:•

Create rawdata/, working/, and products/ subdirectories•

Copy uid*casa_pipescript.py to casa_pipescript.py in the working/ directory.

•

To re-run IF calibration: copy flux.csv, antennapos.csv (if present), anduid*flagtemplate.txt to the working/ directory (there will be one flagtemplate.py
file per EB). Depending on the delivery method, flux.csv and antennapos.csv
are likely to be found in uid*auxproducts.tgz which will need to be unzipped.

•

To re-run IF imaging also:
Copy uid*flagtargetstemplate.txt to the working/ directory (note there is one
per ASDM).
Copy cont.dat (there will only be one per MOUS) to the working/ directory.

•

To re-run SD calibration & imaging: copy jyperk.csv and uid*flagtemplate.txt to
the working/ directory (there will be one file per ASDM).

In the rawdata directory:•

Make sure the naming of the raw ALMA data is consistent with those provided in
the script (e.g. if the data ends in .asdm.sdm then move to names which do nothave this suffix).

•

Modify the pipeline “helper” files as desired (e.g. editing the *flagtemplate.txt file
to add any additional flags – see Sec. 7 for other options).

•

Edit casa_pipescript.py to only include the pipeline steps you wish to repeat (e.g.
commenting out the findcont or imaging steps, which are very computationallyexpensive).

•

Start the version of CASA containing Pipeline using casapy --pipeline

You are now ready to run the script by typing execfile(‘casa_pipescript.py’).
Alternatively, you can sequentially execute individual commands from
casa_pipescript.py, stopping at any point to run other CASA commands (plotms, etc).
Note that to re-run the Pipeline multiple times, it is recommended to start each timefrom a clean working directory containing only CASA “helper” text files and thecasa_pipescript.py script.
20

6.3 Modifying Calibration Commands
The pipeline calibration commands can be modified to produce different results.
For instance, problematic datasets (ASDMs) can be excluded from the processing byediting the “vis=” and “session=” lists in hifa_importdata or hsd_importdata
tasks in the casa_pipescript.py script.
As a second example, a user-specified prioritized reference antenna list can be specified
via the “refant” variable in calibration tasks, over-riding the pipeline reference antenna
heuristics, by switching to pipelinemode=’interactive’ and passing the desired refant list.
E.g.
hifa_bandpass(pipelinemode="interactive", refant=’DV06,DV07’)
See the Pipeline Reference Manual for more options.
Another use case is to keep the default pipeline commands, but to change the values in
the Pipeline “helper” text files to e.g. change the flux scaling, or update antenna positions(see Sec. 7 for details). The new values will be used when the relevant hif_ commands
are run.

6.4 Modifying IF Pipeline Imaging Commands
The pipeline imaging commands can be modified to produce different products. Typicalreasons for re-imaging include:•

Imaging improvements to be gained from interactively editing an emission specificclean mask and cleaning more deeply. The Cycle 5 pipeline generates a cleanmask automatically (see Sec. 9.31 for specifics). Cases with moderate to strong
emission (or absorption) can benefit from deeper clean with additional interactive
clean masking, with the most affected property being the integrated flux density.

•

Non-optimal continuum ranges. The pipeline uses heuristics that attempt to identify
continuum channels over a very broad range of science target line properties.
Particularly for strong line forests (hot-cores) and occasionally for TDM continuum
projects the pipeline ranges can be non-optimal -- too much in the first case andtoo little in the second.

Other science goal driven reprocessing needs may include:•

Desire to use wide image channels in the imaging stage to increase the S/N ofcubes.

•

Desire to use a different Briggs Robust image weighting than the default ofrobust=0.5 (smaller robust = smaller beam, poorer S/N; larger robust = larger
beam, better S/N).

•

Desire to uv-taper images to to increase the S/N for extended emission.

•

Desire to use different continuum frequency ranges than determined by thepipeline, by modifying the cont.dat file (Sec. 7.6).

Some re-imaging examples are given in a “CASA Guide” athttps://casaguides.nrao.edu/index.php/ALMA_Imaging_Pipeline_Reprocessing. There
you will find examples of the following:•

Making aggregate continuum image with all channels of all spectral windows.

•

Redoing continuum subtractions with user-derived continuum ranges.
21•Making a cube of subset of sources, spectral windows, with a different robust
weight and channel binning factor.

6.5 Manual imaging after running casa_pipescript.py
6.5.1 SD Data
After calibration with the script casa_pipescript.py, it is possible to re-image using the
CASA Single Dish task, sdimaging, with user-defined parameters. As mentioned earlier,the Single Dish Pipeline creates a calibrated MS with a filename extension of “*.ms_bl”
for each ASDM. The sdimaging command will make images of all MS that are specified
in the infiles parameter. For other parameters in sdimaging, refer to the*casa_commands.log file.
Note that the images included in the delivery package have the native frequency
resolution and a cell size of one-ninth of the beam size, as recommended in the SD “CASA
Guide” (https://casaguides.nrao.edu/index.php/M100_Band3_SingleDish_4.5). If you
want to change the frequency resolution and cell size, we recommend that you import thedelivered FITS data cubes into CASA and regrid them using the CASA task imregrid.
It is also possible to revise the baseline subtraction using your preferred mask range
instead of the pipeline-defined range. We recommend doing this on the images using the
CASA tasks imcontsub or sdbaseline during your own manual calibration (refer tothe CASA Guides).

6.5.2 IF Data
For IF data that are pipeline calibrated but manually imaged, the imaging commands will
be included in a separate scriptForImaging.py script, containing all the CASA
commands used to create the delivered products. In order to use this imaging script afterusing casa_pipescript.py to recalibrate, the science spectral windows must first be “split”
out from the calibrated measurement sets and the measurement sets output with a.split.cal suffix. To perform the split, in CASA e.g.:
split('uid__A002_X89252c_X852.ms',
outputvis='uid__A002_X89252c_X852.ms.split.cal',
spw='17,19,21,23')
The science spectral windows are specified in the Pipeline WebLog (Home > Observation
Summary > Measurement Set Name > Spectral Setup, in the ID column) or can be
determined
using
the
CASA
task
listobse.g.
listobs(‘uid___A002_X89252c_X852.ms’), where the results will be reported in
the CASA logger.
If the pipeline-calibrated data is restored using scriptForPI.py, that script will perform thesplit command for the user
If a script named scriptForFluxCalibration.py is present in the script directory, thismust also be executed prior to running scriptForImaging.py.

6.6 Manipulating the Pipeline Context
It is recommended to always run the Pipeline using python scripts. New Pipeline
runs/scripts need to be initialized using h_init in order to create an empty pipeline
context.
22

If the script is modified to only run a subset of the pipeline tasks, the context should besaved after the last task by using h_save. To resume the run, use h_resume to load thesaved context before executing any pipeline tasks. See the ALMA Science Pipeline
Reference Manual for more information.
To use the Pipeline to calibrate a dataset but to e.g. insert a different bandpass table into
the processing, the following procedure should be followed:
•
•
•
•
•

7

Run the pipeline until the end of the bandpass table creation task
hifa_bandpass.
View the calibration tables that Pipeline will use with hif_show_calstate.
Export the calibration tables Pipeline uses to a file on disk using
hif_export_calstate.
Edit the calstate file to replace the name of the Pipeline-created bandpass tablewith the one it is wanted to use instead.
Import the edited calstate file back to the context using
hif_import_calstate and resume the processing.

Description of Pipeline “Helper” Text Files

As mentioned in Sec. 5.2, both the IF and SD pipeline use a number of text files that areread by various pipeline tasks (as indicated in Figure 1), and which affect the pipeline
results (e.g. by applying manually identified flags or by updating calibrator fluxes orantenna positions before calculating the calibration tables). These files are particularly
useful for users to over-ride the default pipeline behavior when re-running the pipeline athome, as described in the following section. Below we describe all of the currently
available control files, identifying whether they are used by the IF pipeline, SD pipeline,or both in the subsection heading.

7.1 IF Pipeline: flux.csv
From Cycle 4 onward, the fluxes of standard ALMA quasar calibrators at the observed
frequencies for each spw are written into the ASDM, using extrapolated values calculated
from entries in the ALMA Source Catalog available at the time of observation. These
fluxes are sometimes updated subsequently (thereby bracketing the observation in time),
allowing for more accurate interpolated fluxes to be used for the absolute flux calibration.
Since the pipeline is usually run days to weeks after an observation, ALMA staff run acommand (getALMAFlux) outside of the pipeline to get the best-available fluxes forstandard calibrators at the time the pipeline is run (in a future release, this query will bedone automatically by the pipeline). These are written into the flux.csv text file, which isthen read in by the pipeline hifa_importdata task (if it exists in the directory where thepipeline is run) and used to over-ride the values in the ASDM. The new flux value of theflux calibrator (the source with intent=AMPLITUDE) is then used in the subsequent
hif_setmodels task. Values for the other calibrator intents (BANDPASS, PHASE,CHECK) are also updated, but these values are only shown for comparison against thevalues derived from the pipeline calibration calibration (both are shown in a table in thehifa_gfluxscale stage of the WebLog – see Sec. 9.14). If the flux.csv file is notavailable where the pipeline is run, then the values entered into the ASDM by the online
system are used. If flux.csv file is not available where the pipeline is run and there areno values entered by the online system, then a flux of 1 Jy will be assumed.
23

The format of the flux.csv file is shown in Figure 3 below. It contains one row for every
spw of every calibrator (intents of AMPLITUDE, BANDPASS, PHASE or CHECK ) in every
ASDM in the MOUS. This file can be edited by users and the pipeline re-run in order toscale the fluxes of each ASDMs to a different value for the AMPLITUDE calibrator.
Changing the values of other calibrators will not have an effect on the calibration.
ms,field,spw,I,Q,U,V,spix,comment
uid___A002_Xd0adbe_Xd5a.ms,0,25,0.8818,0.0,0.0,0.0,-0.750167691515,"# field=J1550+0527
intents=AMPLITUDE,ATMOSPHERE,BANDPASS,POINTING,WVR origin=DB age=4 queried_at=2018-09-14 20:47:41 UTC # +0.0849Jy, freq=109.517GHz, spec_index=-0.750+-0.168, Band3/7_separation=0 days, spixAge=-14 days, Band3age=5
days, setjy parameters for field 0 (J1550+0527): spix=-0.7502, reffreq='109.5167GHz',
fluxdensity=[0.881754,0,0,0], au.getALMAFluxcsv v1.4207 executed on 2018-09-14 21:27:01 UT"
uid___A002_Xd0adbe_Xd5a.ms,0,27,0.8926,0.0,0.0,0.0,-0.750167691515,"# field=J1550+0527
intents=AMPLITUDE,ATMOSPHERE,BANDPASS,POINTING,WVR origin=DB age=4 queried_at=2018-09-14 20:47:42 UTC # +0.0868Jy, freq=107.746GHz"
uid___A002_Xd0adbe_Xd5a.ms,0,29,0.9630,0.0,0.0,0.0,-0.750167691515,"# field=J1550+0527
intents=AMPLITUDE,ATMOSPHERE,BANDPASS,POINTING,WVR origin=DB age=4 queried_at=2018-09-14 20:47:42 UTC # +0.1032Jy, freq=97.383GHz"
uid___A002_Xd0adbe_Xd5a.ms,0,31,0.9767,0.0,0.0,0.0,-0.750167691515,"# field=J1550+0527
intents=AMPLITUDE,ATMOSPHERE,BANDPASS,POINTING,WVR origin=DB age=4 queried_at=2018-09-14 20:47:43 UTC # +0.1061Jy, freq=95.563GHz"
uid___A002_Xd0adbe_Xd5a.ms,1,25,0.1435,0.0,0.0,0.0,-0.577110147696,"# field=J1458+0416
intents=ATMOSPHERE,PHASE,WVR origin=Source.xml age=N/A queried_at=N/A # +-0.0076Jy, freq=109.517GHz,
spec_index=-0.577+-0.121, Band3/7_separation=0 days, spixAge=+4 days, Band3age=4 days, setjy parameters forfield 1 (J1458+0416): spix=-0.5771, reffreq='109.5167GHz', fluxdensity=[0.143542,0,0,0]"
uid___A002_Xd0adbe_Xd5a.ms,1,27,0.1449,0.0,0.0,0.0,-0.577110147696,"# field=J1458+0416
intents=ATMOSPHERE,PHASE,WVR origin=Source.xml age=N/A queried_at=N/A # +-0.0077Jy, freq=107.746GHz"

Figure 3: Example of a flux.csv file used by the interferometric pipeline (one per MOUS) – blanklines added here for readability.

The original flux.csv file written by the pipeline upon the initial run of the hif_importdata
task, starts out with "origin=Source.xml" as part of the comment on all lines. From thispoint onward, there are 4 things that can happen depending on the status of database(DB) server.
0) DB server responds to both pipeline and au.getALMAFluxcsv: i.e., normal
Operations.
Result: pipeline will write "origin=DB", and getALMAFluxcsv writes the date/time of thesuccessful query to the end of the first line of each calibrator's entries, and appends "#+/- xxxx Jy, freq=xxxx GHz" to the rest of the lines.
1) DB server is down for a long time: both queries fail (pipeline and subsequentau.getALMAFluxcsv).
Result: pipeline will leave the comment "origin=Source.xml" in place, andau.getALMAFluxcsv appends this string to every line: "# failed to contact calibrator
catalog server!"2) DB server is down briefly (during pipeline query) but is restored before
getALMAFluxcsv runs.
Result: pipeline will leave the comment "origin=Source.xml", and getALMAFluxcsv
writes the date/time of the successful query to the end of the first line of each calibrator's
entries; and in the other lines, it writes the following: "# +/- xxxx Jy, freq=xxxx GHz
3) DB server goes down after the pipeline query but before getALMAFluxcsv runs,
24

Result: pipeline will write "origin=DB", and getALMAFluxcsv will append "# failed tocontact calibrator catalog server!" to every line.

7.2 SD Pipeline: jyperk.csv
ALMA single-dish observations do not include observations of absolute amplitude
calibrators. Instead, the observatory conducts regular observations of standard singledish calibrators and stores them in an observatory database. When the single-dishpipeline is run, ALMA staff run commands outside the pipeline to extract the best value ofthese “Kelvin to Jansky” calibration factors, based on the observing date, frequency, Tsys,and source elevation. The appropriate values are written into a the jyperk.csv text filethat is read and applied when the hsd_k2jycal task is run.
The format of the jyperk.csv file is shown in Figure 4 below. It contains one row for every
spw in every ASDM in the MOUS. This file can be edited by users and the pipeline re-run
in order to scale the fluxes of each ASDM to a different value.
MS,Antenna,Spwid,Polarization,Factor
uid___A002_Xb1d975_Xf65.ms,PM02,17,I,43.785
uid___A002_Xb1d975_Xf65.ms,PM02,19,I,43.782
uid___A002_Xb1d975_Xf65.ms,PM02,21,I,43.664
uid___A002_Xb1d975_Xf65.ms,PM02,23,I,43.63
uid___A002_Xb1d975_Xf65.ms,PM02,25,I,43.638
uid___A002_Xb1d975_Xf65.ms,PM02,27,I,43.64
uid___A002_Xb1d975_Xf65.ms,PM02,29,I,43.641
uid___A002_Xb1d975_Xf65.ms,PM04,17,I,43.785
uid___A002_Xb1d975_Xf65.ms,PM04,19,I,43.782
uid___A002_Xb1d975_Xf65.ms,PM04,21,I,43.664
uid___A002_Xb1d975_Xf65.ms,PM04,23,I,43.63
uid___A002_Xb1d975_Xf65.ms,PM04,25,I,43.638
uid___A002_Xb1d975_Xf65.ms,PM04,27,I,43.64
uid___A002_Xb1d975_Xf65.ms,PM04,29,I,43.641
uid___A002_Xb1cc39_X1e46.ms,PM02,17,I,43.782
uid___A002_Xb1cc39_X1e46.ms,PM02,19,I,43.778
uid___A002_Xb1cc39_X1e46.ms,PM02,21,I,43.661
uid___A002_Xb1cc39_X1e46.ms,PM02,23,I,43.627
uid___A002_Xb1cc39_X1e46.ms,PM02,25,I,43.635
uid___A002_Xb1cc39_X1e46.ms,PM02,27,I,43.636
uid___A002_Xb1cc39_X1e46.ms,PM02,29,I,43.638
uid___A002_Xb1cc39_X1e46.ms,PM04,17,I,43.782
uid___A002_Xb1cc39_X1e46.ms,PM04,19,I,43.778
uid___A002_Xb1cc39_X1e46.ms,PM04,21,I,43.661
uid___A002_Xb1cc39_X1e46.ms,PM04,23,I,43.627
uid___A002_Xb1cc39_X1e46.ms,PM04,25,I,43.635
uid___A002_Xb1cc39_X1e46.ms,PM04,27,I,43.636
uid___A002_Xb1cc39_X1e46.ms,PM04,29,I,43.638

Figure 4: Example of a jyperk.csv file used by the single-dish pipeline (one per MOUS)

7.3 IF Pipeline: antennapos.csv
The position of every antenna in an interferometric observation must be known in orderto properly transfer the calibration from the phase calibrator to the science targets. If these
positions have errors, it will lead to phase errors in the imaging of the science target(increasing with telescope position error and separation between the phase calibrator andscience target).
The antenna positions are calculated by special observatory observations taken outsideof PI science observing, and the positions stored in an observatory database. This
database is queried at the time at the time of an SB execution, and the appropriate
antenna positions are written into the ASDM. These positions are sometimes updated
25

subsequently, especially if the observation happened close to an array reconfiguration or
if an array element was recently moved.
Since the pipeline is usually run days to weeks after an observation, ALMA staff run
commands outside of the pipeline to get the best-available antenna positions at the time
the pipeline is run (in a future release, this query will be done automatically by thepipeline). These are written into the antennapos.csv text file, which is then read in by thepipeline hifa_antpos task (if it exists in the directory where the pipeline is run) and used
to over-ride the values in the ASDM.
The format of the antennapos.csv file is shown in Figure 5 below. It contains one row forevery antenna in every ASDM in the MOUS. This file can be edited by users and thepipeline re-run in order to correct antenna position errors.
name,antenna,xoff,yoff,zoff,comment
uid___A002_Xbb63ba_X18b0.ms,DA42,1.27536e-04,-3.54105e-04,-2.38014e-04,
uid___A002_Xbb63ba_X18b0.ms,DA46,1.98098e-04,-5.34528e-04,-3.65393e-04,
uid___A002_Xbb63ba_X18b0.ms,DA49,1.69321e-04,-2.81896e-04,-1.76309e-04,
uid___A002_Xbb63ba_X18b0.ms,DA52,-4.06882e-05,3.45109e-04,3.15047e-04,
uid___A002_Xbb63ba_X18b0.ms,DA62,-1.79249e-04,2.50696e-04,7.12701e-05,
uid___A002_Xbb63ba_X18b0.ms,DV03,-3.92453e-04,2.85912e-04,3.14499e-04,
uid___A002_Xbb63ba_X18b0.ms,DV08,-2.76083e-04,7.41071e-04,1.87197e-04,
uid___A002_Xbb63ba_X18b0.ms,DV14,-5.41156e-05,2.61746e-04,3.44329e-04,
uid___A002_Xbb63ba_X18b0.ms,DV15,-1.21313e-04,3.67910e-04,1.49062e-04,
uid___A002_Xbb63ba_X18b0.ms,DV23,1.73257e-04,1.36402e-04,-1.23099e-04,
uid___A002_Xbb63ba_X18b0.ms,DV25,3.12879e-03,-5.08802e-03,-2.87630e-03,
uid___A002_Xbb63ba_X18b0.ms,PM03,1.78948e-04,-5.00918e-04,-2.14580e-04,
uid___A002_Xbb63ba_X1626.ms,DA42,1.27536e-04,-3.54105e-04,-2.38014e-04,
uid___A002_Xbb63ba_X1626.ms,DA46,1.98098e-04,-5.34528e-04,-3.65393e-04,
uid___A002_Xbb63ba_X1626.ms,DA49,1.69321e-04,-2.81896e-04,-1.76309e-04,
uid___A002_Xbb63ba_X1626.ms,DA52,-4.06882e-05,3.45109e-04,3.15047e-04,...

Figure 5: Example of a antennapos.csv file used by the interferometric pipeline (one per MOUS); the offset
units are in meters. Corrections that are comparable, or larger than the observing wavelength areconsequential.

7.4 Both IF & SD Pipeline: uid*flagtemplate.txt anduid*flagtsystemplate.txt
The pipeline flagging heuristics may prove inadequate, and users may wish to add
additional flagging commands to exclude these data from the calibration. These manuallyidentified flags can be introduced to any Pipeline reduction by editing theuid*flagtemplate.txt files that are provided with the archived pipeline products andrerunning the pipeline calibration steps. There should be one file for every MS that needs
additional flagging, with a name matching the MS uid. The flag commands can be any
valid CASA flagdata command. For interferometric data, use the <AntID> syntax toflag only cross-correlation data for <AntID>, while for single dish data use the“<AntID>&&*” syntax to flag both cross- and auto-correlation data for <AntID>, and the“<AntID>&&&” syntax to flag auto-correlation data for <AntID>. Examples of the syntax touse in editing these files are given at the top of the files uid*flagtemplate.txt (see Figure
6).
These flag files will be picked up by the hifa_flagdata/hsd_flagdata tasks which
are run before the calibration tasks, therefore excluding the manually identified data frombeing used to generate the calibration tables.
26

Since the tsys spectra are calculated from a different ASDM subtable, any commands
that the user desires to flag the tsys spectral windows have to be applied differently by
the pipeline, so have to be put into the separate flagtsystemplate.txt file. The flagging
syntax is the same, only that those commands should refer to tsys spectral windows inparticular.
# User flagging commands file for the calibration pipeline## Examples# Note: Do not put spaces inside the reason string !
## mode='manual' antenna='DV02;DV03&DA51' spw='22,24:150~175' reason='QA2:applycal_amplitude_frequency'## mode='manual' spw='22' field='1' timerange='2018/02/10/00:01:01.0959~2018/02/10/00:01:01.0961'
reason='QA2:timegaincal_phase_time'## TP flagging: The 'other' option is intended for bad TP pointing# mode='manual' antenna='PM01&&PM01' reason='QA2:other_bad_pointing'## Tsys flagging:# mode='manual' antenna='DV02;DV03&DA51' spw='22,24' reason='QA2:tsysflag_tsys_frequency'#

mode='manual' timerange='2016/12/05/03:55:30.1440'
reason='QA2:applycal_outlier_amp'
mode=’manual’ antenna=’PM02&&&’ reason=’PRTSIR2995’

Figure 6: Example of a uid*flagtemplate.text file used by both the interferometric and single-dish pipeline(one per ASDM)

7.5 IF Imaging Pipeline: uid*flagtargetstemplate.txt
Currently, there are no science target specific flagging heuristics in the IF pipeline, so
errant data may be present, affecting the science imaging products. Users should
examine the science data (e.g. using the CASA task plotms, or examining at the MS
using the CASA viewer). If bad data are found, flagging commands can be added to theuid*flagtargetstemplate.txt files that are provided with the archived pipeline products to
exclude these data from subsequent imaging. There should be one file for every MS thatneeds additional flagging, with a name matching the MS uid. As for theuid*flagtemplate.txt files, the flag commands can be any valid CASA flagdata
command. If these files are found in the directory where the pipeline is run, they will bepicked up by the hifa_flagtargets task and applied to the data before science target
imaging.

7.6 IF Imaging Pipeline: cont.dat
The pipeline-identified continuum frequency ranges, in LSRK units, for each spectral
window of each source are entered into a file called cont.dat that is delivered with thepipeline products. This file lists the LSRK frequency ranges that were used to make theper-spw and aggregate continuum images, and for fitting and subtracting the continuum
for the image cubes. When this file is in the directory where the pipeline is (re)run, thepipeline will use these entries directly instead of using its own heuristics (via thehif_findcont task) to determine them. Therefore, a user can edit this file (or create
their own) in order to use a different continuum range. Alternatively, a user-defined filename can be passed as an argument to the hif_makeimlist task. An example
cont.dat file is shown in Figure 7.

27

Field: G09_0850-0019
SpectralWindow: 17
NONE
SpectralWindow: 19
337.659971874~339.253995016GHz LSRK
SpectralWindow: 21
SpectralWindow: 25
349.755169752~351.067897111GHz LSRK
351.271057297~351.380451244GHz LSRK

Figure 7: Example of a cont.dat file used by the interferometric pipeline (one per MOUS). This example is for
an MOUS that has 5 spectral windows; the entry for spw 21 is empty and spw 23 is omitted, which will resultin the hif_findcont command determining the frequency ranges for these spectral windows.

The behavior of hif_findcont and the subsequent continuum subtraction andcontinuum and line imaging commands is as follows:•

If the spw line in cont.dat is followed by one or more frequency ranges,
hif_findcont will not run its heuristics on the spw. The task hif_uvcontfit
will use these frequency ranges to fit and subtract the continuum from this spw.
Subsequent continuum images will include only these frequency ranges for thisspw, and the spw line cubes will be made from the continuum subtracted data.

•

If the spw line is followed by a line containing “NONE”, hif_findcont will not run
its heuristics on the spw (if the delivered cont.dat file contains spw entries with
“NONE”, this indicates that the hif_findcont task failed to find any continuum
frequency ranges). The task hif_uvcontfit will skip fitting this spw. Subsequent
continuum images will include the full frequency range for this spw (logging awarning), and the spw line cubes will have had no continuum subtraction
performed.

•

If an spw is not followed by a frequency range or is missing from cont.dat when
hif_findcont is run, then it will try to find the frequency ranges, and these will
be used to make subsequent continuum images, and for continuum subtraction.

28

8

The Pipeline WebLog

This section gives an overview of the Pipeline WebLog, which is a collection of webpages
with diagnostic messages, tables, figures, and “Quality Assurance" (QA) scores. It isreviewed, along with the pipeline calibration and imaging products, as part of the ALMA
Quality Assurance process, but also provides important information to investigators on
how the pipeline calibration and imaging steps went.
The section describes common elements to the single dish and interferometric Pipeline
WebLogs. Subsequent sections present descriptions of the SD- or IF- specific “By Task”
part of the WebLog.

8.1 Overview
The WebLog is a set of html pages that give a summary of how the calibration of ALMA
data proceeded, of the imaging products, and provides diagnostic plots and Quality
Assurance (QA) scores. The WebLog will be in the qa directory of an ALMA delivery. Toview the WebLog, untar and unzip the file using e.g. tar zxvf *weblog.tgz . This will
provide a pipeline*/html directory containing the WebLog, which can be viewed using a
web browser e.g. firefox index.html.
The WebLog provides both an overview of datasets and details of each each pipeline
processing. Therefore most calibration pages of the WebLog will first give a single“representative” view, with further links to a more detailed view of all the plots associatedwith that calibration step. Some of these (those produced by the CASA tasks plotms andplotbandpass) will have a “Plot command” link that provides the CASA command toreproduce the plot (see Figure 8). For some stages, the detailed plots can be filtered by
a combination of outlier, antenna and spectral window criteria. Where histograms are
displayed, in modern web browsers it is possible to draw boxes on multiple histograms to
select the plots associated with those data points. All pipeline stages are assigned a QA
score to give an “at a glance” indication of any trouble points.
WebLog Quick Tips
• Any text written in blue, including headings, is a link to further information.
• To go straight to viewing calibrated science target visibilities, go to By Task >hif_applycal and scroll down to the bottom.
• Histograms can have selector boxes drawn on them using the mouse.
• The CASA commands for re-creating many of the WebLog plots are provided.

29

Figure 8: Example of WebLog plot with a "Plot command" link (>_) that provides the CASA command for
reproducing the plot.

8.2 Navigation
To navigate the main pages of the WebLog, click on items given in the bar at the top of
the WebLog home page. Also use the Back button provided at the upper right on someof the WebLog sub-pages. Avoid using “back/previous page” on your web browser(although this can work on modern browsers). Throughout the WebLog, links are denotedby text written in blue and it is usually possible to click on thumbnail plots to enlarge them.

8.3 Home Page
The first page in the WebLog gives an overview of the observations (proposal code, datacodes, PI, observation start and end time), a pipeline execution summary (pipeline &CASA versions, link to the current pipeline documentation, pipeline run date and duration),and an Observation Summary table. Clicking on the “environment” link next to the CASA
version will open a popup detailing hardware and software used, and number of cores if
MPI; see Figure 10). Clicking on the bar at the top of the home page (see Figure 9)
enables navigation to By Topic or By Task.

30

Figure 9: WebLog Home Page. The Navigation Bar is circled in red.

Figure 10: Processing Environment popup window

The Observation Summary table lists all the measurement sets included in the pipeline
processing, grouped by observing “sessions”. Each measurement set is calibrated
independently by the pipeline. For data that have been run through the imaging stages ofthe pipeline, two MS will be listed – the original one including all data and spectral
windows, and a target.ms containing only calibrated science target data. The tableprovides a quick overview of the ALMA receiver band used, the number of antennas, thestart/end date and time, the time spent on source, the array minimum and maximum
baseline length, the rms baseline length and the size of that measurement set. To view
the observational setup of each measurement set in more detail, click on the name of it
to go to its overview page.

8.3.1 Measurement set Overview pages
Clicking on the measurement set name in the Observational Summary table brings up
the Measurement set Overview page (Figure 11). Each measurement set Overview
page has a number of tables: Observation Execution Time, Spatial Setup (includes
mosaic pointings), Antenna Setup, Spectral Setup and Sky Setup (includes
elevation vs. time plot). For more information on the tables titled in blue text, click onthese links. There are additionally links to Weather, PWV, Scans and Telescope
31

Pointings (in the case of Single Dish observations) information. Two thumbnail plots,
which can be enlarged by clicking on them, show the observation structure either as Field
Source Intent vs Time or Field Source ID vs Time. To view the CASA listobs output
from the observation, click on Listobs Output.

Figure 11: Measurement Set Overview Page. Click on the table headings in blue for more information about
each.

8.4 By Topic Summary Page
The By Topic summary page provides an overview of all Warnings and Errors triggered,a Quality Assessment overview in Tasks by Topic and Flagging Summaries for theprocessing.

8.5 By Task Summary Page
The By Task summary page (Figure 12) gives a list of all the pipeline stages performedon the dataset. It is not displayed per measurement set as the Pipeline performs eachstep on every measurement set sequentially before proceeding to the next step; e.g. it
will import and register all measurement sets with the Pipeline before proceeding toperform the ALMA deterministic flagging step on each measurement set. The name ofeach step on the By Task page is a link to more information. On the right hand side of thepage are colored bars and scores that indicate how well the Pipeline processing of thatstage went. Green bars should indicate a fairly problem-free dataset, while blue or redbars indicate less than perfect QA scores. Encircled symbols to the left of each task name(“?”, “!” or “x”), indicate that there are informative QA messages on the subtask pages.
32

Figure 12: By Task summary view. The page has been truncated so both the top and bottom can be seen. Eachpipeline stage is listed, along with its QA score (colored bars to the right), and links to the CASA logs andscripts.

8.5.1 CASA logs and scripts
At the bottom of the By Task summary page are links to the CASA logs and supporting
files and scripts. These include the complete CASA log file produced during the pipeline
run, the pipeline restoration scripts described in Sec. 5.1: casa_pipescript.py andcasa_piperestorescript.py, and the casa_commands.log file described in Sec. 5.5.

8.6 Task Pages
Each task has its own summary page that is accessed by clicking on the task name on
the By Task summary page or in the left navigation menu from other pages. The task
pages provide the outcome, or the representative outcome, of each Pipeline task
executed. For a fast assessment of the calibration results, go straight to theapplycal page. At the top of the page will be any Task Notification (see Figure 13). These
provide informative messages or warnings generated from the QA scoring and should be
reviewed carefully.
33

Figure 13: hifa_tsysflag task page, showing the task notifications at the top, and diagnostic plots (Tsys for each
spw grouped by MS). Further down on the page are flagging summary tables. To see the sub-page for this task,click on the measurement set name in blue above each set of plots. This will take you to a page of detailed plots
for individual MS/antenna/spectral windows (see Figure 15 for an example).

At the bottom of each task page are expandable sections for Pipeline QA, Input
Parameters and Task Execution Statistics, and links to the CASA log commands forthe specific task. An example is given in Figure 14.

Figure 14: Bottom of the hifa_timegaincal page, showing the expanded Pipeline QA section, as well as the
expandable sections for Input Parameters, Task Execution Statistics and link to the CASA logs for this stage.

34

8.6.1 Task sub-pages and plot filtering
Most sub-pages have further links in order to access a more detailed view of the outcomeof each task. These links are often labelled by the measurement set name. Some of theseplots can be filtered by entering one or more MS, antenna, or spectral window in theappropriate box. Still others have histograms of various metrics than can be selectedusing the cursor in a drop-and-drag sense to outline a range of histogram values anddisplays the plots for the MS/antenna/spw combinations that are responsible for thosehistogram values. An example of these subpages and plot filtering is given in Figure 15 –Figure 17 below, using the By Task > hifa_tsysflag: Flag Tsys calibration pages.

Figure 15: Unfiltered view of the hifa_tsysflag sub-page. The page is arrived at by clicking on the measurement
set link from the hifa_tsysflag task page (Figure 13). Only the first row of plots are shown; many more appear
below (one for each MS, antenna, spw combination). This page has histograms of three metric scores based on
the median Tsys that can also be used to filter the plots that are displayed.

35

Figure 16: Same as Figure 15, but with a specific MS, Tsys window, and antenna filter set. The corresponding
plots are displayed below, and their metric scores are shown by blue shading in the histogram plots.

Figure 17: Same as Figure 15, but filtering to the plot of interest by using the mouse to draw a grey box on
the highest histogram values in the RMS deviation from Average Median Tsys histogram plot (upper right).
To clear the grey box filters on the histograms, click on any white space in the histograms.

36

8.7 WebLog Quality Assessment (QA) Scoring
Pipeline tasks have scores associated with them in order to quantify the quality of thedataset and the calibration. When scores are calculated per ASDM, the QA score is taken
as the lowest of the per-ASDM scores. The scores are between 0.0 and 1.0 and arecolorized according to the following table:
Score
0.90-1.00
0.66-0.90
0.33-0.66
0.00-0.33

Color
Green
Blue
YellowRed

Comment
Standard/Good
Below standard
Warning
Error

8.7.1 Interferometric Pipeline QA Scores
Pipeline Task
hifa_importdata

hifa_flagdata

Pipeline QA
Scoring Metric
Checking that therequired calibrators
are present

Determining
percentage ofincremental flagging

Score
1.0 all present
0.1 subtracted for missing bandpass orflux calibrator
1.0 subtracted for missing phasecalibrator or Tsys calibration
0.5 subtracted for existing processing
history
For the following flag types: 'online','shadow', 'qa0', 'before' and 'applycal':Score is 0 if flag fraction is >= 50%
Score is 1 if flag fraction is <= 5%
Score is linearly interpolated between
0 and 1 for fractions between 60% and5%.

hifa_fluxcalflag

Determining
percentage ofincremental flagging

hif_rawflagchans

Determining
percentage of data
flagged due todeviant channels in
rawdata
Determining if areference antenna

hif_refant

The percentages are accumulated foreach flagging type. For example,
13.0% data flagged yields:
1-(0.130-0.05)/(0.60-0.05) = 0.85
Additional 0%-5% flagging: score=1.0;
flagging 5%-50% => 1.0...0.5; >50%:score=0.0. If there are mapped spws
as a result of flagging, the score forthat ms gets set to 0.66.
0% flagged: score=1.0;
100% flagged: score=0.037h_tsyscal
hifa_tsysflag
hifa_antpos

hifa_wvrgcalflag

centrally located andnot flagged a lot
Calculate tsys
calibration tables.
Determining
percentage ofincremental flagging
Determining if
antenna positionalcorrections wereappliedChecking phase
RMS improvement

Additional 0%-5% flagging: score=1.0;
flagging 5%-50% => 1.0...0.5; >50%:score=0.0
1.0 if no corrections needed; 0.9 if one
or more antennas were corrected.

0.0 if RMS(before)/RMS(after) < 1, 0.5... 1.0 for ratios between 1 and 2, and
1.0 for ratios > 2 (See section 9.9)
hif_lowgainflag
Determining
Additional 0%-5% flagging: score=1.0;
percentage offlagging 5%-50% => 1.0...0.5; >50%:
incremental flagging score=0.0
hifa_bandpassflag – Determining fraction For each intent, a linear score
flagging portionof unflagged data
representing the fraction of unflaggedthat is newly flagged data that got newly flagged. I.e., if 60%
was previously flagged, and among
remaining 40%, half got flagged (suchthat total flagging is now 80%), thenthe score for this intent would be 0.5.
hifa_bandpassflag – Judging phase andtwo algorithms: Wiener entropy andbandpass portion
amplitude solution
derivative deviation, and signal-toflatness per antenna, noise ratio (scores: Wiener entropy:
spectral window and error function with 1-sigma deviation ofpolarization (interim
0.001 from 1.0; derivative deviation:measure, futureerror function with 1-sigma deviation ofscoring will be based 0.03 for the outlier fraction; signal-toon data withnoise ratio: error function with 1-sigma
solutions applied)deviation of 1.0 for the signal-to-noise
ratio)
hifa_spwphaseup
Determining fraction Score is the fraction of spectralof spectral windows windows for which phase solutions arewithout phase
unmapped to expected number ofsolutions transferred spectral windows
from other windows
hifa_gfluxscaleflag
Determining fraction First calculate the newly flaggedof previously
fraction per intent, then it combines the
unflagged data thatscores from all intents by multiplying
is newly flagged.
them; i.e., if it evaluated AMPLITUDE
and PHASE, and it got a score of 0.6
and 0.2 for those respectively, then the
total score would be 0.6*0.2=0.12
hifa_gfluxscale
Determining SNR of Fitted flux values with SNR < 5.0 are
fitted flux values
assigned a score of 0.0, SNR > 20.0 ascore of 1.0, and a linearly scaled
value in between. If there are missing
38

hifa_timegaincal

hifa_applycal

Determining X-Y /X2-X1 phasesolution deviations

Determining
percentage ofincremental flagging
hifa_imageprecheck For representative
target report
expectedsynthesized beamand sensitivity andchoose robust to
most closely matchthe PI request

derived fluxes, then the ratio of thenumber of derived values to thenumber of spws is computed, and if
this is lower than the SNR score, it
supercedes the score for that ms.
An additional score assesses
consistency of the derived flux
densities in different spws – it is known
that for low-SNR (e.g. narrow) spws,
the gfluxscale derived flux density canbe systematically biased high.
r_SPW = (derived flux for SPW) /(catalog flux for SPW), K_SPW =r_spw / r_max_snr_spw. Q_total =Max( | 1 - K_spw| ). The QA scorescales from 1.0 for Q_total=0, to ascore of 0.75 if Q_total=20%, andfurther lower QA scores for larger
Q_total.
Standard deviation of X-Y phasedifference converted to path length: 1.0
if lower than 4.25e-6 m, 0.0 if higher
than 7.955e-2 m, with an exponential
decrease in between. Standard
deviation of X2-X1 phase differencesof subsequent integrations converted
to path length: 1.0 if lower than 3.08-e5 m, 0.0 if higher than 2.24e-2 m, withan exponential decrease in between.
NB: The high limits are currently
dummies. Determining their realistic
values is still under development.
Additional 0%-5% flagging: score=1.0;
flagging 5%-50% => 1.0...0.5, >50%:score=0.0
0.5 if no representative target /frequency is found.
If the PI’s desired AR is found in the
ASDM, a QA score is assigned asfollows:
1.0: PI’s AR achieved in both axes withrobust=0.5
0.85: PI’s AR achieved in both axes
with a different robust
0.5: At least one axis is out of range,
but the beam area is still within the PI’s
range
0.25: not even the beam area can bebrought within the PI’s desired range
by changing robust.
39hif_makeimlist

hif_makeimages(non-checksource
calibrators &science targets)

Determine if
expectedtargets/spw will be
imagedDetermine if noise isclose to theoretical

hif_makeimages(Checksources)

Determine if phasetransfer worked forchecksource, by
checking fordecorrelation andpositional shifthif_exportdata

Determining Pipeline
products have been
exported
Determine if proper
files were created
Determine if any
target flags wereappliedDetermine if
continuum could be
identified for all spw

hif_mstransform
hifa_flagtargets
hif_findcont

hif_uvcontfit
hif_uvcontsub

Determine if
continuum could be
fit
Determine if
continuum could be
subtracted

1.0 when all objects with desired intent
appear in list for all science spw
Ratio of sensitivity measured in nonpbcor image in a 0.3 – 0.2 PB annulus
compared to 0.25 times clean
threshold;Score=1 when ratio is 1 or lower
Score=0 when ratio is 5 or higher
Geometric mean of following twoscores:Score1=1.0 – abs[(catalog position –fitted position)/beam size]
Score2=1.0-abs[gfluxscale flux – fitted
image flux)/gfluxscale flux]
Score2 is an indicator of significant
decorrelation, although the score may
be low for other reasons, including low
calibrator S/N.
1.0 when files successfully exported
1.0 when target.ms files successfully
created; otherwise 0.0
1.0 when no flagging commands
applied1.0 if continuum frequency ranges
found for all spw. If not, the score is
the number of spws with ranges
divided by the number of expected
spws.
1.0 if continuum fit table created
Always set = 1.0

8.7.2 Single-Dish Pipeline QA scores
Pipeline Task

Pipeline QA Scoring
Metrichsd_importdata Checking that therequired calibrators arepresent

Score
1.0 ATMOSPHERE intents are present
0.5 subtracted for existing processing
history
0.5 subtracted for existing model data
1.0 one continuous observing session40

1.0 all source coordinate are present
hsd_flagdata

Determining percentage
of incremental flagging

hsd_k2jycal

Checking that all Kelvinto-Jy conversion factors
are provided

hsd_applycal

Determining percentage
of incremental flagging

hsd_baseline

Checking that one ormore than one emission
line is detected by linefinder.
Determining percentage
of incremental flagging
per source per spw.

hsd_blflag

0 < score < 1 === 60% < fraction flagged< 5%" (for 'online', 'shadow', 'qa0', 'before'and 'applycal')
where "0 < score < 1 === HIGH% <fraction flagged < LOW%" means• Score is 0 if flag fraction >= HIGH%• Score is 1 if flag fraction <= LOW%
Score is linearly interpolated between 0
and 1 for fractions between HIGH% andLOW%
0.0 Missing Kelvin-to-Jy conversion
factor for some data
Additional 0%-5% flagging: score=1.0;
flagging 5%-50% => 1.0...0.5, >50%:score=0.0. QA score calculation isrestricted to scans matching the'TARGET' intent if present. If no'TARGET' intent data is present, awarning is raised, and the QA score
calculation reverts to using scans for any
available intent.
1.0 there is more than one emission line
detected in at least one spw.
0.0 No line is detected in all spw.
Additional 0%-5% flagging: score=1.0;
flagging 5%-50% => 1.0...0.5, >50%:score=0.041

9The “By task” WebLog for Interferometric Data

This section describes navigation of the Task sub-pages for each Interferometric Pipelinetask starting from the “By Task” tab. For a fuller description of each task, refer to ALMA
Science Pipeline Reference Manual.

9.1 hifa_importdata
In this task, ASDMs are imported into measurement sets, Binary Data Flags are applied,and some properties of those MSs are calculated. The WebLog page shows a summaryof imported MSs, and flux densities of calibrators. Flux densities are read from the Source
table of the ASDM, which is recorded by the online system at the time of observation byinterpolating in frequency the recent measurements in the calibrator catalog (seeAppendix C of the ALMA Technical Handbook). The “new” ALMA source catalog isqueried, and if that returns sensible values, the ASDM values are replaced with the query
values. This allows observatory measurements performed after the science observation
to be used to obtain a better estimate of the calibrator flux density. The flux densities foreach calibrator in each science spw in each MS are written to the file flux.csv in thecalibration/ subdirectory of a data delivery package. The values in this file can be edited
before continuing with the pipeline execution if you first use the importonly option ofeppr.executeppr.

9.2 hifa_flagdata
In this task, the online (XML format) flags, which includes the QA0 flags for antenna
pointing calibration failures, are applied along with the rest of the deterministic flagging
reasons (unwanted intents, autocorrelations, shadowed antennas, and TDM edge
channels). The WebLog page shows the percentage of flagged data per MS. The “Before
Task” column contains only the effect of the Binary Data Flags (BDF) applied duringhifa_importdata. The additional flags are applied in the order of columns shown in
the table. The percentage in each column reflects the additional amount of data flagged
when applying this flag reason. The QA score for this stage is based on
BDF+QA0+online+template+shadow flagging.

9.3 hifa_fluxcalflag
The WebLog shows any flagging or spwmap that was required. If the flux calibrator is asolar system object, known lines in the object (e.g. CO in Titan’s atmosphere) are flaggedby this task. If more than 75% of a given spw is flagged on the flux calibrator for thisreason, then a spwmap is calculated to transfer the flux scale from another spw. The
WebLog shows if any flagging or spwmap was required.
In Mars, Venus, Titan and Neptune, 12CO is flagged in all ALMA bands. In Mars, Venus,
and Titan, 13CO is also flagged. In Titan, HCN, H13CN and HC15N are also flagged, as
is HCN v2=1. Finally, in Titan CH3CN is flagged up through Band 8. The frequency widththat is flagged is based on published spectra of 1 or 2 transitions of the species. For othertransitions, this frequency width is scaled to maintain consistent velocity widths. Also,
because the flags are applied in topocentric frame, the final width that is flagged is further
broadened by the maximum relative velocity between the object and the geocenter(computed over the next decade). Here is a detailed list of topocentric frequency ranges
flagged:Mars:42

[115.204, 115.338], [230.404, 230.672], [345.595, 345.997], [460.773, 461.309],[691.071, 691.875], [806.184, 807.120], [921.265, 922.335], # CO
[110.190, 110.212], [220.377, 220.421], [330.555, 330.621], [440.721, 440.809],[661.001, 661.133], [771.108, 771.260], [881.196, 881.370], # 13CO
Venus:[115.206, 115.337], [230.407, 230.669], [345.600, 345.992], [460.779, 461.303],[691.081, 691.865], [806.194, 807.110], [921.277, 922.323], # CO
[110.192, 110.210], [220.380, 220.418], [330.560, 330.616], [440.727, 440.803],[661.011, 661.123], [771.118, 771.250], [881.208, 881.358], # 13CO
Titan:[114.92,115.67],[229.49,231.74],[343.82,347.62],[458.29,463.80],[687.75,694.66],[803.4
6,809.85], # CO
[110.18,110.22],[220.28,220.52],[330.36,330.82],[440.42,441.15],[660.60,661.53],[770.6
5,771.72],[880.73,881.82], # 13CO
[88.45,88.81],[176.73,177.80],[264.96,266.81],[353.29,355.72],[441.74,444.52],[618.45,
622.16],[707.01,710.74],[795.56,799.31],[883.82,887.86], # HCN[86.04,86.07],[172.05,172.16],[258.04,258.27],[430.02,430.45],[601.95,602.60],[773.88,
774.64],[859.84,860.62], # HC15N[86.33,86.35],[172.63,172.73],[258.91,259.11],[431.44,431.88],[603.98,604.56],[776.48,777.16],[862.72,863.42] # H13CN[177.192,177.286],[178.088,178.184],[265.782,265.924],[267.128,267.270],[354.365,35
4.555],[356.161,356.351],[442.942,443.178],[445.184,445.422],[620.058,620.390],[623.
197,623.529],[708.596,708.974],[712.182,712.562],[797.117,797.543],[801.149,801.577],[885.619,886.091],[890.096,890.572] # HCN_v2=1[91.938,92.008],[110.304,110.409],[128.659,128.809],[147.039,147.209],[165.415,165.6
08],[183.792,184.006],[202.168,202.403],[220.543,220.798],[238.916,239.194],[257.289,257.587],[275.660,275.980],[294.030,294.371],[312.399,312.761],[330.766,331.149],[349.205,349.534],[367.572,367.920],[385.938,386.303],[404.301,404.683],[422.735,423.062],[441.098,44
1.440],[459.459,459.814],[477.881,478.186],[496.239,496.557] # CH3CN
Neptune:[113.99,116.51],[226.98,234.52],[339.97,351.54],[454.95,467.55],[685.93,696.57],[802.9
2,810.58] # CO

9.4 hif_rawflagchans
This task was designed to detect severe baseline-based anomalies prior to performing
antenna-based calibration. These bad data are often due to hardware problems during
the observation. Outlier channels and outlier baselines are detected in the uncalibrated
visibilities of the bandpass calibrator.
The WebLog page links to the images of the values used for flagging. Any flagged data
are shown on the plots along with a summary of all flagging performed in this task. The
following two rules are used to evaluate the need for flagging:431) "bad quadrant" matrix flagging rule:This starts with the "baseline" vs. "channel" flagging view. In this view, some data
points may already be flagged, e.g. due to an earlier pipeline stage.
First, outliers are identified as those data points in the flagging view whose value
deviates from the median value of all non-flagged data points by a threshold factor
times the median absolute deviation (MAD) of the values of all non-flagged data points,
where the threshold is 'fbq_hilo_limit' (default: 8.0).
In formula: flagging mask = (data - median(all non-flagged data)) > (MAD(all nonflagged data) * fbq_hilo_limit)
Next, the flagging view is considered as split up in 4 quadrants of channels (sincesome problems manifest in only one or more quadrants), and each antenna isevaluated separately as follows:a) Select baselines belonging to antenna and select channels belonging to quadrant.
b) Determine number of newly found outlier datapoints within selection.
c) Determine number of originally unflagged datapoints within selection.
d) Determine fraction of "number of newly found outliers" over "number of originallyunflagged datapoints".
e) If the latter fraction exceeds the fraction threshold 'fbq_antenna_frac_limit' (default:0.2), then a flagging command is generated that will flag all channels within theevaluated quadrant for the evaluated antenna.
f) Otherwise, no action is taken (i.e. the newly found outlier datapoints are notindividually flagged by this rule),
Next, the flagging view is still considered as split up in 4 quadrants of channels, andeach baseline is evaluated separately, as follows:a) Select baseline and select channels belonging to quadrant.
b) Determine number of newly found outlier datapoints within selection.
c) Determine number of originally unflagged datapoints within selection.
d) Determine fraction of "number of newly found outliers" over "number of originallyunflagged datapoints".
e) If the latter fraction exceeds the fraction threshold 'fbq_baseline_frac_limit' (default:1.0), then a flagging command is generated that will flag all channels within theevaluated quadrant for the evaluated baseline.
f) Otherwise, no action is taken (i.e. the newly found outlier datapoints are notindividually flagged by this rule).
2) "outlier" matrix flagging rule:
Data points in the flagging view are identified as outliers if their value deviates fromthe median value of all non-flagged data points by a threshold factor times the medianabsolute deviation of the values of all non-flagged data points, where the threshold is'fhl_limit' (default: 20.0).
In formula: flagging mask = (data - median(all non-flagged data)) > (MAD(all nonflagged data) * fhl_limit)
Flagging commands are generated for each of the identified outlier data points.
If the number of data points in the flagging view are smaller than the minimum sample'fhl_minsample' (default: 5), then no flagging is attempted.

449.5 hif_refant
An ordered list of preferred reference antennas is calculated, with preference given to
central array location and low flagging fraction through the following score:
Refant score = [1-(normalized distance from center)] + [1-(normalized fraction of gooddata)]The WebLog page shows that list, and the score for each antenna can be found in thecasa log for this stage.

9.6 h_tsyscal
System temperature (Tsys) as a function of frequency is calculated from the atmosphericcalibration scan data by the online system at the time of observation. These spectra areimported to a table of the MS during hifa_importdata. In hifa_tsyscal, these
spectra are copied into a CASA calibration table by the gencal task, which flags channels
with zero or negative Tsys. The WebLog shows the mapping of Tsys spectral windows toscience spectral windows, and plots Tsys before flagging. Mapping is often necessary
because in data from Cycles 0-5, Tsys can only be measured in TDM windows on the 64station baseline correlator.

9.7 hifa_tsysflag
This task flags the Tsys cal table created by the hifa_tsyscal pipeline task. Erroneous Tsysmeasurements of several different kinds are detected, including anomalously high Tsys over anentire spectral window, spikes or “birdies” in Tsys, and discrepant “shape” or Tsys as a functionof frequency. Details are provided in the WebLog for each kind of flagging performed, and all of
the Tsys spectra are plotted again. In these plots, all of the anomalies should be gone.
Tsysflag provides six separate flagging metrics, where each metric creates its own flagging viewand has its own corresponding flagging rule(s). In the current standard pipeline, all six metrics areactive, and evaluated in the order set by the parameter "metric_order" (default: 'nmedian,derivative, edgechans, fieldshape, birdies, toomany').

1) Metric 1: "nmedian"A separate view is generated for each polarisation and each spw. Each view is a matrix withaxes "time" vs. "antenna". Each point in the matrix is the median value of the Tsys spectrum
for that antenna/time.
The views are evaluated against the "nmedian" matrix flagging rule, where data points are
identified as outliers if their value is larger than a threshold-factor * median of all non-flagged
data points, where the threshold is 'fnm_limit' (default: 2.0).
Individual sources are evaluated separately with the default setting of fnm_byfield=True; thisis to prevent elevation differences between targets from causing unnecessary flags (mostly
affects high frequencies).
Flagging commands are generated for each of the identified outlier data points.

2) Metric 2: "derivative"
A separate view is generated for each polarisation and each spw. Each view is a matrix withaxes "time" vs. "antenna". Each point in the matrix is calculated as follows:o calculate "valid_data" as the channel-to-channel difference in Tsys for that
antenna/timestamp (for unflagged channels)
o calculate median(abs( valid_data - median(valid_data) ) ) * 100.0

45

The views are evaluated against the "max abs" matrix flagging rule, where data points are
identified as outliers if their absolute value exceeds the threshold "fd_max_limit" (default: 5).
Flagging commands are generated for each of the identified outlier data points.

3) Metric 3: "edgechans"
A separate view is generated for each spw and each of these intents: ATMOSPHERE,
BANDPASS, and AMPLITUDE. Each view contains a "median" Tsys spectrum where for eachchannel the value is calculated as the median value of all selected (spw,intent) Tsys spectrain that channel (this combines data from all antennas together).
The views are evaluated against the "edges" vector flagging rule, which flags all channels fromthe outmost edges (first and last channel) until the first channel for which the channel-tochannel difference first falls below a threshold times the median channel-to-channel difference,
where the threshold is "fe_edge_limit" (default: 3.0).
A single flagging command is generated for all channels newly identified as "edge channels".

4) Metric 4: "fieldshape"
A separate view is generated for each spw and each polarization. Each view is a matrix withaxes "time" vs. "antenna". Each point in the matrix is a measure of the difference of the Tsys
spectrum for that time/antenna from the median of all Tsys spectra for that antenna/spw in the"reference" fields that belong to the reference intent specified by "ff_refintent" (default:"BANDPASS").
The exact fieldshape value is calculated as: 100 * mean(abs(normalized tsys - reference
normalized tsys)), where a 'normalized' array is defined as: "array / median(array)"
The views are evaluated against the "max abs" matrix flagging rule, where data points are
identified as outliers if their absolute value exceeds the threshold "ff_max_limit" (default: 5).

5) Metric 5: "birdies"A separate view is generated for each spw and each antenna. Each view contains a"difference" Tsys spectrum calculated as:"channel-by-channel median of Tsys spectra for antenna within spw" - "channel-bychannel median of Tsys spectra for all antennas within spw".
The views are evaluated against the "sharps" vector flagging rule, which flags each view in
two passes:
a. flag all channels whose absolute difference in value to the following channel exceeds athreshold "fb_sharps_limit" (default: 0.05).
b. around each newly flagged channel, flag neighboring channels until their channel-tochannel difference falls below 2 times the median channel-to-channel difference (this isintended to flag the wings of sharp features).
A single flagging command is generated for all channels newly identified as "birdies".

6) Metric 6: "toomany"
A separate view is generated for each polarisation and each spw. Each view is a matrix withaxes "time" vs. "antenna". Each point in the matrix is the median value of the Tsys spectrum
for that antenna/time. (This is the same as for "nmedian" metric).
The views are evaluated against two separate flagging rules:46

a. "tmf" (too many flags): This evaluates each timestamp one-by-one, flagging an entire
timestamp when the fraction of flagged antennas within this timestamp exceeds thethreshold "tmf1_limit" (default: 0.666). Flagging commands are generated per timestamp.
b. "tmef" (too many entirely flagged): This evaluates all timestamps at once, flagging allantennas for all timestamps within current view (spw, pol) when the fraction of antennas
that are entirely flagged in all timestamps exceeds the threshold "tmef1_limit" (default:
0.666). Flagging commands are generated for each data point in the view that is newly
flagged.

9.8 hifa_antpos
Sometimes the antenna positions were refined after the science data were recorded. If suchrefinements have been located, they are applied in this task. The corrections are listed in the
WebLog, and the data are corrected via a calibration table.

9.9 hifa_wvrgcalflag
Water Vapor Radiometer (WVR) power measurements are converted into a phasecorrection table that can be applied to the science data. The phase rms duringobservation of the bandpass calibrator, with and without the WVR correction, is used 1)
to detect poorly performing WVR units on individual antennas, and 2) to determine if the
WVR correction helps overall.
The WebLog shows the effects of the phase correction in several ways, if any antennas’
WVR data are flagged (the required phase correction is then interpolated from nearby
antennas), and also prints a warning if the correction is deemed not helpful enough toapply at all.
In hifa_wvrgcalflag, a QA score is produced for each measurement set of an OUS. The
QA score is zero if the RMS improvement ratio is less than 1.0, and 1 if it is greater than
2.0. If it is between 1.0 and 2.0, the score will be linearly scaled to be in the range of 0.5
to 1.0. Thus, a ratio of 1.08 will produce a QA score of 0.54. The final score is the lowest
score from all ms. If the ratio for any ms is below 1.32, then a QA notification is also
generated for it at the top of the hifa_wvrgcalflag page.

9.10 hif_lowgainflag
Antennas with persistently discrepant amplitude gains are detected and flagged. The
WebLog links to grayscale images of the relative gain of each antenna calculated using
the observation of the bandpass calibrator, and shows if any antennas are flagged.
This task first creates a bandpass caltable, then a gain phase caltable, and finally a gain
amplitude caltable. This final gain amplitude caltable is used to identify antennas withoutlier gains, for each spw. Flagging commands for outlier antennas (per spw) are applied
to the entire MS.
A separate view is created for spw. Each view is a matrix with axes "time" vs. "antenna".
Each point in the matrix is the absolute gain amplitude for that antenna/timestamp.
The views are evaluated against the "nmedian" matrix flagging rule, where data points
are identified as outliers if:
a. Their value is smaller than a threshold-factor * median of all non-flagged data
points, where the threshold is 'fnm_lo_limit' (default: 0.7), or47

b. Their value is larger than a threshold-factor * median of all non-flagged data
points, where the threshold is 'fnm_hi_limit' (default: 1.3).
Flagging commands are generated for each of the identified outlier data points.

9.11 hif_setmodel
The model flux density of the amplitude calibrator is set, either from an internal CASA
model (solar system objects), or the results of observatory calibrator monitoring (quasars)
which ultimately appear in the file flux.csv (see hifa_importdata). These flux densities
are listed on the WebLog page, along with plots of the amplitude calibrator as a functionof uv distance (which is useful to assess resolved solar system objects). If the bandpass
calibrator is distinct from the amplitude calibrator and is a frequently monitored quasar, its
model is also set at this stage.

9.12 hifa_bandpassflag
In this task, the bandpass calibrator is self-calibrated (phase only is first calibrated on as
short a time interval as allowed by signal-to-noise, listed on the WebLog page). The
antenna-based bandpass phase and amplitude solution is then calculated using a S/Ndependent frequency interval, also listed on the WebLog page.
After calculating an initial bandpass solution, flagging is performed by comparing thescalar difference between the amplitudes of the calibrated visibilities and the model forthe bandpass calibrator (Figure 18). If flags are found, the bandpass phase-up andbandpass calibration tables are recomputed.

Figure 18: bandpass flag removing outlier amplitudes

Finally, the WebLog page links to plots of all of the bandpass solutions, with theatmospheric transmission curve overlaid.
Note about flagging summary table: the “before” flagging fraction in hifa_bandpassflag
may differ from the “after” flagging fraction in hifa_flagdata, because hifa_bandpassflag's"before" summary is done on an MS that has temporarily already had some caltables
applied (and thus some flagging already propagated). This is done because thebefore/after summary in hifa_bandpassflag is intended to clearly show how much new
flagging is done by hifa_bandpassflag.
48

9.13 hifa_spwphaseup
The relative phase offsets between spectral windows are determined for each antennausing the observation of the bandpass calibrator. (The offset is assumed to be constantin time during each execution.) If narrow spectral windows are present, a mapping isdetermined so that the calculated phase calibration as a function of time can be
subsequently transferred (during subsequent gaincal and applycal tasks) from wider,higher S/N spectral windows to the narrow ones. If any such reference spwmaps are
required, then they are listed on the WebLog page. The assumption of constant phaseoffsets vs. time is tested downstream in hif_timegaincal by solving for new time-based
phase solutions per spw with the spwphaseup table (and associated mapping) applied. If
the signal-to-noise is sufficiently poor, all spws are combined for subsequent calculationof time-varying gains, and a warning is printed here to indicate that combination has
occurred.

9.14 hifa_gfluxscaleflag
As with hifa_bandpassflag, outliers in the the calibrated visibilities of the flux and phasecalibrators are flagged before a final fluxscale is calculated. The weblog shows thebefore and after flagging plots as a function of time and of uv distance.

9.15 hifa_gfluxscale
In this task, the absolute flux scale is transferred from the amplitude calibrator to the othercalibrators and ultimately to the science target (via the phase calibrator). A phase-only
self-calibration is performed on all calibrators prior to this flux calculation.
The WebLog for this stage lists the derived flux densities of the non-amplitude calibrators(usually phase and bandpass calibrators), along with the flux values extracted from the
ALMA Source Catalog. Plots of amplitude as a function of uv distance are shown, and If
the absolute flux calibrator is resolved (decreasing flux with increasing uv distance,
usually only the case for solar system objects), only data on short baselines are used to
calculate the flux densities of the secondary calibrators. Any such uv limits are listed in
the table at the top of the WebLog page. Blank entries mean that all UV ranges and/or all
antennas were used, which will be the case for quasars.

Figure 19: uv ranges for fluxscale table.

Derived flux density vs catalog flux density plots are shown for non-amplitude
calibrators. Faint sources can have systematically elevated gfluxscale derived flux
densities in low SNR spws. The pipeline also calculates a QA metric to detect such
spw-spw discrepancies (see QA Table in section 8.7.1), and reports a warning if the
spw-spw variation is greater than 20%.

49

Figure 20: Examples of the plot comparing derived flux densities to source catalog flux densities.

9.16 hifa_timegaincal
In this task, gain as a function of time is calculated from observations of the phasecalibrator. The WebLog page shows plots of this gain, both on a scan timescale (as will
be interpolated to the science target), and on an integration timescale (useful forassessing weather and calibration quality). An additional plot is also provided that helps
to assess time variations of the spw to spw offsets as a function of time, in optimal cases
the scatter about zero should be small.

9.17 hif_applycal
In this task, all previously calculated calibration tables are applied to the sciencedata. Any failed calibration solutions, and flagged Tsys scans, will result in flagging ofactual science data in this stage, so the WebLog shows a summary of that additional
flagging, and high flagging will result in a low QA score. The WebLog page also includes
many useful plots of the calibrated data as a function of time and frequency. Outliers inthese plots can indicate any remaining bad data. A plot of uv coverage after potential
flagging during calibration is provided for the representative Source and spw.

9.18 hif_makeimlist: Set-up parameters for calibrator images
This stage determines image parameters (image size, cell size, etc) to be used in thesubsequent hif_makeimages stage, and reports them on the WebLog page (See Figure21). The “specmode” can be mfs for per-spw continuum multi-frequency synthesis
images, “cont” for mfs continuum images of several spectral windows, or “cube” forspectral cubes. The first time the task is run is in preparation for making per-spw mfs
images of the calibrators.

50

Figure 21: Example of the WebLog for the hif_makeimlist stage. This example is for setting up theparameters for calibrator per-spw multi-frequency synthesis (mfs) continuum images.

9.19 hif_makeimages: calibrator images
This stage actually creates the images requested by the most recent hif_makeimlist.
The first time it is run is to create per-spw mfs continuum images of the calibrators. See
the description in Sec. 9.31 for more information and examples of the hif_makeimages
stage. Low QA scores for non-Check source calibrators may indicate the need for
additional flagging and/or significant decoherence.

9.20 hif_makeimlist: check source imaging
Prepare to create check source images, one per EB

9.21 hif_makeimages: check source imaging and QA information
After creating images, the pipeline calculates and shows a table of check source fitting
results per spw, and per EB:• Check source position offset from catalogue position in mas and synthesized
beams, warning level > 0.15*synthesizedBeam• Fitted [Peak Intensity / Flux Density] Ratio can help to assess decorrelation AND
presence of resolved emission, warning level < 0.8• [Fitted / gfluxscale] Flux Density Ratio, warning level < 0.8• Warning also given if S/N of fitted or gfluxscale flux densities are low < 20
• The images themselves which are per EB / spw are located below the table, andexcept for being per EB are unchanged

51

Figure 22: Check source imaging diagnostic table

9.22 hifa_imageprecheck
The representative source and spw containing the representative frequency selected by
the PI in the OT are used to calculate the synthesized beam and to make sensitivity
estimates for the aggregate bandwidth and representative bandwidth for three values of
the robust parameter. This information is reported in the hifa_imageprecheck weblog,
including a table like the example shown in Figure 23. If no representative
target/frequency information is available, it defaults to the first target and center of first
spw in the data (i.e. pre-Cycle 5 data does not have this information available).
The best Briggs weighting parameter to achieve the PI’s desired angular resolution ischosen automatically:• These robust values are considered in order: +0.5, +1.0, 0.0, +2.0, and for single
fields only, -0.5 (the mathematical assumptions of mosaic synthesis imaging are
not compatible with the mathematical assumptions of uniform Briggs weigthing
and lead to poor results when uv coverage is sparse, so the pipeline does notuse robust<0.0 for mosaics).
• If one value has a predicted beam with both axes within the PI desired range, thatrobust is used. If that robust is not the default 0.5, a warning is printed.
• If no robust produces a beam with both axes within range, the robust that
produces a predicted beam area closest to the mean of the PI’s range is chosen.
• If no robust can produce a beam area within the PI’s range, an error is printed.

52

Figure 23: example imageprecheck page, showing the "Goals from the OT" including the PI desired
sensitivity. The table shows the sensitivity and predicted beam for a range of robust values. The pipeline
then choses the best robust value (see text).

Note: the cell and imsize chosen in these stages is stored in the pipeline context so thatall product images (mfs, cubes) have the same cell and imsize. If these stages are runwith non-default intent/source selections (and thus different , slightly different cell andimsize may naturally result.

9.23 hif_checkproductsize: Mitigation to avoid overly long runs
This function will modify the characteristics of the imaging products in order to decrease
their size, thereby decreasing the time needed to make them so that data can be delivered
to PI’s more expediently. Figure 24 shows an example weblog page for a mitigateddataset. Datasets that have been mitigated will have imaging products with different
characteristics than those that have not been mitigated. Full imaging products can berecreated by users, using the tclean commands that are in the casa_commands.log file,or by calling the appropriate hif_makeimlist, hif_makeimages with the defaults (which will
make full imaging products without mitigations – be aware that this could take many days
to complete).
The mitigations are done in a priority order, with the mitigation halted once the predicted
sizes fall below the thresholds. The mitigation then proceeds according to the following
logic:

53

Figure 24: Screenshot of new hif_checkproductsize stage of IF Pipeline. In this example, the cubes for spw25 had to bebinned by a factor of 2, and the FOV was limited to the 0.5 response point of the primary beam in order to get the
products below the default thresholds. Before the mitigation the maximum cube would have been 96GB; after the
mitigation, it is predicted to be 21.7 GB.

The default limits are:maxcubesize: 40 GB
maxcubelimit: 60 GB
maxproductsize: 350 GBI) The pipeline recipe explicitly encodes these values so it can be easily changed
universally for all pipeline runs
II) The PPR also encodes these values explicitly, so they can be easily changed on aper-MOUS basis, and the pipeline re-run using the modified PPR, or DRMs at a site canchange the default values for all runs at their site.
III) After the pipeline is run, the iptipt.py explicitly encodes the values used, so they canbe easily changed by the user if desired, and new products created.
The size calculations (in GB) are based on the following:
mfssize = 4. * nx * ny / 1e9
cubesize = 4. * nx * ny * nchan / nbin / 1e9
productsize = 2.0 * (mfssize + cubesize)
The mitigation cascade is as follows:
Step 1: If cubesize > maxcubesize, for each spw that exceeds maxcubesize:a. If (nchan == 3840) or (nchan in (1920, 960, 480) AND online channel averaging
was NOT already performed, set nbin=2.
54

b. If still too large, then calculate the Gaussian primary beam (PB) response level atwhich the largest cube size of all targets is equal to the maximum allowed cube
size. The cube sizes are initially calculated at primary beam power level
PB=0.2. For an image of width d, the response level at the edge will be PB=exp(d2*ln(2)/FWHM2), the image size is proportional to d2 µ -ln(PB), and the required
power level to create an image of size = maxcubesize is
PB_mitigation = exp(ln(0.2) * maxcubesize / current_cubesize)i. Then account for imsize padding: PB_mitigation = 1.02 * PB_mitigation
ii. Then limit the size reduction to PB=0.7: PB_mitigation =min(PB_mitigation, 0.7)
iii. Then round to 2 significant digits: PB_mitigation = round(PB_mitigation, 2)
NOTE: this mitigation cannot be applied to mosaics, only single fields.
The same mitigated FoV is used for all science target image products.
c. If still too large, change the pixels per beam (cell size) from 5 to 3.
d. If still too large, stop with error, the largest size cube(s) cannot be mitigated.
Step 2: If productsize > maxproductsize
a. If the number of science targets (single fields or mosaics) is greater than 1,
reduce the number of targets to be imaged until productsize < maxproductsize.
The representative target is always retained.
b. If productsize still too large, repeat steps 1a, 1b, and 1c, recalculating productsize
each time.
c. If productsize is still large, stop with error, the productsize cannot be mitigated.
Step 3: For projects with large cubes that can be mitigated, restrict the number of large
cubes that will be cleaned:a. If there are cubes with sizes greater than 0.5 * maxcubelimit, limit the number oflarge cubes to be cleaned to 1. The spw encompassing the representative
frequency shall always be among the cubes retained.
Step 4: For projects that have many science targets, limit the number to be imaged to30, the representative target is always retained in the list.
When the cube or product size cannot be mitigated, the following warning will appear at
the hif_checkproductsize stage:"QA Maximum cube size cannot be mitigated" and then the pipeline will stop in the first
hif_makeimlist that creates cubes with the message: "Error! Size mitigation had failed. Will not
create any clean targets.”
In the example shown in Figure 24, the initial data products were estimated to include a
cube that would be 96 GB. This triggered two mitigations: spectral window 25 was binned
by a factor of 2, and the FOV was restricted to the 0.5 response point of the PrimaryBeam. This was sufficient to get the cube size down to 21.7 GB, so the mitigation cascade
stopped. The total product size after the cube mitigation is 52.1 GB, so products for all
sources could be made.

55

9.24 hif_exportdata
Calibration tables, calibrator images (exported in fits format), and other products are
moved from the pipeline /working to the /products directory.
NOTE: The subsequent stages are only present in the imaging recipe.

9.25 hif_mstransform
For each execution, calibrated visibilities for the science target(s) are split to a new MS
with “target.ms” in the name, as listed on the front WebLog page.

9.26 hifa_flagtargets
Flagging of the science target data, if determined to be necessary by an observatory
scientist, is performed as listed in the *flagtargetstemplate.txt files linked to the WebLog
page. The WebLog also shows a summary table of any flagging performed.

9.27 hif_makeimlist: Set-up parameters for target per-spw continuum
imaging
Imaging parameters are determined and listed for creation of per-spw mfs continuum
images of each science target. This run of hif_makeimlist also controls theparameters used to create the dirty cubes used by the hif_findcont stage, including
any channel binning (listed in the “nbins” column of the hif_makeimlist table).

9.28 hif_findcont
In this task, dirty image cubes are created for each spectral window of each science target.
The cubes are made at the native channel resolution unless the nbins parameter wasused in the preceding hif_makeimlist stage.
The pipeline then generates and evaluates the mean spectrum of a masked region of thedirty line+cont image constructed from moment0 and moment8 (peak) images.
••

If fewer than 4 pixels of contiguous emission are found after pruning, the whole
field is used.
In either case, what is displayed really is a spectrum of the source (created by theia tool). (see examples in Figure 25).

The mask threshold is based on imstat robust statistics (chauvenet MAD). The mask
image can be viewed if necessary in the working directory (*.joint.mask2 if present,
otherwise *.joint.mask).
Frequency ranges are calculated that are the least likely to contain any line emission orabsorption, and these are listed in the LSRK frame on the WebLog page, as well as being
indicated by the cyan colored horizontal line(s) on the spectra.

56

Figure 25: Two examples of hif_findcont plots, one with the entire window identified as continuum (left),and another with two identified continuum regions (right; identified continuum indicated by cyan lines).

The continuum frequency ranges are also printed to a file called “cont.dat”. If this filealready exists before hif_findcont is executed, then it will first examine the contents.
For any spw that already has frequency ranges defined in this file, it will not perform theanalysis described above in favor of the a priori ranges. For spws not listed in a preexisting file, it will analyze them as normal and update the file. In either case, the filecont.dat is used by the subsequent hif_uvcontfit and hif_makeimages stages.

9.29 hif_uvcontfit
The previously determined continuum frequency ranges as shown in the cont.dat file areused to fit the continuum of each visibility. The fit is performed for each spw independently
using a fitorder=1, and a calibration table is used to store the resulting fits called“uvcont.tbl”. The WebLog for this stage reports the continuum ranges from
hif_findcont in LSRK but translated to the topocentric (TOPO) frame for each MS.

9.30 hif_uvcontsub
The hif_uvcontfit calibration table is applied to the data. After this step, the originalcontinuum + line emission is contained in the DATA column of the MS, while thecontinuum subtracted data are written to the CORRECTED column.

9.31 hif_makeimages: Make target per-spw continuum images
Cleaned continuum images are created for each spectral window, each science target,
using the continuum frequency ranges determined from hif_findcont (as written in thecont.dat file).
The pipeline uses the tclean auto-masking method “auto-multithresh” in all
hif_makeimages stages. This algorithm is intended to mimic what an experienced user
would do when manually masking images while interactively cleaning. The parameterssidelobethreshold and noisethreshold control the masking of the image. The
sidelobethreshold indicates the minimum sidelobe level that should be masked, while thenoisethreshold indicates the minimum signal-to-noise value that should be masked. The
threshold used for masking is the greater of the two values calculated for each minor cycle
based on the rms noise and sidelobe levels in the current residual image. Due to a featurethat “prunes” small (< minbeamfrac) noise-like automask regions real emission can haveall mask regions “pruned” resulting in no clean mask for very compact, typically high S/N57

emission or absorption. For continuum imaging stages, tclean is run again but falling backto the Cycle 4 clean mask that is simply a fraction of the primary beam (0.3 .pb if nomitigation of the field of view has occurred). To save time this is not done for cube imaging
stages.
The pipeline tclean automask parameters vary as a function of imaging type, and the 75th
percentile baseline length, b75. These differences are needed because, for example, thesmaller 12m-array configurations tend to have better uv-coverage and psfs than more
extended configurations.
Automask parameter 7m-array 12m-array b75 < 300 m

12m-array b75 > 300 m

noisethreshold

5.0

4.25

5.0

sidelobethreshold

1.25

2.0

3.0

lownoisethreshold

2.0

1.5

1.5

minbeamfrac

0.1

0.3

0.3

negativethreshold

0.0

0.0 (continuum)/15.0 (line)

0.0 (continuum)/7.0 (line)

Images are cleaned to a threshold of 2 x (predicted rms noise) x (dynamic range correction
factor). The dynamic range correction factor accounts for the fact that sources with a highdynamic range will have larger imaging artifacts, which should not be cleaned. The
artifacts are worse for poorer UV coverage, so different dynamic range (DR) corrections
factors are adopted for 12-m Array and 7-m Array observations, for science targets
according to the following tables:
SourceDynamic
Range≤ 20
20 – 50
50 – 100
100 – 150
≥ 150

12-m Array DR
correction
factor1
1.5
2
2.5
max (2.5, DR/150)

SourceDynamic
Range

7-m Array DR
correction
factor – 1EB

7-m Array DR
correction factor– 2 or more EBs≤4
4 – 10
10 – 20
20 – 30
30 – 40
30 – 75
≥ 75

1
1.5
2
2.5
max (2.5, DR/30)

1
1.5
2
2.5
2.5
3.0
max (3.5, DR/55)

The resulting non-primary beam corrected images are displayed on the WebLog page.
For each image, the properties are shown next to the associated image png (see Figure26). In particular, the following are reported: the center frequency, beam parameters(major and minor FWHM resolution & position angle), theoretical sensitivity, cleaningthreshold, dynamic range of the dirty image (image peak to theoretical noise) andcorresponding DR correction factor, the non-pbcor image rms (the noise measured in the
non-primary beam corrected image over an annulus between the 0.3 to 0.2 response
point of the primary beam, or a smaller analogous annulus if field of view mitigation has
occurred), image max /min of the primary beam corrected image, fractional bandwidth,
aggregate bandwidth, and the image QA score (meant to indicate how close themeasured noise is to the theoretical noise, considering also the DR correction factor –see Sec. 8.7.1).

58

Figure 26: Example of hif_makeimages WebLog page for per-spw images. Clicking on the thumbnail will
enlarge the image. Clicking on the “View other QA images” link will bring up the detailed image page (Figure
27).

The “View Other QA Images’ links for each image show the primary beam corrected
image, residual, clean mask (red area), dirty image, primary beam, psf, and clean model(Figure 27).

59

Figure 27: Details page that is displayed after clicking on the “View other QA images” link on the
hif_makeimages WebLog page.

9.32 hif_makeimlist: Set-up parameters for target aggregate
continuum images
Imaging parameters are calculated and listed for creation of an aggregate (all spectral
windows combined) continuum image (specmode=’cont’) of each science target.

9.33 hif_makeimages: Make target aggregate continuum images
A cleaned aggregate continuum image of each science target is formed from thehif_findcont channels (as listed in the cont.dat file) is created. The aggregate
continuum image(s) are made with nterms=2 if the fractional bandwidth is ≥ 10% (onlycurrently possible for ALMA Bands 3 and 4 data). The resulting non-primary beamcorrected images are displayed on the WebLog page. The “View Other QA Images” links
show the primary beam corrected image, psf, clean model, dirty image, and residual
image (Figure 27).

9.34 hif_makeimlist: Set-up image parameters for target cube imaging
Parameters are calculated and listed for creation of spectral cube images of eachcontinuum-subtracted spectral window of each science target.

9.35 hif_makeimages: Make target cubes
Cleaned continuum-subtracted cubes are created for each science target and spectral
window at the native channel resolution (unless channel binning has been selected usingnbins in the preceding hif_makeimlist) from the CORRECTED column. Cubes are60made in the radio LSRK frequency frame. Only channels that have not been designatedas continuum channels are cleaned. The WebLog page displays non-primary beamcorrected peak intensity images for each cube (“moment 8”) along with properties of the
cubes (see Figure 28). The information is similar to that described in Sec. 9.31 forcontinuum images, except that the noise is the median rms over all channels (stillmeasured in a 0.3 – 0.2 PB annulus), and instead of fractional and aggregate bandwidth
the “channel” information is given as the number of channels imaged times the channel
width. Recall that if no online or nbins (pipeline option) channel averaging is done, thevelocity resolution will be twice the channel width.

Figure 28: Example of hif_makeimages WebLog page for image cubes.

In addition to the “View other QA images” for continuum images demonstrated for Stage27, an additional plot is included for continuum subtracted cubes: an integrated intensity(“moment 0”) and peak intensity (“moment 8”) image using the hif_findcont
continuum frequency ranges (labeled “Line-free Moment 0”; see Figure 29), which should
be noise-like if the continuum subtraction worked well.

61

Figure 29: Example of an image cube details page including the line-free moment 0 and moment 8 images.

9.36 hif_makeimlist: Set-up image parameters for representative
bandwidth target cube imaging
If the PI requested spectral resolution (bandwidth for sensitivity) is at least 4x larger thanthe correlator channel width, then in addition to cubes created at that correlator width,
the representative source and spw are imaged at the PI’s requested resolution, in thisand the next stage.

9.37 hif_makeimages: Make representative bandwidth target cube
If the PI requested bandwidth for sensitivity (representative bandwidth) is significantly
coarser (> 4x) than the native correlator channel width, an additional cube is created at
the PI’s bandwidth (note: this stage is always created even if it is not populated).

9.38 hif_exportdata
Science target images are converted to FITS format and copied to the /products
subdirectory as well as the cont.dat file from the hif_findcont stage. This stage is run
in operations, but is not included in the casa_pipescript.py script.

62

The “By task” WebLog for Single-Dish Data
This section describes navigation of the Task sub-pages for each Single Dish Pipelinetask starting from the “By Task” tab. For a fuller description of each task, refer to the
ALMA Science Pipeline Reference Manual.

9.39 hsd_importdata
The WebLog for hsd_importdata task shows the summary of imported MSs, groupingof spws to be reduced as a group, and spw matching between Tsys and science spws.
This task also generates figures of Telescope Pointings, which are available in the MS
Summary page (i.e. from the Home page, click the MS name, and then click on “Telescope
Pointing”). There are two types of plots that can be found containing full information on all
pointings and just on-source pointings (Figure 30). In these plots, the red circle indicatesthe beam size of the antennas and its location is the starting position of the raster scan.
The Red (small) dot indicates the last position of the raster. The green line represents theantenna slewing motion, and in the right panel of Figure 30 the green line going to/fromthe red dot indicates that the antenna goes to the last scan and returns to the OFF
position. The grey dots indicate flagged data. hsd_importdata generates pointing patternplots with ephemeris correction in addition to the plots without correction if target is moving
source.

Figure 30: The detailed page of Telescope Pointing in the MS summary page.

9.40 hsd_flagdata
The WebLog for the hsd_flagdata task shows the summary of flagged data percentage
per MS due to binary data and online flagging, manually inserted file (*flagtemplate.txt),
shadowing, unwanted intents, and edge channels. Note that the value in the “Before Task”
column corresponds to the percentage of flagged data by binary data flagging (BDF).

9.41 h_tsyscal
This page shows the associations of Tsys and science spectral windows to be used forTsys (amplitude-scale) calibration, and also shows the original Tsys spectra per spectral
window.

9.42 hsd_tsysflag
This page shows the flagged Tsys spectra per spectral window after heuristic flagging isapplied.
63

9.43 hsd_skycal
The WebLog shows the integrated OFF spectra per spw and per source. The y-axis is the
direct output from the correlator, which means the values are dominated by signals fromboth the atmosphere and receivers (Figure 31). The different colors indicate different
scans (times). The magenta lines indicate the atmospheric transmission at eachfrequency.
The time-averaged plots of the OFF spectra are also shown in this page for the purposeof assessing the time variability of the spectra. The different colors here indicate different
spws. Note that the OFF spectrum is not averaged over the spectral windows yet, but it
will be in the future.
The coordinates of the OFF position can be confirmed in the Reference Coordinates table.

Figure 31: An example of OFF spectrum

In addition, amplitude versus time plots for the OFF_SOURCE data and elevationdifference between ON_SOURCE and OFF_SOURCE data plots are shown in thispage.

9.44 hsd_k2jycal
This page shows the list of Kelvin-to-Jansky conversion factors that Pipeline has readfrom a file “jyperk.csv”, which shall contain the factors per spw, per antenna, and perpolarization.

9.45 hsd_applycal
This page shows a list of the calibrated MSs with the name of the applied Tsys, Sky andamplitude calibration (Kelvin-to-Jansky conversion) tables, and also shows the integrated
spectra after calibration.

649.46 hsd_baseline
Spectral data before/after baseline subtraction
The hsd_baseline page of the WebLog shows the three grids of spectra per source:
the one on the top and in the bottom correspond to the spectra before and after thebaseline subtraction, respectively. The one in the middle is obtained by averaging all the
spectra associated with each grid (see Figure 32). Averaging the data improves the S/Nratio making the spectral line features more prominent, which makes it easier to comparethem with the line mask for baseline subtraction (see below). These plots, which appear
just after clicking the hsd_baseline link of the WebLog, show a representative spectral
grid of each spw. Normally they correspond to the spectral grid of a certain antenna. The
spectral grids are shown in R.A./Decl. coordinates. Each small panel shows one
representative spectrum per grid cell (which sometimes we call “sparse profile map”). The
red (horizontal) line over-plotted on the spectrum indicates the fitted function to be used
for baseline subtraction for spectral data before baseline subtraction, while the zero-levelfor spectral data after baseline subtraction.

Figure 32 An example of the summary page of hsd_baseline.
65

On the top panel of each spectral grid, a spatially integrated spectrum per ASDM,
antenna, spw and polarization is shown. The magenta lines indicate the atmospherictransmission at each frequency. The cyan filled regions indicate the mask channels
containing emission line that are identified in the entire map, and red thick bars indicate
the channels masked by a “deviation mask” algorithm, designed to exclude atmosphericlines and lines at the band edge from the baseline fit.
Detailed plots of the spectra can be seen in the detail pages, which can be opened byclicking the “Spectral Window” link below the grid of spectra in the summary page. In thedetail pages the spectral maps of all the antennas are shown. In the upper part of thedetail pages there are boxes that can be used to set filters to plot spectral maps byantenna, field, spectral window and polarization.
R.A. vs Dec. plots
There are four different plots per spw, i.e. “clustering_detection”, “clustering_validation”,“clustering_smoothing”, and “clustering_final”. The number of plots in each figure is thesame as that of the candidate line components. The “cluster_detection” plot (Figure 33a)shows the grid cells having emission line exceeding the threshold. In the plot, yellow gridcells show a region where there is a single time-domain group with detected emission
lines. Cyan squares indicate grid cells where there are more than one time-domain groups
with detected emission lines.
(a)

(b)

(c)

Figure 33 Examples of (a) clustering_detection, (b) clustering_validation, and (c)clustering_smoothing.

After line detection, the algorithm calculates how many spectra containing emission lines
are included in the grid cell in order to judge whether the grid cell possibly contains trueemission lines. At this line detection validation step, the ratio of the number of spectra
having detected emission lines (defined as “Nmember”) per grid cell and the number of
total spectra belonging to the grid cell (“Nspectra”) is calculated. The“clustering_validation” plot (Figure 33b) shows this ratio for each grid cell, i.e., the grid cellis marked as:•••“Validated” if Nmember/Nspectra > 0.5 (Blue squares in Figure 33b)
“Marginally validated” if Nmember/Nspectra > 0.3 (Cyan squares)“Questionable” if Nmember/Nspectra > 0.2 (Yellow squares)

After the validation step, the grid containing the Nmember/Nspectra rate per grid cell issmoothed by a Gaussian-like grid function. This is to eliminate the isolated grid cells
having a single emission line candidate while enhancing the grid cells with detected
emission line in neighboring grid cells.
66

Figure 33c shows an example of “clustering_smoothing”. Blue squares represent the gridcells with points exceeding the defined threshold, i.e., the grid cells having promisingdetections of emission lines that are also found in the neighboring grid cells. Cyan andyellow squares are the grid cells with points slightly below the threshold (Border), or lower
than the threshold (Questionable).
(a)

(b)

Figure 34 (a) An example of how the mask range is calculated. In the blue squares, the mask channel range isthe range obtained at the nearest edge of any validated area by interpolating mask channel ranges in the valid
grid cells (white-filled red circle). (b) An example of clustering_final.

As a final step, the mask region for each grid cell is determined. In the validated area after
the validation and the smoothing steps (blue squares in Figure 33c or green squares in
Figure 34), mask channel ranges are calculated over the spatial domain byinter/extrapolating the mask ranges of the integrated spectra in the validated cells, andover each single non-integrated spectrum. The mask channel range is determined andused in baseline subtraction in the green and blue squares of Figure 34a. An example of“clustering_final” is shown in Figure 34b.
Line Center vs. Line Width plot
This plot shows the extent of each identified emission line candidate on the parameter
space of the line width versus the line center. The small dots indicate spectra containing
identified emission line. The red ovals show each clustering region with a size of thecluster radius.
Number of Clusters vs. Score plot
This plot shows the number of clusters and corresponding scores based on the cluster
size determined from the “line width” v.s. “line center” plot using clustering analysis (Kmeans algorithm). The scoring is empirically defined so that the score gets better (smaller)
when the cluster size is smaller, the number of clusters is smaller, and the number ofoutliers is fewer than those of other clusters. The users will know which number of clusters
is more plausible by searching for the number of clusters with a lower score. This plot isbasically for developers.
Fitting order determination
Pipeline performs baseline fitting using a cubic spline that connects an empirically defined
number of segments of each spectrum.
In a first step, the spectra are grouped in space and time domains. Pipeline groups the
spectra that were observed close in time and position with respect to each other.
Subsequently, Pipeline analyzes each (emission-masked) spectrum through Fast Fourier
Transform (FFT) to obtain the power spectra. Note that input for the discrete Fourier
67

Transform is (spectrum-average)*flag, where “flag” is set to zero for the emission-masked
channels, while set to 1 for other channels. The power spectra of all integrations in agroup are summed together and divided by its average value (averaging over frequency
channels). Based on the peak value of the normalized Fourier spectra (P_FFT), thenumber of segments for cubic spline fitting (N_segment) is defined empirically:• If 1<P_FFT<3 , then N_segment=3,• If 3≦P_FFT<5, then N_segment=4,• If 5≦P_FFT<10, then N_segmen=5,• If P_FFT≧10, then N_segment=F_FFT×2+1, where F_FFT is the frequency
corresponding to the peak P_FFT.
In a second step, in order to take into account the proportion of masked channels, theobtained N_segment is newly defined as N_segment×(Nch-N(mask))/Nch, where Nch isthe number of spectral channel and N(mask) is the number of channels masked. Notethat unmasked channels are equally divided into segments, i.e., number of unmasked
channels is same for all segments. Finally, the Pipeline performs the baseline fitting andbaseline subtraction using cubic splines, which are third order polynomial that meet theboundary condition at the joint between the segments.
Mask range determination
When baseline fitting is performed, the emission channel range is masked out. The mask
range is determined by following equation:
mask={(maxW-width×(2×minW+10)+(width-minW)×maxW}/(maxW-minW), where minW
is minimum channel width, maxW is maximum channel width, width is channel width
where the emission is above a threshold with at least five adjacent channels (see Figure32). The minW and maxW are empirically determined and set to 2 and 500, respectively.
Finally, the range of central channelmask divided by 2 will be masked out. In this definition,
the relatively narrow channel width will have slightly larger mask range. For example, if
width=5channels, the mask range will be 8 channels at the central channel. For width=500
channels, the mask range will be 250 channels at the central channel.

Figure 35: An example of spectrum with Pipeline defined mask range. “Width” corresponds to the number ofchannels where the emission is above the threshold.

68

9.47 hsd_blflag
The WebLog shows the list of flagged data percentage using five criteria that are
explained in the ALMA Pipeline Reference Manual. When you click on “details”, you will
get the detailed figures to evaluate these criteria as a function of rows (one row
corresponds to a spectrum for one integration). The flagged and unflagged data areshown in red and blue, respectively.

9.48 hsd_imaging
Image Sensitivity Table
The achieved sensitivity for the final cubes per Spw and Source are shown in the table.
For the cube of representative Spw, the sensitivity is calculated with the PI's requested
bandwidth, while for the other Spws the sensitivities are measured with the native
resolution.
Profile Map
Figure 36 shows the top of the summary page. Three types of profile maps are available
in the WebLog: 1) The simplified profile map of the combined image per spw at the top,2) a simplified profile map per antenna, and 3) a detailed profile map. In the simplified
profile map, the magenta lines indicate the atmospheric transmission at each frequency.
One transmission profile is plotted for each ASDM processed. To access the simplified
profile map per antenna, click the corresponding “Spectral Window”. Each spectrum of
the simplified profile maps (either 1. or 2.) corresponds to an averaged spectrum in anarea of ⅛ of the image size (imsize), so that the total number of spectra in the profile map
is 8 times 8. If the number of pixels (along x- or y-axis) is less than eight, it shows all
spectrum per pixel. To see the detailed profile maps, click the icon with a symbol ofpolarization in the polarization column (see Figure 36). Each bin of the profile map isequivalent to a pixel, but with an interval of three cells. Due to the limitation of the allowed
number of plots per page (max 5 x 5 plots per page), the rest of the plots are displayed inother pages.

Figure 36 An example of the profile map.

69

Channel Map
The number of channel maps per spw corresponds to the number of emission lines thathave been identified by the clustering analysis. In each channel map (see Figure 37), thetop-middle plot shows the identified emission line and the determined line width(bracketed by two red vertical lines), overplotted on the averaged flux spectrum (in Jy) as
a function of frequency (in GHz).
The top-left plot shows the zoom-up view of the identified emission line, but with velocity
axis. The vertical axis is the averaged flux in Jy and the horizontal axis is in units of km/s.
The (center) velocity of 0 km/s corresponds to the central frequency of the spectral range
where line emission was detected, while the velocity range is equivalent to the masked
region where the emission line was identified. The line velocity width is gridded into 15
bins,
which
are
shown
as
red
verticallines.
The top-right plot shows the total integrated intensity map (in Jy/beam km/s) over the allchannels in the spw. Finally, the channel maps within the velocity range of the identified
emission line are shown in the panel at the bottom. Each channel plot corresponds to abin in the top-left plot.

Figure 37 An example of channel map.

The Baseline RMS Map is created using the baseline RMS stored in the baseline tables.
The baseline RMS is calculated by hsd_baseline using emission free channels.
70

The Integrated Intensity Map for each spw is generated using immoments task with all
the available channel range.

71