How to use ALMA Pipeline when
you have only raw data
Rie E. Miura (NAOJ)
Fumi Egusa (NAOJ)

1CAUSION
Pipeline will expect only processing QA0_PASS
data, and ALMA defined standard mode data. If
you want to use Pipeline reduction on otherdata, you need to run Pipeline on yourresponsibility. Observatory will not do further
support even when you have problems andPipeline errors.

2How to use ALMA Pipeline
1. Start CASA Pipeline$casa --pipeline
2. Confirm your CASA version that you intent to use.
e.g. CASA Version 4.5.0-REL (r35147)
3. Confirm your Pipeline version that you intent to use.
CASA <4>: pipeline.revision
Out[4]: 'r35156 (Pipeline-Cycle3-R3-B)’
You can check the latest version of Pipeline in Science
Portal (https://almascience.nao.ac.jp/documents-andtools).
34. Import ASDMs and Check calibrator
sources
1) Import ASDMs to convert MS, Generate WebLog.

2)

CASA<1>: h_init()
CASA<2>:hifa_importdata(vis=[‘ASDMName’],pipelinemode=‘automatic’)
CASA<3>:h_save()

Confirm that you have the converted MS, “flux.csv”,“*flagonline.txt”, “*flagtemplate.txt”, and “*flagcmds.txt”
in the working directory.
3) Open WebLog
A WebLog (directory) named “pipeline-*” should be
generated in your working directory.
(e.g. $firefox pipeline-2015XXXXTXXXXXX/html/index.html)

45a. Fix MSes (Cycle 0-2 data)
Some Cycle 0 - 2 data need to be *fixed* before running through Pipeline. But
you may want to run all these commands if you are not sure whether it isaffected or not.
NOTE: you need to install analysisUtils package to run these commands (seehttps://casaguides.nrao.edu/index.php?title=Analysis_Utilities).
CASA<1>: import analysisUtils as aU
CASA<2>: es = aU.stuffForScienceDataReduction()

1.
2.
3.

Fix field & source ID (for relatively old data)
CASA<1>: es.fixForCSV2555('uid___A002_X273b43_X146.ms’)
Fix SYSCAL tables:
CASA<2>: from recipes.almahelpers import fixsyscaltimes
CASA<3>: fixsyscaltimes(vis = 'uid___A002_X273b43_X146.ms’)
Fix planets’s coordinate:
CASA<4>: fixplanets(vis = 'uid___A002_X273b43_X146.ms’, field = '1’,fixuvw = T) ß----- Select “fieldID” of a planet in “filed”.
55b. Fix MSes (Cycle 3 data)
Some Cycle3 data may need be *fixed* before running through Pipeline. But you must
*NOT* run all these commands unless you are sure whether it is needed. The
following script is to apply fixes only when necessary (provided by R. Indebetouw).
1. Fix planets’ coordinate (if necessary):
SSOforfix = es.getFieldsForFixPlanets(thisvis)
if (len(SSOforfix) > 0):
print('Running fixplanets...')
for ssofield in SSOforfix:
fd.write(" fixplanets(vis = '" + thisvis + "', field = '" + str(ssofield) +"', fixuvw = T)")
print("# SACM/JAO - Fixes", file=fd)
fixplanets(vis = thisvis, field = str(ssofield), fixuvw = T)
2. Get correct Antenna Position correction info which will be used for hifa_antpos stage.

x=os.getcwd()
if x.count("2015.1")>0:
print("getting antenna positions")
es.correctMyAntennaPositions(mslist,lbc=True)

65c. Fix MSes (Cycle 4 & 5 data)• mslist = glob.glob('uid___A00*_X*_X*.ms')• es.correctMyAntennaPositions(mslist)

76. Prepare flux.csv file•hifa_importdata Search flux values on the calibrators, and fill in “flux.csv”. If
Pipeline create flux.csv with some values (calculated using the database stored inCASA) and needs to be updated, you may want to use:
CASA<1>:aU.getALMAFluxcsv(‘flux.csv’)•Sometimes Pipeline generate flux.csv with blank values, in that case you need tofill in by hand. You may want to use:
CASA<2>:aU.getALMAFlux(source,frequency, date=‘YYYY-MM-DD’)•$cat flux.csv (WARN: Do not use any *space* in the lines)
ms,field,spw,I,Q,U,V,comment
uid___A002_X273b43_X146.ms,2,1,14.1017,0.0,0.0,0.0,"intent=ATMOSPHERE,BANDPASS,WVR # +-0.903Jy, freq=114.664GHz"
uid___A002_X273b43_X146.ms,2,3,14.2698,0.0,0.0,0.0,"intent=ATMOSPHERE,BANDPASS,WVR # +-0.929Jy, freq=112.789GHz"
uid___A002_X273b43_X146.ms,2,5,15.2610,0.0,0.0,0.0,"intent=ATMOSPHERE,BANDPASS,WVR # +-1.088Jy, freq=102.726GHz"
uid___A002_X273b43_X146.ms,2,7,15.4575,0.0,0.0,0.0,"intent=ATMOSPHERE,BANDPASS,WVR # +-1.105Jy, freq=100.914GHz"
uid___A002_X273b43_X146.ms,3,1,1.74095,0.0,0.0,0.0,"intent=ATMOSPHERE,PHASE,WVR # +-0.113Jy, freq=114.664GHz"
uid___A002_X273b43_X146.ms,3,3,1.77152,0.0,0.0,0.0,"intent=ATMOSPHERE,PHASE,WVR # +-0.114Jy, freq=102.789GHz"
uid___A002_X273b43_X146.ms,3,5,1.95524,0.0,0.0,0.0,"intent=ATMOSPHERE,PHASE,WVR # +-0.125Jy, freq=102.726GHz"
uid___A002_X273b43_X146.ms,3,7,1.99233,0.0,0.0,0.0,"intent=ATMOSPHERE,PHASE,WVR # +-0.127Jy, freq=100.914GHz"
uid___A002_X273b43_X146.ms,4,1,14.1017,0.0,0.0,0.0,"intent=ATMOSPHERE,PHASE,WVR # +-0.903Jy, freq=114.664GHz"
uid___A002_X273b43_X146.ms,4,3,14.2698,0.0,0.0,0.0,"intent=ATMOSPHERE,PHASE,WVR # +-0.929Jy, freq=102.789GHz"
uid___A002_X273b43_X146.ms,4,5,15.2610,0.0,0.0,0.0,"intent=ATMOSPHERE,PHASE,WVR # +-1.088Jy, freq=102.726GHz"
uid___A002_X273b43_X146.ms,4,7,15.4575,0.0,0.0,0.0,"intent=ATMOSPHERE,PHASE,WVR # +-1.105Jy, freq=100.726GHz"

8

NOTE
If the absolute flux calibrator is solar systemobject, the flux.csv will be not be used in the
Pipeline processing. The flux density of solar
system is calculated using a model. Even though,the expected flux values of other calibrators influx.csv are useful to compare to the derived
one.
We recommend to use the updated flux values
to appropriate ones, rather than blindly
believing the automatically generated values.
97. Run Pipeline and check results withWebLog• Confirm that you have all ingredients before running Pipeline:$ls
uid___A002_X273b43_X146.ms
uid___A002_X273b43_X146.ms.flagversions
flux.csv• Start Pipeline (from MS)

CASA<1>:import pipeline.recipereducer as recipereducer
CASA<2>:recipereducer.reduce(vis=['uid___A002_X273b43_X146.ms'],procedure='
procedure_hifa_cal.xml’)– Note: if procedure=‘procedure_hifa_cal.xml’, it will terminate after
calibration, does not do imaging of targets. See the next page for moredetail.

• After completion of Pipeline, check WebLog. A WebLog (directory)
named “pipeline-procedure_hifa_cal” should be generated in yourworking directory.
(e.g. $firefox pipeline-procedure_hifa_cal/html/index.html)

10Note: procedure name••

To check which procedures are availble by listing up …/casa-releaseXXX/pipeline/pipeline/recipes/procedure*.xml(before CASA 5.1.1-pipe) e.g. in CASA 4.7-pipe––––•from CASA 5.1.1-pipe––––•

procedure_hifa.xml … Do calibration, imaging of calibrators and targets
procedure_hifacal.xml … Do calibration, imaging of calibrators
procedure_hifatargets.xml … Do restoration, imaging of targets
procedure_hsd.xml … [SD] Do calibration, imaging of targets
procedure_hifa_calimage.xml … Do calibration, imaging of calibrators and targets
procedure_hifa_cal.xml … Do calibration, imaging of calibrators
procedure_hifa_image.xml … Do restoration, imaging of targets
procedure_hsd_calimage.xml … [SD] Do calibration, imaging of targets

Note: even though you choose “procedure_hifa_calimage.xml” to dotarget imaging, if the estimated product size is too large, pipeline run may
terminate without doing target imaging (see description of
hif_checkproductsize in User’s guide)11

7a. Run Imaging Pipeline afterCalibration Pipeline• procedure_hifa_image.xml does not work?
– likely a bug in CASA 5.1.1• instead, use hifatargets– from pipeline.recipes import hifatargets as hifatargets– h_init()– hifatargets.hifatargets(vislist=[‘uid*.ms'])– h_save()• hifatargets– does not combine weblog for calibration and imaging– does not create products (e.g. FITS files,casa_pipescript.py), i.e. no hifa_exportdata
12

Useful Pipeline Documents• Brief description for Pipeline products and WebLog
contents:– “ALMA Science Pipeline User’s Guide” (available in SciencePortal)• List of Pipeline commands and descriptions for each
Pipeline task:– “ALMA Science Pipeline Reference Manual” (available inScience Portal)• When you need tweaking Pipeline after first run:– “Tweaking the Pipeline” (http://almaintweb.mtk.nao.ac.jp/~saigo/EAARC_CASA/reference/TweakPipeline.pptx.pdf)
13