Tweaking the Pipeline Script
Original version by Liz Humphreys (ESO)

Rie E. Miura (NAOJ)

ALMA/ASTE/Mopra Users Meeting 2015 Oct

1	  ALMA Pipeline (PL) Deliveries•

If the Pipeline has been used in a delivery, this will be marked in the
README file of the delivery package•

If the Pipeline has been used, then a special version of CASA
4.3.1 (->CASA 4.5) must be installed which contains Pipeline tasks
to re-create calibrated data•

Start using casapy –pipeline2	  Delivered	  Package	
• $	
  ls	
  …/2013.1.12345.S/science_goal.uid___A001_X100_X10a/group.uid___A001_X100_X10b/member.uid___A001_X100_X10c/	
  
• 	
  	
  	
  	
  |-­‐calibraAon/	
  	
  	
  	
  #ß	
  ﬂagtemplate.txt,	
  ﬂux.csv,	
  jeyperk.csv	
  	
  
	
  	
  	
  	
  |-­‐	
  log/	
  	
  	
  	
  	
  	
  	
  	
  	
  
	
  	
  	
  	
  |-­‐	
  product/	
  	
  	
  	
  	
  	
  	
  	
  #<-­‐	  FITS	  ﬁles	
  
	
  	
  	
  	
  |-­‐qa/	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  #ß	
  WebLog	
  
	
  	
  	
  	
  |-­‐README	
  	
  	
  	
  	
  	
  	
  	
  
	
  	
  	
  	
  |-­‐script/	
  
	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  |-­‐casa_piperestorescript.py	
  
	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  |-­‐casa_pipescript.py	
  
	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  |-­‐PPR_uid___A001_X100_X10d.xml	
  
	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  |-­‐scriptForImaging.py	
  
	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  |-­‐scriptForPI.py	
  
	
  	

Re-creating PL calibrated data
Interferometry data
• If the goal is to re-create the calibrated data produced
during the ALMA Quality Assurance process, thenscriptForPI.py should be used.
• scriptForPI.py calls casa_piperestorescript.py.
scriptForPI.py is the fastest way to obtain calibrated
data, e.g. before re-imaging ALMA data
• the casa_pipescript.py should only be used to fully recalibrate ALMA raw data when it is wanted to tweak an
aspect of the calibration4	  Re-creating PL calibrated data
TP data• Single Dish Pipeline products contain image FITS cubes withnative resolution. If you want to those with preferable
resolution, just use the CASA task imregrid.
• If the goal is to re-create the calibrated data produced during
the ALMA Quality Assurance process, then scriptForPI.py
should be used.
• scriptForPI.py calls casa_pipescript.py which fully recalibrates ALMA raw data when it is wanted to tweak an
aspect of the calibration.
• Note that Single Dish Pipeline does not have restore task. Recalibration will take hours to complete pipeline processing.
5	  _rethrow_casa_excepAons	  =	
  True	  Interferometryh_init()	
  
try:	
  
Pipeline	
	
  	
  	
  	
  hifa_importdata(vis=['uid___A002_X839000_X2a15',	
  'uid___A002_X839000_X2dc6'],	
  session=['session_1',	
  'session_2'])	
  
	
  	
  	
  	
  hifa_ﬂagdata(pipelinemode="automaAc")	
  
	
  	
  	
  	
  hifa_ﬂuxcalﬂag(pipelinemode="automaAc")	
  casa_pipescript.py	
   	
  	
  	
  	
  hifa_rawﬂagchans(pipelinemode="automaAc”)	
  (get	
  from	  the	
  script	
   	
  	
  	
  	
  hif_refant(pipelinemode="automaAc")	
  
	
  	
  	
  	
  hifa_tsyscal(pipelinemode="automaAc")	
  	
  directory)	
  
	
  	
  	
  	
  hifa_tsysﬂag(pipelinemode="automaAc")	
  
	
  	
  	
  	
  hifa_wvrgcalﬂag(pipelinemode="automaAc")	
  To	
  reproduce	
  QA2	
  
	
  	
  	
  	
  hif_lowgainﬂag(pipelinemode="automaAc")	
  ReducAon:	
  ﬂux.csv	
  
	
  	
  	
  	
  hif_setjy(pipelinemode="automaAc")	
  and	
  *ﬂagtemplate.txt	
   	
  	
  	
  	
  hif_bandpass(pipelinemode="automaAc")	
  
must	
  be	
  in	
  the	
  same	
  
	
  	
  	
  	
  hifa_spwphaseup(pipelinemode="automaAc”)	
  directory	
  
	
  	
  	
  	
  hifa_gﬂuxscale(pipelinemode="automaAc")	
  (get	
  from	  the	
  	
  
	
  	
  	
  	
  hifa_Amegaincal(pipelinemode="automaAc")	
  calibraDon	
  directory)	
   	
  	
  	
  	
  hif_applycal(pipelinemode="automaAc")	
  
	
	
  	
  	
  	
  hif_makeimlist(intent='PHASE,BANDPASS,CHECK')	
  
	
  	
  	
  	
  hif_makeimages(pipelinemode="automaAc")	
  ﬁnally:	
  
	
  	
  	
  	
  h_save()	
  _rethrow_casa_excepAons	  =	
  True	  Single Dish h_init()	
  
try:	
  
Pipeline
	
  	
  	
  	
  hsd_importdata(vis=['uid___A002_X839000_X2a15’])	
  Cycle3+ data	
	
  	
  	
  	
  	
  	
  	
  	
  hhsd_ﬂagdata(pipelinemode="automaAc”)	
  ifa_tsyscal(pipelinemode="automaAc")	
  
	
  	
  	
  	
  hifa_tsysﬂag(pipelinemode="automaAc")	
  casa_pipescript.py	
   	
  	
  	
  	
  hsd_mstoscantable(pipelinemode="automaAc")	
  (get	
  from	  the	
  script	
   	
  	
  	
  	
  hsd_inspectdata(pipelinemode="automaAc")	
  
	
  	
  	
  	
  hsd_calsky(pipelinemode="automaAc")	
  	
  directory)	
  
	
  	
  	
  	
  hsd_applycal(pipelinemode="automaAc")	
  
	
  	
  	
  	
  hsd_baseline(pipelinemode="automaAc”)	
  To	
  reproduce	
  QA2	
  
	
  	
  	
  	
  hsd_blﬂag(pipelinemode="automaAc")	
  ReducAon:	
  jyperk.csv	
   	
  	
  	
  	
  hsd_baseline(pipelinemode="automaAc”)	
  and	
  *ﬂagtemplate.txt	
   	
  	
  	
  	
  hsd_blﬂag(pipelinemode="automaAc”)	
  
must	
  be	
  in	
  the	
  same	
  
	
  	
  	
  	
  hsd_imaging(pipelinemode="automaAc")	
  directory	
  
	
  	
  	
  	
  hsd_exportdata(pipelinemode="automaAc")	
  (get	
  from	  the	  	
  ﬁnally:	  calibraDon	
  directory)	
   	
  	
  	
  	
  h_save()	  	

_rethrow_casa_excepAons	  =	
  True	  Single Dish h_init()	
  
try:	
  
Pipeline
	
  	
  	
  	
  hsd_importdata(vis=['uid___A002_X839000_X2a15’])	
  
	
  	
  	
  	
  hsd_ﬂagdata(pipelinemode="automaAc”)	
  Cycle1,2 data	
	
  	
  	
  	
  hifa_tsyscal(pipelinemode="automaAc")	
  
	
  	
  	
  	
  hifa_tsysﬂag(pipelinemode="automaAc")	
  casa_pipescript.py	
   	
  	
  	
  	
  hsd_mstoscantable(pipelinemode="automaAc")	
  (get	
  from	  the	
  script	
   	
  	
  	
  	
  hsd_inspectdata(pipelinemode="automaAc")	
  
	
  	
  	
  	
  hsd_calsky(pipelinemode="automaAc")	
  	
  directory)	
  
	
  	
  	
  	
  hsd_applycal(pipelinemode="automaAc")	
  
	
  	
  	
  	
  hsd_simplescale(pipelinemode="automaAc”)	
  To	
  reproduce	
  QA2	
  
	
  	
  	
  	
  hsd_baseline(pipelinemode="automaAc”)	
  ReducAon:	
  jyperk.csv	
   	
  	
  	
  	
  hsd_blﬂag(pipelinemode="automaAc")	
  and	
  *ﬂagtemplate.txt	
   	
  	
  	
  	
  hsd_baseline(pipelinemode="automaAc”)	
  
must	
  be	
  in	
  the	
  same	
  
	
  	
  	
  	
  hsd_blﬂag(pipelinemode="automaAc”)	
  directory	
  
	
  	
  	
  	
  hsd_imaging(pipelinemode="automaAc")	
  (get	
  from	  the	
  	
  
	
  	
  	
  	
  hsd_exportdata(pipelinemode="automaAc")	
  calibraDon	
  directory)	
   ﬁnally:	
  
	
	
  	
  	
  	
  h_save()	
  Since	
  Single	
  Dish	  Pipeline	
  is	
  opAmized	
  for	  Cycle3	
  data,	
  some	  Cycle1	
  &	
  2	
  data	  needs	  a	
  few	
  tweaks	
  aper	  importdata.	
  	

Using the casa_pipescript.py (1)• To re-create the calibration performed by the ALMA
Observatory: move the ALMA raw data, thecasa_pipescript.py, the flux.csv (for Interferometry data),
the jyperk.csv (for TP data) and *flagtemplate.txt into thesame directory• flux.csv contains calibrator fluxes. The flux calibrator values
may have been edited during the ALMA QA2 process• jyperk.csv contains Jy per Kelvin conversion factors. The
factors are obtained from the AmpCal data belonging to thesame project.
• *flagtemplate.txt files contain any additional flags added to
the PL run manually during QA2

Using the casa_pipescript.py (2)• If flux.csv and *flagtemplate.txt are not present then PL will
still run, but it will create new versions of these files without
any editing that was performed during QA2• [TP data] If jyperk.csv is not present then PL will still run, but
it does not convert the unit of the final images from Kelvin to
Jy/beam.
• Finally, if the rawdata has the suffix *asdm.sdm then thissuffix should be removed to use the script without editing• To run with default values:– casapy –pipeline– execfile(‘casa_pipescript.py’) (for Interferometry data/TP data)10	  A note on the pipelinemode•

You will notice in the script that pipelinemode=‘automatic’ is given
explicitly in each task•

This is to prevent the brackets from being empty, they mustn’t be empty•

But the pipelinemode is important. To see all the parameters that can bechanged in pipelinemode=‘automatic’, try typing e.g. inp
hifa_importdata•

Then type pipelinemode=‘interactive’ before doing the inp again•

Many more PL parameters can be altered by changing the pipelinemode11	
  Tweaking the script: (1) plotlevel• PL has 3 plotlevels:– default: generates all plots apart from for thehif_applycal task.
– summary : omits detail plots (e.g. hifa_timegaincal).
– all: generates all plots.
• Plotting is a major overhead in the current PL• To change from the ‘default’, edit:– h_init() à– h_init(pipelinemode=‘interactive’,plotlevel=‘summary’)13	  Tweaking the script: (2) data
• To change the ALMA raw datasets that need to becalibrated, edit the vis and session list in
hifa_importdata e.g.
•hifa_importdata(vis=['uid___A002_X72e960_X53d','uid___A002_X72e960_X53f'], session=[‘session_1’,‘session_1’]) à•hifa_importdata(vis=['uid___A002_X72e960_X53d’],session=[‘session_1’])

Cycle	  2	  CASA	
  PI	
  Workshop	
  

14	
  

Tweaking the script: (3) editing
the fluxscale• If there was a QSO flux calibrator, the flux used for it is givenin the flux.csv filems,ﬁeld,spw,I,Q,U,V,comment	
  uid___A002_X9fddd8_X2172.ms,0,17,2.9741650916,0.0,0.0,0.0,"intent=ATMOSPHERE,BANDP
uid___A002_X9fddd8_X2172.ms,0,19,2.9741650916,0.0,0.0,0.0,"intent=ATMOSPHERE,BANDP
uid___A002_X9fddd8_X2172.ms,0,21,2.9741650916,0.0,0.0,0.0,"intent=ATMOSPHERE,BANDP
uid___A002_X9fddd8_X2172.ms,0,23,2.9741650916,0.0,0.0,0.0,"intent=ATMOSPHERE,BANDP
uid___A002_X9fddd8_X2172.ms,0,25,2.9741650916,0.0,0.0,0.0,"intent=ATMOSPHERE,BANDP• The easiest way to change the flux used by PL is to edit
this file• Although the fluxes of other calibrators may be listed in thisfile, there is no point in editing them as they are derived during
PL processing	
  Cycle	  2	  CASA	
  PI	
  Workshop	
  

15	
  

Tweaking the script: (4) adding
additional flagging• This can be added to the *flagtemplate.txt files• Parameters are the same as the CASA task flagdata e.g.
– mode=manual spw='25:0~3;122~127'
timerange='2013/01/31/08:09:55.248~2013/01/31/08:10:01.296’
reason='stage8_2’• No white space should be in the parameter settings, only
between parameters .e.g.
– mode=‘manual’ antenna=‘DV07,DV09’ OK– mode=‘manual’ antenna=‘DV07, DV09’ NOT OK• To flag auto-correlation data (for TP data) .e.g.
– mode=‘manual’ antenna=‘PM02&&&’ (auto-correlation)– mode=‘manual’ antenna=‘PM02&&*’ (cross-&auto-correlation)
Cycle	  2	  CASA	
  PI	
  Workshop	
  

16	
  

Tweaking the script: (5) select areference antenna• By default Pipeline creates a ranked list of reference
antennas• This can be over-ridden to select a reference antenna• hif_refant(pipelinemode=‘automatic’) àe.g. hif_refant(pipelinemode=’automatic’,
hm_refant=’manual’, refant=’DV15’)17	
  Tweaking the script: (6) removing/
limiting calibrator imaging• By default Pipeline creates images of the bandpass andphase calibrators and any check source (1 per spectral
window)• To e.g. image the phase calibrator only change••

hif_makecleanlist(intent='PHASE,BANDPASS,CHECK') à
hif_makecleanlist(intent='PHASE')•••

To remove any calibrator imaging, then delete the lines:
hif_makecleanlist(intent='PHASE,BANDPASS,CHECK')
hif_cleanlist(pipelinemode="automatic")18	
  Tweaking the script: (7)changing reference amplitude
calibrator	
• By default Pipeline set fluxes for calibrator with AMPLITUDE
intent by hif_setjy. The flux densities to be set in flux.csv areused or he reference flux densities, if QSO.
• To e.g. use BANDPASS as flux calibrator.	
  • hif_setjy(pipelinemode="automaDc”)	
   à	  • hif_setjy(pipelinemode="interacDve",ﬁeld="J1337-­‐1257",intent="*BANDPASS*")	
  	
  • hifa_gﬂuxscale(pipelinemode="automaDc") à	
  • hifa_gﬂuxscale(pipelinemode="interacDve",reference="J1337-­‐1257",transfer="J1517-­‐243,	
  J1625-­‐2527",	
  reﬁntent="*BANDPASS*",transintent="*AMPLITUDE*,*PHASE*")	
  Save	  context	
•

	
  
•
•

Pipeline	
  reduces	
  data	  automaAcally	
  by	
  
–	
  SelecAng	
  the	  best	  processing	
  strategies	
  →	
  HeurisDcs	
  –	  Organizing	
  the	
  reducAon	
  environment	
  /	
  book-­‐keeping	  →	
  Context	
  	
  
	
  
For	
  diagnosAcs,	
  check	  the	
  WebLog	  aper	  you	
  run	
  each	  task.	
  	
  Pipeline	
  creates	  WebLog	
  each	  task	  soon	  aper	  compleAng	
  the	
  task.	
  	
  Handling	
  context:	
  – Pipeline	
  always	
  run	
  with	  a	
  context	
  h_init()	
  
	
  
– Do	
  not	
  forget	
  to	
  save	  context	
  before	
  closing	  CASA.	
  h_save(‘A.context’)	
  
	
  
– To	
  start	  again	
  but	
  from	  with	  a	
  speciﬁc	
  task:	  h_resume(‘A.context’)	
  ALMA	
  Pipeline	• Pipeline	  CASA	  4.5	
  commissioned	
  for	  at	
  least:	  – 12m	
  array	
  calibraAon	
  (<=	  Band	
  7)	
  	
  
– 7m	
  array	
  calibraAon	
  (<=	  Band	
  7)	
  	
  
– Single-­‐dish	
  end-­‐to-­‐end	
  processing	
  	
  
	
  

• ALMA	
  Data	  Deliveries	  – (1)	
  Keep	  the	
  same	  calibraAon	
  as	
  Observatory	
  	
  
• 	
  Use	
  scriptForPI.py	
  to	
  apply	
  calibraAon	
  tables	  (Faster	
  and	
  recommended!	
  for	  Interferometry	
  data.	
  But	
  slow	
  for	  TP	
  data).	
  	
  
	
  

– 	
  (2)	
  Alter	  the	
  calibraAon	  • Use	
  casapipescript.py	
  (Slower)	
  	
  ALMA	
  Pipeline	  InformaAon	• Available	
  from	  the	
  ALMA	  Science	  Portal	  	
  – hzp://www.almascience.org	
  	
  

You	
  can	
  also	  contact	
  us	
  via	  Helpdesk.	
  (hzps://help.almascience.org)	

Appendix	Overview	• ALMA	  have	  CASA	  Pipelines	
  	
  
	
  
• Currently	
  calibraAon	
  only	  –	
  DiagnosAc	
  calibrator	
  images	  –	
  ALMA	  science	
  target	
  imaging	  being	  commissioned	
  	
  
	
  
• The	
  Pipelines	
  use	
  dedicated	
  Pipeline	
  tasks	
  in	
  CASA	  	  	  • Execute	
  using	
  python	
  scripts	
  provided	
  by	
  the	
  telescopes.	
  Or	
  self-­‐build	
  	
  
	
  
• ALMA	  Pipelines	
  common	
  output:	
  Pipeline	  WebLog	  	  CASA	
  Pipeline	  Versions	• CASA	  4.2.2	
  and	  CASA	  4.3.1	
  have	  special	  versions	
  including	
  the	
  Pipeline	
  	
  
–	
  Obtain	
  from	  hzp://casa.nrao.edu/
casa_obtaining.shtml	
  	
  
	
  
• CASA	  4.4	
  has	  no	
  pipeline	
  version	
  	
  
	
  
• StarAng	
  with	  CASA	  4.5	  (this	  month),	
  one	  version	
  	
  
	
  
• Earliest	
  CASA	  versions	
  to	
  be	
  used	  with	  ALMA	
  Cycle	  3	  	
  data	  (manual	
  or	
  Pipeline)	
  CASA	  4.5	  	
  

Pipeline	  Task	  Types	Preﬁx	h_	

hif_	
hifa_	
hifv_	
hsd_	Task	
  Type	Common	Purpose	
Interferometry	
  and	  single-­‐dish	
  ALMA	
  &	
  
VLA	
Interferometry	
 ALMA	  &	
  VLA	
Interferometry	
 ALMA	
  only	Interferometry	 VLA	
  only	Single-­‐dish	ALMA	
  single-­‐dish	Pipeline	  Tasks	
  vs	
  CASA	  Tasks	
Pipeline	CASA	

hifa_importdata	importasdm	
  hifa_ﬂagdata	

ﬂagdata	hifa_wvrgcalﬂag	

wvrgcal	hifa_bandpass	

bandpass	hifa_gﬂuxscale	

ﬂuxscale	

higa_Ampegaincal	

gaincal	hif_applycal	

applycal	

Pipeline	
  tasks	
  use	  CASA	
  tasks	
  where	
  possible.	
  	
  Pipeline	
  tasks	
  can	
  also	  contain	
  heurisAcs	
  and	
  	
  
may	
  be	
  mulAple	
  CASA	
  tasks	
  bundled	
  together.	
  Pipeline	  CASA	  Commands	
  Log	

Pipeline	
  tasks	
  use	  CASA	
  tasks	
  where	
  possible.	
  	  Pipeline	  Design	
• Pipeline	
  reduces	
  data	  automaAcally	
  by	
  
–	
  SelecAng	
  the	  best	  processing	
  strategies	
  →	
  HeurisDcs	
  –	  Organizing	
  the	
  reducAon	
  environment	
  /	
  book-­‐keeping	
  →	
  Context	
  	
  
	
  
• •	
  ALMA	
  pipeline	
  implements	
  the	
  two	
  aspects	
  using	  the	
  “separaAon	
  of	
  concerns”	
  design	
  principle	
  	
  
• –	
  Mix	
  and	
  match	  of	
  steps	
  is	
  possible	  	
  Pipeline	
  ImplementaDon	
• HeurisDcs	
  and	
  Context	
  are	
  implemented	
  as	
  Python	
  Classes	
  	
  
	
  
• The	
  variables	
  for	
  heurisAcs	
  are	
  “parameters”,	
  the	
  ones	  for	
  context	
  are	
  “inputs”	
  	
  
	
  
• Pipeline	
  runs	  handle	  the	  context	
  automaAcally	
  	
  
	
  
• User	  interacAon	
  mainly	
  via	
  heurisAcs	
  parameters	
  	
  
	
  
• However	
  the	  context	
  can	  be	
  edited	
  to	
  insert	
  own	  calibraAon	
  tables	  	  Appendix:	
  Split	  SPW	  of	
  unwanted	
  Delaycal	
  aeer	  import	
  data	  (Cycle1	
  
and	
  2	
  data)	

• Cycle	  1	
  and	  2	  data	  usually	
  have	  delay	
  cal	  (check	
  source)	
  SPWs.	
  In	
  case	  that	  science	
  SPWs	  are	  not	
  idenAcal,	
  it	
  is	
  needed	
  to	
  split	  the	  MS	  to	
  exclude	
  delay	
  cal	  SPWs	  aper	  imporAng	
  data	  to	
  properly	
  process	
  in	
  Single	
  Dish	  Pipeline.	
CASA<1>:h_init()	
  CASA<2>:hsd_importdata(vis=[‘uid***’])	
  CASA<3>:h_save()	
  CASA<4>:os.system(‘mv	
  uid***.ms	
  uid***.ms.org’)	
  CASA<5>:split2(vis=‘uid***.ms.org’,	
  outputvis=‘uid***.ms.split’,	
  
	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  	
  spw=‘0~8,25~40’,	
  datacolumn=‘all’,keepﬂags=T)	
  CASA<6>:os.system(‘cp	
  –rf	
  uid***.ms.split	
  uid***.ms’)	
  CASA<7>:	
  h_init()	
  CASA<8>:hsd_importdata(vis=[‘uid***.ms’])	
  
….