Using ALMA Archive
ALMA Science Archive substructure and operation

Erik Muller,
Ken’ichi Tatematsu,
Daisuke Iono.
EA-ARC ALMA user support(NAOJ Chile Observatory)

ALMA Science Archive - ASA:Synopsis

1. ASA: Motivated by Scientists
2. Structure of the Archive facility
3. Obtaining data from the archive
4. Structure of Archived data.
5. PI publication requirements
6. Future development.

ALMA Science Archive - ASA:
Motivation and Mission
What do scientists want?
1. Reduced, science-ready, high-level data (PIs want proprietary protection)
2. Processible, accessible data
3. Data they can reprocess/analyse with their “pet” data analysis software
Solution:
1. Provide processed, calibrated data, with clear description of the processing
2. Provide general format data
3. Provide raw data for the PI to completely re-process, if desired.

ALMA science Archive caters for TWO “clients”:

ALMA Science Archive - ASA:
Motivation and Mission
What do scientists want?
1. Reduced, science-ready, high-level data (PIs want proprietary protection)
2. Processible, accessible data
3. Data they can reprocess/analyse with their “pet” data analysis software
Solution:
1. Provide processed, calibrated data, with clear description of the processing
2. Provide general format data
3. Provide raw data for the PI to completely re-process, if desired.

ALMA science Archive caters for TWO “clients”:

1. The principle investigator and“delegated” collaborators.
(Exclusive access during theproprietary period).

2. Future Archive Researchers(After Proprietary period expires,
data is accessible by anyone).

ALMA Science Archive - Archive structure
PINGASNGAS
Archive
Telescope

Processing staff

ALMA Science Archive - Archive structurePI

Proposal submission

NGASNGAS
ArchiveTelescope

Processing staff

ALMA Science Archive - Archive structurePI

Proposal submission

NGASNGAS
ArchiveProposal
retriev

alTelescope

Processing staff

ALMA Science Archive - Archive structurePI

Proposal submission

odr
Ptucuaolp

NGAS
NGAS

dPro

posal
retrievalArchive

Telescoperoc

PProcessing staff

inses.
g..

Ra

wd

ata

uploa

d

ALMA Science Archive - Archive structure
PIArchive query

odr
Ptucuaolp

Proposal submission

NGAS
NGAS

dPro

posal
retriev

alArchive

Telescoperoc

PProcessing staff

inses.
g..

Rawd

ataupload

ALMA Science Archive - Archive structurePI

Data Download

Archive query

Requesthandler

odr
Ptucuaolpd

Proposal submission

NGASNGAS

Proposal
retriev

alArchive

Telescoperoc

PProcessing staff

inses.
g..

Rawd

ata

up

load

ALMA Science Archive - PI delivery:
Current status-Presently, EA is delivering data primarily by FTP transfer.
(Resulting in a shorter delivery time, after staff-processing stage is complete)-We intend to migrate to delivery by Archive internet data distribution, in the near
future (~3-5 weeks).

-Some deliveries are already being made using Archive internet data-distribution.

-Delivery is currently effected through one or more ‘tarballs’.

-Raw data (‘ASDMs’) not ‘delivered’ by default in cycle 1+ data, but areavailable directly by internet request.
Future researchers can optimise pipeline processing for their scienceALMA Science Archive - PI delivery:
PI notice- PI is currently notified by email, using ALMA helpdesk.
- Email will contain instructions on how to obtain your data, one of two methods:- 1. by FTP (File transfer Protocol), from EA-ARC firewall-protected repository- 2. from the Archive data distribution system, using your ALMA login.
PI - Notification for Delivery by FTP transfer
Dear Albert Einstein,
We are pleased to announce that data of the Scheduling Blocks “alpha_cen_12m”
containing execution block uid___A00_X123_X456 from your project 2012.1.00111.Swith the title"Alpha centauri and all that is beautiful"
are now available to you for download. A tarball of these data can be obtained from
(1) Using web browser:
FTP URL: ftp://(2) Using FTP-specific client software:
Host: alma-dl.mtk.nao.ac.jp
User: [user]
Path: [path]
Only passive-mode FTP is available.
The password will be sent to you via separate email. Please reply to this ticket if you donot receive a password within 1hour.
The account will expire at 19th april,16:00 JST.

ALMA Science Archive - PI delivery:
PI notice- PI is currently notified by email, using ALMA helpdesk.
- Email will contain instructions on how to obtain your data, one of two methods:- 1. by FTP (File transfer Protocol), from EA-ARC firewall-protected repository- 2. from the Archive data distribution system, using your ALMA login.
PI - Notification for Delivery by Archive data distribution
Dear Albert Einstein,
We are pleased to announce that data of the Scheduling Blocks “alpha_cen_12m”
containing execution block uid___A00_X123_X456 from your project 2012.1.00111.Swith the title"Alpha centauri and all that is beautiful"
are now available to you for download. A tarball of these data can be obtained from theALMA Science Archive.
Please use your ALMA userid "username" to login to the ALMA Science Portal,
http://almascience.nao.ac.jp
and follow the "Data" -> " link to the archive to search for and download your data,or access tohttp://almascience.nao.ac.jp/aq/asdm_uid= uid___A00_X123_X456
A download request of both processed and raw data can be submitted by selecting alldata (topmost checkbox) and hitting "submit download request".
.
.
.

ALMA Science Archive - ASA:
Delivery: PI and AR - what do you get?
Delivery content for Cycle 1 deliveries + onwards.
FITS-

Data products (FITS cube)

LOGS

-

Observing Logs

Scripts-Calibration, processing + Imaging Scripts

Cal. Tables-Calibration tablesQA rept.

-

Quality Assesment (QA0, QA2) Reports

README-README file (text file describing contents of delivery)

ALMAALMA Science Archive - ASA:
Delivery: PI and AR - what do you get?
Delivery content for Cycle 1 deliveries + onwards.

ALMAREADME
FITS
Scripts
Cal.
Tables
QA
rept.
LOGS

-

Data products (FITS cube)-

Observing Logs-

Calibration, processing + Imaging Scripts-

Calibration tables-

Quality Assesment (QA0, QA2) Reports-

README file (text file describing contents of delivery)

Delivery to PI via network

ALMA Science Archive - Archive researchProprietary rules:- PI Data is available for public use after 12 calendar months(6 months for director discretionary time)- Science verification data available after verification
Accessible only by PI

ALMA PI & delegated COIs

QA2

12 Mnths after delivery

Science verification

QA2

verification

Accessible to anonymous

Proprietary-expired data are all available through the Science portal.

ALMA Science Archive - Archive research
Getting proprietary-expired data:
1. Submit query on ‘Archive query’OR ‘delivery list’
2. Refine query and request data
on Request handler
3. Download.

ALMA Science Archive - Archive research
Getting proprietary-expired data:
1. Submit query on ‘Archive query’OR ‘delivery list’

Access Alma science portal:
http://almascience.nao.ac.jp

2. Refine query and request data
on Request handler
3. Download.

Navigate to
DATA -> Archive
Choose Archive queryALMA Science Archive - Archive research
Getting proprietary-expired data:
1. Submit query on ‘Archive query’OR ‘delivery list’

Access Alma science portal:
http://almascience.nao.ac.jp

2. Refine query and request data
on Request handler
3. Download.

Navigate to
DATA -> Archive
Choose Archive query

1. Enter search parameters
2. Click Search.
3. Refine parameters

ALMA Science Archive - Archive research
Getting proprietary-expired data:
1. Submit query on ‘Archive query’OR ‘delivery list’

Access Alma science portal:
http://almascience.nao.ac.jp

2. Refine query and request data
on Request handler
3. Download.

Navigate to
DATA -> Archive
Choose Archive query

1. Enter search parameters
2. Click Search.
3. Refine parameters
Specify file selection inRequest Handler

ALMA Science Archive - Archive research
Getting proprietary-expired data:
1. Submit query on ‘Archive query’OR ‘delivery list’

Access Alma science portal:
http://almascience.nao.ac.jp

2. Refine query and request data
on Request handler
3. Download.

Navigate toDATA -> Archive
Choose Archive query

1. Enter search parameters
2. Click Search.
3. Refine parameters
Specify file selection in
Request Handler
Download with
provided wget scripts

Download frominternet

ALMA Science Archive - Archive research
Getting proprietary-expired data:
1. Submit query on ‘Archive query’OR ‘delivery list’

Access Alma science portal:
http://almascience.nao.ac.jp

2. Refine query and request data
on Request handler
3. Download.

Navigate to
DATA -> Archive
Choose Archive query

Choose project download(‘delivery list’)

1. Enter search parameters
2. Click Search.
3. Refine parameters
Specify file selection inRequest Handler
Download with
provided wget scripts

Download frominternet

ALMA Science Archive - Archive research
Getting proprietary-expired data:
1. Submit query on ‘Archive query’OR ‘delivery list’

Access Alma science portal:
http://almascience.nao.ac.jp

2. Refine query and request data
on Request handler
3. Download.

Navigate to
DATA -> Archive
Choose Archive query

1. Enter search parameters
2. Click Search.
3. Refine parameters

Choose project download(‘delivery list’)

1. Find target project,
2. Select with checkbox.
3. Click ‘Request marked deliveries’
Specify file selection in
Request Handler

Download with
provided wget scripts

Download frominternet

ALMA Science Archive - Archive researchArchive Query:
When using JAVA 7.51+: security settings should be modified.

- Enables ‘standard’ search patterns- target, position, band. Observing date & conditions. Also: project code, PI name- Enables advanced filtering capbilities- Limit search by linking logical filter parameters (band, sensitivity, etc. etc.)- Download links are returned to user.

ALMA Science Archive - Archive research
Request Handler- Enables selection of specific tarballs (note, duplicity exists in some cases, for cycle 0 data)- Provides a text linux script invoking wget commands, which may be run from the terminal.

ALMA Science Archive - Archive research
Cycle 0 - vs - cycle 1- Cycle 0 data includes the raw data.
-(in an ASDM format file e.g. uis___X123_X123_X12.asdm.sdm.tar)
Cycle 0 data also contains the ‘measurment set’ data, which is calibrated
Cycle 0 data also contains the products in FITS format- Cycle 1 data tarballs to NOT include the raw data, nor the calibrated measurement set.
- Cycle 1 data tarballs include the products, in FITS format.

ALMA Science Archive - Archive research
What next? for PIs and Archive researchers.
- You will receive one, or many tarballs - untar these-(macs can’t do this properly from a on-screen click. Untar the folders from thecommand line)
Examine the README file, it contains information about the contents of the tarballs
Examine the PRODUCTS directory, it contains FITS-format images of the target
frequencies (or continuum), made by the PI

Atacama Large Millimeter/submillimeter Array (ALMA)#####Cycle: 0 (Early science)
Project code: 2011.0.00199.SProject title: Bursting Water Maser Feature in Orion
KLConfiguration: Extended
Proposed rms: 170mJy
CASA version used for reduction: 3.4
Comments from Reducer:
DA42, DV01, DV07, and DV23 were removed because of large pointing error.
DV17 was removed due to negative Trec.
DV19 XX pol. was removed due to out of spec Trec temperature and high Tsys.
DA51 was removed due to bad bandpass gain solutions.
DV05 was removed due to poor S/N in bandpass.
The rms was achieved to be 105 mJy (the required rms is 140 mJy) by theintegration in 7.5 GHz in frequency width but the image may contain non-real detection.
WVR correction seems to make phase worse in some antennas. It is probably
better not to use them in those antennas.
emuller@alma-work:~/2011.0.00199.S$ more `find -name README`
Atacama Large Millimeter/submillimeter Array (ALMA)(C0, C1+) README contains
1. Basic information about the data.
2. Notes from the data reducer:
flagging, high tsys, etc.
3. A directory tree

- 'calibrated' contains the fully calibrated ms for each execution block.
- 'calibration' contains the files needed for calibration starting fromthe initial ms to the fully calibrated data. Plots are included.
- 'script' contains the reduction scripts used to process the initial ms
to calibrated data, but also to obtain concatenated data (if more thanone execution) and imaging products. There are usually several scripts
dealing with different parts of the processing.
- 'product' contains the fits files of the selected image products.
These You
will notincludeall imagesof scientificvalue,
but will indicate
willreceiveone,or many
tarballs- untar thesethe quality of the calibration and images.
(macsthecan’t
do thisfrom
on-screen click. Untar the folders from the- 'qa' contains
qa2 reportsthat properly
show plots andtext ainformationneededto assess the qualitycommandline) of the processing. The resultant image
rms, compared with that proposed, is given.
- 'log' contains
Examinethe casathelogREADME
files.
file, it contains information about the contents of the tarballs

ALMA Science Archive - Archive research
What next? for PIs and Archive researchers.

-- Examine the PRODUCTS directory, it contains FITS-format images of the target#####
frequencies (or continuum), made by the PI|-- 2011.0.00199.S/| |-- sg_ouss_id/| | |-- group_ouss_id/| | | |-- member_ouss_2013-03-01_id/| | | | |-- README| | | | |-- product/| | | | | |-- uid___A002_X590461_X620.ms.split.image.continuum.source4.image.fits| | | | | |-- uid___A002_X590461_X620.ms.split.image.continuum.source4.mask/| | | | |-- calibration/| | | | | |-- uid___A002_X590461_X620.calibration.plots/| | | | | |-- uid___A002_X590461_X620.calibration/| | | | |-- log/
| | | | | |-- uid___A002_X590461_X620.calibration.log| | | | |-- raw/
| | | | | |-- uid___A002_X590461_X620.ms.split/| | | | |-- calibrated/| | | | | |-- uid___A002_X590461_X620.ms.split.cal/| | | | |-- qa/| | | | | |-- uid___A002_X590461_X620__qa2_part1.png| | | | | |-- uid___A002_X590461_X620__qa2_part2.png| | | | | |-- uid___A002_X590461_X620__textfile.txt| | | | | |-- uid___A002_X590461_X620__qa2_part3.png| | | | |-- script/| | | | | |-- uid___A002_X590461_X620.ms.scriptForCalibration.py| | | | | |-- uid___A002_X590461_X620__scriptForImaging.py(C0, C1+) README contains
1. Basic information about the data.
2. Notes from the data reducer:
flagging, high tsys, etc.
3. A directory tree

ALMA Science Archive - Archive research
Raw data, and measurement sets in Cycle 1+

How to make cycle 1 measurement sets (refer to README information):- Cycle 1 data includes the CASA processing script, which runs on the-raw data (asdm format)
PIs and Archive users can obtain the raw data from the archive, put itin the ‘calibrated’ directory, and run the script (or adjust it)

Why do we do this?
- Not all PIs/ARs need the measurement set, it is very bulky (~100’s-GB), and slows transfer
Not all PIs/ARs need the raw data, but it must be available.

ALMA Science Archive & JVO:
http://alma-intweb.mtk.nao.ac.jp/~kt/ALMA_archive_handbook.pdf
locally-stored data query interface

ALMA Science Archive & JVO:
http://alma-intweb.mtk.nao.ac.jp/~kt/ALMA_archive_handbook.pdf
Purpose-built data visualisation with “Vissage”

JVOVissage

ALMA Science Archive - ASA:
Required acknowledgement information.
Please notify EA-ARC and EA-ARC outreach for any publications making
use of ALMA data.

Publications making use of ALMA data (PI and Archive researchers) are required
to include the following acknowledgements in publications:“This paper makes use of the following ALMA data: ADS/
JAO.ALMA#2011.0.01234.S . ALMA is a partnership of ESO (representing its
member states), NSF (USA) and NINS (Japan), together with NRC (Canada)
and NSC and ASIAA (Taiwan), in cooperation with the Republic of Chile. The
Joint ALMA Observatory is operated by ESO, AUI/NRAO and NAOJ.”

ALMA Science Archive - ASA:
Where to now?
Immediate Future improvements
1. Basic layout to be modified slightly
2. ‘Frequency support’ information to be improved
3. File Download destination support improved
4. Filtering improved
5. online help improved.
6. Visual quick-look planned.

Not happy with the Archive functionality?
Or are you having problems?
YOU are the users, but we need your input to make the system better for everyone.
We can only improve the archive if you tell us about problems, or any suggestions.
If you have any comments or suggestions, please submit them to EA-ARC using thehelpdesk system (https://help.almascience.org).
Please also ensure you examine the ‘help’ information on both the Archive query andRequest handler pages.
Please contact EA-ARC for face-to-face support information