# Presentation: 010657

## Suzaku processing and archive

- Lorella Angelini/HEASARC

## Highlights

- Data Processing
- Operations and Data flow
- Pipeline products & versions & current issues
- Upcoming version 2
- US Archive collocated with the HEASARC
- Archive operations
- Archive content and version 2
- Public data
- Software and calibration distribution

## Basic Operations

- At ISAS after an observation is performed  :
- Data are packed in Raw Packet telemetry (RPT)
- RPT are transformed in First FITS Files (FFF)  via mk1stfits
*These are uncalibrated FITS files*
- Orbit, Attitude and time correction (tim)  files are generated
- Observation database (ODB metadata) is updated
- GSFC implements the processing pipeline & installs it at ISAS
- Two sides are  running the same pipeline on all data
- RPT, FFF, orbit, attitude, tim files  and ODB are transferred to the processing sites and used as input to the pipeline
- Generate calibrated and cleaned event files, and average products
- After processing data are transferred to the archive encrypted
- Archives notify the PI that data are ready to be picked-up
- Databases are made at GSFC (post processing) & grabbed by ISAS

## Data flow 

**GSFC processing**
**SFF + Cleaned**** **

**ISAS processing **
**SFF + Cleaned**
** **

**ISAS ****data production**
**FFT+RPT**
**Orbit **
**Attitude **
**Tim**
**ODB**

**GSFC HEASARC **
**PI distribution**
**Archive**

**ISAS**
**PI distribution**
**Archive**

**Databases**

**Pipeline software**

## Transfer rates, size, process time

- Transfer rate to GSFC is 250 kbytes/s
- Transfer occurs without hand-shake  between sites.
- More robust check soon in place for version 2
- Data Size at GSFC :
- Processing : RPT 52 Gb , FFF+Additional files  92 Gb
- Archive  :  558 Gb (output from processing)
- Typical processing time
- ~ 30 minutes small sequences
- 5-7 days large sequences (3 Gb of FFF equivalent to 6.2 Gb after processing)

**Problem in determining the attitude solution**

## Pipeline products (1)  

**Starting from the FFF the pipeline produces**
**    ****Science & housekeeping : data organized by observation on specific target  **
-        =>   Level 1 (unfiltered) , 2 (cleaned), 3 (products) for the HXD and XIS ,
-             their HK,  supporting files and log
**HXD :**
- Unfiltered events  for the WELL, WAM (x RPT) and bursts
- Cleaned event files . WELL RPTs are merged to create GSO and PIN files one for each clock rate.
- Average spectra PIN & GSO; light curves  PIN, GSO, WAM & bursts
**For each of the XIS units**
-  Unfiltered event files x RPT & edit modes. No sub-mode division
- Cleaned event files. All RPTs for specific edit mode are merged and splitted x sub-mode
- Average spectrum & light curve extracted for the XIS configuration with the longer exposure
**   ****=> These data are encrypted & distributed to the PI **
** **TOO & CAL observations are public immediately

## Pipeline products (2)  

**Starting from the FFF the pipeline also produces:**
**  ****Trend data**
- Data containing specific HK parameters or calibration information extracted from the event data
- These data are organized by instrument and data type
- These data are not encrypted and not distributed to the PI
**Starting from the ODB the pipeline produces:**
- Database tables ASCII (suzamaster and suzaxislog) updated when a new sequences is processed
- Used to select data via the Web based search facility (BROWSE)
**File format:**
- Science, HK & calibration data are  provided in FITS (OGIP standard)
- Preview of  summary products   (e.g.  GIF)
- HTML used to record processing

## Pipeline versions  

## Known Pipeline issues  

- XIS Charge Injection data are not processed with the appropriate calibration data
- XIS timing data (psum) are not processed or transferred to the archives
- XIS events are not screened for the Cut-off rigidity
- XIS extraction regions are always circles  not check in place for modes where the circle is outside of the detector window.
- XIS minor modes added in the mkf file to allow event minor mode selection
- HXD gain is applied using two different methods :
- (1) Gain from CALDB or  (2) Gain calculated on the fly
- If (2) users are recommended to reprocess the data soon after the gain CALDB file is available
- HXD background is not generated with the pipeline
- Not all sequences were processed with the same pipeline at ISAS & GSFC  New verification will be in place with version 2 processing.

## Version 2 reprocessing    

- Version 2 processing foreseen to start late June
- All data will be reprocessed (almost 2 years of data)
- Data format will change (mk1stfits)
- All data need to be re-transferred from ISAS to GSFC
- New release of the Suzaku software and calibration data
- New way to calculate the attitude (attitude files longer in time)
- Most of the Known issues will be addressed
- but for the HXD background and the HXD gain
- HXD new products: pseudo event files and a burst ID file
- Not a good estimate to how long will take to reprocess all data
- Mainly driven by the rate on how quickly ISAS can  prepare the data
-  New data will have high priority
- Previous reprocessing uses 5 months to process  1.3 year

## GSFC: Archive operations 

- In US the Suzaku archive is collocate at the HEASARC
- The GSFC processing site transfer data to the HEASARC
- All data (US & Japan) populate the archive
- For US PI and Co-PI :
-  HEASARC notifies the PI via E-mail when data are in the archive
- Same email is ‘bcc’ to ISAS.
-  PI are also notified if data are reprocessed within their propriety period with a new pipeline version.
- The transfers and the archive population are  automatized
- Cron job running hourly retrieve data from the processing site via DTS (data transfer system)
- Science & trend data are archived via DAS (data archive system)
- Science data are kept encrypted in the archive
- Databases are ingested in the W3Browse
- Copies made available in the FTP area

## Data delay before distribution 

## Archive content (1)

- There are 566 observations  at the HEASARC
- 216 are SWG
- 317 are AO1 & 15  A02
- 5 real TOO (public immediately)
- 9 are SWG-TOO, 3 AO1-TOO, 1 AO2-TOO
- There are 127 US observations (PI 112 &  15 Co-PI)
- AO1 and AO2
-  What is missing :
- 8 observations not in the archive compared with the timeline
-  5 problems in processing  (none are PI data), 3 not yet here
- 100 steps pointing (short observations)
- Slew data
- Background HXD

## Archive content (2) 

- Daily retrieval of the Suzaku timeline
- Place on line in the HEASARC timeline Tools
- GSO and PIN background
- Retrieved at HEASARC every month but currently not on-line
- Difficult to keep-up with changes that occur at ISAS
- There is a noticeable time delay between the data distribution and HXD background availability
- Version 2 processing will not produced the background
- HXD team will still produce the background
- A copy will be given  to the archives
- However timescale or the transfer mechanism are not clear

## Archive access 

## Archive: Version 2 processing

- HEASARC data management for version 2 processing:
- The results of the version 2 processing will populate new disks
- The content of the current disks (processing version 1.X.X.X) will be backup into final archive tapes
- New disks will be populated at first with links to the 1.X.X.X data
- The link is substituted with the version 2 soon as ready
- Users will always see the latest version
- Version 1.X.X.X of observations that have already the version 2 available  is kept on disk but not visible from the users
- When all data are reprocessed, the 1.X.X.X version will be moved out from disks

## Archive and Public data 

- Currently the archives includes 56 public  sequences  data
- 5 are TOO , 10 are from SWG time, remaining are calibration observations
- May 27 the propriety period  for all the SWG data ends
- All SWG data will be decrypted, checked for data corruption & put on-line
- Estimate time to conclude operation ~10 days (or earlier)
- AO data. The propriety period is one year.  The clock starts after calibration is made available (CI and Timing) . Public date set at ISAS
- Operationally : ISAS sends to GSFC the list with the observations  that  become public 15 days ahead their public date.
- GSFC notify US PI & Co-PI  within 10d , before data are decrypted
- At the end of the  propriety period  both archive sides decrypt the observation.  It is expected  up to 2 days difference .

##  Software and CALDB

**Calibration data  distributed via the HEASARC  CALDB**
- Deliveries from Japan arrive to the GOF
- GOF checks the agreed file standards and requires CALDB keywords
- GOF deliveries the file to the HEASARC for ingest into CALDB and distribution
-  CALDB updates one a month (end of the month, if any)
**Software distributed in the HEAsoft package**
- Suzaku tools  are developed in Japan & delivered to the HEASARC by the Software Team via CVS
- General support software is provided by the HEASARC
- GSFC performed the software testing before distribution :
- Individual tool with test bed provided from Japan
- Within the pipeline with test bed  generated at GSFC
- Software updated on 3 months timescale (if needed)
- Software allows to reprocess the data (all the steps in the pipeline are reproducible)
- Software available from the HEASARC software distribution page

## Software distribution 

- Next software freeze May 7. Expected release in  early June
- New tools and updates, support for CI .  Attitude library update new rigidity file
- Updates of the help files
- Support for new/old version of the science FITS files
- Support for new/old version of the CALDB files
- This new version will be used in the version 2 of the pipeline

- Four Software releases.  First public software release May 2006
- Most of the software is used within the pipeline
-  Pipeline is running version 2 of the software
-  Version 3:  tools updates  not relevant to the pipeline processing
-  Version 4 : tools updates mostly in parameter file adjustment and CALDB query

## CALDB distribution 

-  Twelve CALDB releases
- First release April 2006
- Include all files necessary for the processing or for the spectral  analysis
- CALDB updated in the pipeline if new files  are relevant to the processing
- Next delivery is  May 7 . Includes :.
- Files necessary for the CI data
- New gain format for the GSO
- Still under discussion the creation of a XIS background data

## Conclusions  

- NEXT two months  :
-    Processing version 2 is coming :
- All data will be reprocessed
- New software and Calibration soon to be released
-  Several Suzaku data will be made public

## Software test procedure (??)  

- Next release foreseen
- Software changes
- Data file changes
- Calibration changes
- Next software release has to be tested against:
-  Two different data formats ,
-  since for a while 2 processing version will be availalble
- New calibration formats , need to work on old and new data format