# Presentation: 414585

## Suzaku processing and archive

- Lorella Angelini/HEASARC

## Highlights

- Data Processing
- Operations and Data flow
- Pipeline products & versions
- Status of Version 2
- US Archive collocated with the HEASARC
- Archive operations
- Archive content and version 2
- Public data
- Software and calibration distribution

## Data flow 

**GSFC processing**
**SFF + Cleaned**** **

**ISAS processing **
**SFF + Cleaned**
** **

**ISAS ****data production**
**FFT+RPT**
**Orbit **
**Attitude **
**Tim**
**ODB**

**GSFC HEASARC **
**PI distribution**
**Archive**

**ISAS**
**PI distribution**
**Archive**

**Databases**

**Pipeline software**

## Pipeline products (1)  

**Starting from the FFF the pipeline produces**
**    ****Science & housekeeping : data organized by observation on specific target  **
-        =>   Level 1 (unfiltered) , 2 (cleaned), 3 (products) for the HXD and XIS ,
-             their HK,  supporting files and log
**HXD :**
- Unfiltered events  for the WELL, WAM (x RPT) and bursts
- Cleaned event files . WELL RPTs are merged to create GSO and PIN files one for each clock rate.
- Average spectra PIN & GSO; light curves  PIN, GSO, WAM & bursts
**For each of the XIS units**
-  Unfiltered event files x RPT & edit modes. No sub-mode division
- Cleaned event files. All RPTs for specific edit mode are merged and splitted x sub-mode
- Average spectrum & light curve extracted for the XIS configuration with the longer exposure
**   ****=> These data are encrypted & distributed to the PI **
** **TOO & CAL observations are public immediately

## Pipeline products (2)  

**Starting from the FFF the pipeline also produces:**
**  ****Trend data**
- Data containing specific HK parameters or calibration information extracted from the event data
- These data are organized by instrument and data type
- These data are not encrypted and not distributed to the PI
**Starting from the ODB the pipeline produces:**
- Database tables ASCII (suzamaster and suzaxislog) updated when a new sequences is processed
- Used to select data via the Web based search facility (BROWSE)
**File format:**
- Science, HK & calibration data are  provided in FITS (OGIP standard)
- Preview of  summary products   (e.g.  GIF)
- HTML used to record processing

## Version 2 reprocessing

- Version 2 processing started late August 7
- All sequences from the start of the mission were reprocessed
- On Aug 28, 607 sequences completed, additional 57 early October
- PI were notified for version 2 availability
- The RPT and FFF were all in house when started
- While re-processing the newly observed data were done first
- Two hiccups while re-processing
- On July 28 the HXD PIN diode anomalous voltage
- GSFC was notified on Aug 29
- 6 affected sequences were already sent out to the PI
- After fix in place data were reprocessed and re-delivered
- 15 Sep 2007 XIS trend night and day changed screening criteria
-  Build separate pipeline to re-create ND files. The XIS NXB is generated from that
- In place by the 1 Oct all data regenerated by the 15

## Version 2 Pipeline (2)

- Data format changed (mk1stfits)  and new attitude files (longer in time)
- Version 6 of the software and  new calibration data
- XIS Charge Injection data  processed with the appropriate CALDB* data
- XIS timing data (psum) processed (minimal) & transferred to the archives
- XIS extraction regions are appropriate for the window mode
- XIS minor modes added in the mkf file to allow minor mode selection
- HXD gain uses CALDB files (previously gain calculated in the pipeline)
- HXD new products: pseudo event files & a burst ID file
- More strict control on the processing version run at ISAS & GSFC
- Known issues :
- The HXD background is not generated in the pipeline.
- Time Lag to when the data are ready & HXD background available
- Software bug in the GSO  PI calculation (fix in software vs 7)
- * The XIS gain/CTI file released in November adjust for an energy shift
- Applicable to data since Sep 2006. Users has to recalculate the gain for data processed with version < 2.1.6.16

## Pipeline versions updates 

## GSFC: Archive operations 

- In US the Suzaku archive is collocate at the HEASARC
- The GSFC processing site transfer data to the HEASARC
- All data (US & Japan) populate the archive
- For US PI and Co-PI :
-  HEASARC notifies the PI via E-mail when data are in the archive
- Same email is ‘bcc’ to ISAS.
-  PI are also notified if data are reprocessed within their propriety period with a new pipeline version.
- PI are notified 15 days before their data are going to be public
- The transfers and the archive population are  automatized
-  Cron job is running hourly to transfer Science &Trend data
- Science data are kept encrypted in the archive
- Databases are ingested in the W3Browse
- Copies made available in the FTP area. JAXA uses the same copy
- Data access via HEASARC Browse , wget or FTP

## Additional data  

- Daily retrieval of the Suzaku timeline from Japan
- Place on line in the HEASARC timeline Tool
- _[http://legacy.gsfc.nasa.gov/cgi-bin/Tools/timeline/timeline.pl ](http://legacy.gsfc.nasa.gov/cgi-bin/Tools/timeline/timeline.pl)_
- PIN background
- Generated in Japan and retrieved at HEASARC
- Background consistent with version 2 processing
- First available on  24 September
-  A newer version after Oct 1 (Fix in the GTIs)
- Available in the HEASARC FTP area
- _[http://legacy.gsfc.nasa.gov/suzaku/data/background/](http://legacy.gsfc.nasa.gov/cgi-bin/Tools/timeline/timeline.pl)_
- Cron job looks at the Japan site for new updates daily
- GSO background is not yet available (end of December)

## Data delay before distribution 

## Archive content (1)

- There are 769 observations  at the HEASARC
- 220 are SWG
- 330 are AO1 & 190  A02
- 7 real TOO (public immediately)
- 9 are SWG-TOO, 3 AO1-TOO, 10 AO2-TOO
- There are 208 US observations (PI 198 & 10 Co-PI)
- AO1 and AO2
-  What is missing :
- 13 observations not in the archive compared with the timeline
-  2 non science in the data recorded (402089010, 702042030)
-  1 invalidated because never made to the source (800013010)
-  10 are from  > Nov 30
- Steps pointing (short observations)
- Slew data

## Archive and Public data 

- May 27 the propriety period  for the SWG data ended
- All SWG data were decrypted, checked for data corruption & put on-line
- The public archive opened on  1 June
- AO1 data also start going into the public archive
- Operationally : ISAS sends to GSFC  list observations to  become public 15 days ahead their public date.
- GSFC notify US PI & Co-PI  within 10day , before data are decrypted
- At the end of the  propriety period  both archive sides decrypt the observation.  It is expected  up to 2 days difference .
- Currently the archives includes 393  public  sequences  data
- TOO , AO1 and SWG data

## Data download

- Typical size sequence 800 MB compressed largest is 3 GB
- 4.2% of the total  HEASARC download in the last 10 months (but archive opens only in June)
- largest is Swift with 48%

##  Software and CALDB

**Calibration data  distributed via the HEASARC  CALDB**
- Deliveries from Japan arrive to the GOF
- GOF checks the agreed file standards and requires CALDB keywords
- GOF deliveries the file to the HEASARC for ingest into CALDB and distribution
-  CALDB updates one a month (end of the month, if any)
**Software distributed in the HEAsoft package**
- Suzaku tools  are developed in Japan & delivered to the HEASARC by the Software Team via CVS
- General support software is provided by the HEASARC
- GSFC performed the software testing before distribution :
- Individual tool test bed provided from Japan
- Within the pipeline with test bed  generated at GSFC
- Software updated on 3 months timescale (if needed)
- Software allows to reprocess the data (all the steps in the pipeline are reproducible)
- Software available from the HEASARC software distribution page

## Software distribution 

- Next software freeze end of February

- Seven  Software releases.  First public software release May 2006
- Most of the software is used within the pipeline
- Version 3:  tools updates  not relevant to the pipeline processing
-  Version 4 : tools updates mostly in parameter file adjustment and CALDB query
-  Version 5 : New tools & updates, support for CI , attitude library, Support for     new/old version of the science and CALDB  files.
-  Version 6: minor bug fixes (running in the pipeline)
- Version 7 : new XIS background generator, GSO bug fix and minor updates
- To be import in the pipeline

## CALDB distribution 

-  21 CALDB releases
- First release April 2006
- Include all files necessary for processing or for the spectral  analysis  . Update if any once a month
- Pipeline always run the latest CALDB if relevant to processing
- Last delivery released with software v7 includes :
- First release of the XIS NXB file (based however on older processing version)
- New XIS gain file (to be applied to all data taken from sep 2006 and processed with version < 2.1.6.16)
- New HXD GSO gain file
- Next release expected end of December
- Updates on contamination
- Updates on HXD gain

## Conclusions  

-    Data have all been processed with version 2
- Pipeline routinely updates for new CALDB files
- Next major reprocessing is TBD
- Just released software version 7 and Calibration file
- Next software release expected in March
- Calibration data released when necessary (time scale 1 month)
-  Several Suzaku data are now public