The Herschel Data Processing
System – HIPE and pipelines –doing well during Herschel’s
early mission phase
Stephan Ott
Herschel Science Centre

on behalf of the Herschel Science Ground Segment
Consortium
http://herschel.esac.esa.int/DpHipeContributors.shtml
PACS

ADASS XIX 06/10/2009
Stephan Ott
VG # 1

http://herschel.esac.esa.int/

Overview• Herschel andHerschel DataProcessing• Key S/W Elements,
Status and further
Milestones ofHerschel DataProcessing• What can HIPE dowith Herschel data?
• SummaryPACS

ADASS XIX 06/10/2009
Stephan Ott
VG # 2

http://herschel.esac.esa.int/

Overview of Herschel• ESA cornerstoneobservatory– Instruments nationally funded• Far Infrared (55 - 672 µm)
space facility– 3.5 m Cassegrain telescope –the largest space telescope everlaunched– 3 science instruments (HIFI,PACS, SPIRE)– 3 years routine operational
lifetime

PACS

ADASS XIX 06/10/2009
Stephan Ott
VG # 3

http://herschel.esac.esa.int/

Current Spacecraft Status
•

••
•
•

Herschel was launched together
with Planck on an Ariane 5 ECA
14th of May 2009 on V188
Herschel reached its operational
orbit around L2Herschel’s Commissioning Phase
successfully completed mid July
Herschel is close to complete its
Performance Verification Phase
PV Phase is followed by a onemonth long Science Demonstration Phase with the Routine
Science Phase starting about 6months after launch

PACS

ADASS XIX 06/10/2009
Stephan Ott
VG # 4

http://herschel.esac.esa.int/

Scope and Methodology of Herschel
Data Processing Development
•••

•

•

Development of the Herschel Data Processing Systemstarted seven years ago to support the data analysis forInstrument Level Tests
The system is coded in Java/Jython to be license free andportable for different operating systems
The system combines for the first time data retrieval,pipeline execution and scientific analysis in one single
environment
All tools for data reduction and analysis, e.g. also the expert
applications for ‘Instrument Calibration’, ‘Trend Analysis’and ‘Quality Control’ systems are part of the Data
Processing System
Herschel Science Centre (ESA), the Instrument Control
Centres (HIFI, PACS and SPIRE) and NHSC jointly manage
and contribute to the Herschel Data Processing System
PACS

ADASS XIX 06/10/2009
Stephan Ott
VG # 5

http://herschel.esac.esa.int/

Key S/W Elements,
Status and further
Milestones of theHerschel Data
Processing System
PACS

ADASS XIX 06/10/2009
Stephan Ott
VG # 6

http://herschel.esac.esa.int/

Installer•

Installers are available for Windows XP, Vista, Linux, Mac
OS X and with or without the Java Runtime Environment for
both user and developer versions

PACS

ADASS XIX 06/10/2009
Stephan Ott
VG # 7

http://herschel.esac.esa.int/

Herschel Interactive Processing
Environment (HIPE)

PACS

ADASS XIX 06/10/2009
Stephan Ott
VG # 8

http://herschel.esac.esa.int/

User friendly tools
Tools to inspect images, data cubes and manipulate tables

PACS

ADASS XIX 06/10/2009
Stephan Ott
VG # 9http://herschel.esac.esa.int/

Documentation•

HIPE provides documentation for the framework and all
instruments, including search functionality

PACS

ADASS XIX 06/10/2009
Stephan Ott
VG # 10

http://herschel.esac.esa.int/

Herschel Pipeline Processing•

Pipeline is executed on the ESAC Grid using the Herschel
Science Archive to produce Herschel Products to different
reduction levels– Level 0– Level 1– Level 2

raw data
instrumental and satellite effects removed
scientific analysis can be performed

PACS

ADASS XIX 06/10/2009
Stephan Ott
VG # 11

http://herschel.esac.esa.int/

Herschel Quality Control••

Data quality control is a combination of automatic screening
and manual inspection
Quality control reports are electronically distributed toexperts

PACS

ADASS XIX 06/10/2009
Stephan Ott
VG # 12

http://herschel.esac.esa.int/

Data Access•

HIPE is connected to the Herschel Science Archive, anddata can be retrieved directly into an Interactive Analysis
SessionPACS

ADASS XIX 06/10/2009
Stephan Ott
VG # 13

http://herschel.esac.esa.int/

Current Status••••••

HIPE 1.1 (Performance Verification Phase version) madeavailable to Herschel Key Program teams
First bulk reprocessing exercise of Herschel data started
First Science Demonstration Phase observations conducted
and data provided to observers using an early version ofHIPE 1.2
HIPE 1.2 (Science Demonstration Phase version) undertesting
HIPE 2.0 (Science Routine Phase version) to be madeavailable to Herschel community early next year
Future HIPE versions will be released regularlyHIPE source code is freely available under the GNU
lesser general public licensePACSADASS XIX 06/10/2009
Stephan Ott
VG # 14

http://herschel.esac.esa.int/

What can HIPE dowith Herschel data?

PACS

ADASS XIX 06/10/2009
Stephan Ott
VG # 15

http://herschel.esac.esa.int/

PACS ‘sneak preview’ of M51 during OD 32

Interactive dataprocessing of the
70 µm channel 30
Threecolour
Elaboratedata
minutes
afterPipeline
processing
picturecombiningprocessingof thedata
receptionwithoperational(prethe160,100, and160µmchannel
70 µm
wavelengthlaunch)pipelineofcombining
severalbands
thescans
70 µm channel

PACS

ADASS XIX 06/10/2009
Stephan Ott
VG # 16

http://herschel.esac.esa.int/

HIFI ‘first light’ of DR21 star forming
region during OD 39
Pipeline
processing
withoperational(pre-launch)pipeline

Interactivedataprocessing

PACS

ADASS XIX 06/10/2009
Stephan Ott
VG # 17

http://herschel.esac.esa.int/

HIFI ‘first light’ of DR21 star forming
region during OD 39 – Final sky spectra
13CO (10-9)•1313
COCO(10-9)(10-9)

Interactive (left – one mapping element) and standard pipeline (right –average of 5 mapping elements) results for HIFI first light using the HRSspectrometer, used simultaneously with the WBS spectrometer.
H2O(1(11) – 0(00))

H2O(1(11)O(1(11) –– 0(00))0(00))H2PACS

ADASS XIX 06/10/2009
Stephan Ott
VG # 18

http://herschel.esac.esa.int/

SPIRE ‘first light’ of M66 during OD 42

Interactive data reduction

Current pipeline processing

PACS

ADASS XIX 06/10/2009
Stephan Ott
VG # 19

http://herschel.esac.esa.int/

SPIRE ‘first light’ of M74 during OD 42

250 µm500 µm350 µmInteractive data reduction

PACS

ADASS XIX 06/10/2009
Stephan Ott
VG # 20

http://herschel.esac.esa.int/

PACS/SPIRE parallel mode observations of
the Milky Way

PACS/SPIRE
5 colour image

PACS 2 colour

SPIRE 3 colour
PACS

ADASS XIX 06/10/2009
Stephan Ott
VG # 21http://herschel.esac.esa.int/

SummaryPACS

ADASS XIX 06/10/2009
Stephan Ott
VG # 22

http://herschel.esac.esa.int/

Summary•

Herschel Data Processing supported Herschel’s Commissioningand Performance Verification Phase•

Herschel Data Processing was able to support the early delivery ofscience data to the users community and is ready for Herschel’s
Science Demonstration and Routine Phase•

During the next months hands-on workshops for Herschel Key
Program Teams on Science Demonstration Phase data reduction
will follow•February 2010 the availability of HIPE 2.0 to the whole Herschelcommunity will be announced together with the opening of theHerschel Science Archive•

The Herschel Data Processing System is in a well advanced state,containing an impressive suite of functionality and documentation

You can follow the Herschel news at
http://herschel.esac.esa.int/Data_Processing.shtml
http://herschel.esac.esa.int/latest_news.shtml
PACSADASS XIX 06/10/2009
Stephan Ott
VG # 23

http://herschel.esac.esa.int/