IRC data &imaging toolkit
Yoshifusa Ita& IRC team

2007/Sep/18

AKARI / IRC DR Workshop @ ESAC

1

Outlinez Part 1

IRC data structurez Characteristics and known problems
z Imaging/Spectroscopic toolkit
z This part is common to both imaging andspectroscopic datazz Part 2

IRC imaging data reduction toolkit
z Things
g that are not included yyetz

2007/Sep/18

AKARI / IRC DR Workshop @ ESAC

2

Part 1:
IRC data
zFocal plane layout
zObservational sequencezStructure of IRC fits files
zCurrent status andcharacteristics/problems
2007/Sep/18

AKARI / IRC DR Workshop @ ESAC

3

Focal-plane layout
The IRC consists of 3 detectors.
z NIR (InSb, 512x412)
z 2
2-5
5 um
z N2,N3,N4 & NP,NGz MIR-S (Si:As, 256x256)
z 5-13
5 13 umz S7,S9W,S11 & SG1,SG2z MIR-L (Si:As, 256x256)
z 12-26 um
z L15,L18W,L24 & LG2z Each camera is equipped withthree filters and two dispersion
elements.
z NIR and MIR
MIR-S
S observe the samesky but MIR-L and NIR&MIR-SAKARI / IRC DR Workshop @ ESAC
4 do
not observe the same sky.
z+Y

2007/Sep/18+X

Observation sequence in one
Exposure Cycle
pointing
Dithering
Filter Change

AOT
00
F1

F1

F1

F1

F1

F1

F1

F1

F1

F1

F1

spectroscopy

S1

03

F1

F2
F2

F2
F3

F1

F1

F2

F3

F2

F1

F1

S2

S2

AFT
TER-DARRKPR
RE-DARK

02
F2
F1

F2

04
S1

S1

S1

S2

S2

S2

05
F1
2007/Sep/18

F1

F1
Time

AKARI / IRC DR Workshop @ ESAC

F1

F1
5

operation of one exposure
1 NIR fits filecycle
image1

1 MIR fits fileimage1

image2

image2

image3

image4

A pair of short and long exposures are carried out for the NIR camera while
one short and three long exposures
are carried out for the MIR detectors. 6
2007/Sep/18
AKARI / IRC DR Workshop @ ESAC
RST indicates a reset of the detectors.
zStructure of IRC fits files
file

z

z
z

file

The IRC FITS data is not a usual 2D one. A raw data FITS file is created for each
Exposure Cycle during a pointing for the NIR and combined MIR-S/MIR-L channels.
(i.e. 1 NIR FITS file and 1 MIR FIT file = 2 FITS files per Exposure Cycle). The
filename format is given as F***** N.fits or F***** M.fits, where ***** is a distinct
incremental reference number.
A NIR raw fits data file is a data cube containing 2 images within it, corresponding to
one short and one long exposure.
Each MIR raw FITS file contains 4 images within it (one short and three longexposures) for both the MIR-S and MIR-L channels respectively making a total of 8frames per MIR FITS file per Exposure Cycle.
Cycle

2007/Sep/18

AKARI / IRC DR Workshop @ ESAC

7

What you will get = rawdata
slit area
MIRL
256
pix

NIR
512
pix
MIRS
observe the same sky

2007/Sep/18

F??????_N.fits(412,1)boresight
AKARI / IRC DR Workshop @ ESAC
412 pix
F??????_M.fits(MIRS:
256 256)
256,
boresight
256pix

256 pix8Difference from the familiar fits
files (summary)z 3-dimensionalz

1 NIR fits file contains:z 1 short exposure imagez 1 long exposure imagez

1 MIRS fits file contains:z 1 short exposure image for MIRS and MIRLz 3 long exposure image for MIRS and MIRLz MIRS and MIRL do not observe the same sky
sky,

but they are stored in one fits file.
2007/Sep/18

AKARI / IRC DR Workshop @ ESAC

9

What you will get = rawdataz Your data can be obtained from:z

http://darts isas jaxa jp/astro/akari/getData html
http://darts.isas.jaxa.jp/astro/akari/getData.html

z The data are encrypted with GPG
ze.g.: AKARI_IRC_2200007_001.tar.gpg
AKARI IRC 2200007 001 tz Decrypted data are stored in a tar ball.
ze.g.: AKARI_IRC_2200007_001.tar

z Unpack the tarballz-> next slide

2007/Sep/18

AKARI / IRC DR Workshop @ ESAC

10

inside the tar ballz README and a directory(TargetID.subID)
z READMEzzzobservation summary, etc.
Please read through this file.

TargetID subID/
TargetID.subID/zzz
2007/Sep/18

e.g.:
e g : 3080003.1
3080003 1

rawdata contains raw data
 F?????????_[N|M].fits:
F????????? [N|M] fits: raw data
 *.lst: list of data files observed with each filter/grism World Coordinate System Parameters are given in the fits header
irc qlirc_ql contains processed data with standard toolkits with standard options.
 only for quick look purpose.

see README ffor ddetails
il off theh text fil
files iinside
id theh two didirectories.
i
AKARI / IRC DR Workshop @ ESAC

11

Very important Fits Header
Keyword = WCSROOTz WCSROOTz

AOCS (Default)
z on the basis of Attitude and Orbit Control Systemz Accuracy is better than 30 arcsec
z

Depends on ecliptic latitude of target because of aberration

2MASS (After using toolkits)
z on the basis of 2MASS coordinatesz matching 2MASS sources with point sources detectedz Accuracy is better than a few arcsec, and in mostcases, better than 1 arcsec
2007/Sep/18



Depends on how
match stars were matched.
AKARI / IRC DR Workshop @ ESAC

12

Current status andcharacteristics/problems (1)z

The number of hot pixels are increasing

PV phase(A il 2006)(April,dark image

4 months Later(August, 2006)
dark image
2007/Sep/18

AKARI / IRC DR Workshop @ ESAC

13

Current status andcharacteristics/problems (2)zDetector linearityzzzmeasured with IRC installed in
the laboratoryy and in flightg likeconfiguration
measurements were madewith calibration lamp that
ill i t dilluminates
detectort t andd anincreasing range of integration
times
after the correction,correction thedeviation from the ideal linear
curve is better than 5% at the
DN of 12000 and 30000 in NIR
and MIR, respectively

2007/Sep/18

AKARI / IRC DR Workshop @ ESAC

14

Current status andcharacteristics/problems (3)Dark and/or Bias stability
z
z

z
z
z

dark level is NOT stable
during 1 pointing.
super-darkd k iimages weremade with pre-dark datataken in LMC survey. They
are “shifted”, i.e.,
add/subtract a certain
constant, by checking slit
area in each image, andthen subtracted.
NIR: no apparent correlation
with detector temperature
MIR: weak correlation with
detector temperaturedark level is very high after
passing SAA

MIRS

temperature

img. dark

img. darkslit. darkttemperaturetslit. dark

dark levelchangeswithin one
pointing

darrk level

z

NIR

time
2007/Sep/18

AKARI / IRC DR Workshop @ ESAC

data from Dr. Wada

15

Current status andcharacteristics/problems (4)
z

z

NIR
z
z

z

z

NEP survey data
S/N : about 10
NEP survey data
S/N : more than 100

Unidentified object
Disappeared after 8th January 2007

Stray light
z
z

z
z

S11: +- 10% L24: +- 10%

MIR
z

z

N4 : +- 5%

Flat fieldpresent in all detectors
unexpected high backgroundprobably due to external(Earth) light, which is not
uniform over the FOV (MIR-L)
flat-field are affected
observation of diffuse
background will suffer

2007/Sep/18

AKARI / IRC DR Workshop @ ESAC

backgroundlevel
changeswithin one
pointing
time
data from Dr. Wada

16

Current status andcharacteristics/problems (5)Encircled EnergyzAperture Correction
z

z

z
z

z

z

IRC flux calibration were made byobserving standard stars (point
source)we assume that all flux are comeinto certain radii (10 and 7.5 pixels
for NIR and MIR).
these radii are large compared to
S/N optimal apertures
the amount of flux enclosed inthese radii is still UNKNOWN (i.e.,flux calibration for diffuse source isstill unknown)
aperture corrections must be
applied when using different
aperturesp
we will provide aperture
corrections, but observers areencouraged to see their own
curves-of-growth because the PSF
diff
differs
fromfpointing
i ti tot pointing
i ti

2007/Sep/18

AKARI / IRC DR Workshop @ ESAC

Flux

S/N
aperture radius
Signal = Const * EE (raperture )Noise =

N pixel∑ noise = N
2

i

pixel

* noise pixeli2
N pixel = π * raperture

EE (raperture )Signal
Const
=
*
Noise
raperture
π noise pixeli.e., S/N ∝ EE(r)/r17

Current status andcharacteristics/problems (6)zGhost in slit-areazz

Ghost of B

NG slit-spectroscopic
data would be damaged
if bright stars were
present in the imaging
area.
Details → Ohyama-san’sypresentation.

Ghost of AA
B

2007/Sep/18

AKARI / IRC DR Workshop @ ESAC

18

Current status andcharacteristics/problems (7)
N3

z

S7

L24

Aspect ratio and Distortion term

The deviation from an ideal lattice is up to 2, 0.6, and 16 pixels at the edge for NIR,MIRS and MIRL,MIRL respectively.
respectively There is little difference with filters among a detector
detector.
z
Main contribution of the deviation comes from aspect ratio disorder.
z
Distortion term (non-linear term) is very small, but present in MIR-L detector.
z
Toolkit corrects aspect ratios, but not distortion.
z
the resultant pixel scales of NIR,MIRS, and MIRL become 1.446, 2.340, and 2.384
arcsec/pixel, respectively. All detectors have wide field of views of 10‘x10.
2006/Oct/11z19

Current status andcharacteristics/problems (8)Latent in MIR-S&L

z
z

z
z

2007/Sep/18

Flat changes after
observing very brightstars.
A few % decrease
Last for several (up to 8)hours

1. observing
bright source

2. latent appears
next pointing

3. latent still
appears next
next pointing

4. latent still
appears ne
nextt
next next pointing

AKARI / IRC DR Workshop @ ESAC

20

Imaging/Spectroscopy toolkits
Common Principles
z Please use our toolkits, not yoursz

our toolkits includes calibration dataz Imaginglinearity,y, flat,, etc.

z Spectroscopyzflux calibration, flat, etc.

calibration data may be updated in the futurez Please send your new tool to helpdesk
helpdesk.
z

we will test it, and maybe ask you to allow us to
distribute.
distribute2007/Sep/18

AKARI / IRC DR Workshop @ ESAC

21

Toolkits: Where to get?
z Imaging/Spectoscopic data reduction toolkits

can be downloaded from:zz

http://www.ir.isas.jaxa.jp/ASTRO-F/Observation/DataReduction/IRC/
Manuals, Tips, and FAQs are also there.

z For Helpzzzsend emails to helpdesk: iris_help@ir.isas.jaxa.jp
Do not forget to tell us your TargetID.
TargetID
Please read the manual and FAQs again before sending emails.

2007/Sep/18

AKARI / IRC DR Workshop @ ESAC

22

Part 2:
IRC imaging toolkit
zRequirements
zOutline
zWhat you will get
zThings not included yet

2007/Sep/18

AKARI / IRC DR Workshop @ ESAC

23

Requirementsz Unix (Linux, BSD, OSX, Solaris)z Perlz GCC3.0 or laterz IRAF version 2.12.2 or laterz curl (for WCS calculation on Solaris)

IImaging
i pipelinei li may work
k on IIntel-Mac
t l M PCz Does not work on Windows PCz

2007/Sep/18

AKARI / IRC DR Workshop @ ESAC

24

Outline of pipelinez

Red Box (pre-pipeline processor)
z
z

z

}

prepipeline

Green Box (pipeline processor)z

instrumental signaturegcorrection
z wraparound, fowler/bit-shift renormalizationz dark subtraction, linearization, flat fielding, distortion
correction etc
etc.
Make Basic Data

Blue Box
z
z

pipe
eline

z

z

header formatting
Make observing log

co-add dithered images (multi-pointing compliant)
Make Post Basic Data

2007/Sep/18

AKARI / IRC DR Workshop @ ESAC

25

What you get from the pipelinez

Basic Datazsingle image
z The unit of resultant image is in [DN]absolute calibration is not applied[DN] depends only on exposure time
time, ii.e.,e you can compare data
among any AOTs and short/long exposure images by only
considering the difference of exposure times.
a table to convert from [DN/sec] to [Jy] for each filter is supplied by
Dr. Tanabe.

z Calibrated:
2007/Sep/18

linearization applied
dark subtracted (using super-dark or self-dark)
flat-fielding
g appliedpp((usingg super-flat)p)distortion corrected
AKARI / IRC DR Workshop @ ESAC

26

What you get from the pipeline
Post Basic Dataz

coadded image for each filterzSometimes pipeline fails to coadd MIRL
(L15 L18W(L15,
L18W, L24) iimages.

zBasic WCS is supplied in the header by
defaultz

2007/Sep/18

However, there is a help tool that uses MIRS shift/rotation to coadd MIRL images.

PipelinePili can automaticallytti ll matcht h2
2mass sources tto iimproveaccuracy by using a tool (not applied by default. this is an option).
sometimes the tool fails to match 2mass sources(L15, L18W, L24)
Accuracy is Better than 30 arcsec after pipeline (WCSROOT=AOCS)Depends on β because of aberration
 Better than a few arcsec after matching 2mass sources(WCSROOT=2MASS)AKARI / IRC DR Workshop @ ESAC
27

Depentds on
how many stars were matchedThings not included yet (1)zMuxbleedzzpresent in NIR
trail of bright pixels inread direction every 4thcolumn2007/Sep/18

AKARI / IRC DR Workshop @ ESAC

28

Things not included yet (2)zColumn Pulldownzzpresent in NIR
a bright source results ina decrease of 20 – 60
DN in the entire
column(s)2007/Sep/18

AKARI / IRC DR Workshop @ ESAC

29

Things not included yet (3)zGhost in imaging-area
z
z

z

z

present in all detectors
position and strength
differ from detector todetector and also fromfilter to filter.
strongest (~( 4% of parent
source) in S11
caused by internalreflections in beam
splitter and also lenses.

2007/Sep/18

Dark increase due to the IRC
all-skyallsky survey

AKARI / IRC DR Workshop @ ESAC

Ghost

30

End
Enjoy IRC data !!

2007/Sep/18

AKARI / IRC DR Workshop @ ESAC

31

Appendix
what you will get from the pipeline&
structure off pipeline

2007/Sep/18

AKARI / IRC DR Workshop @ ESAC

32

What you get from the pipelinez

Other files (0)irclogzzz

created by prepipeline
observing log

darklist before
darklist.before

z

z
z
z

2007/Sep/18

created by prepipeline
a list that contains the name of dark imagesg
taken
before observation.
required if you want to use selfdark image instead ofsuper darksuper-dark

AKARI / IRC DR Workshop @ ESAC

33

What you get from the pipelinez

Other files (1)F??????????_[NSL]00[1-4].fitszcreated by prepipeline
split 3D raw fits files into usual 2D ones.
headers are overwritten

z
z
z



z

2007/Sep/18

CPIX1&2
FILTER

almostl
t equals
l tto raw datad t exceptt forf dimension
di
i and
d
headers.

AKARI / IRC DR Workshop @ ESAC

34

What you get from the pipelinez

Other files (2)ef*F??????????_[NSL]00[1-4].fitszcreated by pipeline just before coadd process.
all things are calibrated

z
z


z

2007/Sep/18

dark flat,
dark,
flat linearity,
linearity distortion,
distortion etc…
etc

Basic Data

AKARI / IRC DR Workshop @ ESAC

35

What you get from the pipelinez

Other files (3)Sef*F??????????_[NSL]00[1-4].fitszf*F?????????_[NSL]00[1-4].fitsz

412x512 (256x256) : NIR (MIRS&L)

copy it to big framez

1024x1024 (512x512) : NIR (MIRS&L)

Sef*F??????????_[NSL]00[1-4].fits.coo.1zzz

result files of source detection
contains xy coordinate and arbitral brightness

Sef*F??????????
Sef
F??????????_[NSL]00[1-4].fits.shift[NSL]00[1 4] fits shiftzzz
2007/Sep/18

result files of calcshift.cl
database file to correct shift and rotation
AKARI / IRC DR Workshop @ ESAC

36

What you get from the pipelinez

Other files (4)RSef*F??????????_[NSL]00[1-4].fitszzrotation and shift corrected Sf**.fits images

ASef*F??????????_[NSL]00[1-4].fitszzz

2007/Sep/18

Sky adjusted Rsef*.fits
Rsef* fits images
These files are actually combined to make the final
coadded images.

AKARI / IRC DR Workshop @ ESAC

37

What you get from the pipeline
Other files (5)zlong(short)_exp.input
z list file containing the name of pair files for long/short
exposure.
exposurez pair000?_[filtername].list
z list file containing the name of files taken by filtername
filter.
filt
z pair000?_[filtername].list_long(short)z split pair000?_[filtername].list according to exposure time(long/short).
z pair000?_[filtername].list_long(short).shift & shift.0
z contains dx[pix],dy[pix],dr[deg][p ], y[p ], [ g] (relative(
to the “first” image),g ),and # of stars matched to calculate dx,dy,dr.
z pair000?_[filtername].list_long(short).combinez contains filename that were combined to create coadded
images. Note thatimages with outlier sky levels were2007/Sep/18
AKARI / IRC DR Workshop @ ESAC
38
excluded from the coaddition.
zWhat you get from the pipelinez

Other files (6)files in logs

zcalcshift.logz

skypair000?_[filtername].list_(long/short)z

2007/Sep/18

log of calcshift.clsee foro tthee rms
soof x aandd y linear
ea fitt results.
esu tslog file of adjust_sky.cl that shows mean,median,mode of
the sky level
level, and sigma
sigma.
any files with outlier sky level will be excluded from the
combine process.

AKARI / IRC DR Workshop @ ESAC

39

What you get from the pipelinez

Other files (7)files in stacked_IMz

Objectname_[filtername]_(long/short).fitsz

plObjectname_[filtername]_(long/short).fitsz

NCOMBINE – values in each pixel = exposure map

sigmaObjectname [filtername] (long/short) fits
sigmaObjectname_[filtername]_(long/short).fitsz

2007/Sep/18

resultpost basbasic
c datastandard deviation of each pixel

AKARI / IRC DR Workshop @ ESAC

40

Structure ofpipeline(1)
z

irc
z

irc.clzz

load.clzz

prepipeline

unlearn_all.clzz

pipeline

prepipeline.clzz

iraf packagespg
preloaded in irc.cl

pipeline.clzz

package root

unlearn iraf
parameterspwelcome.txt

2007/Sep/18

z

AKARI / IRC DR Workshop @ ESAC

welcome message

41

Structure ofpipeline(2)zbinaryzbinary files
z convert2masszconvert2mass all
convert2mass_allzused in flat

mkpair

2007/Sep/18

used in calcshiftmkflatpairzused in putwcs

lonesomezused in putwcs

used in coadd
AKARI / IRC DR Workshop @ ESAC

42

Structure ofpipeline(3)
z

lib
z

library files
z constants.database

all irc constants arein this file.
file
you can edit it atyour own risk

setenv.dat
z setpath.dat
z setpath.pl
setpath plz welcome.org
z2007/Sep/18

AKARI / IRC DR Workshop @ ESAC

43

Structure ofpipeline(4)zlib - subdirectories
z anomalous_pix
z

z

dark
z

z

distortion database

flat
z

z

super-dark images

distortionzz

kno n hot/bad pi
el
known
pixel
map images

super-flat images

slit_maskz

2007/Sep/18

slit-area maskgimagesAKARI / IRC DR Workshop @ ESAC

44

Structure ofpipeline(5)
z

perl
z

perl scripts used inpipeline

2007/Sep/18

AKARI / IRC DR Workshop @ ESAC

45

Structure ofpipeline(6)
z

pkg
z

z

z

z

redboxz prepipeline
greenbox
z pipeline
blueboxz pipelinei li ((coadd)
dd)
z putwcs.cl
irc
tool
irc_toolz coaddLusingS.cl

2007/Sep/18

AKARI / IRC DR Workshop @ ESAC

46

Structure ofpipeline(7)
z

src
z

z

source files to makebinary files in irc/bin.
written in C.

2007/Sep/18

AKARI / IRC DR Workshop @ ESAC

47