# The Data Reduction Pipeline of the Hamburg Robotic Telescope

Authors: Marco Mittag; Alexander Hempelmann; José Nicolás González-Pérez; Jürgen H. M. M. Schmitt
Journal: Advances in Astronomy
 (2010)

---

## Abstract

The fully automatic reduction pipeline for the blue channel of the HEROS
spectrograph of the Hamburg Robotic Telescope (HRT) is presented. This pipeline is started automatically after finishing the night-time observations and calibrations. The pipeline includes all necessary procedures for a reliable and complete data reduction, that is, Bias, Dark, and Flat Field correction. Also the order definition, wavelength calibration, and data extraction are included. The final output is written in a fits-format and ready to use for the astronomer. The reduction pipeline is implemented in IDL and based on the IDL reduction package REDUCE written by Piskunov and Valenti  (2002).


---

## Body

## 1. Introduction

The HRT [1] was built   by Halfmann Teleskoptechnik GmbH (Germany) and installed at Hamburg Observatory in 2005. This Cassegrain-Nasmyth F/8 type telescope  has a 1.2 m aperture, an Alt/Az mounting, and final direct drives with high-precision absolute encoders. The only instrumentation of the HRT is the Heidelberg Extended Range Optical Spectrograph (HEROS). It is connected with the telescope via a polymicro FVP 50/70 μ fused silica fibre, equipped with microlenses on both sides of the fibre to adapt the HRT-HEROS F-ratios. At present, only the blue channel of HEROS (380–570 nm) is operating, while the red channel will start its operations in July 2009. The spectral resolution of the device is R=20000. The telescope is designated for long-time monitoring of active stars at its final location Guanajuato in Mexico.


## 2. Reduction Pipeline

The reduction pipeline is provided as fully automatic reduction pipeline including an automated wavelength calibration. It is started by the Central Control Software of the HRT system after the observations and calibrations have been finished. The reduced data are stored in an archive. This pipeline is implemented in IDL (Interactive Data Language) and uses the reduction package REDUCE, by Piskunov and Valenti [2]. REDUCE is a powerful package providing the required functionality for the HEROS pipeline.In the following the main reduction steps of the pipeline are described and these are represented in flow charts.


### 2.1. Preparation of the Reduction

Before the main reduction starts, it is necessary to prepare the reduction. The flow chart in Figure1 shows the main steps of this part.Figure 1
Flow chart with the main steps of the preparation of the reduction.The first step of the preparation is the definition of the directory holding the raw data. Also a temporary directory and the directory are created, where results are saved on disk.Thereafter the parameters and file names for reduction procedure are defined. All reduction parameters are supplied with the same values to all routines called by the pipeline at the different steps of the reduction. This is made for consistency in the reduction. The parameter values are read out from the parameter file for the blue channel.


### 2.2. Building the Master Calibration Images

The HRT system takes several calibration images: Bias, dark, and flat fields, before and after the observations. The respective images are averaged and then used as master calibration images. The main steps to build the master image are similar for bias, dark, and flat field. The flow chart in Figure2 shows the main steps in this part.Figure 2
Flow chart with the main steps of building the master calibration images.The first step is a check of the variation in the calibration images. The percentage of relative variationrvi between the arithmetic mean of the single images 〈imi〉 and the median of this values median(〈im1〉⋯〈imn〉) is calculated as(1)rvi=(〈imi〉-median(〈im1〉⋯〈imn〉)median(〈im1〉⋯〈imn)〉)·100.
For the calculation of rvi for the darks the single images are corrected by bias and for the rvi for the flats by bias and dark. The percentage of relative variation is plotted in a log file.The calibration images are split in two lists. If the same number of images is taken before and after the observation, then in the first list the images taken at the start of the observation are collected and in the second list those taken after the observation. Then, the percentage of relative variations of the images are checked for both lists. If the percentage of relative variation is higher than a threshold, the corresponding image is not used for building the master calibration image. Normally the total number of images in both lists after the check ofrvi is greater or equals 3. The standard deviations of the percentage of relative variation of images, Equation (1), are calculated for both lists and used as the thresholds to build the master calibration image. If the standard deviation is less than 3.5, then it is reset to the default value 3.5. This is the minimum threshold to find outliers in the images.If the total number of images in one list is less than 3, an error message is obtained and the images are collected in a new list. The content of this new list is checked. Now there are 3 possibilities.(1)
The total number of images is<3. The reduction ends, because one needs at least 3 calibration images to build a master calibration image.(2)
The total number of images is<6. The new list is not split, because in the list must be at least 3 images.(3)
The total number of images is≥6. The new list is split in two lists.Thereafter the single calibration images are averaged. As next step it is checked if an error arose during the combination of the single calibration images. In this case the pipeline stops and an error message is written to a file.The master bias is subtracted from the average dark and flat field and additionally the flat field is corrected from the dark contribution. The dark correction is not performed, if the arithmetic mean of master dark is smaller than a predefined threshold for the dark correction, because in this case a dark contribution is negligible. After the subtraction of the master bias, the dark is time normalised.The results are saved as master calibration images (hereafter, bias, dark, and flat field). To monitor long-term changes, the arithmetic mean of the single images and the corresponding standard deviation are saved in a log file. Also the standard deviation of   both lists and the arithmetic mean, the median, and the standard deviation of the master calibration images are saved.


### 2.3. Order Definition

The flat field is used for the order definition. The central positions of the individual spectral orders are located and defined in this step of the reduction pipeline. If a single order is not found, then the pipeline stops and an error message is written to a file. The results of the order definition are saved in the order definition file. The flow chart in Figure3 shows the main steps in this reduction part.Figure 3
Flow chart with the main steps of the order definition.


### 2.4. Blaze Extraction

During the next step in the reduction pipeline the spectrum of the flat field lamp is extracted.  The flow chart in Figure4 shows the main steps in this part of the pipeline. This spectrum (hereafter, blaze) can be used as blaze function. The blaze is used to eliminate the blaze function  in the science spectrum and to correct for the difference in quantum efficiency of the pixels (Section 2.6). After the background correction the blaze is extracted like a science spectrum. The counts in the blaze are converted in electrons and the blaze is normalised by the exposure time. Finally the blaze is saved in the order definition file.Figure 4
Flow chart with the main steps of the blaze extraction.


### 2.5. Wavelength Calibration

The spectrum of the ThAr lamp is used for the wavelength calibration. One ThAr image will be taken before and one after the observations. The flow chart in Figure5 shows the main steps in this part. The flow of the wavelength calibration can be split in two parts: the spectrum extraction and the new wavelength solution.Figure 5
Flow chart with the main steps of the wavelength calibration.ThAr Spectrum Extraction
The two ThAr spectra are reduced consecutively. The bias is subtracted from the ThAr image. If the arithmetic mean of dark is above the threshold for dark subtraction, then the dark is also subtracted from the ThAr image. Thereafter the ThAr spectrum is extracted and saved on disk.Wavelength Solution
For the automatic wavelength calibration a reference ThAr spectrum with 1D wavelength solutions of the several orders is used. The extracted spectra are compared with the reference spectrum. The shifts are calculated for each order via a cross correlation. After that the order shifts of the two ThAr spectra are averaged for each order.
The new 1D wavelength solutions of all orders are determined with the shifts and the 1D wavelength solution of the reference spectrum. Finally, the results, the shifts, and the spectral resolution of the reference ThAr spectrum are saved in a wavelength file.
To check the results, the pipeline creates plots with the residuals of the 1D wavelength solutions fits and a file containing the arithmetic mean and standard deviation of the shifts and the residuals of the 1D wavelength solution.


### 2.6. Spectrum Extraction

The final part of the pipeline is the extraction of the actual spectra. First, the spectra of all images are extracted. The images of the same object are coadded and then a summed spectrum is created.Spectrum from the Single Science Image
The flow chart in Figure6 shows the main steps in this part of the pipeline. The science images were collected in a list. The bias is subtracted from the science image, and  if the arithmetic mean of dark is greater than a threshold for dark subtraction, the dark is also subtracted from the science image. If sky images from this image (object) exist, then the average of two sky images is created and subtracted from the science image. Next, a background correction is performed. After the background correction the pipeline identifies outliers in the science image and these are flagged. Thereafter, the spectrum is extracted, the counts in the spectrum are converted to electrons, and the spectrum is time normalised. In the following, the spectrum is divided by the blaze [3]. This step eliminates the blaze function. Simultaneously the correction for the quantum efficiency is achieved. If the keyword noautocal is not set, then the wavelengths for the individual objects are corrected by the corresponding barycentric velocity shifts. Finally the spectra are saved in fits file.Figure 6
Flow chart with the main steps of the spectrum extraction form the single science image.Spectrum from the Coadded Science Images
This is the last part of the reduction pipeline. Here the spectrum from the summed science images of the same object is extracted. This reduction procedure is similar to the procedure of extraction the spectrum from the single science image, but additionally at first the images are coadded. The flow chart in Figure7 shows the first steps in this reduction part. The rest is similar to the spectrum extraction a single science image.
The first difference between the both reduction steps is a check which objects have more than one exposure. These objects are collected in a list.
Another difference is the coaddition of the single images for the same object, after the bias and, if necessary, a dark and sky correction. The mean Julian date for the summed image and the barycentric velocity shift are computed.Figure 7
Flow chart of the first steps of the spectrum extraction of the coadd science images.


## 2.1. Preparation of the Reduction

Before the main reduction starts, it is necessary to prepare the reduction. The flow chart in Figure1 shows the main steps of this part.Figure 1
Flow chart with the main steps of the preparation of the reduction.The first step of the preparation is the definition of the directory holding the raw data. Also a temporary directory and the directory are created, where results are saved on disk.Thereafter the parameters and file names for reduction procedure are defined. All reduction parameters are supplied with the same values to all routines called by the pipeline at the different steps of the reduction. This is made for consistency in the reduction. The parameter values are read out from the parameter file for the blue channel.


## 2.2. Building the Master Calibration Images

The HRT system takes several calibration images: Bias, dark, and flat fields, before and after the observations. The respective images are averaged and then used as master calibration images. The main steps to build the master image are similar for bias, dark, and flat field. The flow chart in Figure2 shows the main steps in this part.Figure 2
Flow chart with the main steps of building the master calibration images.The first step is a check of the variation in the calibration images. The percentage of relative variationrvi between the arithmetic mean of the single images 〈imi〉 and the median of this values median(〈im1〉⋯〈imn〉) is calculated as(1)rvi=(〈imi〉-median(〈im1〉⋯〈imn〉)median(〈im1〉⋯〈imn)〉)·100.
For the calculation of rvi for the darks the single images are corrected by bias and for the rvi for the flats by bias and dark. The percentage of relative variation is plotted in a log file.The calibration images are split in two lists. If the same number of images is taken before and after the observation, then in the first list the images taken at the start of the observation are collected and in the second list those taken after the observation. Then, the percentage of relative variations of the images are checked for both lists. If the percentage of relative variation is higher than a threshold, the corresponding image is not used for building the master calibration image. Normally the total number of images in both lists after the check ofrvi is greater or equals 3. The standard deviations of the percentage of relative variation of images, Equation (1), are calculated for both lists and used as the thresholds to build the master calibration image. If the standard deviation is less than 3.5, then it is reset to the default value 3.5. This is the minimum threshold to find outliers in the images.If the total number of images in one list is less than 3, an error message is obtained and the images are collected in a new list. The content of this new list is checked. Now there are 3 possibilities.(1)
The total number of images is<3. The reduction ends, because one needs at least 3 calibration images to build a master calibration image.(2)
The total number of images is<6. The new list is not split, because in the list must be at least 3 images.(3)
The total number of images is≥6. The new list is split in two lists.Thereafter the single calibration images are averaged. As next step it is checked if an error arose during the combination of the single calibration images. In this case the pipeline stops and an error message is written to a file.The master bias is subtracted from the average dark and flat field and additionally the flat field is corrected from the dark contribution. The dark correction is not performed, if the arithmetic mean of master dark is smaller than a predefined threshold for the dark correction, because in this case a dark contribution is negligible. After the subtraction of the master bias, the dark is time normalised.The results are saved as master calibration images (hereafter, bias, dark, and flat field). To monitor long-term changes, the arithmetic mean of the single images and the corresponding standard deviation are saved in a log file. Also the standard deviation of   both lists and the arithmetic mean, the median, and the standard deviation of the master calibration images are saved.


## 2.3. Order Definition

The flat field is used for the order definition. The central positions of the individual spectral orders are located and defined in this step of the reduction pipeline. If a single order is not found, then the pipeline stops and an error message is written to a file. The results of the order definition are saved in the order definition file. The flow chart in Figure3 shows the main steps in this reduction part.Figure 3
Flow chart with the main steps of the order definition.


## 2.4. Blaze Extraction

During the next step in the reduction pipeline the spectrum of the flat field lamp is extracted.  The flow chart in Figure4 shows the main steps in this part of the pipeline. This spectrum (hereafter, blaze) can be used as blaze function. The blaze is used to eliminate the blaze function  in the science spectrum and to correct for the difference in quantum efficiency of the pixels (Section 2.6). After the background correction the blaze is extracted like a science spectrum. The counts in the blaze are converted in electrons and the blaze is normalised by the exposure time. Finally the blaze is saved in the order definition file.Figure 4
Flow chart with the main steps of the blaze extraction.


## 2.5. Wavelength Calibration

The spectrum of the ThAr lamp is used for the wavelength calibration. One ThAr image will be taken before and one after the observations. The flow chart in Figure5 shows the main steps in this part. The flow of the wavelength calibration can be split in two parts: the spectrum extraction and the new wavelength solution.Figure 5
Flow chart with the main steps of the wavelength calibration.ThAr Spectrum Extraction
The two ThAr spectra are reduced consecutively. The bias is subtracted from the ThAr image. If the arithmetic mean of dark is above the threshold for dark subtraction, then the dark is also subtracted from the ThAr image. Thereafter the ThAr spectrum is extracted and saved on disk.Wavelength Solution
For the automatic wavelength calibration a reference ThAr spectrum with 1D wavelength solutions of the several orders is used. The extracted spectra are compared with the reference spectrum. The shifts are calculated for each order via a cross correlation. After that the order shifts of the two ThAr spectra are averaged for each order.
The new 1D wavelength solutions of all orders are determined with the shifts and the 1D wavelength solution of the reference spectrum. Finally, the results, the shifts, and the spectral resolution of the reference ThAr spectrum are saved in a wavelength file.
To check the results, the pipeline creates plots with the residuals of the 1D wavelength solutions fits and a file containing the arithmetic mean and standard deviation of the shifts and the residuals of the 1D wavelength solution.


## 2.6. Spectrum Extraction

The final part of the pipeline is the extraction of the actual spectra. First, the spectra of all images are extracted. The images of the same object are coadded and then a summed spectrum is created.Spectrum from the Single Science Image
The flow chart in Figure6 shows the main steps in this part of the pipeline. The science images were collected in a list. The bias is subtracted from the science image, and  if the arithmetic mean of dark is greater than a threshold for dark subtraction, the dark is also subtracted from the science image. If sky images from this image (object) exist, then the average of two sky images is created and subtracted from the science image. Next, a background correction is performed. After the background correction the pipeline identifies outliers in the science image and these are flagged. Thereafter, the spectrum is extracted, the counts in the spectrum are converted to electrons, and the spectrum is time normalised. In the following, the spectrum is divided by the blaze [3]. This step eliminates the blaze function. Simultaneously the correction for the quantum efficiency is achieved. If the keyword noautocal is not set, then the wavelengths for the individual objects are corrected by the corresponding barycentric velocity shifts. Finally the spectra are saved in fits file.Figure 6
Flow chart with the main steps of the spectrum extraction form the single science image.Spectrum from the Coadded Science Images
This is the last part of the reduction pipeline. Here the spectrum from the summed science images of the same object is extracted. This reduction procedure is similar to the procedure of extraction the spectrum from the single science image, but additionally at first the images are coadded. The flow chart in Figure7 shows the first steps in this reduction part. The rest is similar to the spectrum extraction a single science image.
The first difference between the both reduction steps is a check which objects have more than one exposure. These objects are collected in a list.
Another difference is the coaddition of the single images for the same object, after the bias and, if necessary, a dark and sky correction. The mean Julian date for the summed image and the barycentric velocity shift are computed.Figure 7
Flow chart of the first steps of the spectrum extraction of the coadd science images.


## 3. Conclusions

The data reduction pipeline for the blue channel works fully automatically and stably. In case of an error in some positions in the reduction flow, a message is written in a file. Also log files are written during the reduction. With these error and log files the astronomer can check the reduction flow. In Figure8 an example of the final output of the pipeline is shown. A spectrum of Alcaid taken with HEROS and extracted with MIDAS [3] is shown in Figure 9 for comparison only.Figure 8
A sample result: Vega spectrum relative to the flat field spectrum and normalised to Figure9.Figure 9
A comparative spectrum: Alcaid spectrum [3].One problem in the reduction pipeline is the identification of faint outliers in the science image. Also the continuum normalisation process of late types stars may have problems in order to  find the quasicontinuum segments  of the spectrum.A future task will be the creation of the data reduction pipeline for the red channel. In general this pipeline will have the same structure as the pipeline for the blue channel has.Finally, the optimisation and regular support of   both pipelines are important to obtain the best possible outputs.



---

*Source: 101502-2009-12-02.xml*