Appendix B

Data Processing Plans

Our proposal for the SN survey is to carry out repeated 2.5m imaging at
normal scan rate as often as possible throughout a given 3-month
season.. Unlike the current SDSS survey, we will include imaging during
non-photometric, relatively poor seeing, and bright moon conditions. Our
default plan is to scan for supernovae within the SDSS stripe 82, which
has been scanned several times in the past. Stripe 82 covers 700 fields
for each of 6 camera columns, for RA from 20h to 4h, and Dec from --1.2°
to 1.2°. One possibility is to alternate on successive observing nights
between the two strips of a stripe

We will need to have at APO sufficient computing power and data storage
to run the SDSS Photo pipeline and a frame subtraction pipeline fast
enough to keep up with the data stream and to have a short turn-around
time for spectroscopic follow-up. The plan will be to automate the
pipelines so that the observers can start the processing system at the
end of a night and then go to bed. Members of the SN team should be able
to remotely monitor the processing while it is on-going: checking status
messages and QA outputs as they are generated.

We have obtained and run a frame subtraction pipeline package from the
University of Washington group, with the help of Gajus Miknaitis. The
code consists of Perl scripts wrapped around C routines. This package
has evolved to be used in several variability surveys, including
SuperMacho, Essence, and a previous effort that identified several
supernovae in SDSS data. It is thus well tested, robust, and flexible.
Each stage in the processing has been designed to be modular, and can be
easily replaced, turned-off, or re-run.

Our template frames will consist of the co-added data from these former
scans. Data for the supernova search will be processed by a subset of
the SDSS Photo pipeline stages, in order to produce bias-subtracted,
flat-fielded frames with the astrometric calibration required by the
pipeline. Since the search data will not be confined to photometric and
dark-time conditions, we will do photometry relative to the template
data. The stages in the frame-subtraction processing include:

1)  Match frames from the search data and template data.

2)  Convolve the data to match PSFs in search and template frames.

3)  Subtract the images.

4)  Catalog of candidate transient objects.

5)  Analysis of candidates and flagging for visual inspection.

As the frame subtraction code finishes on a given set of frames and
identifies SN candidates, the remote SN team members will need to be
able to pull over images of template, data, and difference for visual
inspection. On a given night, this might involve inspection of \~20
objects, depending on how we set the thresholds, affecting efficiency
and the rate of false positives.

We have tested the pipeline on SDSS stripe 82 data, and Figure 1 shows
an example for a particularly nearby type Ia supernova discovered by
LOTOSS and also appearing in SDSS data. This image also illustrates some
of the issues of rejecting false positives. These issues include:

1)  Removal of residuals and diffraction spikes from bright starts and
    galaxies. The SDSS catalog can be used to veto candidates near known
    bright objects.

2)  Variable stars, quasars, etc. A catalog of such objects can be built
    up from the set of stripe 82 exposures already in hand.

3)  Asteroids. Since stripe 82 goes through the ecliptic, there will be
    plenty of these. These can often be vetoed from the movement between
    g- and r-band exposures. Also, they would not be confirmed in the
    following scan.

![](media/image1.png){width="5.997916666666667in"
height="2.2555555555555555in"}

Figure 1: Galaxy NGC 6962 (z=0.014) as seen in two g-band SDSS
exposures. The 2nd frame, taken 21 days after the first, contains on
observation of SN 2002ha, a type Ia supernova discovered by LOTOSS. The
frame on the right shows a subtraction of the first two frames.

Our measurements of the execution time are consistent with previous
experience, totaling about 160 sec/field/filter with a 2.8 GHz CPU. We
have been encouraged to propose an engineering run for the Fall 2004
SDSS observing period. Our plan is to use this period to develop and
checkout the hardware platform and pipeline operation in real operating
conditions, to ensure that we've allocated sufficient computing
resources, and to obtain at least some initial science-quality data. Our
estimates of the needed computing resources are listed in Table 1.

In parallel with mountain top processing, data tapes will be shipped to
Fermilab and processed through PHOTO (and the PT pipeline) in the usual
way. The resulting catalogs should be of broad interest for time domain
studies, e.g., QSO variability. We expect to also run the frame
subtraction code at Fermilab; this will provide more accurate
information in all the filters on the SN candidates previously
identified. In addition, we would like to explore the feasibility of
modifying PHOTO to run on the subtracted

frames--- this could provide improved photometric accuracy over that
yielded by the current frame subtraction code. If this is possible, in
the end this will essentially amount to incorporating the relevant
(modified) parts of PHOTO into a modified back-end of the subtraction
pipeline.

We currently envision no strong time constraints on tape shipment or
Fermilab data processing, since this data will not be used for targeting
SDSS spectroscopy. Given that the data will for the most part be of
generally poorer quality than we have currently, the QA requirements on
Fermilab processing can be substantially relaxed: we will mainly want to
be able to ensure that the camera is working properly and that the
non-photometric data can be processed without crashing. In the near
term, we will want to study the non-photometric and moony 2.5m data in
hand and understand what happens to PHOTO as the data quality degrades.

At this time, we do not foresee a requirement for a database for the
repeat scans, since we believe the SN and other time domain science
could be carried out without it. However, this should be given further
thought before being dismissed.

We have also discussed the desirability of making this time domain data
rapidly available to the public---with suitable QA caveats---so that
variable phenomena could be followed up independently. We will need to
explore what this means---which processed data should be released, how
should it be released, how much work is involved and by whom? One
possibility would be to simply make available a subset of what is
generated in the mountain top processing---e.g., the corrected frames
for each night, or corrected frames plus subtracted frames, or corrected
frames plus subtracted frames plus a subtraction catalog (with suitable
warning flags).