Astronomical Data Analysis Software and Systems XIX
ASP Conference Series, Vol. XXX, 2009
Y. Mizumoto, K.-I. Morita, and M. Ohishi, eds.

O07.1

An Overview of the Palomar Transient Factory Pipeline
and Archive at the Infrared Processing and Analysis
CenterC. J. Grillmair, R. Laher, J. Surace, S. Mattingly, E. Hacopians, E.
Jackson, J. van Eyken, B. McCollum, S. Groom, W. Mi, & H. Teplitz
Infrared Processing and Analysis Center, M/S 100-22, California
Institute of Technology, CA, USA
Abstract.
The Palomar Transient Factory is conducting a wide-field, variable-cadenceoptical survey of the northern sky to detect transient, variable, and moving objects. As a member of the PTF collaboration, the Infrared Processing and Analysis Center has developed an image archive, a high-quality photometry pipeline,
and a searchable database of detected astronomical sources. The system is capable of processing and storing 300 Gbytes of data per night over the courseof the 5-year survey. With an expected total of ∼ 20 billion rows, the tablecontaining sources extracted from PTF images will be among the largest astronomical databases ever created. The survey is efficiently discovering transient
sources from asteroids to supernovae, and will inform the development of future
sky surveys like that of the Large Synoptic Survey Telescope.

1.

Introduction

The Palomar Transient Factory (PTF) is a wide-angle, variable cadence, opticalsky survey designed to detect all manner of transient objects, from asteroids tovariable stars to supernovae. The cadences employed and the PTF key projects
currently being pursued are summarized in Figure 1. The 5-day cadence survey
is the largest in terms of sky area (≈ 3π steradians) and is primarily aimed atfinding Type II Supernovae. A more detailed description of the science cases for
PTF is given by Rau et al. 2009.
The survey makes use of of the Oschin 48-inch Schmidt telescope (P48) atop
Palomar Mountain in California. The PTF camera is a 12K × 8K CCD mosaic,originally built for the Canada-France-Hawaii telescope, now modified to provide
faster detector readout and to operate in the more confined focal plane of theP48. The combination of telescope beam and camera provide 1.01 arcsecond
per pixel sampling over a field area of 7.9 deg2 . Typical survey exposure times
are 60 seconds, yielding limiting magnitudes of mg ≈ 21.4 and mR ≈ 20.6. Thelarge-area, 5-day cadence survey will be primarily conducted in the R-band. The
instrument and data taking system are described more fully by Law et al. 2009.
PTF achieved first light in December of 2008 and began survey operationsin May of 2009. Despite high atmospheric ash content (a result of large wildfires
in the area) that precluded observing for extended periods during the last halfof 2009, some 56,000 images (≈ 10 TB) have been collected to date.
1

2

Grillmair et al.

Figure 1.

PTF cadences and the key projects they serve.

A data flow diagram for the PTF system as a whole is shown in Figure 2.
By virtue of two distinct pipelines and a network of follow-up telescopes, PTF
provides automatic, real-time transient classification and follow-up, as well asa database of every source detected in each frame. All images, both raw andprocessed, are stored on spinning disk and will be available for download.
In this contribution we describe the Infrared Processing and Analysis Center (IPAC) PTF pipeline and archive. A separate PTF pipeline at the Lawrence
Berkeley National Laboratory uses image subtraction to detect transient sourcesin near real time and permit rapid classification and same-night follow-up. Conversely, the IPAC PTF pipeline operates on a less rapid time scale and is designed
to achieve the best possible photometric accuracy consistent with hardware capabilities and data rate. Output from the IPAC pipeline is used to feed a large
database that can be queried through the NASA/IPAC Infrared Science Archive(IRSA).
2.

IPAC-PTF Pipeline

In production mode, PTF generates ≈ 60 GB of raw data per night on average,
with peak volumes approaching ≈ 100 GB on clear winter nights. The raw
image data are sent from Mount Palomar to Caltech via fast microwave link andlandline. The raw data are ingested daily at IPAC, which includes storage on
spinning disk, registration of file location and MD5 checksum in the database,and verification against the nightly file manifest that is delivered with the data.
Once all science and calibration files have been received, the photometry pipeline
is initiated. All raw and processed images are stored on spinning disk and will
be made available for public download after an 18 month proprietary period.
A total of ≈ 300 TB of raw and processed image data will be stored over thecourse of the 5 year survey. This is approximately forty times the volume of

The IPAC PTF Pipeline and Archive

Figure 2.

3

Data flow diagram for the PTF system as a whole.

image data stored by the 2MASS (Skrutskie et al. 2006) and SDSS (York et al.
2000) surveys. Pan-STARRS will produce data at ten times the rate of PTF,but nearly all of the image data will be discarded.

3.

IPAC-PTF Computing Resources

The IPAC PTF pipeline hardware currently includes:• 12 Sunfire x4150 8-core pipeline drones• 2 Sunfire x4150 DBMS servers• 1 Sunfire x4150 operations file server (software, sandboxes)• 2 Nexsan SATAbeast 128 TB RAID-5 connected to IRSA file server for
raw and processed data
• 1 Nexsan SATABeast 36 TB RAID-10 for operations file server and sandbox• 1 Nexsan SATABeast 32 TB RAID-5 for database storage• 4 Nexsan SATAblades for primary file server (software, sandboxes)• 5 Nexsan SATAblades and 2 Sun 2540s for secondary database
An additional 60-100 TB of storage will be installed to complete the survey.
Parallel processing on 11 multiprocessor Linux machines is employed to giveprocessing throughput that meets the data-rate requirement, which is 5× thereal-time data acquisition rate. This requirement allows for processing nightly
data as well as reprocessing older data, which will be necessary as the pipeline isrefined and improved over time. Disk storage is configured as RAID-10 for dataintegrity. Both database and raw data are backed up on a weekly basis. In caseof data corruption or loss, up to a week’s worth of processing would have to berepeated in the worst case. A data flow diagram for the IPAC PTF pipeline isshown in Figure 3.

4

Grillmair et al.

Figure 3.

4.

Data flow in the IPAC PTF pipeline.

Pipeline SoftwareStandard image processing modules (bias removal, flat fielding) are written inC. Where possible (e.g. when > 100 sky images have been taken), flat fields
are generated each night by source-masking and median-combining all availablescience images. Though the PTF detectors have been in use for some time andsuffer a considerable range in pixel sensitivities and cosmetic defects (with one
detector currently being nonfunctional), these nightly flat fields are very wellmeasured and reduce such variations to very low levels. Figure 4 shows theeffect of this frame processing on a typical sky image.
Astrometry and photometry are carried out using community software, notably Astrometry.net (Lang et al. 2009), Sextractor (Bertin & Arnouts 1996),and Scamp (Bertin 2006). A high level flow diagram of the pipeline is shownin Figure 4. Sextractor is actually applied to each image a number of times in
the pipeline to mask sources prior to flat field generation, measure the effective
seeing, generate pixel weights and masks, and finally to generate photometry for
all detected sources.

The IPAC PTF Pipeline and Archive

Figure 4.
cessing.

5

A comparison of typical sky image before and after frame pro-

The astrometric solution for each images is independently checked. The
method employed consists of querying the database for appropriately bright andisolated 2MASS sources, matching them to sources extracted from the image,computing the RMS separation of the matches along the image axes, and requiring a minimum number of matches and both RMS values to be less thansome threshold value. Currently a minimum of 20 matches and RMS values lessthan 1.5 arcsec are required. Images not meeting these minimum requirements
are thrown out.
The pipeline currently processes a night’s worth of data in 4 to 5 hours,
ensuring that any necessary reprocessing of older data will be possible during
normal operations without creating a backlog. Multi-threading is used by thepipeline software to achieve this processing rate.
5.

Photometric Calibration

The goal of the IPAC PTF pipeline is to provide photometry accurate to 2%or better. An analysis of pipeline-generated flat fields indicates detector stability of ≈ 1%. Photometric calibration is carried out using stars observed inSloan Digital Sky Survey (SDSS) fields. For all fields that fall within the SDSS
footprint, a photometric solution for zeropoint and color term is computed bydirect comparison with SDSS photometry. This is used to compute a nightly
extinction coefficient, which is subsequently used to calibrate PTF photometry
from images that fall outside the SDSS footprint.
6.

Database

The PTF operations database is being run under open source PostgreSQL. The
primary Sources table will contain astrometry and photometry of every source
detection over the course of the survey. With between 20 and 40 billion rows,

6

Grillmair et al.

Figure 5.

Schematic diagram of the IPAC PTF pipeline.

each with 69 photometric and astrometric parameters, this table will be among
the largest ever created. Load testing under PostgreSQL has thus far shown nounexpected issues with regard to speed or scalability. Care has been taken in
the database schema to ensure that database entries and indices are written tospinning disk via separate I/O channels.
A source association pipeline, run daily, will use each night’s detections toupdate a Merged Sources table. This Merged Sources table will provide timeseries photometry for every detected source and is expected to be the primaryscience output of the IPAC PTF pipeline. We have developed an efficient methodof source association that is done in parallel for independent declination zones
and thus requires no inter-process communication.
7.

IRSA Interface

The NASA/IPAC Infrared Science Archive (IRSA) provides easy access to PTF
data, including both images and catalogs. Archive services for PTF include functionality for both the IPAC pipeline team, the PTF science team, and (eventually) the astronomical community. IRSA provides both raw and processed data,
including full FITS frames, image cut-outs, and supermosaics. Image previews
are augmented by 2MASS catalog overlays. The imaging data can be accessedThe IPAC PTF Pipeline and Archive

7through temporal or spatial searches. Figures 6 and 7 show examples of theimage preview feature.
IRSA provides access to PTF source tables through it’s standard catalog
search tool (“Gator”). The tables include both Sources (extracted from individual frames) and Merged Sources. A lookup functionality between the two tables
will be added in the near future. Records can be accessed with a variety ofsearch constraints for values in the table, including but not limited to position
and time of observation. A current coverage map is shown in Figure 8. Afterseveral years of observation, the PTF Sources table will be among the largest
tables served by IRSA (or any astronomical archive for that matter), rivalingeven the WISE individual sources catalog.

Figure 6.

Screen shot of IRSA PTF nightly summary page.

8

Grillmair et al.

Figure 7.
Screen shot of IRSA exposure preview, with 2MASS sources overlaid on the field.

Figure 8.
2010.

8.

IRSA coverage map of all PTF image data taken as of 15 January,

Future plans

We are currently studying the costs and methods associated with producing andserving:• Co-added images• Differential photometry• Improved photometric calibration• Variability classification and searching

Co-added images would enable detection of sources up to two magnitudes
fainter than the current mR ≈ 20.6 limiting magnitude of individual images. Differential photometry would permit relative photometric accuracies approaching
the stability limits of the detector. Depending on the level of accuracy achieved,
such a capability may enable the routine detection of transiting substellar objects.

The IPAC PTF Pipeline and Archive

9Over the course of the survey we expect to generate a very large catalog ofstars outside the SDSS footprint which have been observed and well calibrated
under photometric conditions. Using this catalog as a reference will ultimately
enable us to observe the entire observable portion of the sky under less than ideal
photometric conditions while still achieving our photometric accuracy goals.
On a somewhat longer time scale, we are studying automated methods ofclassifying variable sources using the Merged Sources table. Suitable metrics
could then be computed, stored, and used to quickly search for variables of aparticular type (e.g. RR Lyrae, cataclysmic variables, eclipsing binaries, etc.).
9.

Conclusion

The IPAC PTF pipeline and archive is an efficient, cost-effective, and userfriendly approach to enabling a new and largely unexplored field of research,
namely wide field, time domain astronomy. In addition to facilitating a large
number of ongoing scientific investigations, the IPAC PTF pipeline is serving
as a useful pathfinder for exploring the processing, handling, and serving of
extremely large data sets. This overview provides little more than an outline ofthe pipeline and archive; a more detailed description of the design and operationof the system is given by Laher et al. (2010).
Acknowledgments. We are grateful to Nouhad Hamam, Eran Ofek, NickLaw, Robert Quimby, John Good, and Serge Monkewitz for their many contributions to the design and development of the IPAC PTF pipeline.
References
Bertin, E. 2006, in ASP Conf. Ser. 351, Astronomical Data Analysis Software andSystems XV, ed. C. Gabriel, C. Arviset, D. Ponz, & S. Enrique, 112
Bertin, E., & Arnouts, S. 1996, A&AS, 117, 393
Laher, R. et al. 2010, in preparation
Lang, D., Hogg, D. W., Mierle, K., Blanton, M., & Roweis, S. 2009, arXiv:0910.2233
Law, N. et al. 2009, PASP, 121, 1395
Rau, A. et al. 2009, PASP, 121, 1334
Skrutskie, M. F. et al. 2006, AJ, 131, 1163
York, D. G., Adelman, J., Anderson, J. E., Jr., Anderson, S. F., Annis, J., Bahcall, N.
A., & Bakken, J. A. 2000, AJ, 120, 1579