NILE: wide-area computing for high energy physics
Authors: Keith Marzullo, Michael Ogg, Aleta Ricciardi, Alessandro Amoroso, F. Andrew Calkins, Eric Rothfus
Venue: Proceedings of the 7th workshop on ACM SIGOPS European workshop: Systems support for worldwide applications (1996)

The CLEO project [2], centered at Cornell University, is a large-scale high energy physics project. The goals of the project arise from an esoteric question---why is there apparently so little antimatter in the universe?---and the computational problems that arise in trying to answer this question are quite challenging. To answer this question, the CESR storage ring at Cornell is used to generate a beam of electrons directed at an equally strong beam of positrons. These two beams meet inside a detector that is embedded in a magnetic field and is equipped with sensors. The collisions of electrons and positrons generate several secondary subatomic particles. Each collision is called an event and is sensed by detecting charged particles (via the ionization they produce in a drift chamber) and neutral particles (in the case of photons, via their deposition of energy in a crystal calorimeter), as well as by other specialized detector elements. Most events are ignored, but some are recorded in what is called raw data (typically 8Kbytes per event). Offline, a second program called pass2 computes, for each event, the physical properties of the particles, such as their momenta, masses, and charges. This compute-bound program produces a new set of records describing the events (now typically 20Kbytes per event). Finally, a third program reads these events, and produces a lossily-compressed version of only certain frequently-accessed fields, written in what is called roar format (typically 2Kbytes per event). The physicists analyze this data with programs that are, for the most part, embarrassingly parallel and I/O limited. Such programs typically compute a result based on a projection of a selection of a large number of events, where the result is insensitive to the order in which the events are processed. For example, a program may construct histograms, or compute statistics, or cull the raw data for physical inspection. The projection is either the complete pass2 record or (much more often) the smaller roar record, and the selection is done in an ad-hoc manner by the program itself. Other programs are run as well. For example, a Monte Carlo simulation of the experiment is also run (called monte carlo) in order to correct the data for detector acceptance and inefficiencies, as well as testing aspects of the model used to interpret the data. This program is compute bound. Another important example is called recompress. Roughly every two years, improvements in detector calibration and reconstruction algorithms make it worthwhile to recompute more accurate pass2 data (and hence, more accurate roar data) from all of the raw data. This program is compute-bound (it currently requires 24 200-MIP workstations running flat out for three months) and so must be carefully worked into the schedule so that it does not seriously impact the ongoing operations. Making this more concrete, the current experiment generates approximately 1 terabyte of event data a year. Only recent roar data can be kept on disk; all other data must reside on tape. The data processing demands consume approximately 12,000 SPECint92 cycles a year. Improvements in the performance of CESR and the sensitivity of the detector will cause both of these values to go up by a factor of ten in the next few years, which will correspondingly increase the storage and computational needs by a factor of ten. The CLEO project prides itself on being able to do big science on a tight budget, and so the programming environment that the CLEO project provides for researchers is innovative but somewhat primitive. Jobs that access the entire data set can take days to complete. To circumvent limited access to tape, the network, or compute resources close to the central disk, physicists often do preliminary selections and projections (called skims) to create private disk data sets of events for further local analysis. Limited resources usually exact a high human price for resource and job management and ironically, can sometimes lead to inefficiencies. Given the increase in data storage, data retrieval, and computational needs, it has become clear that the CLEO physicists require a better distributed environment in which to do their work. Hence, an NSF-funded National Challenge project was started with participants from both high energy physics, distributed computing, and data storage, in order to provide a better environment for the CLEO experiment. The goals of this project, called NILE [7], are: to build a scalable environment for storing and processing High Energy Physics data from the CLEO experiment. The environment must scale to allow 100 terabytes or more of data to be addressable, and to be able to use several hundreds of geographically dispersed processors. to radically decrease the processing time of computations through parallelism. to be practicable. NILE, albeit in a limited form, should be deployed very soon, and evolve to its full form by the end of the project in June 1999. Finally, the CLEO necessity of building on a budget carries over to NILE. There are some more expensive resources, such as ATM switches and tape silos, that it will be necessary to use. However, as far as possible we are using commodity equipment, and free or inexpensive software whenever possible. For example, one of our principal development platforms is Pentium-based PCs, interconnected with 100 Mbps Ethernet, running Linux and the GNU suite of tools.