Data Archiving in Experimental Physics
Leo R. Dalesio, William Watson III and Matthew Bickley (Jefferson Laboratory),
Matthias Clausen (Deutches Elektronen Synchroton)
LANSCE 8, MS H820
Los Alamos National Laboratory
Los Alamos, New Mexico USA 87544

Abstract
In experimental physics, data is archived from a wide
variety of sources and used for a wide variety of purposes.
In each of these environments, trade-offs are made between
data storage rate, data availability, and retrieval rate. This
paper presents archive alternatives in EPICS, the overallarchiver design and details on the data collection andretrieval requirements, performance studies, designchoices, design alternatives, and measurements made on
the beta version of the archiver.

buffers with before event and after event data (like ahardware scope). Channel data is wanted for data analysis
and trending plots of channels through time. The trending
plots of channels through time is the most demanding inthat the last day′s worth of data containing up to 1,000
samples are required to be displayed within one second perchannel. We are allotting 200 msec of this time for retrieval
and 800 msec for display. In addition, 1,000 samples fromthe last 30 days must be displayed within 4 seconds perchannel. We are allotting 3 seconds of this time forretrieval. Requirements were given to view data within aminute of data taking. With these goals in mind, wereviewed the existing archive facilities in EPICS.

1 Introduction
When designing a new archive facility for data archiving
for the Experimental Physics and Industrial Control System(EPICS), a collaborative team was formed from Jefferson
Laboratory (JLAB) and Los Alamos National Laboratory(LANL). The requirements were collected, the work wasdivided among collaborating institutions, and anincremental completion plan was made. As a preliminary
step for the data collection and retrieval portion of theproject, some performance studies were done to investigate
alternative design options.
2 Requirements
Requirements were gathered from JLAB, LANL,
Stanford Linear Accelerator Center (SLAC), and Argonne
National Laboratory (ANL) for a variety of projects.
Archiving at multiple workstations with the ability toarchive the same channel in multiple archives. Data mustbe saved as single channels with varying rates and as sets
where many channels are taken at a single point in time.
Archive data must be available to distributed data viewers,
data management, and data analysis packages, anywhere on
the network. These requirements led to the design of a
distributed data archiver that collects all data types andarrays (one-dimensional) of all data types from any I/OController (IOC) in the control network.
The requirements that affect the design of the archive
file structure are data storage rate, data retrieval rate, andlatency between archiving and using archive data. For longterm storage of system parameters, SLAC requires 1,000
channels every 3 minutes; Jefferson Lab requires 6,000
channels every 10 seconds; and LANSCE requires 15,000
channels every minute. To support analysis of accelerator
transient events, we need to save data on change at 5,000
channels per second and have the ability to trigger data

3 Alternatives existing in EPICS
Several archiving solutions are already available in theEPICS toolkit. They were developed with different aspectsof data archiving in mind. Alternative archiving methods
include: ARR from LANL, a relational database solutionthat was done at Tate Integrated Systems (TIS), an IOC
based archiver done by Kinetic Systems Corporation(KSC) and a distributed archiver and viewer done at
Deutches Elektronen Synchrotronen (DESY).
The archiver that is distributed with EPICS was designed
for archiving commissioning data for a linear accelerator. It
handled five thousand event changes per second. The filesize is defined at invocation and is not able to expand pastthis allocation. The user interface has been found to bedifficult to use and the lacks feedback during data taking.
As a result of the lack of positive feedback on datacollection, several commissioning runs were without dataalthough they believed it was being collected.
The KSC solution was designed for short term, highvolume data in the industrial data acquisition market. It
archives up to fifteen megabytes of data per second. The
data is archived onto mass storage in the IOC undervxWorks. The data is only available for viewing andanalysis after the experiment is complete.
The TIS solution uses relational database technology.
There is support for Dbase and ORACLE. The solution wasdesigned for low volume data that will be kept for manyyears. Access to/from DBASE on a SPARC 5 wasmeasured at 200 records per second, where each record cancontain up to 255 values. On ORACLE, the performanceon a SPARC 20 was measured at 50 records per second.
The latency between data taking and data viewing gets longwith large, high rate channel counts.
At DESY, the archiver was designed to keep facility
information and make it available in a multi-control systemenvironment using CORBA and IDL. In this design, eachchannel is stored in a separate file. This makes access timewithin a channel very fast. Data storage for a high channel
count will slow archiving by requiring many files to beopened and closed. The file access issue is mitigated using
front end buffering and flushing the buffers when needed.
Each of these solutions has been operational andsuccessful. However, the full ranges of requirements were
not met by any of them. Finally, we decided to design analternative archiving facility.
4 Archiver design
The collaborative team from JLAB and LANL
developed a preliminary design to meet the requirements.
(Figure 1) The initial release will use the channel accessprotocol to collect data. There are two data collectionengines included: one for collecting sets of data at a given
point in time and a channel archiver for collecting
individual channels at independent rates. The data retriever
is initially connected directly to a set of data management
routines, and has a set of routines available for use by data
analysis packages and data viewers. There are two dataviewers currently under development in UNIX and JAVA.
Data Management

Time/Data Viewer

Data Analysis

CDEV Service Library

Channel Retriever

Channel Archiver

Save Set Retriever

Save Set Archiver

Channel Access Client Library
Channel AccessArchive Record Any Field

introduces delays on retrieval time by requiring theretrieval routines to follow backward links to pick up amultitude of data chunks. Scrolling forward in time would
be most severely impacted using this method. Creating andmaintaining data areas for each channel, limits the numberof file operations for data retrieval, but puts a burden on thedata storage side to incrementally fill these areas byseeking and writing the next partial set of samples. It
becomes clear that there is a large range of possibilities for
optimizing either storage or retrieval. The optimization ofone – tends to affect the performance of the other. To
decide on an approach, the requirements were reviewed
and some performance tests were made.
6 Performance of file operations
A study was done changing the amount of data, thebuffer size, the number of writes, the number of seeks andthe number of file opens/closes. (Table 1)
Channels

Di
sk
Wr
ites

B
yt
es

See
ks

O
pens

S
ec
on
ds

B
yt
es
/S
ec

1,000
1,000
1,000
100
10
10
10

100
100
10100
1,000
1,000
1,000

1K
100
100
100
100
100
100

100K
100K
10K10K10K
10K
100K

111
1
11K
10K

198.1
19.6
1.2
1.2
1.3
56.5
454

505K
510K
833K
833K
769K
17K
2K

The table shows that opening and closing file descriptors
on each operation results in a 90% reduction inperformance. File open/close needs to be minimized. Fileseeks reduce the performance by about 40%. If we seek
during storage, to group data into one area in the file thenwe reduce the seeks required to fetch it. If we stream data
to disk then we require more seeks when the data isretrieved.
7 Adopted file structureThis paper will primarily focus on the channel-basedarchiver and retriever.
5 Data storage techniques
Optimization for data storage and retrieval isaccomplished using several mechanisms. Data buffering isused to optimize storage rates. When the buffering is donein the front end computers, it optimizes networktransmission and client side CPU utilization by minimizing
the number of packets that are sent over the network andhandled by the client. It also offers a backup storage when
the archiving machines or the network are unavailable. Thebuffering increases the latency for data availability,
however.
Using a streaming write on the client side optimizes disk
access by removing any need to seek to an area before
storing the data. This approach optimizes writing time, but

The file structure chosen attempts to balance therequirements for retrieving data against the storage rate.
(Figure 2) To minimize the search time for a given channel,
a directory is kept containing all channels that have beenarchived. A hash table is used to minimize the seek time.
The entries contain pointers to the most recent and oldest
data collected. Each data file contains a single area for eachchannel for the time period encompassed by the file. If
each file contains 4 hours of time, then there is one area on
the file for 4 hours of data for each channel included in thefile. Each of these data areas contains a header with thefirst time, last time, number of samples, and pointers to thenext and previous time periods. For channels that do notchange in a given time period, no file space will be used.
Using one file per time period minimizes the number of file
opens/closes. Keeping a single file descriptor for a new filewhen the next time period occurs incurs only one
additional file open. Fetching the data from the most recent

time requires only two file opens, one for the directory andone for the data file. Seeking for an arbitrary point in time
will require one file open per time period, with one fileseek and one read of the data header.
8 Configuration parameters provided
Configuration parameters are provided to optimize
F il e Structure Com promise
Directory

H ash Index
Chan 0First file
First offset
Last file
Last offset
First Time
Last Time
Collision PtrChan 1

Data File 0
Chan 0
P rev file
Prev offset
Next file
Next offset
First Time
Last Tim e# Samples¡ ¡- .-..

D a t a File N
Chan 0
P rev file
Prev offset
Next file
Next offset
First Time
Last Time# Samples

Data

Data

Chan 1Chan 1Figure 2

data that is not different. It is best to use channel access“gets” when a channel is changing at a higher rate than it isarchived. It is most efficient to use “monitors” when avalue changes at a rate that is less than or equal to thearchive rate. The channel access monitor threshold iscurrently specified for an instance of the archiver.
9 future plans
In the near future, a standard set of retrieval routines isto be defined. With this standard library, we should be able
to have access to all archive data with any viewers or data
analysis packages. In addition, the use of the DESY archive
buffering techniques will be integrated to allow data
backup when an archiver is not active and higher
bandwidth through front end filtering and buffering. Moreperformance tests need to be done to characterize thebottlenecks and degraded modes of operation. Further
optimization is expected.
10 Conclusions

conditions for different requirements. The configuration
parameters include data buffer rate, data flush to disk rate,
time period per file, and channel access monitor threshold.
The data buffer rate is the maximum frequency that achannel is saved. The data is placed into a circular buffer inthe client side at this rate. The rate is specified as a default
rate in floating point seconds and supports a different rateon each channel.
The rate that the data is flushed to disk is an archiverwide parameter. It is the rate that the channel archiver will
write data to disk. The buffers are allocated to hold twice asmuch data as is collected in the write frequency. This rate
determines the latency between data taking and the use ofdata by the archive data viewers or analysis packages.
The time period per file determines the size of each file.
The longer the time period per file, the larger the file. The
more files that are created, the more file operations that arerequired to retrieve data.
The channel access monitor threshold is used todetermine when the archiver will use channel access“monitors” and when it will use channel access “gets”. If achannel is changing at a much higher rate than the data
buffer rate, there is a waste of network bandwidth and CPU
cycles on the client side for processing events that are
overwritten before they are archived. Using channel access“gets” to archive channels at a higher rate than theychange, wastes resources as above by fetching and storing

The channel archiver is currently running at four
collaborating laboratories. On a SPARC 10 we havemeasured 1,000 channels scanned in the IOC and changingat 2 Hz, sending 2,000 monitors per second to the archive
client. Each entry includes an 8-byte time stamp and 4
bytes of status. The values are buffered and written to disk
every 30 seconds. This was run for 3 hours archiving 432megabytes of data, timestamps, and status with no data
loss. CPU utilization on the SPARC 10 was measured at anaverage of 20%. Retrieval of 1,000 points from a single file
was completed in under 200 msec. Retrieval of 1,000
readings from 30 files was done in under 800 msec. Further
test are required to determine if the 5,000 channel per
second archive rate will be met on a SPARC 10 classmachine. A beta release is available for all code exceptthat in Italics.
Acknowledgements:
This work is part of an extensive effort being put forward
by Jefferson Laboratory and Los Alamos National
Laboratory. Requirements and initial design work was done
by a group from Jefferson Lab, SLAC, ANL, LANL andDESY. Stephanie Allison, Janet Anderson and Marty
Kraimer made significant contributions during the initial
phase of this project.