The History and Future of STScI DADS
Niall Gaffney, Richard Kidwell, Mark Kyprianou, and Faith Abney (STScI)An Introduction to DADS

Changing with Changing Times

DADS (Data Archive and Distribution System) is used for archive and retrieval of the Hubble,
FUSE, Kepler, and the JWST data from STScI. DADS was initially developed by Loral tosupport the HST archive in 1993. In its current incarnation it provides

In 1999, the on the fly reprocessing system (OTFR) was added to DADS. In this system,
data for active instruments are generated at the time of request using the currentcalibration software and reference files. This allowed STScI to deliver the most up to date
data for all instruments that had not been finally calibrated without storing each version on
expensive archive media. Initially this increased our average data delivery time by up to aday, but with improvements in hardware this has decreased to less than an hour. Given thereliability of the current system, even the most skeptical of users eventually approved ofthe change.

• Near-line and deep storage of all data on archival class storage media• A catalog for all data in the system and key associated metadata• A flexible XML interface from which different applications can request data
• Options for different methods of data delivery (CD, DVD, ftp, sftp, or staged for ftp)
• A proprietary rights system to allow only approved users to retrieve data during its
proprietary period.
As technologies evolved over the sixteen years of thissystem, so has both the system and users expectations.
Computations and storage have largely followed Moore’s
Law while network performances have also improved. We
have also witnessed the growth of archival datapublications, growing from a small publication rate to thatwhich currently rivals that of data published by PIs from GO
programs.

$!!"#,!"#+!"#*!"#)!"#-./01223456#(!"#73856#9:/;3<5#=5251:/;#'!"#>?#@AB03/1CD.2#&!"#%!"#$!"#!"#$,,&#$,,*#%!!$#%!!(#%!!,#We discuss that evolution here with a focus on what lessons have been learned to craft a
flexible, extensible, and reusable astronomical data archiving system from our experienceswith DADS and its current (HST, FUSE, KEPLER) and future missions (JWST).

DADS round one

With computational hardware following Moore’s law andarchive media becoming significantly cheaper while
hardware on orbit remains limited by communications
bandwidth, it has become possible to move back to arecalibrate when needed archive. We areinvestigating a system inspired by CADC and ECF to
return to a static archive and reprocess the data aschanges in reference files or calibration files dictate.
The advent of web 2.0 and the VO has created aneed for more immediate data responses. Thesetechnologies allow archives to create moreinteractive interfaces with applications such as theHubble Legacy Archive (HLA) and Aladin.

Life with New Missions and Systems

The DADS system was originally a custom software package designed and developed by
the Loral corporation in 1993. While initially meeting the requirements set forth for thesystem, the system did not perform up to expectations mainly due to its optimization forlarge files, which significantly hindered the ingest of the smaller files. This shortcoming
was not revealed until the vendor delivered the completed software system for validation.
This optimization also pinned the archives storage hardware on the Sony platter system,
neglecting any potential future growth. From these mistakes, we learned that
• Numerous small files can lead to suboptimal file storage issues• Designing a system without an eye towards future technology improvements leads to an
inflexible and stagnant system• Designing purely to requirements without use-cases will often lead to significant
shortcomings in the system delivered• Transparency during the design and development process is important as it can reveal
potential problems before they become immutable and expensive to fix
This system ran from 1993 to 2004 when it was superseded by the next generation ofDADS, designed and developed completely in house at STScI using these lessons toguide us.
After operating for some time, it became clear that the HST archive, with its one year
proprietary data periods for data, was being used more like a library than a classicarchive; with most data being retrieved multiple times during the course of the mission.
Hence the design of the distribution system is at least as important as data ingest.

Migration from FORTRAN to JavaOne of the other major drivers for the redesign of DADS was to liberate it from being aVMS-only system to one that can adapt to changes in hardware, OSes, and capabilities
driven by the needs of new missions. Over this time, we have
• run under 4 OSes including operationally on multiple versions of Solaris and Linux• run on 5 different hardware platforms with different core architectures• upgraded to 3 different long term archive storage media (Sony Platters, MO, UDO)
• ported data between 5 different file systems (UFS, EXT3, QFS, XFS, and ZFS)
• incorporated 3 operational missions, 2 engineering missions, and 1 future mission• accomplished this with a current development staff of less than 3 FTEs.
This would not have been possible without the modular design of the system that isolates
system and mission specifics within the code. Nor could it be done without the virtual andobject oriented nature of the Java system; removing us from the bonds of many OS
specific problems.
Currently we are also migrating our database servers from Sybase to SQLServer. This
migration has been as simple as bcping the current tables from one server to the otherand simply changing the JDBC driver within DADS. The code neutral nature of the JDBC
has saved us significant porting efforts in migration, as illustrated by the efforts needed toport our other archive subsystems.

DADS 2010 and Beyond

From 2001 to 2004 the DADS system was completely redesigned and migrated from
FORTRAN to Java. The intent was to create a more generic astronomical data archiving
system based on software agents interacting with stand alone servers coordinating
individual tasks. While needed at the time, much of the effort went into object serializationand interprocess communications.

As the needs of the astronomical community grows, so will DADS. With the future
missions at STScI planned to be archived using DADS, improvements still need to bemade. During this time we learned that

Were this developed today, the code could be significantly simplified using technologies
such as J2EE to handle object serialization, IPCs, and queue management. That said, theagent based system has lead to more flexibility and extensibility of the system thanpreviously possible due to the isolation of mission specifics within the code. Such mission
specific code need only be in theagent creation step. During thisredesign, as many of the HST
specifics were isolated aspossible. However the initial DB
design could not be fully
restructured while the mission
was active.

• While more expensive to develop, designs that encapsulates mission and hardware
specifics in smaller components that lead to systems that are cheaper to expand andmodify once operational.

XML Request

Validaton andagent creation
ServerQueue Monitor
ServerSerialize
to disk

E-MailStand alone servers
handling related tasks

Data ProcessingServerOrchestrates• It is more expensive and complicated to refractor a system once it is operational• Science archives that make their data public behave more like libraries than classicarchives and should focus heavily on data discovery and distribution.

End User

Data Retrieval
Server

Run Request
Agent• Design without attention to use cases and potential future growth can lead to systems
that do not meet current and future expectations

Data Delivery
Server

Database
Logging Server

E-MailIn addition to mission abstraction, storage media was also abstracted in the archive. By
encapsulating the storage requirements in a factory interface, we are now able to easily
incorporate new storage media as the media technologies improve over time without
significant impact to any other part of the archive.

With redesigns inspired by these lessons learned, DADS has been able to support
missions with very different data requirements (e.g. HST and Kepler) with little to nochanges needed to its core architecture. Many of these lessons were learned while thesystem was actively archiving and distributing HST mission data which lead to more
complexity in their implementation. However, these lessons have been key to thecreation of the current flexible and reusable archive system making it viable for the needsof the JWST project and beyond.