The W. M. Keck Observatory Data Services Initiative
Progress Report, July 2023
The W. M. Keck Observatory (WMKO) and its partners have embarked on a major strategic campaign tomodernize the data flow and increase scientific yield: The Data Services Initiative (DSI). This reportsummarizes the project goals and the project’s progress in the last year made possible by the support ofour Keck Family of Supporters.
The goal of the DSI is to ensure that all data obtained at Keck is useful, discoverable, and quick. The DSI
will transform the end‐to‐end data flow of data delivered to Keck observers and the community at large
through the Keck Observatory Archive (KOA). Realizing the goals of the DSI will require significant data
infrastructure changes at Keck, robust observatory supported reduction pipelines, and extensions to theKOA to support real‐time ingestion of raw data followed by quick ingestion of science‐ready data.

The DSI: Major Components
The DSI will require design, development, prototyping, refinement, and deployment of four major
components: Prepare, Execute, Reduce, and Archive. Prepare refers to the user‐facing components that
result in an observation being ready to perform. Execute refers to the back‐end instrument andtelescope control tools driven by Prepare and results in the execution of observations and related
calibrations. Reduce refers to the set of data reduction routines and their underlying framework. Reduce
results in a set of quick‐look or science‐ready data products for the astronomer. Archive refers to thecuration, storage, and retrieval of data, both in raw and post‐Reduce science‐ready forms to the KOA.
Archive results in data delivery either for the original observer in near real‐time, or for the worldcommunity after the proprietary period, and will integrate with other world‐wide archives to increase
impact. The framework of DSI is shown in Figure 1.

Figure 1: Top‐level architecture of the DSI

Underlying both Prepare and Execute is the system of Database Driven Observing Infrastructure (DDOI).
DDOI, which is already undergoing significant design work at WMKO1 will create a single descriptive
language for observations across the entire WMKO instrumentation suite, similar to systems like theAstronomer’s Proposal Tool (APT) for the Hubble Space Telescope. The system will facilitate target
management, exposure management, instrument configuration, calibration sequencing, andobservation logging. The basic structure of DDOI is shown in Figure 2.

Figure 2: Architecture of the DDOI and interfaces

DDOI will contain an observing database that users can interact with through a set of APIs to create
observing plans, observing logs, and observing ‘templates’ that can be modified in real‐time to maintainthe classical nature of observations at Keck. The database also manages time allocations forobservations triggered as Target of Opportunity (such as gravitational wave EM follow‐up observations)or as part of cadence programs (such as precision radial velocity surveys of stars for exoplanetcharacterization). A critical component of DDOI is automatic and dynamic scheduling of calibrations.
Observers can specify the state of an instrument in the planning tool, and the calibration tool will create
a set of standardized calibrations for that instrument state. Observers will maintain the flexibility toexpand upon these calibrations, but the system will guarantee (within the bounds ofinstrument+telescope availability prior to sunset or after sunrise) that all instrument configurations used
to obtain data in a night have the requisite calibrations to be pipeline processed. WMKO Staff
Astronomers will work with the observing community and pipeline development teams to determine
these requisite calibrations.

The DSI: Progress in the Last YearThe last year has been foundational for DSI across each of the branches. Four data reduction pipelines
are now in nightly service pushing data in near real time to astronomers on the KOA. These instruments
include KCWI (an optical wavelength integral field spectrograph), MOSFIRE (an infrared multi‐objectspectrograph), DEIMOS (an optical wavelength multi‐object spectrograph), and KPF (an ultra‐precise
optical spectrograph for exoplanet characterization). Instead of waiting until after an observing run iscomplete, astronomers using these four instruments are gaining access to science‐grade data during thenight and can often change their observing strategy on the fly as a result. An example is shown in Figure3, which shows the flow of data acquisition, data reduction, and archive ingestion across a night’s

1

Please see the design reference documentation for DDOI here: https://docs.google.com/document/d/1pHaW‐
5fe4UfkooISmUTvCrbLo0GmS2YvMicyn6dRdNE/edit
Headquarters: 65‐1120 Mamalahoa Highway, Kamuela, HI 96743 * Phone (808) 885‐7887 Fax (808) 885‐4464

observing with KCWI. This flow has only been possible due to work on the data reduction pipelines AND
the real‐time ingestion software developed at WMKO under DSI in 2022.

Figure 3: A night observing with KCWI showing data acquisition (blue), ingestion in real time at the archive (green) and data
reduction (orange)

Additionally, significant work has been made to improve the observer portal, shown in Figure 4, and theDDOI system which underpins all DSI. In the coming months, the DDOI system, which already runs theKPF instrument at the execute level, will be deployed fully end‐to‐end on the NIRES instrument, completewith a new DRP interface for real time ingestion. This end‐to‐end demonstration will be a major
milestone for the project and will serve as the template for other instruments under DSI at WMKO.

Figure 4: New Observer’s Portal interface
Note: Report created by Keck Observatory Deputy‐Director and Chief Scientist Dr. John O’Meara.

Headquarters: 65‐1120 Mamalahoa Highway, Kamuela, HI 96743 * Phone (808) 885‐7887 *Fax (808) 885‐4464