U.S. Geological Survey Near Real-Time Dst Index

By J.L. Gannon, J.J. Love, P.A. Friberg, D.C. Stewart and S.W. Lisowski

Open-File Report 2011–1030

U.S. Department of the Interior

U.S. Geological Survey
ii

U.S. Department of the Interior

KEN SALAZAR, Secretary

U.S. Geological Survey

Marcia K. McNutt, Director

U.S. Geological Survey, Reston, Virginia: 2011

For product and ordering information:

World Wide Web: http://www.usgs.gov/pubprod

Telephone: 1-888-ASK-USGS

For more information on the USGS—the Federal source for science about the Earth,

its natural and living resources, natural hazards, and the environment:

World Wide Web: http://www.usgs.gov

Telephone: 1-888-ASK-USGS 

Suggested citation:

Gannon, J.L., Love, J.J., Friberg, P.A., Stewart, D.C., and Lisowski, S.W., 2011, U.S. Geological Survey Near RealTime Dst Index: U.S. Geological Survey Open-File Report 2011–1030, 10 p.

Any use of trade, product, or firm names is for descriptive purposes only and does not imply 

endorsement by the U.S. Government.

Although this report is in the public domain, permission must be secured from the individual 

copyright owners to reproduce any copyrighted material contained within this report.
iii

Contents

Abstract ......................................................................................................................................................................... 1 

Introduction.................................................................................................................................................................... 1

Methods......................................................................................................................................................................... 1

Definitive Dst.............................................................................................................................................................. 1

Near Real-Time Dst ................................................................................................................................................... 2

Data............................................................................................................................................................................... 2

Magworm System.......................................................................................................................................................... 4

Validation....................................................................................................................................................................... 4

Summary ....................................................................................................................................................................... 5

Acknowledgments ......................................................................................................................................................... 5

References Cited........................................................................................................................................................... 6

Figures

1. Spectra showing resolution of periodic disturbances using the full 23-year time series and the 

operational-length 6-month time series.. ........................................................................................................... 7

2. Comparison of operational time series using three- and four-station input over a two-month 

period including the Halloween storm of October and November 2003............................................................ 8 

3. Spectra of Dst variation from three periods showing example storms..................................................................... 9 

4. Comparison of the USGS definitive Dst index, the USGS real-time operational Dst index, the 

difference between the USGS indices, and the Kyoto Final Dst index for the Halloween 2003 

storm period.....................................................................................................................................................10

Table

1. Correlation coefficients and skill scores (prediction efficiencies) for USGS Dst indices. ......................................... 5
1

U.S. Geological Survey Near Real-Time Dst Index

By J.L. Gannon,1 J.J. Love,1 P.A. Friberg,

2 D.C. Stewart1 and S.W. Lisowski2 1U.S. Geological Survey, Box 25046, Mail Stop 966, Federal Center, Denver CO 80225-0046

2Instrumental Software Technologies, Inc., PO Box 963 ,New Paltz, NY 12561 

Abstract

The operational version of the United States Geological Survey one-minute Dst index (a global 

geomagnetic disturbance intensity index) uses either four- or three-station input (Honolulu, Hawaii; San 

Juan, Puerto Rico; Hermanus, South Africa; and Kakioka, Japan; or Honolulu, San Juan and Guam) and 

a method based on the U.S. Geological Survey definitive Dst index. The method uses a combination of 

time-domain techniques and frequency-space filtering to produce the disturbance time series at an 

individual observatory. We compare the operational output to the U.S. Geological Survey one-minute 

Dst index (definitive version) and the Kyoto (Japan) Final Dst and show that the U.S. Geological 

Survey operational output matches both definitive indices well.

Introduction 

The Dst index is a global indicator of the state of the Earth’s geomagnetic activity used in 

scientific studies and in defining space weather effects (for example, McCollough and others, 2008; 

Pulkkinen and others, 2006; Reeves and others, 1999). Because of its proven usefulness in 

understanding space weather phenomena, if Dst could be provided reliably in near real time it could 

also be useful as a specification tool to understand the current state of the ring current and the 

geomagnetic environment, as input to real-time models, and as a monitoring tool for satellite and 

electric power-grid operations. International stations, including Honolulu, Hawaii; San Juan, Puerto 

Rico; Kakioka, Japan; Hermanus, South Africa; and Guam provide necessary geomagnetic data. 

The Kyoto (Japan) World Data Center provides their Dst index (Sugiura, 1964) at several 

processing levels, including Final, Provisional, and Quicklook. Their Final version currently is available 

through the year 2003 and is their most-processed version, using definitive magnetic field data. The 

Provisional version is available for the years 2003–2007 and uses preliminary magnetic field data. 

Quicklook is their near real-time version. All of these processing levels are available with one-hour time 

resolution. 

The U.S. Geological Survey (USGS) now is capable of providing a near-real-time, one-minute 

time-resolution Dst index. We will show that the USGS real-time Dst index closely matches the more 

highly processed indices and provides Dst at a higher (one-minute) time resolution, making it wellsuited for scientific analyses and operational uses. 

Methods

Definitive Dst

The USGS definitive Dst is produced through a combination of time and frequency-space 

analyses (Love and Gannon, 2009; Gannon and Love, 2009). A baseline subtraction is done by fitting a 
2

5th-order polynomial to quiet periods in the time series, effectively detrending the data set, and removing 

the slowly varying crustal and internal field contributions, or secular variation (SV) components. The 

solar quiet (Sq) variation is then estimated by transforming the time series (with large geomagnetic 

storm contributions removed) into frequency space, where components of known Sq frequencies are 

identified and removed. Through this sequence of steps, a time series of magnetic field-disturbance 

values is obtained for each of four contributing stations. Those series are weighted by magnetic latitude 

and averaged, yielding the definitive index. 

Near Real-Time Dst

The near real-time method is based on the definitive method, although it is somewhat scaled 

down because of processing constraints that are needed to produce the index on a near real-time 

schedule. Most of the method differences are a result of the shortened analytical period used for the realtime index. For the definitive one-minute index, an analytical time period of 23 years is used. The 

longer the time period, the more well-resolved the periodic contributions become in the frequency 

domain. For the operational index, an analytical period of six months is used. As new data are included, 

the index is updated and reprocessed using a sliding window, always recalculating using the same 

amount of time. Although some sacrifices in the resolution of Sq harmonics are made, and only daily 

and shorter signals are well-resolved, the largest contributions still are easily identified, such that there 

are only small differences between the operational index and the more computationally intensive 

definitive index. 

As in the definitive version of Dst processing, we remove large storm variations in the time 

domain before extracting Sq in the frequency domain. We do this to minimize any periodic influences 

from times that are obviously storm-driven. The definitive version uses a daily interpolation technique 

to replace large fluctuations with a realistic quiet-time signal. This method does not work at end values, 

as it requires a full day on either side of the disturbance to produce accurately interpolated data. To 

interpolate storm periods that occur in our most recently acquired data, that is, data on the most recent 

end of the time series, we calculate Sq based on the time range up to the storm and then extrapolate an 

Sq signal for the remaining day points. Because Sq does not vary greatly from day to day, extrapolation 

is easily performed.

Another simplification made in the operational procedure is in the secular variation (SV) 

subtraction. In the definitive version, the SV fit is done very precisely using a polynomial of order 5 

over the full 23-year time span. In the operational version, we use a linear fit to the six months of data. 

If we were to do a polynomial fit, we would need fewer degrees of freedom proportional to the 

reduction of time span in order to capture the same time-scale fluctuations. A linear subtraction is 

sufficient for approximating the equivalent fluctuations over a six-month period. In other words, the 

secular trend over the shorter time span is nearly linear. 

Data

To produce the operational index, a time series of one-minute-resolution horizontal magneticfield intensity is required from each contributing station. The definitive version of the algorithm uses 

data from the four observatories traditionally used by the Kyoto data center: Honolulu, San Juan, 

Hermanus and Kakioka. We are not limited, however, to the use of those four stations. The USGS 

realtime Dst currently uses data from the four traditional stations as well as from a secondary, backup 

product using only three USGS stations (with Guam added in place of Hermanus and Kakioka). The 
3

preliminary and definitive data used in testing were obtained from Intermagnet (www.intermagnet.org) 

and the Kyoto World Data Center. 

Differences between the definitive data set used in the definitive index and the preliminary data 

set used in the operational index include sudden spikes in the magnetic field data or baseline offsets due 

to calibration or even to physical changes to the magnetometer instrument, such as physical relocation. 

Spikes occur frequently and can be easily detected. Baseline offsets are more difficult to identify. Both 

variations could have a significant impact on the realtime output. It is worth discussing how these issues 

are detected and how much of an effect they may have on the operational output. 

We remove spikes with a check that flags any variation that is outside of a specified mean and 

standard-deviation range. As data spikes come in, they are flagged as bad data. In the initial processing 

loop, they are ignored and are not included in the final Dst average, and they are interpolated once there 

is at least a day of good data following the spike. The effect on the output time series is the same as a 

data gap, which is discussed in more detail below. 

Baseline offsets occur in the data due to calibration errors or to physical changes in the 

magnetometer environment. These appear as sudden steps in the data. They are easily changed once 

they are detected, by forcing a match in the data across the step. As the data are subsequently detrended, 

the absolute magnitude of the time series does not matter, and this forcing can be done on either side of 

the step. The difficulty lies in the detection of the baseline offset in the first place. Spikes are typically 

very short in duration and the data can be disregarded for a minute or two, but a baseline offset will 

affect all data past the point of occurrence. The most likely cause of a baseline offset is a calibration 

issue between voltage bins in the magnetometer's digitizer. When this type of event occurs, a flag is 

relayed with the data, alerting observers of a bin-change exception. At that point, the data are forced to 

match across the recording interval, thereby ensuring a continuous time series. The more difficult type 

of baseline offset occurs without that type of warning. If a baseline offset occurs, it will continue to 

affect the data until collection time has moved past the six-month input-time window or until the offset

is manually changed. 

The issues related to reduced analytical time span were mentioned in the method section, and 

they include differences in data detrending (the secular variation subtraction) and the frequency-space 

resolution of stationary periodic signals that we label as solar quiet, or Sq. The more interesting 

question is whether Sq can be resolved with 100 times fewer points. Figure 1 shows a comparison of the 

power spectrum of the 23-year time span (used with definitive data) versus the 6-month time span (also 

with definitive data). The power spectrum of the shorter time series is not as resolved, but clear peaks at 

the Sq frequencies still are apparent, although a wider filtering window must be used to remove the Sq 

peaks. 

The operational processing also must make allowances for late or missing data. A balance must 

be made between waiting an allowable time for a segment of data to appear and maintaining sufficient 

timeliness of delivery. This balance is dependent on the data source, as USGS data arrive more quickly 

to our system than do data from Hermanus and Kakioka. For the three-station (that is, USGS-only) 

index, we allow 15 minutes for data arrival. If the data are not received in that time, or if the next point 

is received, the index is calculated based on reduced available data. For the four-station index, the 

maximum wait time is 25 minutes. That increased time allows for the slower nature of FTP and email 

data delivery from Hermanus and Kakioka as compared to the USGS Earthworm data-acquisition 

system.
4

Magworm System

The input and output for the real-time index are done through connection to a Magworm system. 

Magworm is a geomagnetism-data adaptation of the seismological Earthworm system (Johnson and 

others, 1995) for data acquisition, storage and processing. The Magworm real-time index module 

connects to a waveform storage mechanism referred to as a Wave Server. The Wave Server receives and 

serves both magnetometer data from the geomagnetic observatories and the Dst values computed from 

the magnetometer data. Due to the small (1-minute and 1-second) data cadence of geomagnetic data 

compared to seismological data, a Wave Server is capable of storing decades of geomagnetic data, far

beyond the six-month data period needed for the real-time index computation.

The magnetometer data and the computed Dst data are packaged using the standard Earthworm 

time-series packet known as a TRACEBUF2. The magnetometer data arrive at the Wave Server with a 

5-minute packetization latency; the Dst data arrive with a 5- to 10-minute latency. The Wave Server's 

input stream uses a shared-memory transport protocol that is common to all Earthworm and Magworm 

modules. The real-time index module requests magnetometer data packets from the Wave Server using 

a defined communications protocol over a TCP/IP network. Computed Dst values then are added to the 

Wave Server's input stream for storage and dissemination to internal or external sources. 

Validation

In order to validate the near real-time USGS Dst index output, we compared it to the more 

refined definitive USGS 1-minute Dst index, as well as to the Kyoto Final Dst indices. Because all of 

these indices are derived products instead of measured values, there are no physical observables which 

can be used for validation purposes, and there is no correct answer. We wish to know, however, that the 

realtime output can replicate accurately the definitive product and can be used with a high level of 

confidence in the same way. For each of the following examples, we use preliminary data as input to the 

operational algorithm to produce historical time series of the real-time Dst. 

We first compare the three-station (USGS only) and four-station versions of the USGS real-time 

Dst (fig. 2). We expect some variation between the two indices due to the equatorial positioning of the 

Guam observatory. That lower-latitude location subjects the Guam data to increased effects due to the 

equatorial electrojet, an ionospheric current system which is enhanced during a geomagnetic storm but 

is not usually considered a contributor to the intent of the traditionally determined Dst. As expected, the 

largest differences are apparent during the main phase of the storm periods, but in general the indices 

compare well. Because of the storm-time differences and the equatorial influences from Guam, we 

consider the three-station index to be a backup or secondary product to the traditional four-station index. 

Even with these differences, the correlation coefficient and skill score (the prediction efficiencies) 

between the three- and four-station operational indices are very high (see Table 1), suggesting that when 

shorter delay times are desired, the three-station index is a reliable substitute.

Figure 3 illustrates three example time periods that include storms of different sizes and 

characteristics. In each example, the USGS real-time Dst index is shown in blue and the USGS 1-minute 

Definitive Dst is in black. In each case, the real-time index matches the definitive index very well. This 

is particularly true during the main phase (the period of largest minimum values) of each storm. Storm 

periods are the times of greatest interest to the operational community, and it is most critical to have 

confidence in the magnitude of reported storms. The calculated skill score between the definitive and 

real-time indices for the time period 2004–2005 is 0.980, indicating strong prediction efficiencies. 
5

Table 1. Correlation coefficients and skill scores (prediction efficiencies) for USGS Dst indices.

[Abbreviations: ops, operational Dst index; def, definitive Dst index]

Dst Data Source Dst Index Type Correlation

Coefficient

Skill Score

(Prediction Efficiency)

Time Period

(years)

3-station ops 4-station ops 0.986 0.976 2003

4-station ops 4-station def 0.996 0.980 2004–2005

4-station ops 4-station def 0.995 0.972 2003

3-station ops 4-station def 0.988 0.957 2003

One of the largest and most interesting storms in the record of space physics is the Halloween 

storm of 2003. Figure 4 shows the comparison of the real-time and definitive indices for this critical 

time period. In addition, the Kyoto Final Dst is shown. The USGS near real-time index corresponds 

closely to the USGS definitive and the Kyoto Final indices. The 1-hour Kyoto index smooths much of 

the short-scale variation apparent in both versions of the USGS Dst, illustrating the value of a higher 

resolution (1-minute) index. The skill score between the USGS definitive and real-time indices for the 

year 2003 is 0.976, which is nearly identical to the 2004–2005 skill score of 0.980. 

Summary 

The USGS near real-time Dst index is created using a scaled-down version of the USGS 

Definitive Dst algorithm to provide a high (1-minute) time-resolution specification of geomagnetic 

activity in near real time. The primary version uses preliminary data from the four traditional Dst 

observatories (Honolulu, San Juan, Kakioka, and Hermanus) with a 15- to 30-minute delay from real 

time. The secondary version uses only USGS stations, replacing Kakioka and Hermanus with Guam, 

using a 5- to 15-minute delay from real time. Although the data are received with less delay and with 

valuable data-quality information, this product is considered secondary due to the equatorial location of 

the Guam observatory and resulting effects of the equatorial electrojet on the output from Guam. The 

four-station real-time Dst index compares very well to both the USGS definitive index and to the Kyoto 

Final Dst during critical storm periods, indicating a high degree of confidence that the real-time output 

can be used in operational models and analyses. 

Acknowledgments

The results presented in this paper rely on data collected at magnetic observatories. We thank the 

Japan Meteorological Agency, and the Hermanus Magnetic Observatory, for supporting its operation, 

and INTERMAGNET for promoting high standards of magnetic-observatory practice 

(www.intermagnet.org). We thank W.K. Tobiska for his contributions to the project. We thank C.A. 

Finn, Victor Tsai, and Paul Earle for reading a draft manuscript. We thank the IT specialists at the 

USGS for their invaluable assistance, in particular J. Tegeler, C. Bidwell and M. Sanders. 
6

References Cited

Gannon, J.L., and Love, J.J., 2011, The one-minute USGS Dst index: Journal of Atmospheric and SolarTerrestrial Physics, v. 73, p. 323–334 . 

Johnson, C.E., Bittenbinder, Alexander, Bogaert, B., Dietz, L., and Kohler, W., 1995, Earthworm—A 

flexible approach to seismic network processing: Incorporated Research Institutions for Seismology 

Newsletter, v. 14, no. 2, p. 1–4. 

Love, J.J., and Gannon, J.L., 2009, Revised Dst and the epicycles of magnetic disturbance, 1958–2007: 

Annals of Geophysics, v. 27, p. 3101–3131.

McCollough, J.P., Gannon, J.L., Baker, D.N., and Gehmeyr, M., 2008, A statistical comparison of 

commonly used external magnetic field models: Space Weather Journal, vol. 6, p. S10001. 

Pulkkinen, T.I., Ganushkina, N.Y., Tanskanen, E.I., Kubyshkina, M., Reeves, G.D., Thomsen, M.F., 

Russell, C.T., Singer, H.J., Slavin, J.A., and Gjerloev, J., 2006, Magnetospheric current systems 

during stormtime sawtooth events: Jour. Geophysical Research, v. 111, no. A11, A11S17, 13 p. 

Reeves, G.D., McAdams, K.L., Friedel, R.H.W., and O’Brien, T.P., 2003, Acceleration and loss of 

relativistic electrons during geomagnetic storms: Geophysical Research Letters, v. 30, no. 10, p. 1529. 

Sugiura, Masahisa, 1964, Hourly values of equatorial Dst for the International Geophysical Year, in

Annals of the International Geophysical Year: Oxford, Pergamon Press, v. 35, p. 9–45. 
7

. 

Figure 1. Spectra showing resolution of periodic disturbances using the full 23-year time series and the 

operational-length 6-month time series.
8

Figure 2. Comparison of operational time series using three-station (magenta) data from Honolulu (Hawaii), 

San Juan (Puerto Rico), and Guam, and four-station data (including Hermanus, South Africa, and Kakioka, Japan) 

in blue, for a two-month period, including the Halloween storm of October and November 2003. The top panel 

shows the entire Halloween storm period. The bottom panel offers a closer view of the first large main phase of that 

storm. Dates are given in the format mm/dd/yy.
9

Figure 3. Spectra of Dst variation from three periods showing example storms. The black trace in each panel 

is the USGS one-minute Definitive Dst index, the blue trace us the USGS Realtime Dst index, and the green trace 

is the difference between the two, offset by 50 nanoteslas (nT). Dates are shown in the format (m)m/dd/yy.
10

Figure 4. Comparison of the USGS Definitive Dst index (black trace), USGS Realtime operational Dst index 

(blue), the difference between the USGS indices (green), and the Kyoto Final Dst index (red), from the Kyoto World 

Data Center. The top panel shows the entire Halloween 2003 storm period. The bottom panel is a closer view of 

the first large main phase of that storm. Dates are shown in the format mm/dd/yy.