**Some thoughts on how to proceed with the Time-Series Object**

DMS 00-03-09

The general objective of considering the definition of a time-series
object is to point toward possible standardization that will result in
the following benefits:

1\. There can be a set of standard interfaces supported by formatting
system software (e.g., possible extensions to, or built on top of, HDF,
CDF, IDFS), which then allows applications to work in a standard way
with time series data from a variety of instruments and across
disciplines.

2\. Data product developers will know what metadata is needed to support
the standard time-series view of their data, thereby enabling new data
products to be more readily accessible to existing application software.

3\. The understanding of some aspects of data made available as a
time-series object will already be known to data users and will therefor
reduce their learning time in handling new data products.

Within the FEPC we are pursuing a draft \'Time-series\' object to
demonstrate to ourselves (and then to the community) the feasibility and
utility of pursuing this type of standardization. We do not expect this
work to be a standard, but to point the way if this proves productive.
Once we have a draft definition, we can look at the possible
implications for supporting it by some of the current space physics
formatting systems used today, and by new technologies such as XML. We
can also, simultaneously, get wider input on our draft object by sending
it to selected parties for comment.

We can not produce a time-series object that does all that anyone might
want. Each potential user may have some favorite requirements to be
fulfilled. As a way to proceed, I\'m going to summarize what I think a
time-series object looks like based on the material below from Joe King
and Dave Sibeck. Then, I\'m going to list some questions that a
time-series object might be expected to be able to answer. In other
words, from an application point of view, what inputs should I be able
to give it and what type of results should I expect to be able to get
back? These will necessarily reflect my prejudices. Others will have
additional ones, and may think some of mine are not needed. We may
arrive at what we would call a minimum set of services, and one or more
additional sets of services. A real standardization effort would flesh
this out more thoroughly.

**Time-Series Object Summary- Initial Conceptual View**

(note the word \'conceptual\' - this is not an implementation)

1\. A Time-Series Object appears (conceptually) as a sequence of
records.

2\. Each record contains a key date/time value, which is monotonically
increasing or decreasing from record to record.

3\. Each record also contains other data, of arbitrary complexity. This
data can be viewed as a sequence of fields, but may also be viewed as
groups of fields which map to various types of objects such as images,
vectors, matrices, etc. It is not necessary to call out all detailed
structure, so that a whole image may be viewed as one field if so
desired.

4\. Each record also contains one or more \'processing intervals\'. Each
such processing interval gives the time span over which one or more of
the other data fields has been determined. For example, a data field may
be a 1 minute average of other values, in which case the \'processing
interval\' would be 1 minute. It may, of course, be zero as well. The
\'processing interval\' is known for all data fields, identified as
such, in the time-series object.

5\. Each record also contains one or more \'time offset\' values. Each
data field has a time offset value which relates, by the use of
addition, its time of observation/determination to the key date/time
value.

6\. The relationship of the key date/time to all other fields, including
each processing interval field, is known because either it is given
explicitly or it is a part of the standard. Possible relationships
include:

\- For each subset of the data fields, the key date/time, after being
augmented with its relevant offset value, corresponds to the beginning,
the middle, or the end of the associated processing interval value.
Figure 1 below is an example where the offset is to the middle of the
processing interval for data fields \'x\' and \'y\', but to the
beginning of the processing interval for data field \'z\'.

\- (there could be others as well)

7\. The meaning of each field, including key date/time, processing
intervals, time offsets, and data is known and described.

**Questions that might be supported:**

(which may suggest some enhancements to the conceptual view above)

1\. What is the meaning of the key date/time and how is it represented
(integer, multiple fields, etc.)?

2\. What is the start date/time; what is the end date/time?

3\. What is the maximum record size in bytes?

4\. How many records in the object?

5\. How many data fields are present in the object?

6\. What are the data fields available? (Give names)

7\. What is the definition of data field \'x\'.

8\. What is the representation type for data field \'x\' (e.g., integer,

real, image array, character string)

9\. What are the units for data field \'x\'?

10\. What is the \'processing interval\' for data field \'x\'?

11\. What is the offset time for data field \'x\'?

12\. How does the offset time relate to the \'processing

interval\'? (e.g., \'mid point\')

13\. How many records exist between time a and time b?

14\. How many bytes exist between time a and time b?

15\. Get me the records between time a and time b.

16\. Get me the values for fields x, y, and z for their times between a
and b.

17\. Get me the values for fields x, y, and z for the key date/time
between time a and time b.

**====== Following is Joe\'s e-mail on time series object =======**

At Don\'s urging, I\'m forwarding to the whole FEPC the

characterization of \"time series\" as developed to date.

Note that we started with the perception that \"time series\"

was a data object type, more or less as images or other

constructs were also data object types.

However, we have come around to the perception that while

there are a number of issues associated with the time tagging of

data in science data records, virtually all other \"data object types\"

may occur in time-tagged sequential records. \"Time series\" as

one of several more or less mutually exclusive data object types

has disappeared.

Joe

Date: Fri, 11 Feb 2000 14:50:07 -0500

From: \"Joseph H. King\" \<king@ndadsb-f.gsfc.nasa.gov\>

Reply-To: king@ndadsb-f.gsfc.nasa.gov

To: Dave Sibeck \<david.sibeck@jhuapl.edu\>

CC: ad@srl.caltech.edu,

\"Donald M. Sawyer\" \<donald.sawyer@gsfc.nasa.gov\>,

king@ndadsb-f.gsfc.nasa.gov

Subject: Re: test for time tags

Thnaks, Dave. I found your additional last sentence at the very

end, and a couple of small grammatical. I did not find anything

reflective of the ground magnetometer data format, although we have

the case covered in the text. Ed Smith used to build data records this

way also, with a time tag for an hour and then one 1-hr avg and 60 1-min

values.

I threw in the example of the sequence of images to make the point

that \"data object type\" = time series is not necessarily mutually

exclusive from \"data object type\" = image. Maybe the key point

is that \"time series\" is not an \"object type\" like everything else.

Maybe the point is we have images, and numbers of various types

(scalars, vectors, and various kinds of arrays), and any combination

of these can be included in a time ordered and time tagged series of

records.

Maybe our most fundamental \"data object types\" are images and field

vectors and particle distribution functions and moments (derived from

distribution functions) and event data (the multiple delta-E\'s in a

multi-element telescope) and electric/magnetic wave spectra and ???,

and it is these that need to be characterized as to the varieties they
come in

and how they need to be supported.

And we get away from trying to characterize \"time series.\" That\'s not

to say that somewhere all the issues about time tagging identified below

don\'t need to be addressed.

Joe

Dave Sibeck wrote:

\> Dear Joe,

\> I agreed with everything you wrote, but nevertheless made various

\> small changes which you may catch if you look closely!

\> I had in my mind the standard ground magnetometer data format:

\>

\> location, start hour, 60 values at successive minutes, hourly
average.

\>

\> I made some other small changes too.

\>

\> best wishes

\> david

\>

\>
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\--

\> Subject: start of \"time series\" characterization for FEP

\>

\> What is a time series, and to what extent can and should \"time
series\" be

\> considered data structures?

\>

\> A time series is represented by a series of data records, and the
data

\> records may be grouped into files or other \"computer science
objects\" in

\> various ways; these objects may contain descriptive parts in addition
to

\> the actual data records, and/or pointers to associated material
outside the

\> object.

\>

\> A record consists a series of words in binary and/or ASCII
representation.

\> A key element is that a subset of the words convey a time point (time
tag,

\> time stamp). Some additional words convey values of measurements or
of

\> parameters derived from measurements, while yet other words are
sometimes

\> included to convey information about the conditions of the
measurements. A

\> most obvious example of the latter is the specification of the
location

\> where an \"in situ\" measurement is made, or a telescope pointing
direction

\> for remotely sensed observations. Successive records have continually
(but

\> not necessarily uniformly) increasing time stamps.

\>

\> There are some standards for the representation of time tags (e.g.,
IEEE

\> standard =(?) yyyymmddThhmmss.sss) but addressing such standards is
not the

\> point of this note.

\>

\> There are two key elements to be addressed next: how does the time
tag

\> of a record relate to the other words of the record (measurements,

\> measurement-derived parameters, or measurement conditions), and what
more

\> can be said about these other (non-time) words in a time series
record.

\>

\> Any given non-time word in the record may be instantaneously measured
or

\> otherwise determined. In this case, a time point is to be related to
it.

\> The time point may be the record\'s time tag, or may be offset from
it by

\> some amount determinable from the content of the record (e.g., actual
time-

\> offset values) or from accompanying metadata. The offset for multiple

\> instantaneously measured/determined words may be common to all words
or may

\> vary with the words. Again record content or accompanying metadata
needs

\> to make this clear. Fortunately, in (virtually) all cases, the same

\> offsets apply for each word in each record of a file or collection

\> thereof.

\>

\> (\"Instantaneous\" implies measurement intervals short relative to
spacing

\> between successive records and also short relative to time scales for

\> processes to be studied with the data; \"Instantaneous durations\" of

\> measurements should be specified in accompanying metadata.)

\>

\> In many cases, the non-time words in the record are not
instantaneously

\> measured or otherwise determined. In some cases words represent
finite-

\> time-span averages, and in other cases words represent moments or
other

\> parameters derived from distribution functions accumulated over some
finite

\> time. For averages, it should be specified whether the time tag
represents

\> the start point, midpoint, endpoint (or other) of the averaging
interval.

\> The same is true for time tags assigned to parameters derived from
finite-

\> accumulation-interval distribution functions.

\>

\> In addition to instantaneous measurements, averages, and parameters
derived

\> from finite-accumulation-interval distribution functions, on rare
occasions

\> other values are specified. An example is that a parameter value in
the

\> records of AMPTE/CCE 6-sec records is actually the median of the

\> instantaneous values observed during that 6-sec interval. Clearly any
such

\> unusual usages must be carefully specified in accompanying metadata.

\>

\> Relative to the nature of the data in a time series record, SEC
typically

\> thinks of in situ data, where the measurements, parameters derived
from the

\> measurements, or conditions describing the measurements may be

\> intrinsically scalars, vectors, or higher-dimensional arrays. But
also a

\> time sequence of images, e.g, of auroral evolution, can also be
thought to

\> represent a time series. Since a finite time is required to construct
such images, metadata should ideally provide information allowing users
to determine start and end times, as well as the sequence by which the
image is constructed.