Thoughts about\
levels of complexity and specialization in\
data types and structures\
for storing scientific data

Mike Folk, Don Sawyer, Lou Reich and the FEPC

Rev 1: Revised A 2000

Changes:

1.  Additions marked in blue-underline. Deletions in red strikethrough.
    Some additions and deletions are from rev 9, left in because we\'re
    still talking about them.

2.  Significant rewrite of 3.1 and 2 into one (almost) section

3.  Mike redid figures 2(a)-2(e) and moved them into an appendix.

4.  Mike changed Fig 1 into Fig1a and Fig 1b, and added some text. Still
    needs help from Lou.

5.  Mike started work on Section 4.4, which is a listing of simple
    general science types. I\'m using the SI standard quantities as a
    starting point for this. Not sure it will work, but it\'s a start.

6.  Lou. Revision 14 has finished the restructuring most of the material
    in sections 2 and 3. I am leaving the strikeouts from revisions 9-13
    as a historical record and so the authors can see the original
    material. However, to read the document I strongly suggest turning
    strikeout off. I may have gone overboard on my rewriting but I think
    the document is now at the point that we could use it to develop a
    checklist for format selection. I did not address section 4 on the
    how to move around the I am not sure it is still needed and if it is
    needed, I would probably divide it between section 2 and 4

7.  changes to give paper bet

A number of different perspectives can be used to understand the
scientific data formats evolution process.

In this paper, .we provide and overview of scientific data and data
formats and investigate the objects used by scientific data formats from
two

1.  the range and complexity of the object types that can be represented
    in the formats, and

2.  degree to which a format supports general-purpose data objects
    versus data objects that are specific to a particular scientific
    discipline.

We seek to identify common objects and formats that span these two
perspectives, and to describe how they are interrelated. The objective
of the paper is to present a model for understanding different formats
and assessing their utility for a variety of different activities that
are performed on scientific data.

We then look at the concepts of formats from these perspectives.In the
final sections of this paper we present a comprehensive list and
definitions provide a common vocabulary for further discussions in this
area.

# ns

Scientific data is based on tend to be numeric. In order to derive the
scientific information from this data one must understand several
factors such as: the location (space and time), the units of the
measurements and the relationships among multiple measurements. The
addition of these metadata items transforms the measurement data into
scientific information that d and analyzed.

Some metadata is used to aid in interpreting the computational meaning
of data. This is called \"structural metadata\" because it describes the
atomic, simple aggregate, and complex aggregate computing structures
used to represent objects. Structural metadata describes how the are
formatted, and how to interpret them.

For instance, structural metadata tells us that a particular collection
of 4 bytes represents an \"IEEE float.\" Structural metadata also
describe the relationships among objects, and hence is used to describe
complex objects in terms of simpler ones. For instance, structural
metadata can indicate how a certain collection of integers constitutes a
3-dimensional array and that they are stored in row-major order. Or how
a certain collection of arrays and other structures represent a
multi-resolution grid.

Another form of metadata, \"content metadata," transforms strictly CS
datatypes into physical datatypes. Content metadata gives physical
meaning to data by describing its relationships to the real world.
Content metadata is can be varied and vast, and includes physical units,
locations in space and time, physical conditions and constraints, and
instrument information

A core requirement for any tool that handles scientific data is that is
be able to associate both content and structural metadata with numeric
data.

# 2. A look at formats {#a-look-at-formats .unnumbered}

We define a format as follows: a format specifies the organization of
information, at some level of abstraction, in one or more byte streams
that can be exchanged between systems. In this document we focus on
formats designed for scientific data exchange access,rchiving. We
exclude discussion other types of containers, such as relational
databases, although data types that are describe often apply equally
well to databases.

Many scientific data formats are specified with a paper document
describing the meanings, layout, and relationships of objects that the
format supports. These specifications can be abstracted to provide a
less detailed, and more easily understood, view of the supported objects
and relationships. This is often referred to as the *data model* of the
format. A high level of abstraction provides little detail. At the
lowest level of abstraction that data model becomes equivalent to the
format specification. GRIB and FITS, for example, specify objects down
to the bit level. However not all format specifications go to the bit
level. It may be left to a separate specification, or implementation, to
provide this needed level of detail. All these formats require
applications to read and write the data to enable automated processing.
documents are written in a Data Definition Language. There are a wide
variety ranging from Englishted with some diagrams to formal
specification documents, whicha wide variety of object types, layouts,
and relationships to be described The General purpose formats of this
type have different degrees of automated support. Some are simply
documentation and offer no support tools, while other provide tools to
assist in reading and writing descriptions of specific data sets. Some
formats may have readers and writers for converting data from local
storage forms to/from the general format form.

The EAST data description language can describe a sequence of repeating
records where part of the record structure can vary from record to
record, and where a wide variety of standard and custom data types are
supported at the bit level. The PDS ODL and FITS are hybrid DDLs where
binary data objects are delimited (i.e., encapsulaed or referenced) and
describedls. IDFS usepart of its XML has recently emerged as a
general-purpose markup languagethat, in combination with appro Type
Definitions (DTD) can describe almost any character based format.

An alternative paradigm for viewing scientific information is the Object
Oriented or Application Program Interface (API) approach. In this
approach the actual structure of the underlying data is hidden from the
data consumer and scientific information can be gained through a series
of function calls (i.e. methods).

For example, the CDF format has an API for storing and accessing arrays,
and there is an implementation of this API so that the API can be called
by software to access data in CDF. This adds another abstraction level
to a format,in which an application does not have to be concerned with
how the bits are actually organized. It also means that a format, as
defined here, does not have to specify unambiguously a particular
organization for the bits. HDF5, for example, can store its objects in a
number of ways, and indeed provides facilities for applications to
replace its default layouts with ones of their own choosing

There is a good deal of overlap between Data Structure and API formats,
and indeed it is common to use a single format for exchange, access, and
sometimes for archiving as well. Also, it is becoming common to add an
API component to a frmt and develop a DDL to rpresent the
interchange/archival form or a schema specification for an API format.
Examples of this include a C++/C API developed for EAST and hopment
Regardless of whether a format has a DDL specification, an API, or both,
it will have one or more associated data models that are in common use
to understand its objects and their relationships. For example, a
relational database has an abstraction, or data model, of a set of
related tables.

In the following section we investigates the types of objects that are
commonly used in scientific data models.

# Scientific Format Data Object

The data model for a scientific format can be characterized by:

-   hian be associated ata va

This section used the computer science concepts of aggregation and
specialization to discuss these characteristi

#  {#section .unnumbered}

look at how scientific formats organize and support information, we find
they employ a variety of data objects intended to support many kinds of
information. For example CDF employs integers, reals, characters,
arrays, and attributes to support any kind of information that can be
usefully characterized in these ways. Some objects, such as integers,
are very simple, and others, like arrays, are more complex. With formats
like CDF, HDF and others, even more complex objects can be built from
these, such as finite element meshes. And some formats, such as DX,
explicitly support such complex objects.

These distinctions provide one useful way to understand and compare
formats: by the types and complexity of the objects that they support.
Table 1 describes three \'levels of complexity\' for data objects, and
illustrates formats that support them. Read from bottom to top, the
three rows represent increasing levels of complexity, with objects at
each higher level often being built out of those at lower levels.

Table 1. Levels of aggregation of objects for different formats.

  ----------------- ---------------------------- ----------------------------
  **Levels of       **Objects**                  **Example formats that
  complexity                                     support some of the features
  (aggregation)**                                or objects**[^1]

  \(3\) Complex     Raster image, irregular      DX
  aggregates        grid, multi-resolution grid, 
                    index, record of aggregates  
                    and simple types             

  \(2\) Simple      Character string, record of  HDF4/5 Scientific Data Sets,
  aggregates        simple types, array          CDF, netCDF, PDS, FITS\
                                                 (plus all of the above)

  \(1\) Simple      Number (integer, real),      IEEE float, XDR,
  types/bit layouts character, compressed value  ASCII-encoded number (plus
                                                 all of the above)
  ----------------- ---------------------------- ----------------------------

The entries in the \"objects\" column should be interpreted as
abstractions\--this column says nothing about how objects and features
are implemented in a particular format. The third column, \"example
formats...\" refers to actual implementations of these objects.

In object-orient terms, Table 1 describes three levels of *aggregation.*
The more complex objects tend to be built from the simpler ones. Each
level provides information structures that can be used as building
blocks to create structures at higher levels. For instance, integers can
be combined to provide an array structure, which in turn can be used to
create an unstructured grid. (Note: we use the term \'structure\' here
to refer to the way in which an object is implemented.)

\<\<Mike I cut this figure out of a paper by Lloyd Tranish. Both the
figure and the words need modification but I think it adds some clarity
about complex aggregations\>\>

Figure 6 shows sample taxonomies for the aggregation layer, which
describes classes of data that can be broken down into simple fields.
They may range from a series, which is merely a sequence of instances of
a field to cases which can be spatially decomposed into sets of
subfields such as a multizone grid or a collection of spatial partitions
for parallel processing. Adding hierarchy enables a description of an
adaptive mesh. Some data may be sampled over a very topologically
complex mesh. In this case, it may be easier to decompose it into a set
of simpler meshes, each of which is of the same class.

![](media/image1.png){width="4.2in" height="3.9in"}

The structures, semantics and operations of aggregated objects are
implemented using the structures, semantics and operations of the
simpler objects. The aggregated object appears to the user to have its
own interface and has no responsibility to expose the interfaces of the
contained objects. An aggregated object may choose to expose any of
underlying objects

the measurements and associated metadata into known structure/aggregated
type can summarize many of the relationships among various measurements
and metadata. For example a series of soil measurements could be
represented as a table where each row of the table consisted of the
lat/long coordinates, the time of measurement and the measurement value
including units.

This structuring process aids the consumer of the data in the
understanding and visualizing the data. In some cases more complex
mathematical types (which are aggregates of aggregates) are used to
represent complex

As the table suggests, the level of complexity of basic types, can be
related to format. Formats such as XDR and the IEEE float standard
support simple objects, such as standard number representations and
characters, whereas the DX irregular grid is a complex object requiring
several different structures and a great deal of metadata,

## Scientific domain specialization 

Table 1 provides no information about how we extract *scientific*
information from any particular format. To use formats for scientific
purposes, their objects, and the structures that implement them, have to
be assigned scientific meaning. Some formats specify no specific
scientific information, yet they provide useful computing datatypes and
structures for storing scientific data, such as floating point numbers
and multidimensional arrays. Others may specify very general types of
scientific information, such as \"a time sequence of values,\" and still
others support very specific types of information, such as \"magnetic
field observations by the IMP-8 magnetometer.\"

The description of specialized objects in terms of more general ones is
called *specialization* in object terminology. Specialization occurs
when a new *subclass* is created from a more general *superclass* by
constraining the features of the superclass in some way, and/or by
extending the superclass by adding new features. This concept provides
us with another useful criterion for understanding and comparing
formats. Table 2 describes several \'levels of specialization \'for
scientific data objects, and illustrates formats that support them.

Table 2. Levels of specialization of objects supported by different
formats.

+-------------+---------------------------+---------------------------+
| **Levels of | **Features/\              | **Example formats that    |
| speci       | objects**                 | support some of the       |
| alization** |                           | features or objects**[^2] |
+-------------+---------------------------+---------------------------+
| \(d\)       | Instrument X swath        | HDF-EOS MODIS Swath       |
| I           |                           |                           |
| nstrument/\ | Instrument X magnetic     | IMP-8 Magnetometer        |
| applicati   | field                     |                           |
| on-specific |                           |                           |
+-------------+---------------------------+---------------------------+
| \(c\)       | EOS swath                 | HDF-EOS                   |
| Doma        |                           |                           |
| in-specific | Electric and magnetic     | ISTP-CDF                  |
|             | fields                    |                           |
|             |                           |                           |
|             | Something from PDS.       |                           |
+-------------+---------------------------+---------------------------+
| \(b\)       | Observed image, value at  | CDF[^3], PDS,             |
| General     | a location, date/time,    | FlatfileError: Reference  |
| science     | time series               | source not found, FITS    |
+-------------+---------------------------+---------------------------+
| \(a\)       | Number (integer, real),   | DX, HDF, CDF, netCDF,     |
| computer    | record, array, field,     | PDS,\                     |
| science     | raster image,             | IEEE float, XDR,          |
|             | finite-element (FE) mesh, | ASCII-encoded number,     |
|             | vector, tensor, multi-res | ASN.1, XML                |
|             | grid, mesh, index         |                           |
+-------------+---------------------------+---------------------------+

In table 2, the bottom level represents the same math and computer
science datatypes and structures described in Table 1. Formats that
support only these structures can be made to contain useful scientific
information if appropriate documentation, metadata, or interpretive
software accompanies them. There are computer science data formats such
as ASN.1, XDR and XML that support this level of specialization.

Levels (b)-(d) in Table 2 represent increasingly specialized science
datatypes and formats. One crosses the threshold into \"scientific\"
formats when one or more physical dimensions required by the format
itself. This criterion insures that there is a physical interpretation
of the information supported by the format. This may be as simple as a
format providing an ability to specify units for numeric data. Another
example is, an \"observed image\" format that requires earth based
coordinates is a scientific format, whereas a pure image format such GIF
is not.

The \"general science\" level in table 2 introduces concepts such as
time and space, which are common to nearly all sciences. At this level,
spatial and temporal dimensions might be represented without reference
to any specific coordinate systems, so they could be applicable to any
physical science domain. Time and space frequently occur as independent
variables in data collections. Similarly, physical properties such as
temperature and could be represented as having the choice of many
different units. For example, density could be any relationship between
a volume and the number of individual items of some type found within a
unit of that volume. Most of the well-known science data formats such as
HDF, CDF,s, FITS and IDFS support a minimum of this level ociated meta.

Levels (c) and (d) represent project- or discipline-level scientific
formats. They encompass increasingly specific, narrow, scientific
applications, instruments, etc. Domain-specific formats contain a great
deal of contextual information, but provide some generality, so that
data from different instruments, experiments, simulations, etc, can be
encapsulated in the same format. For example in space physics density
becomes number of charged particles/cubic meter. HDF-EOS and ISTP-CDF
are examples of domain-specific formats in that each is designed to be
used for storing data generated by a variety of different instruments.

Formats at the top level, such as the IMP-8 magnetic field format and
the HDF-EOS MODIS Swath, are specific to an instrument or other
application. For example \<\<\<Need Specifics about
IMP8Magnetometr\>\>\>\>\>

The choice of exactly four levels is somewhat arbitrary, but believed to
be useful. The important point is that each level can play a role, and
that the use of levels can have conceptual, engineering, and other
benefits.

A big advantage of viewing formats in terms of domain specialization is
that objects at each level can be built out of those on the lower
levels. In object terms, a more specialized object can inherit the
semantics and methods of the more general objects. This inheritance can
have enormous advantages in terms of software engineering, because it
may permit the higher levels to re-use methods at the lower levels.

## Combining complexity and domain-specialization

We have described two ways to distinguish among scientific formats: by
level of aggregation and by level of specialization. Both views are
useful in understanding the evolution and use of formats, and there is
an interesting interplay between the two views, in Table 3

+---+---------+------------+-------------+-------------+---------------+
|   |         | **Specia   |             |             |               |
|   |         | lization** |             |             |               |
+---+---------+------------+-------------+-------------+---------------+
|   |         | **(a) Math | **(b)       | **(c)       | **(d)         |
|   |         | & Computer | General     | Domain      | Instrument/\  |
|   |         | science**  | science**   | -specific** | a             |
|   |         |            |             |             | pp-specific** |
+---+---------+------------+-------------+-------------+---------------+
| > | **3.    | Irregular  | Wa          | Wa          | Wavelet-based |
|   | Complex | grid,      | velet-based | velet-based | image from    |
| * | aggr    | multi-     | image       | astro       | instrument    |
| * | egate** | resolution |             | image, FE   | X,\           |
| A |         | grid, quad |             | mesh of     | FE mesh for   |
| g |         | trees      |             | materials   | material X    |
| g |         |            |             |             |               |
| r |         |            |             |             |               |
| e |         |            |             |             |               |
| g |         |            |             |             |               |
| a |         |            |             |             |               |
| t |         |            |             |             |               |
| i |         |            |             |             |               |
| o |         |            |             |             |               |
| n |         |            |             |             |               |
| * |         |            |             |             |               |
| * |         |            |             |             |               |
+---+---------+------------+-------------+-------------+---------------+
|   | **2.    | Character  | date/time,  | Geospatial  | Coordinates   |
|   | Simple  | string,    | spatial     | c           | of instrument |
|   | aggr    | record,    | c           | oordinates, | X             |
|   | egate** | array,     | oordinates, | astro       |               |
|   |         | raster     | observed    | raster      |               |
|   |         | image      | image,      | image       |               |
+---+---------+------------+-------------+-------------+---------------+
|   | **1.    | Integer,   | Density     | Solar wind  | Imp-8 solar   |
|   | Simple  | real, bit  |             | density     | wind density  |
|   | types** | string,    |             |             |               |
|   |         | character  |             |             |               |
+---+---------+------------+-------------+-------------+---------------+

##  {#section-1 .unnumbered}

Many formats span the levels of aggregation in Table 1, at least
partially. CDF, netCDF, FITS and HDF all support simple datatypes as
well as certain aggregates, although they don't all support exactly the
same simple types and aggregates. The DX format supports complex
aggregates such as irregular grids. Even though formats like HDF and CDF
do not explicitly include third level objects, complex aggregates, it is
not uncommon for users to create high-level profiles that include these.

Some of these same formats also span different levels of specialization
(Table 2). CDF, FITS and PDS, for instance, are designed to be useful
general-purpose formats, but at the same time provide a certain
scientific context. CDF includes a standard time datatype (\"epoch\"), a
general scientific concept. Similarly, the FITS specification provides
datatypes and aggregates that can be organized to apply to a variety of
different application domains, but it also contains definitions that are
very specific to astronomy.

We\'ll now look at formats in terms of how they span the cells in Table
3, which combines the two dimensions of aggregation and specialization.
Figure 1a is a direct translation of Table 3 onto a graph describing
levels of complexity and levels of specialization. A given format can be
described in terms of what sectors on the graph describe its datatypes
and structures. This is illustrated in Fig. 1b, where HDF4 and HDF-EOS
are described. In this example, HDF4 is seen to contain math and CS
types only, whereas HDF-EOS supports simple and complex aggregates in
the general domain of earth science.

![](media/image2.png){width="4.645138888888889in"
height="3.482638888888889in"}

![](media/image3.png){width="4.645138888888889in"
height="3.482638888888889in"}

#  ormat Cu- Profiles {#ormat-cu--profiles .unnumbered}

Many formats are designed specifically for a particular science or
application, others are formats designed for a wide variety of uses, and
others are somewhere in between. The formats listed at level (a) in
Table 2 (\"computer science\") are general purpose formats. Table 2
lists several formats, such as HDF-EOS and ISTP-CDF, that are
implemented entirely from objects in these general formats. HDF-EOS is
implemented using the HDF format, and ISTP-CDF using CDF. formats like
HDF-EOS and ISTP-CDF *profiles*. We can yze the changes to data model
required by a new profile using the techniques discussed in the previous
section.

ISTP - CDF is an example of a profile that is created by specializing
the objects in the general CDF format. In this case consist of special
rules and guidelines that constrain the use of a more general format,
such as what metadata fields and values to use, and which
format-supported data types and structures to use in what circumstances.
There are no new Object types or aggregates introduced so the only
difference between the diagrammatation of CDF and ISTP-CDF is the fact
that the level of domain specification has increased from the computer
science area to the d

Due to the fact that only specialization was used, any software that can
visualize or analyze CDF formatted data will return accurate results
when operating of ISTP CDF data.

**\<This figure is used for fill, need a CDF/ISTP_CDF Figure\>**

![](media/image4.png){width="4.664583333333334in"
height="3.4972222222222222in"}

HDF-EOS is an example of a profile derived by both specialization and
aggregation. In addition to specifying rules and guidelines to constrain
the more general HDF format, the HDF-EOS profile specifies three
earth-science datatypes: a swath, a grid, and a set of geo-referenced
points. All three of these datatypes are aggregate structures,
implemented by combining HDF arrays, groups, and attributes[^4].
Profiling by extension adds semantics to existing object types or
combinations of object types in essence creating new object types. Since
an object in such a profile may be constructed out of more than one
object in the more general format, profiles may contain more complex
objects than those of the formats from which they are derived. 3 by the
shift in the specialization) axis and in thSince HDF-EOS introduces new
object types and associated methods in addition to those specified in
HDF, any datasets using those types could not be correctly/completely
processed by software that implemented the HDF4 APIs.

#  {#section-2 .unnumbered}

# Guidelines on Profile {#guidelines-on-profile .unnumbered}

A format that facilitates building profiles typically consists of a set
of general objects, a mechanism a specification of the layouts for those
objects, with the idea that those objects could be configured and
described in different ways to apply to different applications. In a
sense, this type of format can be thought of as supporting a family of
customized formats.

Some general purpose formats also provide libraries with features that
are generally useful, starting with the read, write and convert
primitive types and aggregates, and including such services as data
compression, tiling, acalternate storage formats, and parallel I/O.
Furthermore, general-pu are often implemented on a variety of machines

As \"formats on top of formats,\" profiles can combine the context of a
new abstract domain with the features and services of a domain-free
format. This approach lets general format developers concentrate their
efforts on providing features at one level that can be exploited at
higher levels. By using HDF, for instance, the HDF-EOS format is able to
exploit such features as data compression, tiling, and machine
portability without having to re-invent, implement, and maintain them in
a new format. automatically supports the concept ofger and more complex,
requiring greater software development and maintenance costs. If a
general format is used to support many different profiles in different
disciplines, these costs are spread out and can ultimately result in
cost savings, but this requires coordination among a variety of
participantsAlthough higher level objects frequently are implemented in
terms of one or more lower level objects, a format (and API) supporting
a higher level object may or may not make lower level objects
independently available for the user. interface and avoid confusion. For
instance the HDF-EOS API does not have objects like palettes and images

that the general-purpose format will cease to support some of the
required interfaces in future releases. In this case the discipline will
be forced to provide resources to maintain the profile libraries and
software configuration and interoperability with other users of the same
general-purpose format will degrade over time. Another problem is that
the process of creating extensions to the general-purpose formats may
find hidden problems with the implementation of underlying datatypes.
The resources required to fix these problems could be very high and the
provider of the general-purpose format may not be willing to provide
those resources.

\<\<\<\<\<\<End of Former Chapters 1-3\>\>\>\>\>\>

# ISO/IEC 11404 datatypes

\<\<need to revise in view of recent changes\>\>\
To improve our ability to compare formats and to think about the
evolution of formats, it would be useful to be more explicit about the
types of objects that fall into the various levels of complexity and
domain-specificity. For level 1a and 1b types we draw from the ISO/IEC
standard (ISO/IEC 11404: 1996 (E)) for \"Language-independent
datatypes,\" which provides a rich and rigorously defined set of
datatypes for programming languages that maps very well to format
datatypes at this level. Appendix A contains a listing of most of the
ISO 11404 datatypes.

In the next two sections, the appropriate ISO 11404 datatypes are mapped
to the format levels that were defined above. Some format types are
listed that are not included in the ISO 11404 standard, such as
*attribute* and *group*.

The ISO 11404 standard does not deal with how these datatypes are
*represented*, since representation is a language-specific feature. With
formats, as with languages, representation can be very important,
however. \<\<But we have to figure out how best to cover this
material.\>\>

## Level 1(a): simple types/bit layout

Level 1a consists of basic types that are used to represent individual
values, and as elements for aggregate and other higher level types.
Table 2 lists the ISO 11404 types that seem to represent level 1[^5].

  ------------ ---------------------------- ---------------------------- ---------
  **Type**     **Rough definition**         **Examples/comments**        

  Boolean      Math type whose values are   E.g. \"true\", \"false\"     8.1.1
               \"true\" and \"false                                      

  Enumerated   Finite collection of         E.g. \"orange\", \"apple\",  8.1.3
               distinguished values having  \"pear\", where              
               an intrinsic order           orange\<apple\<pear          

  Character    Family of datatypes whose    ASCII most common, but other 8.1.4
               value spaces are             character sets may need to   
               character-sets               be considered.               

  Integer      Math type comprising exact   Many precisions supported.   8.1.7
               integral values              Sometimes signed and         
                                            unsigned are distinguished.  
                                            (Should these be             
                                            enumerated?)                 

  Scaled       Value space is a subset of   Used by some formats to save 8.1.9
               the rationals, each with a   space.                       
               fixed denominator, but                                    
               possessing the concept of                                 
               approximate value.                                        

  Real         Approximations of the math   Size and precision can be    8.1.10
               type \"real numbers.\"       important in scientific      
                                            applications. (Should these  
                                            be enumerated?)              

  Complex      Approximations of the math   Often not a primitive type,  8.1.11
               type \"complex numbers.\"    but represented by a         
                                            predefined convention.       

  Void         Represents an object whose   Perhaps this is the same as  8.1.12
               presence is required, but    a fill-value.\"              
               carries no information.                                   

  Pointer      Values constitute a means of There are a wide variety of  8.3.2
               reference to values of       ways to define and implement 
               another datatype.            pointers. It may be useful   
                                            to delineate some of these.  

  Bit          Represents the finite field                               10.1.3
               of two symbols \"0\" and                                  
               \"1\".                                                    

  Bit string   Datatype of variable length                               10.1.4
               strings of binary digits.                                 

  Octet string Datatype of variable length                               10.1.8
               encodings of 8-bit codes.                                 

  Private\     An application-defined value Makes it possible for a      10.1.9
  AKA \"blob\" space and operation set that non-standard type to be      
               are intentionally concealed  passed between two parties   
               from certain processing      that understand it by a      
               entities. There is no        third party that does not    
               denotation for a value of a  understand it. (See appendix 
               private datatype.            for more details.)           

  Object       Values that uniquely         See ISO/IEC 11404 gives a    10.1.10
  identifier   identify objects in a        precise and thorough         
               communications protocol.     definition.                  

  Optional     A generator that effectively Level 1a when it represents  10.2.4
               adds the \"nil\" value to    a scalar value, but level 1b 
               the value space of a base    when it represents a complex 
               datatype.                    structure.                   

                                                                         
  ------------ ---------------------------- ---------------------------- ---------

## Level 2(a): simple aggregates

The standard defines an *aggregate datatype* as \"a generated datatype
each of whose values is, in principle made up of values of the component
datatypes.\" We use the term \"simple aggregates\" to distinguish these
from more complex aggregates such as multi-resolution grids.

+-------+----------------------------+----------------------------+----+
| **T   | **Rough definition**       | **Examples/comments**      |    |
| ype** |                            |                            |    |
+-------+----------------------------+----------------------------+----+
| R     | Heterogeneous aggregation  | There are often            | 8  |
| ecord | of values of component     | restrictions on how        | .4 |
|       | datatypes, each            | complex the fields of a    | .1 |
|       | aggregation having one     | record can be, but in      |    |
|       | value for each component   | principle a field can be   |    |
|       | datatype, keyed by a fixed | any datatype. In general,  |    |
|       | \"field identifier.\"      | fields do not have to      |    |
|       |                            | occur in any particular    |    |
|       |                            | order, but many formats    |    |
|       |                            | require a fixed order.     |    |
+-------+----------------------------+----------------------------+----+
| Set   | Value space is the set of  | What is the utility of     | 8  |
|       | all subsets of the value   | this for scientific data?  | .4 |
|       | space of the element       |                            | .2 |
|       | datatype, with operations  |                            |    |
|       | appropriate to the         |                            |    |
|       | mathematical *set*.        |                            |    |
+-------+----------------------------+----------------------------+----+
| Bag   | Collection of instance of  | This probably corresponds  | 8  |
|       | values from the element    | to the HDF5 grouping       | .4 |
|       | datatype. Multiple         | structure.                 | .3 |
|       | instance of the same value |                            |    |
|       | may occur, and ordering of |                            |    |
|       | elements is not            |                            |    |
|       | significant.               |                            |    |
+-------+----------------------------+----------------------------+----+
| Seq   | Ordered sequence of values | Differs from *bag* in that | 8  |
| uence | from the element datatype. | the ordering is            | .4 |
|       | The ordering is imposed on | significant. Sequences of  | .4 |
|       | the values and not         | scientific records (e.g.   |    |
|       | intrinsic in the           | observations) are          |    |
|       | underlying datatype. The   | frequently implemented     |    |
|       | same value may occur more  | using this datatype.\      |    |
|       | than once in a given       | Is this the same thing as  |    |
|       | sequence. The only way to  | an array with an unlimited |    |
|       | write to a sequence is to  | dimension?                 |    |
|       | append; there\'s no        |                            |    |
|       | replace operation.         | Is this the same as our    |    |
|       |                            | general \"variable length  |    |
|       |                            | type.\"                    |    |
+-------+----------------------------+----------------------------+----+
| Array | Values are associations    | Index datatypes correspond | 8  |
|       | between the product space  | to dimensions. Some        | .4 |
|       | of one or more finite      | formats allow dimensions   | .5 |
|       | datatypes, designated the  | to grow in size after      |    |
|       | *index datatype*, and the  | creation, and as a result  |    |
|       | value space of the         | they are sometimes treated |    |
|       | *element* datatype, such   | like a *sequence*. In      |    |
|       | that every value in the    | files, elements are        |    |
|       | product space of the index | generally stored           |    |
|       | datatypes associates to    | contiguously, but other    |    |
|       | exactly one value of the   | organizations are          |    |
|       | element datatype.          | possible.                  |    |
+-------+----------------------------+----------------------------+----+
| Table | Unordered collection of    | Note that the ISO          | 8  |
|       | values in the product      | definition specifies       | .4 |
|       | space or one or more       | \"unordered.\" This        | .6 |
|       | *field* datatypes, such    | definition is meant to     |    |
|       | that each value in the     | produce a datatype that    |    |
|       | product space represents   | corresponds to the SQL     |    |
|       | an association among the   | concept of \"table.\"      |    |
|       | values of its fields.      |                            |    |
+-------+----------------------------+----------------------------+----+
| De    | A datatype defined by a    | This makes it possible to  | 8. |
| fined | type-declaration, denoted  | rename an existing         | 4. |
| dat   | by a type-reference.       | datatype or name an        |    |
| atype |                            | existing datatype that     |    |
|       |                            | have a complex syntax.     |    |
|       |                            | This can be useful for     |    |
|       |                            | applications that want to  |    |
|       |                            | store complex datatypes by |    |
|       |                            | name in a file. It is also |    |
|       |                            | used by ISO/IEC 11404 to   |    |
|       |                            | define other common        |    |
|       |                            | datatypes (see next        |    |
|       |                            | section).                  |    |
+-------+----------------------------+----------------------------+----+
| Char  | A family of datatypes that |                            | 10 |
| acter | represent strings of       |                            | .1 |
| s     | symbols from standard      |                            | .5 |
| tring | character sets.            |                            |    |
+-------+----------------------------+----------------------------+----+
| Octet | The datatype of eight-bit  |                            | 10 |
|       | codes, as used for         |                            | .1 |
|       | character-sets and private |                            | .7 |
|       | encodings.                 |                            |    |
+-------+----------------------------+----------------------------+----+
| Opt   | A generator that           | Level 1a when it           | 10 |
| ional | effectively adds the       | represents a scalar value, | .2 |
|       | \"nil\" value to the value | but level 1b when it       | .4 |
|       | space of a base datatype.  | represents a complex       |    |
|       |                            | structure.                 |    |
+-------+----------------------------+----------------------------+----+
| Attr  | A (name, value) pair       | In files, attributes are   | no |
| ibute | attached to some other     | usually only attached to   | ne |
|       | datatype to provide        | certain objects. Although  |    |
|       | additional meaning.        | the \"value\" is usually a |    |
|       |                            | scalar type, it could be   |    |
|       |                            | any of the above-named     |    |
|       |                            | types.                     |    |
+-------+----------------------------+----------------------------+----+
| Group | An organizing structure    | \<\< May be equivalent to  | no |
|       | that contains other data   | a bag or set, depending on | ne |
|       | structures.                | its semantics.\>\>         |    |
+-------+----------------------------+----------------------------+----+

## Level 3(a). Complex aggregates

This category is meant to describe types that build on level 2 types,
either by combining level 2 structures or by providing new access
methods to level 2 structures. It is difficult to define this category
in pure computer science terms, because many of its structures are
created with for specific applications, and some of are unique to a
particular application. Although the category is loosely defined, its
members share one or more of the following characteristics.

8.  They are made up of other aggregate and simple structures.

9.  They may change their size or shape after initial instantiation.

10. Their methods may be more complex than those of the simple types and
    simple aggregates.

11. Different parts may have different size, shape, or resolution.

Many of the formats that we are examining do no directly support these
types, but these types are sometimes created using the formats. For
example, several applications create irregular grids using HDF and
netCDF, although neither format directly supports irregular grids.
Still, there are, such DX, that do define many complex aggretates as
primitives.

12. 

+-------+----------------------------+----------------------------+----+
| **T   | **Rough definition**       | **Examples/comments**      |    |
| ype** |                            |                            |    |
+-------+----------------------------+----------------------------+----+
| Stack | Same as a sequence         | Same as a \"last in first  | 10 |
|       | structure, except the      | out\" (LIFO) list.         | .2 |
|       | insertion operation puts   |                            | .1 |
|       | values at the beginning of |                            |    |
|       | the sequence (push), and a |                            |    |
|       | removal operation takes    |                            |    |
|       | values from the beginning  |                            |    |
|       | of the sequence (pop).     |                            |    |
+-------+----------------------------+----------------------------+----+
| Queue | Same as a sequence         | Same as a \"first in first | no |
|       | structure, except the      | out\" (FIFO) list          | ne |
|       | insertion operations puts  |                            |    |
|       | values at the beginning of |                            |    |
|       | the sequence, and a        |                            |    |
|       | removal operation takes    |                            |    |
|       | values from the end.       |                            |    |
+-------+----------------------------+----------------------------+----+
| Index | This defines a class of    | An index structure is used | No |
|       | level 3 structures in      | to support efficient       | ne |
|       | which an \"index           | search. It supports        |    |
|       | structure\" is first       | essentially the same       |    |
|       | searched for some key      | insertion, deletion, and   |    |
|       | value, whence the location | selection operations as a  |    |
|       | of the desired data can be | table. A given search      |    |
|       | determined.                | structure may be           |    |
|       |                            | categorized according to   |    |
|       |                            | or or more of three        |    |
|       |                            | general types of access:   |    |
|       |                            | random access, multi-view, |    |
|       |                            | and indexed-sequential     |    |
|       |                            | access.                    |    |
|       |                            |                            |    |
|       |                            | Level 3 structures that    |    |
|       |                            | support indexing include   |    |
|       |                            | trees and hash tables.     |    |
+-------+----------------------------+----------------------------+----+
| Tree  | A recursive sequence       | Trees are commonly used    | 10 |
| index | structure in which every   | for data that changes      | .2 |
|       | member is either a tree or | frequently. Trees are      | .2 |
|       | a \"leaf,\" where a leaf   | common index structures. A |    |
|       | can be any datatype.       | B+ tree, which uses a      |    |
|       |                            | B-tree for an index, is    |    |
|       |                            | designed for               |    |
|       |                            | indexed-sequential access. |    |
+-------+----------------------------+----------------------------+----+
| Hash  | A table structure in which | Hash tables can provide    | No |
| index | the position of a key *k*  | very fast random access to | ne |
|       | is determined by a \"hash  | records based on some key. |    |
|       | function,\" *h(k).*        |                            |    |
+-------+----------------------------+----------------------------+----+
|       |                            |                            | No |
|       |                            |                            | ne |
+-------+----------------------------+----------------------------+----+
| Mesh  | A mesh is an n-dimensional | Meshes typically involve   | No |
|       | spatial representation of  | two data structures. One   | ne |
|       | data consisting of a       | structure describes the    |    |
|       | topology and a geometry.   | location of all of the     |    |
|       | The topology is defined by | points in the mesh. A      |    |
|       | a set of vertices and      | second structure describes |    |
|       | connectivity information   | the connectivity that      |    |
|       | describing cells. The      | defines the cells that     |    |
|       | geometry is given by the   | make up the mesh.          |    |
|       | location of the vertices   |                            |    |
|       | and cells in space.        | Another structure          |    |
|       |                            | describes the data         |    |
|       | Types of mesh include:     | assigned to the mesh. Data |    |
|       |                            | are assigned either to     |    |
|       | 13. Structured mesh (grid) | vertices or cells in the   |    |
|       |                            | mesh.                      |    |
|       | Uniform                    |                            |    |
|       |                            |                            |    |
|       | Rectilinear                |                            |    |
|       |                            |                            |    |
|       | Irregular                  |                            |    |
|       |                            |                            |    |
|       | Multi-resolution           |                            |    |
|       |                            |                            |    |
|       | 14. Unstructured           |                            |    |
+-------+----------------------------+----------------------------+----+
| Grid  | 15. A mesh defined in      |                            | No |
| (     |     terms of Cartesian     |                            | ne |
| struc |     coordinates consisting |                            |    |
| tured |     of cells that are      |                            |    |
| mesh) |     quadrilaterals (2-d),  |                            |    |
|       |     hexahedra (3-d), etc., |                            |    |
|       |     whose opposite sides   |                            |    |
|       |     are not necessarily    |                            |    |
|       |     parallel. In the 2-d   |                            |    |
|       |     case, the topology of  |                            |    |
|       |     a grid is defined by 2 |                            |    |
|       |     integers, X and Y,     |                            |    |
|       |     that indicate the      |                            |    |
|       |     number of vertices,    |                            |    |
|       |     X\*Y, and cells, (X-1) |                            |    |
|       |     \* (Y-1). The geometry |                            |    |
|       |     is defined by an array |                            |    |
|       |     (sometimes implied) of |                            |    |
|       |     *x* and *y*            |                            |    |
|       |     coordinates, one for   |                            |    |
|       |     each vertex. Higher    |                            |    |
|       |     dimensions are defined |                            |    |
|       |     simiarly. 3-d and      |                            |    |
|       |     higher cases are       |                            |    |
|       |     defined similarly.     |                            |    |
+-------+----------------------------+----------------------------+----+
| Un    | 16. A grid in which each   | The grid can be defined by | No |
| iform |     cell mesh is a         | providing a start point    | ne |
| grid  |     rectangle of two or    | and delta value for each   |    |
|       |     higher dimension. All  | axis.                      |    |
|       |     cells have the same    |                            |    |
|       |     width and the same     | In 2-d, a uniform grid is  |    |
|       |     height.                | visualized as a set of     |    |
|       |                            | uniformly spaced           |    |
|       |                            | horizontal lines and       |    |
|       |                            | uniformly spaced vertical  |    |
|       |                            | lines.                     |    |
+-------+----------------------------+----------------------------+----+
| R     | A grid whose geometry is   | Each cell of a rectilinear | No |
| ectil | defined by a 1-d           | grid is a rectangle.       | ne |
| inear | \"coordinate array\"       |                            |    |
| grid  | *a~i\ ~*for each axis *i*. | E.g. for a 2-d grid, the   |    |
|       | *a~i\ ~* is the list of    | geometry is defined by two |    |
|       | the values for the *i^th^* | coordinate arrays *x* and  |    |
|       | coordinate of the set of   | *y.* *x* is the list of    |    |
|       | vertices, in increasing    | abscissas of the vertical  |    |
|       | order. The set of vertices | lines, and *y* is the list |    |
|       | is the cross-product of    | of ordinates of the        |    |
|       | the coordinate arrays.     | horizontal lines.          |    |
+-------+----------------------------+----------------------------+----+
| irre  | A grid whose geometry is   | The cells are quadrangles, | No |
| gular | defined by listing the     | hexahedra, etc., whose     | ne |
| grid  | vertices of every point in | opposite sides are not     |    |
|       | the grid.                  | necessarily parallel.      |    |
|       |                            |                            |    |
|       |                            | E.g. for a 2-d grid of     |    |
|       |                            | size m-by-n, m\*n vertices |    |
|       |                            | are provided.              |    |
+-------+----------------------------+----------------------------+----+
| M     | A grid whose resolution    |                            | No |
| ulti- | varies at different        |                            | ne |
| resol | locations.                 |                            |    |
| ution |                            |                            |    |
| grid  |                            |                            |    |
+-------+----------------------------+----------------------------+----+
| Un    | A mesh whose topology is   | Many unstructured meshes   | No |
| struc | defined by the number of   | consist of a single type   | ne |
| tured | nodes, the number of       | of cell, such as a         |    |
| mesh  | cells, and lists of the    | triangle or hexahedron.    |    |
|       | nodes that comprise each   |                            |    |
|       | cell (connectivity lists). |                            |    |
|       | The geometry is defined by |                            |    |
|       | the location of the nodes. |                            |    |
+-------+----------------------------+----------------------------+----+
|       |                            |                            |    |
+-------+----------------------------+----------------------------+----+
| Disc  | Same as a record type, but | Not listed as an ISO 11404 | no |
| rimi- | the record structure many  | type, but this is a type   | ne |
| nant\ | vary in content and length | that occurs in some        |    |
| r     | based on values (the       | applications.              |    |
| ecord | discriminant) stored       |                            |    |
|       | within in the record.      |                            |    |
+-------+----------------------------+----------------------------+----+
|       |                            |                            |    |
+-------+----------------------------+----------------------------+----+

## Level 1(b). Simple science types

Level 1(b) types are atomic types that have general applicability in
many scientific domains. They include information with a physical or
temporal (as opposed to purely computational or mathematical)
interpretation. Of the ISO 11404 types, only two seem to fit well into
this category, \"date and time\" and \"time interval.\"

For others we turn to the International System of Units (SI), which
defines seven *base physical quantities*, all of which can be
represented using atomic datatypes.
(http://physics.nist.gov/cuu/Units/index.html) In addition, there is a
much longer list of SI *derived quantities*, which are defined in terms
of the the seven base quantities.. SI\'s definition of a quantity is

> A **quantity in the general sense** is a property ascribed to
> phenomena, bodies, or substances that can be quantified for, or
> assigned to, a particular phenomenon, body, or substance. Examples are
> mass and electric charge.

The following table lists the two ISO 11404 types, the seven SI base
quantities, and a few derived quantities.

  --------------- ---------------------------- ---------------------------- --------
  **Type**        **Rough definition**         **Examples/comments**        

  Date-and-time   Family of datatypes whose    Resolutions: year, month,    8.1.6
                  values are points in time to etc.                         
                  various resolutions                                       

  Time interval   One of the seven basic SI    The SI standard unit:        10.1.6
                  quantities.\                 second.\                     
                  ISO 11404: A family of       ISO 11404: A form of scaled  
                  datatype representing        datatype. E.g. (second, 10,  
                  elapsed time in seconds or   3) represents 3              
                  factions of a second.        milliseconds.                

  Length          One of the seven basic SI    Standard unit: meter         
                  quantities.                                               

  Mass            One of the seven basic SI    Standard unit: kilogram      
                  quantities.                                               

  Electric        One of the seven basic SI    Standard unit: ampere        
  current         quantities.                                               

  Temperature     One of the seven basic SI    Standard unit: kelvin        
                  quantities.                                               

  Amount          One of the seven basic SI    Standard unit: Mole          
                  quantities.                                               

  Luminous        One of the seven basic SI    Standard unit: candela       
  intensity       quantities.                                               

  Volume          An SI derived quantity.      Standard unit: square meter. 

  Velocity        An SI derived quantity.      Standard unit: meter per     
                                               second                       

  Acceleration    An SI derived quantity.      Standard unit: meter per     
                                               second squared               

  Mass density    An SI derived quantity.      Standard unit: kilogram per  
                                               meter squared                
  --------------- ---------------------------- ---------------------------- --------

Material beyond this point (except Appendix A) is old and needs to be
revised.

## Level 2(b). Aggregate - general science

17. 

```{=html}
<!-- -->
```
1.  Element types (triangle, tetrahedron)

2.  Field value types (vector, tensor)

\<\<\<**What about \"variables?**\" I need to find a place for them, and
I think this is it. Here\'s a related comment by Don, which I will work
from: \"One other point, with regard to CDF and variables. CDF
documentation does refer to variables, including rvaraibles and
zvariables. I think the issue here is how are they defined beyond the
fact that they are arrays, and do we have other, computer science, terms
to refer to them.\"\>\>\>

18. 

19. Spatial context information

```{=html}
<!-- -->
```
1.  Bounding box (identifies the maximum and minimum extent of any
    associated multidimensional array)

```{=html}
<!-- -->
```
3.  Coordinate systems

```{=html}
<!-- -->
```
20. Simple Image and color lookup table

## Level 2c. Instrument-specific 

21. Finite element mesh

```{=html}
<!-- -->
```
1.  Structured

```{=html}
<!-- -->
```
4.  Unstructured

5.  Multi-resolution (hierarchical)

```{=html}
<!-- -->
```
22. Sparse and dense fields (mappings to coordinate systems,
    interpolation schemes)

23. Image with color lookup table

24. Indexed color lookup table

are specific to some scientific domain, instrument type, etc. In this
categorization, the categories can be narrow (Landsat 7) or broad (space
science). We might wish to break this into two levels to cover this
difference.

25. Earth science (Swath, grid, point)

26. Space science (DC fields, particle fluxes)

27. GIS (vector data; raster data)

28. Product design

29. Medicine

30. Etc.

## Library services

In writing these notes, I have been tempted several times to mention the
services that are provided with the various formats. Several of the
formats under discussion come with APIs and libraries for reading and
writing them, as well as other access services, like subsetting. Indeed,
an I/O library seems to be a requirement for those formats that are very
general and provide a lot of different options, formats like HDF, netCDF
and CDF. They are just too complicated to decipher to assume that an
individual could easily write readers and writers for them. On the other
hand, there are formats, like FITS and PDS, that seem to provide both
simplicity and many good options.

Another reason for libraries is to ensure that formats are always
written correctly, and that a given object is written in a consistent
way. This advantage applies to both basic formats and profiles.

More old stuff

# An annotated glossary

I thought it would be useful, not just for this piece, but for our
discussions in general, to begin to define some of the terms we are
working with. This glossary is a first pass at doing that. It still is
missing many terms, and it also ranges beyond the specific \"levels\"
discussion that got it started.

## Types, structures and objects

Some of the ideas for these terms came from \"A Reference Model for
Scientific Information Interchange\" (Reich, Sawyer, Davis).

A *datatype* is a classification identifying one of various types of
data, as floating-point, integer, or Boolean, stating the possible
values for that type, the operations that can be done on that type, and
the way the values of that type are stored. Datatypes specify the set of
values from which a variable, constant, function, or other expression
may take its value.

### *Datatypes*

31. A *primitive datatype* is one whose values are usually treated as a
    single entity with well understood semantics in the computer science
    domain. Primitive datatypes include

32. An *ASCII number* is a textual representation of an integer or
    decimal value. It size and format can sometimes vary from one value
    to another.

33. *Standard integer and floating point types*. These are binary
    fixed-length numeric types that can normally be operated on in
    hardware, such as \"eight-bit signed two\'s complement little endian
    integer\" and \"32-bit IEEE float.\"

34. *Non-standard integer and floating point types*. These are
    fixed-length numeric types that are specially defined by users or
    applications. They can normally be operated on only in software.
    They offer advantages such as space savings that are worth the extra
    costs and troubles that they bring.

35. *String:* A sequence of characters that are treated as a unit.
    Strings can be fixed-length or variable-length.

36. *Boolean:* a type that can have one of two values, \"true\" or
    \"false.\"

37. *Special types. : enumerated type, pixel, pointer, and opaque.*
    These are less well well-defined types, but have the following basic
    meanings:

-   Enumerated type: a type whose value is selected from a defined list
    of values.

-   Pixel (\"picture element\"): the smallest discrete unit in an image,
    either on a screen or stored in memory. A pixel may contain one or
    more bits of information, representing the brightness or color of
    the image at a point and possibly including color information.
    (Adapted from The Free On-line Dictionary of Computing.)

-   Pointer: a value that is understood as a reference to information
    residing at some other location.

-   Opaque type: a sequence of bytes with no associated semantics, as
    far as the format is concerned.

38. *Record* (a.k.a. compound or composite datatype): An ordered
    collection of related elements. Each element is a (name, value)
    pair, where the value may be any allowable type. A record may be
    allowed to contain other records as the value of one or more of its
    elements. (Adapted from ODMG definition of built-in type
    \"structure.\")\
    The elements of compound types are sometimes called \"fields,\" but
    this can be confusing because of the scientific meaning of the term.
    Vectors, tensors, and other datatypes can be implemented using
    records.

Other useful definitions related to types:

A *fixed-length datatype* is a datatype with the constraint that of
every instance in a structure or file occupies a specific number of
bits. An example of a fixed length datatype is \"eight-bit signed two\'s
complement integer\"

A *variable-length datatype* is a data type whose size can vary from one
instance to another. An example of a variable length type is a variable
length string. The distinction between a variable length type and a
variable length array is sometimes murky, and often boils down to the
operations that can be performed on each. For instance, a variable
length string can often be treated as an atomic type when an array
cannot.

### More complex types

39. A *composite data type* is a collection of one or more related
    instances of the available PDTs, stored and accessible as a single
    structure by readers and writers of a file. Aggregates generally
    correspond to the simple data structures supported by most
    languages, such as arrays, tables and trees. They also include
    grouping structures, such as the HDF4 vgroup. CDF and netCDF
    variables are aggregates, as are HDF4 SDS\', but the fact that they
    have associated coordinate systems and other metadata gives them
    meaning beyond what is meant by a pure aggregate.\
    The ODMG\'s Object Database standard uses the term \"collection\" to
    define aggregates. There are four collection types: set, bag, list,
    array. An earlier version of the standard had an object type called
    \"structure,\" which may have been roughly equivalent to a group. It
    might be worth looking at ways to harmonize the types that are
    defined here with the ODMG\'s approach.\
    The following aggregates are listed in our \"levels\" taxonomy.

40. *Array*: an ordered collection of identically typed data items
    distinguished by their indices (or \"subscripts\"). The number of
    dimensions an array can have depends on the language, format, etc.,
    but is often unlimited. The sizes of an array\'s dimensions may be
    fixed, or they may be explicitly or implicitly increased after its
    creation.\
    A single data item could be considered as a zero-dimensional array.
    A one-dimensional array is also known as a \"vector.\" Some data
    models permit only one-dimensional arrays, with higher dimensions
    achieved by forming arrays of arrays. For instance in the C language
    an N-dimensional array is actually a vector, each of whose elements
    is an N-1 dimensional array.\
    Elements of an array are usually stored contiguously, but other
    organizations are possible. Languages differ as to whether the
    leftmost or rightmost index varies most rapidly, i.e. whether each
    row is stored contiguously or each column (for a 2D array).\
    Arrays are appropriate for storing data which must be accessed in an
    unpredictable order, in contrast to \"lists,\" which are best when
    accessed sequentially.

41. *List:* An ordered collection of elements of a given type. Lists
    differ from arrays in that elements may be removed from a list,
    shortening the length of the list. Because of this, the relative
    position of a particular list element may change.

42. *Table:* An collection of records. Depending on the operations
    allowed, a table may be an array or records or a list of records.

43. *Selection region:* a part of certain types of aggregate. A
    selection region for a multidimensional array is typically a smaller
    array that has the same dimensions as the array. This is sometimes
    called a hyperslab.

44. *Group:* An organizing structure that contains other data
    structures. A group may be instantiated in a number of different
    ways. For instance as a list of pointers, where each pointer refers
    to some structure.

45. *Image:* Data representing a two-dimensional scene. A digital image
    is composed of
    [pixel](../../../../../../../../cgi-bin/dict.pl%3Fterm=pixel)s
    arranged in a rectangular array with a certain height and width.
    (Adapted from *The Free On-line Dictionary of Computing*.)

46. *Color lookup table (a.k.a. color palette):* a table that determines
    how to convert pixel values to physical colors represented by a
    color model, such as RGB triplets. A common example would be a
    palette of 256 colors (i.e. addressed by eight-bit pixel values)
    where each color can be chosen from a total of 16.7 million colors
    (i.e. eight bits output for each of red, green and blue). (Adapted
    from *The Free On-line Dictionary of Computing*.)

47. An *attribute* is a (name, value) pair that helps give meaning to
    some other structure. Although the \"value\" part of an attribute is
    usually a scalar value, it could be any of the above-named types.

Some other definitions that describe more complex types and how we
organize them.

48. A *compound aggregate* is a collection of aggregates that implements
    a data structure that is useful for some application. For instance,
    two tables may be combined to describe record data in a way that is
    easy and fast to search and access. CDF and netCDF have a nice way
    of storing and accessing related variables. Such a collection is an
    example of a compound aggregation.

## Files and related things

49. A *data object* is a structure made up of datatypes and aggregates
    that has meaning within some application domain. Object types may be
    included in the specification of a file format (as in the case of a
    domain-specific file format), or they may be specified at a higher
    level, by a particular application of a file format (as in the case
    of HDF-EOS). By this definition, pure domain-specific file formats
    do not directly support aggregations, but rather support objects.
    That said, it is possible to store aggregations in most
    domain-specific format. For instance GRIB, which is designed for
    storing meteorological data, can be used to store large arrays of
    any kind of data.

50. *Structural metadata* is descriptive information about data types
    and aggregations that is sufficient to allow a reader of the data to
    interpret the value or values that the data represents. We assume
    that structural metadata is available in some form for all files.

51. A *file header* is a structure that provides useful information
    about the contents and organization of the file. Typically a header
    is stored at the beginning of a file, but our definition does not
    require this.

52. A *file* is a collection of types and structures, and may include a
    file header.\
    This definition presents some problems. The kinds of things we would
    like a file to be go beyond what we normally think of as a file. I
    found this discussion of the definition of \"file\" in the \"Free
    Online Dictionary of Computing: \"\[The\] prototypical file has
    these characteristics:

It is a single sequence of bytes (but consider
[*Macintosh*](../foldoc.cgi%3FMacintosh) [*resource
forks*](../foldoc.cgi%3Fresource+forks)).

It has a finite length, unlike, e.g. a [*Unix*](../foldoc.cgi%3FUnix)
[*device*](../foldoc.cgi%3Fdevice).

It is stored in a [*non-volatile
storage*](../foldoc.cgi%3Fnon-volatile+storage) medium (but see
[*ramdrive*](../foldoc.cgi%3Framdrive)).

It exists (nominally) in a [*directory*](../foldoc.cgi%3Fdirectory).

It has a name that it can be referred to by in file operations, possibly
in combination with its [*path*](../foldoc.cgi%3Fpath). \"

> Here is a pair of definitions that might help alleviate the problem:

A *file* is a collection of types and structures with a name that it can
be referred to by in file operations, is stored as a sequence of bytes,
is stored on non-volatile storage, and has finite length,

A *virtual file* is a file with the following exceptions: it can exist
in a volatile storage medium (e.g. in memory), its components can be
distributed among more than one device, and it can have unspecified
length (e.g. a UNIX device).

53. A *file format* is an unambiguous specification of how primitive,
    compound and aggregates are organized in a file, sufficiently
    detailed to allow a reader or writer to be written for accessing all
    of the data in the file.

54. A file format is *simple* if it consists only of primitive objects
    and datatypes, stored in some externally-defined order and
    representation. The ASCII export format for Microsoft Excel is a
    simple format; it stores a simple array of numbers in an
    easy-to-read form. Simple formats often support just one type of
    aggregation, but they could consist many structures, such as of a
    sequence of 2D arrays or images, a table and an image, etc.\
    Simple formats have the enormous benefit that it is easy to write
    readers and writes for them. If there is no compelling reason to do
    otherwise, it is probably best to store data in a simple format.

55. A file format is *self-describing* if it contains information about
    how the primitive objects and datatypes are to be interpreted. For
    instance, a format like FITS, with a header that describes the
    datatypes and dimensions of its images is self-describing.
    Self-describing formats are useful when the there is great
    variability in structure of data.\
    This is another difficult thing to define. When we think about
    different file formats we see a continuum of self-describing-ness.
    Furthermore, every file is self-describing in the sense that you
    have to examine its contents to understand its meaning. On the other
    hand, no file is completely self-describing because you always need
    some external information or assumptions in order to decipher the
    contents of a file.

56. A *domain-specific file format* is one that applies to a particular
    application domain, such as space science, meteorology, or moving
    pictures.

57. A *general purpose file format* is one whose primitive objects and
    datatypes do not apply to any particular application-specific
    context. General purpose formats are useful when the data needs to
    be shared among many different application domains, or when there is
    useful, general-purpose software for working with the data, software
    that might span many different domains and therefore need the
    self-description to help decipher any particular instance of the
    data. General-purpose formats can also be useful when it is
    desirable to organize the same basic data differently according to
    how the data might need to be stored or accessed.

58. An *object store file* is a file that is used for storing and
    accessing objects. In the OO world we might call an object store a
    \"container object.\"

59. A *profile* is a file specification that includes data objects and
    metadata specific to a particular application domain.

Appendix A\
ISO/IEC 11404

The ISO/IEC standard (ISO/IEC 11404: 1996 (E)) for
\"Language-independent datatypes\" provides a rich and rigorously
defined set of datatypes for programming languages that maps very well
to format datatypes at this level. This appendix lists many of the
datatypes described in the standard.

# Some definitions from ISO/IEC 11404 

Here are some useful definitions and other comments from the ISO/IEC
11404:1996(E) document, *Information technology\--Programming languages,
their environments and system software interfaces\--Language-independent
datatypes.*

60. A *datatype* is a set of distinct values, characterized by
    properties of those values and by operations on those values.

61. A *value space* is a collection of values for a given datatype. The
    value space of a given datatype can be defined in one of the
    following ways:

Enumerated outright

Defined axiomatically from fundamental notions

Defined as the subset of those values from some already defined value
space which have a given set of properties

Defined as a combination of arbitrary values from some already defined
value spaces by a specified construction procedure.

62. Datatype properties

Equality

Order

Bound

Cardinality

Exact and approximate

Numeric and non-numeric

63. Primitive and non-primitive datatypes

A *primitive* datatype is one which is defined ab initio without
reference to other datatypes.

A *generated* datatype is one what specified, and partly defined, in
terms of other datatypes.

64. Atomic and aggregate

An *atomic* datatype has values which are intrinsically indivisible.

An *aggregate* datatype has values that can be seen as an organization
of specific component datatypes with specific functionalities.

Note: all primitive datatypes are conceptually atomic. Some generated
datatypes are conceptually atomic.

An aggregate data type has the *ordering* property if an only if there
is a canonical first element of each non-empty value in its value-space.
(This allows us to talk about the \"first element\" in an array, for
instance.

65. Homogeneity

An aggregate datatype is *homogeneous* if an only if all components must
belong to a single datatype. A component datatype of a homogeneous
datatype is called the *element datatype*. selves

An aggregate datatype is *heterogeneous* if different components may
belong to different datatypes.

Note: An aggregate datatype is homogeneous even if element datatype is
itself heterogeneous.

66. Size

The *size* of an aggregate-value is the number of component values it
contains.

The size of an aggregate datatype is *fixed* if and only if all values
in its value space contain the same number of component values.

The size of an aggregate datatype is *variable* if different values of
the aggregate datatype may have different numbers of component values.

67. Access method

An access method for an aggregate is the property that determines how
component values can be extracted from a given aggregate-value.

An aggregate datatype has a *direct access method* if and only if there
is an aggregate-imposed mapping between values of one or more \"index\"
(or \"key\") datatypes and the component values of each aggregate value.
Such a mapping is required to be single-valued, i.e. there is at most
one element of each aggregate value which corresponds to each
(composite) value of the index datatype(s).

The *dimension* of a datatype is the number of index or key datatypes
the aggregate has.

# ISO/IEC 11404 datatypes

The following datatypes are based on that standard. The standard
specifies the following categories of datatypes:

68. *Primitive type*: defined either axiomatically or by enumeration.
    E.g. integer; character.

69. *Subtype and extended type*: derived from an existing datatype by
    restricting the value space to a subset type while maintaining the
    same operations. E.g. range subtype. (These are not included in the
    discussion, but we may want to add them later.)

70. *Generated type*: defined by the application of datatype generator
    to one or more previously defined types. E.g. pointer; procedure.

71. *Aggregate type*: generated type each of whose values is made up of
    values of the component datatypes, in the sense that operations on
    all component values are meaningful. E.g. array; record.

72. *Defined type*: can be derived from other types and generators, but
    are treated as primitive types. E.g. character string; bit.

The ISO/IEC 11404 does not deal with how these types are *represented*,
since representation is a language-specific feature. With formats, as
with languages, representation can be very important, however.

## ISO/IEC 11404 primitive types

Certain ISO/IEC 11404 primitive datatypes have been omitted because they
rarely, if ever, occur in formats. These are *state, ordinal,* and
*rational*.

  --------------- ---------------------------- ----------------------------- --------
  **Type**        **Rough definition**         **Examples/comments**         

  Boolean         Math type whose values are   E.g. \"true\", \"false\"      8.1.1
                  \"true\" and \"false                                       

  State           Family of datatypes, each of E.g. for use in switch        8.1.2
                  which comprises a finite     statements. Probably not      
                  number of distinguished but  relevant to scientific data   
                  unordered values.            formats.                      

  Enumerated      Finite collection of         E.g. \"orange\", \"apple\",   8.1.3
                  distinguished values having  \"pear\", where               
                  an intrinsic order           orange\<apple\<pear           

  Character       Family of datatypes whose    ASCII most common, but other  8.1.4
                  value spaces are             character sets may need to be 
                  character-sets               considered.                   

  Ordinal         The infinite enumerated      Is this used in any format?   8.1.5
                  datatype consisting of the                                 
                  mathematical values                                        
                  \"first,\" \"second,\" etc.                                

  Date-and-time   Family of datatypes whose    Resolutions: year, month,     8.1.6
                  values are points in time to etc.                          
                  various resolutions                                        

  Integer         Math type comprising exact   Many precisions supported.    8.1.7
                  integral values              Sometimes signed and unsigned 
                                               are distinguished. (Should    
                                               these be enumerated?)         

  Rational        The \"rational numbers.\"    Number of the form            8.1.8
                                               *signed-number/\[number\]*,   
                                               where *number* cannot equal   
                                               zero.                         

  Scaled          Value space is a subset of   Used by some formats to save  8.1.9
                  the rationals, each with a   space.                        
                  fixed denominator, but                                     
                  possessing the concept of                                  
                  approximate value.                                         

  Real            Approximations of the math   Size and precision can be     8.1.10
                  type \"real numbers.\"       important in scientific       
                                               applications. (Should these   
                                               be enumerated?)               

  Complex         Approximations of the math   Often not a primitive type,   8.1.11
                  type \"complex numbers.\"    but represented by a          
                                               predefined convention.        

  Void            Represents an object whose   Perhaps this is the same as a 8.1.12
                  presence is required, but    fill-value.\"                 
                  carries no information.                                    
  --------------- ---------------------------- ----------------------------- --------

## ISO/IEC 11404 generated types

The ISO standard defines a *datatype generator* as a conceptual
operation on one or more datatypes that yields a datatype. We can use
this concept to describe how higher level, (sometimes) more complex
datatypes are created out of lower level datatypes.

One generated datatype, *choice,* has been omitted because we are not
aware that it occurs in any format. This datatype allows such types as
the C language\'s *union* datatype.

  ----------- ---------------------------- ---------------------------- -------
  **Type**    **Rough definition**         **Examples/comments**        

  Pointer     Values constitute a means of There are a wide variety of  8.3.2
              reference to values of       ways to define and implement 
              another datatype.            pointers. It may be useful   
                                           to delineate some of these.  

  Procedure   A function that maps an      We\'re not aware of formats  8.3.3
              input space to a result      that support this, but it is 
              space.                       frequently mentioned as a    
                                           desirable feature.           

  Aggregate   Values are made up of values Arrays, tables, sets, etc.,  8.4
              of the component datatypes.  are aggregate datatypes. See 
                                           next section.                
  ----------- ---------------------------- ---------------------------- -------

## Simple aggregates

The standard defines an *aggregate datatype* as \"a generated datatype
each of whose values is, in principle made up of values of the component
datatypes.\" We use the term \"simple aggregates\" to distinguish these
from more complex aggregates such as multi-resolution grids.

+-------+----------------------------+----------------------------+----+
| **T   | **Rough definition**       | **Examples/comments**      |    |
| ype** |                            |                            |    |
+-------+----------------------------+----------------------------+----+
| R     | Heterogeneous aggregation  | There are often            | 8  |
| ecord | of values of component     | restrictions on how        | .4 |
|       | datatypes, each            | complex the fields of a    | .1 |
|       | aggregation having one     | record can be, but in      |    |
|       | value for each component   | principle a field can be   |    |
|       | datatype, keyed by a fixed | any datatype. In general,  |    |
|       | \"field identifier.\"      | fields do not have to      |    |
|       |                            | occur in any particular    |    |
|       |                            | order, but many formats    |    |
|       |                            | require a fixed order.     |    |
+-------+----------------------------+----------------------------+----+
| Set   | Value space is the set of  | What is the utility of     | 8  |
|       | all subsets of the value   | this for scientific data?  | .4 |
|       | space of the element       |                            | .2 |
|       | datatype, with operations  |                            |    |
|       | appropriate to the         |                            |    |
|       | mathematical *set*.        |                            |    |
+-------+----------------------------+----------------------------+----+
| Bag   | Collection of instance of  | This probably corresponds  | 8  |
|       | values from the element    | to the HDF5 grouping       | .4 |
|       | datatype. Multiple         | structure.                 | .3 |
|       | instance of the same value |                            |    |
|       | may occur, and ordering of |                            |    |
|       | elements is not            |                            |    |
|       | significant.               |                            |    |
+-------+----------------------------+----------------------------+----+
| Seq   | Ordered sequence of values | Differs from *bag* in that | 8  |
| uence | from the element datatype. | the ordering is            | .4 |
|       | The ordering is imposed on | significant. Sequences of  | .4 |
|       | the values and not         | scientific records (e.g.   |    |
|       | intrinsic in the           | observations) are          |    |
|       | underlying datatype. The   | frequently implemented     |    |
|       | same value may occur more  | using this datatype.\      |    |
|       | than once in a given       | Is this the same thing as  |    |
|       | sequence. The only way to  | an array with an unlimited |    |
|       | write to a sequence is to  | dimension?                 |    |
|       | append; there\'s no        |                            |    |
|       | replace operation.         | Is this the same as our    |    |
|       |                            | general \"variable length  |    |
|       |                            | type.\"                    |    |
+-------+----------------------------+----------------------------+----+
| Array | Values are associations    | Index datatypes correspond | 8  |
|       | between the product space  | to dimensions. Some        | .4 |
|       | of one or more finite      | formats allow dimensions   | .5 |
|       | datatypes, designated the  | to grow in size after      |    |
|       | *index datatype*, and the  | creation, and as a result  |    |
|       | value space of the         | they are sometimes treated |    |
|       | *element* datatype, such   | like a *sequence*. In      |    |
|       | that every value in the    | files, elements are        |    |
|       | product space of the index | generally stored           |    |
|       | datatypes associates to    | contiguously, but other    |    |
|       | exactly one value of the   | organizations are          |    |
|       | element datatype.          | possible.                  |    |
+-------+----------------------------+----------------------------+----+
| Table | Unordered collection of    | Note that the ISO          | 8  |
|       | values in the product      | definition specifies       | .4 |
|       | space or one or more       | \"unordered.\" This        | .6 |
|       | *field* datatypes, such    | definition is meant to     |    |
|       | that each value in the     | produce a datatype that    |    |
|       | product space represents   | corresponds to the SQL     |    |
|       | an association among the   | concept of \"table.\"      |    |
|       | values of its fields.      |                            |    |
+-------+----------------------------+----------------------------+----+
| De    | A datatype defined by a    | This makes it possible to  | 8. |
| fined | type-declaration, denoted  | rename an existing         | 4. |
| dat   | by a type-reference.       | datatype or name an        |    |
| atype |                            | existing datatype that     |    |
|       |                            | have a complex syntax.     |    |
|       |                            | This can be useful for     |    |
|       |                            | applications that want to  |    |
|       |                            | store complex datatypes by |    |
|       |                            | name in a file. It is also |    |
|       |                            | used by ISO/IEC 11404 to   |    |
|       |                            | define other common        |    |
|       |                            | datatypes (see next        |    |
|       |                            | section).                  |    |
+-------+----------------------------+----------------------------+----+

## Simple defined datatypes and generators

The ISO/IEC 11404 standard specifies several commonly occurring
datatypes that are treated as primitive datatypes, but can be derived
from the datatypes and generators defined earlier. Types from the
standard that are not included in the table are *natural number,* and
*modulo.*

  ------------ ---------------------------- ---------------------------- ---------
  **Type**     **Rough definition**         **Examples/comments**        

  Bit          Represents the finite field                               10.1.3
               of two symbols \"0\" and                                  
               \"1\".                                                    

  Bit string   Datatype of variable length                               10.1.4
               strings of binary digits.                                 

  Character    A family of datatypes that                                10.1.5
  string       represent strings of symbols                              
               from standard character                                   
               sets.                                                     

  Octet        The datatype of eight-bit                                 10.1.7
               codes, as used for                                        
               character-sets and private                                
               encodings.                                                

  Octet string Datatype of variable length                               10.1.8
               encodings of 8-bit codes.                                 

  Private      An application-defined value Makes it possible for a      10.1.9
               space and operation set that non-standard type to be      
               are intentionally concealed  passed between two parties   
               from certain processing      that understand it by a      
               entities. There is no        third party that does not.   
               denotation for a value of a  Also allows the value of a   
               private datatype.            datatype that is meaningless 
                                            to all parties but one to be 
                                            used by other parties, as a  
                                            \"handle\" is used by an API 
                                            without the end user         
                                            understanding its semantics. 
                                            (Is this also the definition 
                                            of a blob?) Sometimes called 
                                            \"opaque.\"                  

  Object       Values that uniquely         See ISO/IEC 11404 gives a    10.1.10
  identifier   identify objects in a        precise and thorough         
               communications protocol.     definition.                  
  ------------ ---------------------------- ---------------------------- ---------

## Simple defined generators

The ISO/IEC 11404 standard specifies several commonly occurring
generators that can be derived from the datatypes and generators defined
earlier. They are *stack, tree, cyclic enumerated,* and *optional.* We
include only *optional* here.

  ---------- ----------------------------- ----------------------------- --------
  **Type**   **Rough definition**          **Examples/comments**         

  Optional   A generator that effectively  To be used in those instances 10.2.4
             adds the \"nil\" value to the when an object, such as an    
             value space of a base         array element, a field, or a  
             datatype.                     record, has no value.         
  ---------- ----------------------------- ----------------------------- --------

Appendix B\
How some common formats relate to levels of complexity and abstractness

\<\<This whole section, including these figures, has to be revised, but
I think we should wait until Lou has had a crack at the stuff in the
previous two sections.\>\> We now look at some common scientific formats
in the context of the levels of complexity and abstractness described in
Table 3 and Figure 1. Figures 2(a)-2(e) illustrate where some popular
formats (HDF4, CDF/netCDF, PDS, ISTP-CDF, HDF-EOS) might fit on the
spectrum from domain-free to domain-specific. HDF4 is perhaps the most
domain-free, and HDF-EOS the most domain-specific, but it is clear that
there is a great deal of overlap among many of the formats. Figure 2
illustrates the additional point that one format\'s domain-free
primitives may be the more domain-specific objects supported by another.
For instance, HDF4\'s image is one of the more domain-specific
structures in HDF4, but one of the primitives of PDS.

![](media/image5.png){width="4.664583333333334in"
height="3.4972222222222222in"}

![](media/image4.png){width="4.664583333333334in"
height="3.4972222222222222in"}

![](media/image6.png){width="4.664583333333334in"
height="3.4972222222222222in"}

![](media/image7.png){width="4.664583333333334in"
height="3.4972222222222222in"}

![](media/image8.png){width="4.664583333333334in"
height="3.4972222222222222in"}

Appendix C Requirements of Formats

Specific data structures and implementation details are hidden \<\<Do
you mean that implementation data structure are hidden, but not
necessarily other data structures?\>\>

73. Access is restricted to API/IOLIB so structural metadata is hidden

    \<\<I think the next three bullets also apply to \"data structure
    formats.\>\>

74. Contain adequate semantic metadata required for understanding
    contents

75. Data model may include complex aggregates in additions to simple
    type and aggregates

76. Tend to use well documented standard encodings such as ASCII and
    IEEE floating point for simple types but may optimize for specific
    operating platforms

77. Tend to be optimized for access speed first and size second

78. Instances be self describing through the use of embedded tags
    \<\<DML does this, too, I think.\>\>

-   Data tends to be machine interpretable through the API/IOLIB but
    useless without the API/IOLIB

Sometimes we describe the objects of a format in terms of a Data
Structure. There are two techniques for documenting data structures

-   A separable description written in a *Data Description language
    (DDL)*. \<\<If you are saying that \"data structures formats\" are
    the same as \"DDL\", I\'m not so sure about this. E.g. I think of
    FITS as a data structure format, but it doesn\'t has a DDL, as far
    as I know.\>\>

-   A set of embedded tags which is often describes as a *Data Markup
    Language(DML)*

Typically description/markup languages are defined via a specification
document, and allow a wide variety of object types, layouts, and
relationships to be described. Their data values are then accessed using
software that understands the language. The choice of whether to use a
DML or a DDL is based on the size/granularity of the underlying
elementary data types, the ease of inserting tags into the data, the
ratio of data instances to unique formats and requirements for
self-description and verification. The EAST data description language
can describe a sequence of repeating records where part of the record
structure can vary from record to record, and where a wide variety of
standard and custom data types are supported at the bit level. The PDS
ODL and FITS are hybrid DDL/DML where large binary data objects are
encapsulated by a set of tags. IDFS uses a description language as a
part of its system. XML has recently emerged as a general purpose markup
language that, in combination with appropriate document type definitions
(DTD), can describe almost any format.

\<\<Another use of DML: verification.\>\>

\<\<What about profiles? Seems like ISTP-CDF is a data structure format,
and HDF-EOS is an API format.\>\>

Characteristics of instances of DDL/DML formats include:

79. Contain high level structural Metadata which terminates at the
    underlying elementary datatypes

80. Contain complete semantic metadata required for understanding
    contents

81. Data Model usually contains simple types and aggregates only

82. Use well documented standard encodings for simple types such as
    ASCII character strings and IEEE floating point

83. Are machine interpretable and human understandable

```{=html}
<!-- -->
```
3.  Instances be self describing through the use of embedded tags or
    defined externally in a Data Description written in a Data
    Description Language (DDL)

4.  Not optimized for speed of access or for size. \<\<or are not
    including negative features?\>\>

[^1]: DX = Data Explorer format (IBM); CDF = Common Data Format (NASA);
    FITS = Flexible Image Transport (Astronomy); HDF = Hierarchical Data
    Format (NCSA); netCDF = network Common Data Form (Unidata); PDS =
    Planetary Data System (JPL); XDR = external data representation
    (Sun).

[^2]: DX = Data Explorer format (IBM); CDF = Common Data Format (NASA);
    FITS = Flexible Image Transport (Astronomy); Flatfile = UCLA 2-file
    table format; HDF = Hierarchical Data Format (NCSA); HDF-EOS = Earth
    Observing System HDF-based format; IMP-8 = Interplanetary Monitor
    Platform format; ISTP-CDF = International Solar-Terrestrial Physics
    Project CDF-based format; MPI-IO = message passing interface I/O
    interface, for performing I/O in parallel; netCDF = network Common
    Data Form (Unidata); PDS = Planetary Data System (JPL); XDR =
    external data representation (Sun); zip = a UNIX archiving and
    compression utility.

[^3]: CDF and Flatfile both primarily support 1c object types, but each
    also supports a date/time datatype, and hence are also listed as
    examples of level 2a formats.

[^4]: The design of HDF-EOS assumes, but does not require, that these
    datatypes will be built using objects in the more general HDF
    format.

[^5]: ISO/IEC types that are not included:

    > primitive types *state, ordinal, rational*
    >
    > generated type *choice*
    >
    > simple defined types and generators *natural number, modulo*
    >
    > simple defined generators *stack, tree, cyclic enumerated*