**Binary Data Format Languages and Software across caBIG^TM^**

**Written by members of the Binary Data Format WG**

Jyotishman Pathak, Pankaj Agarwal, Dong Fu, Scott Oster, Patrick
McConnell, Sumit Middha

**[Abstract]{.underline}**

The ability to describe data in a manner that will allow one to simplify
transfer of data across domains without requiring global standard
formats as well as allow third-party tools to easily access multiple
formats of the described data is an important requirement. This is in
particular relevant to the caGrid infrastructure for enabling universal
data exchange. In this whitepaper, we evaluate an existing set of
languages and software implementations for binary data format. The
evaluation parameters focused on various aspects ranging from the
maturity of the language and software to their ability to enable
semantic labeling of individual data objects. Based on the
investigation, the working group recommends that further evaluation and
prototyping efforts should be focused on leveraging the Open Grid Forum
developed Data Format Description Language (DFDL), and its reference
implementations, namely Virtual XML Garden developed by IBM and Defuddle
developed by the Pacific Northwest National Laboratory.

**[1. Introduction]{.underline}**

Grid computing provides a framework and infrastructure for distributed
hardware and software resources to interact and process information.
Inevitably, these resources read and write data represented in different
formats. Consequently, there is always a requirement for translation of
the heterogeneous data into a common representation format. Notably, in
the recent years, evolution of Extensible Markup Language (XML) as an
open-standard has gained the popularity for facilitating the sharing of
(structured) data across different information systems, particularly via
the Internet. Numerous software technologies have been built that enable
authoring, transforming and querying XML-based data. However, there are
many types of data that are not (or should not be) represented as XML
for practical reasons, such as data generated by sensors and data that
are not necessarily in a tree structure (e.g., image metadata).
Consequently, there is a need to develop techniques that would support
processing of non-XML data as though it is XML, and in many cases,
without explicit conversion to XML since such conversions can be an
expensive process. That is, there is a requirement for describing the
data that will allow one to simplify transfer of data across domains
without requiring global standard formats, or to allow third-party tools
to easily access multiple formats of the described data. Such an
approach is in particular relevant to the caGrid infrastructure for
enabling universal data exchange ensuring that data represented/stored
in multiple formats can be transported across the grid in their native
formats, without losing any associated syntactic and semantic metadata,
and processed by applications.

The purpose of this white paper is to provide an overview of available
data format description languages and to: 1.) initiate a process for
evaluating the need for binary data format description technologies
across caBIG^TM^ workspaces; 2.) begin a process for exploring the match
between existing binary data format description technologies and
workspace requirements; and 3.) begin a process of investigating how
binary data format description languages and software can be applied
along with bulk data transfer efforts within caBIG^TM^.

The white paper outlines two case studies that were contributed by the
broader caBIG community and have been used to motivate the initial
detailed examination of binary data format languages and software. These
case studies involve bulk data transfer of (i) imaging and (ii)
microarray data. Moreover, these cases can be extended to next
generation sequencing technology data issues, as discussed later.

The white paper finally provides a recommendation and outlines a process
for systematically evaluating existing binary data format languages and
software that can be useful to provide as well as process syntactic and
semantic descriptions of different data formats.

The following sections of the whitepaper specifically expand upon the
preceding themes, providing:

-   **Section 2:** Background and significance

-   **Section 3:** A survey of existing binary data format languages

-   **Section 4:** A survey of existing binary data format processing
    software

-   **Section 5:** A summary of caBIG community generated use cases for
    binary data formats

-   **Section 6:** A set of recommendations for caBIG binary data
    formats and languages

-   **Section 7:** Concluding remarks

# [2. Background and Significance ]{.underline}

The last decade has brought about an enormous growth of digital data and
content on the Internet. These data are stored in many different digital
forms that vary from character based text format, to packed binary, to
different \"standard\" scientific data formats including various
self-describing data formats. Some examples of commonly used scientific
data formats are GRid In Binary (GRIB), Binary Universal Format
Representation (BUFR), Common Data Format (CDF), network CDF (netCDF),
Hierarchical Data Format (HDF) and HDF for NASA's Earth Observing System
(HDFEOS). There are historical and practical reasons for having all
these different standard data formats. Many of these formats were
developed by different agencies for their own use and needs including
addressing specific aspects such as compactness, portability, efficiency
etc. However, with the advent of the Internet, these communities are
less standalone day-by-day, and becoming more connected and pursuing
inter-disciplinary research, which in turn is fostering data sharing.
This is in particular true within a grid environment where data
interchange is critically important.

However, in the recent years, evolution of Extensible Markup Language
(XML) as an open-standard has gained the popularity for facilitating the
sharing of (structured) data across different information systems,
particularly via the Internet. Numerous tools and technologies have been
built to generate, query, process and store XML data. Along side,
various technologies have evolved to convert non-XML data into XML, and
to convert XML data into other forms. For example, in the world of
relational databases, Structured Query Language (SQL) has been extended
to SQL 2003 to incorporate SQL/XML \[1\]. However, at the same time,
there are many types of data that are not represented as XML for
practical reasons, such as data generated by low-level sensors, data
residing in archives, data not used for exchange with unknown partners,
and data that are not tree structured (e.g., image metadata, video
metadata, microarray and sequencing metadata). The inability to format
data as XML may be caused by the lack of computing power, bandwidth, or
storage capacity, and sometimes, it may even be necessary to transmit
the same data in both XML and non-XML formats as legacy requirements
dictate.

Consequently, there is a need to develop techniques that allows one to
choose an appropriate data representation for an application based on
its needs and at the same time allows providing a \"description\" of the
data in an appropriate format so that multiple programs can directly
interchange the described data. These descriptions can be provided by
the creator of the format, or developed as needed by third parties
intending to use the format and contains few basic building blocks that
can describe different data structures. These basic building blocks are
separately understandable and free from interactions when combined. As a
result, the design becomes scalable to allow the addition of other data
format descriptions without perturbing existing data format modules.

The objective of this working group is to initiate the investigation of
existing languages and technologies for \"describing multiple data
contents\" within the caGrid infrastructure. We believe that having such
a capability will assist in enabling universal data exchange ensuring
that data represented/stored in multiple formats can be transported
across the grid in their native formats, without losing any associated
syntactic and semantic metadata, and processed by applications.

# [3. Binary Data Format Languages]{.underline}

In the following sections, we will introduce a number of binary data
format languages that have been developed to "describe" the content of
different types of data.

**[3.1 Data Format Description Language]{.underline}**

DFDL stands for data format description language and is an XML-based
language for describing data formats \[2\]. It allows one to choose an
appropriate representation of data for an application based on its needs
and provides a mechanism to describe the data format such that multiple
software programs can directly interchange the described data. These
descriptions can be provided by the creator for the format, or developed
as needed by third parties intending to use the format. Thus, in
essence, a DFDL description allows the data to be read from its native
format and presented as an instance of a "logical data model" or enables
conversion to a corresponding XML document. This is achieved via highly
efficient DFDL parsers that are implemented based on lazy evaluation of
the formats and support seekable, random access to data. The following
are the objectives of DFDL implementations:

-   Density: fewest bytes to represent information content (without
    resorting to compression) and fastest possible I/O.

-   Optimized I/O: applications can write data aligned to byte, word, or
    even page boundaries and to use memory-mapped I/O to insure access
    to data content with the smallest number of machine cycles for
    common use cases without sacrificing general access.

DFDL achieves the above requirements by leveraging W3C XML Schema
Definition Language (XSDL) \[2\]. In particular, an XML Schema is
written for the logical model of the data and is augmented with special
DFDL annotations, where the annotations describe the native
representation of the data. Based on this representation, a DFDL parser
interprets the data and represents it as a logical XML model. We explain
the ideas using an example below adopted from \[2\].

Let us assume that we have the following XML data:

\<w\>5\</w\>

\<x\>7839372\</x\>

\<y\>8.6E-200\</y\>

\<z\>-7.1E8\</z\>

The logical model for this data can be represented in the following XML
schema which simply describes the name and type of each element
corresponding to the data:

\<complexType name=\"example1\"\>

\<xs:sequence\>

\<xs:element name=\"w\" type=\"int\"/\>

\<xs:element name=\"x\" type=\"int\"/\>

\<xs:element name=\"y\" type=\"double\"/\>

\<xs:element name=\"z\" type=\"float\"/\>

\</xs:sequence\>

\</complexType\>

Further assume that we have access to this same data, but in non-XML
format. A binary representation of the data can be visualized in the
hexadecimal as follows:

0000 0005 0077 9e8c

169a 54dd 0a1b 4a3f

ce29 46f6

To describe the above in DFDL, we start with the XML schema describing
the data model, and annotate the type definition as follows:

[\<xs:complexType name=\"example1\"\>]{.mark}

[**\<xs:annotation\>**]{.mark}

**[\<xs:appinfo\>]{.mark}**

[]{.mark}$$**\<dfdl:format representation=\"binary\"**

**byteOrder=\"bigEndian\"**

**lengthKind=\"implicit\"**

**applies=\"toScope\"/\>**

**[\</xs:appinfo\>]{.mark}**

[**\</xs:annotation\>**]{.mark}

[\<xs:sequence\>]{.mark}

[\<xs:element name=\"w\" type=\"int\"/\>]{.mark}

[\<xs:element name=\"x\" type=\"int \"/\>]{.mark}

[\<xs:element name=\"y\" type=\"double\"\>]{.mark}

[**\<xs:annotation\>**]{.mark}

**[\<xs:appinfo\>]{.mark}**

[]{.mark}**\<dfdl:format representation=\"binary"**

**floatType="ieeeNonExtended"**

**applies="hereOnly"/\>**

**[\</xs:appinfo\>]{.mark}**

[**\</xs:annotation\>**]{.mark}

[\</xs:element\>]{.mark}

[\<xs:element name=\"z\" type=\"float\" \>]{.mark}

[**\<xs:annotation\>**]{.mark}

**[\<xs:appinfo\>]{.mark}**

[]{.mark} **\<dfdl:format representation=\"binary"**

**floatType="ieeeNonExtended"**

**applies="hereOnly"/\>**

**[\</xs:appinfo\>]{.mark}**

[**\</xs:annotation\>**]{.mark}

[\</xs:element\>]{.mark}

[\</xs:sequence\>]{.mark}

[\</xs:complexType\>]{.mark}

[]{.mark}

This simple DFDL annotation expresses that the data is represented in
binary format and that the byte order will be big endian, and this is
pretty much all the information that the DFDL parser needs to read the
data.[^1]

Similarly, consider if the same data were represented in a text format:

5, 7839372, 8.6E-200, -7.1E8

As before, we can annotate the same data model, although this time we
define the properties that provide the character encoding, the field
separator (comma), and the decimal separator (period):

\<xs:complexType name=\"example1\"\>

**\<xs:annotation\>**

**\<xs:appinfo\>**

**\<dfdl:format representation="text"**

**encoding="UTF-8"**

**decimalSeparator="."\>**

**separator=\",\"**

**applies=\"toScope\" /\>**

**\</xs:appinfo\>**

**\</xs:annotation\>**

\<xs:sequence\>

\<xs:element name=\"w\" type=\"int\"/\>

\<xs:element name=\"x\" type=\"int\"/\>

\<xs:element name=\"y\" type=\"double\"/\>

\<xs:element name=\"z\" type=\"float\"/\>

\</xs:sequence\>

\</xs:complexType\>

This DFDL description is sufficient for a DFDL parser to interpret the
data and represent its logical XML model.

**[3.2 Binary Format Description Language]{.underline}**

BFDL stands for binary format description language and is an XML dialect
based on the eXtensible Scientific Interchange Language (XSIL) \[3\]
that supports the executable documentation of arbitrary binary and ASCII
data sets. Applying a BFD template to a set of files produces an XML
output containing the original data in an XML-tagged format that can be
interpreted by other programs or subjected to further processing (i.e.
using XSLT).

More specifically, the basic framework of BDFL is based on XSIL and BFDL
adds to new elements and several attributes to the XSIL specification.
The elements support an *if* control structure (XBFDif) and dynamic
reference to previously read values (i.e. variables) using XPath
expressions (XBFDvalue-of). The new attributes provide mechanisms to
describe additional file constructs (e.g., fixed length string buffers)
and to reference external files via generic stream numbers instead of
specific file names/URLs. See Figure 1 for an illustration and below for
an example of how a BFD template can be used to generate an XML
representation of data.

![](media/image1.png){width="4.517361111111111in"
height="3.423611111111111in"} Figure 1 **BFDL Workflow**

\<?xml version=\"1.0\"?\>

\<!DOCTYPE XSIL SYSTEM \"bfd.dtd\"\>

\<XSIL\>

\<Param Name=\"month\" Type=\"int\"/\>

\<Param Name=\"day\" Type=\"int\"/\>

\<Param Name=\"year\" Type=\"int\"/\>

\<Param Name=\"numColumns\" Type=\"int\"/\>

\<Param Name=\"flag\" Type=\"int\"/\>

\<XBFDif test=\"/XSIL/Param\[@Name=\'flag\'\] = 5\"\>

\<Array Name=\"frequencyData\" Type=\"double\"\>

\<Dim\>

\<XBFDvalue-of select=\"/XSIL/Param\[@Name=\'numColumns\'\]\"/\>

\</Dim\>

\<Dim\>6\</Dim\>

\</Array\>

\</XBFDif\>

\<XBFDif test=\"/XSIL/Param\[@Name=\'flag\'\] = 9\"\>

\<Array Name=\"timeData\" Type=\"double\"\>

\<Dim\>4\</Dim\>

\<Dim\>

\<XBFDvalue-of select=\"/XSIL/Param\[@Name=\'numColumns\'\]\"/\>

\</Dim\>

\</Array\>

\</XBFDif\>

\<Stream Encoding=\"Binary\" Type=\"Remote\" XBFDStreamnumber = \"1\"\>

\</Stream\>

\</XSIL\>

Figure 2 BFD Template

\<?xml version=\"1.0\"?\>

\<!DOCTYPE XSIL SYSTEM \"bfd.dtd\"\>

\<XSIL\>

\<Param Name=\"month\" Type=\"int\"\>10\</Param\>

\<Param Name=\"day\" Type=\"int\"\>12\</Param\>

\<Param Name=\"year\" Type=\"int\"\>2001\</Param\>

\<Param Name=\"numColumns\" Type=\"int\"\>3\</Param\>

\<Param Name=\"flag\" Type=\"int\"\>9\</Param\>

\<Array Name=\"energyData\" Type=\"double\"\>

\<Dim\>3\</Dim\>

\<Dim\>5\</Dim\>

\<Stream Delimiter=\",\"\>

8.5,9.6,10.7,11.8,1.9,2.0,3.1,4.2

34.1,56.2,68.3,80.4,45.7,49.2,72.7

\</Stream\>

\</Array\>

Figure 3 Product of running BFD template on data file

**[3.3 Earth Science Markup Language]{.underline}**

ESML stands for Earth Science Markup Language and is a specialized
markup language for earth science metadata based on XML \[4\]. The
primary design goal for ESML is to provide an elegant solution allowing
multiple existing data formats to interoperate with different
applications. It has three primary components that enable data and
application interoperability: the ESML Schema, the ESML Description
Files and the ESML Library (Figure 4). The ESML Schema defines the
grammar for generating an ESML Description File (instance document) for
a given dataset in a certain data format. A GUI based Editor (ESML
Editor) is also available for users to generate ESML Description Files.
The ESML Library is the middleware that applications use to parse an
ESML description and retrieve the data in the associated data file(s).

The example below shows an ESML description file for a simple binary
data file with three fields. The data fields are two dimensional arrays.
After declaring the format, the subsequent sequence of declarations
follow the data structure of the file. The entire data file is enclosed
within a single Structure element. The data fields themselves are nested
in two Array elements

> ![](media/image2.wmf){width="4.124305555555556in" height="2.59375in"}
>
> Figure 4 ESML Components

with size specified for each dimension. Data values in the array are
described by the Field element, with a name (Uwind, DimX, DimY,
respectively) and a data type definition to read integer data in base
ten. The ESML core library provides the basic functionality of reading
the structural metadata from the ESML description file and retrieving
the data from the data file. The canonical DOM tree structure stores the
metadata as it is parsed from the ESML description file.

> ![](media/image3.wmf){width="3.5145833333333334in"
> height="3.623611111111111in"}
>
> Figure 5 Sample ESML Description File

**[3.4 Bitstream Syntax Description Language]{.underline}**

BSDL stands for Bitstream Syntax Description Language and is an XML
based language to describe the high-level structure of bitstreams \[5\].
Typically, a multimedia bitstream consists in a structured sequence of
binary symbols; this structure being specific to the coding format. The
objective of BSDL is not meant to replace the original binary format,
but act as an additional layer, similar to metadata. It does not
describe the bitstream on a bit-per-bit basis, but addresses its
high-level structure, i.e. how it is organized in layers or packets of
data. Furthermore, it does not deal with the semantics of the bitstream,
i.e. the original object, image, audio or video it represents, but only
considers it as a sequence of binary symbols.

> ![](media/image4.wmf){width="4.252777777777778in"
> height="2.841666666666667in"}
>
> Figure 6 BSDL Multimedia Content Application

Figure 7 shows an example of a fragment XML description of a JPEG2000
bitstream, an emerging still image coding standard based on a wavelet
transform. In a simplified view, the bitstream is organized as a
sequence of packets containing a header, identified by a two-byte marker
with the hexadecimal value FF91, followed by two bytes indicating the
length of the header, and four bytes indicating the packet number. This
header is followed by the packet data, consisting in a sequence of bytes
resulting from the arithmetic coding of the wavelet coefficients.

\<Packet\>

\<Header\>

\<Marker\>ff91\</Marker\>

\<Length\>4\</Length\>

\<PacketNumber\>1\</PacketNumber\>

\</Header\>

\<PacketData\>155-396\</PacketData\>

\</Packet\>

> Figure 7 Example of a fragment JPEG2000 Bitstream Description

Once we get an XML representation of the bitstream, it is then possible
to define the corresponding editing operations on the XML document,
which will consist in removing the elements corresponding to the packets
of data to be cut off, and in modifying some element or attribute
values. The strength of describing a bitstream structure using XML is
that there are solutions already available to perform such XML-to-XML
editing operations. These operations are performed via XSLT
transformations.

Furthermore, the bitstream description itself contains the data to be
written in the output bitstream, but does not specify how the value
should be encoded. For example, if the description contains an element
\<Length\>4\</Length\>, additional information is required to know
whether the value \"4\" should be encoded as a short integer on two
bytes or as a floating point variable. This information is contained in
a Bitstream Schema as shown in Figure 8.

\<xsd:element name=\"Header\"\>

\<xsd:complexType\>

\<xsd:sequence\>

\<xsd:element name=\"Marker\"

type=\"xsd:hexBinary\"/\>

\<xsd:element name=\"Length\"

type=\"xsd:short\"/\>

\<xsd:element name=\"PacketNumber\"

type=\"xsd:int\"/\>

\</xsd:sequence\>

\</xsd:complexType\>

\</xsd:element\>

> Figure 8 Bitstream Schema

**[3.5 Binary XML Description Language]{.underline}**

BinX (Binary XML description language) is an XML Schema Standard for
representing the "metadata", associated with data stored in binary
files, in XML \[6\]. The metadata can be used to describe information
about the binary data, such as how the data was produced (parameters,
algorithms used etc), when it was produced and by whom. It also contains
a description of the structure and representation of the data itself.

BinX was developed by edikt (e-Science Data, Information and Knowledge
Transformation: [http://www.edikt.org](http://www.edikt.org/)). It is
used to describe the content, structure and physical layout
(endian-ness, blocksize etc.) of binary files. BinX is designed to
enable transparent transfer of data between diverse platforms. It is a
XML-compliant language to annotate data schema for binary files. The
data elements in the binary data files are represented in XML tags.
Therefore, the data schema or metadata of the binary file is separated
from the binary data content, and stored in a separate XML file. Such a
file is readable to the human users as well as computer tools to parse
the datasets.

The following schematic shows ( Figure 9) the flow of information when
BinX files are used for describing binary data files and the BinX
Library (software) is used to parse the original binary data files into
structured data.

A generic example of a BinX file is:

[\<?xml version=\"1.0\" encoding=\"UTF-8\"?\>]{.mark}

[\<dataset xmlns=\"http://http://schemas.nesc.ac.uk/binx/binx\"
xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\"
xsi:schemaLocation=\"http://http://schemas.nesc.ac.uk/binx/binx
binx.xsd\"]{.mark}

[byteOrder=\"bigEndian\" bitOrder=\"bigEndian\"
blockSize=\"32\"\>]{.mark}

[\<definitions\>]{.mark}

[\<typeDef typeName=\"complexType\"\>]{.mark}

[\<struct\>]{.mark}

[\<ieeeFloat-32 varName=\"real\"/\>]{.mark}

[\<ieeeFloat-32 varName=\"imaginary\"/\>]{.mark}

[\</struct\>]{.mark}

[\</typeDef\>]{.mark}

[\</definitions\>]{.mark}

[\<file src=\"http://www.epcc.ed.ac.uk/testFile.bin\"\>]{.mark}

[\<ieeeFloat-32 varName=\"inputParameter1\"/\>]{.mark}

[\<integer-32 varName=\"inputParameter2\"/\>]{.mark}

[\<arrayFixed\>]{.mark}

[\<defType typeName=\"complexType\"/\>]{.mark}

[\<dim indexFrom=\"0\" indexTo=\"99\" name=\"x\"/\>]{.mark}

[\<dim indexFrom=\"0\" indexTo=\"4\" name=\"y\"/\>]{.mark}

[\</arrayFixed\>]{.mark}

[\</file\>]{.mark}

[\</dataset\>]{.mark}

[]{.mark}

The root tag is \<dataset\> which is can contain a set of type
definitions, contained within the \<definitions\> tag, followed by one
or more file descriptions contained within the \<file\> tag. In this
example a new type "ComplexType" has been declared in the definitions
section that is defined to be a struct containing two floats, one called
"real" and one called "imaginary".

> ![](media/image5.jpeg){width="5.763888888888889in"
> height="3.611111111111111in"}
>
> Figure 9 Binary XML workflow

This file contains two numbers a float (inputParameter1) followed by an
integer (inputParameter2) followed by a two dimensional array of our new
complex number type. The \<dataset\> tag allows us to define the byte
order, the bit order and the blocksize. These can be changed for
individual files, or indeed for individual fields. A visualization tool,
called BinX Editor (BinXed), is available which can be used for
designing BinX schema documents. A BinX document is an XML text file
based on the BinX mark-up language for the purpose of describing a
binary data file.

**[3.6 Message Transmission Optimization Mechanism]{.underline}**

# MTOM (SOAP Message Transmission Optimization Mechanism) is a W3C specification \[7\]. It is a method of efficiently sending binary data to and from Web services. It uses XOP (XML-binary Optimized Packaging) to transmit binary data and is intended to replace both MIME and DIME attachments. 

The main focus of MTOM is to enable transmission of binary attachments
such as images, drawings, XML documents etc. It is based on a technique
called "by reference" where data transmission is achieved by attaching
pure binary data as external unparsed general entities outside the XML
document and then embedding reference URIs to those entities as elements
or attribute values. This prevents the unnecessary bloating of data and
wasting of processing power. In particular, the wire format of a MTOM
optimized message is the same as the SOAP with Attachments (SwA) message
\[8\], which also makes it backward compatible with SwA endpoints. The
most notable feature of MTOM is the use of the XOP:Include element,
which is defined in the XML Binary Optimized Packaging (XOP)
specification to reference the binary attachments (external unparsed
general entities) of the message. With the use of this exclusive
element, the attached binary content logically becomes inline (by value)
with the SOAP document even though it is actually attached separately.
This merges the two realms by making it possible to work only with one
data model. This allows the applications to process and describe by only
looking at the XML part, making the reliance on DTDs obsolete. MTOM has
also standardized the referencing mechanism of SwA.

The basic ideas of using XOP packaged data are that:

1.  Binary data can be included alongside plain-text XML data, and the
    XOP package processing would take care of the optimization (which
    reduces the size of the original Base64 encoded data,

2.  The internal references would be resolved, and

3.  All levels of software above the XOP package management would not
    have to worry about managing the binary data either on the encoding
    or the decoding side.

Note that at the conceptual level, this binary data can be thought of as
being base64-encoded in the XML Document. As this conceptual form might
be needed during some processing of the XML document (e.g., for signing
the XML document), it is necessary to have a one-to-one correspondence
between XML Infosets and XOP Packages. Therefore, the conceptual
representation of such binary data is, as if it were base64-encoded,
based on the canonical lexical form of the XML Schema base64Binary
datatype. In the reverse direction, XOP is capable of optimizing only
base64-encoded Infoset data that is in the canonical lexical form.

**[3.7 Hierarchical Data Format]{.underline}**

HDF (Hierarchical Data Format) \[9\] is a data format first developed in
the 1980s and currently in Release 4.x (HDF Release 4.x). HDF version 5,
or HDF5, is a new data format first released in Beta in 1998 and
designed to better meet the ever-increasing demands of scientific
computing and to take better advantage of the ever-increasing
capabilities of computing systems. HDF5 is currently in Release 1.x
(HDF5 Release 1.x). It was created by the NCSA (National Center for
Supercomputing Applications) and is a data model and a programming model
that consists of:

-   a data format specification (multi object file format).

-   a supporting library implementation.

HDF5 supports all types of data stored digitally, regardless of origin
or size. Examples include petabytes of remote sensing data collected by
satellites, terabytes of computational results from nuclear testing
models, and megabytes of high-resolution MRI brain scans can be stored
in HDF5 files, together with metadata necessary for efficient data
sharing, processing, visualization, and archiving. Similar to XML
documents, HDF files are self-describing and allow users to specify
complex data relationships and dependencies. But, in contrast to XML
documents, HDF files can contain binary data (in many representations)
and allow direct access to parts of the file without first parsing the
entire contents. It also supports a rich set of pre-defined datatypes as
well as the creation of an unlimited variety of complex user-defined
datatypes. Datatype definitions include information such as byte order
(endian), size, and floating point representation, to fully describe how
the data is stored, insuring portability to other platforms.

In general, HDF5 files are organized in a hierarchical structure, with
two primary structures: *groups* and *datasets*. An *HDF5 group* is a
structure containing zero or more HDF5 objects. Each group has two
parts: (i) a group header, which contains a group name and a list of
group attributes; and (ii) a group symbol table, which is a list of the
HDF5 objects that belong to the group. An *HDF5 dataset* is stored in a
file in two parts: a header and a data array. The header contains
information that is needed to interpret the array portion of the
dataset, as well as metadata (or pointers to metadata) that describes or
annotates the dataset. Header information includes the name of the
object, its dimensionality, its number-type, information about how the
data itself is stored on disk, and other information used by the library
to speed up access to the dataset or maintain the file\'s integrity. The
Figure 10 below shows a UML representation of the HDF5 data model.

![](media/image6.png){width="5.260416666666667in"
height="5.260416666666667in"}

> Figure 10 UML representation of HDF5 data model

# [4. Binary Data Format Software Implementations]{.underline}

In the following sections, we will introduce a number of binary data
software that act as reference implementations to some of the languages
introduced in the previous sections.

**[4.1 IBM Virtual XML Garden]{.underline}**

Virtual XML is the ability to view and process any data - whether XML or
non-XML - as though it is XML, and in particular allow use of XML
processing languages, such as XPath and XQuery, on the data. IBM Virtual
XML Garden \[11\] adopts this approach for an XML-based virtualization
of data resources. As part of this approach, it separates the
representation of the data from the processing model; that is, we
describe how to support processing of non-XML data as though it is XML
and without explicit conversion to XML. The benefits associated with the
virtual XML approach include leveraging existing interfaces, tools,
knowledge, and communities of interest and also the ability to process
both non-XML and XML data in a uniform way by using XML processing
languages.

To achieve this, IBM Virtual XML Garden adopts W3C XQuery 1.0 and XPath
2.0 data models, which extends the XML Infoset with new features to meet
the requirements for the new XML processing languages such as XPath 2.0,
XSLT 2.0 and XQuery 1.0. An important element of IBM Virtual XML is to
adopt languages that allow XML representation of data in all forms.
IBM's implementation is based on the Data Format Description Language
(DFDL) introduced earlier. Expectedly, an important requirement for
leveraging the power of DFDL is to efficiently implement processing of
the descriptions and ultimately the data. IBM Virtual XML Garden
achieves this by (1) implementing the processing languages lazily such
that processing is driven in an "on demand" fashion by the consumer
accessing the processing result, and (2) analyzing expressions and
queries to understand the data profile properties of each component. The
technical details of how this is achieved are beyond the scope of this
white paper and are available at \[11\]. More details are available at:
<http://www.research.ibm.com/virtualxml/>

**[4.2 Defuddle]{.underline}**

Defuddle \[12\] is a reference implementation for the DFDL specification
\[2\]. It supports translation and metadata extraction of arbitrary text
and binary data through the use of DFDL schema descriptor files. It also
optionally supports the application of style sheets to the output.

The Defuddle parser leverages existing tools for automatically parsing
XML documents within the context of a logical model. It is based on the
JAXB specification which provides a convenient way to use XML Schema to
automatically parse XML instance documents into Java classes
corresponding to that schema. Thus, from a design standpoint, this
solution provides an off-the-shelf ingestion engine for the logical
model (XML Schema), a dynamic logical model compiler (Java classes), and
an XML document generator for streaming data from the classes to XML.
Specifically, at run-time, the DFDL schema is ingested and processed to
generate Java classes representing components of the logical model.
These classes are then compiled using the standard Java compiler. The
translation of the input data source(s) is then initiated using the JAXB
XML marshaller. As the java objects are streamed by the parser, the
logical model is formed by loading the required values from the data
file. The XML model is streamed to an output that can then be processed
by standard XML tools. The class generation process is performed
automatically before translations are performed, but the compiled
classes can be cached to improve performance on subsequent runs. More
details on the software are available at: <http://defuddle.pnl.gov/>

**[4.3 Earth Science Markup Language Implementation]{.underline}**

ESML is an interchange technology that enables data (both structural and
semantic) interoperability with applications without enforcing a
standard format within the Earth science community \[4\]. Users can
write external files using ESML schema to describe the structure of the
data file. Applications can utilize the ESML Library to parse this
description file and decode the data format. As a result, software
developers can now build data format independent scientific applications
utilizing the ESML technology. Furthermore, semantic tags can be added
to the ESML files by linking different domain ontologies to provide a
complete machine understandable data description. This ESML description
file allows the development of intelligent applications that can now
understand and \"use\" the data.

The ESML implementation at its basic comprises of the ESML schema, an
editor for modeling the schema elements and the ESML library that parses
the ESML description files for processing the information. The latest
implementation released by the group was in 2006 and no more active code
development is pursued at present. Furthermore, it is unclear about how
one would associate concepts from ontology to ESML descriptions; at
least, we did not find any existing study on this topic. More details
are available at: <http://esml.itsc.uah.edu/>

**[4.4 Binary XML Implementation]{.underline}**

BinX is an XML schema-based language with which to describe the layout
of binary data files. It is designed to enable transparent transfer of
data between diverse platforms. The BinX library allows the reading and
writing of BinX XML files and the associated binary data. It supports
(or will support) functionality to:

-   Browse -- read data from binary files

-   Extract -- select partial data from the dataset

-   Transform -- use XSL to transform structure of binary data

-   Create -- create binary data and the associated BinX descriptions

The entire library is written in C++. The existing implementation also
comprises of a visualization tool called BinX editor (BinXed) to assist
in designing BinX documents. These documents, as shown in Section 3.5,
is an XML text file based on the BinX mark-up language for the purpose
of describing a binary data file. More details on the software are
available at: <http://www.edikt.org/binx/>

**[4.4 Message Transmission Optimization Mechanism
Implementation]{.underline}**

MTOM (SOAP Message Transmission Optimization Mechanism) is a
specification that enables transmission of \"attached\" binary data to
XML messages. It leverages the XOP:Include element, defined in the XML
Binary Optimized Packaging (XOP) specification, to reference the binary
attachments (external unparsed general entities) of the message such
that the attached binary content logically become inline (by value) with
the SOAP document even though actually it is attached separately.

For our purposes, we investigated the Apache Axis2 project \[13\] which
is a reference implementation of the SOAP (\"Simple Object Access
Protocol\" \[14\]) submission to W3C. Axis2 comes with its own
light-weight object model, AXIOM, for message processing which is
extensible, highly performant and is developer convenient. It is based
on an XML infoset model which refers to the information included inside
the XML and for programmatical manipulation it is convenient to have a
representation of this XML infoset in a language specific manner. For an
object oriented language the obvious choice is a model made up of
objects. For example, DOM and JDOM are two such XML models. AXIOM
leverages a differed built XML info set representation based on StAX
(JSR 173), which is the standard streaming pull parser API. Thus, the
object model can be manipulated as flexibly as any other object model
(Such as JDOM), but underneath the objects will be created only when
they are absolutely required. This leads to much less memory intensive
programming. AXIOM also has the ability to hold binary data. This is
enabled by allowing AXIOM to hold raw binary content in the form of
javax.activation.DataHandler on which MTOM is based. Consequently, using
AXIOM one can send base64encoded data as well as externally attached raw
binary data referenced by \"XOP\" element (optimized content) in a SOAP
message. Additionally, the user can specify whether the binary data or
base64encoded binary data is qualified to be optimized or not at the
construction time of that node or later. In general, to take the optimum
efficiency of MTOM a user is advised to send smaller binary attachments
using base64encoding (none optimized) and larger attachments as
optimized content. More details are available at:
<http://ws.apache.org/axis2/>

**[4.5 Hierarchical Data Format Language Implementation]{.underline}**

HDF (Hierarchical Data Format) technologies are relevant when the data
challenges being faced push the limits of what can be addressed by
traditional database systems, XML documents, or in-house data formats.
Many HDF adopters have very large datasets, very fast access
requirements, or very complex datasets. Others turn to HDF because it
allows them to easily share data across a wide variety of computational
platforms using applications written in different programming languages.
Some use HDF to take advantage of the many open-source and commercial
tools that understand HDF.

HDF technologies at present include two data management formats (HDF4
and HDF5) and libraries, a modular data browser/editor, associated tools
and utilities, and a conversion library. Both HDF4 and HDF5 were
designed to be a general scientific format, adaptable to virtually any
scientific or engineering application, and also have been used
successfully in non-technical areas. The Open Source format is a key
technological foundation for HDF core technologies. It allows users to
collaborate regarding functionality requirements and permits users\'
experience and knowledge to be incorporated into the HDF product when
appropriate. It also permits users and organizations without adequate
technology resources to use a sophisticated and robust data management
tool for no charge. HDF5 is particularly good at dealing with data where
complexity and scalability are important. Data of virtually any type or
size can be stored in HDF5, including complex data structures and data
types. HDF5 is portable, running on most operating systems and machines.
HDF5 is scalable - it works well in high end computing environments, and
can accommodate data objects of almost any size or multiplicity. It is
also efficient, providing fast access to data, including parallel I/O.
It also can store large amounts of data efficiently - it has built-in
compression, or applications can also provide their own special-purpose
compression. HDF4 and HDF5 are both widely used in government, academia,
and industry. There are more than 200 distinct applications of the
formats and an estimated 1.6 million users of NASA data alone. It is
also the base format for a number of community standards, such as
HDF-EOS, the standard for NASA\'s enormous Earth Observing System
http://eospso.gsfc.nasa.gov/, and NeXus, the standard for Neutron, Xray
and Muon Science (<http://www.nexus.anl.gov/index.html>). More details
are available at: <http://hdf.ncsa.uiuc.edu/>

# [5. caBIG™ Case Studies]{.underline}

# 

# The following two case studies, contributed by the members of the caBIG community, describe the need for leveraging descriptive languages during transportation of bulk data.

**[5.1 Use Case# 1: Querying for Imaging Data]{.underline}**

# The diagram in Figure 11, provided by the IVI middleware team, details the imaging use case of 

![](media/image7.jpeg){width="6.500694444444444in"
height="3.2916666666666665in"}

> Figure 11 Sequence diagram depicting querying for images using GridFTP

# querying their Data Service as well as using caGrid BDT to return images using GridFTP (binary transport). The standard Data Service query mechanism (shown in the top QUERY portion of the diagram) maintains all data semantics as the data types are fully described and semantically annotated with the standard Data Service metadata, and data returned from the service is in the XML form of those data types. The mechanism used to return binary images (shown in the bottom RETRIEVE portion of the diagram), however, has a potential loss of semantics, as the content of binary data returned by GridFTP is not described by metadata.

More specifically, the following is the flow of events based on the
above diagram:

1.  The caGrid Client issues a CQL query to the caGrid Data Service, and
    specifies the data should be returned with BDT/GridFTP.

2.  The caGrid Data Service prepares the query results and creates a BDT
    resource for them, associating metadata describing the result type
    (referencing the data service metadata), as well as metadata
    describing the binary data format that will be used to transport
    them over GridFTP.

3.  The caGrid Client (or any client which will be retrieving the data)
    may inspect the BDT resource's metadata to determine the semantics
    of the contained data, as well as the expected binary format of the
    data to be retrieved.

4.  The caGrid Client then retrieves the data in its binary format,
    using GridFTP.

5.  The caGrid Client can then either process the data in its binary
    format, or use the metadata to transform the results back into their
    XML representation.

**[5.2 Use Case# 2: Querying for Microarray Data]{.underline}**

A user queries the caArray data service experiment metadata for
experiments of interest. The results are some sort of pointer to the raw
experiment data, which takes the form of a binary file in CEL format
(for Affymetrix-based experiments). The format is described at
<http://www.stat.lsa.umich.edu/~kshedden/Courses/Stat545/Notes/AffxFileFormats/cel.html>
(look at the version 4 format). CEL data exposed on the grid should be
well formatted and semantically annotated so that users can consume of
the data and discover it if desired. Specifically, there are a number of
fields, which are described and have a particular datatype (integer,
DWORD, etc.). Users of grid services exposing this data should be able
to discover (1) which services are able to transmit CEL data through a
high-throughput mechanism, and (2) what the semantic contents are of the
data.

On a slightly different note, in the domain of next generation
sequencing techniques, technologies such as Roche-454 use binary SFF
data files. These are also well formatted as the CEL file discussed
above, but are typically more bulkier (the format can be accessed from:
<http://www.ncbi.nlm.nih.gov/Traces/trace.cgi?cmd=show&f=formats&m=doc&s=formats#sff>).
Consequently, use case #2 can be extended to work with SFF files as
well. Similarly, one disadvantage of using state-of-the-art high
throughput technologies, such as Illumina-Solexa, is the transfer of a
large number of TIFF image files from the instrument to the workstation
where actual processing takes place. The total size of the image files
can be upto 700GB and expectedly transfer of such bulk data is a time
consuming process. Within this context, the imaging use-case #1 relates
to utilizing GridFTP for images data transfer, and its applicability to
such data will be tremendously useful for Illumina-Solexa sequence
analysis.

[**6. Recommendations for caBIG**^TM^ **Binary Data Format Languages and
Software**]{.underline}

The need for leveraging \"descriptive languages\" for binary data within
caBIG is unprecedented, as had been demonstrated in the aforementioned
use cases. However, each need comes with a different set of concerns and
priorities. In such a setting, a universally useful solution will have
to be described in fairly fundamental terms. Furthermore, any solution
recommended for caBIG should effectively leverage the existing caBIG
knowledge based infrastructure that is expressed in terms of CDEs, UML
models, Interfaces and immutable concepts. In particular, the
recommendation has to consider caDSR's utility and how to maximally
exploit it. The use of caDSR-derived, ontology-anchored knowledge
structures (e.g., the NCI Thesaurus) when designing caBIG applications
provides a means for semantic interoperability between such an
application and others within the caGrid framework.

Based on our evaluation of the binary data format languages and software
introduced in the previous sections, we have developed a "feature
matrix" to compare them and base our recommendation for further
evaluation, prototyping, and eventual implementation within the caGrid
infrastructure. The various elements of the matrix are:

-   Current Status: this indicates whether the particular project/team
    responsible for developing the language/software is active or not.

-   Licensing: this indicates the type of license (open source,
    commercial, limited etc.) used to distribute the language/software
    under consideration.

-   Maturity: this gives an estimate of the development timeframe for a
    particular language/software.

-   Semantic Describability: this indicates how uniformly ontologies and
    vocabularies can be leveraged in associating semantic information to
    the data descriptions.

-   Representative User Community: this indicates whether or not the
    language and its implementation have been adopted by various
    communities and how they are used in different applications.

-   Developer Support: this indicates if there are any tutorials or
    documentation relevant to the language or software that can be
    readily used by developers for prototyping.

-   Grid Compatibility: this indicates whether the language/software
    will be compatible to grid infrastructure, and caGrid, in
    particular.

From both matrices, we made the following observations:

**Binary Data Format Languages:** Out of all the languages that were
considered for evaluation, we believe that the Data Format Description
Language (DFDL) and Earth Science Markup Language (ESML) are the ones
that satisfy most of the requirements in the "feature matrix". In
particular, we consider DFDL to be a front-runner since (i) it is
developed by the Open Grid Forum with inputs and participation from
caBIG community members, (ii) is under active development, and (iii) is
based on use cases and potential applications expanding multiple
communities (as opposed to ESML which is focused only on the earth
science community). Additionally, the DFDL working group \[2\] grew out
of Binary XML Language (BinX) which has been widely used and actively
developed from 2003---2005. With respect to Bitstream Syntax Description
Language (BSDL), we observed that even though the language is relevant
in the context of "describing data", it is more tightly connected to
only bitstream data. A similar observation was also made with respect to
the Message Transmission Optimization Mechanism (MTOM) which is heavily
focused on addressing binary data attachment problems in Web services
(i.e., SOAP with attachments) and does not inherently provide any means
of describing binary data, nor does it propose a binary format for XML
documents. MTOM/XOP is merely a way of serializing XML in a way that
would make it more efficient to include Based64-encoded data.
Furthermore, MTOM has scalability limitations when dealing with large
binary data object. On a different note, the Hierarchical Data Format
(HDF) has the capability to represent a variety of data types and
provided techniques for efficient encoding of the data. However, the HDF
community, unlike DFDL, has not well-addressed the issue of
(semantically) describing the data.

  ---------------- -------------------------------------------- ---------------------------------------------- --------------------------- ----------------------------------------- ---------------------------- --------------------------- -----------------------------------
  **Languages/     **BFDL**                                     **DFDL**                                       **ESML**                    **BSDL**                                  **BinX**                     **HDF**                     **MTOM**
  Standards**                                                                                                                                                                                                                                 

  Name             Binary Format Description Language           Data Format Description Language               Earth Science Markup        Bitstream Syntax Description Language     Binary XML Language          Hierarchical Data Format    Message Transmission Optimization
                                                                                                               Language                                                                                                                       Mechanism

  Web Link         http://collaboratory.emsl.pnl.gov/sam/bfd/   http://forge.gridforum.org/projects/dfdl-wg/   http://esml.itsc.uah.edu/   http://www2002.org/CDROM/alternate/334/   http://www.edikt.org/binx/   http://hdf.ncsa.uiuc.edu/   http://www.w3.org/TR/soap12-mtom/

  Current Status   Inactive since 2003                          Active                                         Inactive since 2006         Inactive since 2003                       Inactive since 2005          Active                      Active

  Maturity         Developed since 2000 till 2003               Developed since 2003 till date                 Developed since 2003 till   N/A                                       Developed since 2003 till    Developed since early 90's  Developed since 2005 till date
                                                                                                               2006                                                                  2005                         till date                   

  Semantic         N/A                                          Allows semantic labeling of individual data    Allows elements to be       N/A                                       N/A                          N/A                         N/A
  Describability                                                objects and groups using meta-data elements.   hooked up to concepts from                                                                                                     
                                                                                                               an ontology                                                                                                                    

  Representative   N/A                                          Open Grid Forum???                             Earth Science community     N/A                                       Astronomy, Bioinformatics,   High energy physics,        Web services???
  User Community                                                                                                                                                                     Particle Physics,            Aeronautics, Atmospheric    
                                                                                                                                                                                     CAD/CAM/CAE.                 science                     
  ---------------- -------------------------------------------- ---------------------------------------------- --------------------------- ----------------------------------------- ---------------------------- --------------------------- -----------------------------------

> Table 1 "Feature Matrix" for Binary Format Description Languages
>
> Table 2 "Feature Matrix" for Binary Format Description Language
> Implementations

**Binary Data Format Languages Implementations:** Since DFDL is one of
the languages that fit our requirements, naturally its reference
implementations, namely IBM Virtual XML Garden and PNNL Defuddle, were
relevant. Both the implementations are actively developed and
well-supported, although Virtual XML is based on IBM Alphaworks license
which might limit its adoption in the open-source world. Additionally,
ESML implementation met the requirements although it is no longer under
development and support. Both HDF and MTOM implementations, HDF5 and
Apache Axis 2, respectively, are very widely used by different
communities and actively developed. They also have very good support and
documentation to assist in building applications. The BinX
implementation, on the other hand, lacks these aspects and in some sense
remained more of an "academic pursuit".

Based on the observations made by our group, we believe that there are
many statements of further work that needs to be accomplished. A summary
of those tasks is presented here.

1.  **Further Evaluation of Binary Data Format Languages and Software:**
    as identified in this white paper, our small working group, due to
    limited resources and time constraints, barely scratched the surface
    in evaluating the existing binary data format languages and
    software. We believe that this evaluation should be continued, in
    particular, with the participation of members responsible for
    development of the languages/software to get their feedback and
    (expert) opinions in understanding the details. One aspect that
    warrants further investigation is the issue of whether or not the
    languages/software are caGrid compatible, and if not, what can be
    done to achieve the desired compatibility level.

2.  **Experimentation and Developing Prototypes:** Our current
    evaluation did not include any prototyping efforts. We believe this
    is an important requirement, in particular, to measure the
    scalability and performance related issues. Furthermore, such
    efforts will provide hands-on experience in addressing the
    compatibility issues.

3.  **Active Engagement of the Community:** We believe that successful
    incorporation of "descriptive languages" within the caGrid
    infrastructure will heavily impact how different types of data are
    interpreted. Consequently, this will influence the development of
    various applications that leverage the caGrid SDK and its associated
    software components. Hence, engaging the wider caBIG community
    during the various phases of evaluation, implementation and adoption
    of descriptive languages in caGrid will be beneficial.

# [7. Conclusions]{.underline} 

There exist compelling use cases within the caBIG community that could
greatly benefit from leveraging binary data format languages and
software. The existence of multiple, and often overlapping technologies
necessitates the selection of a sufficiently extensible and evolvable
system that is both able to meet currently perceived need, and that will
also be able to adapt to future need and technologies. The
recommendations provided in Section 6, which include further evaluation
and prototyping with existing binary data format languages and software
system should provide such benefits to the caBIG community.

# [8. References]{.underline}

[^1]: Note that there are many elements of the DFDL schema such as
    applies="hereOnly" that are applied when generating a DFDL
    annotation. A detailed discussion of the DFDL schema is beyond the
    scope of this paper and more information is available at:
    <http://forge.gridforum.org/projects/dfdl-wg/>