# Presentation: 950215

## Binary Data Format (BDF)

**Binary Data Format (BDF)**

**ARCH/VCDE Small Working Group**
**03/28/2008**

- _[https://](https://gforge.nci.nih.gov/projects/binary/)__[gforge.nci.nih.gov](https://gforge.nci.nih.gov/projects/binary/)__[/projects/binary/](https://gforge.nci.nih.gov/projects/binary/)_** **

## Background: Data Interchange Standards

**Data Interchange Standards are required for enabling interoperability between multiple information systems**
**2 types of data interchange standards exist:**
- Prescriptive standards: “put data in this format!”
- Textual – XML
- Binary – ASN.1, XDR, EBML
- You use the defined encodings and syntax
- Descriptive standards: “choose your own format!”
- Textual, binary, mixture
- You use your own encodings and syntax
- No common description

- [DFDL OGF-20, May 2007]

## Background: Data Interchange Formats

**There will never be a standard data **_***format***_
- Legacy formats
- Application specific formats
- eXtensible formats (XML) doesn’t help
- verbose, tree-based
- One size will never fit all
**But we could provide a language for **_***describing ***_**formats**
- Transparency of physical representation
- Unambiguous description of data
- Automatic format conversion

- [DFDL OGF-20, May 2007]

## Requirements: Descriptive Standards

**Data format independent**
- Textual, Binary...
**Descriptions can be shared**
- “Machine interpretable” and “Human readable”
**Builds on existing standards**
- Leverage XML technology and concepts
**Supports move to SOA and grid computing**
- Interoperability
**Spans commercial and scientific worlds**

## Objectives of BDF Working Group

**Evaluate existing descriptive data formats/languages**
**Evaluate existing descriptive data format software and implementations**
- E.g., efficient parsers/formatters
**Develop or leverage existing use-cases**
- E.g., Imaging and Microarray
**Propose a recommendation**
- White paper

## Example of OGF DFDL: XML

**<w>****5****</w>**
**<x>****7839372****</x>**
**<y>****8.6E-200****</y>**
**<z>****-7.1E8****</z>**

## Example of OGF DFDL: XSD

**<xs:complexType name=“example1”>**
**	****<xs:sequence>**
**	****	****<xs:element name="w" type=“int"/>**
**	****	****<xs:element name="x" type=“int"/>**
**	****	****<xs:element name="y" type=“double"/>**
**	****	****<xs:element name="z" type=“float"/>**
**	****</xs:sequence>**
**</xs:complexType>**

## Example of OGF DFDL: Binary

**0000 0005 0077 9e8c **
**169a 54dd 0a1b 4a3f **
**ce29 46f6**

## Example of OGF DFDL: Binary DFDL

**<xs:complexType name="example1">**
**  ****<xs:annotation>**
**	****  ****<xs:appinfo source="http://www.ogf.org/dfdl/v1.0"> **
**	****          ****<dfdl:format repType="binaryInteger"**
**                         ****byteOrder="bigEndian”**
**	****	****	****	****	**** ****lengthKind=“fixed”**
**	****	****	****	****	**** ****length=“4”/>**
**	****  ****</xs:appinfo>**
**	****</xs:annotation>**
**	****<xs:sequence>**
**	****	****<xs:element name="w" type="int"/>**
**	****	****<xs:element name="x" type="int“/>**
**	****	****<xs:element name="y" type="double“**
**	****	****	****	****   ****dfdl:length=“8”**** />**
**	****	****<xs:element name="z" type="float“/>**
**	****</xs:sequence>**
**</xs:complexType>**

## Binary Data Format Languages Evaluated

**BFDL: Binary Format Description Language**
**DFDL: Data Format Description Language**
**ESML: Earth Science Markup Language**
**BSDL: Bitstream Syntax Description Language**
**BinX: Binary XML Language**
**HDF: Hierarchical Data Format**
**MTOM: Message Transmission Optimization Mechanism**

## Binary Data Format Software Evaluated

**IBM Virtual XML Garden (DFDL)**
**PNNL Defuddle (DFDL)**
**ESML Implementation (ESML)**
**BinX Implementation (BinX)**
**HDF Implementation (HDF)**
**Apache Axis 2 (MTOM)**

## Use Cases: Querying for Imaging/Microarray Data

## Use Cases: Querying for Imaging/Microarray Data

**Querying procedure:**
- caGrid client issues a CQL query to the caGrid data service, and specifies that the data should be returned with BDT/GridFTP
- caGrid data service prepares the query results and creates a BDT resource. Associates metadata describing:
- Result type (referencing the data service metadata)
- Binary data format used to transport the results over GridFTP
- Client inspects the BDT resource’s metadata to determine the semantics of the contained data, as well as the expected binary format of data to be retrieved

## Use Cases: Querying for Imaging/Microarray Data

- caGrid client retrieves data in its “original” binary format using GridFTP
- Client either processes the binary data or uses metadata to transform into an XML representation

## Evaluation Criteria

**Current Status: **indicates whether a particular language/software is actively developed or not
**Licensing: **indicates the type of license (open-source, commercial, limited etc.) used to distribute the language/software
**Maturity: **gives an estimate of the development timeframe of the language/software
**Semantic Describability: **indicates how uniformly ontologies can be leveraged in associating semantic information to data descriptions
**User Community: **indicates how widely the language/software is adopted and by what types of user communities
**Developer Support: **indicates whether the language/software is actively supported; availability of documentation (tutorials, API docs etc.)
**Grid Compatibility: **is the language/software compatible to grid infrastructure, and in particular, caGrid?

## Evaluation Results - 1

| Languages/ Standards | BFDL | DFDL | ESML |
| --- | --- | --- | --- |
| Name | Binary Format Description 
Language | Data Format Description 
Language | Earth Science Markup 
Language |
| Web Link | http://collaboratory.emsl.pnl.gov/sam/bfd/ | http://forge.gridforum.org/projects/dfdl-wg/ | http://esml.itsc.uah.edu/ |
| Current Status | Inactive since 2003 | Active | Inactive since 2006 |
| Maturity | Developed since 2000 till 
2003 | Developed since 2003 till
 date | Developed since 2003 till 
2006 |
| Semantic
Describability | N/A | Allows semantic labeling
of individual data objects
and groups using meta-
data elements. | Allows elements to be 
hooked up to concepts from an ontology |
| Representative User 
Community | N/A | Open Grid Forum | Earth Science community |

## Evaluation Results - 1

| Languages/
Standards | BSDL | BinX | HDF | MTOM |
| --- | --- | --- | --- | --- |
| Name | Bitstream Syntax 
Description Language | Binary XML Language | Hierarchical Data 
Format | Message 
Transmission 
Optimization 
Mechanism |
| Web Link | http://www2002.org/CDROM/alternate/334/ | http://www.edikt.org/binx/ | http://hdf.ncsa.uiuc.edu/ | http://www.w3.org/TR/soap12-mtom/ |
| Current Status | Inactive since 2003 | Inactive since 2005 | Active | Active |
| Maturity | N/A | Developed since 2003
 till 2005 | Developed since early
 90’s till date | Developed since 2005
 till date |
| Semantic
Describability | N/A | N/A | N/A | N/A |
| Representative User
 Community | N/A | Astronomy,
 Bioinformatics, 
Particle Physics,
 CAD/CAM/CAE. | High energy physics, 
Aeronautics,
 Atmospheric science | Web services |

## Slide 18

| Software | IBM Virtual XML Garden | PNNL Defuddle | ESML Implementation |
| --- | --- | --- | --- |
| Reference implementation of which language? | Data Format Description Language | Data Format Description Language | Earth Science Markup Language |
| Web Link | http://collaboratory.emsl.pnl.gov/sam/bfd/ | http://forge.gridforum.org/projects/dfdl-wg/ | http://esml.itsc.uah.edu/ |
| Current Status | Active | Active | Inactive since 2006 |
| Licensing | IBM Alphaworks | Apache License 2.0 | GNU LGPL 2.1 |
| Maturity | Developed since 2004 till 
date | Developed since 2005 till date | Developed since 2003 till 2006 |
| Developer Support/ Usability | Good;  developer documentation is available | Average | Good; developer document is available |
| Grid Compatibility | Supports OGF developed DFDL | Reference implementation of OGF DFDL | N/A |

## Evaluation Results - 2

| Software | BinX Implementation | HDF Implementation | Apache Axis 2 |
| --- | --- | --- | --- |
| Reference implementation of which language? | Binary XML Language | Hierarchical Data Format | Message Transmission Optimization Mechanism |
| Web Link | http://www.edikt.org/binx/ | http://hdf.ncsa.uiuc.edu/ | http://ws.apache.org/axis2 |
| Current Status | Inactive since 2005 | Active | Active |
| Licensing | University of Edinburgh | N/A | Apache License 2.0 |
| Maturity | Developed since 2003 till 2005 | Developed since early 90’s till date | Developed since 2002 till date |
| Developer Support/ Usability | Poor | Good; developer documentation is available | Good; developer documentation is available |
| Grid Compatibility | N/A | N/A | N/A |

## Observations: Languages

**DFDL and ESML satisfy most requirements in the “feature matrix”; DFDL is the front-runner:**
- Developed by Open Grid Forum with inputs from caBIG community members
- Under active development
- Based on much broader use cases and applications
- Leverages experience/lessons learned from BinX
**BSDL even though relevant for “describing data”, is more tightly connected to bitstream data**
**MTOM is focused on dealing with binary data attachments (e.g., SOAP with attachments)**
**HDF enables representation of variety of data type and efficient encodings; not semantically describing data**

## Observations: Software

**IBM Virtual XML Garden and PNNL Defuddle are reference implementations of DFDL**
- Both are active
- However, Virtual XML is based on IBM Alphaworks license
**ESML implementation meets the requirements, but no longer under active development**
**HDF5 and Apache Axis 2 are not highly relevant, although both have good user support and are active**
**BinX implementation seemed more “academic” and is inactive since 2005**

## Recommendations

**Further Evaluation of Languages and Software**
- Limited time and resources for the WG
- Invite participation of members responsible for development of languages/software
- Investigate caGrid compatibility
**Experimentation and Prototyping**
- Adopt a particular language/software and leverage an existing use case
- Study scalability and performance related issues
**Community Engagement**
- Engage broader caBIG community during various phases of evaluation, implementation and adoption

## Whitepaper and other documents

**Whitepaper:**
- _[https://gforge.nci.nih.gov/docman/view.php/455/10992/BDF-Whitepaper-Final.pdf](https://gforge.nci.nih.gov/docman/view.php/455/10992/BDF-Whitepaper-Final.pdf)_
**Presentations, charter, meeting minutes etc.**
- _[https://gforge.nci.nih.gov/docman/?group_id=455](https://gforge.nci.nih.gov/docman/?group_id=455)_

## Participants

**Mayo Clinic**
- Jyotishman Pathak
- Sumit Middha
**Duke University**
- Patrick McConnell
- Pankaj Agarwal
**Northwestern University**
- Dong Fu
**Ohio State University**
- Scott Oster
- Shannon Hastings
**Booz Allen Hamilton**
- Mike Keller

## Discussion

**Q & A**

- _[https://](https://gforge.nci.nih.gov/projects/binary/)__[gforge.nci.nih.gov](https://gforge.nci.nih.gov/projects/binary/)__[/projects/binary/](https://gforge.nci.nih.gov/projects/binary/)_** **