# Presentation: 199044

## Automated Network Synchronization of NetCDF Datasets

**FY2004 HPCC Funded Proposal Report**
**L. Charles Sun, NODC**
**Joe Sirott, PMEL**
**Don Denbo, PMEL**

- FY2005 HPCC Annual Review Meeting May 31 – June 2, 2005, PMEL, Seattle, WA

## Acknowledgements

**The NOAA HPCC Program Office**
**Co-PIs**
- Joe Sirott, PMEL (Developer)
- Donald W. Denbo, PMEL
**Argo/GTSPP Team**
- Norman Hall, NODC
- Mike Simmons, NODC
- Melanie Hamilton, NODC
- Shannon Niou, STG Int’l

## Outline

**Argo Project Background**
**Argo Data Management System**
**Problem Statement and Solution**
**Future Direction**
**Publications**

## Argo Float Operation

**Argo Float Operation**

## Argo Data Management System

**Argo Data Management System**

**1. Argo Data Assembly Centers (DACs):**

**2. Argo Global Data Assembly Centers (GDACs)**

**3. Global Argo Data Repository - NODC**

- NODC

## Data Transferring Method: FTP

**Data Transferring Method: FTP**

**NODC**

**FNMOC**

**FTP**

**Notes:**
After the Web server at PMEL searches data sets from the Data Servers at NODC and JOSS, it then returns the results that meet users search criteria to users.

## Global Argo Data Repository

**http://www.nodc.noaa.gov/argo**

## Argo Data Usage

## Data Transferring Problem

**Data Transferring Problem**

**NODC**

**FNMOC**

**FTP**

**Notes:**
After the Web server at PMEL searches data sets from the Data Servers at NODC and JOSS, it then returns the results that meet users search criteria to users.

## Rsync Algorithm

**Client divides file into blocks**
**Calculates a hash based signature for each block**
**Client sends signatures to server**
**Server compares signatures from client and only sends data that isn’t already on client**

## CdfSync Enhancements

**Take advantage of netCDF block structure**
**Compress file metadata for efficient updates of large number of small files**
**In-place updates for small updates to large files**

## Benchmark I: Use NetCDF Blocks 

**Synchronize identical 512 MB netCDF files, compare with synchronization of identical 512 MB non-netCDF files. **
**Use –in-place to measure only disk reads, not writes:**
**Rsync: 105 sec**
**Cdfsync: 72 sec  **

**Speedups of 1.45**

## Benchmark II: compressed file list

**1.5 million identical netCDF files over low bandwidth (256 Kb/sec) link**
**Rsync: 910 sec**
**Cdfsync: 206 sec**

**Speedups of 4.41**

## Benchmark III:in-place updates

**1.5 GB netCDF file with extra data appended to record dimension:**
**Rsync: 434 sec**
**Cdfsync: 175 sec**

**Speedups of 2.48**

## Availability

**http://www.epic.noaa.gov/epic/software/cdfsync****/**

## Future Direction I

**Disseminate Cdfsync at the NODC Web site, after security review by the NODC Web team.**
**Report the success of this project to the international Argo Management Team at the upcoming 6th Argo Data Management Team (ADMT) Meeting, November 8 to 11, 2005, Tokyo, Japan.**

## Future Direction II

**Seek mirror sites of Serving Argo data, using cdfsync for data transferring**
**Possible sites:**
**PMEL**
**Asia Pacific Data Research Center (APDRC) at the Univ. of Hawaii **
**Taiwan National Center for Ocean Research (NCOR)**
**Seek support for cdfsync protocol to Dapper OPeNDAP server.**
**Add Web client and incorporate user feedback in future releases. **

## Publication

**Sirott, J., L.C.Sun, and D.W. Denbo, 2005, **_**Networked Synchronization of netCDF Datasets**_**, Proceedings of the 22st International Conference on Interactive Information and Processing Systems (IIPS) for Meteorology, Oceanography, and Hydrology, 2005 AMS Annual Meeting, San Diego, CA. AMS 2005 http://ams.confex.com/ams/pdfpapers/87337.pdf**