# Presentation: 458163

## Experiences in using the CDP to support Scientific Data Workflows

- Presented by:
- Michael Burek

## Acknowledgments 

- Luca Cinquini
- Don Middleton
- Louisa Emmons

## Introduction

- In early 2006 the CDP was selected to support some of the data archiving requirements for the Milagro project
- The Milagro field campaign was a decentralized collection effort that required coordination of data collection efforts throughout the campaign
- The role of the CDP was to collect data from field operations and make them available to all of the researchers

## Introduction to Milagro

- Milagro is an international effort to characterize atmospheric pollution in large cities, Mexico City in particular.
- Measurements were made in the city and in likely paths of atmospheric movement to study dispersion and evolution of pollutants
- Data collection was done from 7 aircraft, 14 ground measurement sites, mm mobile collection points, and ww met data sites.  Available relevant satellite data was also archived.
-  ~200 instruments were used to collect data

## Introduction to Milagro (2)

- During the collection phase ~450 researchers and technicians, 30 Nationalities, 150 institutions, were in the field collecting data
- Data collection took place over ~30 days in March 2006
- Uploaded Measurements were folded in with model output data  and MET data to inform decisions about where collections would be taking place
- Scientific analysis of these data is ongoing “dozens” of  scientific papers are in process

## Requirements

- An access control system was required to allow only Milagro participants to view and download data
- A participant’s data uploads were to assigned areas in the Milagro hierarchy
- Participants were allowed and encouraged to enter metadata about data uploads to their respective areas
- Participants have the option to create hierarchies that would extend from the assigned starting point in the hierarchy
- Wide variety of file formats

## Requirements (2)

- The administration of the collection would be placed in the hands of a group admin that was in the Milagro project
- The group admin would have control of group membership, read, data upload and publication, and metadata editing privileges for the participants
- Participants would be allowed to upload tar/compressed files that could later be uncompressed and un-tarred
- An easy registration process for non-UCAR participants
- Upload and download metrics were collected and made available

## Solutions

- Controls for the participant user interface were integrated into the browse pages that they had permissions to upload and publish to, and we unavailable otherwise
- A group administration interface was created and integrated into the existing access control mechanisms
- New reporting options were added to the CDP metrics system
- Web based workflows were created for upload, publication, and metadata editing

## Publication Block diagram

## Security issues

- Uploads have risks involved
- A layered approach was used to mitigate potential problems
- Users are known to group admins
- Users have a limited area that they are allowed to upload to
- Users are limited in the number of GB they can upload
- Uploading of files can be prohibited on the basis of file extension and  binary signature
- Metrics reports to group admins allows the group admin to flag problems

## Results

- The system was designed for preliminary, first look Milagro data during collection but later it was also used for the first round of results for the first Milagro scientific meeting.
- Metadata entry was uneven (no surprise here)
- The ability to create hierarchies was extensively used
- The ability to assign upload, publish and metadata privileges was extensively used
- ~2500 items were uploaded, many from non-native English speakers
- The PI and participants *use* the upload/publish system.
- This work could be extended to distributed data systems, the data do not have to be moved to be published.
- This work could be extended to operate with web services for automatic publication

## Demo

- Demo of user upload/publish
- Will not demo group admin tools or registration tools

## Questions/Discussion?