# Presentation: 148717

## Using Data Grid Technology to Manage Distributed Data

**Using Data Grid Technology to Manage Distributed Data**
**Reagan W. Moore**
**University of California, San Diego**
**San Diego Supercomputer Center**
**moore@sdsc.edu**
**http://www.npaci.edu/DICE/**

## Generic Technology Evolution

- Community specification of intent
- Identification of common infrastructure across
- Digital libraries
- Data grids
- Persistent archives
- Focus on scalability and robustness of unifying infrastructure

## Slide 3

## Topics

- Data management systems
- Data Grids, Digital Libraries, Persistent Archives
- Common data management technology
- Logical name space, storage abstraction
- Collection federation
- Knowledge management systems

## Digital Libraries

- Provide services on the data collection
- Ingestion, loading of attribute values
- Extensibility, definition of new attributes
- Discovery, queries on attributes
- Browsing, hierarchical listing
- Presentation, formatting specified data models
- Communities
- Digital library
- Global Grid Forum, Databases and the Grid working group
- OMG, Common Warehouse Metamodel

## Data Grids

- Manage data in a distributed environment
- Logical name space, provide global identifier
- Data access, storage system abstraction
- Replication, disaster back up
- Uniform access, common API across file systems, archives, and databases
- Single sign-on, authenticate across administration domains
- Communities
- Global Grid Forum, data grids
- Discipline specific data management systems

## Persistent Archives

- Manage technology evolution
- Storage system abstraction, support data migration across storage systems
- Information repository abstraction, support catalog migration to new databases
- Logical name space, support global persistent identifier
- Communities
- Persistent archive community
- Global Grid Forum, Persistent archive working group

## Common Capabilities

- Logical name space
- Registration of digital entities
- Storage repository abstraction
- Operations used to manipulate data in a storage system
- Information repository abstraction
- Operations used to manipulate a catalog in a database

## Data Grid(Storage Resource Broker)

- Integration of collection-based management of digital entities, with
- Remote data access through storage system abstraction
- Catalog access through information repository abstraction
- Automation through collection-owned data

## Storage Abstraction

- Provide common access semantics
- Archival storage systems
- File systems
- Databases
- Support Unix file system operations
- Map from the interface preferred by your application to the interfaces required by legacy storage systems
- Support database interactions
- Map from information repository abstraction to database commands

## SDSC Storage Resource Broker & Meta-data Catalog

**Java, NT**
**Browsers**

**Web**
**WSDL**

**GridFTP**

**SDSC Storage Resource Broker & Meta-data Catalog**
**Storage Abstraction**

**Application**

**HRM**

- Storage Abstraction

- Catalog Abstraction

**Databases**
**DB2, Oracle, Sybase**

- Logical Name
- Space

- Latency
- Management

- Data
- Transport

- Metadata
- Transport

- Consistency Management / Authorization-Authentication

**Linux **
**I/O**

**DLL /**
**Python**

- GridFTP

- GridFTP

## Logical Name Space(Data Grid Transparencies)

- Naming transparency - find a data set without knowing its name
- Map from attributes to a global file name
- Location transparency - access a data set without knowing where it is
- Map from global file name to local file name
- Access transparency - access a data set without knowing the type of storage system
- Federated client-server architecture

## Logical Name Space Operations

- Replication
- One to many mapping from logical name to physical name
- Containers
- Mapping from logical name to location in a physical container
- Shadow links
- Registration of user owned data into the collection

## SDSC Storage Resource Broker & Meta-data Catalog

**Java, NT**
**Browsers**

**Web**
**WSDL**

**GridFTP**

**SDSC Storage Resource Broker & Meta-data Catalog**
**Logical Name Space**

**Application**

**HRM**

- Access
- APIs

- Servers

- Storage Abstraction

- Catalog Abstraction

**Databases**
**DB2, Oracle, Sybase**

- Logical Name
- Space

- Latency
- Management

- Data
- Transport

- Metadata
- Transport

- Consistency Management / Authorization-Authentication

- Prime
- Server

**Linux **
**I/O**

**DLL /**
**Python**

## Digital Entities

- Digital entities are “images of reality”, made of
- Data, the bits (zeros and ones) put on a storage system
- Information, the attributes used to assign semantic meaning to the data
- Knowledge, the semantic and structural relationships described by a data model
- Every digital entity requires information and knowledge to correctly interpret and display

## Types of Digital Entities

- Files
- Physical files in the collection ID space
- Shadow links to files in your user ID space
- Directories
- Shadow links to directories in your user ID space
- Databases
- Shadow links to tables
- SQL command strings
- URLs

## Preservation(Similar requirements to a data grid)

- Name transparency
- Find a file by attributes (map from attributes to global name)
- Location transparency
- Access a file by a global identifier (map from global to local file name)
- Access transparency
- Use same API to access data in archive or file cache
- Authenticity
- Disaster recovery, replicate data across storage systems
- Audit and process management

## SDSC Storage Resource Broker & Meta-data Catalog

**Java, NT**
**Browsers**

**Web**
**WSDL**

**GridFTP**

**SDSC Storage Resource Broker & Meta-data Catalog**
**Preservation**

**Application**

**HRM**

- Access
- APIs

- Servers

- Storage Abstraction

- Catalog Abstraction

**Databases**
**DB2, Oracle, Sybase**

- Logical Name
- Space

- Latency
- Management

- Data
- Transport

- Metadata
- Transport

- Consistency Management / Authorization-Authentication

- Prime
- Server

**Linux **
**I/O**

**DLL /**
**Python**

## Convergence of Technologies

- Data grids as basis for distributed data management
- Federation of distributed resources
- Creation of logical name space to automate discovery
- Digital libraries
- Discovery based on attributes
- Hierarchical collection management
- Extensible schema through information repository abstraction
- Persistent archives
- Data replication
- Persistence management

## Data Naming Ontologies

| Concept space | Discipline concepts |
| --- | --- |
| Collection | Discipline attributes |
| Data grid | Global Identifier |
| Archive / file systems | Local file name |
| Data model | Attributes that describe data structure |

## Differentiating between Data, Information, and Knowledge

**Differentiating between Data, Information, and Knowledge**

**Data**
- Digital object
- Objects are streams of bits
**Information**
- Any tagged data, which is treated as an attribute.
- Attributes may be tagged data within the digital object, or tagged data that is associated with the digital object
**Knowledge**
- Relationships between attributes
- Relationships can be procedural/temporal, structural/spatial, logical/semantic, functional

## Knowledge Creation Roadmap

- Knowledge syntax (consensus)
- RDF, XMI, Topic Map
- Knowledge management (recursive operations)
- Oracle parallel database
- Knowledge manipulation (spatial/procedural rules)
- Generation of inference rules and mapping to data models
- Knowledge generation (scalable inference engine)
- Application of inference rules in inference engine

## Knowledge Based Data Grid Roadmap

- Attributes
- Semantics

- Knowledge

- Information

- Data

- Ingest
- Services

- Management

- Access
- Services

- (Model-based Access)

- (Data Handling System - SRB)

- MCAT/HDF

- Grids

- XML DTD

- SDLIP

- XTM DTD

- Rules - KQL

- Information
- Repository

- Attribute- based
- Query

- Feature-based
- Query

- Knowledge or
- Topic-Based
- Query / Browse

- Knowledge
- Repository for
- Rules

- Relationships
- Between
- Concepts

- Fields
- Containers
- Folders

- Storage
- (Replicas,
- Persistent IDs)