# Presentation: 893070

## GridFTP: Easy access to HPSS for Grid Applications

- Bill Allcock
- Argonne National Lab

## Outline of my Talk

- Because this topic may not be familiar to many here, I am going to do this in two stages:
- Part I:
- What is GridFTP and why you should care
- How it works with HPSS
- Part II (time permitting):
- Introduction to Grids
- More detail on GridFTP
- Even more detail on the interface to HPSS

## Why should you care?

- Some of the HPSS community are also involved in Grid computing and want their storage resources accessible via standard Grid protocols
- Grid computing may not be all it is hyped to be, but it does not appear that it is going away, so having GridFTP access makes HPSS available to the entire Grid community
- It provides a secure, fast, robust, standard, open source access solution to HPSS data.

## What is GridFTP (short version)

- A secure, robust, fast, efficient, standards based, widely accepted data transfer protocol
- Secure
- GSI vs Kerberos: discussed later
- robust: we send restart markers and can restart failed transfers
- Fast: we use multiple TCP streams, proccesses and hosts to move data (similar to HPSS)
- Standards based: Protocol is documented in GGF
- Widely Accepted: Used in production in many projects

## Widely Accepted

- GridFTP (and GSI) are by far the most widely used tools in the Globus Toolkit
- GridFTP is at the heart of almost every major Grid project.
- It is used in production on a daily basis:
- LIGO moves a TB a day
- HEP physics uses it (ATLAS, CMS, OSG, PPDG, LCG).
- Just did a “service challenge” where they maintained 600 MB/s average for 10 days straight, moving 500 TB of data.
- UK QCD does automatic replication
- The Earth Science Grid uses it for climate studies

## Shameless Marketing SlideorWhy you should use the new Server

- Its way cool 
- Hit 27 Gbs on the TG 30 Gbs network (90% utilization)
- Had a single host supporting 1800 simultaneous clients.
- coadd runs with the 2.4.3 server were having O(1000) failures.  A test run with the new server had zero failures
- Our support focus is the new server
- Trivial to install
- Need the new server to use the HPSS module 
***SUMMARY: YOU REALLY WANT******TO USE THE NEW SERVER!***

## Parallelism vs Striping

## A note about GSI vs Kerberos

- First, this is not my area of expertise and I don’t want to have a religious debate 
- GSI and Kerberos are actually very similar
- There are a couple of key differences:
- cross-realm agreements vs user-to-user
- speed at which changes can be made
- GSI calls user-to-user delegation
- This is critical for Grid apps and GridFTP
- Kerberos *can* do this, but has not yet made it into official distributions.

## How does it work? (0)

- To make a “normal” GridFTP server an HPSS GridFTP server:
- copy the HPSS DSI to the right directory
- add –dsi HPSS to the server command line
- put a grid-mapfile with the DN to Kerberos mappings in place.
- One issue that is not resolved is the distribution of the DSI
- not clear what IBM license will be
- get it from us, IBM
- source or binary

## How does it work (1)?

- GridFTP client does GSI *authentication* to the GridFTP server
- GSI is strong authentication
- Credentials are signed, you choose whose signature you trust
- You end up with a globally unique ID called a distinguished name or DN
- /DC=org/DC=doegrids/OU=People/CN=William E. Allcock 337378

## How does it work? (2)

- We have authenticated, now we authorize
- The server looks the DN up in the grid-mapfile and determines what Kerberos user it should run as
- This *is* an extra admin step
- There are tools out there to help automate this.
- HPSS might be able to automate this fairly easy since I believe you already have an LDAP registry of users.
- The GridFTP server runs as the same user as pftp and so is a privileged HPSS user
- The server does a load_thread_state, which essentially does a setuid.  This determines the access rights.
- Note: You must be in the grid-mapfile to have access

## How does it work? (3)

- The server is now essentially an HPSS client.
- The module that did calls like fopen, fread, frwrite is replaced with a module that does HPSS_open, HPSS_readlist, HPSS_writelist
- Actually, we are using a higher level API (pio) that calls back to us and asks for blocks.
- This was done to prevent any intellectual property issues from arising.
- For a put, a buffer arrives over the network using the GridFTP protocol, and then is written (without a copy) to HPSS using whatever protocol is behind the pio API.
- For a get, a buffer is read from HPSS and then written to the network (without a copy).

## How does it work? Caveats

- The GridFTP protocol allows for writing blocks out of order
- obviously this doesn’t work so well for tape
- The current solution is that we will buffer blocks until we have consumed a specified max amount of memory and then the transfer will fail.
- Initial tests indicate this wont happen under “reasonable” conditions
- For GridFTP, the sending side of the transfer MUST do the TCP connect.
- Potential firewall issues
- Note that the authentication and authorization is very similar to the GSI-authenticated pftpd

## Performance

- We are just starting to get performance numbers, but I don’t trust them enough yet, to show them.
- However, with no particular tuning, globus-url-copy (our client) got about 20% better performance than pftp on a 2 way stripe to disk (no tape data yet)
- There are things that bother me about the data, like in all cases it is significantly slower than I would expect.
- About the only thing I would take from this so far is that it does not look like we will have significant problems at least matching the pftp performance.

## Part II:  More Details

## Introduction to Grid Computing

- Term comes from the Power Grid
- When you flip your light switch, you don’t care if the power company generated the power or bought, it you just want the light to come on for an agreed upon price.
- NOTE FOR CALIFORNIANS: Yes, the light *is* supposed to come on.
- Idea is for computing to work the same
- Lots of names: Utility computing, on-demand computing, heterogeneous computing, Metacomputing, etc..
- Lots of hype
- Much of what you hear called Grid Computing is not
- However, people don’t agree

## What is Grid Computing

- Resource sharing
- Computers, storage, sensors, networks, ...
- Sharing always conditional: issues of trust, policy, negotiation, payment, ...
- Coordinated problem solving
- Beyond client-server: distributed data analysis, computation, collaboration, ...
- Dynamic, multi-institutional virtual orgs
- Community overlays on classic org structures
- Large or small, static or dynamic
- What does it mean to be on the Grid?
- Mostly, running certain protocols, same as the internet
- Two common ones are:
- Grid Security Infrastructure (GSI)
- GridFTP

## Why is this hard/different?

- Lack of central control
- Where things run
- When they run
- Shared resources
- Contention, variability
- Communication
- Different sites implies different sys admins, users, institutional goals, and often “strong personalities”

## So why do it and why now?

- Why do it?
- Computations that need to be done with a time limit
- Data that can’t fit on one site
- Data owned by multiple sites
- Applications that need to be run bigger, faster, etc
- Why now?
- idea has been around since the 60’s
- Compute power, network speeds, storage cost, begin to make it practical

## Where are we?

- Things that are working
- Key standards are emerging
- Open source infrastructure appearing
- Success stories & experience gained
- Challenges that remain
- Still requires significant manual admin
- Complexity of some infrastructure
- Missing specifications
- Limited practical experience
- Progress being made on all fronts

## Introduction to GridFTP

## What is GridFTP?

- A secure, robust, fast, efficient, standards based, widely accepted data transfer protocol
- A Protocol
- Multiple Independent implementation can interoperate
- This works.  Both the Condor Project at Uwis and Fermi Lab have home grown servers that work with ours.
- Lots of people have developed clients independent of the Globus Project.
- The Globus Toolkit  supplies a reference implementation:
- Server
- Client tools (globus-url-copy)
- Development Libraries

## GridFTP: The Protocol (cont)

- Existing standards
- RFC 959: File Transfer Protocol
- RFC 2228: FTP Security Extensions
- RFC 2389: Feature Negotiation for the File Transfer Protocol
- Draft: FTP Extensions
- GridFTP: Protocol Extensions to FTP for the Grid
- Grid Forum Recommendation
- GFD.20
- http://www.ggf.org/documents/GWD-R/GFD-R.020.pdf

## wuftpd based GridFTP

- _Existing Functionality_
- Security (gss api)
- Reliability / Restart
- Parallel Streams
- Third Party Transfers
- Manual TCP Buffer Size
- Server Side Processing
- Partial File Transfer
- Large File Support
- Data Channel Caching
- Integrated Instrumentation
- De facto standard on the Grid

- _New Functionality in 3.2_
- Server Improvements
- Structured File Info
- MLST, MLSD
- checksum support
- chmod support
- globus-url-copy changes
- File globbing support
- Recursive dir moves
- RFC 1738 support
- Control of restart
- Control of DC security

## GridFTP: Caveats

- Protocol requires that the sending side do the TCP connect (possible Firewall issues)
- Working on V2 of the protocol
- Add explicit negotiation of streams to relax the directionality requirement above
- Optionally adds block checksums and resends
- Add a unique command ID to allow pipelining of commands
- Client / Server
- Currently, no server library, therefore Peer to Peer type apps VERY difficult
- Generally needs a pre-installed server
- Looking at a “dynamically installable” server

## The New GridFTP Server

## GT4 GridFTP Server Implementation

- 100% Globus code.  No licensing issues.
- Striping support is provided in 4.0
- Has IPV6 support included (EPRT, EPSV), but we have limited environment for testing.
- Extremely modular to allow integration with a variety of data sources (files, mass stores, etc.)
- Based on XIO
- wuftpd specific functionality, such as virtual domains, is NOT present

## New Server Architecture

- GridFTP (and normal FTP) use (at least) two separate socket connections:
- A control channel for carrying the commands and responses
- A Data Channel for actually moving the data
- Control Channel and Data Channel can be (optionally) completely separate processes.
- A single Control Channel can have multiple data channels behind it.
- This is how a striped server works.
- In the future we would like to have a load balancing proxy server work with this.

## Possible Configurations

- Control

- Data

- Typical Installation

- Control

- Data

- Separate Processes

- Striped Server

- Control

- Striped Server (future)

- Data

## New Server Architecture

- Data Transport Process (Data Channel) is architecturally, 3 distinct pieces:
- The protocol handler.  This part talks to the network and understands the data channel protocol
- The Data Storage Interface (DSI). A well defined API that may be replaced to access things other than POSIX filesystems
- ERET/ESTO processing.  Ability to manipulate the data prior to transmission.
- Not implemented as a separate module for 4.0, but planned for 4.2
- Working with several groups to on custom DSIs
- LANL / IBM for HPSS
- UWis / Condor for NeST
- SDSC for SRB

## Third Party Transfer

- RFT Service

- RFT Client

- SOAP Messages

- Notifications(Optional)

## The Data Storage Interface (DSI)

- Conceptually, the DSI is very simple.
- There are a few required functions (init, destroy)
- Most of the interface is optional, and you can only implement what is needed for your particular application.
- There are a set of API functions provided that allow the DSI to interact with the server itself.
- Note that the DSI could be given significant functionality, such as caching, proxy, backend allocation, etc..

## Developer Implemented Functions

- Below is the structure used to hold the pointers to your functions.
- This can be found in <install>/source-trees/gridftp/server/src

- typedef struct globus_gfs_storage_iface_s
- {
-     int                                 descriptor;
-     /* session initiating functions */
-     globus_gfs_storage_init_t           init_func;
-     globus_gfs_storage_destroy_t        destroy_func;
-     /* transfer functions */
-     globus_gfs_storage_transfer_t       list_func;
-     globus_gfs_storage_transfer_t       send_func;
-     globus_gfs_storage_transfer_t       recv_func;
-     globus_gfs_storage_trev_t           trev_func;

-     /* data conn funcs */
-     globus_gfs_storage_data_t           active_func;
-     globus_gfs_storage_data_t           passive_func;
-     globus_gfs_storage_data_destroy_t   data_destroy_func;
-     globus_gfs_storage_command_t        command_func;
-     globus_gfs_storage_stat_t           stat_func;
-     globus_gfs_storage_set_cred_t       set_cred_func;
-     globus_gfs_storage_buffer_send_t    buffer_send_func;
- } globus_gfs_storage_iface_t;

## Master vs. Slave DSI

- If you wish to support striping, you will need two DSIs
- The Master DSI will be in the PI or front end.  It must implement all functions (that you want to support).
- Usually, this is relatively trivial and involves minor processing and then “passing” the command over the IPC channel to the slave DSI
- Any functions not implemented will be handled by the server if possible (non-filesystem, active, list)
- All DSI’s must implement the init_func and destroy_func functions.

## Slave Functions

- The slave DSI does the real work.  It typically implements the following functions:
- send_func: This function is used to send data from the DSI to the server (get or RETR)
- recv_func: This function is used to receive data from the server (put or STOR)
- stat_func: This function performs a unix stat, i.e. it returns file info.  Used by the list function
- command_func:  This function handles simple (succeed/fail or single line response) file system operations such as mkdir, site chmod, etc.

## IPC Calls

- These calls are how the master DSI “passes” the call to the slave DSI
- The IPC calls are basically the same as the DSI calls.
- globus_gfs_ipc_iface_stat_t	stat_func;
- globus_gfs_storage_stat_t	stat_func;
- These calls implement an internal, binary protocol to transfer the necessary structures between the front end and the back end.
- The IPC receiver receives the message and then invokes the sender DSI.  The sender DSI does not know, nor does it need to know, whether it is local or remote.