# Presentation: 947805

## Parallel HDF5

- Tutorial

## Outline

- Prerequisites
- Why parallel HDF5?
- Overview of Parallel HDF5 Design
- Setting up Parallel Environment
- Creating and Accessing a File
- Creating and Accessing a Dataset
- Writing and Reading Hyperslabs

## Prerequisites

- Basic knowledge of serial  HDF5 Library and APIs
- Basic knowledge of MPI and MPI I/O
- Basic knowledge of C and/or Fortran90(95)
- No Open MP
- No Thread Programming

## Why parallel HDF5?

- Parallel computing environment for scalable performance and computational power
- Problem size is so big that does not fit to one computer (node)
- HDF5 file portability between parallel file systems (?)

## Overview of Parallel HDF5 DesignFunction requirements

- An API to support parallel file access for HDF5 files in a message passing environment
- Collective API calls only for file structural changes
- Supports independent and collective I/O for datatsets in the same or different HDF5 files
- Minimum deviation from HDF5 interface

## Overview of Parallel HDF5 DesignSystem requirements

- C and F90 language interfaces
- MPI for interprocess communication
- MPI I/O calls for parallel access
- Platforms supported: IBM SP2 and SP3, Intel TFLOPS, SGI Origin 2000, Linux

## Setting up Parallel Environment

- Set up environment variables needed on your system (?)
- ?

## Creating and Accessing a FileProgramming model

- HDF5 uses optional access template object to control the file access mechanism
- General model to access HDF5 file in parallel:
***Setup access template***
***Open File ***
***Close File ***

## Setup access template

- Each process of the MPI communicator creates an
- access template and sets it up with MPI parallel
- access information
- C:

**herr_t H5Pset_fapl_mpio(hid_t plist_id, **
**       ****MPI_Comm comm,  MPI_Info info);**

- F90:

-  **h5pset_fapl_mpio_p(plist_id, comm, info);**
** **
** ****integer(hid_t) :: plist_id**
** ****integer        :: comm, info**

## C Example

**    ****23          comm = MPI_COMM_WORLD;**
**    ****24          info = MPI_INFO_NULL; **
**    **
**    ****26          /***
**    ****27****	****     * Initialize MPI**
**    ****28****	****     */**
**    ****29****	****    MPI_Init(&argc, &argv);**
**    ****30****	****    MPI_Comm_size(comm, &mpi_size);**
**    ****31****	****    MPI_Comm_rank(comm, &mpi_rank);  **
**    ****32****	**** **
**    ****33****	****    /* **
**    ****34****	****     * Set up file access property list with **
**                 ***** parallel I/O access**
**    ****35****	****     */**
**    ****36****	****     plist_id = H5Pcreate(H5P_FILE_ACCESS);**
**    ****37****	****     H5Pset_fapl_mpio(plist_id, comm, info);**
**    ****38****	**

## F90 Example

-    **                   **
**    ****23      comm = MPI_COMM_WORLD**
**    ****24****	****info = MPI_INFO_NULL**
**    ****25****	**
**    ****26****	****CALL MPI_INIT(mpierror)**
**    ****27****	****CALL MPI_COMM_SIZE(comm, mpi_size, mpierror)**
**    ****28****	****CALL MPI_COMM_RANK(comm, mpi_rank, mpierror) **
**    ****29****	****!**
**    ****30****	****! Initialize FORTRAN predefined datatypes**
**    ****31****	****!**
**    ****32****	****CALL h5open_f(error) **
**    ****33****	**
**    ****34****	****! **
**    ****35****	****! Setup file access property list with parallel **
**              ****I/O access.**
**    ****36****	****!**
**    ****37****	****CALL h5pcreate_f(H5P_FILE_ACCESS_F, plist_id, error)**
**    ****38****	****CALL h5pset_fapl_mpio_f(plist_id, comm, info, error)**

## Creating and Accessing a Dataset

- All processes of the MPI communicator open a dataset by a *collective call*
**H5Dcreate or H5Dopen; H5Dclose**
- All processes of the MPI communicator extend dataset with unlimited dimensions by a *collective call*
**H5Dextend**

## Accessing a Dataset

- All processes that have opened dataset may do collective I/O
- Each process may do independent and arbitrary number of data I/O access calls
**C:   H5Dwrite and H5Dread**
**F90: h5dwrite_f and h5dread_f**

## Accessing a DatasetProgramming model

- Create and set dataset transfer property
**C:   H5Pset_dxpl_mpio**
**H5FD_MPIO_COLLECTIVE**
**H5FD_MPIO_INDEPENDENT**
**F90: h5pset_dxpl_mpio_f**
**H5FD_MPIO_COLLECTIVE_F**
**H5FD_MPIO_INDEPENDENT_F**
- Access dataset with the defined transfer property

## C Example: Collective write

**    ****95    /***
**    ****96     * Create property list for collective dataset write.**
**    ****97     */**
**    ****98     plist_id = H5Pcreate(H5P_DATASET_XFER);**
**    ****99     H5Pset_dxpl_mpio(plist_id, H5FD_MPIO_COLLECTIVE);**
**   ****100****	****    **
**   ****101     status = H5Dwrite(dset_id, H5T_NATIVE_INT,,**
**   ****102****	****	****	****     memspace, filespace, plist_id, **
**                             ****data);**
**   **
**	**

## F90 Example: Collective write

**    ****87  !**
**    ****88  ! Create property list for collective dataset write**
**    ****89  !**
**    ****90  CALL h5pcreate_f(H5P_DATASET_XFER_F, plist_id, error) **
**    ****91  CALL h5pset_dxpl_mpio_f(plist_id, &**
**                                ****H5FD_MPIO_COLLECTIVE_F, error)**
**    ****92****	****     **
**    ****93  !**
**    ****94  ! Write the dataset collectively. **
**    ****95  !**
**    ****96  CALL h5dwrite_f(dset_id, H5T_NATIVE_INTEGER, data, &**
**                        ****error, &**
**                        ****file_space_id = filespace, & **
**                        ****mem_space_id = memspace, &**
**                        ****xfer_prp = plist_id)**
**	**

## Writing and Reading HyperslabsProgramming model

- Each process defines memory and file hyperslabs
- Each process executes partial write/read call
- Collective calls
- Independent calls

##             Example 1

- P0

- P1

- File

-             Example 1
*Writing dataset by rows*

- P2

- P3

## Writing by rowsOutput of h5dump utility

**HDF5 "SDS_row.h5" {**
**GROUP "/" {**
**   ****DATASET "IntArray" {**
**      ****DATATYPE  H5T_STD_I32BE  **
**      ****DATASPACE  SIMPLE { ( 8, 5 ) / ( 8, 5 ) } **
**      ****DATA {**
**         ****10, 10, 10, 10, 10,**
**         ****10, 10, 10, 10, 10,**
**         ****11, 11, 11, 11, 11,**
**         ****11, 11, 11, 11, 11,**
**         ****12, 12, 12, 12, 12,**
**         ****12, 12, 12, 12, 12,**
**         ****13, 13, 13, 13, 13,**
**         ****13, 13, 13, 13, 13**
**      ****} **
**   ****} **
**} **
**} **

##             Example 1

- P1 (memory space)

- File

-             Example 1
*Writing dataset by rows*

**count[0] = dimsf[0]/mpi_size**
**count[1] = dimsf[1];**
**offset[0] = mpi_rank * count[0];**
**offset[1] = 0;**

**count[0]**

**count[1]**

**offset[0]**

**offset[1]**

## C Example 1   

-     **71  /* **
**    ****72   * Each process defines dataset in memory and**
**         ***** writes it to the hyperslab**
**    ****73   * in the file.**
**    ****74   */**
**    ****75   count[0] = dimsf[0]/mpi_size;**
**    ****76   count[1] = dimsf[1];**
**    ****77   offset[0] = mpi_rank * count[0];**
**    ****78   offset[1] = 0;**
**    ****79   memspace = H5Screate_simple(RANK,count,NULL);**
**    ****80****	**
**    ****81  /***
**    ****82   * Select hyperslab in the file.**
**    ****83   */**
**    ****84   filespace = H5Dget_space(dset_id);**
**    ****85   H5Sselect_hyperslab(filespace, **
**         ****H5S_SELECT_SET,offset,NULL,count,NULL);**
**    ****86****	**

## P0

- P1

- File

-             Example 2
*Writing dataset by columns*

## Writing by columnsOutput of h5dump utility

**HDF5 "SDS_col.h5" {**
**GROUP "/" {**
**   ****DATASET "IntArray" {**
**      ****DATATYPE  H5T_STD_I32BE  **
**      ****DATASPACE  SIMPLE { ( 8, 6 ) / ( 8, 6 ) } **
**      ****DATA {**
**         ****1, 2, 10, 20, 100, 200,**
**         ****1, 2, 10, 20, 100, 200,**
**         ****1, 2, 10, 20, 100, 200,**
**         ****1, 2, 10, 20, 100, 200,**
**         ****1, 2, 10, 20, 100, 200,**
**         ****1, 2, 10, 20, 100, 200,**
**         ****1, 2, 10, 20, 100, 200,**
**         ****1, 2, 10, 20, 100, 200**
**      ****} **
**   ****} **
**} **
**} **

## P1

- P0

- File

-             Example 2
*Writing dataset by columns*

- Memory

**block[1]**

**block[0]**

**P0 offset[1]**

**P1 offset[1]**

**stride[1]**

**dimsm[0]**
**dimsm[1]**

## C Example 2

**85**	    **/***
**86****	****     * Each process defines hyperslab in **
**          ***** the file**
**88****	****     */**
**89****	****    count[0] = 1;**
**90****	****    count[1] = dimsm[1];**
**91****	****    offset[0] = 0;**
**92****	****    offset[1] = mpi_rank;**
**93****	****    stride[0] = 1;**
**94****	****    stride[1] = 2;**
**95****	****    block[0] = dimsf[0];**
**96****	****    block[1] = 1;**
**97****	**
**98****	****    /***
**99****	****     * Each process selects hyperslab.**
**100****	****     */**
**101****	****    filespace = H5Dget_space(dset_id);**
**102****	****    H5Sselect_hyperslab(filespace,**
**         ****H5S_SELECT_SET, offset, stride, **
**         ****count, block);**
**103****	**

## P0

- P2

- File

-             Example 3
*Writing dataset by pattern*

- P3

- P1

## Writing by patternOutput of h5dump utility

**HDF5 "SDS_pat.h5" {**
**GROUP "/" {**
**   ****DATASET "IntArray" {**
**      ****DATATYPE  H5T_STD_I32BE  **
**      ****DATASPACE  SIMPLE { ( 8, 4 ) / ( 8, 4 ) } **
**      ****DATA {**
**         ****1, 3, 1, 3,**
**         ****2, 4, 2, 4,**
**         ****1, 3, 1, 3,**
**         ****2, 4, 2, 4,**
**         ****1, 3, 1, 3,**
**         ****2, 4, 2, 4,**
**         ****1, 3, 1, 3,**
**         ****2, 4, 2, 4**
**      ****} **
**   ****} **
**} **
**} **

## P2

- File

-             Example 3
*Writing dataset by pattern*

**	****    **
**           ****offset[0] = 0;**
**           ****offset[1] = 1;**
**           ****count[0] = 4;**
**           ****count[1] = 2;**
**           ****stride[0] = 2;**
**           ****stride[1] = 2;**

- Memory

**stride[0]**

**stride[1]**

**offset[1]**

**count[1]**

## C Example 3Writing by pattern

-     **89****	****    /* **
**    ****90****	****     * Each process defines dataset in memory and**
**            ***** writes it to the hyperslab**
**    ****91****	****     * in the file.**
**    ****92****	****     */**
**    ****93****	****    count[0] = 4;**
**    ****94****	****    count[1] = 2;**
**    ****95****	****    stride[0] = 2;**
**    ****96****	****    stride[1] = 2;**
**    ****97****	****    if(mpi_rank == 0) {**
**    ****98****	****       offset[0] = 0;**
**    ****99****	****       offset[1] = 0;**
**   ****100****	****    } **
**   ****101****	****    if(mpi_rank == 1) {**
**   ****102****	****       offset[0] = 1;**
**   ****103****	****       offset[1] = 0;**
**   ****104****	****    }**
**   ****105****	****    if(mpi_rank == 2) {**
**   ****106****	****       offset[0] = 0;**
**   ****107****	****       offset[1] = 1;**
**   ****108****	****    } **
**   ****109****	****    if(mpi_rank == 3) {**
**   ****110****	****       offset[0] = 1;**
**   ****111****	****       offset[1] = 1;**
**   ****112****	****    }**

## P0

- P2

- File

-             Example 4
*Writing dataset by chunks*

- P1

- P3

## Writing by chunksOutput of h5dump utility

**HDF5 "SDS_chnk.h5" {**
**GROUP "/" {**
**   ****DATASET "IntArray" {**
**      ****DATATYPE  H5T_STD_I32BE  **
**      ****DATASPACE  SIMPLE { ( 8, 4 ) / ( 8, 4 ) } **
**      ****DATA {**
**         ****1, 1, 2, 2,**
**         ****1, 1, 2, 2,**
**         ****1, 1, 2, 2,**
**         ****1, 1, 2, 2,**
**         ****3, 3, 4, 4,**
**         ****3, 3, 4, 4,**
**         ****3, 3, 4, 4,**
**         ****3, 3, 4, 4**
**      ****} **
**   ****} **
**} **
**} **

## P2

- File

-             Example 4
*Writing dataset by chunks*

- Memory

**block[0] = chunk_dims[0];**
**block[1] = chunk_dims[1];**
**offset[0] = chunk_dims[0];**
**offset[1] = 0;**

**chunk_dims[0]**

**chunk_dims[1]**

**block[0]**

**block[1]**

**offset[0]**

**offset[1]**

## C Example 4Writing by chunks

-     **97****	****    count[0] = 1;**
**    ****98****	****    count[1] = 1 ;**
**    ****99****	****    stride[0] = 1;**
**   ****100****	****    stride[1] = 1;**
**   ****101****	****    block[0] = chunk_dims[0];**
**   ****102****	****    block[1] = chunk_dims[1];**
**   ****103****	****    if(mpi_rank == 0) {**
**   ****104****	****       offset[0] = 0;**
**   ****105****	****       offset[1] = 0;**
**   ****106****	****    } **
**   ****107****	****    if(mpi_rank == 1) {**
**   ****108****	****       offset[0] = 0;**
**   ****109****	****       offset[1] = chunk_dims[1];**
**   ****110****	****    }**
**   ****111****	****    if(mpi_rank == 2) {**
**   ****112****	****       offset[0] = chunk_dims[0];**
**   ****113****	****       offset[1] = 0;**
**   ****114****	****    } **
**   ****115****	****    if(mpi_rank == 3) {**
**   ****116****	****       offset[0] = chunk_dims[0];**
**   ****117****	****       offset[1] = chunk_dims[1];**
**   ****118****	****    }**** **