# Presentation: 917699

## Overview of the Global ArraysParallel Software Development Toolkit  Introduction to Global Address Space Programming Models

- P. Saddayappan2
-  Jarek Nieplocha1 , Bruce Palmer1, Manojkumar Krishnan1, Vinod Tipparaju1
- 1Pacific Northwest National Laboratory
- 2Ohio State University

## Outline of the Tutorial

- Parallel programming models
- Global Arrays (GA) programming model
- GA Operations
- Writing, compiling and running GA programs
- Basic, intermediate, and advanced calls
- With C and Fortran examples
- GA Hands-on session

- Global Arrays Tutorial

## Parallel Programming Models

- Single Threaded
- Data Parallel, e.g. HPF
- Multiple Processes
- Partitioned-Local Data Access
- MPI
- Uniform-Global-Shared Data Access
- OpenMP
- Partitioned-Global-Shared Data Access
- Co-Array Fortran
- Uniform-Global-Shared + Partitioned Data Access
- UPC, Global Arrays, X10

- Global Arrays Tutorial

## High Performance Fortran

- Single-threaded view of computation
- Data parallelism and parallel loops
- User-specified data distributions for arrays
- Compiler transforms HPF program to SPMD program
- Communication optimization critical to performance
- Programmer may not be conscious of communication implications of parallel program
- DO I = 1,N              DO I = 1,N
-     DO J = 1,N              DO J = 1,N
-          A(I,J) = B(J,I)         A(I,J) = B(I,J)
-     END                        END
- END                        END

- Global Arrays Tutorial

## Message Passing Interface

- Most widely used parallel programming model today
- Bindings for Fortran, C, C++, MATLAB
- P parallel processes, each with local data
- MPI-1: Send/receive messages for inter-process communication
- MPI-2: One-sided get/put data access from/to local data at remote process
- Explicit control of all inter-processor communication
- Advantage: Programmer is conscious of communication overheads and attempts to minimize it
- Drawback: Program development/debugging is tedious due to the partitioned-local view of the data

- Global Arrays Tutorial

## OpenMP

- Uniform-Global view of shared data
- Available for Fortran, C, C++
- Work-sharing constructs (parallel loops and sections) and global-shared data view ease program development
- Disadvantage: Data locality issues obscured by programming model
- Only available for shared-memory systems

- Global Arrays Tutorial

## Co-Array Fortran

- Partitioned, but global-shared data view
- SPMD programming model with local and shared variables
- Shared variables have additional co-array dimension(s), mapped to process space; each process can directly access array elements in the space of other processes
- A(I,J) = A(I,J)[me-1] + A(I,J)[me+1]
- Compiler optimization of communication critical to performance, but all non-local access is explicit

- Global Arrays Tutorial

## Unified Parallel C (UPC)

- SPMD programming model with global shared view for arrays as well as pointer-based data structures
- Compiler optimizations critical for controlling inter-processor communication overhead
- Very challenging problem since local vs. remote access is not explicit in syntax (unlike Co-Array Fortran)
- Linearization of multidimensional arrays makes compiler optimization of communication very difficult
- Performance study with NAS benchmarks (PPoPP 2005, Mellor-Crummey et. al.) compared CAF and UPC
- Co-Array Fortran had significantly better scalability
- Linearization of multi-dimensional arrays in UPC was a significant source of overhead

- Global Arrays Tutorial

## Slide 9

- Global Arrays Tutorial

## Global Arrays

- single, shared data structure/ global indexing
- e.g., access A(4,3) rather than buf(7) on task 2

- Physically distributed data

- Distributed dense arrays that can be accessed through a shared memory-like style

- Global Address Space

- Global Arrays Tutorial

## Global Array Model of Computations

- Shared memory view for distributed dense arrays
- Get-Local/Compute/Put-Global model of computation
- MPI-Compatible; Currently usable with Fortran, C, C++, Python
- Data locality and granularity control similar to message passing model

- Global Arrays Tutorial

## Global Arrays vs. Other Models

- Advantages:
- Inter-operates with MPI
- Use more convenient global-shared view for multi-dimensional arrays, but can use MPI model wherever needed
- Data-locality and granularity control is explicit with GA’s get-compute-put model, unlike the non-transparent communication overheads with other models (except MPI)
- Library-based approach: does not rely upon smart compiler optimizations to achieve high performance
- Disadvantage:
- Only useable for array data structures

- Global Arrays Tutorial

## How to Achieve Performance?

- Important considerations in achieving high performance:
- Specifying parallelism in computation
- Load balancing of computation
- Minimization of communication overheads
- All parallel programming models address first two bullets adequately, but differ w.r.t. communication optimization
- GA acknowledges that remote data access is expensive and encourages programmer to optimize communications
- Explicit rather than implicit communication
- Block access
- Nonblocking operations for latency hiding
- Consider matrix-matrix multiply example
- Parallelism is easily apparent: I,J loops
- But communication required is not obvious;
- compiler optimization crucial for performance
- With GA, user is responsible for, and optimizes
- communication consciously, aided by global view

*Matrix Multiplication*
- Do I = 1,N
-  Do J = 1,N
-   Do K = 1,N
-      C(I,J) = C(I,J) + A(I,K)*B(K,J)
-    End Do
-   End Do
- End Do

- Global Arrays Tutorial

## SRUMMA Matrix MultiplicationGA version

**A**

**B**

**C=A.B**

- Computation

- Comm.
- (Overlap)

**Issue NB Get A and B blocks**
**do**** (until last chunk)**
**    ****issue NB Get to the next blocks**
**    ****wait for previous issued call**
**    ****compute A*B (sequential dgemm)**
**    ****NB atomic accumulate into “C”      **
**        ****matrix**
**done**

**Advantages:**
** ****- Minimum memory**
** ****- Highly parallel**
** ****- Overlaps computation and communication**
**          ****- latency hiding**
** ****- exploits data locality**
** ****- patch matrix multiplication (easy to use)**
** ****- dynamic load balancing**

**patch matrix multiplication**

**=**

- Global Arrays Tutorial

## SUMMA Matrix Multiplication:Improvement over PBLAS/ScaLAPACK

**Papers at IPDPS’04**
**ICPADS’05**
**ACM Frontiers’06**

- Global Arrays Tutorial

## Overview of the Global ArraysParallel Software Development Toolkit Global Arrays Programming Model

- Jarek Nieplocha1
- Bruce Palmer1, Manojkumar Krishnan1, Vinod Tipparaju1,  P. Saddayappan2
- 1Pacific Northwest National Laboratory
- 2Ohio State University

## Basic Communication Models

***message passing***
***2-sided model***

- P1

- P0

*receive*

*send*

- P1

- P0

*put*

***remote memory access (RMA)***
***1-sided model***

- A

- B

- P1

- P0

*A=B*

***shared memory load/stores***
***0-sided model***

- A

- A

- B

- A

- B

- Global Arrays Tutorial

## Distributed Data vs Shared Memory

**Distributed Data:**
- Data is explicitly associated with each processor, accessing data requires specifying the location of the data on the processor and the processor itself.

- Data locality is explicit but data access is complicated. Distributed computing is typically implemented with message passing (e.g. MPI)

- Global Arrays Tutorial

## Distributed Data vs Shared Memory (Cont).

- Shared Memory:
- Data is in a globally accessible address space, any processor can access data by specifying its location using a global index

- Data is mapped out in a natural manner (usually corresponding to the original problem) and access is easy. Information on data locality is obscured and leads to loss of performance.

- (1,1)

- (150,200)

- (47,95)

- (106,171)

- Global Arrays Tutorial

## Global Arrays

- single, shared data structure/ global indexing
- e.g., access A(4,3) rather than buf(7) on task 2

- Physically distributed data

- Distributed dense arrays that can be accessed through a shared memory-like style

- Global Address Space

- Global Arrays Tutorial

## Global Arrays (cont.)

- Shared memory model in context of distributed dense arrays
- _Much_ simpler than message-passing for many applications
- Complete environment for parallel code development
- Compatible with MPI
- Data locality control similar to distributed memory/message passing model
- Extensible
- Scalable

- Global Arrays Tutorial

## Global Array Model of Computations

- Global Arrays Tutorial

## Creating Global Arrays

- g_a = NGA_Create(type, ndim, dims, name, chunk)

- float, double, int, etc.

- dimension

- array of dimensions

- character string

- minimum block size on each processor

- integer array handle

- Global Arrays Tutorial

## Shared Memory Style Communication in GA

- Global Arrays Tutorial

## Remote Data Access in GA

- Message Passing:
- identify size and location of data blocks
- loop over processors:
- if (me = P_N) then
- pack data in local message buffer
- send block of data to message buffer on P0
- else if (me = P0) then
- receive block of data from P_N in message buffer
- unpack data from message buffer to local buffer
- endif
- end loop
- copy local data on P0 to local buffer

- Global Arrays:
**          ****NGA_Get(g_a, lo, hi, buffer, ld);**

- Global Array handle

- }

- Global upper and lower indices of data patch

- Local buffer and array of strides

- }

- P0

- P1

- P2

- P3

- Global Arrays Tutorial

## Data Locality

- What data does a processor own?
-                  NGA_Distribution(g_a, iproc, lo, hi);
- Where is the data?
-                  NGA_Access(g_a, lo, hi, ptr, ld)
- Use this information to organize calculation so that maximum use is made of locally held data

- Global Arrays Tutorial

## Example: Matrix Multiply C=AxB

- Example: Matrix Multiply C=AxB

- local buffers on the processor

- global arrays representing matrices

- •

- •

- =

- =

- nga_get

- nga_put or
- ga_acces

- dgemm

*block*
*owner*

- Global Arrays Tutorial

## Matrix Multiply C=AxB(more memory efficient)

- Matrix Multiply C=AxB(more memory efficient)

- local buffers on the processor

**more scalable!**
- (less memory, higher parallelism)

- •

- •

- =

- =

- get

- atomic accumulate

*dgemm*

*multiple processors contribute*

- Global Arrays Tutorial

## Core Capabilities

- Distributed array library
- dense arrays 1-7 dimensions
- four data types (Fortran): *integer*,* real,* *double precision*, *double complex*
- five data types (C): *int, long, float, double, double complex*
- global rather than per-task view of data structures
- user control over data distribution: regular and irregular
- Collective and shared-memory style operations
- ga_sync, ga_scale, etc
- ga_put, ga_get, ga_acc
- nonblocking ga_put, ga_get, ga_acc
- Interfaces to third party parallel numerical libraries
- PeIGS, Scalapack, SUMMA, Tao
- example: to solve a linear system using LU factorization
**call ga_lu_solve(g_a, g_b)**
-                   instead of
**call pdgetrf(n,m, locA, p, q, dA, ind, info)**
**call pdgetrs(trans, n, mb, locA, p, q, dA,dB,info)**

- Global Arrays Tutorial

## Structure of GA

- Message Passing
*Global operations*

- ARMCI
*portable 1-sided*
*communication*
*put,get, locks, etc*

- distributed arrays layer
*memory management, index translation*

- system specific interfaces
*LAPI, GM/Myrinet, threads, VIA,..*

- Global Arrays and MPI are completely interoperable. Code can contain calls to both libraries.

- Fortran 77

- C

- C++

- Babel

- F90

- Python

- Java

- Application programming language interface

- Global Arrays Tutorial

## Global Arrays and CCA

- Common Component Architecture (CCA)
- Standard for Plug N’ Play HPC Component technology
- Advantages: software complexity and interoperability within and across scientific domains, addressing issues in programming language interoperability, domain-specific common interfaces, dynamic composability
- GA Component provides explicit interfaces (CCA *ports*) to other systems that expand functionality of GA
- For example: GA’s interoperability with TAO (Toolkit for Advanced Optimization – ANL) optimization component
- Language interoperable Fortran, C, C++, Python, Java and F90
- Multi-level Parallelism in applications using CCA’s MCMD Programming Model and GA Processor groups

- CCA-based quantum chemistry application which integrates NWChem, GA, TAO, PETSc, MPQC Components

- Global Arrays Tutorial

## Disk Resident Arrays

- Extend GA model to disk
- system similar to Panda (U. Illinois) but higher level APIs
- Provide easy transfer of data between N-dim arrays stored on disk and  distributed arrays stored in memory
- Use when
- Arrays too big to store in core
- checkpoint/restart
- out-of-core solvers

*global array*

***disk resident array***

- Global Arrays Tutorial

## Application Areas

- thermal flow simulation

- Visualization and image analysis

- electronic structure chemistry
**Major area**

- glass flow simulation

- material sciences

- molecular dynamics

- Others: financial security forecasting, astrophysics, geosciences, atmospheric chemistry

- Biology: organ simulation

- bioinformatics

- Global Arrays Tutorial

## Source Code and MoreInformation

- Version 4.0 available
- Homepage at http://www.emsl.pnl.gov/docs/global/
- Platforms (32 and 64 bit)
- IBM SP, IBM Blue Gene/L (see Thursday IBM technical paper)
- Cray X1, XD1, XT3 in beta version
- Linux Cluster with Ethernet, Myrinet, Infiniband, or Quadrics
- Solaris
- Fujitsu
- Hitachi
- NEC
- HP
- Windows

- Global Arrays Tutorial

## Overview of the Global ArraysParallel Software Development Toolkit Getting Started, Basic Calls

- Manojkumar Krishnan1
-  Jarek Nieplocha1 , Bruce Palmer1, Vinod Tipparaju1, P. Saddayappan2
- 1Pacific Northwest National Laboratory
- 2Ohio State University

## Outline

- Writing, Building, and Running GA Programs
- Basic Calls
- Intermediate Calls
- Advanced Calls

- Global Arrays Tutorial

## Writing, Building and Running GA programs

- Writing GA programs
- Compiling and linking
- Running GA programs
- For detailed information
- GA Webpage
- GA papers, APIs, user manual, etc.
- (Google: Global Arrays)
- _[http://](http://www.emsl.pnl.gov/docs/global/)__[www.emsl.pnl.gov](http://www.emsl.pnl.gov/docs/global/)__[/docs/global/](http://www.emsl.pnl.gov/docs/global/)_
- GA User Manual
- http://www.emsl.pnl.gov/docs/global/user.html
- GA API Documentation
- GA Webpage => User Interface
- http://www.emsl.pnl.gov/docs/global/userinterface.html
- GA Support/Help
- hpctools@pnl.gov or hpctools@emsl.pnl.gov
- 2 mailing lists: GA User Forum, and GA Announce

- Global Arrays Tutorial

## Writing GA Programs

- GA Definitions and Data types
- C programs include files:* *ga.h, macdecls.h
- Fortran programs should include the files: mafdecls.fh, global.fh.
-  GA Initialize, GA_Terminate --> initializes and terminates GA library

**#include <stdio.h> **
**#include "mpi.h“**
**#include "ga.h"**
**#include "macdecls.h"**
**int main( int argc, char **argv ) { **
**	****MPI_Init( &argc, &argv ); **
**	****GA_Initialize();**
**	****printf( "Hello world\n" ); **
**	**
**	****GA_Terminate();**
**	****MPI_Finalize(); **
**	****return 0; **
**}**** **

- Global Arrays Tutorial

## Writing GA Programs

- GA requires the following functionalities from a message passing library (MPI/TCGMSG)
- initialization and termination of processes
- Broadcast, Barrier
- a function to abort the running parallel job in case of an error
- The message-passing library has to be,
- initialized before the GA library
- terminated after the GA library is terminated
- GA Compatible with MPI

**#include <stdio.h> **
**#include "mpi.h“**
**#include "ga.h"**
**#include "macdecls.h"**
**int main( int argc, char **argv ) { **
**	****MPI_Init( &argc, &argv ); **
**	****GA_Initialize();**
**	****printf( "Hello world\n" ); **
**	**
**	****GA_Terminate();**
**	****MPI_Finalize(); **
**	****return 0; **
**} **

- Global Arrays Tutorial

## Compiling and Linking GA Programs

- 2 ways
- GA Makefile in global/testing
- Your Makefile
- GA Makefile in global/testing
- To compile and link your GA based program, for example “app.c” (or “app.f”, ..)
- Copy to $GA_DIR/global/testing, and type
- make app.x or gmake app.x
- Compile any test program in GA testing directory, and use the appropriate compile/link flags in your program

- Global Arrays Tutorial

## Compiling and Linking GA Programs (cont.)

- Your Makefile
- Please refer to the INCLUDES, FLAGS and LIBS variables, which will be printed at the end of successful GA installation on your platform
- You can use these variables in your Makefile
- For example: gcc $(INCLUDES) $(FLAGS) –o ga_test ga_test.c $(LIBS)

**INCLUDES**	= -I./include -I/msrc/apps/mpich-1.2.6/gcc/ch_shmem/include
**LIBS**  	= -L/msrc/home/manoj/GA/cvs/lib/LINUX -lglobal -lma  -llinalg
- 	  -larmci -L/msrc/apps/mpich-1.2.6/gcc/ch_shmem/lib -lmpich –lm
**For Fortran Programs:**
**FLAGS****	**= -g  -Wall -funroll-loops -fomit-frame-pointer -malign-double
- 	  -fno-second-underscore -Wno-globals
**For C Programs:**
**FLAGS**	= -g  -Wall -funroll-loops -fomit-frame-pointer -malign-double
- 	  -fno-second-underscore -Wno-globals

- NOTE: Please refer to GA user manual chapter 2 for more information

- Global Arrays Tutorial

## Running GA Programs

- Example: Running a test program “ga_test” on 2 processes
- mpirun -np 2 ga_test
- Running GA program is same as MPI

- Global Arrays Tutorial

## Setting GA Memory Limits

- GA offers an optional mechanism that allows a programmer to limit aggregate memory consumption used by GA for storing Global Array data
- Good for systems with limited-memory (e.g. BlueGene/L)
- GA uses Memory Allocator (MA) library for internal temporary buffers
- allocated dynamically, and deallocated when operation  completes.
- Users *should* specify memory usage of their application during GA initialization
- Can be specified in MA_Init(..) call after GA_Initialize()

**! Initialization of MA and setting GA memory limits**
**call ga_initialize()**
**if (ga_uses_ma()) then**
**    ****status = ma_init(MT_DBL, stack, heap+global)**
**else**
**    ****status = ma_init(mt_dbl,stack,heap)**
**    ****call ga_set_memory_limit(ma_sizeof(MT_DBL,global,MT_BYTE))**
**endif**
**if(.not. status) ... !we got an error condition here **

- Global Arrays Tutorial

## Outline

- Writing, Building, and Running GA Programs
- Basic Calls
- Intermediate Calls
- Advanced Calls

- Global Arrays Tutorial

## GA Basic Operations

- GA programming model is very simple.
- Most of the parallel programs can be written with these basic calls
**GA_Initialize, GA_Terminate**
**GA_Nnodes, GA_Nodeid**
**GA_Create, GA_Destroy**
**GA_Put, GA_Get**
**GA_Sync**

- Global Arrays Tutorial

## GA Initialization/Termination

- There are two functions to *initialize* GA:
**Fortran**
- subroutine ga_initialize()
- subroutine ga_initialize_ltd(limit)
**C**
- void GA_Initialize()
- void GA_Initialize_ltd(size_t limit)
- To *terminate* a GA program:
**Fortran**	subroutine ga_terminate()
**C**			void GA_Terminate()
- integer	limit	- amount of memory in bytes per process     [input]

-       **program main**
**      ****#include “mafdecls.h”**
**      ****#include “global.fh”**
**      ****integer ierr**
**c**
**      ****call mpi_init(ierr)**
**      ****call ga_initialize()**
**c**
**      ****write(6,*) ‘Hello world’**
**c**
**      ****call ga_terminate()**
**      ****call mpi_finilize()**
**      ****end**

- Global Arrays Tutorial

## Parallel Environment - Process Information

**Parallel Environment:**
- how many processes are working together (*size*)
- what their IDs are (ranges from 0 to *size*-1)
- To return the process ID of the current process:
**Fortran****	**integer function ga_nodeid()
**C****	****	****	**int GA_Nodeid()
- To determine the number of computing processes:
**Fortran****	**integer function ga_nnodes()
**C****	****	****	**int GA_Nnodes()

- Global Arrays Tutorial

## Parallel Environment - Process Information (EXAMPLE)

-  **program main**
** ****#include “mafdecls.h”**
** ****#include “global.fh”**
** ****integer ierr,me,nproc**
** ****call mpi_init(ierr)**
** ****call ga_initialize()**
** ****me = ga_nodeid()**
** ****size = ga_nnodes()**
** ****write(6,*) ‘Hello world: My rank is ’ +  me + ‘ out of ‘ + **
**!****	****	****size + ‘processes/nodes’**
** ****call ga_terminate()**
** ****call mpi_finilize()**
** ****end**

**$ mpirun –np 4 helloworld**
- Hello world: My rank is 0 out of 4 processes/nodes
- Hello world: My rank is 1 out of 4 processes/nodes
- Hello world: My rank is 3 out of 4 processes/nodes
- Hello world: My rank is 2 out of 4 processes/nodes

- Global Arrays Tutorial

## GA Data Types

- C Data types
- C_INT		- int
- C_LONG	- long
- C_FLOAT	- float
- C_DBL		- double
- C_DCPL	- double complex
- Fortran Data types
- MT_F_INT	- integer (4/8 bytes)
- MT_F_REAL	- real
- MT_F_DBL	- double precision
- MT_F_DCPL	- double complex

- Global Arrays Tutorial

## Creating/Destroying Arrays

- To *create* an array with regular distribution:
**Fortran****	****	**logical function _[nga_create](http://www.emsl.pnl.gov/docs/global/ga_ops.html#ga_create)_(type,				ndim, dims, name, chunk, g_a)
**C****	****	****	****	**int _[NGA_Create](http://www.emsl.pnl.gov/docs/global/c_nga_ops.html#ga_create)_(int type, int ndim,				int dims[], char *name, int chunk[])
- character*(*)	name	- a unique character string			[input]
- integer	type		- GA data type				[input]
- integer	dims() 	- array dimensions                      	[input]
- integer	chunk()	- minimum size that dimensions				  should be chunked into			[input]
- integer	g_a		- array handle for future references	[output]

-       **dims(1) = 5000**
**      ****dims(2) = 5000**
**      ****chunk(1) = -1     ****!Use defaults**
**      ****chunk(2) = -1**
**      ****if (.****not.****nga_create(MT_F_DBL,2,dims,’Array_A’,chunk,g_a)****)**
**     ****+    call ga_error(“Could not create global array A”,g_a)**

- Global Arrays Tutorial

## Creating/Destroying Arrays (cont.)

- To *create* an array with irregular distribution:
**Fortran****	****	**logical function _[nga_create_irreg](http://www.emsl.pnl.gov/docs/global/ga_ops.html#ga_create_irreg)_ 				(type, ndim, dims, array_name,				map, nblock, g_a)
**C**				int _[NGA_Create_irreg](http://www.emsl.pnl.gov/docs/global/c_nga_ops.html#ga_create_irreg)_(int type, 				int ndim, int dims[], char* 					array_name, nblock[], map[])
- character*(*)	name	- a unique character string				[input]
- integer 	type		- GA datatype					[input]
- integer	dims    	- array dimensions					[input]
- integer	nblock(*)	- no. of blocks each dimension is divided into	[input]
- integer	map(*)	- starting index for each block			[input]
- integer	g_a		- integer handle for future references		[output]

- Global Arrays Tutorial

## Creating/Destroying Arrays (cont.)

- Example of irregular distribution:
- The distribution is specified as a Cartesian product of distributions for each dimension. The array indices start at 1.
- The figure demonstrates distribution of a 2-dimensional array 8x10 on 6 (or more) processors. *block[2]*={3,2}, the size of *map* array is *s*=5 and array *map* contains the following elements *map*={1,3,7, 1, 6}.
- The distribution is nonuniform because, P1 and P4 get 20 elements each and processors P0,P2,P3, and P5 only 10 elements each.

-       **block(1) = 3**
**      ****block(2) = 2**
**      ****map(1) = 1**
**      ****map(2) = 3**
**      ****map(3) = 7**
**      ****map(4) = 1**
**      ****map(5) = 6**
**      ****if (.****not.****nga_create_irreg(MT_F_DBL,2,dims,’Array_A’,map,block,g_a)****)**
**     ****+    call ga_error(“Could not create global array A”,g_a)**

- 2

- P5

- P2

- 4

- P4

- P1

- 2

- P3

- P0

- 5

- 5

- Global Arrays Tutorial

## Creating/Destroying Arrays (cont.)

- To *duplicate* an array:
**Fortran**		logical function _[ga_duplicate](http://www.emsl.pnl.gov/docs/global/ga_ops.html#ga_duplicate)_(g_a, g_b, name)
**C**				int _[GA_Duplicate](http://www.emsl.pnl.gov/docs/global/c_nga_ops.html#ga_duplicate)_(int g_a, char *name)
- Global arrays can be *destroyed* by calling the function:
**Fortran****	**subroutine _[ga_destroy](http://www.emsl.pnl.gov/docs/global/ga_ops.html#ga_destroy)_(g_a)
**C**			void _[GA_Destroy](http://www.emsl.pnl.gov/docs/global/c_nga_ops.html#ga_destroy)_(int g_a)
- integer	g_a, g_b;
- character*(*)	name;
- name		- a character string				[input]
- g_a			- Integer handle for reference array		[input]
- g_b			- Integer handle for new array			[output]

-  **call ****nga_create(MT_F_INT,dim,dims,**
**+****    ‘array_a’,chunk,g_a)**
** ****call ****ga_duplicate(g_a,g_b,‘array_b’)**
** ****call ****ga_destroy(g_a)**

- Global Arrays Tutorial

## Put/Get

*Put* copies data from the local array to the global array section:
**Fortran****	****	**subroutine _[nga_put](http://www.emsl.pnl.gov/docs/global/ga_ops.html#ga_put)_(g_a, lo, hi, buf, ld)
**C**				void _[NGA_Put](http://www.emsl.pnl.gov/docs/global/c_nga_ops.html#ga_put)_(int g_a, int lo[], int hi[], void *buf, int ld[])
*Get *copies data from a global array section to the local array:
**Fortran****	****	**subroutine _[nga_get](http://www.emsl.pnl.gov/docs/global/ga_ops.html#ga_get)_(g_a, lo, hi, buf, ld)
**C**				void _[NGA_Get](http://www.emsl.pnl.gov/docs/global/c_nga_ops.html#ga_get)_(int g_a, int lo[], int hi[], void *buf, int ld[])
- integer	g_a global array handle				[input]
- integer	lo(),hi() limits on data block to be moved	[input]
- double		precision/complex/integer buf	local buffer	[output]
- integer	ld() array of strides for local buffer		[input]

- Global Arrays Tutorial

## Put/Get (cont.)

- Example of *put* operation:
- transfer data from a local buffer (10 x10 array)* *to (7:15,1:8) section of a 2-dimensional 15 x10 global array into *lo*={7,1}, *hi*={15,8}, ld={10}

**double precision buf(10,10)**
**  ****:**
**  ****:**
**call ****nga_put(g_a,lo,hi,buf,ld)**

- lo

- hi

- global

- local

- Global Arrays Tutorial

## Atomic Accumulate

*Accumulate* combines the data from the local array with data in the global array section:
**Fortran****	****	**subroutine _[nga_acc](http://www.emsl.pnl.gov/docs/global/ga_ops.html#ga_acc)_(g_a, lo, hi,
- 						buf, ld, alpha)
**C**				void _[NGA_Acc](http://www.emsl.pnl.gov/docs/global/c_nga_ops.html#ga_acc)_(int g_a, int lo[], 				int hi[], void *buf, int ld[], 				void *alpha)
- integer	g_a array handle							[input]
- integer 	lo(), hi() limits on data block to be moved 		[input]
- double 	precision/complex buf	local buffer				[input]
- integer 	ld() array of strides for local buffer				[input]
- double 	precision/complex alpha arbitrary scale factor		[input]

- ga(i,j) = ga(i,j)+alpha*buf(k,l)

- global

- local

- Global Arrays Tutorial

## Memory and Task Synchronization

*Sync* is a collective operation
- It acts as a barrier, which synchronizes all the processes and ensures that all the Global Array operations are complete at the call
- The functions are:
**Fortran****	**subroutine _[ga_sync](http://www.emsl.pnl.gov/docs/global/ga_ops.html#ga_sync)_()
**C****	****	****	**void _[GA_Sync](http://www.emsl.pnl.gov/docs/global/c_nga_ops.html#ga_sync)_()

- sync

- Global Arrays Tutorial

## Message-Passing Wrappers to Reduce/Broadcast Operations

- To broadcast from process root to all other processes:
**Fortran****	**subroutine _[ga_brdcst](http://www.emsl.pnl.gov/docs/global/ga_ops.html#ga_brdcst)_(type, buf, lenbuf,			root)
**C****	****	****	**void _[GA_Brdcst](http://www.emsl.pnl.gov/docs/global/c_nga_ops.html#ga_brdcst)_(void *buf, int lenbuf,			int root)
- integer	type       		[input] ! message type for broadcast
- byte		buf(lenbuf)		[input/output]
- integer	lenbuf			[input]
- integer	root			[input]

- root

**double precision buf(lenbuf)**
**  ****:**
**size = 8*lenbuf**
**call ****ga_brdcst(msg_id,buf,size,0)**

- Global Arrays Tutorial

## Message-Passing Wrappers to Reduce/Broadcast Operations (cont.)

- To sum elements across all nodes and broadcast result to all nodes:
**Fortran**
- Subroutine _[ga_igop](http://www.emsl.pnl.gov/docs/global/ga_ops.html#ga_igop)_(type, x, n, op)
- Subroutine _[ga_dgop](http://www.emsl.pnl.gov/docs/global/ga_ops.html#ga_igop)_(type, x, n, op)
**C**
- Void _[GA_Igop](http://www.emsl.pnl.gov/docs/global/c_nga_ops.html#ga_igop)_(long x[], int n, char *op)
- Void _[GA_Dgop](http://www.emsl.pnl.gov/docs/global/c_nga_ops.html#ga_dgop)_(double x[], int n, char *op)
- integer		type		[input]
- <type>		x(n)	vector of elements	[input/output]
- character*(*)		OP	operation	[input]
- integer		n	number of elements	[input]

**double precision rbuf(len)**
**   ****:**
**call ****ga_dgop(msg_id,rbuf,len,’+’)**

- Global Arrays Tutorial

## Locality Information

- Discover array elements held by each processor
**Fortran****	**nga_distribution(g_a,proc,lo,hi)
**C****	**		void NGA_Distribution(int g_a, int proc, int *lo, int *hi)
- integer		g_a		array handle		[input]
- integer		proc		processor ID		[input]
- integer		lo(ndim)	lower index		[output]
- integer		hi(ndim)	upper index		[output]

**do iproc = 1, nproc**
**  ****write(6,*) ‘Printing g_a info for processor’,iproc**
**  ****call ****nga_distribution(g_a,iproc,lo,hi)**
**  ****do j = 1, ndim**
**    ****write(6,*) j,lo(j),hi(j)**
**  ****end do**
**dnd do**

- Global Arrays Tutorial

## Example: Matrix Multiply

*/* Determine which block of data is locally owned. Note that*
*   **the same block is locally owned for all GAs. */*
- NGA_Distribution(g_c, me, lo, hi);
*/* Get the blocks from g_a and g_b needed to compute this block in*
*   **g_c and copy them into the local buffers a and b. */*
- lo2[0] = lo[0]; lo2[1] = 0; hi2[0] = hi[0]; hi2[1] = dims[0]-1;
- NGA_Get(g_a, lo2, hi2, a, ld);
- lo3[0] = 0; lo3[1] = lo[1]; hi3[0] = dims[1]-1; hi3[1] = hi[1];
- NGA_Get(g_b, lo3, hi3, b, ld);
*/* Do local matrix multiplication and store the result in local*
*   **buffer c. Start by evaluating the transpose of b. */*
**for**(i=0; i < hi3[0]-lo3[0]+1; i++)
-   **for**(j=0; j < hi3[1]-lo3[1]+1; j++)
-      btrns[j][i] = b[i][j];
*/* Multiply a and b to get c */*
**for**(i=0; i < hi[0] - lo[0] + 1; i++) {
-   **for**(j=0; j < hi[1] - lo[1] + 1; j++) {
-     c[i][j] = 0.0;
-     **for**(k=0; k<dims[0]; k++)
-       c[i][j] = c[i][j] + a[i][k]*btrns[j][k];
-   }
- }
*/* Copy c back to g_c */*
- NGA_Put(g_c, lo, hi, c, ld);

- •

- •

- =

- =

- nga_get

- nga_put

- dgemm

- Global Arrays Tutorial

## Overview of the Global ArraysParallel Software Development Toolkit Intermediate and Advanced Calls

- Bruce Palmer1
-  Jarek Nieplocha1 , Manojkumar Krishnan1, Vinod Tipparaju1, P. Saddayappan2
- 1Pacific Northwest National Laboratory
- 2Ohio State University

## Outline

- Writing, Building, and Running GA Programs
- Basic Calls
- Intermediate Calls
- Advanced Calls

- Global Arrays Tutorial

## Basic Array Operations

- Whole Arrays:
- To set all the elements in the array to zero:
**Fortran****	**subroutine _[ga_zero](http://www.emsl.pnl.gov/docs/global/ga_ops.html#ga_zero)_(g_a)
**C****	**	void _[GA_Zero](http://www.emsl.pnl.gov/docs/global/c_nga_ops.html#ga_zero)_(int g_a)
- To assign a single value to all the elements in array:
**Fortran****	**subroutine _[ga_fill](http://www.emsl.pnl.gov/docs/global/ga_ops.html#ga_fill)_(g_a, val)
**C****	****	**void _[GA_Fill](http://www.emsl.pnl.gov/docs/global/c_nga_ops.html#ga_fill)_(int g_a, void *val)
- To scale all the elements in the array by factor*val*:
**Fortran****	**subroutine _[ga_scale](http://www.emsl.pnl.gov/docs/global/ga_ops.html#ga_scale)_(g_a, val)
**C****	****	**void _[GA_Scale](http://www.emsl.pnl.gov/docs/global/c_nga_ops.html#ga_scale)_(int g_a, void *val)

- Global Arrays Tutorial

## Basic Array Operations (cont.)

- Whole Arrays:
**To copy data between two arrays:**
**Fortran****	**subroutine _[ga_copy](http://www.emsl.pnl.gov/docs/global/ga_ops.html#ga_copy)_(g_a, g_b)
**C****	****	**void _[GA_Copy](http://www.emsl.pnl.gov/docs/global/c_nga_ops.html#ga_copy)_(int g_a, int g_b)
- Arrays must be same size and dimension
- Distribution may be different

**call ****ga_create(MT_F_INT,ndim,dims,**
**     ‘****array_A’,chunk_a,g_a)**
**call ****nga_create(MT_F_INT,ndim,dims,**
**     ‘****array_B’,chunk_b,g_b)**
**    ****... Initialize g_a ....**
**call ****ga_copy(g_a, g_b)**

- “g_a”

- “g_b”

- Global Arrays g_a and g_b distributed on a 3x3 process grid

- Global Arrays Tutorial

## Basic Array Operations (cont.)

- Patch Operations:
- The copy patch operation:
**Fortran****	****	**subroutine _[nga_copy_patch](http://www.emsl.pnl.gov/docs/global/ga_ops.html#ga_copy_patch)_(trans, g_a,			alo, ahi, g_b, blo, bhi)
**C****	****	****	**void _[NGA_Copy_patch](http://www.emsl.pnl.gov/docs/global/c_nga_ops.html#ga_copy_patch)_(char trans, int g_a,			int alo[], int ahi[], int g_b, int blo[], 			int bhi[])
- Number of elements must match

- “g_a”

- “g_b”

**Copy**

- Global Arrays Tutorial

## Basic Array Operations (cont.)

- Patches (Cont):
- To set only the region defined by *lo* and *hi* to zero:
**Fortran****	**subroutine _[nga_zero_patch](http://www.emsl.pnl.gov/docs/global/ga_ops.html#ga_zero_patch)_(g_a, lo, hi)
**C****	****	**void _[NGA_Zero_patch](http://www.emsl.pnl.gov/docs/global/c_nga_ops.html#ga_zero_patch)_(int g_a, int lo[] int hi[])
- To assign a single value to all the elements in a patch:
**Fortran****	**subroutine _[nga_fill_patch](http://www.emsl.pnl.gov/docs/global/ga_ops.html#ga_fill_patch)_(g_a, lo, hi, val)
**C****	****	****	**void_[NGA_Fill_patch](http://www.emsl.pnl.gov/docs/global/c_nga_ops.html#ga_fill_patch)_(int g_a, 			int lo[] int hi[], void *val)
- To scale the patch defined by lo and hi by the factor val:
**Fortran****	**subroutine _[nga_scale_patch](http://www.emsl.pnl.gov/docs/global/ga_ops.html#ga_scale_patch)_(g_a, lo, hi,			val)
**C****	****	****	**void_[NGA_Scale_patch](http://www.emsl.pnl.gov/docs/global/c_nga_ops.html#ga_scale_patch)_(int g_a, 			int lo[] int hi[], void *val)
- The copy patch operation:
**Fortran****	**subroutine _[nga_copy_patch](http://www.emsl.pnl.gov/docs/global/ga_ops.html#ga_copy_patch)_(trans, g_a,			alo, ahi, g_b, blo, bhi)
**C****	****	****	**void _[NGA_Copy_patch](http://www.emsl.pnl.gov/docs/global/c_nga_ops.html#ga_copy_patch)_(char trans, int g_a,			int alo[], int ahi[], int g_b, int blo[], 			int bhi[])

- Global Arrays Tutorial

## Scatter/Gather

*Scatter* puts array elements into a global array:
**Fortran****	****	**subroutine _[nga_scatter](http://www.emsl.pnl.gov/docs/global/ga_ops.html#ga_scatter)_(g_a, v,				subscrpt_array, n)
**C**				void _[NGA_Scatter](http://www.emsl.pnl.gov/docs/global/c_nga_ops.html#ga_scatter)_(int g_a, void *v,				int *subscrpt_array[], int n)
*Gather* gets the array elements from a global array into a local array:
**Fortran****	****	**subroutine _[nga_gather](http://www.emsl.pnl.gov/docs/global/ga_ops.html#ga_gather)_(g_a, v,				subscrpt_array, n)
**C**				void _[NGA_Gather](http://www.emsl.pnl.gov/docs/global/c_nga_ops.html#ga_gather)_(int g_a, void *v,				int *subscrpt_array[], int n)
- integer		g_a array handle		 				[input]
- double precision	v(n) array of values						         [input/output]
- integer 		n number of values						[input]
- integer		subscrpt_array location of values in global array	[input]

- Global Arrays Tutorial

## Scatter/Gather (cont.)

- Example of *scatter* operation:
- Scatter the 5 elements into a 10x10 global array
- Element 1		v[0] = 5	subsArray[0][0] = 2					subsArray[0][1] = 3
- Element 2		v[1] = 3	subsArray[1][0] = 3					subsArray[1][1] = 4
- Element 3		v[2] = 8	subsArray[2][0] = 8					subsArray[2][1] = 5
- Element 4		v[3] = 7	subsArray[3][0] = 3					subsArray[3][1] = 7
- Element 5		v[4] = 2	subsArray[4][0] = 6					subsArray[4][1] = 3
- After the *scatter* operation, the five elements would be scattered into the global array as shown in the figure.

|  | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 0 |  |  |  |  |  |  |  |  |  |  |
| 1 |  |  |  |  |  |  |  |  |  |  |
| 2 |  |  |  | 5 |  |  |  |  |  |  |
| 3 |  |  |  |  | 3 |  |  | 7 |  |  |
| 4 |  |  |  |  |  |  |  |  |  |  |
| 5 |  |  |  |  |  |  |  |  |  |  |
| 6 |  |  |  | 2 |  |  |  |  |  |  |
| 7 |  |  |  |  |  |  |  |  |  |  |
| 8 |  |  |  |  |  | 8 |  |  |  |  |
| 9 |  |  |  |  |  |  |  |  |  |  |

**integer subscript(ndim,nlen)**
**   ****:**
**call ****nga_scatter(g_a,v,subscript,nlen)**

- Global Arrays Tutorial

## Read and Increment

*Read_inc* remotely updates a particular element in an integer global array:
**Fortran****	**integer function _[nga_read_inc](http://www.emsl.pnl.gov/docs/global/ga_ops.html#ga_read_inc)_(g_a,				subscript, inc)
**C**				long _[NGA_Read_inc](http://www.emsl.pnl.gov/docs/global/c_nga_ops.html#ga_read_inc)_(int g_a,				int subscript[], long inc)
- Applies to integer arrays only
- Example: can be used as a global counter
- integer	g_a									[input]
- integer 	subscript(ndim), inc							[input]

**c  Create task counter**
**      ****call ****nga_create(MT_F_INT,one,one,chunk,g_counter)**
**      ****call ****ga_zero(g_counter)**
**         ****:**
**      ****itask = ****nga_read_inc(g_counter,one,one)**
**      ****... Translate itask into task ...**

- Global Arrays Tutorial

## Outline

- Writing, Building, and Running GA Programs
- Basic Calls
- Intermediate Calls
- Advanced Calls

- Global Arrays Tutorial

## Access

- To provide direct access to local data in the specified patch of the array owned by the calling process:
**Fortran****	****	**subroutine _[nga_access](http://www.emsl.pnl.gov/docs/global/ga_ops.html#ga_access)_(g_a, lo, hi,				index, ld)
**C****	****	****	****	**void _[NGA_Access](http://www.emsl.pnl.gov/docs/global/c_nga_ops.html#ga_access)_(int g_a,
- 						int lo[], int hi[], void *ptr, int ld[])
- Processes can access the local position of the global array
- Process “0” can access the specified patch of its local position of the array
- Avoids memory copy

**call ****nga_create(MT_F_DBL,2,dims,’Array’,chunk,g_a)**
**  ****:**
**call ****nga_distribution(g_a,me,lo,hi)**
**call ****nga_access(g_a,lo,hi,index,ld)**
**call do_subroutine_task(****dbl_mb(index)****,ld(1))**
**call ****nga_release(g_a,lo,hi)**
**subroutine do_subroutine_task(a,ld1)**
**double precision a(ld1,*)**

- 0

- 1

- 2

- 3

- 4

- 5

- 6

- 7

- 8

**Access**: gives a pointer to this local patch

- Global Arrays Tutorial

## Locality Information (cont.)

- Global Arrays support abstraction of a distributed array object
- Object is represented by an integer handle
- A process can access its portion of the data in the global array
- To do this, the following steps need to be taken:
- Find the distribution of an array, i.e. which part of the data the calling process owns
- Access the data
- Operate on the data: read/write
- Release the access to the data

- Global Arrays Tutorial

## Non-blocking Operations

- The non-blocking APIs are derived from the blocking interface by adding a handle argument that identifies an instance of the non-blocking request.
**Fortran**
- subroutine _[nga_nbput](http://www.emsl.pnl.gov/docs/global/ga_ops.html#nga_nbput)_(g_a, lo, hi, buf, ld, nbhandle)
- subroutine _[nga_nbget](http://www.emsl.pnl.gov/docs/global/ga_ops.html#nga_nbget)_(g_a, lo, hi, buf, ld, nbhandle)
- subroutine _[nga_nbacc](http://www.emsl.pnl.gov/docs/global/ga_ops.html#nga_nbacc)_(g_a, lo, hi, buf, ld, alpha, nbhandle)
- subroutine _[nga_nbwait](http://www.emsl.pnl.gov/docs/global/ga_ops.html#nga_nbwait)_(nbhandle)
**C**
- void _[NGA_NbPut](http://www.emsl.pnl.gov/docs/global/c_nga_ops.html#ga_nbput)_(int g_a, int lo[], int hi[], void *buf,int ld[], ga_nbhdl_t* nbhandle)
- void _[NGA_NbGet](http://www.emsl.pnl.gov/docs/global/c_nga_ops.html#ga_nbget)_(int g_a, int lo[], int hi[], void *buf, int ld[], ga_nbhdl_t* nbhandle)
- void _[NGA_NbAcc](http://www.emsl.pnl.gov/docs/global/c_nga_ops.html#ga_nbacc)_(int g_a, int lo[], int hi[], void *buf, int ld[], void *alpha, ga_nbhdl_t* nbhandle)
- int _[NGA_NbWait](http://www.emsl.pnl.gov/docs/global/c_nga_ops.html#ga_nbwait)_(ga_nbhdl_t* nbhandle)
- integer	nbhandle	- non-blocking request handle			[output/input]

- Global Arrays Tutorial

## Non-Blocking Operations

**double precision buf1(nmax,nmax)**
**double precision buf2(nmax,nmax)**
**   ****:**
**call ****nga_nbget(g_a,lo1,hi1,buf1,ld1,nb1)**
**ncount = 1**
**do while(.....)**
**  ****if (mod(ncount,2).eq.1) then**
**     ****... Evaluate lo2, hi2**
**     ****call ****nga_nbget(g_a,lo2,hi2,buf2,nb2)**
**     ****call ****nga_wait(nb1)**
**     ****... Do work using data in buf1**
**  ****else**
**     ****... Evaluate lo1, hi1**
**     ****call ****nga_nbget(g_a,lo1,hi1,buf1,nb1)**
**     ****call ****nga_wait(nb2)**
**     ****... Do work using data in buf2**
**  ****endif**
**  ****ncount = ncount + 1**
**end do**

- Global Arrays Tutorial

## SUMMA Matrix Multiplication

**A**

**B**

**C=A.B**

- Computation

- Comm.
- (Overlap)

**Issue NB Get A and B blocks**
**do**** (until last chunk)**
**    ****issue NB Get to the next blocks**
**    ****wait for previous issued call**
**    ****compute A*B (sequential dgemm)**
**    ****NB atomic accumulate into “C”      **
**        ****matrix**
**done**

**Advantages:**
** ****- Minimum memory**
** ****- Highly parallel**
** ****- Overlaps computation and communication**
**          ****- latency hiding**
** ****- exploits data locality**
** ****- patch matrix multiplication (easy to use)**
** ****- dynamic load balancing**

**patch matrix multiplication**

**=**

- Global Arrays Tutorial

## SUMMA Matrix Multiplication:Improvement over PBLAS/ScaLAPACK

- Global Arrays Tutorial

## Cluster Information

- To return the total number of nodes that the program is running on:
**Fortran**	integer function ga_cluster_nnodes()
**C****	****	****	**int GA_Cluster_nnodes()
- To return the node ID of the process:
**Fortran****	**integer function ga_cluster_nodeid()
**C****	****	****	**int GA_Cluster_nodeid()

- N0

- N1

- Global Arrays Tutorial

## Cluster Information (cont.)

- To return the number of processors available on node inode:
**Fortran****	**integer function ga_cluster_nprocs(inode)
**C****	****	****	**int GA_Cluster_nprocs(int inode).
- To return the processor ID associated with node inode and the local processor ID iproc:
**Fortran****	**integer function ga_cluster_procid(inode,			iproc)
**C****	****	****	**int GA_Cluster_procid(int inode, int iproc)
- integer	inode				[input]
- 			inode				[input]
- integer	inode,iproc			[input]
- 			inode,iproc			[input]

- 0(0)

- 1(1)

- 2(2)

- 3(3)

- 4(0)

- 5(1)

- 6(2)

- 7(3)

- Global Arrays Tutorial

## Cluster Information (cont.)

_*Example:*_
- 2 nodes with 4 processors each. Say, there are 7 processes created.
- ga_cluster_nnodes returns 2
- ga_cluster_nodeid returns 0 or 1
- ga_cluster_nprocs(inode) returns 4 or 3
- ga_cluster_procid(inode,iproc) returns a processor ID

- Global Arrays Tutorial

## Processor Groups

- To create a new processor group:
**Fortran**	integer function _[ga_pgroup_create](http://www.emsl.pnl.gov/docs/global/ga_ops.html#GA_PGROUP_CREATE)_(list, size)
**C****	****	****	**int** **_**[GA_Pgroup_create](http://www.emsl.pnl.gov/docs/global/c_nga_ops.html#GA_PGROUP_CREATE)**_(int *list, int size)
- To assign a processor groups:
**Fortran****	**logical function nga_create_config(type,				ndim, dims, name, chunk, p_handle, g_a)
**C****	****	****	**int NGA_Create_config(int type, int ndim,				int dims[], char *name, int p_handle, int chunk[])
- integer	g_a		- global array handle			[input]
- integer	p_handle	- processor group handle			[output]
- integer	list(size)	list of processor IDs in group		[input]
- integer	size		number of processors in group		[input]

- Global Arrays Tutorial

## Processor Groups

- world group

- group A

- group B

- group C

- Global Arrays Tutorial

## Processor Groups (cont.)

- To set the default processor group
**Fortran****	**subroutine _[ga_pgroup_set_default](http://www.emsl.pnl.gov/docs/global/ga_ops.html#GA_PGROUP_SET_DEFAULT)_(p_handle)
**C****	****	****	**void** **_**[GA_Pgroup_set_default](http://www.emsl.pnl.gov/docs/global/c_nga_ops.html#GA_PGROUP_SET_DEFAULT)**_(int p_handle)
- To access information about the processor group:
**Fortran**
- integer function _[ga_pgroup_nnodes](http://www.emsl.pnl.gov/docs/global/ga_ops.html#GA_PGROUP_NNODES)_(p_handle)
- integer function _[ga_pgroup_nodeid](http://www.emsl.pnl.gov/docs/global/ga_ops.html#GA_PGROUP_NODEID)_(p_handle)
**C**
- int** **_**[GA_Pgroup_nnodes](http://www.emsl.pnl.gov/docs/global/c_nga_ops.html#GA_PGROUP_NNODES)**_(int p_handle)
- int** **_**[GA_Pgroup_nodeid](http://www.emsl.pnl.gov/docs/global/c_nga_ops.html#GA_PGROUP_NODEID)**_(int p_handle)**      **
- integer	p_handle	- processor group handle			[input]

- Global Arrays Tutorial

## Processor Groups (cont.)

- To determine the handle for a standard group at any point in the program:
**Fortran**
- integer function _[ga_pgroup_get_default](http://www.emsl.pnl.gov/docs/global/ga_ops.html#GA_PGROUP_GET_DEFAULT)_()
- integer function _[ga_pgroup_get_mirror](http://www.emsl.pnl.gov/docs/global/ga_ops.html#GA_PGROUP_GET_MIRROR)_()
- integer function _[ga_pgroup_get_world](http://www.emsl.pnl.gov/docs/global/ga_ops.html#GA_PGROUP_GET_WORLD)_()
**C**
- int _[GA_Pgroup_get_default](http://www.emsl.pnl.gov/docs/global/c_nga_ops.html#GA_PGROUP_GET_DEFAULT)_()
- int _[GA_Pgroup_get_mirror](http://www.emsl.pnl.gov/docs/global/c_nga_ops.html#GA_PGROUP_GET_MIRROR)_()
- int _[GA_Pgroup_get_world](http://www.emsl.pnl.gov/docs/global/c_nga_ops.html#GA_PGROUP_GET_WORLD)_()

- Global Arrays Tutorial

## Processor Groups (cont.)

- Collective Operations on Groups:
- Identical to the standard global operations, except they have an extra argument that takes a group handle
**Fortran**
- subroutine _[ga_pgroup_sync](http://www.emsl.pnl.gov/docs/global/ga_ops.html#GA_PGROUP_SYNC)_(p_handle)
- subroutine _[ga_pgroup_brdcst](http://www.emsl.pnl.gov/docs/global/ga_ops.html#GA_PGROUP_BRDCST)_(p_handle,type,buf,lenbuf,root)
- subroutine _[ga_pgroup_dgop](http://www.emsl.pnl.gov/docs/global/ga_ops.html#GA_PGROUP_DGOP)_(p_handle, type, buf, lenbuf, op)
- subroutine _[ga_pgroup_sgop](http://www.emsl.pnl.gov/docs/global/ga_ops.html#GA_PGROUP_SGOP)_(p_handle, type, buf, lenbuf, op)
- subroutine _[ga_pgroup_igop](http://www.emsl.pnl.gov/docs/global/ga_ops.html#GA_PGROUP_IGOP)_(p_handle, type, buf, lenbuf, op)
**C**
- void** **GA_Pgroup_sync(p_handle)
- void** **GA_Pgroup_brdcst(int p_handle, void *buf, root)
- void** **_**[GA_Pgroup_dgop](http://www.emsl.pnl.gov/docs/global/c_nga_ops.html#GA_PGROUP_DGOP)**_(int p_handle, double *buf, int lenbuf, char *op)
- void** **_**[GA_Pgroup_fgop](http://www.emsl.pnl.gov/docs/global/c_nga_ops.html#GA_PGROUP_FGOP)**_(int p_handle, float *buf, int lenbuf, char *op)
- void** **_**[GA_Pgroup_igop](http://www.emsl.pnl.gov/docs/global/c_nga_ops.html#GA_PGROUP_IGOP)**_(int p_handle, int *buf, int lenbuf, char *op)
- void** **_**[GA_Pgroup_lgop](http://www.emsl.pnl.gov/docs/global/c_nga_ops.html#GA_PGROUP_LGOP)**_(int p_handle, long *buf, int lenbuf, char *op)

- Global Arrays Tutorial

## c
c  create subgroup p_a
c
      p_a=ga_pgroup_create(list, nproc)
      call ga_pgroup_set_default(p_a)
      call parallel_task(p_a)
      call ga_pgroup_set_default(ga_pgroup_get_world())

            subroutine parallel_task(p_b)
            p_b=ga_pgroup_create(list, nproc)
            call ga_pgroup_set_default(p_b)
            call parallel_subtask(p_b)
            call ga_pgroup_set_default(p_b)

- c
- c  create subgroup p_a
- c
-       p_a=ga_pgroup_create(list, nproc)
-       call ga_pgroup_set_default(p_a)
-       call parallel_task(p_a)
-       call ga_pgroup_set_default(ga_pgroup_get_world())
-             subroutine parallel_task(p_b)
-             p_b=ga_pgroup_create(list, nproc)
-             call ga_pgroup_set_default(p_b)
-             call parallel_subtask(p_b)
-             call ga_pgroup_set_default(p_b)

- Default Processor Group

- Global Arrays Tutorial

## MD Application on Groups

- Global Arrays Tutorial

## Creating Arrays with Ghost Cells

- To create arrays with ghost cells:
- For arrays with regular distribution:
**Fortran****	****	**logical function _[nga_create_ghosts](http://www.emsl.pnl.gov/docs/global/ga_ops.html#ga_create_ghosts)_(type,			dims, width, array_name, chunk, g_a)
**C**			int _[int](http://www.emsl.pnl.gov/docs/global/c_nga_ops.html#nga_create_ghosts)__[ ](http://www.emsl.pnl.gov/docs/global/c_nga_ops.html#nga_create_ghosts)__[NGA_Create_ghosts](http://www.emsl.pnl.gov/docs/global/c_nga_ops.html#nga_create_ghosts)_(int type, 			int ndim, int dims[], int width[], 			char *array_name, int chunk[])
- For arrays with irregular distribution:
**n-d Fortran****	**logical function			_[nga_create_ghosts_irreg](http://www.emsl.pnl.gov/docs/global/ga_ops.html#ga_create_ghosts_irreg)_(type,			dims, width, array_name, map, block,			g_a)
**C****	****	****	**int _[int](http://www.emsl.pnl.gov/docs/global/c_nga_ops.html#nga_create_ghosts_irreg)__[ ](http://www.emsl.pnl.gov/docs/global/c_nga_ops.html#nga_create_ghosts_irreg)__[NGA_Create_ghosts_irreg](http://www.emsl.pnl.gov/docs/global/c_nga_ops.html#nga_create_ghosts_irreg)_(int type,			int ndim, int dims[], int width[], 			char *array_name, int map[], int block[])
- integer	width(ndim)		- array of ghost cell widths		[input]

- Code

- Global Arrays Tutorial

## Ghost Cells

- normal global array

- global array with ghost cells

- Operations:
- 	NGA_Create_ghosts      - creates array with ghosts cells
- 	GA_Update_ghosts	     - updates with data from adjacent processors
- 	NGA_Access_ghosts	     - provides access to “local” ghost cell elements
- 	NGA_Nbget_ghost_dir   - nonblocking call to update ghosts cells

- Global Arrays Tutorial

## Ghost Cell Update

- Automatically update ghost cells with appropriate data from neighboring processors. A multiprotocol implementation has been used to optimize the update operation to match platform characteristics.

- Global Arrays Tutorial

## Periodic Interfaces

- Periodic interfaces to the one-sided operations have been added to Global Arrays in version 3.1 to support computational fluid dynamics problems on multidimensional grids.
- They provide an index translation layer that allows users to request blocks using put, get, and accumulate operations that possibly extend beyond the boundaries of a global array.
- The references that are outside of the boundaries are wrapped around inside the global array.
- Current version of GA supports three periodic operations:
*periodic get*
*periodic put*
*periodic acc*

**call ****nga_periodic_get(g_a,lo,hi,buf,ld)**

- global

- local

- Global Arrays Tutorial

## Periodic Interfaces (cont.)

- Example of periodic interfaces:
- Assume a two dimensional global array g_a with dimensions 5 X 5 (Figure 1)
- To access a patch [2:4,-1:3], one can assume that the array is wrapped over in the second dimension (Figure 2)
- Therefore the patch [2:4, -1:3] is as shown in Figure 3

**Figure 1**

**Figure 2**

**Figure 3**

- Global Arrays Tutorial

## Periodic Get/Put/Accumulate

**Fortran****	**subroutine _[nga_periodic_get](http://www.emsl.pnl.gov/docs/global/ga_ops.html#ga_periodic_get)_(g_a, lo, hi, buf, ld)
**C**			void _[NGA_Periodic_get](http://www.emsl.pnl.gov/docs/global/c_nga_ops.html#ga_periodic_get)_(int g_a, int lo[], int hi[], void *buf, int ld[])
-  **Fortran****	**subroutine _[nga_periodic_put](http://www.emsl.pnl.gov/docs/global/ga_ops.html#ga_periodic_put)_(g_a, lo, hi, buf, ld)
**C****	****	****	**void _[NGA_Periodic_put](http://www.emsl.pnl.gov/docs/global/c_nga_ops.html#ga_periodic_put)_(int g_a, int lo[], int hi[], void *buf, int ld[])
**Fortran****	**subroutine _[nga_periodic_acc](http://www.emsl.pnl.gov/docs/global/ga_ops.html#ga_periodic_acc)_(g_a, lo, hi, buf, ld, alpha)
**C****	****	****	**void _[NGA_Periodic_acc](http://www.emsl.pnl.gov/docs/global/c_nga_ops.html#ga_periodic_acc)_(int g_a, int lo[], int hi[], void *buf, int ld[],				void *alpha)

- Global Arrays Tutorial

## Lock and Mutex

*Lock* works together with *mutex*.
- Simple synchronization mechanism to protect a critical section
- To enter a critical section, typically, one needs to:
- Create mutexes
- Lock on a mutex
- Do the exclusive operation in the critical section
- Unlock the mutex
- Destroy mutexes
- The *create mutex* functions are:
**Fortran****	**logical function _[ga_create_mutexes](http://www.emsl.pnl.gov/docs/global/ga_ops.html#ga_create_mutex)_(number)
**C****	****	****	**int _[GA_Create_mutexes](http://www.emsl.pnl.gov/docs/global/c_nga_ops.html#ga_create_mutexes)_(int number)
- number	- number of mutexes in mutex array				[input]

- Global Arrays Tutorial

## Lock and Mutex (cont.)

- The *destroy mutex* functions are:
**Fortran****	**logical function _[ga_destroy_mutexes](http://www.emsl.pnl.gov/docs/global/ga_ops.html#ga_destroy_mutex)_()
**C****	****	****	**int _[GA_Destroy_mutexes](http://www.emsl.pnl.gov/docs/global/c_nga_ops.html#ga_destroy_mutexes)_()
- The *lock *and* unlock* functions are:
**Fortran**
- subroutine _[ga_lock](http://www.emsl.pnl.gov/docs/global/ga_ops.html#ga_lock)_(int mutex)
- subroutine _[ga_unlock](http://www.emsl.pnl.gov/docs/global/ga_ops.html#ga_unlock)_(int mutex)
**C**
- void _[GA_lock](http://www.emsl.pnl.gov/docs/global/c_nga_ops.html#ga_lock)_(int mutex)
- void _[GA_unlock](http://www.emsl.pnl.gov/docs/global/c_nga_ops.html#ga_unlock)_(int mutex)
- integer	mutex			[input]  ! mutex id

- Global Arrays Tutorial

## Fence

*Fence* blocks the calling process until all the data transfers corresponding to the Global Array operations initiated by this process complete
- For example, since ga_put might return before the data reaches final destination, ga_init_fence and ga_fence allow process to wait until the data transfer is fully completed
- ga_init_fence();
- ga_put(g_a, ...);
- ga_fence();
- The *initialize fence* functions are:
**Fortran**	subroutine _[ga_init_fence](http://www.emsl.pnl.gov/docs/global/ga_ops.html#ga_init_fence)_()
**C****	****	****	**void _[GA_Init_fence](http://www.emsl.pnl.gov/docs/global/c_nga_ops.html#ga_init_fence)_()
- The *fence* functions are:
**Fortran****	**subroutine _[ga_fence](http://www.emsl.pnl.gov/docs/global/ga_ops.html#ga_fence)_()
**C****	****	****	**void _[GA_Fence](http://www.emsl.pnl.gov/docs/global/c_nga_ops.html#ga_fence)_()

- Global Arrays Tutorial

## Synchronization Control in Collective Operations 

- To eliminate redundant synchronization points:
**Fortran****	**subroutine _[ga_mask_sync](http://www.emsl.pnl.gov/docs/global/ga_ops.html#ga_mask_sync)_(prior_sync_mask,			post_sync_mask)
**C****	****	****	**void _[GA_Mask_sync](http://www.emsl.pnl.gov/docs/global/c_nga_ops.html#ga_mask_sync)_(int prior_sync_mask,			int post_sync_mask)
- logical		first	- mask (0/1) for prior internal synchronization		[input]
- logical		last	- mask (0/1) for post internal synchronization		[input]

- Global Arrays Tutorial

## Linear Algebra

- Whole Arrays:
- To add to arrays:
**Fortran****	**subroutine _[ga_add](http://www.emsl.pnl.gov/docs/global/ga_ops.html#ga_add)_(alpha, g_a, beta, g_b, g_c)
**C****	****	**void_[GA_Add](http://www.emsl.pnl.gov/docs/global/c_nga_ops.html#ga_add)_(void *alpha, int g_a, void *beta, int g_b, int g_c)
- To multiply arrays:
**Fortran****	**subroutine _[ga_dgemm](http://www.emsl.pnl.gov/docs/global/ga_ops.html#ga_dgemm)_(transa, transb, m, n, k, alpha, g_a, g_b, beta, g_c)
**C****	****	**void_[GA_Dgemm](http://www.emsl.pnl.gov/docs/global/c_nga_ops.html#ga_dgemm)_(char ta, char tb, int m, int n, int k, double alpha, int g_a,
- 				int g_b, double beta, int g_c )
- double precision/complex/integer	alpha, beta				[input]
- integer 				g_a, g_b, g_c	- array handles	[input]
- double/complex/int			*alpha		- scale factor	[input]
- double/complex/int			*beta		- scale factor	[input]
- character*1				transa, transb			[input]
- integer				m, n, k				[input]
- double precision			alpha, beta				[input] (DGEMM)
- double complex			alpha, beta				[input] (ZGEMM)
- integer				g_a, g_b				[input]
- integer				g_c					[output]

- Global Arrays Tutorial

## Linear Algebra (cont.)

- Whole Arrays (cont.):
- To compute the element-wise dot product of two arrays:
- Three separate functions for data types
- Integer
**Fortran****	**function ga_idot(g_a, g_b)
**C****	****	**_**[GA_Idot(](http://www.emsl.pnl.gov/docs/global/c_nga_ops.html#ga_dot)**_int g_a, int g_b)
- Double precision
**Fortran****	**function_[ga_ddot](http://www.emsl.pnl.gov/docs/global/ga_ops.html#ga_ddot)_(g_a, g_b)
**C****	****	**_**[GA_Ddot](http://www.emsl.pnl.gov/docs/global/c_nga_ops.html#ga_dot)**_(int g_a, int g_b)
- Double complex
**Fortran****	**function _[ga_zdot](http://www.emsl.pnl.gov/docs/global/ga_ops.html#ga_zdot)_(g_a, g_b)
**C****	****	**_**[GA_Zdot](http://www.emsl.pnl.gov/docs/global/c_nga_ops.html#ga_dot)**_(int g_a, int g_b)
- integer		g_a, g_b							[input]
- integer 		GA_Idot(int g_a, int g_b)
- long 			GA_Ldot(int g_a, int g_b)
- float 			GA_Fdot(int g_a, int g_b)
- double 		GA_Ddot(int g_a, int g_b)
- DoubleComplex 	GA_Zdot(int g_a, int g_b)

- Global Arrays Tutorial

## Linear Algebra (cont.)

- Whole Arrays (cont.):
- To symmetrize a matrix:
**Fortran****	**subroutine _[ga_symmetrize](http://www.emsl.pnl.gov/docs/global/ga_ops.html#ga_symmetrize)_(g_a)
**C****	****	****	**void _[GA_Symmetrize](http://www.emsl.pnl.gov/docs/global/c_nga_ops.html#ga_symmetrize)_(int g_a)
- To transpose a matrix:
**Fortran****	**subroutine _[ga_transpose](http://www.emsl.pnl.gov/docs/global/ga_ops.html#ga_transpose)_(g_a, g_b)
**C****	****	****	**void _[GA_Transpose](http://www.emsl.pnl.gov/docs/global/c_nga_ops.html#ga_transpose)_(int g_a, int g_b)

- Global Arrays Tutorial

## Linear Algebra (cont.)

- Patches:
- To add element-wise two patches and save the results into another patch:
**Fortran****	****	**subroutine _[nga_add_patch](http://www.emsl.pnl.gov/docs/global/ga_ops.html#ga_add_patch)_(alpha, g_a,			alo, ahi, beta,  g_b, blo, bhi, g_c, clo, chi)
**C****	****	****	**void _[NGA_Add_patch](http://www.emsl.pnl.gov/docs/global/c_nga_ops.html#ga_add_patch)_(void *alpha, 			int g_a, int alo[], int ahi[], void *beta,			int g_b, int blo[], int bhi[], int g_c,			int clo[], int chi[])
- integer		g_a, g_b, g_c			[input]
- dbl prec/comp/int	alpha, beta		scale factors	[input]
- integer  		ailo, aihi, ajlo, ajhi	g_a patch coord	[input]
- integer		bilo, bihi, bjlo, bjhi	g_b patch coord	[input]
- integer		cilo, cihi, cjlo, cjhi	g_c patch coord	[input]

- Global Arrays Tutorial

## Linear Algebra (cont.)

- Patches (cont.):
- To perform matrix multiplication:
**Fortran****	****	**subroutine _[ga_matmul_patch](http://www.emsl.pnl.gov/docs/global/ga_ops.html#ga_matmul_patch)_(transa, transb, alpha, beta, g_a, ailo,
- 					aihi, ajlo, ajhi, g_b, bilo, bihi, bjlo, bjhi, g_c, cilo, cihi, cjlo, cjhi)
**C****	****	****	**void _[GA_Matmul_patch](http://www.emsl.pnl.gov/docs/global/c_nga_ops.html#ga_matmul_patch)_(char *transa, char* transb, void* alpha,
- 					void *beta, int g_a, int ailo, int aihi, int ajlo, int ajhi, int g_b, int bilo,
- 					int bihi, int bjlo, int bjhi, int g_c, int cilo, int cihi, int cjlo, int cjhi)
- integer		g_a, ailo, aihi, ajlo, ajhi	patch of g_a	[input]
- integer		g_b, bilo, bihi, bjlo, bjhi	patch of g_b	[input]
- integer		g_c, cilo, cihi, cjlo, cjhi	patch of g_c	[input]
- dbl prec/comp	alpha, beta			scale factors	[input]
- character*1		transa, transb		transpose flags	[input]

- Global Arrays Tutorial

## Linear Algebra (cont.)

- Patches (cont.):
- To compute the element-wise dot product of two arrays:
- Three separate functions for data types
- Integer
**Fortran****	**nga_idot_patch(g_a, ta, alo, ahi, g_b, tb, blo, bhi)
**C****	****	**_**[NGA_Idot_patch](http://www.emsl.pnl.gov/docs/global/c_nga_ops.html#ga_dot_patch)**_(int g_a, char* ta, int alo[], 		int ahi[], int g_b, char* tb, int blo[], int bhi[])
- Double precision
**Fortran****	**function_[nga_ddot_patch](http://www.emsl.pnl.gov/docs/global/ga_ops.html#ga_ddot_patch)_(g_a, ta, alo, ahi, g_b, 	tb, blo, bhi)
**C****	****	**_**[NGA_Ddot_patch](http://www.emsl.pnl.gov/docs/global/c_nga_ops.html#ga_dot_patch)**_(int g_a, char* ta, int alo[], 		int ahi[], int g_b, char* tb, int blo[], int bhi[])
- Double complex
**Fortran****	**function_[nga_zdot_patch](http://www.emsl.pnl.gov/docs/global/ga_ops.html#ga_zdot_patch)_(g_a, ta, alo, ahi, g_b, 	tb, blo, bhi)
**C****	****	**_**[NGA_Zdot_patch](http://www.emsl.pnl.gov/docs/global/c_nga_ops.html#ga_dot_patch)**_(int g_a, char* ta, int alo[], 		int ahi[], int g_b, char* tb, int blo[], int bhi[])
- integer		g_a, g_b							[input]
- integer 		GA_Idot(int g_a, int g_b)
- long 			GA_Ldot(int g_a, int g_b)
- float 			GA_Fdot(int g_a, int g_b)
- double 		GA_Ddot(int g_a, int g_b)
- DoubleComplex 	GA_Zdot(int g_a, int g_b)

- Global Arrays Tutorial

## Interfaces to Third Party Software Packages

- Scalapack
- Solve a system of linear equations
- Compute the inverse of a double precision matrix
- PeIGS
- Solve the generalized Eigen-value problem
- Solve the standard (non-generalized) Eigenvalue problem
- Interoperability with Others
- PETSc
- CUMULVS

- Global Arrays Tutorial

## Locality Information

- To determine the process ID that owns the element defined by the array subscripts:
**n-DFortran****	****	**logical function _[nga_locate](http://www.emsl.pnl.gov/docs/global/ga_ops.html#ga_locate)_(g_a, 					subscript, owner)
**C**				int _[NGA_Locate](http://www.emsl.pnl.gov/docs/global/c_nga_ops.html#ga_locate)_(int g_a,				int subscript[])
- integer	g_a			array handle				[input]
- Integer	subscript(ndim)	element subscript			[input]
- integer 	owner			process id				[output]

- 0

- 4

- 8

- 1

- 5

- 9

- 2

- 6

- 10

- 3

- 7

- 11

**owner=5**

- Global Arrays Tutorial

## Locality Information (cont.)

- To return a list of process IDs that own the patch:
**Fortran****	****	**logical function 							_[nga_locate_region](http://www.emsl.pnl.gov/docs/global/ga_ops.html#ga_locate_region)_(g_a, lo,
- 						hi, map,proclist, np)
**C**				int _[NGA_Locate_region](http://www.emsl.pnl.gov/docs/global/c_nga_ops.html#ga_locate_region)_(int g_a,				int lo[], int hi[],int *map[],				int procs[])
- integer 	np			- number of processors that own		[output]
- 						  a portion of block
- integer	g_a			- global array handle			[input]
- integer 	ndim			- number of dimensions of the 					  global array
- integer	lo(ndim)		- array of starting indices for array					  section					[input]
- integer	hi(ndim)		- array of ending indices for array					  section					[input]
- integer	map(2*ndim,*)	- array with mapping information		[output]
- integer	procs(np)		- list of processes that own a part 					  of array section				[output]

- 0

- 4

- 8

- 1

- 5

- 9

- 2

- 6

- 10

- 3

- 7

- 11

- procs = {0,1,2,4,5,6}
- map = {lo01,lo02,hi01,hi02,
-        lo11,lo12,hi11,hi12,
-        lo21,lo22’hi21,hi22,
-        lo41,lo42,hi41,hi42,
-        lo51,lo52,hi51,hi52’
-        lo61’lo62,hi61,hi62}

- Global Arrays Tutorial

## New Interface for Creating Arrays

- Developed to handle the proliferating number of properties that can be assigned to Global Arrays
**Fortran****	**integer function ga_create_handle()
- 					subroutine ga_set_data(g_a, dim, dims, type)
- 					subroutine ga_set_array_name(g_a, name)
- 					subroutine ga_set_chunk(g_a, chunk)
- 					subroutine ga_set_irreg_distr(g_a, map, nblock)
- 					subroutine ga_set_ghosts(g_a, width)
- 					logical function ga_allocate(g_a)
**C****	****	****	**int GA_Create_handle()
- 					void GA_Set_data(int g_a, int dim, int *dims, int type)
- 					void GA_Set_array_name(int g_a, char* name)
- 					void GA_Set_chunk(int g_a, int *chunk)
- 					void GA_Set_irreg_distr(int g_a, int *map, int *nblock)
- 					void GA_Set_ghosts(int g_a, int *width)
- 					int GA_Allocate(int g_a)

- Global Arrays Tutorial

## New Interface for Creating Arrays (Cont.) 

-       **integer ndim,dims(2),chunk(2)**
**      ****integer g_a, g_b**
**      ****logical status**
**c**
**      ****ndim = 2**
**      ****dims(1) = 5000**
**      ****dims(2) = 5000**
**      ****chunk(1) = 100**
**      ****chunk(2) = 100**
**c**
**c Create global array A using old interface**
**c**
**      ****status = ****nga_create(MT_F_DBL, ndim, dims, chunk, ‘array_A’, g_a)**
**c**
**c Create global array B using new interface**
**C**
**      ****g_b = ****ga_create_handle()**
**      ****call ****ga_set_data(g_b, ndim, dims, MT_F_DBL)**
**      ****call ****ga_set_chunk(g_b, chunk)**
**      ****call ****ga_set_name(g_b, ‘array_B’)**
**      ****call ****ga_allocate(g_b)**

- Global Arrays Tutorial

## Example Code

- 1-D Transpose (Fortran)
- 1-D Transpose (C)
- Matrix Multiply (Fortran)
- Matrix Multiply (C)

- Global Arrays Tutorial

## Example: 1-D Transpose

- Global Arrays Tutorial

## Transpose Example – C

**int** ndim, dims[1], chunk[1], ld[1], lo[1], hi[1];
**int** lo1[1], hi1[1], lo2[1], hi2[1];
**int** g_a, g_b, a[MAXPROC*TOTALELEMS],b[MAXPROC*TOTALELEMS];
**int** nelem, i;
*/* Find local processor ID and number of processors */*
**int** me=GA_Nodeid(), nprocs=GA_Nnodes();
*/* Configure array dimensions. Force an unequal data distribution */*
- ndim     = 1; */* 1-d transpose */*
- dims[0]  = nprocs*TOTALELEMS + nprocs/2;
- ld[0]    = dims[0];
- chunk[0] = TOTALELEMS; */* minimum data on each process */*
*/* create a global array g_a and duplicate it to get g_b */*
- g_a = NGA_Create(C_INT, 1, dims, **"array A"**, chunk);
**if** (!g_a) GA_Error(**"create failed: A"**, 0);
**if** (me==0) printf(**"  Created Array A\n"**);
- g_b = GA_Duplicate(g_a, **"array B"**);
**if** (! g_b) GA_Error(**"duplicate failed"**, 0);
**if** (me==0) printf(**"  Created Array B\n"**);

- Global Arrays Tutorial

## Transpose Example – C (cont.)

*/* initialize data in g_a */*
**if** (me==0) {
-    printf(**"  Initializing matrix A\n"**);
-    **for**(i=0; i<dims[0]; i++) a[i] = i;
-    lo[0]  = 0;
-    hi[0] = dims[0]-1;
-    NGA_Put(g_a, lo, hi, a, ld);
- }
*/* Synchronize all processors to guarantee that everyone has data before proceeding **   to the next step. */*
- GA_Sync();
*/* Start initial phase of inversion by inverting the data held locally on each processor.**   Start by finding out which data each processor owns. */*
- NGA_Distribution(g_a, me, lo1, hi1);
*/* Get locally held data and copy it into local buffer a  */*
- NGA_Get(g_a, lo1, hi1, a, ld);
*/* Invert data locally */*
- nelem = hi1[0] - lo1[0] + 1;
**for** (i=0; i<nelem; i++) b[i] = a[nelem-1-i];

- Global Arrays Tutorial

## Transpose Example – C (cont.)

*/* Invert data globally by copying locally inverted blocks into** * their inverted positions in the GA */*
- lo2[0] = dims[0] - hi1[0] -1;
- hi2[0] = dims[0] - lo1[0] -1;
- NGA_Put(g_b,lo2,hi2,b,ld);
*/* Synchronize all processors to make sure inversion is complete */*
- GA_Sync();
*/* Check to see if inversion is correct */*
**if**(me == 0) verify(g_a, g_b);
*/* Deallocate arrays */*
- GA_Destroy(g_a);
- GA_Destroy(g_b);

- Global Arrays Tutorial

## Transpose Example - Fortran

- 	integer dims(3), chunk(3), nprocs, me, i, lo(3), hi(3), lo1(3)
- 	integer hi1(3), lo2(3), hi2(3), ld(3), nelem
- 	integer g_a, g_b, a(MAXPROC*TOTALELEMS), b(MAXPROC*TOTALELEMS)
- 	integer heap, stack, ichk, ierr
- 	logical status
- 	heap = 300000
- 	stack = 300000
- c
- c   Initialize communication library
- c
**#ifdef** USE_MPI
-       call mpi_init(ierr)
**#else**
-       call pbeginf
**#endif**
- c
- c   Initialize GA library
- c
-       call ga_initialize()

- Global Arrays Tutorial

## Transpose Example – Fortran (cont.)

- c
- c   Find local processor ID and number of processors
- c
-       me = ga_nodeid()
-       nprocs = ga_nnodes()
-       **if** (me.eq.0) write(6,101) nprocs
-   101 format('Using ',i4,' processors')
- c
- c   Allocate memory **for** GA library
- c
-       status = ma_init(MT_F_DBL, stack/nprocs, heap/nprocs)
- c
- c   Configure array dimensions. Force an unequal data distribution.
- c
-       dims(1) = nprocs*TOTALELEMS + nprocs/2
-       ld(1) = MAXPROC*TOTALELEMS
-       chunk(1) = TOTALELEMS     ! Minimum data on each processor
- c
- c   Create global array g_a and then duplicate it to get g_b
- c
-       status = nga_create(MT_F_INT, NDIM, dims, **"Array A"**, chunk, g_a)
-       status = ga_duplicate(g_a, g_b, **"Array B"**)

- Global Arrays Tutorial

## Transpose Example – Fortran (cont.)

- c
- c   Initialize data in g_a
- c
-       **do** i = 1, dims(1)
-         a(i) = i
-       end **do**
-       lo1(1) = 1
-       hi1(1) = dims(1)
- c
- c   Copy data from local buffer a to global array g_a. Only **do this for**
- c   processor 0.
- c
-       **if** (me.eq.0) call nga_put(g_a, lo1, hi1, a, ld)
- c
- c   Synchronize all processors to guarantee that everyone has data
- c   before proceeding to the next step.
- c
-       call ga_sync

- Global Arrays Tutorial

## Transpose Example – Fortran (cont.)

- c
- c   Start initial phase of inversion by inverting the data held locally on
- c   each processor. Start by finding out which data each processor owns.
- c
-       call nga_distribution(g_a, me, lo, hi)
- c
- c   Get locally held data and copy it into local buffer a
- c
-       call nga_get(g_a, lo, hi, a, ld)
- c
- c   Invert local data
- c
-       nelem = hi(1) - lo(1) + 1
-       **do** i = 1, nelem
-         b(i) = a(nelem - i + 1)
-       end **do**
- c
- c   Do global inversion by copying locally inverted data blocks into
- c   their inverted positions in the GA
- c
-       lo2(1) = dims(1) - hi(1) + 1
-       hi2(1) = dims(1) - lo(1) + 1
-       call nga_put(g_b, lo2, hi2, b, ld)

- Global Arrays Tutorial

## Transpose Example – Fortran (cont.)

- c
- c   Synchronize all processors to make sure inversion is complete
- c
-       call ga_sync()
- c
- c   Check to see **if** inversion is correct. Start by copying g_a into local
- c   buffer a, and g_b into local buffer b.
- c
-       call nga_get(g_a, lo1, hi1, a, ld)
-       call nga_get(g_b, lo1, hi1, b, ld)
-       ichk = 0
-       **do** i = 1, dims(1)
-         **if** (a(i).ne.b(dims(1)-i+1) .and. me.eq.0) then
-           write(6,111) i,a(i),b(dims(1)-i+1)
-   111     format('Mismatch at ',3i8)
-           ichk = ichk + 1
-         endif
-       end **do**
-       **if** (ichk.eq.0.and.me.eq.0) write(6,*) 'Transpose OK'

- Global Arrays Tutorial

## Transpose Example – Fortran (cont.)

- c
- c   Deallocate memory **for** arrays and clean up GA library
- c
-       **if** (me.eq.0) write(6,*) 'Terminating...'
-       status = ga_destroy(g_a)
-       status = ga_destroy(g_b)
-       call ga_terminate
**#ifdef** USE_MPI
-       call mpi_finalize
**#else**
-       call pend
**#endif**
-       stop
-       end

- Global Arrays Tutorial

## Matrix Multiply Example – C

**int** dims[NDIM], chunk[NDIM], ld[NDIM];
**int** lo[NDIM], hi[NDIM], lo1[NDIM], hi1[NDIM];
**int** lo2[NDIM], hi2[NDIM], lo3[NDIM], hi3[NDIM];
**int** g_a, g_b, g_c, i, j, k, l;
*/* Find local processor ID and the number of processors */*
**int** me=GA_Nodeid(), nprocs=GA_Nnodes();
*/* Configure array dimensions. Force an unequal data distribution */*
**for**(i=0; i<NDIM; i++) {
-    dims[i]  = TOTALELEMS;
-    ld[i]= dims[i];
-    chunk[i] = TOTALELEMS/nprocs-1; */*minimum block size on each process*/*
- }

- Global Arrays Tutorial

## Matrix Multiply Example – C (cont.)

*/* create a global array g_a and duplicate it to get g_b and g_c*/*
- g_a = NGA_Create(C_DBL, NDIM, dims, "array A", chunk);
**if** (!g_a) GA_Error("create failed: A", NDIM);
**if** (me==0) printf("  Created Array A\n");
- g_b = GA_Duplicate(g_a, "array B");
- g_c = GA_Duplicate(g_a, "array C");
**if** (!g_b || !g_c) GA_Error("duplicate failed",NDIM);
**if** (me==0) printf("  Created Arrays B and C\n");
*/* initialize data in matrices a and b */*
**if**(me==0)printf("  Initializing matrix A and B\n");
- k = 0; l = 7;
**for**(i=0; i<dims[0]; i++) {
-    **for**(j=0; j<dims[1]; j++) {
-   a[i][j] = (**double**)(++k%29);
-   b[i][j] = (**double**)(++l%37);
-    }
- }

- Global Arrays Tutorial

## Matrix Multiply Example – C (cont.)

*/*  Copy data to global arrays g_a and g_b */*
- lo1[0] = 0;
- lo1[1] = 0;
- hi1[0] = dims[0]-1;
- hi1[1] = dims[1]-1;
**if** (me==0) {
-    NGA_Put(g_a, lo1, hi1, a, ld);
-    NGA_Put(g_b, lo1, hi1, b, ld);
- }
*/*  Synchronize all processors to make sure everyone has data */*
- GA_Sync();
*/* Determine which block of data is locally owned. Note that*
*   **the same block is locally owned for all GAs. */*
- NGA_Distribution(g_c, me, lo, hi);

- Global Arrays Tutorial

## Matrix Multiply Example – C (cont.)

*/* Get the blocks from g_a and g_b needed to compute this block in*
*   **g_c and copy them into the local buffers a and b. */*
- lo2[0] = lo[0];
- lo2[1] = 0;
- hi2[0] = hi[0];
- hi2[1] = dims[0]-1;
- NGA_Get(g_a, lo2, hi2, a, ld);
- lo3[0] = 0;
- lo3[1] = lo[1];
- hi3[0] = dims[1]-1;
- hi3[1] = hi[1];
- NGA_Get(g_b, lo3, hi3, b, ld);
*/* Do local matrix multiplication and store the result in local*
*   **buffer c. Start by evaluating the transpose of b. */*
**for**(i=0; i < hi3[0]-lo3[0]+1; i++)
-    **for**(j=0; j < hi3[1]-lo3[1]+1; j++)
-   btrns[j][i] = b[i][j];

- Global Arrays Tutorial

## Matrix Multiply Example – C (cont.)

*/* Multiply a and b to get c */*
**for**(i=0; i < hi[0] - lo[0] + 1; i++) {
-    **for**(j=0; j < hi[1] - lo[1] + 1; j++) {
-   c[i][j] = 0.0;
-   **for**(k=0; k<dims[0]; k++)
-  c[i][j] = c[i][j] + a[i][k]*btrns[j][k];
-    }
- }
*/* Copy c back to g_c */*
- NGA_Put(g_c, lo, hi, c, ld);
- verify(g_a, g_b, g_c, lo1, hi1, ld);
*/* Deallocate arrays */*
- GA_Destroy(g_a);
- GA_Destroy(g_b);
- GA_Destroy(g_c);

- Global Arrays Tutorial