XcalableMP⟨ex-scalable-em-p⟩
Language Specification

Version 1.4
XcalableMP Specification Working Group

November, 2018

c
Copyright ⃝2008-2018
XcalableMP Specification Working Group. Permission to copy without fee all or part of this material is granted, provided the XcalableMP Specification Working
Group copyright notice and the title of this document are displayed. Notice is given that copying
requires the explicit permission of the XcalableMP Specification Working Group.

2

History
Version 1.4: November, 2018
• 4.1.2 Combined Directive• F DRAFT: Tasklet of upcoming XcalableMP 2.0
Version 1.3: August, 2017
• 2.8.2 Node Terminology• 2.8.6 Local-view Terminology• 3.1 Array Section Notation• 3.2 Array assignment statements in C
• 3.5 Dynamic Allocation of Global Data in C
• 4.2 nodes Directive• 4.3.1 template Directive• 4.3.4 align Directive• 4.4.3 loop Construct• 4.5.1 reflect Construct• 4.5.6 wait async Construct• 4.5.8 reduce shadow Construct• 5.7 Coarrays in XcalableMP C• 5.8 Directives for the Local-view Programming• 7.2.2 xmpc all node num• 7.2.5 xmpc node num• 7.2.6 xmpc this image• 7.2.8 xmp num images• 7.3 Execution Control Functions• 7.5.1 xmp malloc• 7.6.4 xmp nodes attr
• B.2.1 xmp array gtol
• B.2.2 xmp array lsize• B.2.4 xmp array lda• 7.9.1 xmp scatter• 7.9.3 xmp pack• 7.9.4 xmp unpack• C Memory-layout Model
3

Version 1.2.1: November, 2014

Corrections and clarifications to Version 1.2.

• 4.3.4 align Directive• 4.4.3 loop Directive• 5.7.1 [C] Declarations of Coarrays• 5.8.1 [F] local alias Directive• 3.1 Array Section Notation• 3.2 Array Assignment Statement
Version 1.2: November 20, 2013
Version 1.1: November 13, 2012
Version 1.0: November 14, 2011

4

Contents
1 Introduction
1.1 Features of XcalableMP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.2 Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.3 Organization of this Document . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.4 Changes to Version 1.4 from Version 1.3 . . . . . . . . . . . . . . . . . . . . . . .
1.5 Changes to Version 1.3 from Version 1.2.1 . . . . . . . . . . . . . . . . . . . . . .
1.6 Changes to Version 1.1 from Version 1.2 . . . . . . . . . . . . . . . . . . . . . . .

1
1
2
2
2
2
3

2 Overview of the XcalableMP Model and Language
2.1 Hardware Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.2 Execution Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.3 Data Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.4 Global-view Programming Model . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.5 Local-view Programming Model . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.6 Interactions between Global View and Local View . . . . . . . . . . . . . . . . .
2.7 Base Languages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.8 Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.8.1 Language Terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.8.2 Node Terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.8.3 Data Terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.8.4 Work Terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.8.5 Communication and Synchronization Terminology . . . . . . . . . . . . .
2.8.6 Local-view Terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . .

5
5
5
6
6
7
8
8
9
9
11
12
12
12
13

3 Base Language Extensions in XcalableMP C
3.1 Array Section Notation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.2 Array Assignment Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.3 Built-in Functions for Array Section . . . . . . . . . . . . . . . . . . . . . . . . .
3.4 Pointer to Global Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.4.1 Name of Global Array . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.4.2 Address-of Operator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.5 Dynamic Allocation of Global Data . . . . . . . . . . . . . . . . . . . . . . . . . .
3.6 Descriptor-of Operator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

15
15
16
17
17
17
17
17
18

4 Directives
4.1 Directive Format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.1.1 General Rule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.1.2 Combined Directive . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.2 nodes Directive . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.2.1 Node Reference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

19
19
19
21
22
23

i

4.3

Template and Data Mapping Directives . . . . . . . . . . . . . . . . . . . . . . .
4.3.1 template Directive . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.3.2 Template Reference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.3.3 distribute Directive . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.3.4 align Directive . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.3.5 shadow Directive . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.3.6 template fix Construct . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Work Mapping Construct . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.4.1 task Construct . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.4.2 tasks Construct . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.4.3 loop Construct . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.4.4 array Construct . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Global-view Communication and Synchronization Constructs . . . . . . . . . . .
4.5.1 reflect Construct . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.5.2 gmove Construct . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.5.3 barrier Construct . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.5.4 reduction Construct . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.5.5 bcast Construct . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.5.6 wait async Construct . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.5.7 async Clause . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.5.8 reduce shadow Construct . . . . . . . . . . . . . . . . . . . . . . . . . . .

24
24
25
26
29
31
33
35
35
36
38
47
48
48
50
52
52
55
56
57
57

5 Support for the Local-view Programming
5.1 Rules Determining Image Index . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.1.1 Primary Image Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.1.2 Image Index Determined by a task Directive . . . . . . . . . . . . . . . .
5.1.3 Current Image Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.1.4 Image Index Determined by a Non-primary Node Array . . . . . . . . . .
5.1.5 Image Index Determined by an Equivalenced Node Array . . . . . . . . .
5.1.6 On-node Image Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.2 Basic Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.2.1 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.3 coarray Directive . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.3.1 Purpose and Form of the coarray Directive . . . . . . . . . . . . . . . . .
5.3.2 An Example of the coarray Directive . . . . . . . . . . . . . . . . . . . .
5.4 image Directive . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.4.1 Purpose and Form of the image Directive . . . . . . . . . . . . . . . . . .
5.4.2 An Example of the image Directive . . . . . . . . . . . . . . . . . . . . . .
5.5 Image Index Translation Intrinsic Procedures . . . . . . . . . . . . . . . . . . . .
5.5.1 Translation to the Primary Image Index . . . . . . . . . . . . . . . . . . .
5.5.2 Translation to the Current Image Index . . . . . . . . . . . . . . . . . . .
5.6 Examples of Communication between Tasks . . . . . . . . . . . . . . . . . . . . .
5.7 [C] Coarrays in XcalableMP C . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.7.1 [C] Declaration of Coarrays . . . . . . . . . . . . . . . . . . . . . . . . . .
5.7.2 [C] Reference of Coarrays . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.7.3 [C] Synchronization of Coarrays . . . . . . . . . . . . . . . . . . . . . . .
5.8 Directives for the Local-view Programming . . . . . . . . . . . . . . . . . . . . .
5.8.1 [F] local alias Directive . . . . . . . . . . . . . . . . . . . . . . . . . .
5.8.2 post Construct . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5.8.3 wait Construct . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

61
61
61
62
62
62
62
63
63
63
64
64
65
66
66
66
67
67
68
68
71
71
72
72
73
73
76
77

4.4

4.5

ii

5.8.4[C] lock/unlock Construct . . . . . . . . . . . . . . . . . . . . . . . . . .

78

6 Procedure Interfaces
6.1 General Rule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.2 Argument Passing Mechanism in XcalableMP Fortran . . . . . . . . . . . . . . .
6.2.1 Sequence Association of Global Data . . . . . . . . . . . . . . . . . . . . .
6.2.2 Descriptor Association of Global Data . . . . . . . . . . . . . . . . . . . .
6.3 Argument-Passing Mechanism in XcalableMP C . . . . . . . . . . . . . . . . . .

81
81
81
82
85
88

7 Intrinsic and Library Procedures93
7.1 Intrinsic Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
7.1.1 xmp desc of . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
7.2 System Inquiry Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
7.2.1 xmp all node num . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
7.2.2 [C] xmpc all node num . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
7.2.3 xmp all num nodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
7.2.4 xmp node num . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
7.2.5 [C] xmpc node num . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
7.2.6 [C] xmpc this image . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
7.2.7 xmp num nodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
7.2.8 xmp num images . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
7.2.9 xmp wtime . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
7.2.10 xmp wtick . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
7.3 [C] Execution Control Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
7.3.1 xmp exit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
7.4 Synchronization Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
7.4.1 xmp test async . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
7.5 Memory Allocation Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
7.5.1 [C] xmp malloc . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
7.6 Mapping Inquiry Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
7.6.1 xmp nodes ndims . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
7.6.2 xmp nodes index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
7.6.3 xmp nodes size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
7.6.4 xmp nodes attr . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
7.6.5 xmp nodes equiv . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
7.6.6 xmp template fixed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
7.6.7 xmp template ndims . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
7.6.8 xmp template lbound . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
7.6.9 xmp template ubound . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
7.6.10 xmp dist format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
7.6.11 xmp dist blocksize . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
7.6.12 xmp dist gblockmap . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
7.6.13 xmp dist nodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
7.6.14 xmp dist axis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
7.6.15 xmp align axis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
7.6.16 xmp align offset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
7.6.17 xmp align replicated . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
7.6.18 xmp align template . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
7.6.19 xmp array ndims . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
7.6.20 xmp array lshadow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
iii

7.7
7.8
7.9

7.6.21 xmp array ushadow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
7.6.22 xmp array lbound . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
7.6.23 xmp array ubound . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108[F] Array Intrinsic Functions of the Base Language . . . . . . . . . . . . . . . . 109[C] Built-in Elemental Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
Intrinsic/Built-in Transformational Procedures . . . . . . . . . . . . . . . . . . . 109
7.9.1 xmp scatter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
7.9.2 xmp gather . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
7.9.3 xmp pack . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
7.9.4 xmp unpack . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
7.9.5 xmp transpose . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
7.9.6 xmp matmul . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
7.9.7 xmp sort up . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
7.9.8 xmp sort down . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113

8 OpenMP in XcalableMP Programs

115

Bibliography

117

A Programming Interface for MPI
119
A.1 Call MPI functions from an XcalableMP program . . . . . . . . . . . . . . . . . . 119
A.1.1 xmp get mpi comm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
A.1.2 xmp init mpi . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
A.1.3 xmp finalize mpi . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
A.2 Call XcalableMP functions from an MPI program . . . . . . . . . . . . . . . . . . 121
A.2.1 xmp init . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
A.2.2 xmp finalize . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
B Interface to Numerical Libraries
125
B.1 Interface Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
B.2 Extended Mapping Inquiry Functions . . . . . . . . . . . . . . . . . . . . . . . . 125
B.2.1 xmp array gtol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
B.2.2 xmp array lsize . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
B.2.3 xmp array laddr . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
B.2.4 xmp array lda . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
B.3 Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
C Memory-layout Model131

D XcalableMP I/O
133
D.1 Categorization of I/O . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
D.1.1 Local I/O . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
D.1.2 Master I/O[F] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
D.1.3 Global I/O . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
D.2 File Connection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
D.2.1 File Connection in Local I/O . . . . . . . . . . . . . . . . . . . . . . . . . 135
D.2.2 [F] File Connection in Master I/O . . . . . . . . . . . . . . . . . . . . . . 135
D.2.3 File Connection in Global I/O . . . . . . . . . . . . . . . . . . . . . . . . 135
D.3 Master I/O . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
D.3.1 master io Construct . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
D.4 [F] Global I/O . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
iv

D.4.1 Global I/O File Operation . . . . . . . . . . . . . . . . . . . . . . . . . . . 138
D.4.2 Collective Global I/O Statement . . . . . . . . . . . . . . . . . . . . . . . 140
D.4.3 Atomic Global I/O Statement . . . . . . . . . . . . . . . . . . . . . . . . . 140
D.4.4 Direct Global I/O Statement . . . . . . . . . . . . . . . . . . . . . . . . . 141
D.5 [C] Global I/O Library . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
D.5.1 Global I/O File Operation . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
D.5.2 Collective Global I/O Functions . . . . . . . . . . . . . . . . . . . . . . . 146
D.5.3 Atomic Global I/O Functions . . . . . . . . . . . . . . . . . . . . . . . . . 148
D.5.4 Direct Global I/O Functions . . . . . . . . . . . . . . . . . . . . . . . . . 149
E Memory Consistency Model
151
E.1 Execution Traces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151
E.1.1 Common Constraints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152
E.1.2 Constraints for Synchronous Communications . . . . . . . . . . . . . . . . 152
E.1.3 Constraints for Asynchronous Communications . . . . . . . . . . . . . . . 152
F DRAFT: Tasklet of upcoming XcalableMP 2.0
157
F.1 XcalableMP Extended Execution Model . . . . . . . . . . . . . . . . . . . . . . . 157
F.2 Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
F.2.1 Node Terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
F.2.2 Thread Terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
F.2.3 Tasklet Terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
F.3 Directives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158
F.3.1 Tasklet Constructs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158
F.3.2 Tasklet Scheduling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166
F.3.3 Communication Tasklet Constructs . . . . . . . . . . . . . . . . . . . . . . 166
G Sample Programs

171v

List of Figures
2.1
2.2
2.3
2.4

Hardware model. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Parallelization using the global-view programming model. . . . . . . . . . . . . .
Local-view programming model. . . . . . . . . . . . . . . . . . . . . . . . . . . .
Global view and local view. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

5
7
8
9

4.1
4.2

Example showing shadow of a two-dimensional array. . . . . . . . . . . . . . . . .
Example of periodic shadow reflection. . . . . . . . . . . . . . . . . . . . . . . . .

33
49

Sequence association with a global dummy argument. . . . . . . . . . . . . . . .
Sequence association with a local dummy argument. . . . . . . . . . . . . . . . .
Sequence association of a section of a global data object as an actual argumentwith a local dummy argument. . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.4 Sequence association of an element of a global data object as an actual argumentwith a local dummy argument. . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.5 Sequence association with a global dummy argument that has a full shadow. . .
6.6 Descriptor association with a global dummy argument. . . . . . . . . . . . . . . .
6.7 Descriptor association with a local dummy argument. . . . . . . . . . . . . . . .
6.8 Passing to a global dummy argument. . . . . . . . . . . . . . . . . . . . . . . . .
6.9 Passing to a local dummy argument. . . . . . . . . . . . . . . . . . . . . . . . . .
6.10 Passing an element of a global data object as an actual argument to a local dummy
argument. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

83
84

6.1
6.2
6.3

85
86
86
88
89
90
91
91

B.1 Invocation of a library routine using an interface procedure. . . . . . . . . . . . . 125
C.1 Example of memory layout in the Omni XcalableMP compiler. . . . . . . . . . . 132
E.1 Constraints that are required by the XcalableMP memory consistency model. . . 152

vi

List of Tables
7.1

Built-in elemental functions in XcalableMP C . . . . . . . . . . . . . . . . . . . . 110

D.1 Global I/O. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
D.2 Operations for I/O. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137

vii

Acknowledgment
The XcalableMP specification is designed by the XcalableMP Specification Working Group,
which consists of the following members from academia, research laboratories, and industries.
• Tatsuya Abe . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Chiba Institute of Technology• Tokuro Anzaki . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hitachi• Taisuke Boku . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . University of Tsukuba• Toshio Endo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . TITECH• Yoshinari Fukui . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . JAMSTEC• Yasuharu Hayashi . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . NEC• Atsushi Hori . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . RIKEN• Kohichiro Hotta . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fujitsu• Hidetoshi Iwashita . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fujitsu• Susumu Komae . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . AXE• Atsushi Kubota . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hiroshima City University• Jinpil Lee . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . RIKEN• Toshiyuki Maeda . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Chiba Institute of Technology• Motohiko Matsuda . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . RIKEN• Yuichi Matsuo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . JAXA
• Kazuo Minami . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . RIKEN• Shoji Morita . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . AXE• Hitoshi Murai . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . RIKEN• Kengo Nakajima . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . University of Tokyo• Takashi Nakamura . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . JAXA
• Tomotake Nakamura . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fujitsu• Mamoru Nakano . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . CRAY
• Masahiro Nakao . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . RIKEN• Takeshi Nanri . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Kyushu University• Kiyoshi Negishi . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hitachi• Satoshi Ohshima . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Kyushu University• Yasuo Okabe . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Kyoto University• Hitoshi Sakagami . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . NIFS
• Tomoko Sakari . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fujitsu• Shoich Sakon . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . NEC• Mitsuhisa Sato . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . RIKEN• Taizo Shimizu . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . PC Cluster Consortium• Takenori Shimosaka . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hitachi• Yoshihisa Shizawa . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . RIST
viii

• Shozo Takeoka . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . AXE• Hitoshi Uehara . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . JAMSTEC• Eiji Yamanaka . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Fujitsu• Masahiro Yasugi . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Kyushu Institute of Technology• Mitsuo Yokokawa . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Kobe University
This work was supported by “Seamless and Highly-productive Parallel Programming Environment for High-performance Computing” project funded by Ministry of Education, Culture,
Sports, Science and Technology, Japan, and is supported by PC Cluster Consortium.

ix

x

1

2

Chapter 1

Introduction

6

This document defines the XcalableMP specification, which is a directive-based language extension of Fortran and C for scalable and performance-aware parallel programming. The specification includes a collection of compiler directives and intrinsic and library procedures, andprovides a model of parallel programming for distributed memory multiprocessor systems.

7

1.1

8

The features of XcalableMP are summarized as follows:

3
4
5

9
10
11
12
13

14
15
16
17
18

19
20

21
22
23

24
25
26

27
28

29
30

Features of XcalableMP• XcalableMP supports typical parallelization based on the data-parallel paradigm and work
mapping under the “global-view” programming model, and it enables the parallelizationof the original sequential code using minimal modification with simple directives such as
OpenMP [1]. Many ideas on “global-view” programming are inherited from High Performance Fortran (HPF) [2].
• The important design principle of XcalableMP is “performance-awareness.” All actions
related to communication and synchronization are taken by directives (and coarray features), which is diﬀerent from automatic parallelizing compilers. The user should be awareof the eﬀect of the XcalableMP directives in the execution model for distributed-memoryarchitecture.
• XcalableMP also includes features from Partitioned Global Address Space (PGAS) languages, such as coarray of the Fortran 2008 standard, for “local-view” programming.
• An extension of existing base languages with directives is useful to reduce code-rewritingand education costs. The XcalableMP language specification is defined as an extension to
the Fortran and C base languages.
• For flexibility and extensibility, the execution model enables us to combine XcalableMPwith explicit Message Passing Interface (MPI) [3] coding for more complicated and tuned
parallel codes and libraries.
• For multi-core and SMP clusters, OpenMP directives can be combined into XcalableMPfor thread programming inside each node as a hybrid programming model.
XcalableMP is being designed based on experiences gained during the development of HPF,
HPF/JA [4], Fujitsu XPF (VPP FORTRAN) [5, 6], and OpenMPD [7].
1

2

CHAPTER 1. INTRODUCTION

1.2

Scope

The XcalableMP specification covers only user-directed parallelization, where the user explicitly
specifies the behavior of the compiler and the runtime system in order to execute the program in parallel in a distributed-memory system. XcalableMP-compliant implementations are
not required to automatically distribute data, detect parallelism, parallelize loops, or generatecommunications and synchronizations.

1.3

Organization of this Document

The remainder of this document is structured as follows:

1

2
3
4
5
6

7

8

• Chapter 2: Overview of the XcalableMP Model and Language

9• Chapter 3: Base Language Extensions in XcalableMP C

10• Chapter 4: Directives

11• Chapter 5: Support for Local-view Programming

12• Chapter 6: Procedure Interface

13• Chapter 7: Intrinsic and Library Procedures

14• Chapter 8: OpenMP in XcalableMP Programs

15

In addition, the following appendices are included in this document as proposals.

16• Appendix A: Programming Interface for MPI

17• Appendix B: Interface to Numerical Libraries

18• Appendix C: Memory-layout Model

19• Appendix D: XcalableMP I/O

20• Appendix E: Memory Consistency Model

21• Appendix F: DRAFT: Tasklet of upcoming XcalableMP 2.0

22

1.4

Changes to Version 1.4 from Version 1.3

23• Combined directives in XcalableMP C are allowed.

24• Add an appendix about the tasklet featues of upcoming XcalableMP 2.0.

25

1.5

Changes to Version 1.3 from Version 1.2.1• In XcalableMP C, a square bracket is available in nodes-decl, nodes-ref, template-ref, andtemplate-decl.

26

27
28

• Add the orthogonal clause to the reflect directive in Section 4.5.1.

29• Add xmpc all node num() in Section 7.2.2.

30

1.6. CHANGES TO VERSION 1.1 FROM VERSION 1.2

1• Add xmpc node num() in Section 7.2.5.

2• Add xmpc this image() in Section 7.2.6.

3• Add xmp num images() in Section 7.2.8.

4• Modify xmp array gtol() in Section B.2.1.

5• Change xmp array lsize() not to include shadow object in Section B.2.2.

6• Create xmp array lda() from xmp array lead dim() in Section B.2.4.

7• In XcalableMP C, the dynamic allocataion of multi-dimensional global data is allowed.

8• A restriction on the align directive is added.

9• The expand and margin clauses of the loop construct are added.

10
11

3

• The meaning of a reduction-kind “-” in the reduction clause of the loop construct ischanged.

13• The treatment for async-id not associated with any asynchronous communication is specified.

14• The reduce shadow construct is added.

15• The description of the local alias directive is modified.

16• The xmp exit library function is added.

17• The specifications of xmp scatter, xmp pack, and xmp unpack are modified.

18• The memory consistency model of XcalableMP is discussed in the appendix.

12

19

1.6

Changes to Version 1.1 from Version 1.2

20• The position of align directives for dummy arguments in XcalableMP C is specified.

21• It is specified that aligned arrays cannot be initialized.

22• The interpretation of a reduction clause of the loop directive is corrected.

23• The syntax for declaring coarrays is changed.

24• An assumed-shape array can be the target of the local alias directive.

25• The syntax and the semantics of the array section notation in XcalableMP C is modified.

26• The syntax of the array assignment statement in XcalableMP C is extended.

1

2

3

4

5
6
7
8
9
10

Chapter 2

Overview of the XcalableMP Modeland Language
2.1

Hardware Model

The target of XcalableMP is distributed-memory multicomputers (Figure 2.1). Each computation node, which may contain several cores, has its own local memory (shared by the cores, ifany), and is connected with the others via an interconnection network. Each node can accessits local memory directly and remote memory (the memory of another node) indirectly (i.e.,through inter-node communication). However, it is assumed that accessing remote memory ismuch slower than accessing local memory.

Figure 2.1: Hardware model.

11

12
13
14
15

2.2

Execution ModelAn XcalableMP program execution is based on the Single Program Multiple Data (SPMD)
model, where each node starts execution from the same main routine, and continues to execute the same code independently (i.e., asynchronously), which is referred to as the replicatedexecution, until it encounters an XcalableMP construct.
56

CHAPTER 2. OVERVIEW OF THE XCALABLEMP MODEL AND LANGUAGE

A set of nodes that executes a procedure, statement, loop, a block, etc. is referred toas its executing node set, and is determined by the innermost task, loop, or array directive
surrounding it dynamically, or at runtime. The current executing node set is an executing node
set of the current context, which is managed by the XcalableMP runtime system on each node.
The current executing node set at the beginning of the program execution, or entire node set,
is a node set that contains all the available nodes, which can be specified in an implementationdefined way (e.g., through a command-line option).
When a node encounters at runtime either a loop, array, or task construct, and is containedby the node set specified by the on clause of the directive, it updates the current executing node
set with the specified one and executes the body of the construct, after which it resumes thelast executing node set and proceeds to execute the subsequent statements.
In particular, when a node in the current executing node set encounters a loop or an array
construct, it executes the loop or the array assignment in parallel with other nodes, so thateach iteration of the loop or element of the assignment is independently executed by the nodein which a specified data element resides.
When a node encounters a synchronization or a communication directive, synchronization orcommunication occurs between it and other nodes. That is, such global constructs are performed
collectively by the current executing nodes. Note that neither synchronization nor communication occurs unless these constructs are being specified.

10

2.3

20

Data Model

There are two classes of data in XcalableMP: global data and local data. Data declared in an
XcalableMP program are local by default.
Global data are distributed onto the executing node set by the align directive (see section
4.3.4). Each fragment of distributed global data is allocated in the local memory of a node in
the executing node set.
Local data comprises all data that are not global. They are replicated within the local
memory of each of the executing nodes.
A node can access directly only local data and sections of global data that reside in its local
memory. To access data in remote memory, explicit communication must be specified in ways
such as global communication constructs and coarray assignments.
In particular, in XcalableMP Fortran, for common blocks that include any global variables,
it is implementation-defined what storage sequences they occupy and how storage association isdefined between two of them.

2.4

Global-view Programming Model

The global-view programming model is useful when, starting from a sequential version of a program, the programmer parallelizes it in data-parallel style by adding directives with minimum
modification. In the global-view programming model, the programmer describes the distribution of data among nodes using the data distribution directives. The loop construct assigns
each iteration of a loop to the node at which the computed data is located. The global-viewcommunication directives are used to synchronize nodes, maintain the consistency of shadow
areas, and move sections of distributed data globally. Note that the programmer must specify
explicitly communications to make all data references in the program local, and this is doneusing appropriate directives.
In many cases, the XcalableMP program according to the global-view programming modelis based on a sequential program, and it can produce the same results, regardless of the number

1
2
3
4
5
6
7
8
9

11
12
13
14
15
16
17
18
19

21
22
23
24
25
26
27
28
29
30
31
32
33

34

35
36
37
38
39
40
41
42
43
44
45

2.5. LOCAL-VIEW PROGRAMMING MODEL

7

4

of nodes (Figure 2.2).
There are three groups of directives for the global-view programming model. Because these
directives are ignored as a comment by the compilers of base languages (Fortran and C), an
XcalableMP program can be compiled by them to ensure that they run properly.

5

Data Mapping

6

Specifies the data distribution and mapping to nodes (partially inherited from HPF).

7

Work Mapping (Parallelization)

1
2
3

10

Assigns a work to a node set. The loop construct maps each iteration of a loop to nodes owninga specific data elements. The task construct defines a set amount of work as a task, and assigns
it to a specific node set.

11

Communication and Synchronization

8
9

12
13
14

Specifies how to communicate and synchronize with the other compute nodes. In XcalableMP,inter-node communication must be explicitly specified by the programmer. The compiler guarantees that no communication occurs unless it is explicitly specified by the programmer.

Figure 2.2: Parallelization using the global-view programming model.

15

16
17
18
19

2.5

Local-view Programming Model

The local-view programming model is suitable for programs that explicitly describe an algorithm
and a remote data reference that are to be executed by each node (Figure 2.3).
For the local-view programming model, some language extensions and directives are provided.
The coarray notation, which is imported from Fortran 2008, is one such extension, and can be

8

CHAPTER 2. OVERVIEW OF THE XCALABLEMP MODEL AND LANGUAGE

used to specify which replica of a local data is to be accessed. For example, the expression ofA(i)[N] is used to access an array element of A(i) located on the node N. If the access is areference, then a one-sided communication to get the value from the remote memory (i.e., theget operation) is issued by the executing node. If the access is a definition, then a one-sided
communication to put a value to the remote memory (i.e., the put operation) is issued by the
executing node.

1
2
3
4
5
6

Figure 2.3: Local-view programming model.

2.6Interactions between Global View and Local View

In the global view, nodes are used to distribute data and works. In the local view, nodes are used
to address data in the coarray notation. In application programs, programmers should choose
an appropriate data model according to the structure of the program. Figure 2.4 illustrates theglobal view and the local view of data.
Data may have both a global view and a local view, and can be accessed from either.
XcalableMP provides some directives to give the local name (alias) to the global data declaredin the global-view programming model to enable them to also be accessed in the local-viewprogramming model. This feature is useful to optimize a certain part of the program by using
explicit remote data access in the local-view programming model.

2.7

Base Languages

The XcalableMP language specification is defined based on Fortran and C as the base languages.
More specifically, the base language of XcalableMP Fortran is Fortran 90 or later, and that of
XcalableMP C is ISO C90 (ANSI C89) or later.

7

8
9
10
11
12
13
14
15
16

17

18
19
20

2.8. GLOSSARY

9!$xmp nodes P(4)!$xmp template T(100)!$xmp distribute T(block) onto P
real G(80, 100)!$xmp align G(*, i) with T(i)real L(50, 40)! global variable! local variable (default)

Global name space (virtual)G(80, 100)

Data allocation

G(80, 1:25)

G(80, 26:50)

G(80, 51:75)

G(80, 76:100)

L(50,40)

L(50,40)

L(50,40)

L(50,40)

node P(1)node P(2)node P(3)node P(4)Figure 2.4: Global view and local view.

1

2.8

Glossary2

2.8.1

Language Terminology

4base language A programming language that serves as the foundation of the XcalableMP specification.

5base program A program written in a base language.

3

6
7

8
9

10
11
12
13
14

15
16

17
18

XcalableMP
Fortran The XcalableMP specification for a base language Fortran, abbreviatedas XMP/F.
XcalableMP C The XcalableMP specification for a base language C, abbreviated as
XMP/C.
structured block For C, an executable statement, possibly compound, with a single
entry at the top and a single exit at the bottom, or an XcalableMP
construct. For Fortran, a block of executable statements with a single
entry at the top and a single exit at the bottom, or an XcalableMP
construct.
procedure A generic term used to refer to “procedure” (including subroutine andfunction) in XcalableMP Fortran and “function” in XcalableMP C.
directive In XcalableMP Fortran, a comment, and in XcalableMP C, a #pragma,that specifies XcalableMP program behavior.

10

CHAPTER 2. OVERVIEW OF THE XCALABLEMP MODEL AND LANGUAGE
declarative
directive An XcalableMP directive that may only be placed in a declarative
context. A declarative directive has no associated executable user
code; instead, it has one or more associated user declarations.
executabledirective An XcalableMP directive that is not declarative; it may be placed in
an executable context.
construct An XcalableMP executable directive (and for Fortran, the paired end
directive, if any) and the associated statement, loop, or structured
block, if any.
global construct A construct that is executed collectively and synchronously by every
node in the current executing node set. Global constructs are further
classified into two groups of global communication constructs, such as
gmove and barrier, which specify communication or synchronization,
and work mapping constructs, such as loop, array, and tasks, which
specify parallelization of loops, array assignments, or tasks.
template A dummy array that represents an index space to be distributed onto
a node set, which serves as the “template” of parallelization in XcalableMP and can be considered to represent a set of, for example, grid
points in the grid method and particles in the particle method. A template is used in an XcalableMP program to specify the data and work
mapping. Note that the lower bound of each dimension of a templateis one in both XcalableMP Fortran and XcalableMP C.
data mapping Allocating elements of an array to nodes in a node set by specifying
with the align directive that the array is aligned with a distributed
template.
work mapping Assigning each of the iterations of a loop, the elements of an arrayassignment, or the tasks to nodes in a node set. Such work mapping is
specified by aligning it with a template or distributing it onto a node
set.
global A data or a work is global if and only if there are one or more replicatedinstances of it, each of which is shared by the executing nodes.
local A data or a work is local if and only if there is a replicated instance ofit on each of the executing nodes.
global-viewmodel A programming or parallelization model on which parallel programs
are written by specifying how to map global data and works onto
nodes.
local-view model A programming or parallelization model on which parallel programs
are written by specifying how each node owns local data and performs
local works.

1
2
3

4
5

6
7
8

9
10
11
12
13
14

15
16
17
18
19
20
21

22
23
24

25
26
27
28

29
30

31
32

33
34
35

36
37
38

2.8. GLOSSARY

1

2
3
4

5

6
7
8
9

10
11
12
13
14
15

2.8.2

11

Node Terminology
node An execution entity managed by the XcalableMP runtime system,
which has its own memory and can communicate with other nodes.
A node can execute one or more threads concurrently.
node set A totally ordered set of nodes.
entire node set A node set that contains all of the nodes participating in the execution of an XcalableMP program. It is the current executing node
set specified explicitly or implicitly at the beginning of the program
execution.
executing node
set A node set that contains all of the nodes participating in the executionof a procedure, statement, construct, etc. of an XcalableMP programis called its executing node set. In this document, this term is used torepresent the current executing node set unless it is ambiguous. The
executing node set at the beginning of the program execution is theentire node set.

18

current
executing node
set An executing node set of the current context, which is managed by the
XcalableMP runtime system. The current executing node set can be
modified by the task, array, or loop constructs.

19

executing node A node in the executing node set.

16
17

20
21
22
23
24
25

26
27

28
29
30

31
32
33

34
35
36

node array An XcalableMP entity of the same form as a Fortran array that represents a node set in XcalableMP programs. Each element of a nodearray represents a node in the corresponding node set. A node arrayis declared by the nodes directive. Note that the lower bound of eachdimension of a node array is one in both XcalableMP Fortran and
XcalableMP C.
entire nodearray A node array corresponding to the entire node set. An entire nodearray can be declared by a NODES directive without “=node-ref”.
executing nodearray A node array corresponding to the executing node set. An executing node array corresponding to the procedure can be declared by a
NODES directive with the node reference “*”.
parent node set The parent node set of a node set is the last executing node set, which
encountered the innermost task, loop, or array construct that is being executed.
node number A unique number assigned to each node in a node set, which starts
from one and corresponds to its position within the node set that is
totally ordered.

12

CHAPTER 2. OVERVIEW OF THE XCALABLEMP MODEL AND LANGUAGE

2.8.3

Data Terminology
variable A named data storage block whose value can be defined and redefined
during the execution of a program. Note that variables include arraysections.
global data An array that is aligned with a template. Elements of a global data
object are distributed onto nodes according to the distribution of thetemplate. As a result, each node owns a part of the global data (calleda local section), and can access it directly, but cannot access those on
the other nodes.
local data Data that are not global. Each node owns a replica of a local data
object, and can access it directly, but cannot access those on the othernodes. Note that the replicas of a local data object do not always havethe same value.
replicated data Data whose storage is allocated on multiple nodes. A replicated data
is either a local data object or a global data object replicated by analign directive.
distribution Assigning each element of a template to nodes in a node set in a
specified manner. In a broad sense, it refers to assigning each elementof an array, loop, etc.
alignment Associating each element of an array, loop, etc. with an element of
the specified template. An element of the aligned array, loop, etc. isnecessarily mapped to the same node as its associated element of thetemplate.
local section A section of a global data object that is allocated as an array on eachnode at runtime. The local section of a global data object includes its
shadow objects.
shadow An additional area of the local section of a distributed array, which isused to keep elements to be moved in from neighboring nodes.

2.8.4

Work Terminology
task A specific instance of executable codes that is defined by the task
construct and executed by a node set specified by its on clause.

2.8.5

Communication and Synchronization Terminology
communication The movement of data between nodes. Communication in XcalableMP
occurs only when the programmer instructs it explicitly using a global
communication construct or a coarray reference.
reduction A procedure involving combining variables from each node in a specified manner and returning the result value. A reduction always involves
communication. A reduction is specified by either the on clause of theloop construct or the reduction construct.

1

2
3
4

5
6
7
8
9

10
11
12
13

14
15
16

17
18
19

20
21
22
23

24
25
26

27
28

29

30
31

32

33
34
35

36
37
38
39

2.8. GLOSSARY

synchronization Synchronization is a mechanism to ensure that multiple nodes do not
execute specific portions of a program at the same time. Synchronization between any number of nodes is specified by the barrier
construct, and that between two nodes by the post and wait constructs.

1
2
3
4
5

asynchronous
communication Communication that does not block, and which returns before it iscomplete. Thus, statements that follow it can overtake it. An asynchronous communication is specified by the async clause of global
communication constructs or the async directive for a coarray reference.

6
7
8
9
10

11

12
13
14

15
16

17

13

2.8.6

Local-view Terminology
local alias An alias to the local section of a global data object, that is, a distributed array. A local alias can be used in XcalableMP programs inthe same way as normal local data.
image An instance of an XcalableMP program corresponding to a respective
node.
image set A totally ordered set of images.

19

image index An integer value that identifies an image in an image set, whose range
is from one to the size of the image set.

20

entire image set The image set corresponding to the entire node set one to one in turn.

21

executing image
set An image set corresponding to the executing node set one to one in
turn.

18

22
23
24

25
26
27
28
29
30

The executing image set at the beginning of the program execution
comprises the entire image set.
allocation image
set An image set on which the coarray data object is allocated.
The allocation image set for a non-allocatable [F] or a static [C] coarray variable comprises the entire image set. Otherwise, the allocation
image set for an allocatable [F] or an auto [C] coarray variable is theexecuting image set on which it is allocated unless it is specified by
the COARRAY directive.

1

2

3

Chapter 3

Base Language Extensions in
XcalableMP C

5This chapter describes base language extensions in XcalableMP C that are not described in any
other chapters.

6

3.1

7

Synopsis4

Array Section Notation

9

The array section notation is a notation to describe a part of an array, which is adapted in
Fortran.

10

Syntax

8

11

12

13

14

15
16
17
18
19
20
21
22
23
24
25
26
27
28

[C]

array-sectionisarray-name[ { triplet | int-expr } ]...

where triplet is:[base] : [length] [: step]
Description
In XcalableMP C, the base language C is extended so that a part of an array, i.e., an arraysection, can be put in an array assignment statement, which is described in 3.2, and some
XcalableMP constructs. An array section is built from a subset of the elements of an array,
which is specified by this notation including at least one triplet.
When step is positive, the triplet specifies a set of subscripts that is a regularly spaced integer
sequence of length length beginning with base and proceeding in increments of step up to thelargest. When step is negative, the triplet specifies a set of subscripts that is a regularly spaced
integer sequence of length length beginning with base and proceeding in increments of step down
to the smallest.
When base is omitted, it is assumed to be 0. When length is omitted, it is assumed to accountfor the remainder of the array dimension. When step is omitted, it is assumed to be 1.
An array section can be considered as a virtual array containing the set of elements fromthe original array, which is determined by all possible subscript lists that are specified by thesequence of triplets or int-expr’s in square brackets.
15

16

CHAPTER 3. BASE LANGUAGE EXTENSIONS IN XCALABLEMP C

Restrictions

1• [C] Each of base, length, and step must be an integer expression.

2• [C] length must be greater than zero.

3• [C] step must not be zero.

4

Example5

Assuming that an array A is declared by the following statement,6

int A[100];7some array sections can be specified as follows:
A[10:10]A[10:]
A[:10]
A[10:5:2]
A[:]

3.2

array section of 10 elements from A[10] to A[19]array section of 90 elements from A[10] to A[99]array section of 10 elements from A[0] to A[9]array section of 5 elements from A[10] to A[18] by step 2
the whole of A

Array Assignment Statement

8

9

10

Synopsis11

An array assignment statement copies a value into each element of an array section.

12

Syntax13[C]

array-section [:[int-expr]...] = expression;

Description
The value of each element of the result of the right-hand side expression is assigned to thecorresponding element of the array section on the left-hand side. When an operator or anelemental function (see section 7.8) is applied to array sections in the right-hand side expression,
it is evaluated to an array section that has the same shape as that of the operands or arguments,and each element of which is the result of the operator or function applied to the corresponding
element of the operands or arguments. A scalar object is assumed to be an array section thathas the same shape as that of the array section(s), and where each element has its value.
Note that an array assignment is a statement, and therefore cannot appear as an expression
in any other statements.
Restrictions• [C] any array section appearing in the right-hand side expression and the left-hand side
must have the same shape, i.e., the same number of dimensions and size of each dimension.
• [C] If array-section on the left-hand side is followed by “:[int-expr]...”, it must be acoarray.

14

15

16
17
18
19
20
21
22
23
24

25

26
27

28
29

3.3. BUILT-IN FUNCTIONS FOR ARRAY SECTION

1

2
3

17

ExamplesAn array assignment statement in the fourth line copies the elements B[0] through B[4] into
the elements A[5] through A[9].

4

XcalableMP C
int A[10];int B[5];...
A[5:5] = B[0:5];

5

3.3

Built-in Functions for Array Section

10

Some built-in functions are defined that can accept one or more array sections as arguments. In
addition, some of them are array-valued. Such array-valued functions can appear in the righthand side of an array assignment statement, and should be preceded by the array directive if
the array section is distributed.
All of the built-in functions for array sections are described in Sections 7.8 and 7.9.

11

3.4

Pointer to Global Data

12

3.4.1

Name of Global Array

6
7
8
9

17

The name of a global array is considered to represent an abstract entity in the XcalableMP
language. It is not interpreted as the pointer to the array, while the name of a local array is.
However, the name of a global array that appears in an expression is evaluated to the baseaddress of its local section on each node. The pointer can be operated on each node as if it werea normal (local) pointer.

18

3.4.2

13
14
15
16

Address-of Operator

23

The result of the address-of operator (“&”) applied to an element of a global array is the pointerto the corresponding element of its local section. Note that the value of the result pointer isdefined only on the node that owns the element. The pointer can be operated on the node as if
it were a normal (local) pointer.
As a result, for a global array a, a and &a[0] are not always evaluated to the same value.

24

3.5

19
20
21
22

25
26

Dynamic Allocation of Global Data

In XcalableMP C, it is possible to allocate global arrays at runtime. Such an allocation is doneby performing the following steps.

27

1. Declare a pointer to an object of the type of the global array to be allocated.

28

2. Align the pointer with a template as if it were an array.

2930

3. Allocate a block of memory of the global size using the xmp malloc library procedure, andassign the return value to the pointer on each node.

18

CHAPTER 3. BASE LANGUAGE EXTENSIONS IN XCALABLEMP C

XcalableMP C#pragma nodes p(NP1,NP2)#pragma xmp template t(:,:)#pragma xmp distribute t(block,block) onto p
5

float (*pa)[N2];#pragma xmp align pa[i][j] with t(i,j)#pragma xmp template_fix t(0:N1-1,0:N2-1)
pa = (float (*)[N2])xmp_malloc(xmp_desc_of(pa), N1, N2);

3.6

Descriptor-of Operator

The descriptor-of operator (“xmp desc of”) is introduced as a built-in operator in XcalableMPC.
The result of the descriptor-of operator applied to XcalableMP entities such as node arrays,
templates, and global arrays is their descriptor, which can be used in various ways, including as
an argument of some inquiry procedures. The type of the result, xmp desc t, is implementationdefined, and is defined in the xmp.h header file in XcalableMP C.
For the xmp desc of intrinsic function in XcalableMP Fortran, refer to section 7.1.1.

1

2
3
4
5
6
7
8

1

2

3
4

Chapter 4

Directives
This chapter describes the syntax and behavior of XcalableMP directives. In this document,
the following notation is used to describe XcalableMP directives.
xxx
xxx...
[xxx]

5

[F]
[C]

type-face characters are used to indicate literal-type characters.
If the line is followed by “...”, then xxx can be repeated.
xxx is optional.
The syntax rule continues.
The following lines are eﬀective only in XcalableMP Fortran.
The following lines are eﬀective only in XcalableMP C.

6

4.1

Directive Format

7

4.1.1

General Rule

8
9
10
11

In XcalableMP Fortran, XcalableMP directives are specified using special comments that are
identified by unique sentinels !$xmp. An XcalableMP directive follows the rules for comment
lines of either the Fortran free or fixed source form, depending on the source form of the surrounding program unit1 . XcalableMP Fortran directives are case insensitive.
[F]

12

13
14

In XcalableMP C, XcalableMP directives are specified using the #pragma mechanism provided by the C standards. XcalableMP C directives are case-sensitive.
[C]

15

16
17
18
19
20
21
22
23

!$xmp directive-name clause#pragma xmp directive-name clauseDirectives are classified as declarative directives and executable directives.
The declarative directive is a directive that may only be placed in a declarative context.
A declarative directive has no associated executable user code. The scope rule of declarative
directives obeys that of the declaration statements in the base language. For example, in XcalableMP Fortran, a node array declared by a nodes directive is visible only within either the
program unit, the derived-type declaration, or the interface body that immediately surrounds
the directives, unless it is overridden in the inner blocks or is use or host associated. Further, in
XcalableMP C, a node array declared by a nodes directive is visible only in the range from the1Consequently, the rules of comment lines that an XcalableMP directive follows are the same as the ones
followed by an OpenMP directive.

19

20

CHAPTER 4. DIRECTIVES

declaring point to the end of the block when placed within a block, or of the file when placedoutside any blocks, unless overridden in the inner blocks.
Note that in XcalableMP Fortran, node arrays and templates in other scoping units areaccessible by use or host association.
The following directives are declarative directives.

1
2
3
4
5

• nodes6• template

7• distribute8• align9• shadow

10• coarray

11

The executable directives are placed in an executable context. A stand-alone directive is an
executable directive that has no associated user code, such as a barrier directive. An executabledirective and its associated user code make up an XcalableMP construct, as in the following
format:
[F]

[C]

12
13
14
15

!$xmp directive-name clause ...
structured-block16#pragma xmp directive-name clause ...
structured-block17

Note that in XcalableMP Fortran, a corresponding end directive is required for some executable directives such as task and tasks, and in XcalableMP C, the associated statement canbe a compound one.
The following directives are executable directives.

18
19
20
21

• template fix

22• task23• tasks

24• loop25• array26• reflect27• gmove28• barrier29• reduction

30• bcast31• wait async

32

4.2. NODES DIRECTIVE

1

4.1.2

2

Synopsis

21

Combined Directive

4

Multiple attributes can be specified by one combined declarative directive, which is analogous
to type declaration statements using the “::” punctuation.

5

Syntax

3

6

7

8

[F]

!$xmp combined-directive

is

[C]

#pragma xmp combined-directive

iscombined-attribute is one of:nodes
template
distribute (dist-format [, dist-format]... ) onto nodes-namealign ( align-source [, align-source]... )with template-name (align-subscript [, align-subscript]... )shadow ( shadow-width [, shadow-width]... )[F] dimension ( explicit-shape-spec [, explicit-shape-spec]... )9and combined-decl is one of:10

nodes-decl
template-decl
array-name11

combined-attribute [, combined-attribute ]... ::combined-decl [, combined-decl ]...
combined-attribute [, combined-attribute ]... ::combined-decl [, combined-decl ]...

Description

14

A combined directive is interpreted as if an object corresponding to each combined-decl is declared in a directive corresponding to each combined-attribute, where all restrictions of eachdirective, in addition to the following ones, are applied.

15

Restrictions

12
13

16
17

18
19

20
21

22
23

24
25

• The same kind of combined-attribute must not appear more than once in a given combineddirective.
• If the nodes attribute appears in a combined-directive, each combined-decl must be anodes-decl.
• If the template or distribute attribute appears in a combined-directive, each combineddecl must be a template-decl.
• If the align or shadow attribute appears in a combined-directive, each combined-decl must
be an array-name.
• [F] If the dimension attribute appears in a combined-directive, any object to which it
applies must be declared using either the template or the nodes attribute.

22

CHAPTER 4. DIRECTIVES

4.2

nodes Directive

1

Synopsis2

The nodes directive declares a named node array.

3

Syntax4[F]!$xmp nodes nodes-decl [, nodes-decl ]...

[C]

#pragma xmp nodes nodes-decl [, nodes-decl ]...

5

where nodes-decl is one of:nodes-name ( nodes-spec [, nodes-spec ]... )nodes-name ( nodes-spec [, nodes-spec ]... ) = nodes-ref[C] nodes-name [ nodes-spec ][ [ nodes-spec ]... ][C] nodes-name [ nodes-spec ][ [ nodes-spec ]... ] = nodes-refand nodes-spec must be one of:int-expr*
Description
The nodes directive declares a node array that corresponds to a node set.
The first and third forms of the nodes directive are used to declare a node array that
corresponds to the entire node set. The second and fourth forms are used to declare a nodearray, each element of which is assigned to the node of the node set specified by nodes-refat the corresponding position of its elements order. In the first and second forms, which use
parentheses, the element order of the declared node array is based on Fortran s. In the thirdand fourth forms, which use square brackets, the element order of the declared node array isbased on C s.
If node-size in the last dimension is “*” in the first and second forms, or if that in the first
dimension is “*” in the third and fourth forms, then the size of the node array is automaticallyadjusted according to the total size of either the entire node set in the first and third forms or
the referenced node set in the second and fourth forms.
Restrictions• nodes-name must not conflict with any other local name in the same scoping unit.
• nodes-spec can be “*” only in the last dimension in the first and second forms, and nodesspec can be “*” only in the first dimension in the third and fourth forms.
• nodes-ref must not reference nodes-name either directly or indirectly.
• If no nodes-spec is “*”, then the product of all nodes-spec must be equal to the total sizeof the entire node set in the first and third forms, or the referenced node set in the secondand fourth forms.
• nodes-subscript in nodes-ref must not be “*”.

6

7

8

9

10

11
12
13
14
15
16
17
18
19
20
21
22

23

24

25
26

27

28
29
30

31

4.2. NODES DIRECTIVE

1

2
3
4

5

6
7
8
9
10

11

23

ExamplesThe following are examples of the first and the third forms that appears in the main program.
Because the node array p, which corresponds to the entire node set, is declared to be of size 16,
this program must be executed by 16 nodes.
XcalableMP Fortranprogram main!$xmp nodes p(16)!$xmp nodes q(4,*)!$xmp nodes r(8)=p(3:10)
5 !$xmp nodes z(2,3)=p(1:6)...
end program

XcalableMP C
int main() {#pragma xmp nodes p[16]#pragma xmp nodes q[*][4]#pragma xmp nodes r[8]=p[2:8]#pragma xmp nodes z[3][2]=p[0:6]
...
}

5

The following are examples of a node declaration in a procedure. Because p is declared in
the second and fourth forms to have a size of 16 and corresponds to the executing node set, theinvocation of the foo function must be executed by 16 nodes. The node array q is declared in
the first and third forms, and corresponds to the entire node set. The node array r is declared
as a subset of p, and x as a subset of q.
XcalableMP Fortranfunction foo()!$xmp nodes p(16)=*!$xmp nodes q(4,*)!$xmp nodes r(8)=p(3:10)
5 !$xmp nodes x(2,3)=q(1:2,1:3)...
end function

XcalableMP C
void foo(){#pragma xmp nodes p[16]=*#pragma xmp nodes q[*][4]#pragma xmp nodes r[8]=p[2:8]#pragma xmp nodes x[3][2]=q[0:3][0:2]
...
}

12

4.2.1

13

Synopsis

14

The node reference is used to reference a node set.

15

Syntax

16

A node reference nodes-ref is specified by either the name of a node array or the “*” symbol.

17

18

Node Reference

nodes-ref[C] nodes-ref

is
is
or

nodes-name [( nodes-subscript [, nodes-subscript ]... )]nodes-name [[ nodes-subscript ][ [ nodes-subscript ]... ]]*where nodes-subscript must be one of:

19

20

21

22
23

int-expr
triplet*Description
A node reference by nodes-name represents a node set corresponding to the node array specified
by the name or its subarray. It is totally ordered in Fortran’s array element order in the first5

24

CHAPTER 4. DIRECTIVES

form, and in C’s array element order in the second form. A node reference by “*” represents
the executing node set.
Specifically, the “*” symbol that appears as nodes-subscript in a dimension of nodes-ref isinterpreted by each node at runtime as its position (coordinate) in the dimension of the referenced
node array. Thus, a node reference p(s1 , ..., sk−1 , *, sk+1 , ..., sn ) is interpreted asp(s1 , ..., sk−1 , jk , sk+1 , ..., sn ) on the node p(j1 , ..., jk−1 , jk , jk+1 , ..., jn ).
Note that “*” can be used only as the node reference in the on clause of some executabledirectives.

1
2
3
4
5
6
7
8

Examples9

Assume that p is the name of a node array and that m is an integer variable.

10• As a target node array in the distribute directive,
XcalableMP Fortran!$xmp distribute a(block) onto p

XcalableMP C#pragma xmp distribute a(block) onto p• To specify the node array to which the declared node array corresponds in the second andfourth forms of the nodes directive,

11
12
13

14
15
16

XcalableMP Fortran
XcalableMP C!$xmp nodes r(2,2,4) = p(1:4,1:4) #pragma xmp nodes r[4][2][2] = p[0:4][0:4]17!$xmp nodes r(2,2,4) = p(1:16)#pragma xmp nodes r[4][2][2] = p[0:16]• To specify the node array that corresponds to the executing node set of a task in the task
directive,
XcalableMP Fortran
XcalableMP C!$xmp task on p(1:4,1:4)#pragma xmp task on p[0:4][0:4]!$xmp task on p(1:16)#pragma xmp task on p[0:16]!$xmp task on p(:,*)#pragma xmp task on p[*][:]!$xmp task on p(m)#pragma xmp task on p[m]• To specify the node array that corresponds to the executing node set in the barrier andthe reduction directive,
XcalableMP Fortran!$xmp barrier on p(5:8)!$xmp reduction (+:a) on p(*,:)

XcalableMP C#pragma xmp barrier on p[4:4]#pragma xmp reduction (+:a) on p[:][*]• To specify the source node and the node array that corresponds to the executing node set
in the bcast directive,
XcalableMP Fortran
XcalableMP C!$xmp bcast (b) from p(k) on p(:) #pragma xmp (b) from p[k-1] on p[:]4.3

Template and Data Mapping Directives

4.3.1

template Directive

18
19

20

21
22
23

24

25
26
27
28

29

30

Synopsis

31

The template directive declares a template.

32

4.3. TEMPLATE AND DATA MAPPING DIRECTIVES

1

25

Syntax[F]!$xmp template template-decl [, template-decl ]...

[C]

#pragma xmp template template-decl [, template-decl ]...

2

3

where template-decl is:

4[C]

template-name ( template-spec [, template-spec ]... )template-name [ template-spec-c ] [ [ template-spec-c ]... ]5and template-spec must be one of:6[int-expr :] int-expr:

7

and template-spec-c must be one of:int-expr:

89Description

14

The template directive declares a template with the shape specified by the sequence of templatespec’s or template-spec-c’s. If every template-spec or template-spec-c is “:”, then the shape ofthe template is initially undefined. This template must not be referenced until the shape isdefined by a template fix directive (see section 4.3.6) at runtime. If only int-expr is specified
as template-spec, the default lower bound is one.

15

Restrictions

10
11
12
13

16

• template-name must not conflict with any other local name in the same scoping unit.

17• Every template-spec must be either [int-expr :] int-expr or “:”.

18

• Every template-spec-c must be either int-expr or “:”.

19

4.3.2

Template Reference20

Synopsis22

The template reference expression specified in the on or the from clause of some directives isused to indirectly specify a node set.

23

Syntax21

24

25

template-ref[C] template-reftemplate-name [( template-subscript [, template-subscript]... )]template-name [[ template-subscript ] [ [ template-subscript ]...] ]where template-subscript must be one of:

26

27

is
is

int-expr
triplet*26

CHAPTER 4. DIRECTIVES

Description

1

Being specified in the on or the from clause of some directives, the template reference refers to
a subset of a node set in which the specified subset of the template resides.
Specifically, the “*” symbol that appears as template-subscript in a dimension of template-ref
is interpreted by each node at runtime as the indices of the elements in the dimension that residein the node. “*” in a template reference is similar to “*” in a node reference.

2
3
4
5
6

Examples7

Assume that t is a template.

8• In the task directive, the executing node set of the task can be indirectly specified using
a template reference in the on clause.
XcalableMP Fortran!$xmp task on t(1:m,1:n)!$xmp task on t

XcalableMP C#pragma xmp task on t[0:n][0:m]#pragma xmp task on t

• In the loop directive, the executing node set of each iteration of the following loop isindirectly specified using a template reference in the on clause.
XcalableMP Fortran!$xmp loop (i) on t(i-1)

XcalableMP C#pragma xmp loop (i) on t[i-1]• In the array directive, the executing node set on which the associated array-assignment
statement is performed in parallel is indirectly specified using a template reference in theon clause.
XcalableMP Fortran!$xmp array on t(1:n)

XcalableMP C#pragma xmp array on t[0:n]• In the barrier, reduction, and bcast directives, the node set that is to perform theoperation collectively can be indirectly specified using a template reference in the on clause.
XcalableMP Fortran!$xmp barrier on t(1:n)!$xmp reduction (+:a) on t(*,:)!$xmp bcast (b) on t(1:n)

4.3.3

XcalableMP C#pragma xmp barrier on t[0:n]#pragma xmp reduction (+:a) on t[:][*]#pragma xmp bcast (b) on t[0:n]distribute Directive

9
10

11

12
13

14

15
16
17

18

19
20

21

22

Synopsis23

The distribute directive specifies the distribution of a template.

24

Syntax25[F]!$xmp distribute template-name (dist-format [, dist-format]... ) onto nodes-name[C]#pragma xmp distribute template-name (dist-format [, dist-format]... )onto nodes-name#pragma xmp distribute template-name [ dist-format ] [ [ dist-format ] ... ]onto nodes-name

[C]

26

4.3. TEMPLATE AND DATA MAPPING DIRECTIVES

1

2

3

27

where dist-format must be one of:*block [ ( int-expr ) ]cyclic [ ( int-expr ) ]gblock ( { * | int-array } )Description

10

According to the specified distribution format, a template is distributed onto a specified nodearray. The dimension of the node array that appears in the onto clause corresponds, in orderof left-to-right, to the dimension of the distributed template for which the corresponding distformat is not “*”.
Let d be the size of the dimension of the template, p be the size of the corresponding
dimension of the node array, ceiling and mod be Fortran’s intrinsic functions, and each of thearithmetic operators be that of Fortran. The interpretation of dist-format is as follows:

11“*” The dimension is not distributed.

12

block Equivalent to block(ceiling(d/p)).

4
5
6
7
8
9

18

block(n) The dimension of the template is divided into contiguous blocks of size n, which are
distributed onto the corresponding dimension of the node array. The dimension of thetemplate is divided into d/n blocks of size n, and one block of size mod(d,n) if any, andeach block is assigned sequentially to an index along the corresponding dimension of thenode array. Note that if k = p-d/n-1 > 0, then there is no block assigned to the last k
indices.

19

cyclic Equivalent to cyclic(1).

13
14
15
16
17

20
21
22

23
24
25

cyclic(n) The dimension of the template is divided into contiguous blocks of size n, and theseblocks are distributed onto the corresponding dimension of the node array in a round-robinmanner.
gblock(m) m is referred to as a mapping array. The dimension of the template is divided into
contiguous blocks so that the i’th block is of size m(i), and these blocks are distributed
onto the corresponding dimension of the node array.

28

If at least one gblock(*) is specified in dist-format, then the template is initially undefinedand must not be referenced until the shape of the template is defined by template fix directives
at runtime.

29

Restrictions

26
27

30
31

32
33

34
35
36

• [C] template-name must be declared by a template directive that lexically precedes the
directive.
• The number of dist-format that is not “*” must be equal to the rank of the node arrayspecified by nodes-name.
• The size of the dimension of the template specified by template-name that is distributedby block(n) must be equal to or less than the product of the block size n and the size ofthe corresponding dimension of the node array specified by nodes-name.

28

CHAPTER 4. DIRECTIVES• The array int-array in parentheses following gblock must be an integer one-dimensional
array, and its size must be equal to the size of the corresponding dimension of the nodearray specified by nodes-name.
• Every element of the array int-array in parentheses following gblock must have a value ofa nonnegative integer.
• The sum of the elements of the array int-array in parentheses following gblock must be
equal to the size of the corresponding dimension of the template specified by templatename.
• [C] A distribute directive for a template must precede any of its references in theexecutable code in the block.

1
2
3

4
5

6
7
8

9
10

Examples11

Example 1

12

XcalableMP Fortran!$xmp nodes p(4)!$xmp template t(64)!$xmp distribute t(block) onto p

XcalableMP C#pragma xmp nodes p[4]#pragma xmp template t[64]#pragma xmp distribute t[block] onto p

The template t is distributed in block format, as shown in the following table.
p(1)
p(2)
p(3)
p(4)

t(1:16)t(17:32)t(33:48)t(49:64)

p[0]
p[1]
p[2]
p[3]

t[0:16]t[16:16]t[32:16]t[48:16]

Example 2

13

14

15

16

XcalableMP Fortran!$xmp nodes p(4)!$xmp template t(64)!$xmp distribute t(cyclic(8)) onto p

XcalableMP C#pragma xmp nodes p[4]#pragma xmp template t[64]#pragma xmp distribute t[cyclic(8)] onto p

The template t is distributed in cyclic format of size eight, as shown in the following
table.
p(1)
p(2)
p(3)
p(4)

t(1:8) t(33:40)
t(9,16) t(41:48)
t(17,24) t(49:56)
t(25,32) t(57:64)

p[0]
p[1]
p[2]
p[3]

t[0:8] t[32:8]
t[8:8] t[40:8]
t[16:8] t[48:8]
t[24:8] t[56:8]

Example 3
XcalableMP Fortran!$xmp nodes p(8,5)!$xmp template t(64,64,64)!$xmp distribute t(*,cyclic,block) onto p

17

18
19

20

21

XcalableMP C#pragma xmp nodes p[5][8]#pragma xmp template t[64][64][64]22#pragma xmp distribute t[block][cyclic][*] onto p

The first dimension of the template t is not distributed. The second dimension is distributed onto the first dimension of the node array p in cyclic format. The third dimension is distributed onto the second dimension of p in block format. The results are as
follows:

23
24
25
26

4.3. TEMPLATE AND DATA MAPPING DIRECTIVES

p(1,1)
p(2,1)...
p(8,5)

1t(1:64, 1:57:8, 1:13)t(1:64, 2:58:8, 1:13)...
t(1:64, 8:64:8, 53:64)

p[0][0]p[0][1]...
p[4][7]29

t[0:13][0:8:8][0:64]t[0:13][1:8:8][0:64]...
t[52:12][7:8:8][0:64]

Note that the “64” in template t is not divisible by “5” in node p. Thus, the sizes of theblocks are diﬀerent among nodes.

2
3

4

4.3.4

align Directive

5

Synopsis

7

The align directive specifies that an array is to be mapped in the same way as a specified
template.

8

Syntax

6

9

10

11

12

13

[F]

!$xmp align array-name ( align-source [, align-source]... )with template-name (align-subscript [, align-subscript]... )[C]#pragma xmp align array-name [align-source] [[align-source]]...
with template-name (align-subscript [, align-subscript]... )orwith template-name [align-subscript ] [ [ align-subscript ]... ]where align-source must be one of:scalar-int-variable*:
and align-subscript must be one of:scalar-int-variable [ { + | - } int-expr ]*:15Note that the variable scalar-int-variable that appears in align-source is referred to as an“align dummy variable” and int-expr appearing in align-subscript as an “align oﬀset.”

16

Description

14

17
18
19
20

21
22
23

24
25

The array specified by array-name is aligned with the template that is specified by templatename so that each element of the array indexed by the sequence of align-sources is aligned withthe element of the template indexed by the sequence of align-subscripts, where align-sources andalign-subscripts are interpreted as follows:
1. The first form of align-source and align-subscript represents an align dummy variable andan expression of it, respectively. The align dummy variable is considered to range over all
valid index values in the corresponding dimension of the array.
2. The second form “*” of align-source and align-subscript represents a dummy variable (notan align dummy variable) that does not appear anywhere in the directive.

30

CHAPTER 4. DIRECTIVES• The second form of align-source is said to “collapse” the corresponding dimension of
the array. As a result, the index along the corresponding dimension does not aﬀect
the determination of the alignment.
• The second form of align-subscript is said to “replicate” the array. Each element of thearray is replicated, and is aligned to all index values in the corresponding dimensionof the template.
3. The third form of align-source and the matching align-subscript represents the same aligndummy variable whose range spans all valid index values in the corresponding dimensionof the array. The matching of colons (“:”) in the sequence of align-sources and alignsubscripts is determined as follows:• [F] Colons in the sequence of align-sources and those in the sequence of alignsubscripts are matched in corresponding left-to-right order, where any align-source
and align-subscript that is not a colon is ignored.
• [C] Colons in the sequence of align-sources in right-to-left order, and those in thesequence of (align-subscript)’s in left-to-right order are matched, or those in thesequence of [align-subscript]’s in right-to-left order are matched, where any alignsource and align-subscript that is not a colon is ignored.
In XcalableMP C, an align directive for a dummy argument can be placed either outside
the function body (as in the old style of C) or in it (as in the ANSI style).
Restrictions• [C] array-name must be declared by a declaration statement that lexically precedes the
directive.

1
2
3
4
5
6

7
8
9
10

11
12
13
14
15
16
17

18
19

20

21
22

• An align dummy variable may appear at most once in the sequence of align-sources.

23• An align dummy variable may appear at most once in the sequence of align-subscripts.

24• An align-subscript may contain at most one occurrence of an align dummy variable.

25• The int-expr in an align-subscript may not contain any occurrence of an align dummy
variable.
• The sequence of align-sources must contain exactly as many colons as contained by thesequence of align-subscripts.
• [F] The array specified by array-name must not appear as an equivalence-object in an
equivalence statement.
• [C] An align directive for an array must precede any of its appearances in the executablecode in the block.

26
27

28
29

30
31

32
33

• [F] The array specified by array-name shall not be initially defined.

34• [C] The array specified by array-name shall not be initialized through an initializer.

35

4.3. TEMPLATE AND DATA MAPPING DIRECTIVES

1

Examples2

Example 1
XcalableMP Fortran!$xmp align a(i) with t(i)331

XcalableMP C#pragma xmp align a[i] with t[i]6

In XcalableMP Fortran, the array element a(i) is aligned with the template elementt(i). In XcalableMP C, the array element a[i] is aligned with the template elementt[i]. These are equivalent to the following codes.

7

XcalableMP Fortran!$xmp align a(:) with t(:)

4
5

8

Example 2
XcalableMP Fortran!$xmp align a(*,j) with t(j)

9

11
12

Example 3
XcalableMP Fortran!$xmp align a(j) with t(*,j)

14XcalableMP C#pragma xmp align a[j] with t[j][*]In XcalableMP Fortran, the array element a(j) is replicated and aligned with each template element of t(:,j). In XcalableMP C, the array element a[j] is replicated andaligned with each template element of t[j][:].

15
16
17

18

XcalableMP C#pragma xmp align a[j][*] with t[j]In XcalableMP Fortran, the subarray a(:,j) is aligned with the template element t(j).
Note that the first dimension of a is collapsed. In XcalableMP C, the subarray a[j][:] isaligned with the template element t[j]. Note that the second dimension of a is collapsed.

10

13

XcalableMP C#pragma xmp align a[:] with t[:]Example 4
XcalableMP Fortran!$xmp template t(n1,n2)
real a(m1,m2)!$xmp align a(*,j) with t(*,j)

19XcalableMP C#pragma xmp template t[n2][n1]double a[m2][m1]#pragma xmp align a[j][*] with t[j][*]In XcalableMP Fortran, the subarray a(:,j) is aligned with each template element oft(:,j). In XcalableMP C, the subarray a[j][:] is aligned with each template elementof t[j][:].

20
21
22

By replacing “*” of the array a and “*” of the template t with a dummy variable i andk, respectively, this alignment can be interpreted as the following mapping.

23
24[F] a(i, j) → t(k, j) | (i, j, k) ∈ (1 : n1, 1 : n2, 1 : m1)[C] a[j][i] → t[j][k] | (i, j, k) ∈ (0 : n1, 0 : n2, 0 : m1)

25
26

27

4.3.5

shadow Directive

28

Synopsis

29

The shadow directive allocates the shadow area for a distributed array.

30

Syntax[F]!$xmp shadow array-name ( shadow-width [, shadow-width]... )[C]#pragma xmp shadow array-name [shadow-width][[shadow-width]]...

31

32

CHAPTER 4. DIRECTIVES

where shadow-width must be one of:int-expr
int-expr : int-expr*

Description
The shadow directive specifies the width of the shadow area of an array specified by array-name,which is used to communicate the neighbor element of the block of the array. When shadowwidth is of the form “int-expr : int-expr,” the shadow area of the width specified by the firstint-expr is added at the lower bound, and that specified by the second one is added at the upperbound in the dimension. When shadow-width is of the form int-expr, the shadow area of thesame width specified is added at both the upper and lower bounds in the dimension. Whenshadow-width is of the form “*”, the entire area of the array is allocated on each node, and thearea that it does not own is regarded as a shadow. This type of shadow is sometimes referred
to as a “full shadow.”
Note that the shadow area of a multi-dimensional array includes “obliquely-neighboring”
elements, which are owned by the node whose indices are diﬀerent in more than one dimension,and that the shadow area can also be allocated at the global lower and upper bounds of anarray.
The data stored in the storage area declared by the shadow directive is referred to as ashadow object. A shadow object represents an element of a distributed array, and corresponds
to the data object that represents the same element as itself. The corresponding data object isreferred to as the reflection source of the shadow object.

Restrictions• [C] array-name must be declared by a declaration statement that lexically precedes the
directive.

• The value specified by shadow-width must be a nonnegative integer.

• The number of shadow-width must be equal to the number of dimensions (or rank) of thearray specified by array-name.

• [C] A shadow directive for an array must precede any of its appearances in the executablecode in the block.

1

2

3

4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

21

22
23

24

25
26

27
28

4.3. TEMPLATE AND DATA MAPPING DIRECTIVES

33

Example1a(16,17:32)

XcalableMP Fortran!$xmp nodes p(4,4)!$xmp template t(64,64)!$xmp distribute t(block,block) onto pa(16,33)a(16,16)a(17:32,16)a(17:32,
17:32)a(17:32,33)

25real a(64,64)!$xmp align a(i,j) with t(i,j)!$xmp shadow a(1,1)

a(33,33)a(33,16)a(33,17:32)

Figure 4.1: Example showing shadowof a two-dimensional array.

6

The node p(2,2) has a(17:32,17:32) as a data object, and a(16,16), a(17:32,16),a(33,16), a(16,17:32), a(33,17:32), a(16,33), a(17:32,33), and a(33,33) as shadow objects (Figure 4.1). Among them, a(16,16), a(33,16), a(16,33), and a(33,33) are “obliquelyneighboring” elements of p(2,2).

7

4.3.6

8

Synopsis

9This construct fixes the shape and/or the distribution of an undefined template.

10

Syntax

3
4
5

11

12

13

14

15

16

17

18
19
20
21

template fix Construct[F]!$xmp template fix[( dist-format [, dist-format]... )] template-name [(template-spec [, template-spec]... )][C]#pragma xmp template fix[( dist-format [, dist-format]...)] template-name [(template-spec [, template-spec]... )]or[[ dist-format [, dist-format]...]] template-name [ template-spec ] [ [template-spec ]... ]where template-spec is:[int-expr :] int-expr
and dist-format is one of:*block [( int-expr )]cyclic [( int-expr )]gblock ( int-array )Description
The template fix construct fixes the shape and/or the distribution of the template that is
initially undefined, by specifying the sizes and/or the distribution format of each dimension
at runtime. Arrays that are aligned with an initially undefined template must be allocatable
arrays, in XcalableMP Fortran, or a pointer (see Section 3.5), in XcalableMP C, which cannot

34

CHAPTER 4. DIRECTIVES

be allocated until the template is fixed by the template fix construct. No constructs thathave such a template in their on clause should be encountered until the template is fixed by the
template fix construct. Any undefined template can be fixed only once by the template fix
construct in its scoping unit.
The meaning of the sequence of dist-formats is the same as that in the distribute directive.

Restrictions• When a node encounters a template fix construct at runtime, the template specified bytemplate-name must be undefined.

1
2
3
4
5

6

7
8

• If the sequence of dist-formats exists in a template fix construct, it must be identicalto the sequence of dist-formats in the distribute directive for the template specified bytemplate-name, except for int-array specified in the parenthesis immediately after gblock.

10• Either the sequence of dist-formats or the sequence of template-spec’s should be given.

12

Example9

11

13

XcalableMP Fortran
XcalableMP C!$xmp nodes p(*)#pragma xmp nodes p[*]!$xmp template t(:)#pragma xmp template t[:]!$xmp distribute t(gblock(*)) onto p #pragma xmp distribute t[gblock(*)] onto p
real, allocatable :: a(:)
double *a;
5 !$xmp align a(i) with t(i)#pragma xmp align a[i] with t[i]
5
...
...
14
N = ...
N = ...;
M(...) = ...
M[] = {...};...
...
10 !$xmp template_fix(gblock(M)) t(N)#pragma xmp template_fix[gblock(M)] t[N] 10...
...
allocate (a(N))
a = xmp_malloc(xmp_desc_of(a), N);
Because the shape is t(:) or t[:] and the distribution format is gblock(*), the templatet is initially undefined. The allocatable array a is aligned with t. After the size N and themapping array M is defined, t is fixed by the template fix construct and a is allocated.
In XcalableMP C, it is possible to allocate global arrays at runtime only when they are
one-dimensional. Such an allocation is done by perfoming the following steps.

15
16
17
18
19

1. Declare a pointer to an object of the type of the global array to be allocated.

20

2. Align the pointer with a template as if it were a one-dimensional array.

21

3. Allocate a storage of the global size with the function xmp malloc() and assign the result
value to the pointer on each node.
The functions xmp desc of() and xmp malloc() are described in section 3.6 and 7.5.1, respectively.

22
23

24
25

4.4. WORK MAPPING CONSTRUCT

1

4.4

Work Mapping Construct

2

4.4.1

task Construct3

Synopsis

4

The task construct defines a task that is executed by a specified node set.

5

Syntax[F]!$xmp task on {nodes-ref | template-ref}structured-block!$xmp end task[C]#pragma xmp task on {nodes-ref | template-ref}structured-block

35

6

7

Description

16

When a node encounters a task construct at runtime, it executes the associated block (called atask) if it is included by the node set specified by the on clause; otherwise, it skips the executionof the block.
Unless a task construct is surrounded by a tasks construct, nodes-ref or template-ref in theon clause is evaluated by the executing node set at the start of the task; otherwise, nodes-refand template-ref of the task construct are evaluated by the executing node set at the entry of
the tasks construct that immediately surrounds it. The current executing node set is set to bethat specified by the on clause at the entry of the task construct, and it is rewound to the last
one at the exit.

17

Restrictions

8
9
10
11
12
13
14
15

18
19

20

21
22
23
24

25
26
27

28

• The node set specified by nodes-ref or template-ref in the on clause must be a subset ofthe parent node set.

ExampleExample 1 In XcalableMP Fortran, copies of variables a and b are replicated on nodes nd(1)through nd(8). A task defined by the task construct is executed only on nd(1), anddefines the copies of a and b on a node nd(1). The copies on nodes nd(2) through nd(8)are not defined.
In XcalableMP C, copies of variables a and b are replicated on nodes nd[0] through nd[7].
A task defined by the task construct is executed only on nd[0], and defines the copies ofa and b on a node nd[0]. The copies on nodes nd[1] through nd[7] are not defined.

36

CHAPTER 4. DIRECTIVES

XcalableMP Fortran!$xmp nodes nd(8)!$xmp template t(100)!$xmp distribute t(block) onto ndreal a, b;

5

XcalableMP C#pragma xmp nodes nd[8]#pragma xmp template t[100]#pragma xmp distribute t[block] onto nd
float a, b;

5
1!$xmp task on nd(1)read(*,*) ab = a*1.e-6
10 !$xmp end task

#pragma xmp task on nd[0]{scanf ("%f", &a);b = a*1.e-6;}

10

Example 2 According to the on clause with a template reference, an assignment statement in
the task construct is executed by the owner of the array element a(:,j) or a[j][:].

5

XcalableMP Fortran!$xmp nodes nd(8)!$xmp template t(100)!$xmp distribute t(block) onto nd

XcalableMP C#pragma xmp nodes nd[8]#pragma xmp template t[100]#pragma xmp distribute t(block) onto nd

integer i,jreal a(200,100)!$xmp align a(*,j) with t(j)int i,j;float a[100][200];#pragma align a[j][*] with t[j]

2
3
4

5

5

i = ...
j = ...

10!$xmp task on t(j)a(i,j) = 1.0!$xmp end task

4.4.2

i = ...;
j = ...;#pragma xmp task on t[j]a[j][i] = 1.0;}

tasks ConstructSynopsis
The tasks construct is used to instruct the executing nodes to execute the multiple tasks thatit surrounds in an arbitrary order.
Syntax[F]!$xmp tasks
task-construct...
!$xmp end tasks[C]#pragma xmp tasks{
task-construct
...
}

10

6

7

8
9

10

11

4.4. WORK MAPPING CONSTRUCT

1

37

Description

7

task constructs surrounded by a tasks construct are executed in arbitrary order without implicit
synchronization at the start of each task. As a result, if there are no overlaps between theexecuting node sets of the adjacent tasks, they can be executed in parallel.
nodes-ref or template-ref of each task immediately surrounded by a tasks construct is evaluated by the executing node set at the entry of the tasks construct.
No implicit synchronization is performed at the start and end of the tasks construct.

8

Example

2
3
4
5
6

9
10
11

12

13
14

Example 1 Three instances of subroutine task1 are concurrently executed by node sets p(1:500),
p(501:800), and p(801:1000).
XcalableMP Fortran
subroutine caller!$xmp nodes p(1000)!$xmp template tp(100)!$xmp distribute t(block) onto p5real a(100,100)!$xmp align a(*,k) with t(k)...
!$xmp tasks!$xmp task on p(1:500)
10
call task1(a)!$xmp end task!$xmp task on p(501:800)
call task1(a)!$xmp end task
15 !$xmp
task on p(801:1000)
call task1(a)!$xmp end task!$xmp end tasks...
20
end subroutine

XcalableMP Fortran
subroutine task1(a)...
!$xmp nodes q(*)=*!$xmp nodes p(1000)!$xmp distribute t(block) onto preal a(100,100)!$xmp align a(*,k) with t(k)...
end subroutine

5

10

Example 2 The first node p(1) executes the first and second tasks, the final node p(8) the
second and the third tasks, and the other nodes p(2) through p(7) only the second task.

15

XcalableMP Fortran!$xmp nodes p(8)!$xmp template t(100)!$xmp distribute t(block) onto preal a(100)
5 !$xmp align a(i) with t(i)...
!$xmp tasks!$xmp task on t(1)

38

CHAPTER 4. DIRECTIVES

10a(1) = 0.0!$xmp end task!$xmp task on t(2:99)!$xmp loop on t(i)15do i=2,99a(i) = foo(i)enddo!$xmp end task
20!$xmp task on t(100)a(100) = 0.0!$xmp end task!$xmp end tasks

4.4.3

loop ConstructSynopsis

1

2

The loop construct specifies that each iteration of the following loop is executed by a node set
that is specified by the on clause, so the iterations are distributed among nodes and executed inparallel.

Syntax

3
4
5

6

[F]

!$xmp loop [ ( loop-index [, loop-index]... ) ] on {nodes-ref | template-ref}[ expand( expand-width [, expand-width]... ) ][ margin( margin-width [, margin-width]... ) ][ reduction-clause ]...
do-loops[C]#pragma xmp loop [ ( loop-index [, loop-index]... ) ] on {nodes-ref | template-ref}[ expand( expand-width [, expand-width]... ) ][ margin( margin-width [, margin-width]... ) ][ reduction-clause ]...
for-loops

7

where expand-width and margin-width must be one of:[/unbound/] int-expr[/unbound/] int-expr : int-expr
reduction-clause is:reduction( reduction-kind : reduction-spec [, reduction-spec ]... )reduction-kind is one of:

8

9

10

11

12

4.4. WORK MAPPING CONSTRUCT

[F]

+
*
.and.
.or.
.eqv.
.neqv.
max
min
iand
ior
ieor
firstmax
firstminlastmax
lastmin

[C]

+
*
&
|
^
&&
||
max
min
firstmax
firstminlastmax
lastmin

1

2

3

4

5
6
7
8
9
10
11
12
13
14
15
16
17

39

and reduction-spec is:reduction-variable [ / location-variable [, location-variable ]... / ]Description
A loop directive is associated with a loop nest consisting of one or more tightly nested loops
that follow the directive, and it distributes the execution of their iterations onto the node set
specified by the on clause.
The sequence of loop-indexes in parenthesis denotes an index of an iteration of the loop nests.
If a control variable of a loop does not appear in the sequence, it is assumed that each of its
possible values is specified in the sequence. The sequence can be considered to denote a set ofindices of iterations. When the sequence is omitted, it is assumed that the control variables of
all the loops in the associated loop nests are specified.
When a template-ref is specified in the on clause, the associated loop is distributed so thatthe iteration (set) indexed by the sequence of loop-indexes is executed by the node onto whicha template element specified by the template-ref is distributed.
When a nodes-ref is specified in the on clause, the associated loop is distributed so that theiteration (set) indexed by the sequence of loop-indexes is executed by a node specified by the

40

CHAPTER 4. DIRECTIVES

nodes-ref.
In addition, the executing node set is updated to the node set specified by the on clause at
the beginning of every iteration, and it is restored to the last one at the end of it.
When a reduction-clause is specified, a reduction operation of the kind specified by reductionkind for a variable specified by reduction-variable is executed just after the execution of the loop
nest.
When the expand clause is specified, and is of the form “int-expr : int-expr” in a dimension,
the first int-expr is subtracted from the local lower bound in that dimension, and the second
one is added to the local upper bound. When the expand clause is specified, and is of the form
int-expr, the int-expr is subtracted from the local lower bound in that dimension, and is added
to the local upper bounds. However, an “expanded” local iteration space does not spread outof the original global iteration space unless the /unbound/ modifier is specified in expand-width.
When the margin clause is specified, the loop is transformed so that its local iteration space,margin, is:margin = expand △ orig
where expand is a local iteration space when an expand clause with the same argument(s) is
specified, orig is a local iteration space when neither expand nor margin, and △ is the symmetricdiﬀerence operator.
(Advice to programmers and implementers) Using the expand and margin clauses
and asynchronous communication, programmers can overlap computation and communication as in the code left below. It is recommended for the implementation to
support an extension that is a syntactic sugar for those sequence of constructs, suchas the peel and wait clause in the code immediately following.

1
2
3
4
5
6
7
8
9
10
11
12

13
14
15

16
17
18
19
20

XcalableMP Fortran!$xmp reflect (a) async(10)!$xmp loop (i,j) on t(i,j)!$xmp+
expand(-1,-1)
5
do j = 1, 16
do i = 1, 16...
end doend do10!$xmp wait_async (10)!$xmp loop (i,j) on t(i,j)!$xmp+
margin(-1,-1)
15
do j = 1, 16
do i = 1, 16...
end doend doXcalableMP Fortran!$xmp reflect (a) async(10)!$xmp loop (i,j) on t(i,j)!$xmp+
peel_and_wait(10, -1,-1)
do j = 1, 16
do i = 1, 16...
end doend do5

The reduction operation that is executed, except in cases with reduction-kind of FIRSTMAX,FIRSTMIN, LASTMAX, or LASTMIN, is equivalent to the reduction construct with reduction-kindof “+” for “-” in the clause and the same reduction-kind for the other kinds, the same reductionvariable, and an on clause obtained from that of the loop directive by replacing each loop-index

21

22
23
24
25

4.4. WORK MAPPING CONSTRUCT

1
2

3

41

in the nodes-ref or the template-ref with a triplet representing the range of its value. As an
example, the two codes below are therefore equivalent.
XcalableMP Fortran! Initialize s_tmp to the identity! element of the op operator
s_tmp = ...
XcalableMP Fortran!$xmp loop (j) on t(:,j)!$xmp loop (j) on t(:,j)
5!$xmp+
reduction(op:s)do j = js, je
do j = js, je...
...
do i = 1, N
5
do i = 1, N
s_tmp = s_tmp op a(i,j)
s = s op a(i,j)
end do10
end do...
...
end doend do!$xmp reduction(op:s_tmp)!$xmp+on t(*,js:je) 15
s = s op s_tmp

4
5
6

7
8
9
10

11
12
13
14

15

In particular, for the reduction kinds of FIRSTMAX, FIRSTMIN, LASTMAX, and LASTMIN, in
addition to a corresponding MAX or MIN reduction operation, the location-variables are set after
executing the loop construct as follows:• For FIRSTMAX and FIRSTMIN, they are set to their values at the end of the first iterationin which the reduction-variable takes the value of the reduction result, where first refers
to the first position in the sequential order in which iterations of the associated loop nest
were executed without parallelization.
• For LASTMAX and LASTMIN, they are set to their values at the end of the last iteration inwhich the reduction-variable takes the value of the reduction result, where last refers to thelast position in the sequential order in which iterations of the associated loop nest were
executed without parallelization.
Restrictions

16• loop-index must be a control variable of a loop in the associated loop nest.

17• A control variable of a loop can appear as loop-index at most once.

19• The node set specified by nodes-ref or template-ref in the on clause must be a subset ofthe parent node set.

20

• The template specified by template-ref must be fixed before the loop construct is executed.

18

23• The loop construct is global, which means that it must be executed by all of the executing
nodes with the same values for each local variable referenced in the directive, and the lower
bound, upper bound, and step of the associated loop.

24• Either of the expand or margin clause, if any, can be specified.

21
22

25
26

• The number of expand-width, if any, must be equal to the number of dimensions (or rank)of the template specified by template-ref or of the node array specified by node-ref.

42

CHAPTER 4. DIRECTIVES• The number of margin-width, if any, must be equal to the number of dimensions (or rank)of the template specified by template-ref or of the node array specified by node-ref.
• reduction-spec must have one or more location-variable’s if and only if reduction-kind iseither FIRSTMAX, FIRSTMIN, LASTMAX, or LASTMIN.

1
2

3
4

Examples5

Example 1

6

XcalableMP Fortran!$xmp distribute t(block) onto p!$xmp align (i) with t(i) :: a, b...
!$xmp loop (i) on t(i)5do i = 1, Na(i) = 1.0b(i) = a(i)end doThe loop construct determines the node that executes each of the iterations, according to
the distribution of template t, and distributes the execution. This example is syntacticallyequivalent to the one shown below, but will be faster because the iterations to be executedby each node can be determined before executing the loop.
XcalableMP Fortran!$xmp distribute t(block) onto p!$xmp align (i) with t(i) :: a, b...
do i = 1, N
5 !$xmp task on t(i)a(i) = 1.0b(i) = a(i)!$xmp end task
end doExample 2

7
8
9
10

11

XcalableMP Fortran!$xmp distribute t(*,block) onto p!$xmp align (i,j) with t(i,j) :: a, b...
!$xmp loop (i,j) on t(i,j)
5
do j = 1, M
do i = 1, Na(i,j) = 1.0b(i,j) = a(i,j)
end do10
end doBecause the first dimension of template t is not distributed, only the j loop, which isaligned with the second dimension of t, is distributed. This example is syntacticallyequivalent to the task construct shown below.

12
13
14

4.4. WORK MAPPING CONSTRUCT

43

XcalableMP Fortran!$xmp distribute t(*,block) onto p!$xmp align (*,j) with t(*,j) :: a, b...
do j = 1, M
5 !$xmp task on t(*,j)
do i = 1, Na(i,j) = 1.0b(i,j) = a(i,j)
end do10 !$xmp end task
end do1

Example 3
XcalableMP Fortran!$xmp distribute t(block,block) onto p!$xmp align (i,j) with t(i,j) :: a, b...
!$xmp loop (i,j) on t(i,j)
5
do j = 1, M
do i = 1, Na(i,j) = 1.0b(i,j) = a(i,j)
end do10
end do

2
3
4
5
6

7

The distribution of loops in the nested loop can be specified using the sequence of loopindexes in one loop construct. This example is equivalent to the loop shown below, butwill run faster because the iterations to be executed by each node can be determinedoutside of the nested loop. Note that the node set specified by the inner on clause is asubset of that specified by the outer one.
XcalableMP Fortran!$xmp distribute t(block,block) onto p!$xmp align (i,j) with t(i,j) :: a, b...
!$xmp loop (j) on t(:,j)
5do j = 1, M!$xmp loop (i) on t(i,j)
do i = 1, Na(i,j) = 1.0b(i,j) = a(i,j)
10
end doend doExample 4
XcalableMP Fortran!$xmp nodes p(10,3)...
!$xmp loop on p(:,i)
do i = 1, 3

44

CHAPTER 4. DIRECTIVES

call subtask ( i )end do5

Three node sets p(:,1), p(:,2), and p(:,3) are created as the executing node sets,
and each of them executes iterations 1, 2, and 3 of the associated loop, respectively.
This example is equivalent to the loop containing task constructs (below left) or static
tasks/task constructs (below right).

XcalableMP Fortran!$xmp nodes p(10,3)...
do i = 1, 3!$xmp task on p(:,i)
5call subtask ( i )!$xmp end task
end doXcalableMP Fortran!$xmp nodes p(10,3)...
!$xmp tasks!$xmp task on p(:,1)
call subtask ( 1 )!$xmp end task!$xmp task on p(:,2)
call subtask ( 2 )!$xmp end task!$xmp task on p(:,3)
call subtask ( 3 )!$xmp end task!$xmp end tasks

1
2
3
4

5

5

10

Example 5

6

XcalableMP Fortran...
lb(1) = 1
iub(1) = 10
lb(2) = 115
iub(2) = 25
lb(3) = 26
iub(3) = 50!$xmp loop (i) on p(lb(i):iub(i))
do i = 1, 3
10call subtask ( i )end doThe executing node sets of diﬀerent sizes are created by p(lb(i):iub(i)) with diﬀerent
values of i for unbalanced workloads. This example is equivalent to the loop containing
task constructs (below left) or static tasks/task constructs (below right).

789

4.4. WORK MAPPING CONSTRUCT

1

2

XcalableMP Fortrando i = 1, 3!$xmp task on p(lb(i):iub(i))
call subtask ( i )!$xmp end task5
end do...

45
XcalableMP Fortran!$xmp tasks!$xmp task on p(1:10)
call subtask ( 1 )!$xmp end task!$xmp task on p(11:25)
call subtask ( 2 )!$xmp end task!$xmp task on p(25:50)
call subtask ( 3 )!$xmp end task!$xmp end tasks

5

10

Example 6
XcalableMP Fortran...
s = 0.0!$xmp loop (i) on t(i) reduction(+:s)
do i = 1, N
5s = s + a(i)end do345This loop computes the sum of a(i) into the variable s on each node. Note that only thepartial sum is computed on s without the reduction clause. This example is equivalent tothe code given below.
XcalableMP Fortran...
s = 0.0!$xmp loop (i) on t(i)do i = 1, N
5s = s + a(i)end do!$xmp reduction(+:s) on t(1:N)

6

Example 7
XcalableMP Fortran...
amax = -1.0e30
ip = -1
jp = -1
5 !$xmp loop (i,j) on t(i,j) reduction(firstmax:amax/ip,jp/)
do j = 1, M
do i = 1, N
if( 1(i,j) .gt. amx ) thenamx = a(i,j)
10
ip = ijp = jend if

46

CHAPTER 4. DIRECTIVES

end doend doThis loop computes the maximum value of a(i,j) and stores it into the variable amax ineach node. In addition, the first indices for the maximum element of a are obtained in
ip and jp after executing the loops. Note that this example cannot be written using the
reduction construct.
Example 8

1
2
3
4

5

XcalableMP Fortran!$xmp loop (i,j) on t(i,j) expand(/unbound/1,/unbound/1)
do j = 1, 16
do i = 1, 16...
5
end doend do!$xmp loop (i,j) on t(i,j) margin(/unbound/1,/unbound/1)
do j = 1, 16
10
do i = 1, 16...
end doend doAssuming that the template t(100,100) is distributed in (block,block) onto a node arrayp(4,4), the original local iteration space on p(1,1), orig1,1 is:orig1,1 = { (1, 1),(1, 2),(1, 3),(1, 4),(2, 1),(2, 2),(2, 3),(2, 4),(3, 1),(3, 2),(3, 3),(3, 4),(0, 1),(1, 1),(2, 1),(3, 1),(4, 1),(5, 1),(0, 2),(1, 2),(2, 2),(3, 2),(4, 2),(5, 2),

7(4, 1),(4, 2),(4, 3),(4, 4) }and it is expanded using the expand clause for the first loop, as follows:
expand(1, 1)1,1 = { (0, 0),(1, 0),(2, 0),(3, 0),(4, 0),(5, 0),

6(0, 3),(1, 3),(2, 3),(3, 3),(4, 3),(5, 3),(0, 4),(1, 4),(2, 4),(3, 4),(4, 4),(5, 4),

8(0, 5),(1, 5),(2, 5),(3, 5),(4, 5),(5, 5) }Note that expand(1, 1)1,1 spreads out of the original global iteration space {(i, j) | 1 ≤i, j ≤ 16} because the /unbound/ specifier is specified in the expand clause.
The local iteration space for the second loop with the margin clause is defined using thesymmetric diﬀerence operator, as follows:margin(1, 1)1,1 = expand(1, 1)1,1 △orig1,1= { (0, 0), (0, 1), (0, 2), (0, 3), (0, 4),(1, 0),(2, 0),(3, 0),(4, 0),(5, 0), (5, 1), (5, 2), (5, 3), (5, 4),(0, 5),(1, 5),(2, 5),(3, 5),(4, 5),(5, 5) }

9
10
11
12

4.4. WORK MAPPING CONSTRUCT

1

4.4.4

2

Synopsis

3

The array construct divides the work of an array assignment between nodes.

4

Syntax47

array Construct[F]!$xmp array on template-refarray-assignment-statement[C]#pragma xmp array on template-refarray-assignment-statement

5

6

Description

12

The array assignment is an alternative to a loop that performs an assignment to each elementof an array. This directive specifies the parallel execution of an array assignment, where each
sub-assignment and sub-operation of an element is executed by a node that is determined by
the on clause.
Note that array assignments can also be used in XcalableMP C, which is one of the language
extensions introduced by XcalableMP (see Section 3.2).

13

Restrictions

7
8
9
10
11

14
15

16
17

18
19

• The node set specified by template-ref in the on clause must be a subset of the parent nodeset.
• The template section specified by template-ref must have the same shape as the associatedarray assignment.
• The array construct is global and must be executed by all of the executing nodes withthe same valuse for the variables that appear in the construct.

20

Examples21

Example 1
XcalableMP Fortran!$xmp distribute t(block) onto p!$xmp align (i) with t(i) :: a...
!$xmp array on t(1:N)
5a(1:N) = 1.0

22This example is equivalent to the code shown below.
XcalableMP Fortran!$xmp distribute t(block) onto p!$xmp align (i) with t(i) :: a...
!$xmp loop on t(1:N)
5
do i = 1, Na(i) = 1.0
end do

48

CHAPTER 4. DIRECTIVES

Example 2

1

XcalableMP Fortran!$xmp template t(100,20)!$xmp distribute t(block,block) onto p
dimension a(100,20), b(100,20)!$xmp align (i,j) with t(i,j) :: a, b
5...
!$xmp array on t
a = b + 2.0
This example is equivalent to the code shown below.

2

XcalableMP Fortran!$xmp template t(100,20)!$xmp distribute t(block,block) onto p
dimension a(100,20), b(100,20)!$xmp align (i,j) with t(i,j) :: a, b
5...
!$xmp loop (i,j) on t(i,j)
do j = 1, 20
do i = 1, 100a(i,j) = b(i,j) + 2.0
10
end doend do4.5

Global-view Communication and Synchronization Constructs 3

4.5.1

reflect ConstructSynopsis5

The reflect construct assigns the value of a reflection source to the corresponding shadow
object.
Syntax
[F]
[C]

4

6
7

8

!$xmp reflect ( array-name [, array-name]... )[width ( reflect-width [, reflect-width]... )] [orthogonal] [async ( async-id )]#pragma xmp reflect ( array-name [, array-name]... )[width ( reflect-width [, reflect-width]... )] [orthogonal] [async ( async-id )]where reflect-width must be one of:[/periodic/] int-expr[/periodic/] int-expr : int-expr
Description
The reflect construct updates each of the shadow objects of the array specified by array-namewith the value of its corresponding reflection source. Note that the shadow objects corresponding
to elements at the non-orthogonal positions are also updated with this construct, unless theorthogonal clause is specified.

9

10

11

12

13
14
15
16

4.5. GLOBAL-VIEW COMMUNICATION AND SYNCHRONIZATION CONSTRUCTS

49

14

When the width clause is specified and takes the form “int-expr : int-expr” in a dimension,
the shadow area having the width specified by the first int-expr at the lower bound and thatspecified by the second one at the upper bound in the dimension are updated. When the widthclause is specified, and takes the form int-expr, the shadow areas having the same width specified
at both the upper and lower bounds in the dimension are updated. When the width clause isomitted, the whole shadow area of the array is updated.
In particular, when the /periodic/ modifier is specified in reflect-width, the update of the
shadow object in the dimension is “periodic,” which means that the shadow object at the global
lower (upper) bound is treated as if it corresponds to the data object of the global upper (lower)bound, and is updated with that value by the reflect construct.
When the orthogonal clause is specified, only the shadow objects corresponding to elements
at the orthogonal positions are updated by the reflect construct.
When the async clause is specified, the statements following this construct may be executed
before the operation is complete.

15

Restrictions

1
2
3
4
5
6
7
8
9
10
11
12
13

• The arrays specified by the sequence of array-names must be mapped onto the executing
node set.

16
17• The reflect width of each dimension specified by the reflect-width must not exceed the
shadow width of the arrays.

18
19• The reflect construct is global, which means that it must be executed by all nodes in
the current executing node set, and each local variable referenced in the construct musthave the same value among all of them.

20
21
22

• async-id must be an expression of type default integer in XcalableMP Fortran or type int
in XcalableMP C.

23
24

Example25

XcalableMP Fortran!$xmp nodes p(4)!$xmp template t(100)!$xmp distribute t(block) onto p
5

10real a(100)!$xmp align a(i) with t(i)!$xmp shadow a(1)...
!$xmp reflect (a) width (/periodic/1)

a
p(1)

p(2)

p(3)

1

p(4)
100

Figure 4.2: Example of periodic shadow reflection.

50

CHAPTER 4. DIRECTIVESThe shadow directive allocates “periodic” shadow areas of the array a. The reflect construct updates “periodically” the shadow area of a (Figure 4.2). A periodic shadow at the lower
bound on the node p(1) is updated with the value of a(100) and that at the upper bound onp(4) with the value of a(1).

4.5.2

gmove ConstructSynopsis
The gmove construct allows an assignment statement, which may cause communication, to be
executed possibly in parallel by the executing nodes.
Syntax[F][C]!$xmp gmove [in | out] [async ( async-id )]#pragma xmp gmove [in | out] [async ( async-id )]Description
This construct copies the value of the right-hand side variable into the left-hand side of theassociated assignment statement, which may cause communication between the executing nodes.
Such communication is detected, scheduled, and performed by the XcalableMP runtime system.
There are three operating modes of the gmove construct:• collective mode
When neither the in nor the out clause is specified, the copy operation is performed
collectively, and results in implicit synchronization among the executing nodes.
If the async clause is not specified, then the construct is “synchronous,” and it is guaranteed that the left-hand side data can be read and overwritten, the right-hand side data can
be overwritten, and all of the operations of the construct on the executing nodes are completed when returning from the construct; otherwise, the construct is “asynchronous,” andit is not guaranteed that the operations are completed, until the associating wait async
construct (Section 4.5.6) is completed.
• in mode
When the in clause is specified, the right-hand side data of the assignment, all or part ofwhich may reside outside the executing node set, can be transferred from its owner nodes
to the executing nodes by this construct.
If the async clause is not specified, then the construct is “synchronous,” and it is guaranteed that the left-hand side data can be read and overwritten, and that all of the operationsof the construct on the owner nodes of the right-hand side and the executing nodes are completed when returning from the construct; otherwise, the construct is “asynchronous,” andit is not guaranteed that the operations are completed, until the associating wait async
construct (Section 4.5.6) is completed.
• out mode
When the out clause is specified, the left-hand side data of the assignment, all or part ofwhich may reside outside the executing node set, can be transferred from the executing
nodes to its owner nodes by this construct.
If the async clause is not specified, then the construct is “synchronous,” and it is guaranteed that the right-hand side data can be overwritten, and that all of the operations of

1
2
3
4

5

6

7
8

9

10

11

12
13
14
15

16
17
18
19
20
21
22
23
24

25
26
27
28
29
30
31
32
33
34

35
36
37
38
39
40

4.5. GLOBAL-VIEW COMMUNICATION AND SYNCHRONIZATION CONSTRUCTS

1
2
3
4

51

the construct on the owner nodes of the left-hand side and the executing nodes are completed when returning from the construct; otherwise, the construct is “asynchronous,” andit is not guaranteed that the operations are completed, until the associating wait async
construct (Section 4.5.6) is completed.

6

When the async clause is specified, the statements following this construct may be executed
before the operation is complete.

7

Restrictions

5

8
9

10
11
12

13
14
15

16
17
18

19
20
21

22
23

24

25
26
27
28

29

30
31
32
33

34

• The gmove construct must be followed by (i.e., associated with) a simple assignment statement that contains neither arithmetic operations nor function calls.
• The gmove construct is global, which means that it must be executed by all nodes in thecurrent executing node set, and each local variable referenced in the construct must havethe same value.
• If the gmove construct is in the collective mode, then all elements of the distributed arrays
appearing on both the left-hand side and the right-hand side of the associated assignment
statement must reside in the executing node set.
• If the gmove construct is in the in mode, then all elements of the distributed array appearing
on the left-hand side of the associated assignment statement must reside in the executing
node set.
• If the gmove construct is in the out mode, then all elements of the distributed arrayappearing on the right-hand side of the associated assignment statement must reside in
the executing node set.
• async-id must be an expression of type default integer in XcalableMP Fortran or type int
in XcalableMP C.
ExamplesExample 1: Array assignment If the arrays on both the left-hand side and the right-handside are distributed, then the copy operation is performed using all-to-all communication.
If the left-hand side is a replicated array, this copy is performed using multi-cast communication. If the right-hand side is a replicated array, then no communication is required.
XcalableMP Fortran!$xmp gmovea(:,1:N) = b(:,3,0:N-1)

XcalableMP C#pragma xmp gmove
a[1:N][:] = b[0:N][3][:];

Example 2: Scalar assignment to an array When the right-hand side is an element of a
distributed array, the copy operation is performed by broadcast communication from theowner of the element. If the right-hand side is a replicated array, then no communicationis required.
XcalableMP Fortran!$xmp gmovea(:,1:N) = c(k)XcalableMP C#pragma xmp gmove
a[1:N][:] = c[k]52

CHAPTER 4. DIRECTIVES

Example 3: in mode assignment Because b(3) referenced on the right-hand side of the
gmove construct does not reside in the executing node set (p(1:2)), the construct is executed in the in mode. Thus, b(3) is transferred from its owner node p(3) to the executing
node set.
Until p(1:2) returns from the construct, there is no gurantee that any node can read andoverwrite a(1:2), and that any relevant operations on p(1:2) and p(3) are completed.

1
2
3
4
5
6

XcalableMP Fortran!$xmp nodes p(4)!$xmp template t(4)!$xmp distribute t(block) onto preal a(4), b(4)!$xmp align (i) with t(i) : a, b...
!$xmp task on p(1:2)...
10 !$xmp gmove ina(1:2) = b(2:3)...
!$xmp end task
5

4.5.3

barrier Construct7

Synopsis

8

The barrier construct specifies an explicit barrier at the point at which the construct appears.

9

Syntax

10

[F]
[C]

!$xmp barrier [on nodes-ref |template-ref]#pragma xmp barrier [on nodes-ref | template-ref]Description
The barrier operation is performed among the node set specified by the on clause. If no on
clause is specified, then it is assumed that the current executing node set is specified in it.
Note that an on clause may represent multiple node sets. In such a case, a barrier operationis performed in each node set.
Restriction• The node set specified by the on clause must be a subset of the executing node set.

4.5.4

reduction Construct

11

12

13
14
15
16

17

18

19

Synopsis

20

The reduction construct performs a reduction operation among nodes.

21

4.5. GLOBAL-VIEW COMMUNICATION AND SYNCHRONIZATION CONSTRUCTS

1

Syntax
[F]

2

3

4

6

7

8

9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

!$xmp reduction ( reduction-kind : variable [, variable ]... )[on node-ref | template-ref] [async ( async-id )]where reduction-kind is one of:
+
*
.and.
.or.
.eqv.
.neqv.
max
min
iand
ior
ieor
[C]

5

53

#pragma xmp reduction ( reduction-kind : variable [, variable ]... )[on node-ref | template-ref] [async ( async-id )]where reduction-kind is one of:
+
*
&
|
^
&&
||
max
min
Description
The reduction construct performs a type of reduction operation specified by reduction-kindfor the specified local variables among the node set specified by the on clause, and it sets
the reduction results to the variables on each of the nodes. Note that some of the reduction
operations, namely, FIRSTMAX, FIRSTMIN, LASTMAX, and LASTMIN, which can be specified in the
reduction clause of the loop directive, cannot be specified in the reduction construct because
their semantics are not defined for it. The variable specified by variable, which is the target ofthe reduction operation, is referred to as the “reduction variable.” After the reduction operation,
the value of a reduction variable becomes the same in every node that performs the operation.
The reduction result is computed by combining the reduction variables on all of the nodes
using the reduction operator. The ordering of this reduction is unspecified.
When the async clause is specified, the statements following this construct may be executed
before the operation is complete.
When template-ref is specified in the on clause, the operation is performed in a node set
that consists of nodes onto which the specified template section is distributed. Therefore, before
the reduction construct is executed, the referenced template must be fixed. When nodes-ref is
specified in the on clause, the operation is performed in the specified node set. When the onclause is omitted, the operation is performed in the executing node set.

54

CHAPTER 4. DIRECTIVES

Note that an on clause may represent multiple node sets. In such a case, a reduction operationis performed in each node set.
Restrictions

1
2

3• The variables specified by the sequence of variables must either not be aligned or must be
replicated among nodes of the node set specified by the on clause.
• The reduction construct is global, which means that it must be executed by all nodes in
the current executing node set, and each local variable referenced in the construct musthave the same value.

4
5

6
7
8

• async-id must be an expression of type default integer in XcalableMP Fortran or type int
in XcalableMP C.

10• The node set specified by the on clause must be a subset of the executing node set.

11

Examples9

12

Example 1

13

XcalableMP Fortran!$xmp reduction(+:s)!$xmp reduction(max:aa) on t(*,:)!$xmp reduction(min:bb) on p(10:30)
In the first line, the reduction operation calculates the sum of the scalar variable s in theexecuting node set, and the result is stored in the variable in each node.
The reduction operation in the second line computes the maximum value of the variable
aa in each node set onto which each of the template sections specified by t(*,:) isdistributed.
In the third line, the minimum value of the variable bb in the node set specified by p(10:30)
is calculated. This example is equivalent to the following code using the task construct.
XcalableMP Fortran!$xmp task on p(10:30)!$xmp reduction(min:bb)!$xmp end task
Example 2
XcalableMP Fortrandimension a(n,n), p(n), w(n)!$xmp align a(i,j) with t(i,j)!$xmp align p(i) with t(i,*)!$xmp align w(j) with t(*,j)
5...
!$xmp loop (j) on t(:,j)
do j = 1, n
sum = 0!$xmp loop (i) on t(i,j) reduction(+:sum)10
do i = 1, n
sum = sum + a(i,j) * p(i)end do

14
15
16
17
18
19
20

21

4.5. GLOBAL-VIEW COMMUNICATION AND SYNCHRONIZATION CONSTRUCTS

55

w(j) = sum
end do12This code computes the matrix vector product, where a reduction clause is specified forthe loop construct of the inner loop. This is equivalent to the following code snippet.
XcalableMP Fortran!$xmp loop (j) on t(:,j)
do j = 1, n
sum = 0!$xmp loop (i) on t(i,j)
5
do i = 1, n
sum = sum + a(i,j) * p(i)end do!$xmp reduction(+:sum) on t(1:n,j)w(j) = sum10
end do

3
4
5
6
7
8
9

10
11

In these cases, the reduction operation on the scalar variable sum is performed for every
iteration in the outer loop, which may cause a large overhead. To reduce this overhead,
the reduction clause should be specified in the loop construct for the outer loop. This isbecause the node set in which the reduction operation is performed is determined on thebasis of its on clause (see 4.4.3), and the on clause of the outer loop construct is diﬀerent
from that of the inner one. However, this code can be modified using the reduction
construct as follows:
XcalableMP Fortrandimension a(n,n), p(n), w(n)!$xmp align a(i,j) with t(i,j)!$xmp align p(i) with t(i,*)!$xmp align w(j) with t(*,j)
5...
!$xmp loop (j) on t(:,j)
do j = 1, n
sum = 0!$xmp loop (i) on t(i,j)
10
do i = 1, n
sum = sum + a(i,j) * p(i)end dow(j) = sum
end do15 !$xmp reduction(+:w) on t(1:n,*)This code performs a reduction operation on the array w only once, which may result infaster operation.

12

4.5.5

bcast Construct13

Synopsis

14

The bcast construct performs broadcast communication from a specified node.

56

CHAPTER 4. DIRECTIVES

Syntax[F][C]1!$xmp bcast ( variable [, variable]... ) [from nodes-ref | template-ref][on nodes-ref] | template-ref] [async ( async-id )]#pragma xmp bcast ( variable [, variable]... ) [from nodes-ref | template-ref][on nodes-ref | template-ref] [async ( async-id )]Description
The values of the variables specified by the sequence of variables (called broadcast variables) are
broadcasted from the node specified by the from clause (called the source node) to each of thenodes in the node set specified by the on clause. After executing this construct, the values of thebroadcast variables become the same as those in the source node. If the from clause is omitted,
then the first node, that is, the leading one in Fortran’s array element order, of the node set
specified by the on clause is assumed to be a source node. If the on clause is omitted, then it isassumed that the current executing node set is specified in it.
When the async clause is specified, the statements following this construct may be executed
before the operation is complete.
Restrictions• The variables specified by the sequence of variables must either not be aligned or must be
replicated among nodes of the node set specified by the on clause.
• The bcast construct is global, which means that it must be executed by all nodes in thecurrent executing node set, and each local variable referenced in the construct must havethe same value among all of them.
• async-id must be an expression of type default integer in XcalableMP Fortran or type int
in XcalableMP C.
• The node set specified by the on clause must be a subset of the executing node set.
• The source node specified by the from clause must belong to the node set specified by theon clause.
• The source node specified by the from clause must be one node.

4.5.6

wait async ConstructSynopsis
The wait async construct guarantees that asynchronous communications specified by async-id
are complete.
Syntax[F][C]!$xmp wait async ( async-id [, async-id ]... ) [on nodes-ref | template-ref]#pragma xmp wait async ( async-id [, async-id ]... ) [on nodes-ref | template-ref]Description
The wait async construct will block, and therefore statements following it will not be executed,
until the completion of all of the asynchronous communications that are specified by async-id’sand issued on the node set specified by the on clause. If an async-id is not associated with any
asynchronous communication, the wait async construct ignores it.

2

3

4
5
6
7
8
9
10
11
12

13

14
15

16
17
18

19
20

21

22
23

24

25

26

27
28

29

30

31

32
33
34
35

4.5. GLOBAL-VIEW COMMUNICATION AND SYNCHRONIZATION CONSTRUCTS

1

2
3

4
5

6
7
8

9
10

57

Restrictions• async-id must be an expression of type default integer in XcalableMP Fortran or type int
in XcalableMP C.
• async-id must be associated with an asynchronous communication using the async clauseof a communication construct.
• The wait async construct is global, which means that it must be executed by all nodes in
the current executing node set, and each local variable referenced in the construct musthave the same value among all of them.
• The node set specified by the on clause must be the same as those of the global constructs
that initiate the asynchronous communications specified by async-id.

11

4.5.7

async Clause

12

Synopsis

14

The async clause of the reflect, gmove, reduction, and bcast constructs enables the corresponding communication to be performed asynchronously.

15

Description

13

18

Communication corresponding to the construct with an async clause is performed asynchronously,that is, it is initiated but not completed, and therefore, statements following it may be executed
before the communication is complete.

19

Example16
17

XcalableMP Fortran!$xmp reflect (a) async(1)S1!$xmp wait_async(1)S2

23

The reflect construct on the first line matches the wait construct on the third line because
both of their async id evaluate to one. These constructs ensure that statements in S1 can be
executed before the reflect communication is complete, and no statement in S2 is executed
until the reflect communication is complete.

24

4.5.8

25

Synopsis26

The reduce shadow construct adds values of shadow objects to their reflection source.

27

Syntax

20
21
22

[F]
28

[C]

reduce shadow Construct!$xmp reduce shadow ( array-name [, array-name]... )[width ( reflect-width [, reflect-width]... )] [orthogonal] [async ( async-id )]#pragma xmp reduce shadow ( array-name [, array-name]... )[width ( reflect-width [, reflect-width]... )] [orthogonal] [async ( async-id )]58

CHAPTER 4. DIRECTIVES

Description

1

The reduce shadow construct adds values of shadow objects of the array specified by arrayname to their reflection source. Note that the shadow objects corresponding to elements at the
non-orthogonal positions are also added as the default behavior.
When the width clause is specified and has the form “int-expr : int-expr” in a dimension,
the shadow areas having the width specified by the first int-expr at the lower bound, and thatspecified by the second one at the upper bound in the dimension are added. When the widthclause is specified and has the form int-expr, the shadow areas having the same width specified
at both the upper and lower bounds in the dimension are added. When the width clause isomitted, the whole shadow area of the array is added.
In particular, when the /periodic/ modifier is specified in reflect-width, the addition of the
shadow object in the dimension is “periodic,” which means that the shadow object at the global
lower (upper) bound is treated as if it corresponds to the data object of the global upper (lower)bound and is added by the reduce shadow construct.
When the orthogonal clause is specified, the shadow object is added only by orthogonal
nodes.
When the async clause is specified, the statements following this construct may be executed
before the operation is complete.

10

Restrictions

19• The arrays specified by the sequence of array-names must be mapped onto the executing
node set.
• The width of each dimension specified by reflect-width must not exceed the shadow widthof the arrays.
• The reduce shadow construct is global, which means that it must be executed by all nodes
in the current executing node set, and each local variable referenced in the construct musthave the same value among all of them.
• async-id must be an expression of type default integer in XcalableMP Fortran or type int
in XcalableMP C.
Examples

3
4
5
6
7
8
9

11
12
13
14
15
16
17
18

20
21

22
23

24
25
26

27
28

29

XcalableMP Fortran
real rho(n,n)!$xmp align rho(i,j) with t1(i,j)!$xmp shadow rho(1:1)
52real f(m)integer x(m), y(m)!$xmp align (k) with t2(k) : f, x, y!$xmp loop on t2(k)do i = 1, noix = x(i)iy = y(i)dx = x(i) - ix
dy = y(i) - iy
15
rho(ix ,iy ) = rho(ix10,iy) + (1.0-dx)*(1.0-dy)*f(i)4.5. GLOBAL-VIEW COMMUNICATION AND SYNCHRONIZATION CONSTRUCTS

59

rho(ix+1,iy ) = rho(ix+1,iy ) +dx *(1.0-dy)*f(i)rho(ix ,iy+1) = rho(ix ,iy+1) + (1.0-dx)*
dy *f(i)rho(ix+1,iy+1) = rho(ix+1,iy+1) +dx *dy *f(i)end do20!$xmp reduce_shadow (rho)!$xmp reflect (rho)
1
2
3
4
5

Assume that a two-dimensional field rho and m particles are both distributed onto nodes.
On each node, a contribution of a particle f(k) is added to the nearest grid point of the field
and its neighbors, which may be in the shadow area on the node. In the last two lines, thevalues of the shadow area from neighboring nodes are added to the corresponding data object,
and the results are then copied back to the shadow area on the neighboring nodes.

1

2

3

4
5
6
7
8

9
10
11
12
13

14
15
16

17
18
19
20

21
22
23
24

25

Chapter 5

Support for the Local-view
Programming
This chapter describes the coarray features in XcalableMP, which are based on that of Fortran2008. Note that they are also available in XcalableMP C. In addition, this chapter describes
some directives for the local-view programming.
The coarray features in Fortran 2008 are extended and integrated into XcalableMP. The
specifications in this chapter are designed to achieve the following purposes:• Upward compatibility to the Fortran 2008 coarray features
If an XcalableMP Fortran program does not contain any XMP directives, any standardconforming Fortran 2008 program remains standard conforming under XcalableMP. In thissense, the interpretations and extensions defined in this chapter are upward compatible
with the Fortran International Standard, ISO/IEC 1539-1:2010 (Fortran 2008).
• Support for task parallelism
XcalableMP makes it possible to construct a task parallel program by combining multiple
Fortran 2008 codes, which may be developed independently, with minimum modifications.
• Integration of global-view style programming and local-view style programming
In XcalableMP, users can appropriately use global-view style programming of XcalableMPor local-view style programming, which is typically used in MPI or Fortran 2008 programs,
according to the characteristics of the code in a program.
• Possibility of support for multiple topologies of a computing system
An XcalableMP processor may allow users to specify the correspondence between nodearrays and the topologies of a computing system, and to exploit the full potential of aparticular system.

5.1

Rules Determining Image Index

27

This section defines how the image index of an image in a set of images is determined inassociation with a node array and a task construct.

28

5.1.1

26

29
30
31

Primary Image Index

Every image has a default image index in all of the images at the invocation of a program. In
XcalableMP, the default image index is the primary image index, and is an integer value that
ranges from one to the number of images at the invocation of a program.
61

62

CHAPTER 5. SUPPORT FOR THE LOCAL-VIEW PROGRAMMING

A primary node array corresponds to all of the images at the invocation of a program, andit also corresponds to all of the nodes at the invocation of a program. The primary image index
of an image is the (Fortran) subscript order value of the uniquely corresponding element of a
primary node array.

5.1.2

Image Index Determined by a task Directive

1
2
3
4

5

The execution of a task directive determines that a set of nodes (and the corresponding set
of images) forms an executing node set. If a name of a node array or a subobject of a nodearray appears in the task directive, the nodes and the corresponding images in the executing
node set are ordered in (Fortran) array element order in the node array or the subobject of thenode array. If a name of a template array or a subobject of a template array appears in thetask directive, the nodes and the corresponding images in the executing node set are orderedin (Fortran) array element order in the corresponding subobject of the node array. The image
index of an image in the determined set of images is the integer order value in the range one tothe cardinality of the set of images.

10

5.1.3

15

Current Image Index

The image index of an image in the current set of images is the current image index.
A current executing node array corresponds to the current set of images and also the current
executing node set at the evaluation of the declaration of the node array. Each image in thecurrent set of images corresponds to the element of an executing node array whose subscript ordervalue is the same as the current image index of the image when the evaluation of the declarationof the executing node array is being evaluated. In particular, when all task directive constructs
are completed, the current image index of an image is the same as the primary image index.

5.1.4

Image Index Determined by a Non-primary Node Array

A non-primary node array corresponds to all of the images at the invocation of a program, andit also corresponds to all the nodes at the invocation of a program. The correspondence between
each image and each element of a non-primary node array is processor dependent. A processor
may support any means to specify the correspondence.
The image index of an image in all of the images at the invocation of a program is thesubscript order value of the corresponding element of a non-primary node array. This is thecase if and only if the current set of images corresponds to the non-primary node whole array inwhich the nodes in the executing node set are ordered in (Fortran) array-element order in the
non-primary node whole array. The image index is a non-primary image index.
The correspondence between the primary image index and a non-primary image index of
the same image is processor dependent. Between any two distinct non-primary node arrays, thecorrespondence between a non-primary image index and the other non-primary image index of
the same image is processor dependent unless they have the same shape. If two non-primarynode arrays have the same shape, the corresponding elements of the node arrays correspond to
the same image.

5.1.5

Image Index Determined by an Equivalenced Node Array

A nodes directive with “=node-ref” that is not “=*” or “=**” specifies that each element of
the declared node array corresponds in (Fortran) array-element order to that of the node-ref,which is the name of a node array or a subobject of a node array. The nodes in the declared
node array and the corresponding images are ordered in (Fortran) array-element order in the

6
7
8
9

11
12
13
14

16
17
18
19
20
21
22

23

24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

39

40
41
42
43

5.2. BASIC CONCEPTS

63

2

node-ref. The image index of an image in the set of images corresponding to the declared nodearray is the integer order value ranging from one to the cardinality of the set of images.

3

5.1.6

1

On-node Image Index

10

XcalableMP supports the coarray directive and the image directive to specify that an image
index indicates the image corresponding to the element of a particular node array whose subscript
order value is the same as the image index. The image index is an on-node image index for the
specified node array. Because an evaluation of the declaration of a node array determines a setof images corresponding to the node array, the directives specify that the set of images is the“all images” for the image indices aﬀected by the directives. In particular, the on-node image
index for a primary node array is the primary image index.

11

5.2

4
5
6
7
8
9

12
13
14
15
16
17
18
19
20
21

22
23
24
25

Basic Concepts

In XcalableMP, “all images” in Fortran 2008 changes coupled with the execution of task constructs, and refers to the current set of images. In particular, when an allocate statement is
executed for which an allocate-object is a coarray, there is an implicit synchronization of all theimages in the current set of images. On each image in the current set of images, execution of thesegment following the statement is delayed until all other images in the set have executed thesame statement the same number of times. When a deallocate statement is executed for whichan allocate-object is a coarray, there is an implicit synchronization of all the images in the current
set of images. On each image in the current set of images, execution of the segment following
the statement is delayed until all other images in the set have executed the same statement thesame number of times.
• When an allocatable coarray is allocated during the execution of task constructs, thecoarray shall be subsequently deallocated before the completion of the task constructwhose task directive is the most recently executed one in the task constructs that are notcompleted at the allocation.

29

The image index determined by an image selector indicates the current image index bydefault. Coarrays are visible within the range of the “all images,” and are accessed using thecurrent image index by default. The image index that appears in an executable statement
indicates the current image index by default.

30

5.2.1

26
27
28

31
32
33

Examples• In the following code fragment, the value of a coarray b on the images 1, 2, 3, and 4,
which constitute the executing node set and correspond to node(5), node(6), node(7),and node(8) respectively, is defined with the value of the coarray a on node(5).
XcalableMP Fortranprogram xmpcoarray!$xmp nodes node(8)=**! A primary node array.
!$xmp task on node(5:8) ! The executing node set
call sub! corresponds to node(5:8).
5 !$xmp end task
endsubroutine sub
real, save :: a[*], b[*] ! The images 1, 2, 3,

64

CHAPTER 5. SUPPORT FOR THE LOCAL-VIEW PROGRAMMING:b = a[1]10! and 4 correspond to node(5:8),! respectively.

• In the following code fragment, an allocatable coarray a is allocated on the images 1, 2,
3, and 4, which constitute the executing node set and correspond to node(5), node(6),node(7), and node(8), respectively.
XcalableMP Fortranprogram xmpcoarray!$xmp nodes node(8)=**!$xmp task on node(5:8) ! The executing node set
call sub2! corresponds to node(5:8).
5 !$xmp end task
end

1
2
3

subroutine sub2
real, allocatable :: a(:)[:]:
allocate(a(0:99)[*])

10

Note

4

• The result of xmp num nodes() is always the same as that of NUM IMAGES().

5• The result of xmp node num() is always the same as that of THIS IMAGE().

6• In a read statement, an io-unit that is an asterisk identifies an external unit that ispreconnected for a sequential formatted input only on the image whose primary image
index is one.

5.3

coarray Directive

5.3.1

Purpose and Form of the coarray Directive

The coarray directive maps coarrays onto a node array and the set of images that corresponds
to the node array. An image index determined by an image selector for a coarray that appears
in a coarray directive always indicates the on-node image index for the node array; that is, thespecified image corresponds to the node whose subscript order value in the node array is thesame as the image index.
A coarray appearing in a coarray directive is an on-node coarray of the node array that is
specified in the coarray directive.
[F]!$xmp coarray on node-name :: object-name-list[C]#pragma xmp coarray on node-name :: object-name-list

7
8
9

10

11

12
13
14
15
16
17
18

19

• An object-name shall be a name of a coarray declared in the same scoping unit.
• The same object-name shall not appear more than once in coarray directives in a scoping
unit.

20

21
22

5.3. COARRAY DIRECTIVE

1
2
3
4

5
6
7

8
9
10
11

12
13

14
15
16

17

18
19
20
21
22
23
24

25
26
27
28
29
30
31
32

33

65

• If an object-name is a name of an allocatable object, the current set of images at theallocation and the deallocation of the object shall correspond to the node array specified
as the node-name, and the current image index of each image shall be the same as thesubscript order value of the corresponding element of the node array.
• If an object-name is the name of an allocated allocatable dummy argument, the set ofimages onto which it is mapped shall be a subset of the set of images that has mostrecently allocated the corresponding argument in the chain of argument associations.
• If an object-name is the name of a nonallocatable dummy argument whose ultimate argument has an allocatable attribute, the set of images onto which the object-name is mapped
shall be a subset of the set of images that has most recently allocated the correspondingargument in the chain of argument associations.
• The image index determined by an image selector for an on-node coarray shall be within
the range of one to the size of the node array onto which the on-node coarray is mapped.
• THIS IMAGE(COARRAY[,DIM]) shall be invoked by the image contained in the set ofimages onto which the COARRAY argument is mapped if the COARRAY argument appears in a coarray directive.
Note• The result value of THIS IMAGE(COARRAY) is the sequence of cosubscript values for the
COARRAY argument that would specify the current image index of the invoking image,if the COARRAY argument does not appear in a coarray directive. The result valueof THIS IMAGE(COARRAY) is the sequence of cosubscript values for the COARRAY
argument that would specify the on-node image index of the invoking image for the nodearray onto which the COARRAY argument is mapped if the COARRAY argument appears
in a coarray directive.
• The result value of THIS IMAGE(COARRAY,DIM) is the value of cosubscript DIM in
the sequence of cosubscript values for the COARRAY argument that would specify thecurrent image index of the invoking image if the COARRAY argument does not appear ina coarray directive. The result value of THIS IMAGE(COARRAY,DIM) is the value ofcosubscript DIM in the sequence of cosubscript values for the COARRAY argument thatwould specify the on-node image index of the invoking image for the node array onto which
the COARRAY argument is mapped if the COARRAY argument appears in a coarraydirective.

5.3.2

An Example of the coarray Directive
XcalableMP Fortranmodule global!$xmp nodes node(8)=**real s[*]!$xmp coarray on node :: s
5
end global
program coarrayuse global!$xmp task on node(5:8)
10call sub! The coarray s is always! visible on node(1:8).

! The executing node set! consists of node(5:8).

66

CHAPTER 5. SUPPORT FOR THE LOCAL-VIEW PROGRAMMING!$xmp end task
endsubroutine subuse global
real, save :: a[*]! The images 1, 2, 3, and 4:! correspond to node(5:8), respectively.
if(this_image().eq.1)then ! The value of the coarray a on node(5)s[1] = a! defines that of the coarray s on node(1)
endif

15

20

5.4

image Directive

5.4.1

Purpose and Form of the image Directive

1

The image directive specifies that an image index in the following executable statement indicatesthe on-node image index of the node array specified in the image directive unless the image index
is determined by an image selector.
The image directive also specifies that the execution of a sync all statement performs a
synchronization of all of the images corresponding to the node array specified in the image
directive.
[F]!$xmp image ( node-name )[C]#pragma xmp image ( node-name )

2

3
4
5
6
7
8

9

• An image directive shall be followed by a sync all statement, an image control statementthat contains image-set, or a reference to an intrinsic procedure that has IMAGES argument.

5.4.2

An Example of the image Directive
XcalableMP Fortranmodule global!$xmp nodes node(8)=**real s[*]!$xmp coarray on node :: s
5
end global! The coarray s is always visible! on node(1:8).

program image
use global!$xmp tasks
10 !$xmp task on node(1:4)
call subA ! The executing node set consists of node(1:4).
!$xmp end task!$xmp task on node(5:8)
call subB ! The executing node set consists of node(5:8).
15 !$xmp end task!$xmp end tasks
end

10
11

12

5.5. IMAGE INDEX TRANSLATION INTRINSIC PROCEDURES

67

subroutine subA
use global
real, save :: a[*] ! The images 1, 2, 3, and 4:! correspond to node(1:4), respectively.
!$xmp image(node)! Synchronization between node(1:4) andsync images(5)! node(5).
25a = s[1]! a on node(1:4) is defined using:! the value of s on node(1).
end subroutine
20

subroutine subB
use global
real, save :: b[*] ! The images 1, 2, 3, and 4:! correspond to node(5:8), respectively.
if(this_image() .eq. 1)then ! The image 1 indicates node(5).
s[1] = b! s on node(1) is defined using the value of35! b on node(5).
!$xmp
image(node)! Synchronization between
sync images((/1,2,3,4/)) ! node(5) and node(1:4).
endif
:
40
end subroutine
30

1

5.5

Image Index Translation Intrinsic Procedures

3

XcalableMP supports intrinsic procedures to translate image indices between diﬀerent sets ofimages.

4

5.5.1

5

xmp get primary image index(NUMBER,INDEX,PRI INDEX,NODE DESC)

6

Description. Translate image indices to the primary image indices.

7

Class. Subroutine.

8

Arguments.

2

9
10
11
12
13
14
15
16
17
18
19
20

Translation to the Primary Image Index

NUMBER shall be a scalar of type default integer. It is an INTENT(IN) argument.
INDEX shall be a rank-one array of type default integer. The size of INDEX shall begreater than or equal to the value of NUMBER. It is an INTENT(IN) argument.
The value of each element of INDEX shall be within the range one to the size of thenode array specified in NODE DESC if NODE DESC appears. The value of eachelement of INDEX shall be within the range one to the cardinality of the current
set of images if NODE DESC does not appear.
PRI INDEX shall be a rank-one array of type default integer. The size of PRI INDEX
shall be greater than or equal to the value of NUMBER. It is an INTENT(OUT)
argument. If NODE DESC appears, PRI INDEX(i) is assigned the primary image index corresponding to the element of the node array specified in NODE DESC
whose subscript order value is INDEX(i); otherwise, PRI INDEX(i) is assigned

68

CHAPTER 5. SUPPORT FOR THE LOCAL-VIEW PROGRAMMING

the primary image index corresponding to the image whose current image index is
INDEX(i).
NODE DESC (optional) shall be a descriptor of a node array. It is an INTENT(IN)
argument. NODE DESC shall appear in XcalableMP C.
Example. In the following code fragment, the value of index(1:4) is (/5,6,7,8/).
XcalableMP Fortran!$xmp nodes node(1:8)=**! A primary node array!$xmp nodes subnode(4)=node(5:8)
integer index(4)call xmp_get_primary_image_index&
5&(4,(/1,2,3,4/),index,xmp_desc_of(subnode))

5.5.2

Translation to the Current Image Index

1
2
3
4

5

6

xmp get image index(NUMBER,INDEX,CUR INDEX,NODE DESC)

7

Description. Translate image indices to the current image indices.

8

Class. Subroutine.

9

Arguments.

10

NUMBER shall be a scalar of type default integer. It is an INTENT(IN) argument.
INDEX shall be a rank-one array of type default integer. The size of INDEX shall begreater than or equal to the value of NUMBER. It is an INTENT(IN) argument.
The value of each element of INDEX shall be within the range one to the size of thenode array specified in NODE DESC.

11
12
13
14
15

CUR INDEX shall be a rank-one array of type default integer. The size of CUR INDEX 16
shall be greater than or equal to the value of NUMBER. It is an INTENT(OUT) 17
argument. If the current image index corresponding to the element of the node- 18
array specified in NODE DESC whose subscript order value is INDEX(i) exists, 19
CUR INDEX(i) is assigned the current image index; otherwise, CUR INDEX(i) 20
is assigned zero.
21
NODE DESC shall be a descriptor of a node array. It is an INTENT(IN) argument.
Example. In the following code fragment, the value of index(1:4) is (/1,2,3,4/).
XcalableMP Fortran!$xmp nodes node(1:8)=**integer index(4)!$xmp task on node(5:8)
call xmp_get_image_index&
5&(4,(/5,6,7,8/),index,xmp_desc_of(node))!$xmp end task

5.6

Examples of Communication between Tasks• In the following program fragment, two tasks communicate with each other with synchronization.

22

23

24

25
26

5.6. EXAMPLES OF COMMUNICATION BETWEEN TASKS

XcalableMP Fortranmodule nodes!$xmp nodes node(8)=**! A primary node array
integer, parameter :: n=2!$xmp nodes subnodeA(n)=node(1:n)! subnodeA is for taskA.
5 !$xmp nodes subnodeB(8-n)=node(n+1:8)! subnodeB is for taskB.
endmodule

10

module intertask
use nodes
real,save :: dA[*],dB[*]
endmodule

use nodes!$xmp tasks
15 !$xmp task on subnodeA
call taskA!$xmp end task!$xmp task on subnodeB
call taskB
20 !$xmp end task!$xmp end tasks
end! The taskA is invoked on subnodeA.

! The taskB is invoked on subnodeB.

subroutine taskA
use intertask:
me = this_image()! The value of me is i on subnodeA(i).
if(me.eq.1)thencall xmp_get_primary_image_index& ! The value of iyouabs
30&(1,(/1/),iyouabs,subnodeB) ! is n+1.
!$xmp
image(node)! Synchronization between
sync images(iyouabs)! node(1) and node(n+1).
call exchange(dA,dB,iyouabs)!$xmp
image(node)! Synchronization between
35
sync images(iyouabs)! node(1) and node(n+1).
endifsync all! Synchronization within subnodeA.
if(me.ne.1)dA = dA[1]sync all! Synchronization within subnodeA.
40
:
end
25

45

subroutine taskB
use intertask:
me = this_image()! The value of me is i on subnodeB(i).
if(me.eq.1)thencall xmp_get_primary_image_index& ! The value of iyouabs&(1,(/1/),iyouabs,subnodeA)! is 1.

69

70

CHAPTER 5. SUPPORT FOR THE LOCAL-VIEW PROGRAMMING

image(node)! Synchronization between
sync images(iyouabs)! node(n+1) and node(1).
call exchange(dB,dA,iyouabs)!$xmp
image(node)! Synchronization between
sync images(iyouabs)! node(n+1) and node(1).
55
endifsync all! Synchronization within subnodeB.
if(me.ne.1)dB = dB[1]sync all! Synchronization within subnodeB.
50

60

!$xmp

end

subroutine exchange(mine,yours,iput)
use nodes
real :: mine[*],yours[*]! mine and yours are always
65 !$xmp coarray on node :: mine,yours! visible on node(1:8).
yours[iput] = mine

! node(1) puts mine to yours[n+1] and! node(n+1) puts mine to yours[1].

end• In the following program fragment, two tasks communicate with each other without oneto-one synchronization.
XcalableMP Fortran!$xmp nodes node(8)=**! A primary node array:!$xmp tasks!$xmp
task on(node(1:n))
5
call taskA(n)! The taskA is invoked on node(1:n)!$xmp
end task!$xmp
task on(node(n+1:8))
call taskB(8-n)! The taskB is invoked on node(n+1:8)!$xmp
end task
10 !$xmp end tasksendsubroutine taskA(n)real,save :: yours[*],mine[*]15 !$xmp nodes subnode(n)=*! An executing node array

2025

me = this_image()if(me.eq. NUM_IMAGES())thencall xmp_get_primary_image_index(1,me,meabs) ! meabs=n.
call exchange(yours,mine,meabs,meabs+1,NUM_IMAGES())
endif
sync all! Synchronization within node(1:n).
if(me.ne.NUM_IMAGES())mine = mine[NUM_IMAGES()]sync all! Synchronization within node(1:n).
end

1
2

5.7. [C] COARRAYS IN XCALABLEMP C

subroutine taskB(m)real,save :: yours[*],mine[*]!$xmp nodes subnode(m)=*! An executing node array30

me = this_image()if(me.eq.1)thencall xmp_get_abs_image_index(1,me,meabs) ! meabs=n+1.
call exchange(yours,mine,meabs,meabs-1,NUM_IMAGES())
endif
sync all! Synchronization within node(n+1:8).
if(me.ne.1)mine = mine[1]sync all! Synchronization within node(n+1:8).
end

35

40

subroutine exchange(yours,mine,meabs,iyouabs,nnodes)
USE, INTRINSIC :: ISO_FORTRAN_ENV
real :: yours[*],mine[*]real, save :: s[*]! only for exchage.
45
TYPE(LOCK_TYPE),save :: lock[*]! for lock.
!$xmp nodes subnode(nnodes)=*! An executing node array.
!$xmp nodes node(8)=**! The coarrays s and lock are!$xmp coarray on node :: s,lock! always visible on node(1:8).
LOCK(lock[meabs])! node(n) puts yours[n] to s[n] ands[meabs] = yours! node(n+1) puts yours[n+1] to s[n+1].
UNLOCK(lock[meabs])

50

LOCK(lock[iyouabs]) ! node(n) gets s[n+1] into mine[n] andmine = s[iyouabs]! node(n+1) gets s[n] into mine[n+1].
UNLOCK(lock[iyouabs])
end

55

1

5.7

2

This section describes the coarray features for XcalableMP C.

3

5.7.1

4

Synopsis5

Coarrays are declared in XcalableMP C.

6

Syntax

7

8

9

[C]

[C] Coarrays in XcalableMP C[C] Declaration of Coarrays

data-type variable : codimensions

where codimensions is:[[int-expr]...][*]

71

72

CHAPTER 5. SUPPORT FOR THE LOCAL-VIEW PROGRAMMING

Description

1

For XcalableMP C, coarrays are declared with a colon and square bracket, where codimensions
specify the coshape of a variable.
Note that the coarray directive for defining a coarray in the XcalableMP specification 1.0(page 49) is obsolete.
Restrictions

2
3
4
5

6

• A coarray variable must have a global scope.

7

Examples8

XcalableMP C#pragma xmp nodes p[16]float x:[*];
A variable x that has a global scope is declared as a coarray.

5.7.2

9

[C] Reference of Coarrays

10

Synopsis

11

A coarray can be directly referenced or defined by any node. The target node is specified using
an extended notation in XcalableMP C.
Syntax[C]

13

14

variable : [int-expr]...

15

Description

16

A sequence of [int-expr]’s preceded by a colon in XcalableMP C determines the image index
for a coarray to be accessed. Note that the image index in XcalableMP C is 0-origin while theimage index in XcalableMP Fortran is 1-origin.
A reference of coarrays can appear in the same place as an reference of normal variables inthe base languages.
Examplesif(xmpc_this_image() == 1){
A[:] = B[:]:[0];
5 }

17
18
19
20
21

22

In the following codes, the second image ([C] image index 1/[F] image index 2) gets all valuesof array B on the first image ([C] image index 0/[F] image index 1) to array A on the second
image.
XcalableMP C
XcalableMP Fortranint A[100]:[*], B[100]:[*];
integer :: A(100)[*], B(100)[*]

5.7.3

12

if (this_image() == 2) thenA(:) = B(:)[1]
end if[C] Synchronization of Coarrays

23
24
25

26

5

27

Synopsis

28

XcalableMP C provides synchronization functions for coarrays.

29

5.8. DIRECTIVES FOR THE LOCAL-VIEW PROGRAMMING

1

2

3

Format
[C]
[C]
[C]
[C]
[C]

void
void
void
void
void

xmp sync all(int* status)
xmp sync memory(int* status)
xmp sync image(int image, int* status)
xmp sync images(int num, int* image set, int* status)
xmp sync images all(int* status)

Description

4• xmp sync all is equivalent to the sync all statement in Fortran 2008.

5• xmp sync memory is equivalent to the sync memory statement in Fortran 2008.

67• A combination of xmp sync image, xmp sync images, and xmp sync images all is equivalent to the sync images statement in Fortran 2008.

8– xmp sync image is to synchronize one image.

9– xmp sync images is to synchronize some images.

10– xmp sync images all is to synchronize all images.

11

12

Arguments• The argument status is defined with one of the follow symbolic constants.

13– XMP STAT SUCCESS

14– XMP STAT STOPPED IMAGE
If an execution of the function is successful, the status is defined using XMP STAT SUCCESS.
The condition where the status is defined using XMP STAT STOPPED IMAGE is the sameas that where the status is defined using STAT STOPPED IMAGE in Fortran 2008. Thesesymbolic constants are defined in “xmp.h.” If any other error condition occurs during theexecution of these functions, the status is defined with a value that is diﬀerent from thevalue of XMP STAT SUCCESS and XMP STAT STOPPED IMAGE.

15
16
17
18
19
20

21

22
23

73

• In xmp sync image, the variable image determines a target image index.
• In xmp sync images, the variable num is a number of target images, and the variable
image set is an array in which the target image set is defined.

24

5.8

Directives for the Local-view Programming

25

5.8.1

[F] local alias Directive

26

Synopsis

28

In XcalableMP Fortran, the local alias directive declares a local data object as an alias tothe local section of a mapped array.

29

Syntax

27

30

[F]

!$xmp local alias local-array-name => global-array-name

74

CHAPTER 5. SUPPORT FOR THE LOCAL-VIEW PROGRAMMING

Description
The LOCAL ALIAS directive associates a non-mapped array with an explicitly mapped array.
The non-mapped array is an associating local array and the explicitly mapped array is anassociated global array. The shape of the associating local array is the same as that of thenode-local portion of the associated global array including the shadow area. Each elementof the associating local array shares the same storage unit in array-element order with thatof the node-local portion of the associated global array. An associating local array and thecorresponding global array always have the same allocation status. An associating local arrayalways has the dynamic type and type parameter values of the corresponding associated global
array.
An associating local array may be a coarray. An associating local array that is a coarray is
an on-node coarray of the node array onto which the corresponding associated global array ismapped. All specifications and restrictions on coarrays are also applied to an associating local
array that is a coarray, with the exception that an associating local array is always declaredwith assumed-shape-spec-list of the same rank as the associated global array. In particular, a
processor shall ensure that an associating local array that is a coarray has the same bounds on
all the images corresponding to the node array onto which the corresponding associated global
array is mapped. The mapping attributes that are allowed for an associated global array areprocessor dependent.
Note that the base language Fortran is extended so that a deferred-shape array that is neither
an allocatable array nor an array pointer is declared if it is specified as a local-array-name inthe local alias directive.
In XcalableMP C, the address-of operator is applied to global data substitutes for the
local alias directive (see 3.4).
Restrictions• A global-array-name shall be the name of an explicitly mapped array declared in the same
scoping unit.
• A local-array-name shall be the name of a non-mapped array declared in the same scopingunit.
• A local-array-name shall not be a dummy argument.
• An associating local array shall have the declared type and type parameters of the corresponding associated global array.
• An associating local array shall be declared with assumed-shape-spec-list of the same rank
as the corresponding associated global array.
• A local-array-name shall appear in a coarray directive in the same scoping unit and thenode-name in the coarray directive shall be the name of the node array onto which theassociated global array is mapped.
• If an associated global array is a dummy argument and corresponds to an associating
local array that is a coarray, the corresponding eﬀective argument shall be an explicitly
mapped array or a subobject of an explicitly mapped array whose name appears in a
LOCAL ALIAS directive, and the corresponding associating local array shall be a coarray.
• If a dummy argument is a coarray and the corresponding ultimate argument is a coarrayappearing in a LOCAL ALIAS directive, the dummy argument shall appear in a COAR-

1

2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

25

26
27

28
29

30

31
32

33
34

35
36
37

38
39
40
41

42
43

5.8. DIRECTIVES FOR THE LOCAL-VIEW PROGRAMMING75

RAY directive with a node array corresponding to a subset of the set of images that
corresponds to the node array onto which the ultimate argument is mapped.

1
2

3

Examples4

Example 1
XcalableMP Fortran!$xmp nodes n(4)!$xmp template :: t(100)!$xmp distribute (block) onto n :: t5real :: a(100)!$xmp align (i) with t(i) :: a!$xmp shadow (1) :: areal :: b(:)
10 !$xmp local_alias b => a
The array a is distributed by block onto four nodes. The node n(2) has its local section of
25elements (a(25:50)) with shadow areas of size one on both the upper and lower bounds.
The local alias b is an array of 27 elements (b(1:27)) on n(2). The table below shows
the correspondence of each element of a to that of b on n(2).

5
6
7
8

a
lower shadow
26
27
28
...
50
upper shadow

9

10

b
1
2
3
4
...
26
27

Example 2
XcalableMP Fortran!$xmp nodes n(4)!$xmp template :: t(100)!$xmp distribute (cyclic) onto n :: t5real :: a(100)!$xmp align (i) with t(i) :: areal :: b(0:)!$xmp local_alias b => a

11
12
13
14

An array a is distributed cyclically onto four nodes. Node n(2) has its local section of
25 elements (a(2:100:4)). The lower bound of local alias b is declared to be zero. As aresult, b is an array of size 25 whose lower bound is zero (b(0:24)) on n(2). The tablebelow shows the correspondence of each element of a to that of b on n(2).

76

CHAPTER 5. SUPPORT FOR THE LOCAL-VIEW PROGRAMMING

a
2
6
10
...
98

b
0
1
2
...
24

Example 3

1

2

XcalableMP Fortran!$xmp nodes n(4)!$xmp template :: t(:)!$xmp distribute (block) onto n :: treal, allocatable :: a(:)!$xmp align (i) with t(i) :: a5real :: b(:)[*]!$xmp local_alias b => a10…

!$xmp template_fix :: t(128)
allocate (a(128))15

if (me < 4) b(4) = b(4)[me +1]
Because the global array a is an allocatable array, its local alias b is not defined when thesubroutine starts execution. b is defined when a is allocated at the allocate statement.
Note that b is declared as a coarray, and can therefore be accessed in the same manner as
a normal coarray.

5.8.2

post ConstructSynopsis

3
4
5
6

7

8

The post construct, in combination with the wait construct, specifies the point-to-point synchronization.

10

Syntax

11

[F]
[C]

!$xmp post ( nodes-ref, tag )#pragma xmp post ( nodes-ref, tag )Description
This construct ensures that the execution of statements that precede it is completed before
statements that follow the matching wait construct start are executed.
A post construct issued with the arguments of nodes-ref and tag on a node (called a postingnode) dynamically matches at most one wait construct issued with the arguments of the postingnode (unless omitted) and the same value as tag (unless omitted) by the node specified by nodesref.

9

12

13

14
15
16
17
18
19

5.8. DIRECTIVES FOR THE LOCAL-VIEW PROGRAMMING

1

2

3
4

Restriction• nodes-ref must represent one node.
• tag must be an expression of type default integer, in XcalableMP Fortran, or type int, in
XcalableMP C.

5

Example6

Example 1
XcalableMP Fortran
S1!$xmp post (p(2), 1)

7

XcalableMP Fortran!$xmp wait (p(1), 1)
S2

It is assumed that the code of the left is executed by the node p(1), while that on theright is executed by node p(2).

89The post construct on the left matches the wait construct on the right because their nodesrefs represent each other and both tags’s have the same value of one. These constructs
ensure that no statement in S2 is executed by p(2) until the execution of all statements
in S1 is completed by p(1).

10
11
12
13

14

77

Example 2
XcalableMP Fortran!$xmp wait
S3
It is assumed that this code is executed by node p(2).

15The post construct in the left code in Example 1 may match this wait construct because
both nodes-ref and tag are omitted in this wait construct.

16
17

18

5.8.3

wait Construct19

Synopsis

21

The wait construct, in combination with the post construct, specifies a point-to-point synchronization.

22

Syntax

20

23

24

25
26
27
28
29
30
31
32
33

[F]
[C]

!$xmp wait [( nodes-ref [, tag] )]#pragma xmp wait [( nodes-ref [, tag] )]Description
This construct prohibits statements that follow from being executed until the execution of allstatements preceding a matching post construct is completed on the node specified by node-ref.
A wait construct that is issued with the arguments of nodes-ref and tag on a node (called a
waiting node) dynamically matches a post construct issued with the arguments of the waiting
node and the same value as tag by the node specified by nodes-ref.
If tag is omitted, then the wait construct can match a post construct that is issued withthe arguments of the waiting node and any tag by the node specified by nodes-ref. If both tag
and nodes-ref are omitted, then the wait construct can match a post construct that is issued
with the arguments of the waiting node and any tag on any node.

78

CHAPTER 5. SUPPORT FOR THE LOCAL-VIEW PROGRAMMING

Restriction• nodes-ref must represent one node.
• tag must be an expression of type default integer, in XcalableMP Fortran, or type int, in
XcalableMP C.

5.8.4[C] lock/unlock Construct

1

2

3
4

5

Synopsis

6

The lock/unlock constructs are equivalent to the lock/unlock statements in Fortran 2008.

7

Syntax

8

[C]
[C]
[C]
[C]

#include <xmp.h>
xmp lock t lock-object [, lock-object ]...
#pragma xmp lock (lock-object) [ acquired lock (success) ] [ stat (status) ]#pragma xmp unlock (lock-object) [ stat (status) ]Please note the following points:• The type xmp lock t is defined in “xmp.h”.
• The variable lock-object is a coarray.
• The variable success is an expression of type int.
• The variable status is an expression of type int.
Description

9

10
11
12
13
14

15

The lock construct, in combination with the unlock construct, is used to control a lock-object. 16
The lock-object must be defined as a coarray to control it on a target node. The lock-object must 17
be an expression of type xmp lock t, which is an opaque object defined in “xmp.h”.
18
If the acquired lock clause is not used in the lock construct and the lock-object is locked, 19
the node stops at the lock construct until the lock-object is unlocked by a diﬀerent node. If the 20
acquired lock clause is used in the lock construct and the lock-object is locked by a diﬀerent 21
node, the node does not stop at the lock construct and the variable success is defined with 22
the value false; lock construct leaves the lock-object unchanged. If the acquired lock clause is 23
used in the lock construct and the lock-object is unlocked, the variable success is defined with 24
the value true.
25
The status is defined with one of the follow symbolic constants when executing the lock/unlock 26
construct.
27• XMP STAT SUCCESS• XMP STAT LOCKED• XMP STAT UNLOCKED• XMP STAT LOCKED OTHER IMAGE

28
29
30
31

If the execution of the lock/unlock construct is successful, the status is defined with 32
XMP STAT SUCCESS. The condition where the status is defined with XMP STAT LOCKED, XMP STAT UNLOCKED,
33
or XMP STAT LOCKED OTHER IMAGE is the same as that where the status is defined with STAT LOCKED, 34
STAT UNLOCKED, or STAT LOCKED OTHER IMAGE in Fortran 2008. These symbolic constants are 35
defined in “xmp.h”. If any other error condition occurs during the execution of these con- 36
structs, the status is defined with a value that is diﬀerent from the value of XMP STAT SUCCESS, 37
XMP STAT LOCKED, XMP STAT UNLOCKED, and XMP STAT LOCKED OTHER IMAGE.
38

5.8. DIRECTIVES FOR THE LOCAL-VIEW PROGRAMMING

1

ExampleXcalableMP C#include "xmp.h"

xmp_lock_t lock_obj:[*];
int A:[*], B;
5 #pragma xmp nodes p[2]...
#pragma xmp lock(lock_obj:[2])if(xmp_node_num() == 1){
A:[2] = B;
10}
#pragma xmp unlock(lock_obj:[2])

79

1

2

Chapter 6

Procedure Interfaces

7This chapter describes the procedure interfaces, that is, how procedures are invoked and arguments are passed, in XcalableMP.
In order to achieve high composability of XcalableMP programs, it is one of the most important requirement that XcalableMP procedures can invoke procedures written in the baselanguage with as few restrictions as possible.

8

6.1

3
4
5
6

9
10
11
12
13
14
15
16
17
18
19
20

21
22

General Rule

In XcalableMP, a procedure invocation is itself a local operation, and does not cause any communication or synchronization at runtime. Thus, a node can invoke any procedure, whether
written in XcalableMP or in the base language, at any point during the execution. There is norestriction on the characteristics of procedures invoked by an XcalableMP procedure, except fora few ones on its argument, which are explained below.
Local data in the actual or dummy argument list (referred to as a local actual argument
and a local dummy argument, respectively) are treated by the XcalableMP compiler in the samemanner as the compiler of the base language. This rule makes it possible for a local actualargument in a procedure written in XcalableMP to be associated with a dummy argument of aprocedure written in the base language.
If both an actual argument and its associated dummy argument are coarrays, they must be
declared on the same node set.
Implementation The XcalableMP compiler does not transform either local actual or dummy
arguments, so the backend compiler of the base language can treat them in its usual way.

25

The rest of this chapter specifies how global data appearing in an actual and/or dummy argument
list (referred to as a global actual argument and a global dummy argument, respectively) areprocessed by the XcalableMP compiler.

26

6.2

27

Either of the following global data can be put in the actual argument list:

23
24

Argument Passing Mechanism in XcalableMP Fortran28• an array name;29• an array element; or30• an array section that satisfies both of the following conditions:81

82

CHAPTER 6. PROCEDURE INTERFACES– its subscript list is a list of zero or more colons (“:”) followed by zero or more
int-expr’s;– the subscript of the dimension having a shadow is int-expr unless it is the last dimension.
There are two kinds of argument association for global data in XcalableMP Fortran: oneis sequence association, which is for global dummies that are an explicit-shape or assumed-size
array, and the other is descriptor association, which is for all other.

6.2.1

Sequence Association of Global Data

The concept of sequence association in Fortran is extended for global actual and dummy arguments in XcalableMP as follows.
If the actual argument is an array name or an array section that satisfies the above conditions,
it represents an element sequence consisting of the elements of its local section in Fortran’s arrayelement order on each node. In addition, if the actual argument is an element of a global data
object, it represents an element sequence consisting of the corresponding element in the local
section and each element that follows it in array element order on each node.
An global actual argument that represents an element sequence and corresponds to a global
dummy argument is sequence associated with the dummy argument if the dummy argument is an
explicit-shape or assumed-size array. According to this (extended) rule of sequence association,
each element of the element sequence represented by the global actual argument is associatedwith the element of the local section of the global dummy argument that has the same positionin array element order.
Sequence association is the default rule of association for global actual arguments, and itis therefore applied unless it is obvious from the interface of the invoked procedure that thecorresponding dummy argument is neither an explicit-shape nor assumed-size array.
Implementation In order to implement sequence association, the name, a section, or anelement of global data appearing as an actual argument is treated by the XcalableMP compiler
as the base address of its local section on each node, and the global data appearing as thecorresponding dummy argument is initialized at runtime so that it is composed of the local
sections, each of which starts from the address received as the argument. On a node that does
not have the local section corresponding to the actual argument, an unspecified value (e.g. null)is received.
Such an implementation implies that in many cases, in order to associate properly a global
actual argument with the global dummy argument, their mappings (including their shadow
attributes) must be identical.
ExamplesExample 1 Both the actual argument a and the dummy argument x are global explicit-shapearrays, and therefore, a is sequence associated with x.
The base address of the local section of a is passed between these subroutines on eachnode. Each of the local sections of x starts from the received address (Figure 6.1).
XcalableMP Fortran
subroutine xmp_sub1!$xmp nodes p(4)!$xmp template t(100)!$xmp distribute t(block) onto p

1
2
3
4

5
6
7

8

9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

25
26
27
28
29
30
31
32
33
34

35

36
37
38
39

6.2. ARGUMENT PASSING MECHANISM IN XCALABLEMP FORTRAN

583

real a(100)!$xmp align a(i) with t(i)!$xmp shadow a(1:1)
call xmp_sub2(a)end subroutine

10

subroutine xmp_sub2(x)!$xmp nodes p(4)!$xmp template t(100)!$xmp distribute t(block) onto p
15
real x(100)!$xmp align x(i) with t(i)!$xmp shadow x(1:1)
...

p(1)

p(2)

p(3)

p(4)

a
1

100

xmp_sub1

sequence association

xmp_sub2
1

100

x
p(1)

p(2)

p(3)

p(4)

Figure 6.1: Sequence association with a global dummy argument.

1
2
3
4
5

Example 2 The actual argument a is a global explicit-shape array, and the dummy argument
x is a local explicit-shape. Sequence association is also applied in this case.
The caller subroutine xmp sub1 passes the base address of the local section of a on eachnode, and the callee f sub2 receives it and initializes x with the storage starting from it(Figure 6.2).
XcalableMP Fortran
subroutine xmp_sub1!$xmp nodes p(4)!$xmp template t(100)!$xmp distribute t(block) onto p5real a(100)!$xmp align a(i) with t(i)!$xmp shadow a(1:1)
n = 1 + 100/4 + 1
call f_sub2(a,n)
10
end subroutine
Fortransubroutine f_sub2(x,n)
real x(n)...

84

CHAPTER 6. PROCEDURE INTERFACES

p(1)

p(2)

p(3)

p(4)

a
1

25

26

xmp_sub1

50

51

75

76

100

27

1

27

sequence association

f_sub2
1

27

1

27

1

x
p(1)

p(2)

p(3)

p(4)

Figure 6.2: Sequence association with a local dummy argument.

Example 3 The actual argument a(:,1) is a contiguous section of a global data object, andthe dummy argument x is a local explicit-shape array. Sequence association is applied inthis case, but only the node p(1) owns the section. Hence, f sub2 is invoked only by p(1)(Figure 6.3).
XcalableMP Fortran
subroutine xmp_sub1!$xmp nodes p(4)!$xmp template t(100,100)!$xmp distribute t(*,block) onto p5real a(100,100)!$xmp align a(i,j) with t(i,j)!$xmp shadow a(0,1:1)
n = 100!$xmp task on p(1)10call f_sub2(a(:,1),n)!$xmp end task
end subroutine
Fortransubroutine f_sub2(x,n)
real x(n)
...

1
2
3
4

6.2. ARGUMENT PASSING MECHANISM IN XCALABLEMP FORTRAN

1

25

26

p(1)

50

51

p(2)

75

85

76 100

p(3)

p(4)

a

xmp_sub1
f_sub2

sequence
associationxp(1)Figure 6.3: Sequence association of a section of a global data object as an actual argument witha local dummy argument.

1
2
3

Example 4 The actual argument a(1) is an element of the global data, and the dummy argument x is a local explicit-shape array. Sequence association is applied in this case, but
only the node p(1) owns the element. Hence, f sub2 is invoked only by p(1)(Figure 6.4).
XcalableMP Fortran
subroutine xmp_sub1!$xmp nodes p(4)!$xmp template t(100)!$xmp distribute t(block) onto p5real a(100)!$xmp align a(i) with t(i)!$xmp shadow a(1:1)
n = 100/4!$xmp task on p(1)10call f_sub2(a(1),n)!$xmp end task
end subroutine
Fortransubroutine f_sub2(x,n)
real x(n)...

7Example 5 Even if either the global actual or dummy argument has a full shadow, the rule ofsequence association is the same in principle. Hence, the base address of the local sectionof a is passed between these subroutines on each node, and each local section of x starts
from the received address (Figure 6.5).

8

6.2.2

4
5
6

9
10
11
12
13

Descriptor Association of Global Data

When the actual argument is a global data object, and it is obvious from the interface of
the invoked procedure that the corresponding dummy argument is neither an explicit-shape
nor assumed-size array, the actual argument is descriptor associated with the dummy argument.
According to the descriptor association rule, the dummy argument inherits its shape and storage
from the actual argument.

86

CHAPTER 6. PROCEDURE INTERFACES

p(1)

p(2)

p(3)

p(4)

a
1

xmp_sub1
f_sub2

25

26

50

51

75

76

100

sequence
association
1

25

x
p(1)

Figure 6.4: Sequence association of an element of a global data object as an actual argumentwith a local dummy argument.

1

100

p(1)
p(2)

a
p(3)
p(4)

xmp_sub1

sequence association

xmp_sub2

p(1)
p(2)

x
p(3)
p(4)
1

100

Figure 6.5: Sequence association with a global dummy argument that has a full shadow.

Implementation In order to implement the descriptor association, a global actual argument
is treated by the XcalableMP compiler:• as if it were the global-data descriptor of the actual array, which is an internal data structure
managed by the XcalableMP runtime system to store information on a global data object(see 7.1.1) if the dummy is a global data object; or• as it is an array representing the local section of the actual array, which is to be processedby the backend Fortran compiler in the same manner as usual data if the dummy is a local
data object.

For the first case, a global dummy is initialized at runtime with a copy of the global-data
descriptor received.
When an actual argument is descriptor associated with the dummy argument and their
mappings are not identical, the XcalableMP runtime system may detect and report the error.

1
2

3
4
5

6
7
8

9
10
11
12

6.2. ARGUMENT PASSING MECHANISM IN XCALABLEMP FORTRAN

1

2
3
4
5

87

ExamplesExample 1 There is an explicit interface of the subroutine xmp sub2 specified by an interface
block in the subroutine xmp sub1, from which it is found that the dummy argument x
is a global assumed-shape array. Therefore, the global actual argument a is descriptor
associated with the global dummy argument x.
It is the global-data descriptor of a that is passed between these subroutines. The dummy
argument x is initialized by the XcalableMP runtime system on the basis of the information
extracted from the descriptor received (Figure 6.6).
XcalableMP Fortran
subroutine xmp_sub1

678!$xmp nodes p(4)!$xmp template t(100)
5 !$xmp distribute t(block) onto preal a(100)!$xmp align a(i) with t(i)!$xmp shadow a(1:1)
interfacesubroutine xmp_sub2(x)!$xmp nodes p(4)!$xmp template t(100)!$xmp distribute t(block) onto p
15
real x(:)!$xmp align x(i) with t(i)!$xmp shadow a(1:1)
end subroutine xmp_sub2
end interface
10

20

call xmp_sub2(a)end subroutine
subroutine xmp_sub2(x)!$xmp nodes p(4)!$xmp template t(100)!$xmp distribute t(block) onto preal x(:)
30 !$xmp align x(i) with t(i)!$xmp shadow a(1:1)
...
25

9
10
11
12
13
14
15

Example 2 There is the explicit interface of the subroutine f sub2, which is written in Fortran,
specified by an interface block in the subroutine xmp sub1, and the dummy argument x isa local (i.e., non-mapped) assumed-shape array. Therefore, the global actual argument a
is descriptor associated with the local dummy argument x.
The global actual argument is replaced with its local section by the XcalableMP compiler,
and the association of the local section with the dummy argument is to be processed by
the backend Fortran compiler in the same manner as usual data (Figure 6.7).

88

CHAPTER 6. PROCEDURE INTERFACES

p(1)

p(2)

p(3)

p(4)

a
1

100

global-data descriptor

xmp_sub1

descriptor association

xmp_sub2
global-data descriptor
1

100

x
p(1)

p(2)

p(3)

p(4)

Figure 6.6: Descriptor association with a global dummy argument.

XcalableMP Fortran
subroutine xmp_sub1!$xmp nodes p(4)!$xmp template t(100)
5 !$xmp distribute t(block) onto preal a(100)!$xmp align a(i) with t(i)!$xmp shadow a(1:1)
10interfacesubroutine f_sub2(x)real x(:)
end subroutine f_sub2
end interface

15

call f_sub2(a)end subroutine
Fortransubroutine f_sub2(x)real x(:)...

6.3

Argument-Passing Mechanism in XcalableMP C

When an actual argument is a global data object, it is passed by the address of its local section.
When a dummy argument is a global data object , an address is received and used as the baseaddress of each of its local sections.
Implementation The name of a global data object appearing as an actual argument is treatedby the XcalableMP compiler as the pointer to the first element of its local section on each

1

2
3
4

5
6

6.3. ARGUMENT-PASSING MECHANISM IN XCALABLEMP C

p(1)

p(2)

p(3)

89

p(4)

a
1

xmp_sub1

25

26

50

51

75

100

76

descriptor association by the backend compiler

f_sub2

1

27

1

27

1

27

1

27

x
p(1)

p(2)

p(3)

p(4)

Figure 6.7: Descriptor association with a local dummy argument.

10

node. On a node onto which no part of the global data object is mapped, the pointer is set
to an unspecified value (e.g., null). Note that an element of a global data object in the actualargument list is treated in the same manner as those in other usual statements because an arrayelement is passed by value, as in C.
The name of a global data object appearing as a dummy argument is treated by the XcalableMP compiler as the pointer to the first element of its local section on each node. As a result,
it is initialized at runtime so that it is composed of the local sections on the executing nodes.
Such an implementation implies that in many cases, in order to pass properly a global actualargument to the corresponding global dummy argument, their mappings (including their shadow
attributes) must be identical.

11

Examples

1
2
3
4
5
6
7
8
9

12
13

Example 1 The global actual argument a is treated by the XcalableMP compiler as the pointerto the first element of its local section, which is passed to the callee, on each node.
The global dummy argument x is initialized so that each of its local sections starts fromthe address held by the received pointer (Figure 6.8).

14
15

XcalableMP C
void xmp_func1(){#pragma xmp nodes p[4]#pragma xmp template t[100]5 #pragma xmp distribute t[block] onto p
float a[100];#pragma xmp align a[i] with t[i]#pragma xmp shadow a[1:1]
xmp_func2(a);10}void xmp_func2(float x[100]){
15 #pragma xmp nodes p[4]#pragma xmp template t[100]#pragma xmp distribute t[block] onto p

90

CHAPTER 6. PROCEDURE INTERFACES#pragma xmp align x[i] with t[i]#pragma xmp shadow a[1:1]
20
...

p[0]

p[1]

p[2]

p[3]

a
0

99

xmp_func1

pass by address

xmp_func2
0

99

x
p[0]

p[1]

p[2]

p[3]

Figure 6.8: Passing to a global dummy argument.

Example 2 The global actual argument a is treated by the XcalableMP compiler as the pointerto the first element of its local section, which is passed to the callee, on each node.
The local dummy argument x on each node starts from the address held by the received
pointer (Figure 6.9).

1
2
3
4

XcalableMP C
void xmp_func1(){#pragma xmp nodes p[4]#pragma xmp template t[100]5 #pragma xmp distribute t[block] onto p
float a[100];#pragma xmp align a[i] with t[i]#pragma xmp shadow a[1:1]
c_func2(a);10}Cvoid c_func2(float x[27]){...

Example 3 The actual argument a[0] is an element of a global data object, and the dummy
argument x is a scalar, where the normal argument-passing rule of C for variables of a basic
type (i.e., “pass-by-value”) is applied. However, only the node p[0] owns the element.
Hence, c func2 is invoked only by p[0] (Figure 6.10).
XcalableMP C
void xmp_func1(){#pragma xmp nodes p[4]#pragma xmp template t[100]5 #pragma xmp distribute t[block] onto p

5
6
7
8

6.3. ARGUMENT-PASSING MECHANISM IN XCALABLEMP C

p[0]

p[1]

p[2]

91

p[3]

a
0

24

25

49

xmp_func1

50

74

75

99

26

0

26

pass by addressc_func2
0

26

0

26

0

x
p[0]

p[1]

p[2]

p[3]

Figure 6.9: Passing to a local dummy argument.

float a[100];#pragma xmp align a[i] with t[i]#pragma xmp shadow a[1:1]10#pragma xmp task on p[0]c_func2(a[0]);}Cvoid c_func2(float x)
{
...

p[0]

p[1]

p[2]

p[3]

a
0

24

25

49

50

74

75

99

xmp_func1 pass byvalue
c_func2xp[0]Figure 6.10: Passing an element of a global data object as an actual argument to a local dummy
argument.

1

2

Chapter 7

Intrinsic and Library Procedures

5This specification defines various procedures that perform a system inquiry, synchronization,
computation, etc. The procedures are provided as intrinsic procedures in XcalableMP Fortran,
and as library procedures in XcalableMP C.

6

7.1

Intrinsic Functions

7

7.1.1

xmp desc of

8

Format

3
4

9

[F]

type(xmp desc)

xmp desc of(xmp entity)

11

Note that xmp desc of is an intrinsic function in XcalableMP Fortran or a built-in operatorin XcalableMP C. For the xmp desc of operator, refer to section 3.6.

12

Synopsis1018

xmp desc of returns a descriptor to retrieve information of the specified global array, template,or node array. The resulting descriptor can be used as an input argument of mapping inquiry
functions.
The type of descriptors, type(xmp desc), in XcalableMP Fortran, and xmp desc t, in XcalableMP C, is implementation-defined, and it is defined in a Fortran module named xmp lib or
a Fortran include file named xmp lib.h.

19

Arguments

13
14
15
16
17

21

The argument or operand xmp entity is the name of either a global array, a template, or a nodearray.

22

7.2

20

System Inquiry Functions

23• xmp all node num

24• [C] xmpc all node num

25• xmp all num nodes

26• xmp node num

27• [C] xmpc node num
93

94

CHAPTER 7. INTRINSIC AND LIBRARY PROCEDURES• [C] xmpc this image

1• xmp num nodes2• xmp num images

3• xmp wtime

4• xmp wtick

5

7.2.1

xmp all node num

6Format[F][C]

7

integer function
int

xmp all node num()xmp all node num(void)

Synopsis

89The xmp all node num routine returns the node number, within the entire node set, of the nodethat calls xmp all node num.

10
11

Arguments

12

none.

13

7.2.2

[C] xmpc all node num

Format
[C]

14

15

int

xmpc all node num(void)

Synopsis

16

17

The xmpc all node num routine returns the node number −1, within the entire node set, of thenode that calls xmpc all node num.

18
19

Arguments

20

none.

21

7.2.3

xmp all num nodes

Format[F][C]

integer function
int

22

23

xmp all num nodes()xmp all num nodes(void)

24

Synopsis25

The xmp all num nodes routine returns the number of nodes in the entire node set.

26

Arguments

27

none.

28

7.2. SYSTEM INQUIRY FUNCTIONS

1

7.2.4

2

Format

3

4

[F]
[C]

95

xmp node num

integer function
int

xmp node num()
xmp node num(void)

Synopsis6

The xmp node num routine returns the node number, within the current executing node set, ofthe node that calls xmp node num.

7

Arguments

8

none.

9

7.2.5

10

Format

5

11

12

[C]

[C] xmpc node num

int

xmpc node num(void)

Synopsis14

The xmpc node num routine returns the node number −1, within the current executing node set,of the node that calls xmpc node num.

15

Arguments

16

none.

17

7.2.6

18

Format

13

19

[C]

[C] xmpc this image

int

xmpc this image(void)20

Synopsis21

The xmpc this image routine is identical to the xmpc node num routine.

22

Arguments

23

none.

24

7.2.7

25

Format

26

[F]
[C]

xmp num nodes

integer function
int

xmp num nodes()xmp num nodes(void)

27

Synopsis28

The xmp num nodes routine returns the number of the executing nodes.

96

CHAPTER 7. INTRINSIC AND LIBRARY PROCEDURES

Arguments

1

none.

2

7.2.8

xmp num images

3

Format[F][C]

4

integer function
int

xmp num images()
xmp num images(void)

5

Synopsis6

The xmp num images routine is identical to the xmp num nodes routine.

7

Arguments

8

none.

9

7.2.9

xmp wtime10

Format[F][C]

11

double precision function
doublexmp wtime()
xmp wtime(void)

Synopsis

12

13

The xmp wtime routine returns elapsed wall-clock time in seconds since some time in the past.
The “time in the past” is guaranteed not to change during the life of the process. There is norequirement that diﬀerent nodes return “the same time.”

14
15
16

Arguments

17

none.

18

7.2.10xmp wtick

19

Format[F][C]

double precision function
double20

xmp wtick()
xmp wtick(void)

Synopsis
The xmp wtick routine returns the resolution of the timer used by xmp wtime. It returns adouble-precision value that is equal to the number of seconds between successive clock ticks.

21

22

23
24

Arguments

25

none.

26

7.3. [C] EXECUTION CONTROL FUNCTIONS

1

7.3[C] Execution Control Functions

2

7.3.1

xmp exit

3

Format

4

5

[C]

void

97

xmp exit(int status)

Synopsis10

xmp exit terminates an XcalableMP program normally. The value of the argument status
returned to the host environment is the same as that by the exit standard library function of
the base language.
xmp exit must be collectively invoked by every node in the entire node set; otherwise, thebehavior is undefined.

11

Arguments

12

The argument status is a status code to be returned to the host environment.

13

7.4

Synchronization Functions

14

7.4.1

xmp test async[F]logical function
integer

xmp test async(async id)
async id[C]

int

xmp test async(int async id)

6
7
8
9

15

16

Synopsis19

The xmp test async routine returns .true. in XcalableMP Fortran, or 1 in XcalableMP C, if
an asynchronous communication specified by the argument async id is complete; otherwise, it
returns .false. or 0.

20

Arguments

17
18

22

The argument async id is an integer expression that specifies an asynchronous communication
initiated by a global communication construct with the async clause.

23

7.5

Memory Allocation Functions

24

7.5.1

[C] xmp malloc

25

void*

xmp malloc(xmp desc t d, size t size0, size t size1, ...)

26

Synopsis

21

27
28
29

The xmp malloc routine allocates storage for the local section of a global array of size size0×size1× . . .
that is associated with the descriptor specified by d, and returns the pointer to it on each node.
For an example of xmp malloc, refer to section 3.5.

98

CHAPTER 7. INTRINSIC AND LIBRARY PROCEDURES

Arguments

1• d is the descriptor associated with the pointer to a global array to be allocated.

2• size0, size1, ... are the sizes of the dimensions of the global array to be allocated.

3

7.6

Mapping Inquiry Functions

All mapping inquiry functions are specified as integer functions. These functions return zero
upon success and an implementation-defined negative integer value upon failure.

7.6.1

xmp nodes ndims

[C]

5
6

7

Format[F]

48integer function
type(xmp desc)
integerint

xmp nodes ndims(d, ndims)dndims
xmp nodes ndims(xmp desc t d, int *ndims)

9

Synopsis10

The xmp nodes ndims function provides the rank of the target node array.

11

Input Arguments

12• d is a descriptor of a node array.
Output Arguments

14• ndims is the rank of the node array specified by d.

7.6.2

xmp nodes index[C]

integer function
type(xmp desc)
integer
integer
int

15

16

Format[F]

13

17

xmp nodes index(d, dim, index)
d
dim
index
xmp nodes index(xmp desc t d, int dim, int *index)

Synopsis
The xmp nodes index function provides the indices of the executing node in the target nodearray.
Input Arguments

18

19

20
21

22

• d is a descriptor of a node array.

23• dim is the target dimension of the node array.

24

Output Arguments• index is an index of the target dimension of the node array specified by d.

25

26

7.6. MAPPING INQUIRY FUNCTIONS

1

7.6.3

2

Format[F]

3[C]

xmp nodes size

integer function
type(xmp desc)
integer
integerint

xmp nodes size(d, dim, size)
d
dim
size
xmp nodes size(xmp desc t d, int dim, int *size)

4

Synopsis5

The xmp nodes size function provides the size of each dimension of the target node array.

6

Input Arguments

7• d is a descriptor of a node array.

8• dim is the target dimension of the node array.

9

10
11

Output Arguments• size is the extent of the target dimension of the node array specified by
t d.

12

7.6.4

13

Format
[F]

14

[C]
15

99

xmp nodes attr

integer function
type(xmp desc)
integerint

xmp nodes attr(d, attr)d
attr
xmp nodes attr(xmp desc t d, int *attr)

Synopsis21

The xmp nodes attr function provides the attribute of the target node array. The output valueof the argument attr is one of:
XMP ENTIRE NODES(Entire nodes)
XMP EXECUTING NODES(Executing nodes)
XMP EQUIVALENCE NODES (Equivalence nodes)
These are named constants that are defined in module xmp lib and in the include file
xmp lib.h in XcalableMP Fortran, and symbolic constants that are defined in the header file
xmp.h in XcalableMP C.

22

Input Arguments

16
17

18

19
20

23

24

25

• d is a descriptor of a node array.
Output Arguments• attr is an attribute of the target node array specified by d.

100

CHAPTER 7. INTRINSIC AND LIBRARY PROCEDURES

7.6.5

xmp nodes equiv

1Format[F][C]

2

integer function
type(xmp desc)
type(xmp desc)
integer
integer
integerint

xmp nodes equiv(d, dn, lb, ub, st)
d
dn
lb(*)
ub(*)
st(*)
xmp nodes equiv(xmp desc t d, xmp desc t *dn,
int lb[], int ub[], int st[])Synopsis4

The xmp nodes equiv function provides the descriptor of a node array as well as a subscript listthat represents a node set that is assigned to the target node array in the nodes directive. Thisfunction returns with a failure when the target node array is not declared as equivalenced.
Input Arguments

Output Arguments• lb is a one-dimensional integer array the extent of which must be more than or equal to
the rank of the referenced node array. The i-th element of lb is set to the lower bound of
the i-th subscript of the node reference unless it is “*”, or to undefined otherwise.
• ub is a one-dimensional integer array the extent of which must be more than or equal to
the rank of the referenced node array. The i-th element of ub is set to the upper bound of
the i-th subscript of the node reference unless it is “*”, or to undefined otherwise.
• st is a one-dimensional integer array the extent of which must be more than or equal to
the rank of the referenced node array. The i-th element of st is set to the stride of thei-th subscript of the node reference unless it is “*”, or to zero otherwise.

xmp template fixed

Format[C]

integer function
type(xmp desc)
logical
int

6
7

9

10

• dn is the descriptor of the referenced node array if the target node array is declared as
equivalenced; otherwise, dn is set to undefined.

[F]

5

8

• d is a descriptor of a node array.

7.6.6

3

11
12

13
14
15

16
17
18

19
20
21

22

23

xmp template fixed(d, fixed)dfixed
xmp template fixed(xmp desc t d, int *fixed)

Synopsis
The xmp template fixed function provides the logical value that shows whether the templateis fixed or not.

24

25

26
27

7.6. MAPPING INQUIRY FUNCTIONS

1

2

3

4
5
6

Input Arguments• d is a descriptor of a template.
Output Arguments• fixed is set to true in XcalableMP Fortran and an implementation-defined non-zero integer
value in XcalableMP C if the template specified by d is fixed; otherwise, it is set to false
in XcalableMP Fortran and zero in XcalableMP C.

7

7.6.7

8

Format[F]

9[C]

xmp template ndims
integer function
type(xmp desc)
integerint

xmp template ndims(d, ndims)d
ndims
xmp template ndims(xmp desc t d, int *ndims)

10

Synopsis11

The xmp template ndims function provides the rank of the target template.

12

Input Arguments

13

14

15

• d is a descriptor of a template.
Output Arguments• ndims is the rank of the template specified by d.

16

7.6.8

17

Format
[F]

18

[C]
19

101

xmp template lboundinteger function
type(xmp desc)
integer
integerint

xmp template lbound(d, dim, lbound)ddimlboundxmp template lbound(xmp desc t d, int dim, int *lbound)

Synopsis21

The xmp template lbound function provides the lower bound of each dimension of the template.
This function returns with a failure when the lower bound is not fixed.

22

Input Arguments

20

23• d is a descriptor of a template.

24• dim is the target dimension of the template.

25

26
27

Output Arguments• lbound is the lower bound of the target dimension of the template specified by d. Whenthe lower bound is not fixed, it is set to undefined.

102

CHAPTER 7. INTRINSIC AND LIBRARY PROCEDURES

7.6.9

xmp template uboundFormat
[F]

[C]

1

2

integer function
type(xmp desc)
integer
integerint

xmp template ubound(d, dim, ubound)ddimuboundxmp template ubound(xmp desc t d, int dim, int *ubound)

Synopsis3

4

The xmp template ubound function provides the upper bound of each dimension of the template.
This function returns with a failure when the upper bound is not fixed.
Input Arguments

5
6

7• d is a descriptor of a template.

8• dim is the target dimension of the template.

9Output Arguments

10• ubound is an upper bound of the target dimension of the template specified by d. Whenthe upper bound is not fixed, it is set to undefined.

7.6.10xmp dist format[C]

integer function
type(xmp desc)
integer
integer
int

12

13

Format[F]

11

14

xmp dist format(d, dim, format)ddimformatxmp dist format(xmp desc t d, int dim, int *format)

Synopsis
The xmp dist format function provides the distribution format of a dimension of a template.
The output value of the argument format is one of:
XMP NOT DISTRIBUTED (not distributed)
XMP BLOCK(block distribution)
XMP CYCLIC(cyclic distribution)
XMP GBLOCK(gblock distribution)
These symbolic constants are defined in “xmp.h”.
Input Arguments

15

16

17
18

19

20

21

• d is a descriptor of a template.

22• dim is the target dimension of the template.

23

Output Arguments• format is a distribution format of the target dimension of the template specified by d.

24

25

7.6. MAPPING INQUIRY FUNCTIONS

1

7.6.11

2

Format[F]

xmp dist blocksize

integer function
type(xmp desc)
integer
integer
int

3

[C]

xmp dist blocksize(d, dim, blocksize)ddimblocksize
xmp dist blocksize(xmp desc t d, int dim, int *blocksize)

4

Synopsis5

The xmp dist blocksize function provides the block width of a dimension of a template.

6

Input Arguments

7• d is a descriptor of a template.

8• dim is the target dimension of the template.

910Output Arguments• blocksize is the block width of the target dimension of the template specified by d.

11

7.6.12

12

Format
[F]

13

[C]
14

103

xmp dist gblockmap

integer function
type(xmp desc)
integer
integerint

xmp dist gblockmap(d, dim, map)ddim
map(N)
xmp dist gblockmap(xmp desc t d, int dim, int map[])

Synopsis18

The xmp dist gblockmap function provides the mapping array of the gblock distribution.
When the dim-th dimension of the global array is distributed by gblock and its mapping
array is fixed, this function returns zero; otherwise, it returns an implementation-defined negative
integer value.

19

Input Arguments

15
16
17

20

• d is a descriptor of a template.

21• dim is the target dimension of the template.

22

Output Arguments24• map is a one-dimensional integer array the extent of which is more than the size of thecorresponding dimension of the node array onto which the template is distributed.

25

The i-th element of map is set to the value of the i-th element of the target mapping array.

23

104

CHAPTER 7. INTRINSIC AND LIBRARY PROCEDURES

7.6.13xmp dist nodes

1

Format[F][C]

2

integer function
type(xmp desc)
type(xmp desc)int

xmp dist nodes(d, dn)ddn
xmp dist nodes(xmp desc t d, xmp desc t *dn)

Synopsis4

The xmp dist nodes function provides the descriptor of the node array onto which a template
is distributed.
Input Arguments

Output Arguments

xmp dist axis

[C]

8

10

11

Formatinteger function
type(xmp desc)
integer
integerint

6

9• dn is the descriptor of the node array.

[F]

5

7

• d is a descriptor of a template.

7.6.143

12

xmp dist axis(d, dim, axis)
d
dim
axis
xmp dist axis(xmp desc t d, int dim, int *axis)

Synopsis
The xmp dist axis function provides the dimension of the node array onto which a dimensionof a template is distributed. This function returns with a failure when the dimension of thetemplate is not distributed.
Input Arguments

13

14

15
16
17

18

• d is a descriptor of a template.

19• dim is the target dimension of the template.

20Output Arguments• axis is a dimension of the node array onto which the target dimension of the template
specified by d is distributed. When the dimension of the template is not distributed, it is
set to undefined.

21

22
23
24

7.6. MAPPING INQUIRY FUNCTIONS

1

7.6.152

Format
[F]

3

[C]
4

105

xmp align axis

integer function
type(xmp desc)
integer
integer
int

xmp align axis(d, dim, axis)
d
dim
axis
xmp align axis(xmp desc t d, int dim, int *axis)

Synopsis7

The xmp align axis function provides the dimension of the template with which a dimension ofa global array is aligned. This function returns with a failure when the dimension of the global
array is not aligned.

8

Input Arguments

56

9• d is a descriptor of a global array.

10• dim is the target dimension of the global array.

11

12
13
14

Output Arguments• axis is the dimension of the template with which the target dimension of the global arrayspecified by d is aligned. When the dimension of the global array is not aligned, or iscollapsed, it is set to undefined.

15

7.6.16

16

Format
[F]

17

[C]
18

xmp align offset

integer function
type(xmp desc)
integer
integer
int

xmp align offset(d, dim, offset)ddim
offsetxmp align offset(xmp desc t d, int dim, int *offset)

Synopsis20

The xmp align offset function provides the align oﬀset for a dimension of a global array. Thisfunction returns with a failure when there is no oﬀset.

21

Input Arguments

19

22• d is a descriptor of a global array.

23• dim is the target dimension of the global array.

24

25
26

Output Arguments• offset is the align oﬀset for the target dimension of the global array specified by d. When
there is no oﬀset, it is set to undefined.

106

CHAPTER 7. INTRINSIC AND LIBRARY PROCEDURES

7.6.17

xmp align replicatedFormat
[F]

[C]

1

2

integer function
type(xmp desc)
integerlogicalint

xmp align replicated(d, dim, replicated)ddim
3
replicatedxmp align replicated(xmp desc t d, int dim, int *replicated)

Synopsis4

The xmp align replicated function provides the logical value that shows whether or not thedimension of the template with which a global array is aligned is replicated.
Input Arguments

5
6

7• d is a descriptor of a global array.

8• dim is the target dimension of the template with which the global array is aligned.

9Output Arguments

10• replicated is a logical scalar, which is set to true if the dimension of the template is
replicated.

7.6.18xmp align template

Format[F][C]

xmp align template(d, dt)
ddt
xmp align template(xmp desc t d, xmp desc t *dn)

Synopsis

Input Arguments

15

Output Arguments

18

20

21

• dt is the descriptor of the template.

xmp array ndims

22

23

Formatinteger function
type(xmp desc)
integer
int

17

19

• d is a descriptor of a global array.

[C]

13

16

The xmp align template function provides the descriptor of the template with which a global
array is aligned.

[F]

12

14

integer function
type(xmp desc)
type(xmp desc)
int

7.6.191124

xmp array ndims(d, ndims)d
ndims
xmp array ndims(xmp desc t d, int *ndims)

25

7.6. MAPPING INQUIRY FUNCTIONS

1

Synopsis2

The xmp array ndims function provides the rank of a global array.

3

Input Arguments

4

5

6

• d is a descriptor of a global array.
Output Arguments• ndims is the rank of the global array specified by d.

7

7.6.208Format
[F]

9

[C]
10

107

xmp array lshadow

integer function
type(xmp desc)
integer
integer
int

xmp array lshadow(d, dim, lshadow)ddimlshadow
xmp array lshadow(xmp desc t d, int dim, int *lshadow)

Synopsis12

The xmp array lshadow function provides the size of the lower shadow of a dimension of a global
array.

13

Input Arguments

1114• d is a descriptor of a global array.

15• dim is the target dimension of the global array.

16

17
18

Output Arguments• lshadow is the size of the lower shadow of the target dimension of the global array specified
by d.

19

7.6.2120

Format
[F]

21

[C]
22

23
24

xmp array ushadow

integer function
type(xmp desc)
integer
integer
int

xmp array ushadow(d, dim, ushadow)ddimushadow
xmp array ushadow(xmp desc t d, int dim, int *ushadow)

Synopsis
The xmp array ushadow function provides the size of the upper shadow of a dimension of a
global array.

108

CHAPTER 7. INTRINSIC AND LIBRARY PROCEDURES

Input Arguments

1• d is a descriptor of a global array.

2• dim is the target dimension of the global array.

3

Output Arguments

4• ushadow is the size of the upper shadow of the target dimension of the global array specified
by d.

7.6.22

xmp array lboundFormat
[F]

[C]

5
6

7

8

integer function
type(xmp desc)
integer
integer
int

xmp array lbound(d, dim, lbound)ddimlboundxmp array lbound(xmp desc t d, int dim, int *lbound)

Synopsis

910The xmp array lbound function provides the lower bound of a dimension of a global array. Thisfunction returns with a failure when the lower bound is not fixed.
Input Arguments

11
12

13

• d is a descriptor of a global array.

14• dim is the target dimension of the global array.

15

Output Arguments

16• lbound is the lower bound of the target dimension of the global array specified by d. Whenthe lower bound is not fixed, it is set to undefined.

7.6.23

xmp array uboundFormat[F][C]

integer function
type(xmp desc)
integer
integer
int

17
18

19

20

xmp array ubound(d, dim, ubound)ddimuboundxmp array ubound(xmp desc t d, int dim, int *ubound)

Synopsis
The xmp array ubound function provides the upper bound of a dimension of a global array. Thisfunction returns with a failure when the upper bound is not fixed.
Input Arguments

21

22

23
24

25

• d is a descriptor of a global array.

26• dim is the target dimension of the global array.

27

7.7. [F] ARRAY INTRINSIC FUNCTIONS OF THE BASE LANGUAGE

1

2
3

4

5
6
7
8

9

Output Arguments• ubound is the upper bound of the target dimension of the global array specified by d.
When the upper bound is not fixed, it is set to undefined.

7.7• Inquiry functions
The inquiry functions with a global array or its subobject being an argument are regarded
as inquiries about the global array, and return its “global” properties as if it were notdistributed.

11
12• Elemental functions
The result of the elemental functions with a global array or its subobject being an argument
has the same shape and mapping as the argument. Note that such a reference of theseelemental functions is in eﬀect limited to be in the array construct.

14
15
16

17

• Transformational functions
It is unspecified how the transformational functions work when a global array or its subobject appears as an argument. A processor shall detect such a reference of these functions
and issue a warning message for it. Some intrinsic transformational subroutines are definedin section 7.9 as alternatives to these transformational functions.

18
19
20
21

22

[F] Array Intrinsic Functions of the Base Language

The array intrinsic functions of the base language Fortran are classified into three classes: inquiry, elemental, and transformational.
This section specifies how these functions work in the XMP/F programs when a global arrayappears as an argument.

10

13

109

7.8

[C] Built-in Elemental Functions

32

Some built-in elemental functions that can operate each element of array arguments are definedin XcalableMP C. Such a built-in function accepts one or more array sections as its arguments
and returns an array-valued result having the same shape and mapping as the argument. The
values of the elements of the result are the same as what would have been obtained if the scalarfunction of the C standard library had been applied separately to the corresponding elementsof each array argument.
These functions may appear on the right-hand side of an array assignment statement, andit should be preceded by the array directive if the array section is distributed.
Table 7.1 shows the list of built-in elemental functions in XcalableMP C. Their elementwise
behavior is the same as those of the corresponding functions in the C standard library.

33

7.9

23
24
25
26
27
28
29
30
31

34
35
36
37
38

Intrinsic/Built-in Transformational ProceduresSome intrinsic/built-in transformational procedures are defined for the non-elemental operationsof arrays.
Note that each “array argument” of the following procedures must be an array name or anarray section, in XcalableMP Fortran, or an array section, in XcalableMP C, that represents
the whole array.

110

CHAPTER 7. INTRINSIC AND LIBRARY PROCEDURES

Table 7.1: Built-in elemental functions in XcalableMP C. (The first line refers to the element
type of their argument(s) and return value.)
double
acos
asin
atan
atan2
cos
sin
tan
cosh
sinh
tanh
exp
frexp
ldexp
log
log10
fabs
pow
sqrt
ceil
floor
fmod

7.9.1

float
acosf
asinf
atanf
atan2f
cosf
sinf
tanf
coshf
sinhf
tanhf
expf
frexpfldexpflogf
log10f
fabsf
powf
sqrtf
ceilf
floorffmodf

long double
acosl
asinl
atanl
atan2l
cosl
sinl
tanl
coshl
sinhl
tanhl
expl
frexplldexpllogl
log10l
fabsl
powl
sqrtl
ceill
floorlfmodl

xmp scatter

Format
[F]
[C]

void

1

2

xmp scatter(x, a, idx1, ..., idxn)
xmp scatter(x[:]..., a[:]..., idx1[:]..., ..., idxn[:]...)

Synopsis
The xmp scatter procedure copies the value of each element of an array a to the corresponding
element of an array x that is determined by vectors idx1, ..., idxn.
This procedure produces the same result as the following Fortran assignment statement whenx, a, and idx1, ..., idxn are not mapped.
x(idx1(:,:,...), ..., idxn(:,:,...)) = a(:,:,...)
If any of the vectors idx1, ..., idxn have two or more elements with the same value, thebehavior and the result of xmp scatter is unspecified.
Output Arguments• x is an array of any type, shape, and mapping.

3

4

5
6
7
8

9

10
11

12

13

7.9. INTRINSIC/BUILT-IN TRANSFORMATIONAL PROCEDURES

1

2

3
4

Input Arguments• a is an array of the same type as x and any shape and mapping.
• idx1, ..., idxn are integer arrays of the same shape and mapping as a. The number of
idx’s is equal to the rank of x.

5

7.9.2

6

Format

7

8

111

[F]
[C]

xmp gather

void

xmp gather(x, a, idx1, ..., idxn)
xmp gather(x[:]..., a[:]..., idx1[:]..., ..., idxn[:]...)

Synopsis12

The xmp gather procedure copies the value of each element of an array a determined by vectors
idx1, ..., idxn to the corresponding element of an array x.
This procedure produces the same result as the following Fortran assignment statement whenx, a, and idx1, ..., idxn are not mapped.

13x(:,:,...) = a(idx1(:,:,...), ..., idxn(:,:,...))

14

Output Arguments

9
10
11

15

16

17

18
19

• x is an array of any type, shape, and mapping.
Input Arguments• a is an array of the same type as x and any shape and mapping.
• idx1, ..., idxn are integer arrays of the same shape and mapping as x. The number of
idx’s is equal to the rank of a.

20

7.9.3

21

Format

22

23

[F]
[C]

xmp packvoid

xmp pack(v, a, [mask])
xmp pack(v[:], a[:]..., [mask[:]...])

Synopsis26

The xmp pack procedure packs all of the elements of an array a, if mask is not specified, orthe elements selected by mask, to a vector v according to the array element order of the baselanguage.

27

Output Arguments

24
25

28

29

30

31
32

• v is a one-dimensional array of any type, size, and mapping.
Input Arguments• a is an array of the same type as v and any shape and mapping.
• (optional) mask is an array of default logical, in XcalableMP Fortran, or of type Bool, in
XcalableMP C, that has the same shape and mapping as a.

112

CHAPTER 7. INTRINSIC AND LIBRARY PROCEDURES

7.9.4

xmp unpack

Format
[F]
[C]

2

void

xmp unpack(a, v, [mask])
xmp unpack(a[:]..., v[:], [mask[:]...])

Synopsis

Output Arguments• a is an array of any type, shape, and mapping.
Input Arguments• v is a one-dimensional array of the same type of a and any shape and mapping.
• (optional) mask is an array of default logical, in XcalableMP Fortran, or of type Bool, in
XcalableMP C, that has the same shape and mapping as a.

xmp transpose

Format
[F]
[C]

3

4

The xmp unpack procedure unpacks a vector v to all the elements of an array a, if mask is not
specified, or the elements selected by a mask mask according to the array element order of thebase language.

7.9.5

1

5
6
7

8

9

10

11

12
13

14

15

void

xmp transpose(x, a, opt)
xmp transpose(x[:][:], a[:][:], int opt)

16

Synopsis17

The xmp transpose procedure sets the result obtained by transposing a matrix a to a matrix x.

18

Output Arguments

19• x is a two-dimensional array of any type, shape, and mapping.
Input Arguments• a is a two-dimensional array of the same type as x and any mapping. The extent of thefirst dimension is equal to that of the second dimension of x, and the extent of the second
dimension is equal to that of the first dimension of x.
• opt is an integer scalar. If opt is 0, the value of a remains unchanged after calling thisprocedure. If opt is 1, the value may be changed.

7.9.6

xmp matmul

Format
[F]
[C]

void

20

21

22
23
24

25
26

27

28

xmp matmul(x, a, b)
xmp matmul(x[:][:], a[:][:], b[:][:])

29

7.9. INTRINSIC/BUILT-IN TRANSFORMATIONAL PROCEDURES

1113

Synopsis3

The xmp matmul procedure computes the product of matrices a and b, and it sets the result toa matrix x.

4Output Arguments

2

5

6

7
8

9
10
11

• x is a two-dimensional array of any numerical type, shape and mapping.
Input Arguments• a is a two-dimensional array of the same type of x and any mapping. The extent of thefirst dimension is equal to that of x.
• b is a two-dimensional array of the same type of x and any mapping. The extent of thefirst dimension is equal to that of the second dimension of a, and the extent of the second
dimension is equal to that of x.

12

7.9.7

13

Format

14

15

[F]
[C]

xmp sort up

void

xmp sort up(v1, v2)
xmp sort up(v1[:], v2[:])

Synopsis17

The xmp sort up procedure sets the result obtained by sorting elements of a vector v2 in ascending order to a vector v1.

18

Output Arguments

16

19

20

21

• v1 is a one-dimensional array of any numerical type, shape, and mapping.
Input Arguments• v2 is a one-dimensional array of the same type, shape, and mapping as v1.

22

7.9.8

23

Format

24

25

[F]
[C]

xmp sort down

void

xmp sort down(v1, v2)
xmp sort down(v1[:], v2[:])

Synopsis27

The xmp sort down procedure sets the result obtained by sorting elements of a vector v2 indescending order to a vector v1.

28

Output Arguments

26

29• v1 is a one-dimensional array of any numerical type, shape and mapping.

114

CHAPTER 7. INTRINSIC AND LIBRARY PROCEDURES

Input Arguments• v2 is a one-dimensional array of the same type, shape, and mapping as v1.

1

2

1

2

3
4

5
6
7
8

Chapter 8

OpenMP in XcalableMP Programs
The usage of OpenMP directives in XcalableMP programs is subjected to the following basicrule.
• XcalableMP directives and the invocation of an XcalableMP intrinsic/built-in procedure
should be single-threaded, and they may therefore be placed in the sequential part, or
one of the single, master, or critical regions that are closely nested inside a parallel
region whose parent thread is the initial thread;11• with the exception that the XcalableMP’s loop directive that controls a loop can be placed
immediately inside the OpenMP’s parallel loop directive (parallel do for Fortran andparallel for for C), which controls the identical loop.

12

The behavior of coarray references in a parallel region is implemetation-defined.

9
10

13

14

ExamplesAssume that the following codes are placed in the sequential part of the program.
XcalableMP C#pragma omp parallel forfor (...){#pragma xmp barrier // NG because not single-threaded}
XcalableMP C#pragma omp parallel forfor (...){#pragma omp single{
5#pragma xmp barrier // OK because single-threaded// (inside a single region)}}
XcalableMP C#pragma omp parallel for#pragma xmp loop // OK because immediately nestedfor (...){
...
5 }

115

116

CHAPTER 8. OPENMP IN XCALABLEMP PROGRAMS

XcalableMP C#pragma xmp loop // OK because single-threaded (not nested)#pragma omp parallel forfor (...){...
5 }XcalableMP C#pragma xmp loop // OK because single threaded (not nested)for (...){#pragma omp parallel forfor (...) { ... }5 }XcalableMP C#pragma omp parallel forfor (...){#pragma xmp loop // NG because not immediately nestedfor (...) { ... }
5 }

1

2
3

4
5

6
7

8
9

10
11
12
13

14
15
16

17
18
19

Bibliography[1] OpenMP Architecture Review Board, “OpenMP Application Program Interface Version
3.1”, http://www.openmp.org/mp-documents/OpenMP3.1.pdf (2011).
[2] High Performance Fortran Forum, “High Performance Fortran Language Specification Version 2.0”, http://hpff.rice.edu/versions/hpf2/hpf-v20.pdf (1997).
[3] Message Passing Interface Forum, “MPI: A Message-Passing Interface Standard Version2.2”, http://www.mpi-forum.org/docs/mpi-2.2/mpi22-report.pdf (2009).
[4] Japan Association of High Performance Fortran, “HPF/JA Language Specification”, http://www.hpfpc.org/jahpf/spec/hpfja-v10-eng.pdf (1999).
[5] Yuanyuan Zhang, Hidetoshi Iwashita, Kuninori Ishii, Masanori Kaneko, Tomotake Nakamura, and Kohichiro Hotta, “Hybrid Parallel Programming on SMP Clusters Using XPFortran and OpenMP”, Proceedings of the International Workshop on OpenMP (IWOMP
2010), Vol. 6132 of Lecture Notes in Computer Science, pp. 133–148, Springer (2010).
[6] Hidetoshi Iwashita, Naoki Sueyasu, Sachio Kamiya, and Matthijs van Waveren, “VPP
Fortran and the design of HPF/JA extensions”, Concurrency and Computation — Practice& Experience, Vol. 14, No. 8–9, pp. 575–588, Wiley (2002).
[7] Jinpil Lee, Mitsuhisa Sato, and Taisuke Boku, “OpenMPD: A Directive-Based Data Parallel
Language Extension for Distributed Memory Systems”, Proceedings of the 2008 International Conference on Parallel Processing, pp. 121-128 (2008).

117

1

2

Appendix A

Programming Interface for MPI

5This chapter describes the programming interface for MPI, which is widely used for parallel
programming in cluster computing. Users can introduce MPI functions to XcalableMP using
the interface.

6

A.1

3
4

7
8

Call MPI functions from an XcalableMP program

XcalableMP provides the following user API functions to call MPI functions from an XcalableMP
program.

9• xmp get mpi comm10

• xmp init mpi

11• xmp finalize mpi

12

A.1.1

13

Format

14

15

[F]
[C]

xmp get mpi comminteger function
MPI Comm

xmp get mpi comm()
xmp get mpi comm(void)

Synopsis17

xmp get mpi comm returns the handle of the communicator associated with the executing node
set.

18

Arguments

19

none.

20

A.1.2

21

Format

16

22

[F]
[C]

xmp init mpi

void

xmp init mpi()
xmp init mpi(int *argc, char ***argv)

23

Synopsis24

xmp init mpi initializes the MPI execution environment.
119

120

APPENDIX A. PROGRAMMING INTERFACE FOR MPI

Arguments

1

In XcalableMP C, the command-line arguments argc and argv should be given to xmp init mpi.

2

A.1.3

3

xmp finalize mpi

Format
[F]
[C]

void

4

xmp finalize mpi()
xmp finalize mpi(void)

Synopsis6

xmp finalize mpi terminates the MPI execution environment.

7

Arguments

8

none.

9

Example10

XcalableMP C#include <stdio.h>#include "mpi.h"#include "xmp.h"5#pragma xmp nodes p[4]int main(int argc, char *argv[]) {xmp_init_mpi(&argc, &argv)
int rank, size;
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &size);

10

15

5

#pragma xmp task on p[1:2]{
MPI_Comm comm = xmp_get_mpi_comm(); // get the MPI communicator of p[1:2]
int rank, size;
MPI_Comm_rank(comm, &rank);
MPI_Comm_size(comm, &size);

20}
xmp_finalize_mpi();
return 0;25}

A.2. CALL XCALABLEMP FUNCTIONS FROM AN MPI PROGRAM

1

2
3

A.2

121

Call XcalableMP functions from an MPI program

XcalableMP provides the following user API functions to call XcalableMP functions from an
MPI program.

4• xmp init

5

• xmp finalize

8

The XcalableMP functions should appear between xmp init and xmp finalize. Please referto chapter 6 and examples in this section about how arguments are passed to the XcalableMPfunctions.

9

A.2.1

10

Format6
7

xmp init

[F]
11

[C]
12

integervoid

xmp init(comm)comm
xmp init(MPI Comm comm)

Synopsis15

xmp init initializes the XcalableMP execution environment. The MPI communicatior indicatedin xmp init is used as an executing node set in the XcalableMP functions. xmp init should
appear after MPI Init.

16

Arguments

17

MPI Communicator comm should be given to xmp init.

18

A.2.2

19

Format

13
14

20

21

[F]
[C]

xmp finalize

void

xmp finalize()
xmp finalize(void)

Synopsis23

xmp finalize finalizes the XcalableMP execution environment. xmp finalize should appear
before MPI Finalize.

24

Arguments

25

none.

26

Example (C language)

22

MPI C
#include <mpi.h>#include <xmp.h>#define N 5
int main(int argc, char **argv)
{

5

122

APPENDIX A. PROGRAMMING INTERFACE FOR MPI

int a[N], comm_size;
MPI_Init(&argc, &argv);
MPI_Comm_size(MPI_COMM_WORLD, &comm_size);

10

xmp_init(MPI_COMM_WORLD);
foo(N*comm_size, a);// foo() is an XMP function
xmp_finalize();
15

MPI_Finalize();
return 0;}
XcalableMP C
void foo(int total_elements, int a[total_elements]){#pragma xmp nodes p[*]#pragma xmp template t[total_elements]
5 #pragma xmp distribute t[block] onto p#pragma xmp align a[i] with t[i]#pragma xmp loop on t[i]for(int i=0;i<total_elements;i++)10a[i] = i;}

Example (Fortran)

1

MPI Fortranprogram test
include ’mpif.h’
integer, parameter :: N = 5
integer :: a(N), ierr, comm_size
5call MPI_INIT(ierr)call MPI_COMM_SIZE(MPI_COMM_WORLD, comm_size, ierr)call xmp_init(MPI_COMM_WORLD)
call foo(comm_size*5, a)call xmp_finalize()call MPI_FINALIZE(ierr)end program test
XcalableMP Fortran
subroutine foo(total_elements, a)
integer total_elements, a(total_elements)!$xmp nodes p(*)!$xmp template t(total_elements)
5 !$xmp distribute t(block) onto p!$xmp align a(i) with t(i)10

A.2. CALL XCALABLEMP FUNCTIONS FROM AN MPI PROGRAM!$xmp loop on t(i)do i=1, total_elements
10a(i) = iend doend subroutine foo

123

1

2

Appendix B

Interface to Numerical Libraries

4

This chapter describes the XcalableMP interfaces to existing MPI parallel libraries, which is
eﬀective to achieve high productivity and performance of XcalableMP programs.

5

B.1

6

The recommended design of the interface is as follows:378Interface Design• Numerical library routines can be invoked by an XcalableMP procedure through an interface procedure (Figure B.1).
XcalableMP program
Program foo!$xmp…
!$xmp…
…
call ixmp_xxx(…)
…
stop
end

XcalableMP interface

Numerical library routine

subroutine ixmp_xxx(…)…
call sub(…)…
returnend

subroutine sub(…)…
returnend

Figure B.1: Invocation of a library routine using an interface procedure.

9
10
11

12
13

14

15
16
17

• When the numerical library routine requires information regarding a global array, theinterface extracts it from the descriptor using query routines provided by XcalableMP,and passes it to the numerical library routine as an argument.
• The interface does not aﬀect the behavior of numerical library routines except for restrictions concerning the XcalableMP specification.

B.2

Extended Mapping Inquiry Functions

In this section, the extended mapping inquiry functions, which are implementation-defined, areshown. Specifications of the functions below are obtained from the Omni XcalableMP compiler(http://www.xcalablemp.org/download.html).
125

126

APPENDIX B. INTERFACE TO NUMERICAL LIBRARIES

B.2.1

xmp array gtol

[F]

integer function
type(xmp desc)
integer
integer
integer

xmp array gtol(d, dim, g idx, l idx)
d
dim
g idx
l idx

void

xmp array gtol(xmp desc t d, int dim, int g idx, int* l idx)

[C]1

Synopsis2

3

The xmp array gtol function translates a global index specified by g idx of a global arrayspecified by d into the corresponding index of its local section, and sets it to an array specified
by l idx. If the element of the specified index does not reside in the caller of the function, theresulting array is set to an unspecified value.
Input Arguments

4
5
6
7

8

• d is a descriptor of the global array.

9• dim is the target dimension of the global array.

10• g idx is an index of the global array.

11

Output Argument12• l idx is an index of the local array.

B.2.2

xmp array lsize

14

Format[F][C]

integer function
type(xmp desc)
integer
integer
int

13

15

xmp array lsize(d, dim, lsize)
d
dim
lsize
xmp array lsize(xmp desc t d, int dim, int *lsize)

Synopsis
The xmp array lsize function provides the local size of each dimension of the target global
array. Note that the local size does not include the size of the shadow.
Input Arguments

16

17

18
19

20

• d is a descriptor of a global array.

21• dim is the target dimension of the global array.

22

Output Argument• lsize is the local size of the target dimension of the global array.

23

24

B.3. EXAMPLE

1

B.2.3

2

Format

3

[C]

127

xmp array laddr

int

xmp array laddr(xmp desc t d, void **laddr)

4

Synopsis5

The xmp array laddr function provides the local address of the target global array.

6

Input Arguments

7

8

9

• d is a descriptor of a global array.
Output Arguments• laddr is the local address of the target global array.

10

B.2.4

11

Format
[F]

12

[C]
13

xmp array lda

integer function
type(xmp desc)
integer
int

xmp array lda(d, lda)d
lda
xmp array lda(xmp desc t d, int* lda)

Synopsis15

The xmp array lda function provides the leading dimension of the two-dimensional global array.
This function is used to call numerical libraries, such as BLAS.

16

Input Argument

14

17

18

19

20

21
22
23
24
25
26

27
28

• d is a descriptor of a global array, which must be a two-dimensional array.
Output Argument• lda is a leading dimension of the target global array.

B.3

ExampleThis section shows the interface to ScaLAPACK as an example of the XcalableMP interface tonumerical libraries.
ScaLAPACK is a linear algebra library for distributed-memory. Communication processes
in the ScaLAPACK routines depend on BLACS (Basic Linear Algebraic Communication Subprograms). ScaLAPACK library routines invoked from XcalableMP procedures also depend on
BLACS.
Example 1 This example shows an implementation of the interface for the ScaLAPACK driver
routine pdgesv.

128

APPENDIX B. INTERFACE TO NUMERICAL LIBRARIES

XcalableMP Fortran
subroutine ixmp_pdgesv(n,nrhs,a,ia,ja,da,ipiv,b,ib,jb,db,ictxt,info)
use xmp_lib
5

10

15

20

25

integer n,nrhs,ia,ja,ib,jb,ictxt,info,desca(9),descb(9),ierrdouble precision a,btype(xmp_desc) da,db,dta,dtb
integer lbound_a1,ubound_a1,lbound_a2,ubound_a2
integer blocksize_a1,blocksize_a2,lead_dim_a
integer lbound_b1,ubound_b1,lbound_b2,ubound_b2
integer blocksize_b1,blocksize_b2,lead_dim_b
ierr=xmp_array_lbound(da,1,lbound_a1)
ierr=xmp_array_ubound(da,1,ubound_a1)
ierr=xmp_array_lbound(da,2,lbound_a2)
ierr=xmp_array_ubound(da,2,ubound_a2)
ierr=xmp_align_template(da,dta)
ierr=xmp_dist_blocksize(dta,1,blocksize_a1)
ierr=xmp_dist_blocksize(dta,2,blocksize_a2)
ierr=xmp_array_lead_dim(da,1,lead_dim_a)ierr=xmp_array_lbound(db,1,lbound_b1)
ierr=xmp_array_ubound(db,1,ubound_b1)
ierr=xmp_array_lbound(db,2,lbound_b2)
ierr=xmp_array_ubound(db,2,ubound_b2)
ierr=xmp_align_template(db,dtb)
ierr=xmp_dist_blocksize(dtb,1,blocksize_b1)
ierr=xmp_dist_blocksize(dtb,2,blocksize_b2)
ierr=xmp_array_lead_dim(db,1,lead_dim_b)

30

35

desca(1)=1
desca(2)=ictxt
desca(3)=ubound_a1-lbound_a1+1
desca(4)=ubound_a2-lbound_a2+1
desca(5)=blocksize_a1
desca(6)=blocksize_a2
desca(7)=0
desca(8)=0
desca(9)=lead_dim_a4045descb(1)=1
descb(2)=ictxt
descb(3)=ubound_b1-lbound_b1+1
descb(4)=ubound_b2-lbound_b2+1
descb(5)=blocksize_b1
descb(6)=blocksize_b2
descb(7)=0
descb(8)=0
descb(9)=lead_dim_bB.3. EXAMPLE

129

50

call pdgesv(n,nhrs,a,ia,ja,desca,ipiv,b,ib,jb,descb,info)
returnend
55

1

Example 2 This example shows an XcalableMP procedure using the interface of Example 1.
XcalableMP Fortranprogram xmptdgesv
use xmp_lib
double precision a(1000,1000)
double precision b(1000)
integer ipiv(2*1000,2)!$xmp nodes p(2,2)!$xmp template t(1000,1000)
10 !$xmp template t1(2*1000,2)!$xmp distribute t(block,block) onto p!$xmp distribute t1(block,block) onto p!$xmp align a(i,j) with t(i,j)!$xmp align ipiv(i,j) with t1(i,j)
15 !$xmp align b(i) with t(i,*)...
integer i,j,ictxt
integer m=1000,n=1000,nprow=2,npcol=2
integer icontxt=-1,iwhat=020
integer nrhs=1,ia=1,ja=1,ib=1,jb=1,info
character*1 order...
order="C"...
25
call blacs_get(icontxt,iwhat,ictxt)
call blacs_gridinit(ictxt,order,nprow,npcol)...
!$xmp loop (i,j) on t(i,j)
do j=1,n30
do i=1,ma(i,j) = ...
end doend do...
35 !$xmp loop on t(i,*)
do i=1,mb(i)= ...
end do...
40
call ixmp_pdgesv(n,nrhs,a,ia,ja,xmp_desc_of(a),ipiv,*
b,ib,jb,xmp_desc_of(b),ictxt,info)
5

130

APPENDIX B. INTERFACE TO NUMERICAL LIBRARIES

45...
call blacs_gridexit(ictxt)
...
stop
end

Appendix C

1

Memory-layout Model2

16

In this chapter, the memory-layout model of global data in the Omni XcalableMP compiler(http://www.xcalablemp.org/download.html) is presented for reference.
According to the XcalableMP specification, a global array is distributed onto a node arrayaccording to the data-mapping directives, and as a result, a node owns a set of elements.
On each node, all and only the elements of the global array that it owns are gathered to form
the local array having the same rank as the global. For each axis of the global data, all and only
the indices that the node owns are packed to the axis of the local array so that the sequencecan be maintained, with the shadow area, if any, added at the lower and/or upper bound of theaxis.
Eventually, the local array is stored in memory on each node according to the rule for storingarrays in the base language, that is, in row-major order in XMP/Fortran and in column-major
order in XMP/C.
Note that owing to the model above, the memory usage may be non-uniform among thenodes.

17

Example

3
4
5
6
7
8
9
10
11
12
13
14
15

XcalableMP Fortran!$xmp nodes p(4,4)!$xmp template t(64,64)!$xmp distribute t(block,block) onto p
5

18
19
20

real a(64,64)!$xmp align a(i,j) with t(i,j)!$xmp shadow a(1,1)
The array a is distributed by a format of (block,block) onto a two-dimensional node arrayp, and each node owns a local array including a shadow area. Then, the local array is stored inmemory on each node, as shown in Figure C.1.

131

132

APPENDIX C. MEMORY-LAYOUT MODEL

shadowp(1,1) p(1,2) p(1,3) p(1,4)
p(2,4)
p(2,1) p(2,2) p(2,3) p(2,4)
distribute

allocatep(3,1) p(3,2) p(3,3) p(3,4)

p(4,1) p(4,2) p(4,3) p(4,4)

lowglobal array

local arrayhighmemoryFigure C.1: Example of memory layout in the Omni XcalableMP compiler.

12

Appendix D

XcalableMP I/O

3

D.1

Categorization of I/O

4

XcalableMP has three kinds of I/O.

5

D.1.1

Local I/O

16

Local I/O is a method that is employed to use I/O statements and standard I/O functions in
the base languages, where I/O statements and functions are used without any directives.
I/O statements (in XcalableMP Fortran) and I/O functions (in XcalableMP C) are executed
locally similar to other execution statements. It depends on the system which nodes can handle
the I/O statements and functions.
Local I/O can read a file written by the base language, and vice versa.
[F] The name of a global array in the I/O list describes the entire area of the array locatedin each node.
An array element of a global array can be referred to as an I/O item only in the node where
it is located.
[F] No array section of a global array can be referred to as an I/O item.

17

D.1.2

6
7
8
9
10
11
12
13
14
15

Master I/O[F]29

Master I/O is input and output for the file that corresponds to an executing node set. Master
I/O is a collective execution.
In master I/O, a global data object is input and output as if it was executed only by a master
node, which represents the executing node set, through its local copy of the data.
The master node is chosen by the system arbitrarily from among the executing node set,
and is unique to the executing node set during execution of the program.
Master I/O is provided in the form of directives of XcalableMP Fortran.
A global array as an I/O item is accessed in the sequential order of array elements. Whena local variable is read from a file, the value is copied to all nodes of the executing node set.
When a local variable or an expression is written to a file, only the value of the data on themaster node is written.
Master I/O can read a file written in the base language, and vice versa.

30

D.1.3

18
19
20
21
22
23
24
25
26
27
28

31
32

Global I/O

Global I/O is input and output for a file that corresponds to an executing node set. Some
executions of the global I/O are collective and the others are independent. In a large system with
133

134

APPENDIX D. XCALABLEMP I/O

Collective I/O
Atomic I/O
Direct I/OTable D.1: Global I/O.
independent/collective access methodcollective
sequential accessindependent
sequential accessindependent
direct accessmany nodes, the global I/O can be expected to have higher speed and less memory consumption
execution than master I/O.
[F] It is provided in the form of directives for some of I/O statements, such as OPEN,
CLOSE, READ, and WRITE statements.
[C] It is provided in the form of service functions and an include file.
Global I/O can handle only unformatted (binary) files. In XcalableMP Fortran, implied DOloops and some specifiers cannot be used. In XcalableMP C, formatted I/O libraries, including
fprintf() and fscanf(), are not provided.
Global I/O can read a file written in MPI-IO, and vice versa.
[F] File formats are not compatible between XcalableMP Fortran and the base language
because global I/O does not generate or access file headers and footers that are specific to thebase language.
There are three kinds of global I/O, as shown in Table D.1. Collective global I/O isfor collective execution and sequential file access. It handles global data in a sequential order,
similar to master I/O. Atomic global I/O is for independent execution and sequential file access.
Executing nodes share file positioning of the global I/O file and execute each I/O statement andlibrary call mutually. Direct global I/O is for independent execution and direct file access.
Each executing node has its own file positioning and accesses a shared file independently.
Restriction• The name of a global array may not be declared in a namelist group. That is, NAMELISTI/O is not allowed for global arrays.
Advice to programmers
Local I/O is useful for debugging that is focused on a node because local I/O is executed oneach node individually.
Master I/O is a directive extension, where the execution result matches that of the baselanguage, ignoring directive lines.
Global I/O aims for highly parallel I/O using thousands of nodes. It is limited to binary
files, and it avoids the extreme concentration of computational load and memory consumption
to specific nodes using MPI-IO or other parallel I/O techniques.

D.2

File Connection

A file is connected to a unit in XcalableMP Fortran and to a file handler in XcalableMP C.
This operation is called file connection. Local I/O connects a file to each node independently.
Master I/O and global I/O connect a file to an executing node set collectively.
There are two ways of specifying file connections, dynamic connection and preconnection.
Dynamic connection connects a file during the execution of the program. Preconnection connects
a file at the beginning of execution of the program, and it can therefore execute I/O statements

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

19

20
21

22

23
24
25
26
27
28
29

30

31
32
33
34
35
36

D.3. MASTER I/O

135

2

and functions without the prior execution of an OPEN statement or a function call to open the
file.

3

D.2.1

1

File Connection in Local I/O

7

The language processor of the base language connects the file to each node. It is implementationdefined which nodes can access the standard input, output, and error files. The behavior of theaccesses to files having the same name on multiple nodes is also implementation-defined. The
only primary node can access the standard input, output, and error files.

8

D.2.2

4
5
6

[F] File Connection in Master I/O

19

An OPEN statement that is specified with a master I/O directive connects a file to the executing
node set. When a master I/O file is connected by a READ statement or a WRITE statement
without encountering any OPEN statement, the name and attributes of the file depend on thelanguage system of the base language. Disconnection from a master I/O file is executed by a
CLOSE statement or by the termination of the program.
The dynamic connection must be executed collectively by all nodes sharing the file with thesame unit number. Two executing node sets may employ the same unit number only if they
have no common node.
The standard input, output, and error files are preconnected to the entire node set. Therefore, master I/O executed on the entire node set is always allowed without OPEN or CLOSE
statements.

20

D.2.3

9
10
11
12
13
14
15
16
17
18

File Connection in Global I/O

22

The dynamic connection of global I/O is a collective execution, and is valid for the executing
node set. Global I/O files cannot be preconnected.

23

[F]

21

29

An OPEN statement that is specified with a global I/O directive connects a file to the executing
node set. Disconnection from a global I/O file is executed by a CLOSE statement or by thetermination of the program.
The dynamic connection must be executed collectively by all nodes sharing the file with thesame unit number. Two executing node sets may employ the same unit number only if they
have no common node.

30

[C]

24
25
26
27
28

33

A library function to open a global I/O file connects the file to the executing node set. Disconnection from a global I/O file is executed by a library function to close the file or terminate theprogram.

34

D.3

31
32

35
36
37

Master I/O

A master I/O construct executes data transfer between a file and an executing node set via a
master node of the executing node set. For a global array, the virtual sequential order of thearray elements is visible.

136

APPENDIX D. XCALABLEMP I/O

D.3.1

master io Construct

Syntax[F]

1

2!$xmp master io
io-statement!$xmp master io begin
io-statement...
!$xmp master io end
where io-statement is one of:

[F]

3

4

• OPEN statement

5• CLOSE statement

6• READ statement

7• WRITE statement

8• PRINT statement

9• BACKSPACE statement

10• ENDFILE statement

11• REWIND statement12• INQUIRE statement

13

Restriction• The following items, including a global array or a subobject of a global array, must notappear in an input item or output item.

14

15
16

– A substring-range

17– A section-subscript18– An expression including operators

19– An io-implied-do-control

20• An I/O statement specified with a master I/O directive must be executed collectively on
the node set that is connected to the file.
• Internal file I/O is not permitted to be a master I/O.

Description
An I/O statement that is specified with a master I/O directive accesses a file whose format is
the same as that of the base language. The access, including connection, disconnection, input
and output, file positioning, and inquiry, is collective, and must be executed on the same node
set as the one on which the file was connected.
A master node, which is a unique node to an executing node set, is chosen by the language
system. Master I/O works as if all file accesses were executed only on the master node.
The operations for I/O items are summarized in Table D.2.

21
22

23

24

25
26
27
28
29
30
31

D.4. [F] GLOBAL I/O

input item

output item

137

Table D.2: Operations for I/O.
I/O item
operation
name of global array
The data elements that are read from the file inthe sequential order of array elements are distributed onto the global array on the node set.
The file positioning increases according to thesize of data.
array element of global The data element that is read from the file isarraycopied to the element of the global array on thenode to which it is mapped. The file positioning
increases according to the size of data.
local variable
The data element that is read from the file isreplicated to the local variables on all nodes ofthe executing node set. The file positioning increases according to the size of data.
implied DO loop
For each input item, repeat the above operation.
name of global array
The data elements of the global array are collected and are written to the file in the sequential order of array elements. The file positioning
increases according to the size of data.
array element of global The element of the global array is written toarray
the file. A file position increases according to
the size of data.
local variable and ex- The value evaluated on the master node is writpression
ten to the file. The file positioning increases
according to the size of data.
implied DO loop
For each output item, repeat the above operation.

9

Namelist input and output statements cannot treat global arrays. A namelist output statement writes the values on the master node to the file. In the namelist input, each item of thenamelist is read from the file to the master node if it is recorded in the file. Then, all items ofthe namelist are replicated onto all nodes of the executing node set from the master node even
if some items are not read from the file.
IOSTAT and SIZE specifiers and specifiers of the INQUIRE statement that can return values
always return the same value among the executing node set.
When a condition that is specified by the ERR, END, or EOR specifier is satisfied, all nodesof the executing node set are branched together to the same statement.

10

Advice to implementers

1
2
3
4
5
6
7
8

12

It is recommended to provide such a compiler option that local I/O statements (specified without
directives) are regarded as master I/O statements (specified with master io directives).

13

D.4

11

14
15

[F] Global I/O

Global I/O performs unformatted data transfer, and can be expected to have a higher performance and lower memory consumption than master I/O. The file format is compatible with the138

APPENDIX D. XCALABLEMP I/O

one in MPI-IO.
There are three kinds of Global I/O, namely collective I/O, atomic I/O, and direct I/O.

D.4.1

1
2

Global I/O File Operation

3

global io construct is defined as follows.

4

Syntax5[F]!$xmp global io [atomic / direct]
io-statement!$xmp global io [atomic / direct] begin
io-statement...
!$xmp end global io
The first syntax is just a shorthand of the second syntax.

[F]

6

7

Restriction

8

I/O statements and specifiers that are available for an io-statement are shown in the following
table. The definition of each specifier is described in the specification of the base language.
Case of global io construct without a direct clause:10

I/O statement
OPEN
CLOSE
READ
WRITE

9

available specifiers
UNIT, IOSTAT, FILE, STATUS, POSITION, ACTION, ACCESS, FORM
UNIT, IOSTAT, STATUS
UNIT, IOSTAT
UNIT, IOSTAT
11

Case of global io construct with a direct clause:
I/O statement
OPEN
CLOSE
READ
WRITE

available specifiers
UNIT, IOSTAT, FILE, STATUS, RECL, ACTION, ACCESS, FORM
UNIT, IOSTAT, STATUS
UNIT, REC, IOSTAT
UNIT, REC, IOSTAT
12

The input item and output item of a data transfer statement with a global io directive
must be the name of a variable.
Description
Global I/O construct connects, disconnects, inputs, and outputs the global I/O file, which iscompatible with MPI-IO.
The standard input, output, and error files cannot be a Global I/O file. A Global I/O filecannot preconnect to any unit or any file handler, and must be explicitly connected by the
OPEN statement that is specified with a global io directive.
The OPEN statement that is specified with a global io directive is collective execution, andthe file is shared among the executing node set. A file that has already been opened by another

13
14

15

16
17
18
19
20
21
22

D.4. [F] GLOBAL I/O

1
2
3
4
5
6
7
8

9

OPEN statement with a global io directive cannot be reopened by an OPEN statement withor without a global io directive before closing it.
A global I/O file must be disconnected explicitly by a CLOSE statement that is specified witha global io directive; otherwise, the result of I/O is not guaranteed. The CLOSE statementthat is specified with a global io directive is a collective execution, and must be executed by
the same executing node set as the one where the OPEN statement is executed.
Utilizable values of the specifiers in I/O statements are shown in the following table. Definitions of the specifiers are described in the specification of the base language.
• OPEN statement
specifiers
UNIT
FILE
STATUSPOSITIONACTIONRECL
ACCESS
FORM

10
11
12
13
14
15
16

17

19

20
21

value
external file unit (scalar constant
expression)
file name (scalar CHARACTER
expression)’OLD’, ’NEW’, ’REPLACE’ or’UNKNOWN’’ASIS’, ’REWIND’ or ’APPEND’’READ’, ’WRITE’ or ’READWRITE’
the value of the record length(scalar constant expression)’SEQUENTIAL’ or ’DIRECT’’FORMATTED’ or ’UNFORMATTED’default
not omissible
not omissible’UNKNOWN’’ASIS’
implementation-defined
not omissible’SEQUENTIAL’For direct access, UNFORMATTED. For
sequential access, this specifier shall not
be omitted.

POSITION is available only if the directive has no direct clause. RECL is available only
if the directive has a direct clause. For direct I/O, the ACCESS specifier shall appear andthe value shall be evaluated to DIRECT. For collective I/O and atomic I/O, the value ofthe ACCESS specifier shall be evaluated to SEQUENTIAL if this specifier appears. For
collective I/O and atomic I/O, the FORM specifier shall appear and the value shall be
evaluated to UNFORMATTED. For direct I/O, the value of the FORM specifier shall be
evaluated to UNFORMATTED if this specifier appears.
• CLOSE statement
specifiers
UNIT
STATUS

18

139

value
external file unit (scalar constant expression)’KEEP’ or ’DELETE’default
not omissible.
’KEEP’• READ/WRITE statement
REC is available only if the directive has a direct clause.
• When a scalar variable of default INTEGER is specified to the IOSTAT specifier, it becomes defined with an error code after execution.

140

APPENDIX D. XCALABLEMP I/O

specifiers
UNIT
REC

value
external file unit (scalar constant expression)
the value of the number of record (scalar constant expression)

default
not omissible
not omissible

OPEN, CLOSE, READ, and WRITE statements that are specified with global io directives without atomic or direct clauses are called collective OPEN, collective CLOSE, collective
READ, and collective WRITE statements, respectively. All of these statements are called collective I/O statements.
OPEN, CLOSE, READ, and WRITE statements specified with global io directives having
atomic clauses are called atomic OPEN, atomic CLOSE, atomic READ, and atomic WRITE
statements, respectively. All of these statements are called atomic I/O statements.
OPEN, CLOSE, READ, and WRITE statements specified with global io directives withdirect clauses are called direct OPEN, direct CLOSE, direct READ, and direct WRITE statements, respectively. All of these statements are called direct I/O statements.
The file connected by a collective, atomic, or direct OPEN statement can be read/written
only by the same type of READ/WRITE statements. The file can be disconnected by the same
type of CLOSE statement. Diﬀerent types of global I/O cannot be executed together for thesame file or the same unit. For example, atomic I/O statements cannot be executed for the unit
connected by a collective OPEN statement.
D.4.1.1

file sync all Directive

Two data accesses cause a conflict if they access the same absolute byte displacements of thesame file, and at least one is a write access. When two accesses to the same file conflict in direct
or collective I/O, the following file sync all directive to the file must be executed.
Syntax

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

16

17
18
19

20

!$xmp file sync all([UNIT=]file-unit-number)
21

The file sync all directive is an execution directive and collective to the nodes connected tothe specified file-unit-number. The execution of a file sync all directive first synchronizes all thenodes connected to the specified file-unit-number, and then causes all previous writes to the fileby the nodes to be transferred to the storage device. If some nodes have made updates to thefile, then all such updates become visible to subsequent reads of the file by the nodes.

D.4.2

Collective Global I/O Statement

Collective I/O statements read/write shared files and can handle global arrays.
All collective I/O statements execute collectively. In collective I/O, all accesses to a file,such as connection, disconnection, input, and output, must be executed on the same executing
node set.
The operations for I/O items are summarized in the following table.

D.4.3

Atomic Global I/O Statement

Atomic I/O statements read/write shared files exclusively among executing nodes in arbitraryorder. Because it is a nondeterministic parallel execution, the results may diﬀer every time it isexecuted, even for the same program.

22
23
24
25
26

27

28
29
30
31
32

33

34
35
36

D.5. [C] GLOBAL I/O LIBRARY

input
item

I/O itemname of global array

local variable

output name of global array
item
local variable, expression

141

operation
The values read from a file are assigned to the elements ofthe global array. The file positioning increases according to
the size of the data.
The values read from the file are replicated into the local
array on all executing nodes. The file positioning increases
according to the length of the data.
The values of a global array are written to the file in thesequential order of the array elements. The file positioning
increases according to the size of the data.
The values evaluated on a node are arbitrarily selected by
the language processor from the executing node set. The filepositioning increases according to the size of the data.

10

Atomic OPEN and CLOSE statements are executed collectively, while atomic READ and
WRITE statements are executed independently. A file connected by an atomic OPEN statement
can be disconnected only by an atomic CLOSE statement executed on the same executing nodeset. Atomic READ and WRITE statements can be executed on any single node of the same
executing node set.
Atomic READ and WRITE statements are exclusively executed. The unit of exclusive
operation is a single READ statement or a single WRITE statement.
The initial file positioning is determined by the POSITION specifier of the atomic OPEN
statement. Then, the file positioning seeks in every READ and WRITE statement according to
the length of the input/output data.

11

D.4.4

1
2
3
4
5
6
7
8
9

Direct Global I/O Statement25

Direct I/O statements read/write shared files by specifying the file positioning for each node.
Direct OPEN and CLOSE statements are executed collectively, while direct READ and
WRITE statements are executed independently. A file connected by a direct OPEN statement
can be disconnected only by a direct CLOSE statement executed on the same executing nodeset. Direct READ and WRITE statements can be executed on any single node of the same
executing node set.
Direct READ and WRITE statements read/write local data at the file positioning specified
by the REC specifier independently. The file positioning is shifted from the top of the file on
the basis of the product of the specifiers RECL (of OPEN statement) and REC (of READ and
WRITE statement).
In order to guarantee the order of direct I/O statements to the same file position, the fileshould be closed or the file sync all directive should be executed between these statements.
Otherwise, the outcome of multiple accesses to the same file position, in which at least one is awrite access, is implementation-defined.

26

D.5

12
13
14
15
16
17
18
19
20
21
22
23
24

27
28
29
30

[C] Global I/O Library

XcalableMP C provides some data types defined in the include file “xmp.h”, a set of libraryfunctions with arguments of the data types, and built-in operators to get values of the datatypes from names of a variable, a template, etc.
The following types are provided.

142

APPENDIX D. XCALABLEMP I/O• xmp file t : file handle

1• xmp rang t : descriptor of array section

2

The following library functions are provided. Collective function names end with all.

3

• global I/O file operation

4– xmp fopen all : file open
– xmp fclose all : file close– xmp fseek : setting (individual) file pointer– xmp fseek shared all : setting shared file pointer– xmp ftell : displacement of (individual) file pointer– xmp ftell shared : displacement of shared file pointer– xmp file sync all : file synchronization• collective I/O– xmp file set view all : setting file view
– xmp file clear view all : initializing file view
– xmp fread all : collective read of local data
– xmp fwrite all : collective write of local data
– xmp fread darray all : collective read of global data
– xmp fwrite darray all : collective write of global data
• atomic I/O– xmp fread shared : atomic read
– xmp fwrite shared : atomic write• direct I/O– xmp fread : direct read
– xmp fwrite : direct write

5
6
7
8
9
10
11

12

13
14
15
16
17
18

19

20
21

22

23
24

Data type25

The following data types are defined in include file xmp io.h.

26

xmp file t A file handler. It is connected to a file when the file is opened. It has a shared file
pointer and an individual file pointer to point where data should be read/written in thefile.
A shared file pointer is a shared resource among all nodes of the node set that has opened
the file. Atomic I/O uses a shared file pointer. An (individual) file pointer is an individual
resource on each node. Collective I/O and direct I/O use individual file pointers.
These two file pointers are managed in the structure xmp file t, and can be controlled andreferenced only through the provided library functions.
xmp range t Descriptor of array section, including lower bound, upper bound, and stride
for each dimension. Functions for operating the descriptor are shown in the following
table. The xmp allocate range() function is used to allocate memory. The xmp set range()function is used to set ranges of an array section. The xmp free range() function releases
the memory for the descriptor.

27
28
29
30
31
32
33
34

35
36
37
38
39

D.5. [C] GLOBAL I/O LIBRARY

143

function nameargument
return value

xmp range t ∗xmp allocate range(n dim)int n dim the number of dimensions
xmp range t∗ descriptor of array section. NULL is returned when aprogram abends.

function nameargument

void xmp set range(rp, i dim, lb, length, step)
xmp range t ∗rp descriptor
int i dim target dimension
int lb lower bound of array section in the dimension i dim
int length length of array section in the dimension i dim
int step stride of array section in the dimension i dim

144

APPENDIX D. XCALABLEMP I/Ofunction nameargument

xmp range t ∗rpvoid xmp free range(rp)descriptor of array section.

D.5.1

Global I/O File Operation

D.5.1.1

xmp fopen all

1

2

xmp fopen all opens a global I/O file. Collective execution.
function nameargument
return value3

xmp file t ∗xmp fopen all(fname, amode)
const char ∗fname file nameconst char ∗amode equivalent to fopen of POSIX. combination of “rwa+”
xmp file t∗ file structure. NULL is returned when a program
abend.

File view is initialized, where file view is based on the MPI-IO file view mechanism. The
value of shared and individual file pointers depends on the value of amode.
amode
r
r+
w
w+

a
a+

D.5.1.2

5intended purpose
Open for reading only. File pointer points to the beginning of the file.
Open an existing file for update (reading and writing). File pointerpoints to the beginning of the file.
Create for writing. If a file having that name already exists, it will beoverwritten. File pointer points to the beginning of the file.
Create a new file for update (reading and writing). If a file having thatname already exists, it will be overwritten. File pointer points to thebeginning of the file.
Append; open for writing at end-of-file or create for writing if the filedoes not exist. File pointer points to the end of the file.
Open for append; open (or create if the file does not exist) for update
at the end of the file. File pointer points to the beginning of the file.

xmp fclose all

6

xmp fclose all closes a global I/O file. Collective execution.
function nameargument
return value

D.5.1.3

4

xmp file t ∗fh
int

int ∗xmp fclose all(fh)file structure
0: normal termination
1: abnormal termination. fh is NULL.
2:abnormal termination.
error
MPI File close.

xmp fseek

xmp fseek sets the individual file pointer in the file structure. Independent execution.

7

in

8

9

D.5. [C] GLOBAL I/O LIBRARY

function nameargument

return value

145

int xmp fseek(fh, oﬀset, whence)xmp file t ∗fh file structure
long long oﬀset displacement of current file view from position of whence
int whence choose file position
SEEK SET: the beginning of the file
SEEK CUR: current position
SEEK END: the end of the file
int 0: normal termination
an integer other than 0: abnormal termination1

D.5.1.4

xmp fseek shared2

xmp fseek shared sets the shared file pointer in the file structure. Independent execution.
function nameargument

return value

3

4
5

D.5.1.5

xmp ftell

xmp ftell returns the position of the individual file pointer in the file structure. Independent
execution.
function nameargument
return value

67
8

int xmp fseek shared(fh, oﬀset, whence)xmp file t ∗fh file structure
long long oﬀset displacement of current file view from positionof whence
int whence choose file position
SEEK SET: the beginning of the file
SEEK CUR: current position
SEEK END: the end of the file
int 0: normal termination
an integer other than 0: abnormal termination

D.5.1.6

xmp file t ∗fhlong long

long long xmp ftell(fh)file structure
Upon successful completion, the function shallopen the file and return a non-negative integer
representing the lowest-numbered unused file descriptor. Otherwise, a negative number shall be
returned.

xmp ftell shared

xmp ftell shared returns the position of the shared file pointer in the file structure. Independent
execution.

146

APPENDIX D. XCALABLEMP I/Ofunction nameargument
return value

D.5.1.7

long long xmp ftell shared(fh)xmp file t ∗fh file structure
long long Upon successful completion, the function shallopen the file and return a non-negative integer
representing the lowest numbered unused file descriptor. Otherwise, negative number shall be
returned.

xmp file sync all1

xmp file sync all guarantees completion of access to the file from nodes sharing the file. Twodata accesses conflict if they access the same absolute byte displacements of the same file, andat least one is a write access. When two accesses A1 and A2 to the same file conflict in direct orcollective I/O, an xmp file sync all to the file must be invoked between A1 and A2; otherwise,
the outcome of the accesses is undefined. Collective execution.
function nameargument
return value

D.5.2

xmp file t ∗fhintCollective I/O is executed collectively, but using the individual pointer. It reads/writes datafrom the position of the individual file pointer and moves the position forward by the length ofthe data.
Before the file access, a file view is often specified. A file view, like a window to the file,spans the positions corresponding to the array elements that are owned by each node. For moredetails of file view, refer to the MPI 2.0 specification.
xmp file set view all

xmp file set view all sets a file view to the file. Collective execution.
function nameargument

return value

3
4
5
6

int xmp file sync all(fh)file structure
0: normal termination
an integer other than 0: abnormal termination

Collective Global I/O Functions

D.5.2.1

2

7

8
9
10
11
12
13

14

15

int xmp file set view all(fh, disp, desc, rp)
xmp file t ∗fh file structure
long long disp displacement from the beginning of thefile.
xmp desc t desc descriptor
xmp range t ∗rp range descriptor
int 0: normal termination
an integer other than 0: abnormal termination

The file view of distributed desc limited to range rp is set into file structure f h.

16

D.5. [C] GLOBAL I/O LIBRARY

1

2
3
4

D.5.2.2

xmp file clear view all

xmp file clear view all clears the file view. Collective execution.
The positions of the shared and individual file pointers are set to disp, and the elemental
data type and the file type are set to MPI BYTE.
function nameargument
return value

56
7

D.5.2.3

xmp fread all reads the same data from the position of the shared file pointer onto all of theexecuting nodes. Collective execution.

return value

9
10
11
12

D.5.2.4

xmp fwrite all writes individual data on all of the executing nodes to the position of the shared
file pointer. Collective execution.
It is assumed that the file view is set in advance. Each node writes its data into its own fileview.

return value

14
15
16

ssize t xmp fread all(fh, buﬀer, size, count)xmp file t ∗fh file structure
void ∗buﬀer beginning address of read variables
size t size the size of a read data element
size t count the number of read data elements
ssize t Upon successful completion, return the size ofread data. Otherwise, negative number shall be
returned.

xmp fwrite allfunction nameargument13

int xmp file clear view all(fh, disp)
xmp file t ∗fh file structure
long long disp displacement from the beginning of the file.
int 0: normal termination
an integer other than 0: abnormal termination

xmp fread allfunction nameargument

8147

D.5.2.5

ssize t xmp fwrite all(fh, buﬀer, size, count)xmp file t ∗fh file structure
void ∗buﬀer beginning address of written variables
size t size the size of a written data element
size t count the number of written data elements
ssize t Upon successful completion, return the size of
written data. Otherwise, negative number shallbe returned.

xmp fread darray all

xmp fread darray all reads data cooperatively to the global array from the position of the shared
file pointer.
Data is read from the file to distributed desc limited to range rp.

148

APPENDIX D. XCALABLEMP I/Ofunction nameargument

return value

D.5.2.6

ssize t xmp fread darray all(fh, desc, rp)
xmp file t ∗fh file structure
xmp desc t desc descriptor
xmp range t ∗rp range descriptor
ssize t Upon successful completion, return the size ofread data. Otherwise, negative number shall be
returned.

xmp fwrite darray all

xmp fwrite darray all writes data cooperatively from the global array to the position of theshared file pointer.
function nameargument

return value

Atomic Global I/O Functions

Atomic I/O is executed independently, but using the shared pointer. It exclusively reads/writes
local data from the position of the shared file pointer, and moves the position forward by thelength of the data.
Before atomic I/O is executed, the file view must be cleared.
[Rationale]
Although the file views must be the same on all processes in order to use the shared file
pointer, the xmp file set view all function may set diﬀerent file views for all nodes. Thus, before
atomic I/O is used, the file view must be cleared.
D.5.3.1

xmp fread shared

xmp fread shared exclusively reads local data from the position of the shared file pointer, andmoves the position forward by the length of the data. Independent execution.
function nameargument

return value

2
3

ssize t xmp fwrite darray all(fh, desc, rp)
xmp file t ∗fh file structure
xmp desc t desc descriptor
xmp range t ∗rp range descriptor
ssize t Upon successful completion, return the size ofread data. Otherwise, negative number shall be
returned.

Data is written from distributed desc limited to range rp to the file.

D.5.3

1

ssize t xmp fread shared(fh, buﬀer, size, count)xmp file t ∗fh file structure
void ∗buﬀer beginning address of read variables
size t size the size of a read data element
size t count the number of read data elements
ssize t Upon successful completion, return the size ofread data. Otherwise, negative number shall be
returned.

4

5

6
7
8
9
10
11
12
13

14

15
16

D.5. [C] GLOBAL I/O LIBRARY

12
3

D.5.3.2

xmp fwrite shared

xmp fwrite shared exclusively writes local data to the position of the shared file pointer andmoves the position forward by the length of the data. Independent execution.
function nameargument

return value

4

149

D.5.4

ssize t xmp fwrite shared(fh, buﬀer, size, count)xmp file t ∗fh file structure
void ∗buﬀer beginning address of written variables
size t size the size of a written data element
size t count the number of written data elements
ssize t Upon successful completion, return the size of
written data. Otherwise, negative number shallbe returned.

Direct Global I/O Functions

11

Direct I/O is executed independently and uses the individual pointer. It individually reads/writes
local data from the position of the individual file pointer, and moves the position forward by
the length of the data, considering the file view.
In order to guarantee the order by xmp fread and xmp fwrite functions to the same fileposition, the file should be closed or the xmp file sync all function should be executed between
these functions. Otherwise, the outcome of multiple accesses to the same file position, in which
at least one is the xmp fwrite function, is implementation dependent.

12

Advice to programmers

5
6
7
8
9
10

14

Function xmp fseek is useful for setting the individual file pointer. It is not recommended touse it together with the file view because of its complexity.

15

D.5.4.1

13

16
17

xmp fread

xmp fread reads data from the position of the individual file pointer and moves the position
forward by the length of the data. Independent execution.
function nameargument

return value

18

19
20

D.5.4.2

ssize t xmp fread(fh, buﬀer, size, count)xmp file t ∗fh file structure
void ∗buﬀer beginning address of read variables
size t size the size of a read data element
size t count the number of read data elements
ssize t Upon successful completion, return the size ofread data. Otherwise, negative number shall be
returned.

xmp fwrite

xmp fwrite writes data to the position of the individual file pointer and moves the position
forward jby the length of the data. Independent execution.

150

APPENDIX D. XCALABLEMP I/Ofunction nameargument

return value

ssize t xmp fwrite(fh, buﬀer, size, count)xmp file t ∗fh file structure
void ∗buﬀer beginning address of written variables
size t size the size of a written data element
size t count the number of written data elements
ssize t Upon successful completion, return the size of
written data. Otherwise, negative number shallbe returned.

1

2

Appendix E

Memory Consistency Model

14

This chapter explains the memory consistency model that is adopted by XcalableMP.
Memory consistency models have specified rules regarding multiple data accesses to memory.
Because XcalableMP is an extension of the base languages, and its memory consistency model
is defined as an extension to them, that is, XcalableMP follows all of the rules that are adopted
by base languages.
In addition, XcalableMP introduces some rules about global view. In global view, global
communication constructs are used to access distributed data. Furthermore, distributed data
can be accessed by designating data in local view. Conversely, non-distributed data can beaccessed by designating distributed data using global communication constructs in global view.
These are not considered under the memory consistency models of the base language because
global view is a new concept that was introduced by XcalableMP.
Please recall that global communication constructs are collective as described in Section 2.8.

15

E.1

3
4
5
6
7
8
9
10
11
12
13

16
17
18

Execution Traces

This section explains execution traces that are enabled by the Xcalable memory consistencymodel.
First, instructions are defined asi : = xmp syn | xmp asyn(async-id) | wait async(async-id) | f stmt

19
20
21
22

where xmp syn denotes a global communication construct with no async clause, xmp asyn(async-id)
denotes a global communication construct with the clause async(async-id), and f stmt is astatement.
Next, operations are defined aso : = Fetchj i | Executej i | Reflectj i

23
24
25
26
27
28
29
30
31

where j is a positive integer.
Operation Fetchj i denotes that instruction i is fetched j times. The integer j is incremented
whenever a loop is exited. The instructions that are called multiple times in loops are identified
by js. Operation Executej i denotes that instruction i is executed, while operation Reflectj idenotes that the eﬀect of instruction i is saved to physical memories.
Finally, the memory consistency model defines constraints written by a partial order ≤on operations as described below. Execution traces are defined as sequences of operations that
follow the order. In the following, o1 < o2 denotes o1 ≤ o2 and o1 ̸≡ o2 . In addition, o1 < o2 < o3denotes o1 < o2 and o2 < o3 .
151

152

APPENDIX E. MEMORY CONSISTENCY MODEL

Fetchj1 i1 < Fetchj2 i2 implies Executej1 i1 < Executej2 i2
j1

j2

j1

(i)
j2

Execute xmp syn < Execute i2 implies Reflect xmp syn < Execute i2
j1

j3

(ii)
j2

Execute xmp asyn(async-id) < Execute wait async(async-id) < Execute i2 implies
Reflectj1 xmp asyn(async-id) < Executej2 i2(iii)

Figure E.1: Constraints that are required by the XcalableMP memory consistency model.

E.1.1

Common Constraints

In this subsection, we explain some constraints that are common to both synchronous andasynchronous communications.
In the XcalableMP memory consistency model, instructions are executed in the order inwhich they are fetched. This is represented by i in Figure E.1.

E.1.2

Constraints for Synchronous Communications

The constructs reflect, gmove (and its subsequent assignment statement), reduction, andbcast are synchronous if async is not specified. This means that executions of these constructs
guarantee the completion of data synchronization. That is, global communication constructs
read data that are written by previously executed statements, and their subsequent statements
and global communication constructs read data that are written by global communication constructs. This is given by ii in Figure E.1
For example, in the following code, the assignment statement g(:)=h(:) is guaranteed tobe completed before the second gmove construct is executed. Therefore, the value of g(i) mustbe i when the assignment statement x(:)=g(6:10) is executed.
Finally, the value of x(i) on p(1) should be i+5.
XcalableMP Fortran!$xmp nodes p(2)!$xmp template t(10)!$xmp distribute (block) onto p :: tinteger :: g(10), h(10)
5 !$xmp align (i) with t(i) :: g, hinteger x(5)

1

2
3
4
5

6

7
8
9
10
11
12
13
14
15
16

!$xmp loop on t(i)do i=1,1010h(i)=iend do!$xmp gmove
g(:)=h(:)
15 !$xmp gmovex(:)=g(6:10)

E.1.3

Constraints for Asynchronous Communications

The constructs reflect, gmove (and its following assignment statement), reduction, and bcast
are asynchronous if asyncs are specified. Completions of data read and written by these global

17

18
19

E.1. EXECUTION TRACES

153

communication constructs are not guaranteed until wait asyncs are executed. This is represented by iii in Figure E.1.
3
For example, in the following code, the assignment statement g(:)=h(:) may not be com4
pleted before the second gmove construct is executed as the first gmove construct has async
5clause. Therefore, the value of g(i) is not guaranteed to be i+5. Of course, the value of x(i)6on p(1) is not guaranteed to be i+5.
XcalableMP Fortran!$xmp nodes p(2)!$xmp template t(10)!$xmp distribute (block) onto p :: tinteger :: g(10), h(10)
5 !$xmp align (i) with t(i) :: g, hinteger x(5)12!$xmp loop on t(i)do i=1,1010h(i)=iend do!$xmp gmove async(1)g(:)=h(:)
15 !$xmp gmovex(:)=g(6:10)!$xmp wait_async(1)The wait async(async-id ) guarantees the completion of a global communication constructthat has async-id. Therefore, the value of x(i) is not guaranteed to be i+5 in the following
9
program:
XcalableMP Fortran!$xmp nodes p(2)!$xmp template t(10)!$xmp distribute (block) onto p :: tinteger :: g(10), h(10)
5 !$xmp align (i) with t(i) :: g, hinteger x(5)78!$xmp loop on t(i)do i=1,1010h(i)=iend do!$xmp gmove async(1)g(:)=h(:)
15 !$xmp wait_async(1)!$xmp gmovex(:)=g(6:10)

10
11

Assignment statements in local view and gmove constructs in global view may race. The
value of x(5) is not guaranteed to be 6, and may be 10 in the following program:

154

APPENDIX E. MEMORY CONSISTENCY MODEL

XcalableMP Fortran!$xmp nodes p(2)!$xmp template t(10)!$xmp distribute (block) onto p :: tinteger :: g(10), h(10)
5 !$xmp align (i) with t(i) :: g, hinteger x(5)integer l(5), m(5)!$xmp local_alias l => g
10 !$xmp local_alias m => h!$xmp loop on t(i)do i=1,10h(i)=i15
end do!$xmp gmove async(1)g(:)=h(:)l(5)=6
20 !$xmp wait_async(1)x(5)=l(5)By avoiding the race, the value of x(5) is guaranteed to be 6 as follows:
XcalableMP Fortran!$xmp nodes p(2)!$xmp template t(10)!$xmp distribute (block) onto p :: tinteger :: g(10), h(10)
5 !$xmp align (i) with t(i) :: g, hinteger x(5)integer l(5), m(5)!$xmp local_alias l => g!$xmp local_alias m => h1

10!$xmp loop on t(i)do i=1,10h(i)=iend do15!$xmp gmove async(1)g(:)=h(:)!$xmp wait_async(1)l(5)=620
x(5)=l(5)Please note that function calls have no synchronization at its entrance/exit. In the following
program, the value of x(5) is not guaranteed to be 6:
XcalableMP Fortran!$xmp nodes p(2)!$xmp template t(10)23E.1. EXECUTION TRACES!$xmp distribute (block) onto p :: tinteger :: g(10), h(10)
5 !$xmp align (i) with t(i) :: g, hinteger x(5)integer l(5), m(5)!$xmp local_alias l => g!$xmp local_alias m => h10!$xmp loop on t(i)do i=1,10h(i)=iend do15!$xmp gmove async(1)call sub(g,h)l(5)=6!$xmp wait_async(1)20x(5)=l(5)

155

1

2

3

Appendix F

DRAFT: Tasklet of upcoming
XcalableMP 2.0

6This chapter shows a draft of the specification of the tasklet features from upcoming XcalableMP
2.0. The models and any other rules of XcalableMP 2.0 follows those of XcalableMP unlessspecified.

7

F.1

4
5

XcalableMP Extended Execution Model

18

When a node encounters a tasklets construct at runtime, a set of threads are created, an
implicit tasklet for structured-block is generated on the node, and one of the threads begins
execution of it.
When a thread encounters a tasklet construct at runtime, a new tasklet is generated on thenode. Execution of generated tasklets is assigned to one of the threads on the node, subject tothe thread’s availability to execute work. Thus, execution of the new tasklet could be immediate,or deferred until later according to the tasklet scheduling constraint (Section F.3.2) and thread
availability.
At the end of the tasklets construct, there is an implicit taskletwait construct to complete
all of the tasklets generated in the construct, after which the threads created at the beginningof the construct are terminated, and the node resumes the execution.

19

F.2

Glossary

20

F.2.1

Node Terminology

8
9
10
11
12
13
14
15
16
17

node A logical entity managed by the XcalableMP runtime system, which
has its own local memory and can communicate with each other, andon which one or more threads can execute inside the tasklets region.

21
22
23

24

F.2.2

thread An execution entity of tasklets, which execute on a node inside atasklets region.

25
26

27

28
29

Thread Terminology

F.2.3

Tasklet Terminology
tasklet A specific instance of executable code and its data environment, generated when a thread encounters a tasklet construct.
157

158

APPENDIX F. DRAFT: TASKLET OF UPCOMING XCALABLEMP 2.0
taskletdependence An ordering relation between two sibling tasklets: the dependent
tasklet and a previously generated predecessor tasklet. The tasklet
dependence is fulfilled when the predecessor tasklet has completed.

123

dependent
tasklet A tasklet that because of a tasklet dependence cannot be executed
until its predecessor tasklets have completed.

4predecessor
tasklet A tasklet that must complete before its dependent tasklets can be
executed.

6

5

7

F.3

Directives

8

F.3.1

Tasklet Constructs

9

F.3.1.1

tasklets Construct10

Synopsis
The tasklets construct starts a region for tasklet execution. See Section ?? for a generaldescription of the XMP’s tasklet execution model.
Syntax
[F]

11

12
13

14

!$xmp tasklets [on {nodes-ref | template-ref}]structured-block!$xmp end tasklets
15[C]#pragma xmp tasklets [on {nodes-ref | template-ref}]structured-block

Description
When a node encounters a tasklets construct at runtime, an implicit tasklet for structured-blockis created and a thread on the node begins execution of it. There is an implicit taskletwait
construct at the end of the tasklets construct.
In a tasklets region, only directives listed below can be specified.

16

17
18
19
20

• tasklet

21• taskletyield22• taskletwait23• tasklet reflect24• tasklet gmove

25• tasklet bcast

26• tasklet reduction

27• tasklet reduce_shadow

28

F.3. DIRECTIVES

159

3

Note that a tasklets region may not contain another tasklets construct, that is, tasklets
cannot be nested.
In addition, no OpenMP directives can be specified in a tasklets region.

4

Restrictions

1
2

7• The tasklets construct is global, which means that it must be executed by all nodes in
the current executing node set, and each local variable referenced in the construct musthave the same value.

8• The node set specified by the on clause must be a subset of the executing node set.

56

9

F.3.1.2

tasklet Construct10

Synopsis

11

The tasklet construct defines a tasklet that will be executed by a specified node set.

12

Syntax[F]!$xmp tasklet [on {nodes-ref | template-ref}] [ depend-clause ]...
structured-block!$xmp end tasklet[C]#pragma xmp tasklet [on {nodes-ref | template-ref}] [ depend-clause ]...
structured-block13

14

where depend-clause is one of:15

in ( variable [, variable]... )out ( variable [, variable]... )inout ( variable [, variable]... )16

pro_post ( {nodes-ref | template-ref} [, tag] )pro_wait [( {nodes-ref | template-ref} [, tag] )]epi_post ( {nodes-ref | template-ref} [, tag] )epi_wait [( {nodes-ref | template-ref} [, tag] )]17

remote_in [( variable [, {nodes-ref | template-ref}] [, tag] )]remote_out [( variable [, {nodes-ref | template-ref}] [, tag] )]accept_remote_in ( variable, {nodes-ref | template-ref} [, tag] )accept_remote_out ( variable, {nodes-ref | template-ref} [, tag] )

18

19
20
21
22
23
24
25
26

Description
When a thread encounters a tasklet construct at runtime, it generates a tasklet from the codefor the associated block and put the tasklet into the tasklet pool if it is included by the node
set specified by the on clause; otherwise, it skips the block.
If the on clause is omitted, it is assumed that the nodes that owns a variable selected by the
implementation from the variables specified in the out clause, if any, is specified in it; otherwise
it is assumed that the current executing node set is specified in it.
The tasklet may have some scheduling constraints defined by depend-clause. The specification of each depend-clause is described in the following sections.

160

APPENDIX F. DRAFT: TASKLET OF UPCOMING XCALABLEMP 2.0

Restrictions• The node set specified by nodes-ref or template-ref in the on clause must be a subset ofthe parent node set.
F.3.1.3

in/out/inout Clauses

1

2
3

4

Synopsis

5

The in, out, and inout clauses specify dependences of tasklets within a node.

6

Syntax7

in ( variable [, variable]... )out ( variable [, variable]... )inout ( variable [, variable]... )Description
The in, out, and inout clauses work in the same way as the depend clause of the task directive
in OpenMP 4.0 or later, with regard to dependences of tasklets within a node.
The in clause. The generated tasklet will be a dependent tasklet of all previously generated
sibling tasklets that reference at least one of the list items in an out or inout clause.
The out and inout clauses. The generated tasklet will be a dependent tasklet of all previously generated sibling tasklets that reference at least one of the list items in an in, out, or
inout clause.
Restrictions• variables used in these clauses of the same tasklet or sibling tasklets must indicate identicalstorage locations or disjoint storage locations.
• variables cannot be zero-length array sections.
• A variable that is part of another variable (such as an element of a structure) but is not
an array element or an array section cannot appear in these clause.
F.3.1.4

pro post Clause

Synopsis
The pro_post clause, in combination with the pro_wait or epi_wait construct, specifies apoint-to-point synchronization between tasklets.
Syntaxpro_post ( {nodes-ref | template-ref} [, tag] )Description
This clause ensures that the predecessor tasklets have completed on the local node before tasklets
that have a matching pro_wait clause are scheduled, or tasklets that have a matching epi_wait
clause are completed on remote nodes.
A pro_post clause having the arguments of nodes-ref/template-ref and tag, of a tasklet on
a node (called a posting node) dynamically matches at most one pro_wait or epi_wait clause

8

9

10
11
12
13
14
15
16

17

18
19

20

21
22

23

24

25
26

27

28

29

30
31
32
33
34

F.3. DIRECTIVES

161

2

having the arguments of the posting node (unless omitted) and the same value as tag (unlessomitted), of a tasklet on the node specified by nodes-ref/template-ref.

3

Restrictions

1

4
5• tag must be an expression of type default integer, in XcalableMP Fortran, or type int, in
XcalableMP C.

6

F.3.1.5

7

Synopsis

pro wait Clause

9

The pro_wait clause, in combination with the pro_post or epi_post clause, specifies a pointto-point synchronization between tasklets.

10

Syntax81112

pro_wait [( {nodes-ref | template-ref} [, tag] )]Description

23

This clause prohibits the tasklet from being scheduled on the local node until tasklets that havea matching pro_post clause are scheduled, or tasklets that have a matching epi_post clauseare completed on remote nodes.
A pro_wait clause having the arguments of nodes-ref/template-ref and tag, of a tasklet on
a node (called a waiting node) dynamically matches a pro_post clause having the argumentsof the waiting node and the same value as tag, of a tasklet on the node specified by nodesref/template-ref.
If tag is omitted, then the pro_wait construct can match a pro_post or epi_post clausehaving the arguments of the waiting node and any tag, of tasklet on the node specified by nodesref/template-ref. If both nodes-ref/template-ref and tag are omitted, then the pro_wait clausecan match a pro_post or epi_post clause having the arguments of any node and any tag.

24

Restrictions

13
14
15
16
17
18
19
20
21
22

25
26

• tag must be an expression of type default integer, in XcalableMP Fortran, or type int, in
XcalableMP C.

27

F.3.1.6

28

Synopsis

epi post Clause

30

The epi_post clause, in combination with the pro_wait or epi_wait construct, specifies apoint-to-point synchronization between tasklets.

31

Syntax29

32

epi_post ( {nodes-ref | template-ref} [, tag] )162

APPENDIX F. DRAFT: TASKLET OF UPCOMING XCALABLEMP 2.0

Description
This clause ensures that the tasklet have completed on the local node before tasklets that havea matching pro_wait clause are scheduled, or tasklets that have a matching epi_wait clauseare completed on remote nodes.
A pro_post clause having the arguments of nodes-ref/template-ref and tag, of a tasklet on
a node (called a posting node) dynamically matches at most one pro_wait or epi_wait clausehaving the arguments of the posting node (unless omitted) and the same value as tag (unlessomitted), of a tasklet on the node specified by nodes-ref/template-ref.
Restrictions• tag must be an expression of type default integer, in XcalableMP Fortran, or type int, in
XcalableMP C.
F.3.1.7

epi wait Clause

Synopsis

2
3
4
5
6
7
8

9

10
11

12

13

The epi_wait clause, in combination with the pro_post or epi_post clause, specifies a pointto-point synchronization between tasklets.
Syntax1415

16

epi_wait [( {nodes-ref | template-ref} [, tag] )]Description
This clause prohibits the tasklet from being completed on the local node until tasklets that havea matching pro_post clause are scheduled, or tasklets that have a matching epi_post clauseare completed on remote nodes.
A pro_wait clause having the arguments of nodes-ref/template-ref and tag, of a tasklet on
a node (called a waiting node) dynamically matches a pro_post clause having the argumentsof the waiting node and the same value as tag, of a tasklet on the node specified by nodesref/template-ref.
If tag is omitted, then the pro_wait construct can match a pro_post or epi_post clausehaving the arguments of the waiting node and any tag, of tasklet on the node specified by nodesref/template-ref. If both nodes-ref/template-ref and tag are omitted, then the pro_wait clausecan match a pro_post or epi_post clause having the arguments of any node and any tag.
Restrictions• tag must be an expression of type default integer, in XcalableMP Fortran, or type int, in
XcalableMP C.
F.3.1.8

1

remote in Clauses

Synopsis
The remote_in clause specifies a predecessor tasklet with regard to a data dependency derived
from a remote-read (get) operation.

17

18

19
20
21
22
23
24
25
26
27
28
29

30

31
32

33

34

35
36

F.3. DIRECTIVES

1

2

3

163

Syntaxremote_in [( {variable | *} [, {nodes-ref | template-ref}] [, tag] )]Description

10

A remote_in clause can be regarded as a syntactic sugar for the combination of in, pro_wait,
and epi_post clauses.
If * is specified as the first argument or all of the arguments are omitted, no in clause isimplied.
If the second argument is omitted, the implied pro_wait clause will match the first incoming
pro_post or epi_post clause from a node and the implied epi_post will work as if the node
was specified in it.

11

Restrictions

4
5
6
7
8
9

13

• variables used in these clauses of the same tasklet or sibling tasklets must indicate identicalstorage locations or disjoint storage locations.

14• variables cannot be zero-length array sections.

12

15
16

17
18

• A variable that is part of another variable (such as an element of a structure) but is not
an array element or an array section cannot appear in these clause.
• tag must be an expression of type default integer, in XcalableMP Fortran, or type int, in
XcalableMP C.

19

F.3.1.9

20

Synopsis

accept remote in Clause

22

The accept_remote_in clause specifies a dependent tasklet with regard to a data dependency
derived from a remote-read (get) operation.

23

Syntax

21

24

25

26
27

28
29
30

31

accept_remote_in ( variable, {nodes-ref | template-ref} [, tag] )Description
A accept_remote_in clause can be regarded as a syntactic sugar for the combination of in,
pro_post, and epi_wait clauses.
(Advice to implementers) The action for the implied epi_wait clause may bedeferred until a dependent tasklet of the tasklet with respect to the the implied inclause is scheduled.
Restrictions

33• variables used in these clauses of the same tasklet or sibling tasklets must indicate identicalstorage locations or disjoint storage locations.

34• variables cannot be zero-length array sections.

32

164

APPENDIX F. DRAFT: TASKLET OF UPCOMING XCALABLEMP 2.0• A variable that is part of another variable (such as an element of a structure) but is not
an array element or an array section cannot appear in these clause.
• tag must be an expression of type default integer, in XcalableMP Fortran, or type int, in
XcalableMP C.
F.3.1.10

remote out Clause

Synopsis

2

3
4

5

6

The remote_out clause specifies a predecessor tasklet with regard to a data dependency derived
from a remote-write (put) operation.
Syntax

78

9remote_out [( {variable | *} [, {nodes-ref | template-ref}] [, tag] )]Description
A remote_out clause can be regarded as a syntactic sugar for the combination of out, pro wait,
and epi post clauses.
If * is specified as the first argument or all of the arguments are omitted, no in clause isimplied.
If the second argument is omitted, the implied pro_wait clause will match the first incoming
pro_post or epi_post clause from a node and the implied epi_post will work as if the node
was specified in it.
Restrictions• variables used in these clauses of the same tasklet or sibling tasklets must indicate identicalstorage locations or disjoint storage locations.
• variables cannot be zero-length array sections.
• A variable that is part of another variable (such as an element of a structure) but is not
an array element or an array section cannot appear in these clause.
• tag must be an expression of type default integer, in XcalableMP Fortran, or type int, in
XcalableMP C.
F.3.1.11

1

accept remote out Clause

Synopsis
The accept_remote_out clause specifies a dependent tasklet with regard to a data dependency
derived from a remote-write (put) operation.
Syntaxaccept_remote_out ( variable, {nodes-ref | template-ref} [, tag] )

10

11

12
13
14
15
16
17
18

19

20
21

22

23
24

25
26

27

28

29
30

31

32

F.3. DIRECTIVES

1

2
3

Description
A accept_remote_out clause can be regarded as a syntactic sugar for the combination of out,
pro_post, and epi_wait clauses.
(Advice to implementers) The action for the implied pro_post clause may be
advanced after the scheduling constraint enforced by the implied out clause is met.

4
5

6

165

Restrictions

8• variables used in these clauses of the same tasklet or sibling tasklets must indicate identicalstorage locations or disjoint storage locations.

9• variables cannot be zero-length array sections.

7

10
11

12
13

• A variable that is part of another variable (such as an element of a structure) but is not
an array element or an array section cannot appear in these clause.
• tag must be an expression of type default integer, in XcalableMP Fortran, or type int, in
XcalableMP C.

14

F.3.1.12

15

Synopsistaskletyield Construct

17

The taskletyield construct specifies that the current tasklet can be suspended in favor of
execution of a diﬀerent tasklet.

18

Syntax16[F]!$xmp taskletyield[C]#pragma xmp taskletyield

19

20

Description

21

The taskletyield construct includes an explicit task scheduling point in the current tasklet.

22

F.3.1.13

23

Synopsistaskletwait Construct

25

The taskletwait construct specifies a wait on the completion of child tasklets of the current
tasklet.

26

Syntax24[F]!$xmp taskletwait[C]#pragma xmp taskletwait

27

28

29
30
31

Description
The taskletwait construct includes an implicit task scheduling point in the current tasklet.
The current tasklet is suspended at the tasklet scheduling point until all child tasks that it
generated before the taskletwait construct complete execution.

166

APPENDIX F. DRAFT: TASKLET OF UPCOMING XCALABLEMP 2.0

F.3.2

Tasklet Scheduling

Whenever a thread reaches a tasklet scheduling point, the implementation may cause it to perform a tasklet switch, beginning or resuming execution of a diﬀerent tasklet. Tasklet schedulingpoints are implied at the following locations:• the point immediately following the generation of a tasklet• the point of completion of a tasklet that has one or more implicit or explicit epi_wait
clauses

1

2
3
4

5

6
7

• after the point of completion of a tasklet

8• at a taskletyield construct

9• at an implicit and explicit taskletwait construct

10

When a thread encounters a tasklet scheduling point, it may do either of the following,
subject to the Tasklet Scheduling Constraints (below):

11
12• begin execution of a tasklet

13• resume any suspended tasklet

14

If more than one of the above choices is available, it is unspecified as to which will be chosen.
Tasklet Scheduling Constraints are as follows:
1. A dependent tasklet shall not be scheduled until its tasklet dependences are fulfilled.
2. A tasklet that has one or more implicit or explicit pro_wait clauses shall not be scheduled
until all of the pro_wait clauses are matched with pro_post or epi_post clauses.

F.3.3

Communication Tasklet Constructs

F.3.3.1

Overview

The communication tasklet constructs define inter-node interactions between tasklets on thebasis of XMP’s global-view communication constructs.
They are executed as a set of tasklets, each of which produces the same eﬀect on a node as
the corresponding global-view communication construct does.
(Advice to implementers) To implement the above feature of the communication
tasklet constructs, they should be based on remote read (get) operations and pointto-point synchronizations.
F.3.3.2

tasklet reflect ConstructSynopsis
The tasklet reflect construct defines tasklets that update the shadow area of a global array
like the reflect directive.

15
16

17

18
19

20

21

22
23
24
25

26
27
28

29

30

31
32

F.3. DIRECTIVES

1

Syntax
[F]

2

3

167

[C]

!$xmp tasklet reflect ( array-name [, array-name]... )[width ( reflect-width [, reflect-width]... )] [orthogonal][on {nodes-ref | template-ref}]#pragma xmp tasklet reflect ( array-name [, array-name]... )[width ( reflect-width [, reflect-width]... )] [orthogonal][on {nodes-ref | template-ref}]Description

10

The tasklet reflect construct generates a tasklet on each of the nodes specified by the onclause, which will produce the same eﬀect as the reflect construct having the same clauses
does. In addition, for each of the arrays specified by the sequence of array-names, an inoutclause for it implicitly added to the generated tasklets.
Note that tasklet reflect is a local construct, unlike reflect, and therefore the node set
specified by the on clause need not include all of the nodes onto which the target arrays are
mapped.

11

Restrictions

4
5
6
7
8
9

13

• The reflect width of each dimension specified by the reflect-width must not exceed the
shadow width of the arrays.

14• The node set specified by the on clause must be a subset of the executing node set.

12

15

Example16

F.3.3.3

17

Synopsistasklet gmove Construct

19

The tasklet gmove construct defines tasklets that copy the variable from the right-hand side(rhs) into the left-hand side (lhs) of the associated assignment statement like the gmove construct.

20

Syntax

18

21

22

[F]
[C]

!$xmp tasklet gmove [on {nodes-ref | template-ref}]#pragma xmp tasklet gmove [on {nodes-ref | template-ref}]Description

29

The tasklet gmove construct generates a tasklet on each of the nodes specified by the onclause, which will produce the same eﬀect as the gmove construct having the same clauses does.
In addition, for the variables on the right-hand and left-hand side of the associated assignment
statement, in and out clauses are implicitly added to the generated tasklets, respectively,
Note that tasklet gmove is a local construct, unlike gmove, and therefore the node set
specified by the on clause need not include all of the nodes onto which global arrays appearing
in the associated statement, if any, are mapped.

30

Restrictions

23
24
25
26
27
28

31
32

• The tasklet gmove construct must be followed by (i.e., associated with) a simple assignment statement that contains neither arithmetic operations nor function calls.

168

APPENDIX F. DRAFT: TASKLET OF UPCOMING XCALABLEMP 2.0• The node set specified by the on clause must be a subset of the executing node set.
F.3.3.4

tasklet bcast ConstructSynopsis

Syntax[C]

Description
The tasklet bcast construct generates a tasklet on each of the nodes specified by the onclause, which will produce the same eﬀect as the bcast construct having the same clauses does.
In addition, for each of the variables specified by the sequence of variables, an inout clause for
it implicitly added to the generated tasklets.
Note that tasklet bcast is a local construct, unlike bcast.
Restrictions• The variables specified by the sequence of variables must either not be aligned or must be
replicated among nodes of the node set specified by the on clause.
• The node set specified by the on clause must be a subset of the executing node set.
• The source node specified by the from clause must belong to the node set specified by theon clause.
• The source node specified by the from clause must be one node.
tasklet reduction ConstructSynopsis
The tasklet reduction construct defines tasklets that perform a reduction operation like thereduction construct.
Syntax
[F]
[C]

4
5

6

!$xmp tasklet bcast ( variable [, variable]... ) [from nodes-ref | template-ref][on nodes-ref] | template-ref]#pragma xmp tasklet bcast ( variable [, variable]... ) [from nodes-ref | template-ref][on nodes-ref | template-ref]F.3.3.5

2

3

The tasklet bcast construct defines tasklets that perform broadcast communication from a
specified node like the bcast construct.

[F]1!$xmp tasklet reduction ( reduction-kind : variable [, variable ]... )[on node-ref | template-ref]#pragma xmp tasklet reduction ( reduction-kind : variable [, variable ]... )[on node-ref | template-ref]

7

8

9
10
11
12
13

14

15
16

17

18
19

20

21

22

23
24

25

26

F.3. DIRECTIVES

1

169

Description

6

The tasklet reduction construct generates a tasklet on each of the nodes specified by the onclause, which will produce the same eﬀect as the reduction construct having the same clauses
does. In addition, for each of the variables specified by the sequence of variables, an inoutclause for it implicitly added to the generated tasklets.
Note that tasklet reduction is a local construct, unlike reduction.

7

Restrictions

2
3
4
5

9

• The variables specified by the sequence of variables must either not be aligned or must be
replicated among nodes of the node set specified by the on clause.

10• The node set specified by the on clause must be a subset of the executing node set.

8

11

F.3.3.6

12

Synopsistasklet reduce shadow Construct

14

The tasklet reduce_shadow construct defines tasklets that add values of shadow objects totheir reflection source like the reduce_shadow construct.

15

Syntax

13

[F]

16

17

[C]

!$xmp tasklet reduce shadow ( array-name [, array-name]... )[width ( reflect-width [, reflect-width]... )] [orthogonal][on {nodes-ref | template-ref}]#pragma xmp tasklet reduce shadow ( array-name [, array-name]... )[width ( reflect-width [, reflect-width]... )] [orthogonal][on {nodes-ref | template-ref}]Description

24

The tasklet reduce_shadow construct generates a tasklet on each of the nodes specified by
the on clause, which will produce the same eﬀect as the reduce_shadow construct having thesame clauses does. In addition, for each of the arrays specified by the sequence of array-names,
an inout clause for it implicitly added to the generated tasklets.
Note that tasklet reduce_shadow is a local construct, unlike reduce_shadow, and therefore
the node set specified by the on clause need not include all of the nodes onto which the target
arrays are mapped.

25

Restrictions

18
19
20
21
22
23

27

• The width of each dimension specified by reflect-width must not exceed the shadow widthof the arrays.

28• The node set specified by the on clause must be a subset of the executing node set.

26

1

2

3

Appendix G

Sample Programs
Example 1
XcalableMP C/** A parallel explicit solver of Laplace equation in \XMP*/#pragma xmp nodes p(NPROCS)
5 #pragma xmp template t(1:N)#pragma xmp distribute t(block) onto p
double u[XSIZE+2][YSIZE+2],
uu[XSIZE+2][YSIZE+2];
10 #pragma xmp align u[i][*] to t(i)#pragma xmp align uu[i][*] to t(i)#pragma xmp shadow uu[1:1][0:0]
lap_main(){
int x,y,k;double sum;for(k = 0; k < NITER; k++){/* old <- new */20 #pragma xmp loop on t(x)for(x = 1; x <= XSIZE; x++)for(y = 1; y <= YSIZE; y++)uu[x][y] = u[x][y];#pragma xmp reflect (uu)
25 #pragma xmp loop on t(x)for(x = 1; x <= XSIZE; x++)for(y = 1; y <= YSIZE; y++)u[x][y] = (uu[x-1][y] + uu[x+1][y] +uu[x][y-1] + uu[x][y+1])/4.0;30}
15

sum = 0.0;
#pragma xmp loop on t[x] reduction(+:sum)for(x = 1; x <= XSIZE; x++)35
for(y = 1; y <= YSIZE; y++)
171

172

APPENDIX G. SAMPLE PROGRAMS

sum += (uu[x][y]-u[x][y]);#pragma xmp task on p(1)printf("sum = %g\n",sum);}
Example 2

1

XcalableMP C/** Linpack in XcalableMP (Gaussian elimination with partial pivoting)*
1D distribution version*/
5 #pragma xmp nodes p(*)#pragma xmp template t(0:LDA-1)#pragma xmp distribute t(cyclic) onto p
double pvt_v[N];// local10/*
gaussian elimination with partial pivoting*/
dgefa(double a[n][LDA],int lda, int n,int ipvt,int *info)#pragma xmp align a[:][i] with t(i){15
REAL t;int idamax(),j,k,kp1,l,nm1,i;
REAL x_pvt;

20

nm1 = n - 1;
for (k = 0; k < nm1; k++) {kp1 = k + 1;/* find l = pivot indexl = A_idamax(k,n-k,a[k]);ipvt[k] = l;*/25/* if (a[k][l] != ZERO) */#ifdef XMP#pragma xmp gmove
pvt_v[k:n-k] = a[l][k:n-k];
30 #elsefor(i = k; i < n; i++) pvt_v[i] = a[i][l];#endif/* interchange if necessary */if (l != k){#ifdef XMP#pragm xmp gmove
a[l][:] = a[k][:];#pramga xmp gmove
40
a[k][:] = pvt_v[:];#else
for(i = k; i< n; i++) a[i][l] = a[i][k];for(i = k; i< n; i++) a[i][k] = pvt_v[i];
35

173

45

#endif}/* compute multipliers */t = -ONE/pvt_v[k];A_dscal(k+1, n-(k+1),t,a[k]);/* row elimination with column indexing */for (j = kp1; j < n; j++) {t = pvt_v[j];A_daxpy(k+1,n-(k+1),t,a[k],a[j]);}50}
ipvt[n-1] = n-1;

55

}

60

dgesl(double a[n][LDA],int lda,int n,int pvt[n],double b,int job)#pragma xmp align a[:][i] with t(i)#pragma xmp align b[i] with t(i){REAL t;int k,kb,l,nm1;

65

nm1 = n - 1;/* job = 0 , solve a * x = b, first solvefor (k = 0; k < nm1; k++) {l = ipvt[k];70 #pragma xmp gmove
t = b[l];if (l != k){#pragma xmp gmove
b[l] = b[k];75 #pragma xmp gmove
b[k] = t;}A_daxpy(k+1,n-(k+1),t,a[k],b);}80/* now solve u*x = y */for (kb = 0; kb < n; kb++) {k = n - (kb + 1);#pragma xmp task on t(k)85 {b[k] = b[k]/a[k][k];t = -b[k];}#pragma xmp bcast (t) from t(k)90A_daxpy(0,k,t,a[k],b);}}l*y = b*/

174

APPENDIX G. SAMPLE PROGRAMS/** distributed array based routine*/
A_daxpy(int b,int n,double da,double dx[n],double dy[n])#pragma xmp align dx[i] with t(i)#pragma xmp align dy[i] with t(i)100 {int i,ix,iy,m,mp1;
if(n <= 0) return;if(da == ZERO) return;/* code for both increments equal to 1 */105 #pragma xmp loop on t(b+i)
for (i = 0;i < n; i++) {dy[b+i] = dy[b+i] + da*dx[b+i];
}
}
95

110

int A_idamax(int b,int n,double dx[n])#pragma xmp align dx[i] with t(i){double dmax, g_dmax;
115
int i, ix, itemp;
if(n == 1) return(0);/* code for increment equal to 1 */itemp = 0;120
dmax = 0.0;
#pragma xmp loop on t(i) reduction(lastmax:dmax/itemp/)
for (i = b; i < n; i++) {if(fabs((double)dx[i]) > dmax) {itemp = i;125
dmax = fabs((double)dx[i]);}}return (itemp);}
130

A_dscal(int b,int n,double da,double dx[n])#pragma xmp align dx[i] with t(i)#pragma xmp align dy[i] with t(i){135
int i;
if(n <= 0)return;/* code for increment equal to 1 */#pragma xmp loop on t(i)140
for (i = b; i < n; i++)dx[i] = da*dx[i];}Index/periodic/ modifier, 49, 58/unbound/ modifier, 40
address-of operator, 17align, 29align dummy variable, 29align oﬀset, 29alignment, 12
allocation image set, 13
array, 47
array assignment in XMP/C, 16
array intrinsic functions, 109
array section in XMP/C, 15
async clause, 57
asynchronous communication, 13
barrier, 52
base language, 9base program, 9bcast, 55
block, 27
broadcast variables, 56
built-in elemental functions, 109
built-in functions of XMP/C, 17
built-in transformational procedures, 109
coarray reference, 72
collapse, 30
collective mode (of gmove), 50
combined directive, 21
communication, 12
construct, 10
current executing node set, 6, 11
cyclic, 27
data mapping, 10
declarative directive, 10
declarative directives, 19
dependent tasklet, 158
descriptor, 18
descriptor association, 82, 85
descriptor-of operator, 18, 93
Directivealign, 29
array, 47
async clause, 57
barrier, 52
bcast, 55
distribute, 26
gmove, 50
local alias, 73
lock, 78loop, 38, 55
nodes, 22post, 76, 77
reduce shadow, 57
reduction, 52
reflect, 48
shadow, 31
task, 35, 37
tasklet, 159
tasklets, 158
tasks, 35, 36
template, 24
unlock, 78
wait, 76, 77
wait async, 56
directive, 9, 19, 158
distribute, 26
distribution, 12
distribution format*, 27
block, 27
cyclic, 27
gblock, 27
entire image set, 13
entire node array, 11
entire node set, 6, 11
Examplealign, 31, 129
array, 47
array assignment in XMP/C, 17
array section in XMP/C, 16
async, 57
175

176

coarray, 72
coarray reference, 72
distribute, 129
dynamic allocation in XMP/C, 17
end task, 35, 37
end tasks, 37
gmove, 51
library interface, 127
local alias, 75
loop, 41, 42, 129
memory-layout, 131
MPI interface, 120
node reference, 24
nodes, 23, 129
OpenMP in XcalableMP programs, 115
post, 77
procedure interface, 82, 87
reduce shadow, 58
reduction, 54
reflect, 49
shadow, 33, 49
task, 35, 37, 42
tasks, 37
template, 26, 129
template fix, 17, 34, 76
wait, 77
wait async, 57
xmp desc of, 17
xmp malloc, 17
executable directive, 10
executable directives, 19
executing image set, 13
executing node, 11
executing node array, 11
executing node set, 6, 11
full shadow, 32
gblock, 27
global, 10
global actual argument, 81
global communication constructs, 10
global construct, 10
global constructs, 6
global data, 6, 12
global dummy argument, 81
global-view model, 10
gmove, 50
image, 13
image index, 13

INDEX

image set, 13
in mode (of gmove), 50
Intrinsic and Library Procedures
xmp all node num, 94
xmp all num nodes, 94
xmp array gtol, 126
xmp exit, 97
xmp malloc, 17, 97
xmp node num, 95
xmp num nodes, 95
xmp test async, 97
xmp wtick, 96
xmp wtime, 96
xmpc all node num, 94
intrinsic transformational procedures, 109
Laplace, 171
library interface, 125
Linpack, 172
local, 10local actual argument, 81
local alias, 13
local data, 6, 12
local dummy argument, 81
local section, 12
local-view model, 10
local alias, 73
location-variable, 41
lock, 78loop, 38, 55
node, 11, 157
node array, 11node number, 11
node reference, 23, 24
node set, 11
nodes, 22
out mode (of gmove), 50
parent node set, 11
post, 76, 77
posting node, 76
predecessor tasklet, 158
procedure, 9
procedure interface, 81
reduce shadow, 57
reduction, 12
reduction, 52
reduction variable, 53
reflect, 48

INDEX

reflection source, 32
replicate, 30
replicated data, 12
replicated execution, 5Sample Program
Laplace, 171
Linpack, 172
sequence association, 82
shadow, 12
shadow, 31
shadow object, 32
source node, 56
structured block, 9
synchronization, 13
Syntaxaccept remote in, 163
accept remote out, 164align, 29
array, 47
array assignment in XMP/C, 16
array section in XMP/C, 15
barrier, 52
bcast, 56
coarray, 71
coarray reference, 72
directive, 19
distribute, 26
epi wait, 161, 162
gmove, 50
in, 160
inout, 160
local alias, 73
lock, 78loop, 38
node reference, 23
nodes, 22out, 160
post, 76pro post, 160
pro wait, 161
reduce shadow, 57
reduction, 53
reflect, 48
remote in, 163
remote out, 164
shadow, 31
task, 35
tasklet, 158, 159
tasklet bcast, 168

177

tasklet gmove, 167
tasklet reduce shadow, 169
tasklet reduction, 168
tasklet reflect, 167
taskletwait, 165
taskletyield, 165
tasks, 36
template, 25
template reference, 25
template fix, 33
wait, 77
wait async, 56
task, 7, 12
task, 35, 37
tasklet, 157
tasklet, 159
tasklet dependence, 158
tasklets, 158
tasks, 35, 36
template, 10
template, 24
template reference, 25
thread, 157
unlock, 78
variable, 12
wait, 76, 77
wait async, 56
waiting node, 77
work mapping, 10
work mapping constructs, 10
XcalableMP C, 9
XcalableMP Fortran, 9
xmp align axis, 105
xmp align offset, 105
xmp align replicated, 106
xmp align template, 106
xmp all node num, 94
xmp all num nodes, 94
xmp array gtol, 126
xmp array laddr, 127
xmp array lbound, 108
xmp array lshadow, 107
xmp array lsize, 126
xmp array ndims, 106
xmp array ubound, 108
xmp array ushadow, 107
xmp desc of, 18, 93

178

xmp dist axis, 104
xmp dist blocksize, 103
xmp dist gblockmap, 103
xmp dist format, 102
xmp dist nodes, 104
xmp exit, 97
xmp finalize, 121
xmp finalize mpi, 120
xmp gather, 111
xmp get mpi comm, 119
xmp init, 121
xmp init mpi, 119
xmp malloc, 17, 97
xmp matmul, 112
xmp node num, 95
xmp nodes size, 99
xmp nodes attr, 99
xmp nodes equiv, 100
xmp nodes index, 98
xmp nodes ndims, 98
xmp num nodes, 95
xmp pack, 111
xmp scatter, 110
xmp sort down, 113
xmp sort up, 113
xmp template fixed, 100
xmp template lbound, 101
xmp template ndims, 101
xmp template ubound, 102
xmp test async, 97
xmp transpose, 112
xmp unpack, 112
xmp wtick, 96
xmp wtime, 96
xmpc all node num, 94

INDEX