୪ิ᭷㝈せ⣲ἲ䛻䜘䜛୍ḟඖᐃᖖ⇕ఏᑟゎᯒ䝥䝻䜾䝷䝮
Fortran⦅୰ᓥ ◊࿃ᮾி኱Ꮫ᝟ሗᇶ┙䝉䞁䝍䞊

S2-ref• ၥ㢟ࡢᴫせ㸪ᐇ⾜᪉ἲ• ᒁᡤศᩓࢹ࣮ࢱࡢ⪃࠼᪉• ࣉࣟࢢ࣒ࣛࡢㄝ᫂
• ィ⟬౛

2

FEM1D

3ᑐ㇟࡜ࡍࡿၥ㢟㸸୍ḟඖ⇕ఏᑟၥ㢟య✚ᙜ䛯䜚୍ᵝⓎ⇕Q

w § wT · ¨O¸Qwx © wx ¹
x= xmaxx=0 (xmin)• ୍ᵝ࡞㸸᩿㠃✚A㸪⇕ఏᑟ⋡O• య✚ᙜࡓࡾ୍ᵝⓎ⇕㸦᫬㛫ᙜࡓࡾ㸧䛊QL-3T-1䛋
• ቃ⏺᮲௳
– x=0 㸸T= 0

㸦ᅛᐃ㸧

wT
– x=xmax㸸
wx

㸦᩿⇕㸧
0

Q

0

FEM1D

4ᑐ㇟࡜ࡍࡿၥ㢟㸸୍ḟඖ⇕ఏᑟၥ㢟య✚ᙜ䛯䜚୍ᵝⓎ⇕Q

w § wT · ¨O¸Qwx © wx ¹
x= xmaxx=0 (xmin)• ୍ᵝ࡞㸸᩿㠃✚A㸪⇕ఏᑟ⋡O• య✚ᙜࡓࡾ୍ᵝⓎ⇕㸦᫬㛫ᙜࡓࡾ㸧䛊QL-3T-1䛋
• ቃ⏺᮲௳
– x=0 㸸T= 0

㸦ᅛᐃ㸧

wT
– x=xmax㸸
wx

㸦᩿⇕㸧
0

Q

0

FEM1D5ゎᯒゎయ✚ᙜ䛯䜚୍ᵝⓎ⇕Q

w § wT · ¨O¸Qwx © wx ¹
x= xmaxx=0 (xmin)

T

wT
wx

0@ x

0

O T cc Q Q x  C  C

OT c

1

1

0@ x

Q x max , T c

1  2
O T  Q x  C1 x  C 2  C 2
2
1  2 Q x max
? T 
Qx 
x
2O
O

x max

0@ x

0, T

x max

0@ x

0

0

S2-ref6䝣䜯䜲䝹䝁䝢䞊䠈䝁䞁䝟䜲䝹䠄1/2䠅
ࢹ࢕ࣞࢡࢺࣜ⏕ᡂࣇ࢓࢖ࣝࢥࣆ࣮
>$>$ cd
cd <$T-fem2>
ྛ⮬సᡂࡋࡓࢹ࢕ࣞࢡࢺࣜ
>$pFEM>$ mkdir
cp /home/t00000/fem2/s2r.tar
.
>$
pFEM
>$ cd
tar
xvf s2r.tar
FORTRAN࣮ࣘࢨ࣮┤ୗ࡟ࠕmpi/S2-refࠖ࡜࠸࠺ࢹ࢕ࣞࢡࢺࣜࡀ࡛ࡁ࡚࠸ࡿࠋ
>$cd ~/pFEM<$T-fem2>/mpi/S2-refࢆ<$T-S2r>࡜࿧ࡪࠋ
>$ cp /home/ss/aics60/2014Summer/F/1d.tar .
>$ tar xvf 1d.tar

䝁䞁䝟䜲䝹
C࣮ࣘࢨ࣮
>$ cd <$T-S2r>>$ cd ~/pFEM>$ mpicc –Os –noparallel 1d.c>$ cp /home/ss/aics60/2014Summer/C/1d.tar .
>$ tar xvf 1d.tar

S2-ref7䝣䜯䜲䝹䝁䝢䞊䠈䝁䞁䝟䜲䝹䠄2/2䠅
ࢹ࢕ࣞࢡࢺࣜ☜ㄆ࣭ࢥࣥࣃ࢖ࣝ
ࣇ࢓࢖ࣝࢥࣆ࣮
>$>$cdcd~/pEFM/1d<$T-fem2>ྛ⮬సᡂࡋࡓࢹ࢕ࣞࢡࢺࣜ
>$>$mpifrtpx–Kfast 1d.f
cp /home/t00000/fem2/s2r.tar.
>$>$mpifccpx–Kfast1d.c
tar xvfs2r.tarS2-ref8ไᚚࣇ࢓࢖ࣝ㸸input.datไᚚࣇ࢓࢖ࣝ
4
1.0 1.0100
1.e-8

input.dat

1.0

NE㸦せ⣲ᩘ㸧'x㸦せ⣲㛗ࡉL㸧㸪Q, A, O཯᚟ᅇᩘ㸦CGἲᚋ㏙㸧CGἲࡢ཯᚟ᡴษㄗᕪ

1.0

'x=1
1
x=0

2
x=1

3
x=2

5

4

3

2

1

4
x=3

x=4

せ⣲␒ྕ
⠇Ⅼ␒ྕ䠄඲య䠅S2-ref9䝆䝵䝤䝇䝥䝸䜽䝖䠖go.sh#!/bin/sh#PJM -L "node=4"#PJM -L "elapse=00:10:00"#PJM -L "rscgrp=school"#PJM -j#PJM -o "test.lst"#PJM --mpi "proc=64"
mpiexec ./a.out

8ศ๭“node=1““proc=8”16ศ๭
“node=1““proc=16”32ศ๭
“node=2““proc=32”

64ศ๭
“node=4““proc=64”

192ศ๭
“node=12““proc=192”

S2-ref10䛂୪ิィ⟬䛃䛾ᡭ㡰• ไᚚ䝣䜯䜲䝹䠈䛂඲せ⣲ᩘ䛃䜢ㄞ䜏㎸䜐• ෆ㒊䛷䛂ᒁᡤศᩓ䝯䝑䝅䝳䝕䞊䝍䛃䜢⏕ᡂ䛩䜛• 䝬䝖䝸䜽䝇⏕ᡂ• ඹᙺ໙㓄ἲ䛻䜘䜚䝬䝖䝸䜽䝇䜢ゎ䛟• ඖ䛾䝥䝻䜾䝷䝮䛸䜋䛸䜣䛹ኚ䜟䜙䛺䛔

S2-ref• ၥ㢟ࡢᴫせ㸪ᐇ⾜᪉ἲ• ᒁᡤศᩓࢹ࣮ࢱࡢ⪃࠼᪉• ࣉࣟࢢ࣒ࣛࡢㄝ᫂
• ィ⟬౛

11

S2-ref12᭷㝈せ⣲ἲࡢฎ⌮㸸ࣉࣟࢢ࣒ࣛ• ึᮇ໬– ไᚚኚᩘㄞࡳ㎸ࡳ– ᗙᶆㄞࡳ㎸ࡳэせ⣲⏕ᡂ㸦N:⠇Ⅼᩘ㸪NE㸸せ⣲ᩘ㸧– 㓄ิึᮇ໬㸦඲య࣐ࢺࣜࢡࢫ㸪せ⣲࣐ࢺࣜࢡࢫ㸧– せ⣲э඲య࣐ࢺࣜࢡࢫ࣐ࢵࣆࣥࢢ㸦Index㸪Item㸧• ࣐ࢺࣜࢡࢫ⏕ᡂ– せ⣲༢఩ࡢฎ⌮㸦do icel= 1, NE㸧• せ⣲࣐ࢺࣜࢡࢫィ⟬• ඲య࣐ࢺࣜࢡࢫ࡬ࡢ㔜ࡡྜࢃࡏ– ቃ⏺᮲௳ࡢฎ⌮• 㐃❧୍ḟ᪉⛬ᘧ– ඹᙺ໙㓄ἲ㸦CG㸧ᅄゅᙧせ⣲
5

6

7

1

2

1

2

5

8
3

3

6

4
7

1
1
5

1

6

S2-ref

3
7

6

2
2

8

䛣䜜䛷䛿䝬䝖䝸䜽䝇⏕ᡂ䛻ᚲせ䛺᝟ሗ䛿୙༑ศ

3
2

1

䛂⠇Ⅼ䝧䞊䝇䠄㡿ᇦ䛤䛸䛾⠇Ⅼᩘ䛜䝞䝷䞁䝇䛩䜛䠅䛃䛾ศ๭⮬⏤ᗘ䠖⠇Ⅼୖ䛷ᐃ⩏

4
7

2
3

2

8
3

3

䝬䝖䝸䜽䝇⏕ᡂ䛾䛯䜑䛻䛿䠈䜸䞊䝞䞊䝷䝑䝥㒊ศ䛾せ⣲䛸⠇Ⅼ䛾᝟
ሗ䛜ᚲせ

4
13

୪ิ᭷㝈せ⣲ἲࡢᒁᡤࢹ࣮ࢱᵓ㐀⠇Ⅼ࣮࣋ࢫ㸸Node-based partitioningz ᒁᡤࢹ࣮ࢱ࡟ྵࡲࢀࡿࡶࡢ :
z

z
z
z

z

ࡑࡢ㡿ᇦ࡟ᮏ᮶ྵࡲࢀࡿ⠇Ⅼࡑࢀࡽࡢ⠇Ⅼࢆྵࡴせ⣲ᮏ᮶㡿ᇦእ࡛࠶ࡿࡀ㸪ࡑࢀࡽࡢせ⣲࡟ྵࡲࢀࡿ⠇Ⅼ⠇Ⅼࡣ௨ୗࡢ3✀㢮࡟ศ㢮zzzෆⅬ㸸Internal nodes
እⅬ㸸External nodes
ቃ⏺Ⅼ㸸Boundary nodesࡑࡢ㡿ᇦ࡟ᮏ᮶ྵࡲࢀࡿ⠇Ⅼᮏ᮶㡿ᇦእ࡛࠶ࡿࡀ࣐ࢺࣜࢡࢫ⏕ᡂ࡟ᚲせ࡞⠇Ⅼ௚ࡢ㡿ᇦࡢࠕእⅬࠖ࡜࡞ࡗ࡚࠸ࡿ⠇Ⅼz㡿ᇦ㛫ࡢ㏻ಙࢸ࣮ࣈࣝz㡿ᇦ㛫ࡢ᥋⥆ࢆࡢࡒࡃ࡜㸪኱ᇦⓗ࡞᝟ሗࡣ୙せz

S2-ref᭷㝈せ⣲ἲࡢ≉ᛶ㸸せ⣲࡛㛢ࡌࡓィ⟬
14

Node-based Partitioning
internal nodes - elements - external nodes
3(
3(






3(


























㸰

3(
S2-ref

㸱

㸲

3(






























































㸯



3(





































3(









3(


15

Node-based Partitioning
internal nodes - elements - external nodes
zPartitioned nodes themselves (Internal Nodes) ෆⅬzElements which include Internal Nodes ෆⅬ䜢ྵ䜐せ⣲zExternal Nodes included in the Elements እⅬin overlapped region among partitions.

zInfo of External Nodes are required for completely local
element–based operations on each processor.
15

8
S2-ref

6

7

14

13

4

5

10

1

2

3

9

11

12
16

S2-ref17୍ḟඖၥ㢟䠖11せ⣲䠈12⠇Ⅼ䠈3㡿ᇦ
1

1

1

1

1

1

2

2

2

2

2

2

3

3

3

3

3

3

4

4

4
4

4

4

4
4

5

5

5

5

6

6

6

6

7

7

7

7

8

8

8

8

9

9

9

9

10

10

10

10

11

11

11

11

12

12

5
5

5

6

6

7

7

8

8

8

8

9

9

9

10

10

11

11

12

S2-ref18୍ḟඖၥ㢟䠖11せ⣲䠈12⠇Ⅼ䠈3㡿ᇦᒁᡤ␒ྕ䠖⠇Ⅼ䞉せ⣲䛸䜒1䛛䜙䜅䜛

#0

1

1

2

2

3

#1

3

4

5

4

4

5

1

1

2

2

3

3

#2

4

5

5

4

6

1

1

2

2

3

3

4

S2-ref19୍ḟඖၥ㢟䠖11せ⣲䠈12⠇Ⅼ䠈3㡿ᇦእⅬ䞉ቃ⏺Ⅼ

#0

1

1

2

2

3

#1

3

4

5

4

4

5

1

1

2

2

3

3

#2

4

5

5

4

6

1

1

2

2

3

3

4

S2-ref20୍ḟඖၥ㢟䠖୍⯡ⓗ䛺ᒁᡤ␒ྕ䛾௜䛡᪉
1

N+1

N+1

N

N

1

1

1

1

1

2

2

2

2

2

2

…

…

…

N-1

N-1

N-1

N

N

N+1

N

N

#0: N+1⠇Ⅼ䠈Nせ⣲#PETot-1: N+1⠇Ⅼ䠈Nせ⣲

N+1

N+2

୍⯡䛾㡿ᇦ䠖
N+2⠇Ⅼ䠈N+1せ⣲S2-ref21୍ḟඖၥ㢟䠖11せ⣲䠈12⠇Ⅼ䠈3㡿ᇦ
せ⣲✚ศ䠈せ⣲䝬䝖䝸䜽䝇э඲య䝬䝖䝸䜽䝇
ෆⅬ䠈䛭䜜䜢ྵ䜐せ⣲䠈እⅬ䛷ྍ⬟
#0

1

1

2

2

3

#1

3

4

5

4

4

5

1

1

2

2

3

3

#2

4

5

5

4

6

1

1

2

2

3

3

4

S2-ref22๓ฎ⌮௜䛝ඹᙺ໙㓄ἲ
Preconditioned Conjugate Gradient Method 䠄CG䠅
Compute r(0)= b-[A]x(0)for i= 1, 2, …
solve [M]z(i-1)= r(i-1)
Ui-1= r(i-1) z(i-1)
if i=1p(1)= z(0)else
Ei-1= Ui-1/Ui-2p(i)= z(i-1) + Ei-1 p(i-1)
endifq(i)= [A]p(i)Di = Ui-1/p(i)q(i)x(i)= x(i-1) + Dip(i)r(i)= r(i-1) - Diq(i)check convergence |r|end๓ฎ⌮䠖ᑐゅ䝇䜿䞊䝸䞁䜾S2-ref23๓ฎ⌮䠈䝧䜽䝖䝹ᐃᩘಸ䛾ຍῶᒁᡤⓗ䛺ィ⟬䠄ෆⅬ䛾䜏䠅䛜ྍ⬟э୪ิฎ⌮1
&& ^]` >0LQY@^U`
GRL 1: L= : L''  : L5
HQGGR

2
3
4
5

&
& ^[` ^[`$/3+$ ^S`&^U` ^U` $/3+$ ^T`
GRL 1
3+, L 3+, L $/3+$ : L3: L5 : L5  $/3+$ : L4
HQGGR

6
7
8
9
10
11
12

S2-ref24ෆ✚඲య䛷࿴䜢䛸䜛ᚲせ䛜䛒䜛э㏻ಙ?
1&
& $/3+$ 5+2^S`^T`& G
GRL 1& &: L3 : L4
HQGGR$/3+$ 5+2&

2
3
4
5
6
7
8
9
10
11
12

S2-ref25⾜ิ䝧䜽䝖䝹✚እⅬ䛾್䠄᭱᪂䛾p䠅䛜ᚲせэ1ᑐ1㏻ಙ
&
& ^T` >$@^S`
GRL 1: L4  ',$* L : L3
GRM ,1'(; L ,1'(; L: L4  : L4 $0$7 M : ,7(0 M 3
HQGGR
HQGGR

5

1

2

3

4

6

S2-ref26୍ḟඖၥ㢟䠖11せ⣲䠈12⠇Ⅼ䠈3㡿ᇦእⅬ䞉ቃ⏺Ⅼ

#0

1

1

2

2

3

#1

3

4

5

4

4

5

1

1

2

2

3

3

#2

4

5

5

4

6

1

1

2

2

3

3

4

S2-ref27⾜ิ䝧䜽䝖䝹✚䠖䝻䞊䜹䝹䛻ィ⟬ᐇ᪋ྍ⬟
1
2
3
4
5
6

1

1

2

2

3

3

4

4

5

5

6
7
7
9
10
11
12

=

6

7

7

8

8

9

9

10

10

11

11

12

12

S2-ref28⾜ิ䝧䜽䝖䝹✚䠖䝻䞊䜹䝹䛻ィ⟬ᐇ᪋ྍ⬟
1
2
3
4
5
6

1

1

2

2

3

3

4

4

5

5

6
7
8
9
10
11
12

=

6

7

7

8

8

9

9

10

10

11

11

12

12

S2-ref29⾜ิ䝧䜽䝖䝹✚䠖䝻䞊䜹䝹䛻ィ⟬ᐇ᪋ྍ⬟
1
2
3
4
5
6

1

1

2

2

3

3

4

4

5

5

6
7
8
9
10
11
12

=

6

7

7

8

8

9

9

10

10

11

11

12

12

S2-ref30

⾜ิ䝧䜽䝖䝹✚䠖䝻䞊䜹䝹ィ⟬ #1
1

1
2

2
3
4

1
2

4

=

2

3

3

4

4

1

1

2
3

1

=

2

3

3

4

4

5
6

5

1

2

3

4

6

S2-ref

31㏦ಙ䠄MPI_Isend/Irecv/Waitall䠅
SENDbuf

neib#1neib#2neib#3neib#4BUFlength_e

BUFlength_e

BUFlength_e

BUFlength_e

export_index(0)+1

export_index(1)+1 export_index(2)+1

export_index(3)+1

export_index(4)do neib= 1, NEIBPETOT
do k= export_index(neib-1)+1, export_index(neib)
kk= export_item(k)SENDbuf(k)= VAL(kk)㏦ಙ䝞䝑䝣䜯䜈䛾௦ධ
enddo
enddo
ᗘ䛺䛹䛾ኚᩘ䜢┤᥋㏦ಙ䠈ཷಙ䛻౑do neib= 1, NEIBPETOTiS_e= export_index(neib-1) + 1
iE_e= export_index(neib )BUFlength_e= iE_e + 1 - iS_e䛖䛾䛷䛿䛺䛟䠈䛣䛾䜘䛖䛺䝞䝑䝣䜯䜈୍ᅇ௦ධ䛧䛶ィ⟬䛩䜛䛣䛸䜢່䜑䜛䚹call MPI_ISEND&&(SENDbuf(iS_e), BUFlength_e, MPI_INTEGER, NEIBPE(neib), 0,&&
MPI_COMM_WORLD, request_send(neib), ierr)enddo
call MPI_WAITALL (NEIBPETOT, request_send, stat_recv, ierr)S2-ref32ཷಙ䠄MPI_Isend/Irecv/Waitall䠅
do neib= 1, NEIBPETOTiS_i= import_index(neib-1) + 1
iE_i= import_index(neib )BUFlength_i= iE_i + 1 - iS_i
call MPI_IRECV&&(RECVbuf(iS_i), BUFlength_i, MPI_INTEGER, NEIBPE(neib), 0,&&
MPI_COMM_WORLD, request_recv(neib), ierr)enddo
call MPI_WAITALL (NEIBPETOT, request_recv, stat_recv, ierr)do neib= 1, NEIBPETOT
do k= import_index(neib-1)+1, import_index(neib)
kk= import_item(k)VAL(kk)= RECVbuf(k)
enddo
enddo

RECVbufཷಙ䝞䝑䝣䜯䛛䜙௦ධ

neib#1neib#2neib#3neib#4BUFlength_i

BUFlength_i

BUFlength_i

BUFlength_i

import_index(0)+1

import_index(1)+1 import_index(2)+1

import_index(3)+1

import_index(4)S2-ref• ၥ㢟ࡢᴫせ㸪ᐇ⾜᪉ἲ• ᒁᡤศᩓࢹ࣮ࢱࡢ⪃࠼᪉• ࣉࣟࢢ࣒ࣛࡢㄝ᫂
• ィ⟬౛

33

S2-ref䝥䝻䜾䝷䝮䠖1d.f䠄1/11䠅ㅖኚᩘ
SURJUDPKHDW'S
LPSOLFLW5($/  $+2=LQFOXGH PSLIK
LQWHJHU113/8,7(5PD[
LQWHJHU5=34''UHDO NLQG  G;5(6,'(36
UHDO NLQG  $5($49&21'
UHDO NLQG  GLPHQVLRQ  DOORFDWDEOH3+,5+6
UHDO NLQG  GLPHQVLRQ  DOORFDWDEOH',$*$0$7
UHDO NLQG  GLPHQVLRQ  DOORFDWDEOH:UHDO NLQG  GLPHQVLRQ  .0$7(0$7
LQWHJHUGLPHQVLRQ  DOORFDWDEOH,&(/12'
LQWHJHUGLPHQVLRQ  DOORFDWDEOH,1'(;,7(0
LQWHJHU NLQG  1(,%3(727%8)OHQJWK3(727
LQWHJHU NLQG  GLPHQVLRQ  1(,%3(
LQWHJHU NLQG  GLPHQVLRQ  LPSRUWBLQGH[H[SRUWBLQGH[
LQWHJHU NLQG  GLPHQVLRQ  LPSRUWBLWHPH[SRUWBLWHPUHDO NLQG  GLPHQVLRQ  6(1'EXI5(&9EXI
LQWHJHU NLQG  GLPHQVLRQ  DOORFDWDEOHVWDWBVHQG
LQWHJHU NLQG  GLPHQVLRQ  DOORFDWDEOHVWDWBUHFY
LQWHJHU NLQG  GLPHQVLRQ  DOORFDWDEOHUHTXHVWBVHQG
LQWHJHU NLQG  GLPHQVLRQ  DOORFDWDEOHUHTXHVWBUHFY

34

S2-ref䝥䝻䜾䝷䝮䠖1d.f䠄2/11䠅
ไᚚ䝕䞊䝍ㄞ䜏㎸䜏&&
&_,1,7_&
&&& 03,LQLW
FDOO03,B,QLW LHUU
FDOO03,B&RPPBVL]H 03,B&200B:25/'3(727LHUU
FDOO03,B&RPPBUDQN 03,B&200B:25/'P\BUDQNLHUU

03,ึᮇ໬㸸ᚲ㡲඲ࣉࣟࢭࢫᩘ㸸3(727⮬ศࡢࣛࣥࢡ␒ྕ㸦㹼3(727㸧㸸P\BUDQN&& &75/GDWD
LI P\BUDQNHT WKHQ
RSHQ ILOH LQSXWGDW VWDWXV XQNQRZQUHDG  1(JUHDG  G;49$5($&21'
UHDG  ,7(5PD[
UHDG  (36
FORVH 
HQGLI
FDOO03,B%FDVW 1(J03,B,17(*(503,B&200B:25/'LHUU
FDOO03,B%FDVW ,7(5PD[03,B,17(*(503,B&200B:25/'LHUU
FDOO03,B%FDVW G;03,B'28%/(B35(&,6,2103,B&200B:25/'LHUU
FDOO03,B%FDVW 4903,B'28%/(B35(&,6,2103,B&200B:25/'LHUU
FDOO03,B%FDVW $5($03,B'28%/(B35(&,6,2103,B&200B:25/'LHUU
FDOO03,B%FDVW &21'03,B'28%/(B35(&,6,2103,B&200B:25/'LHUU
FDOO03,B%FDVW (3603,B'28%/(B35(&,6,2103,B&200B:25/'LHUU

35

S2-ref䝥䝻䜾䝷䝮䠖1d.f䠄2/11䠅
ไᚚ䝕䞊䝍ㄞ䜏㎸䜏&&
&_,1,7_&
&&& 03,LQLW
FDOO03,B,QLW LHUU
FDOO03,B&RPPBVL]H 03,B&200B:25/'3(727LHUU
FDOO03,B&RPPBUDQN 03,B&200B:25/'P\BUDQNLHUU
&& &75/GDWD
LI P\BUDQNHT WKHQ
RSHQ ILOH LQSXWGDW VWDWXV XQNQRZQUHDG  1HJ
UHDG  G;49$5($&21'
UHDG  ,7(5PD[
UHDG  (36
FORVH 
HQGLI

03,ึᮇ໬㸸ᚲ㡲඲ࣉࣟࢭࢫᩘ㸸3(727⮬ศࡢࣛࣥࢡ␒ྕ㸦㹼3(727㸧㸸P\BUDQNP\BUDQN ࡢ࡜ࡁไᚚࢹ࣮ࢱࢆㄞࡳ㎸ࡴ
1HJࠕ඲ࠖせ⣲ᩘ

FDOO03,B%FDVW 1(J03,B,17(*(503,B&200B:25/'LHUU
FDOO03,B%FDVW ,7(5PD[03,B,17(*(503,B&200B:25/'LHUU
FDOO03,B%FDVW G;03,B'28%/(B35(&,6,2103,B&200B:25/'LHUU
FDOO03,B%FDVW 4903,B'28%/(B35(&,6,2103,B&200B:25/'LHUU
FDOO03,B%FDVW $5($03,B'28%/(B35(&,6,2103,B&200B:25/'LHUU
FDOO03,B%FDVW &21'03,B'28%/(B35(&,6,2103,B&200B:25/'LHUU
FDOO03,B%FDVW (3603,B'28%/(B35(&,6,2103,B&200B:25/'LHUU

36

S2-ref䝥䝻䜾䝷䝮䠖1d.f䠄2/11䠅
ไᚚ䝕䞊䝍ㄞ䜏㎸䜏&&
&_,1,7_&
&&& 03,LQLW
FDOO03,B,QLW LHUU
FDOO03,B&RPPBVL]H 03,B&200B:25/'3(727LHUU
FDOO03,B&RPPBUDQN 03,B&200B:25/'P\BUDQNLHUU
&& &75/GDWD
LI P\BUDQNHT WKHQ
RSHQ ILOH LQSXWGDW VWDWXV XQNQRZQUHDG  1HJ
UHDG  G;49$5($&21'
UHDG  ,7(5PD[
UHDG  (36
FORVH 
HQGLI

03,ึᮇ໬㸸ᚲ㡲඲ࣉࣟࢭࢫᩘ㸸3(727⮬ศࡢࣛࣥࢡ␒ྕ㸦㹼3(727㸧㸸P\BUDQNP\BUDQN ࡢ࡜ࡁไᚚࢹ࣮ࢱࢆㄞࡳ㎸ࡴ
1HJࠕ඲ࠖせ⣲ᩘ

FDOO03,B%FDVW 1(J03,B,17(*(503,B&200B:25/'LHUU ␒࠿ࡽྛࣉࣟࢭࢫ࡟ࢹ࣮ࢱ㏦ಙ
FDOO03,B%FDVW ,7(5PD[03,B,17(*(503,B&200B:25/'LHUU
FDOO03,B%FDVW G;03,B'28%/(B35(&,6,2103,B&200B:25/'LHUU
FDOO03,B%FDVW 4903,B'28%/(B35(&,6,2103,B&200B:25/'LHUU
FDOO03,B%FDVW $5($03,B'28%/(B35(&,6,2103,B&200B:25/'LHUU
FDOO03,B%FDVW &21'03,B'28%/(B35(&,6,2103,B&200B:25/'LHUU
FDOO03,B%FDVW (3603,B'28%/(B35(&,6,2103,B&200B:25/'LHUU

37

S2-ref38

MPI_BCAST•P#0 A0 B0 C0 D0
P#1

P#0 A0 B0 C0 D0BroadcastP#1 A0 B0 C0 D0

P#2

P#2 A0 B0 C0 D0

P#3

P#3 A0 B0 C0 D0䝁䝭䝳䝙䜿䞊䝍䞊 䛂comm䛃ෆ䛾୍䛴䛾㏦ಙඖ䝥䝻䝉䝇䛂root䛃䛾䝞䝑䝣䜯䛂buffer䛃䛛䜙䠈䛭䛾௚඲䛶䛾䝥䝻䝉䝇䛾䝞䝑䝣䜯䛂buffer䛃䛻䝯䝑䝉䞊䝆䜢㏦ಙ䚹• call MPI_BCAST (buffer,count,datatype,root,comm,ierr)– buffer௵ពI/O– countᩚᩘI䝞䝑䝣䜯䛾ඛ㢌䜰䝗䝺䝇䠈䝍䜲䝥䛿䛂datatype䛃䛻䜘䜚Ỵᐃ䝯䝑䝉䞊䝆䛾䝃䜲䝈– datatype ᩚᩘI䝯䝑䝉䞊䝆䛾䝕䞊䝍䝍䜲䝥

FORTRANC– root– comm– ierr

MPI_INTEGER, MPI_REAL, MPI_DOUBLE_PRECISION, MPI_CHARACTER etc.
MPI_INT, MPI_FLOAT, MPI_DOUBLE, MPI_CHAR etc.

ᩚᩘ
ᩚᩘ
ᩚᩘ

I
I
O

㏦ಙඖ䝥䝻䝉䝇䛾ID䠄䝷䞁䜽䠅䝁䝭䝳䝙䜿䞊䝍䜢ᣦᐃ䛩䜛᏶஢䝁䞊䝗

S2-ref䝥䝻䜾䝷䝮䠖1d.f䠄3/11䠅
ᒁᡤศᩓ䝯䝑䝅䝳䝕䞊䝍&
& /RFDO0HVK6L]H
1J 1(J
1 1J3(727

⥲⠇Ⅼᩘ
ᒁᡤ⠇Ⅼᩘ

QU 1J 1 3(727LI P\BUDQNOWQU 1 11Jࡀ3(727࡛๭ࡾษࢀ࡞࠸ሙྜ

1( 1 
13 1
LI P\BUDQNHT 1( 1 
LI P\BUDQNHT 13 1
LI P\BUDQNHT3(727 1( 1 
LI P\BUDQNHT3(727 13 1
LI 3(727HT 1( 1LI 3(727HT 13 1
&&̽ $55$<V
DOORFDWH 3+, 13 ',$* 13 $0$7  13 5+6 13
DOORFDWH ,&(/12'  1(
DOORFDWH ,1'(; 13 ,7(0  13 : 13
3+, G
$0$7 G
',$* G
5+6 G

39

S2-ref40䝥䝻䜾䝷䝮䠖1d.f䠄3/11䠅
ᒁᡤศᩓ䝯䝑䝅䝳䝕䞊䝍䠈ྛせ⣲ĺ୍ᵝ&& /RFDO0HVK6L]H
1J 1(J
1 1J3(727

⥲⠇Ⅼᩘ
ᒁᡤ⠇Ⅼᩘ

QU 1J 1 3(727LI P\BUDQNOWQU 1 11Jࡀ3(727࡛๭ࡾษࢀ࡞࠸ሙྜ

1( 1 
13 1

1(㸸ᒁᡤせ⣲ᩘ13㸸ᒁᡤ⠇Ⅼᩘ㸦ෆⅬ㸩እⅬ㸧

LI P\BUDQNHT 1( 1 
LI P\BUDQNHT 13 1
LI P\BUDQNHT3(727 1( 1 
LI P\BUDQNHT3(727 13 1
LI 3(727HT 1( 1LI 3(727HT 13 1

N+1

N

1

1

2

&
&̽ $55$<V
DOORFDWH 3+, 13 ',$* 13 $0$7  13 5+6 13
DOORFDWH ,&(/12'  1(
DOORFDWH ,1'(; 13 ,7(0  13 : 13
3+, G
$0$7 G
',$* G
5+6 G

2

…

N-1

N

N+1

N+2

୍⯡䛾㡿ᇦ䠖
N+2⠇Ⅼ䠈N+1せ⣲S2-ref䝥䝻䜾䝷䝮䠖1d.f䠄3/11䠅
ᒁᡤศᩓ䝯䝑䝅䝳䝕䞊䝍䠈ྛせ⣲ĺ୍ᵝ&& /RFDO0HVK6L]H
1J 1(J
1 1J3(727

⥲⠇Ⅼᩘ
ᒁᡤ⠇Ⅼᩘ

QU 1J 1 3(727LI P\BUDQNOWQU 1 11Jࡀ3(727࡛๭ࡾษࢀ࡞࠸ሙྜ

1( 1 
13 1

1(㸸ᒁᡤせ⣲ᩘ13㸸ᒁᡤ⠇Ⅼᩘ㸦ෆⅬ㸩እⅬ㸧

LI P\BUDQNHT 1( 1 
LI P\BUDQNHT 13 1
LI P\BUDQNHT3(727 1( 1 
LI P\BUDQNHT3(727 13 1
LI 3(727HT 1( 1LI 3(727HT 13 1

1

1

2

2

&
&̽ $55$<V
DOORFDWH 3+, 13 ',$* 13 $0$7  13 5+6 13
DOORFDWH ,&(/12'  1(
DOORFDWH ,1'(; 13 ,7(0  13 : 13
3+, G
$0$7 G
',$* G
5+6 G

…

N-1

N

N

N+1

#0: N+1⠇Ⅼ䠈Nせ⣲41

S2-ref䝥䝻䜾䝷䝮䠖1d.f䠄3/11䠅
ᒁᡤศᩓ䝯䝑䝅䝳䝕䞊䝍䠈ྛせ⣲ĺ୍ᵝ&& /RFDO0HVK6L]H
1J 1(J
1 1J3(727

⥲⠇Ⅼᩘ
ᒁᡤ⠇Ⅼᩘ

QU 1J 1 3(727LI P\BUDQNOWQU 1 11Jࡀ3(727࡛๭ࡾษࢀ࡞࠸ሙྜ

1( 1 
13 1

1(㸸ᒁᡤせ⣲ᩘ13㸸ᒁᡤ⠇Ⅼᩘ㸦ෆⅬ㸩እⅬ㸧

LI P\BUDQNHT 1( 1 
LI P\BUDQNHT 13 1
LI P\BUDQNHT3(727 1( 1 
LI P\BUDQNHT3(727 13 1
LI 3(727HT 1( 1N+1
LI 3(727HT 13 1

N

1

1

2

2

…

&
&̽ $55$<V
DOORFDWH 3+, 13 ',$* 13 $0$7  13 5+6 13
DOORFDWH ,&(/12'  1(
DOORFDWH ,1'(; 13 ,7(0  13 : 13
3+, G
$0$7 G
',$* G
5+6 G

N-1

N#PETot-1: N+1⠇Ⅼ䠈Nせ⣲42

S2-ref䝥䝻䜾䝷䝮䠖1d.f䠄3/11䠅
ᒁᡤศᩓ䝯䝑䝅䝳䝕䞊䝍&
& /RFDO0HVK6L]H
1J 1(J
1 1J3(727

⥲⠇Ⅼᩘ
ᒁᡤ⠇Ⅼᩘ

QU 1J 1 3(727LI P\BUDQNOWQU 1 11Jࡀ3(727࡛๭ࡾษࢀ࡞࠸ሙྜ

1( 1 
13 1

1(㸸ᒁᡤせ⣲ᩘ13㸸ᒁᡤ⠇Ⅼᩘ㸦ෆⅬ㸩እⅬ㸧

LI P\BUDQNHT 1( 1 
LI P\BUDQNHT 13 1
LI P\BUDQNHT3(727 1( 1 
LI P\BUDQNHT3(727 13 1
LI 3(727HT 1( 1LI 3(727HT 13 1
&&̽ $55$<V
DOORFDWH 3+, 13 ',$* 13 $0$7  13 5+6 13
DOORFDWH ,&(/12'  1(
DOORFDWH ,1'(; 13 ,7(0  13 : 13
3+, G
$0$7 G
',$* G
5+6 G

1࡛࡞ࡃ13࡛㓄ิࢆᐃ⩏ࡋ࡚࠸ࡿⅬ࡟ὀព

43

S2-ref䝥䝻䜾䝷䝮䠖1d.f䠄4/11䠅
㓄ิึᮇ໬䠈せ⣲䡚⠇Ⅼ
GRLFHO 1(,&(/12'  LFHO LFHO,&(/12'  LFHO LFHO
HQGGR
LI 3(727JW WKHQ
LI P\BUDQNHT WKHQLFHO 1(,&(/12'  LFHO 1,&(/12'  LFHO 1
HOVHLI P\BUDQNHT3(727 WKHQLFHO 1(,&(/12'  LFHO 1
,&(/12'  LFHO 
HOVH
LFHO 1( ,&(/12'  LFHO,&(/12'  LFHOLFHO 1(,&(/12'  LFHO,&(/12'  LFHO
HQGLI
HQGLI

1

1
1

icel
Icelnod(2*icel-1)=icel

Icelnod(2*icel)=icel+144

S2-ref䝥䝻䜾䝷䝮䠖1d.f䠄4/11䠅
㓄ิึᮇ໬䠈せ⣲䡚⠇Ⅼ
GRLFHO 1(,&(/12'  LFHO LFHO,&(/12'  LFHO LFHO
HQGGR

䛂1-2䛃䛾せ⣲䜢䛂1䛃䛸䛩䜛LI 3(727JW WKHQ
LI P\BUDQNHT WKHQLFHO 1(,&(/12'  LFHO 1,&(/12'  LFHO 1

1

1

2

HOVHLI P\BUDQNHT3(727 WKHQLFHO 1(,&(/12'  LFHO 1
N+1
1,&(/12'  LFHO N

HOVH
LFHO 1( ,&(/12'  LFHO,&(/12'  LFHOLFHO 1(,&(/12'  LFHO,&(/12'  LFHO
HQGLI
HQGLI

1

1
1

N+1

N

1

2

1

1

…

2

2

N-1

2

2

N

…

…

N

N-1

N-1

N+1

#0: N+1⠇Ⅼ䠈Nせ⣲N

N#PETot-1: N+1⠇Ⅼ䠈Nせ⣲

N+1

N+2

୍⯡䛾㡿ᇦ䠖
N+2⠇Ⅼ䠈N+1せ⣲45

S2-ref䝥䝻䜾䝷䝮䠖1d.f䠄5/11䠅
Indexᐃ⩏.0$7  G
.0$7  G
.0$7  G
.0$7  G&&&
&_&211(&7,9,7<_&
&,1'(; 

1

N+1

N+1

N

N

1

1

,1'(;  ,1'(; 1 ,1'(; 13 
LI P\BUDQNHT ,1'(;  
LI P\BUDQNHT3(727 ,1'(; 1 
GRL 13,1'(; L ,1'(; L ,1'(; L
HQGGR
13/8 ,1'(; 13,7(0 

1

1

1

2

2

2

2

2

2

…

…

…

N-1

N-1

N-1

N

N

N+1

N

N

#0: N+1⠇Ⅼ䠈Nせ⣲#PETot-1: N+1⠇Ⅼ䠈Nせ⣲

N+1

N+2

୍⯡䛾㡿ᇦ䠖
N+2⠇Ⅼ䠈N+1せ⣲46

S2-ref䝥䝻䜾䝷䝮䠖1d.f䠄6/11䠅
Itemᐃ⩏
GRL 1
M6 ,1'(; L
LI P\BUDQNHTDQGLHT WKHQ,7(0 M6 L
HOVHLI P\BUDQNHT3(727DQGLHT1 WKHQ,7(0 M6 L
HOVH
,7(0 M6 L
,7(0 M6 LLI LHT ,7(0 M6 1
LI LHT1 ,7(0 M6 1
LI P\BUDQNHTDQGLHT1 ,7(0 M6 1
HQGLI
HQGGR
1

L 1
M6 ,1'(; L
LI P\BUDQNHT WKHQ,7(0 M6 1
HOVH,7(0 M6 
HQGLI

N+1

N+1

N

N

1

1

1

1

1

2

2

2

2

2

2

L 1
LI P\BUDQNQHDQGP\BUDQNQH3(727 WKHQ
M6 ,1'(; L,7(0 M6 1
HQGLI

…

…

…

N-1

N-1

N-1

N

N

N+1

N

N

#0: N+1⠇Ⅼ䠈Nせ⣲#PETot-1: N+1⠇Ⅼ䠈Nせ⣲

N+1

N+2

୍⯡䛾㡿ᇦ䠖
N+2⠇Ⅼ䠈N+1せ⣲47

S2-ref䝥䝻䜾䝷䝮䠖1d.f䠄7/11䠅
㏻ಙ䝔䞊䝤䝹ᐃ⩏&
& &20081,&$7,21
1(,%3(727 
LI P\BUDQNHT 1(,%3(727 
LI P\BUDQNHT3(727 1(,%3(727 LI 3(727HT 1(,%3(727 
1(,%3(  P\BUDQN 
1(,%3(  P\BUDQN
LI P\BUDQNHT 1(,%3(  P\BUDQN
LI P\BUDQNHT3(727 1(,%3(  P\BUDQN %8)OHQJWK LPSRUWBLQGH[ LPSRUWBLQGH[ LPSRUWBLWHP 
LPSRUWBLWHP 



1
1

H[SRUWBLQGH[ 
H[SRUWBLQGH[ 
H[SRUWBLWHP 
H[SRUWBLWHP 




1

LI P\BUDQNHT WKHQ
LPSRUWBLWHP  1
H[SRUWBLWHP  1
HQGLI

1

N+1

N+1

N

N

1

1

1

1

1

2

2

2

2

2

2

…

…

…

N-1

N-1

N-1

N

N

N+1

N

N

#0: N+1⠇Ⅼ䠈Nせ⣲#PETot-1: N+1⠇Ⅼ䠈Nせ⣲

N+1

N+2

୍⯡䛾㡿ᇦ䠖
N+2⠇Ⅼ䠈N+1せ⣲&& ,1,7DUUD\VIRU03,B:DLWDOODOORFDWH VWDWBVHQG 03,B67$786B6,=(1(,%3(727 VWDWBUHFY 03,B67$786B6,=(1(,%3(727
DOORFDWH UHTXHVWBVHQG 1(,%3(727 UHTXHVWBUHFY 1(,%3(72748

S2-ref49

MPI_ISEND•㏦ಙ䝞䝑䝣䜯䛂sendbuf䛃ෆ䛾䠈㐃⥆䛧䛯䛂count䛃ಶ䛾㏦ಙ䝯䝑䝉䞊䝆䜢䠈䝍䜾䛂tag䛃䜢௜䛡䛶䠈䝁䝭䝳䝙䜿䞊䝍ෆ䛾䠈䛂dest䛃䛻㏦ಙ䛩䜛䚹䛂MPI_WAITALL䛃䜢࿧䜆䜎䛷䠈
㏦ಙ䝞䝑䝣䜯䛾ෆᐜ䜢᭦᪂䛧䛶䛿䛺䜙䛺䛔䚹• call MPI_ISEND(sendbuf,count,datatype,dest,tag,comm,request, ierr)sendbuf ௵ព
countᩚᩘ
datatype ᩚᩘ
dest
ᩚᩘ
tag
ᩚᩘ

I
I
I
I
I

– comm– request

ᩚᩘ
ᩚᩘ

I
O

– ierr

ᩚᩘ

O

–
–
–
–
–

㏦ಙ䝞䝑䝣䜯䛾ඛ㢌䜰䝗䝺䝇䠈䝯䝑䝉䞊䝆䛾䝃䜲䝈䝯䝑䝉䞊䝆䛾䝕䞊䝍䝍䜲䝥ᐄඛ䝥䝻䝉䝇䛾䜰䝗䝺䝇䠄䝷䞁䜽䠅䝯䝑䝉䞊䝆䝍䜾䠈㏦ಙ䝯䝑䝉䞊䝆䛾✀㢮䜢༊ู䛩䜛䛸䛝䛻౑⏝䚹
㏻ᖖ䛿䛂0䛃䛷䜘䛔䚹ྠ䛨䝯䝑䝉䞊䝆䝍䜾␒ྕྠኈ䛷㏻ಙ䚹䝁䝭䝳䝙䜿䞊䝍䜢ᣦᐃ䛩䜛
㏻ಙ㆑ูᏊ䚹MPI_WAITALL䛷౑⏝䚹䠄㓄ิ䠖䝃䜲䝈䛿ྠᮇ䛩䜛ᚲせ䛾䛒䜛䛂MPI_ISEND䛃࿧䜃ฟ䛧ᩘ䠄㏻ᖖ䛿㞄᥋䝥䝻䝉䝇ᩘ䛺䛹䠅䠅᏶஢䝁䞊䝗

S2-ref50

MPI_IRECV•ཷಙ䝞䝑䝣䜯䛂recvbuf䛃ෆ䛾䠈㐃⥆䛧䛯䛂count䛃ಶ䛾㏦ಙ䝯䝑䝉䞊䝆䜢䠈䝍䜾䛂tag䛃䜢௜䛡䛶䠈䝁䝭䝳䝙䜿䞊䝍ෆ䛾䠈䛂dest䛃䛛䜙ཷಙ䛩䜛䚹䛂MPI_WAITALL䛃䜢࿧䜆䜎䛷䠈ཷಙ䝞䝑䝣䜯䛾ෆᐜ䜢฼⏝䛧䛯ฎ⌮䜢ᐇ᪋䛧䛶䛿䛺䜙䛺䛔䚹• call MPI_IRECV(recvbuf,count,datatype,dest,tag,comm,request, ierr)recvbuf ௵ព
countᩚᩘ
datatype ᩚᩘ
dest
ᩚᩘ
tag
ᩚᩘ

I
I
I
I
I

– comm– request

ᩚᩘ
ᩚᩘ

I
O

– ierr

ᩚᩘ

O

–
–
–
–
–

ཷಙ䝞䝑䝣䜯䛾ඛ㢌䜰䝗䝺䝇䠈䝯䝑䝉䞊䝆䛾䝃䜲䝈䝯䝑䝉䞊䝆䛾䝕䞊䝍䝍䜲䝥ᐄඛ䝥䝻䝉䝇䛾䜰䝗䝺䝇䠄䝷䞁䜽䠅䝯䝑䝉䞊䝆䝍䜾䠈ཷಙ䝯䝑䝉䞊䝆䛾✀㢮䜢༊ู䛩䜛䛸䛝䛻౑⏝䚹
㏻ᖖ䛿䛂0䛃䛷䜘䛔䚹ྠ䛨䝯䝑䝉䞊䝆䝍䜾␒ྕྠኈ䛷㏻ಙ䚹䝁䝭䝳䝙䜿䞊䝍䜢ᣦᐃ䛩䜛
㏻ಙ㆑ูᏊ䚹MPI_WAITALL䛷౑⏝䚹䠄㓄ิ䠖䝃䜲䝈䛿ྠᮇ䛩䜛ᚲせ䛾䛒䜛䛂MPI_IRECV䛃࿧䜃ฟ䛧ᩘ䠄㏻ᖖ䛿㞄᥋䝥䝻䝉䝇ᩘ䛺䛹䠅䠅᏶஢䝁䞊䝗

S2-ref

51

MPI_WAITALL• 1ᑐ1㠀䝤䝻䝑䜻䞁䜾㏻ಙ䝃䝤䝹䞊䝏䞁䛷䛒䜛䛂MPI_ISEND䛃䛸䛂MPI_IRECV䛃䜢౑⏝䛧䛯ሙྜ䠈䝥䝻䝉䝇䛾ྠᮇ䜢ྲྀ䜛䛾䛻౑⏝䛩䜛䚹• ㏦ಙ᫬䛿䛣䛾䛂MPI_WAITALL䛃䜢࿧䜆๓䛻㏦ಙ䝞䝑䝣䜯䛾ෆᐜ䜢ኚ᭦䛧䛶䛿䛺䜙䛺䛔䚹ཷಙ᫬䛿䛂MPI_WAITALL䛃䜢࿧䜆๓䛻ཷಙ䝞䝑䝣䜯䛾ෆᐜ䜢฼⏝䛧䛶䛿䛺䜙䛺䛔䚹• ᩚྜᛶ䛜ྲྀ䜜䛶䛔䜜䜀䠈 䛂MPI_ISEND䛃䛸䛂MPI_IRECV䛃䜢ྠ᫬䛻ྠᮇ䛧䛶䜒䜘䛔䚹– 䛂MPI_ISEND/IRECV䛃䛷ྠ䛨㏻ಙ㆑ูᏊ䜢౑⏝䛩䜛䛣䛸

•

䛂MPI_BARRIER䛃䛸ྠ䛨䜘䛖䛺ᶵ⬟䛷䛒䜛䛜䠈௦⏝䛿䛷䛝䛺䛔䚹– ᐇ⿦䛻䜒䜘䜛䛜䠈䛂request䛃䠈䛂status䛃䛾ෆᐜ䛜ṇ䛧䛟᭦᪂䛥䜜䛪䠈ఱᗘ䜒
䛂MPI_ISEND/IRECV䛃䜢࿧䜃ฟ䛩䛸ฎ⌮䛜㐜䛟䛺䜛䠈䛸䛔䛖䜘䛖䛺⤒㦂䜒䛒䜛䚹• call MPI_WAITALL(count,request,status,ierr)– count– request

ᩚᩘ
ᩚᩘ

I
I/O

– statusᩚᩘO– ierrᩚᩘOྠᮇ䛩䜛ᚲせ䛾䛒䜛䛂MPI_ISEND䛃 䠈䛂MPI_RECV䛃࿧䜃ฟ䛧ᩘ䚹
㏻ಙ㆑ูᏊ䚹䛂MPI_ISEND䛃䠈䛂MPI_IRECV䛃䛷฼⏝䛧䛯㆑ูᏊྡ䛻ᑐᛂ䚹䠄㓄ิ䝃䜲䝈䠖䠄count䠅䠅≧ἣ䜸䝤䝆䜵䜽䝖㓄ิ䠄㓄ิ䝃䜲䝈䠖䠄MPI_STATUS_SIZE,count䠅䠅
MPI_STATUS_SIZE䠖 䇾mpif.h”,”mpi.h”䛷ᐃ䜑䜙䜜䜛
䝟䝷䝯䞊䝍
᏶஢䝁䞊䝗

S2-ref52୍⯡໬䛥䜜䛯㏻ಙ䝔䞊䝤䝹䠖㏦ಙ
• ㏦ಙ┦ᡭ
– NEIBPETOT䠈NEIBPE(neib)• 䛭䜜䛮䜜䛾㏦ಙ┦ᡭ䛻㏦䜛䝯䝑䝉䞊䝆䝃䜲䝈– export_index(neib), neib= 0, NEIBPETOT• 䛂ቃ⏺Ⅼ䛃␒ྕ– export_item(k), k= 1, export_index(NEIBPETOT)• 䛭䜜䛮䜜䛾㏦ಙ┦ᡭ䛻㏦䜛䝯䝑䝉䞊䝆– SENDbuf(k), k= 1, export_index(NEIBPETOT)S2-ref

53㏦ಙ䠄MPI_Isend/Irecv/Waitall䠅
SENDbuf

neib#1neib#2neib#3neib#4BUFlength_e

BUFlength_e

BUFlength_e

BUFlength_e

export_index(0)+1

export_index(1)+1 export_index(2)+1

export_index(3)+1

export_index(4)do neib= 1, NEIBPETOT
do k= export_index(neib-1)+1, export_index(neib)
kk= export_item(k)SENDbuf(k)= VAL(kk)㏦ಙ䝞䝑䝣䜯䜈䛾௦ධ
enddo
enddo
ᗘ䛺䛹䛾ኚᩘ䜢┤᥋㏦ಙ䠈ཷಙ䛻౑do neib= 1, NEIBPETOTiS_e= export_index(neib-1) + 1
iE_e= export_index(neib )BUFlength_e= iE_e + 1 - iS_e䛖䛾䛷䛿䛺䛟䠈䛣䛾䜘䛖䛺䝞䝑䝣䜯䜈୍ᅇ௦ධ䛧䛶ィ⟬䛩䜛䛣䛸䜢່䜑䜛䚹call MPI_ISEND&&(SENDbuf(iS_e), BUFlength_e, MPI_INTEGER, NEIBPE(neib), 0,&&
MPI_COMM_WORLD, request_send(neib), ierr)enddo
call MPI_WAITALL (NEIBPETOT, request_send, stat_recv, ierr)S2-ref54㏦ಙ䠖୍ḟඖၥ㢟• ㏦ಙ┦ᡭ
– NEIBPETOT䠈NEIBPE(neib)

5

1

2

SENDbuf(1)=BUF(1)

3

4

6

SENDbuf(2)=BUF(4)• NEIBPETOT=2, NEIB(1)= my_rank-1, NEIB(2)= my_rank+1• 䛭䜜䛮䜜䛾㏦ಙ┦ᡭ䛻㏦䜛䝯䝑䝉䞊䝆䝃䜲䝈– export_index(neib), neib= 0, NEIBPETOT• export_index(0)=0, export_index(1)= 1, export_index(2)= 2• 䛂ቃ⏺Ⅼ䛃␒ྕ– export_item(k), k= 1, export_index(NEIBPETOT)• export_item(1)= 1, export_item(2)= N• 䛭䜜䛮䜜䛾㏦ಙ┦ᡭ䛻㏦䜛䝯䝑䝉䞊䝆– SENDbuf(k), k= 1, export_index(NEIBPETOT)• SENDbuf(1)= BUF(1), SENDbuf(2)= BUF(N)S2-ref55୍⯡໬䛥䜜䛯㏻ಙ䝔䞊䝤䝹䠖ཷಙ• ཷಙ┦ᡭ
– NEIBPETOT䠈NEIBPE(neib)• 䛭䜜䛮䜜䛾ཷಙ┦ᡭ䛛䜙ཷ䛡ྲྀ䜛䝯䝑䝉䞊䝆䝃䜲䝈– import_index(neib), neib= 0, NEIBPETOT• 䛂እⅬ䛃␒ྕ– import_item(k), k= 1, import_index(NEIBPETOT)• 䛭䜜䛮䜜䛾ཷಙ┦ᡭ䛛䜙ཷ䛡ྲྀ䜛䝯䝑䝉䞊䝆– RECVbuf(k), k= 1, import_index(NEIBPETOT)S2-ref56

ཷಙ䠄MPI_Isend/Irecv/Waitall䠅
do neib= 1, NEIBPETOTiS_i= import_index(neib-1) + 1
iE_i= import_index(neib )BUFlength_i= iE_i + 1 - iS_i
call MPI_IRECV&&(RECVbuf(iS_i), BUFlength_i, MPI_INTEGER, NEIBPE(neib), 0,&&
MPI_COMM_WORLD, request_recv(neib), ierr)enddo
call MPI_WAITALL (NEIBPETOT, request_recv, stat_recv, ierr)do neib= 1, NEIBPETOT
do k= import_index(neib-1)+1, import_index(neib)
kk= import_item(k)VAL(kk)= RECVbuf(k)
enddo
enddo

RECVbufཷಙ䝞䝑䝣䜯䛛䜙௦ධ

neib#1neib#2neib#3neib#4BUFlength_i

BUFlength_i

BUFlength_i

BUFlength_i

import_index(0)+1

import_index(1)+1 import_index(2)+1

import_index(3)+1

import_index(4)S2-ref57ཷಙ:୍ḟඖၥ㢟5• ཷಙ┦ᡭ

1

2

BUF(0)=RECVbuf(1)

3

4

6

BUF(5)=RECVbuf(2)– NEIBPETOT䠈NEIBPE(neib)• NEIBPETOT=2, NEIB(1)= my_rank-1, NEIB(2)= my_rank+1• 䛭䜜䛮䜜䛾ཷಙ┦ᡭ䛛䜙ཷ䛡ྲྀ䜛䝯䝑䝉䞊䝆䝃䜲䝈– import_index(neib), neib= 0, NEIBPETOT• import_index(0)=0, import_index(1)= 1, import_index(2)= 2• 䛂እⅬ䛃␒ྕ– import_item(k), k= 1, import_index(NEIBPETOT)• import_item(1)= N+1, import_item(2)= N+2• 䛭䜜䛮䜜䛾ཷಙ┦ᡭ䛛䜙ཷ䛡ྲྀ䜛䝯䝑䝉䞊䝆– RECVbuf(k), k= 1, import_index(NEIBPETOT)• BUF(N+1)=RECVbuf(1), BUF(N+2)=RECVbuf(2)S2-ref58୍⯡໬䛥䜜䛯㏻ಙ䝔䞊䝤䝹䠖Fortran5

1

2

SENDbuf(1)=BUF(1)5

1

BUF(5)=RECVbuf(1)

2

3

4

6

SENDbuf(2)=BUF(4)

3

4

6

BUF(6)=RECVbuf(2)NEIBPETOT= 2
NEIBPE(1)= my_rank - 1
NEIBPE(2)= my_rank + 1
import_index(1)= 1
import_index(2)= 2
import_item (1)= N+1
import_item (2)= N+2
export_index(1)= 1
export_index(2)= 2
export_item (1)= 1
export_item (2)= N
if (my_rank.eq.0) thenimport_item (1)= N+1
export_item (1)= N
NEIBPE(1)= my_rank+1
endifS2-ref59୍⯡໬䛥䜜䛯㏻ಙ䝔䞊䝤䝹䠖Cゝㄒ
4

0

1

SENDbuf[0]=BUF[0]4

0

BUF[4]=RECVbuf[0]

1

2

3

5

SENDbuf[1]=BUF[3]

2

3

5

BUF[5]=RECVbuf[1]NEIBPETOT= 2
NEIBPE[0]= my_rank - 1
NEIBPE[1]= my_rank + 1
import_index[1]= 0
import_index[2]= 1
import_item [0]= N
import_item [1]= N+1
export_index[1]= 0
export_index[2]= 1
export_item [0]= 0
export_item [1]= N-1
if (my_rank.eq.0) thenimport_item [0]= N
export_item [0]= N-1
NEIBPE[0]= my_rank+1
endifS2-ref60䝥䝻䜾䝷䝮䠖1d.f䠄8/11䠅
඲య䝬䝖䝸䜽䝇⏕ᡂ䠖1CPU䛾䛸䛝䛸඲䛟ྠ䛨䠖ྛせ⣲ĺ୍ᵝ&&
&_0$75,;$66(0%/(_&
#0&GRLFHO 1(
LQ ,&(/12'  LFHO
LQ ,&(/12'  LFHO'/ G;
F. $5($ &21''/(0$7  &N .0$7 (0$7  &N .0$7 (0$7  &N .0$7 (0$7  &N .0$7 

1

1

2

2

',$* LQ ',$* LQ (0$7 ',$* LQ ',$* LQ (0$7 LI P\BUDQNHTDQGLFHOHT WKHQ
N ,1'(; LQ HOVH
N ,1'(; LQ HQGLI
N ,1'(; LQ $0$7 N $0$7 N (0$7 $0$7 N $0$7 N (0$7 &

41 G 49 $5($ '/5+6 LQ 5+6 LQ 41
5+6 LQ 5+6 LQ 41
HQGGR

3

#1

3

4

5

4

4

5

1

1

2

2

3

3

#2

4

5

5

4

6

1

1

2

2

3

3

4

FEM3D

61

Local Matrix䠖ྛ䝥䝻䝉䝇䛻䛚䛡䜛ಀᩘ⾜ิ
NP
N

NP

N

NP

N

internal

external

FEM3D62ᮏᙜ䛻ᚲせ䛺䛾䛿䛣䛾㒊ศ
NP
N

NP

N

NP

N

internal

external

pFEM3D-2

63

MAT_ASS_MAIN: Overview*DXVVLDQ4XDGSRLQWVLQ ]GLUHFWLRQ*DXVVLDQ4XDGSRLQWVLQ KGLUHFWLRQ*DXVVLDQ4XDG3RLQWHLQ [GLUHFWLRQ'HILQH6KDSH)XQFWLRQDW*DXVVLDQ4XDG3RLQWV SRLQWV,WVGHULYDWLYHRQQDWXUDOORFDOFRRUGLQDWHLVDOVRGHILQHG

GRNSQ 
GRMSQ 
GRLSQ 

HQGGR
HQGGR
HQGGR
GRLFHO ,&(/727/RRSIRU(OHPHQW-DFRELDQDQGGHULYDWLYHRQJOREDOFRRUGLQDWHRIVKDSHIXQFWLRQVDW*DXVVLDQ4XDG3RLQWVDUHGHILQHGDFFRUGLQJWRFRRUGLQDWHVRIQRGHV㸦-$&2%,㸧
GRLH 
GRMH 

/RFDO1RGH,'/RFDO1RGH,'*OREDO1RGH,'LSMS$GGUHVVRI$LSMS LQ͆LWHP͇NN

GRNSQ *DXVVLDQ4XDGSRLQWVLQ ]GLUHFWLRQGRMSQ *DXVVLDQ4XDGSRLQWVLQ KGLUHFWLRQGRLSQ  *DXVVLDQ4XDGSRLQWVLQ[GLUHFWLRQie
LQWHJUDWLRQRQHDFKHOHPHQWFRHIILFLHQWVRIHOHPHQWPDWULFHVDFFXPXODWLRQWRJOREDOPDWUL[
HQGGR
HQGGR
HQGGR
HQGGR
HQGGR
HQGGR

je

pFEM3D-2

64

඲䛶䛾せ⣲䛾ィ⟬䜢ᐇ᪋䛩䜛
እⅬ䜢ྵ䜐䜸䞊䝞䞊䝷䝑䝥㡿ᇦ䛾せ⣲䛾ィ⟬䜒ᐇ᪋
3(
3(







3(



























㸰
3(

㸱

㸲
3(






























































㸯



3(




































3(








3(



FEM3D65ᚑ䛳䛶⤖ᯝⓗ䛻䛿䛣䛾䜘䛖䛺⾜ิ䜢ᚓ䜛䛜
NP
N

NP

N

NP

N

internal

external

FEM3D66㯮ᯟ䛷ᅖ䜣䛰㒊ศ䛾⾜ิ䛿୙᏶඲䛧䛛䛧䠈ィ⟬䛻䛿౑⏝䛧䛺䛔䛾䛷䛣䜜䛷Ⰻ䛔
NP
N

NP

N

NP

N

internal

external

S2-ref67䝥䝻䜾䝷䝮䠖1d.f䠄9/11䠅ቃ⏺᮲௳䠖1CPU䛾䛸䛝䛸䜋䛸䜣䛹ྠ䛨&&
&_%281'$5<&21',7,216_
&
&

#0

1

1

2

&
& ; ;PLQ
LI P\BUDQNHT WKHQ
L 
M6 ,1'(; L$0$7 M6 G
',$* L G
5+6 L G
GRN 13/8
LI ,7(0 N HT $0$7 N G
HQGGR
HQGLI
&

2

3

#1

3

4

5

4

4

5

1

1

2

2

3

3

#2

4

5

5

4

6

1

1

2

2

3

3

4

S2-ref䝥䝻䜾䝷䝮䠖1d.f䠄10/11䠅
ඹᙺ໙㓄ἲ

&
&
&_&*LWHUDWLRQV_&
&
5 
= 
4 
3 
'' 
GRL 1: L'' '',$* L
HQGGR&& ^U` ^E` >$@^[LQL`_
&

LQLW
GRQHLE 1(,%3(727GRN H[SRUWBLQGH[ QHLE H[SRUWBLQGH[ QHLE
NN H[SRUWBLWHP N
6(1'EXI N 3+, NN
HQGGR
HQGGR

Compute r(0)= b-[A]x(0)for i= 1, 2, …
solve [M]z(i-1)= r(i-1)
Ui-1= r(i-1) z(i-1)
if i=1p(1)= z(0)else
Ei-1= Ui-1/Ui-2p(i)= z(i-1) + Ei-1 p(i-1)
endifq(i)= [A]p(i)Di = Ui-1/p(i)q(i)x(i)= x(i-1) + Dip(i)r(i)= r(i-1) - Diq(i)check convergence |r|end

68S2-ref69ඹᙺ໙㓄ἲ• ⾜ิ䝧䜽䝖䝹✚• ෆ✚• ๓ฎ⌮䠖1CPU䛾䛸䛝䛸ྠ䛨• DAXPY䠖1CPU䛾䛸䛝䛸ྠ䛨

S2-ref70๓ฎ⌮䠈DAXPY&& ^]` >0LQY@^U`
GRL 1: L= : L''  : L5
HQGGR&& ^[` ^[`$/3+$ ^S`&^U` ^U` $/3+$ ^T`
GRL 1
3+, L 3+, L $/3+$ : L3: L5 : L5  $/3+$ : L4
HQGGRS2-ref71⾜ิ䝧䜽䝖䝹✚䠄1/2䠅
㏻ಙ䝔䞊䝤䝹౑⏝䠈{p}䛾᭱᪂್䜢ィ⟬๓䛻ྲྀᚓ&
& ^T` >$@^S`
GRQHLE 1(,%3(727GRN H[SRUWBLQGH[ QHLE H[SRUWBLQGH[ QHLE
NN H[SRUWBLWHP N
6(1'EXI N : NN3
HQGGR
HQGGR
GRQHLE 1(,%3(727
LV H[SRUWBLQGH[ QHLE OHQBV H[SRUWBLQGH[ QHLE  H[SRUWBLQGH[ QHLE
FDOO03,B,VHQG 6(1'EXI LV OHQBV03,B'28%/(B35(&,6,21
1(,%3( QHLE 03,B&200B:25/'UHTXHVWBVHQG QHLE LHUU
HQGGR
GRQHLE 1(,%3(727
LU LPSRUWBLQGH[ QHLE OHQBU LPSRUWBLQGH[ QHLE  LPSRUWBLQGH[ QHLE
FDOO03,B,UHFY 5(&9EXI LU OHQBU03,B'28%/(B35(&,6,21
1(,%3( QHLE 03,B&200B:25/'UHTXHVWBUHFY QHLE LHUU
HQGGR
FDOO03,B:DLWDOO 1(,%3(727UHTXHVWBUHFYVWDWBUHFYLHUU
GRQHLE 1(,%3(727GRN LPSRUWBLQGH[ QHLE LPSRUWBLQGH[ QHLE
NN LPSRUWBLWHP N: NN3 5(&9EXI N
HQGGR
HQGGR

S2-ref72

⾜ิ䝧䜽䝖䝹✚䠄2/2䠅
{q}= [A]{p}
FDOO03,B:DLWDOO 1(,%3(727UHTXHVWBVHQGVWDWBVHQGLHUU
GRL 1: L4  ',$* L : L3
GRM ,1'(; L ,1'(; L: L4  : L4 $0$7 M : ,7(0 M 3
HQGGR
HQGGR

S2-ref73ෆ✚ྛ䝥䝻䝉䝇䛷ィ⟬䛧䛯್䜢䠈MPI_Allreduce䛷ྜィ&& 5+2 ^U`^]`
5+2 G
GRL 1
5+2 5+2: L5 : L=
HQGGR
FDOO03,B$OOUHGXFH 5+25+203,B'28%/(B35(&,6,21
03,B68003,B&200B:25/'LHUU

S2-ref74

MPI_REDUCE•

P#0 A0 B0 C0 D0P#1 A1 B1 C1 D1P#0 op.A0-A3 op.B0-B3 op.C0-C3 op.D0-D3
Reduce

P#1

P#2 A2 B2 C2 D2

P#2

P#3 A3 B3 C3 D3

P#3䝁䝭䝳䝙䜿䞊䝍䞊 䛂comm䛃ෆ䛾䠈ྛ䝥䝻䝉䝇䛾㏦ಙ䝞䝑䝣䜯䛂sendbuf䛃䛻䛴䛔䛶䠈₇⟬
䛂op䛃䜢ᐇ᪋䛧䠈䛭䛾⤖ᯝ䜢1䛴䛾ཷಙ䝥䝻䝉䝇䛂root䛃䛾ཷಙ䝞䝑䝣䜯䛂recbuf䛃䛻᱁⣡䛩䜛䚹– ⥲࿴䠈✚䠈᭱኱䠈᭱ᑠ ௚• call MPI_REDUCE(sendbuf,recvbuf,count,datatype,op,root,comm,ierr)
௵ព
௵ព

I
O

– countᩚᩘ– datatype ᩚᩘII– sendbuf– recvbuf

FORTRANC– op㏦ಙ䝞䝑䝣䜯䛾ඛ㢌䜰䝗䝺䝇䠈ཷಙ䝞䝑䝣䜯䛾ඛ㢌䜰䝗䝺䝇䠈䝍䜲䝥䛿䛂datatype䛃䛻䜘䜚Ỵᐃ䝯䝑䝉䞊䝆䛾䝃䜲䝈䝯䝑䝉䞊䝆䛾䝕䞊䝍䝍䜲䝥

MPI_INTEGER, MPI_REAL, MPI_DOUBLE_PRECISION, MPI_CHARACTER etc.
MPI_INT, MPI_FLOAT, MPI_DOUBLE, MPI_CHAR etc

ᩚᩘ

I

ィ⟬䛾✀㢮

MPI_MAX, MPI_MIN, MPI_SUM, MPI_PROD, MPI_LAND, MPI_BAND etc䝴䞊䝄䞊䛻䜘䜛ᐃ⩏䜒ྍ⬟䠖 MPI_OP_CREATE– rootᩚᩘIཷಙඖ䝥䝻䝉䝇䛾ID䠄䝷䞁䜽䠅– commᩚᩘI䝁䝭䝳䝙䜿䞊䝍䜢ᣦᐃ䛩䜛– ierrᩚᩘO᏶஢䝁䞊䝗

S2-ref

75㏦ಙ䝞䝑䝣䜯䛸ཷಙ䝞䝑䝣䜯• MPI䛷䛿䛂㏦ಙ䝞䝑䝣䜯䛃䠈䛂ཷಙ䝞䝑䝣䜯䛃䛸䛔䛖ኚᩘ䛜䛧䜀
䛧䜀Ⓩሙ䛩䜛䚹
• ㏦ಙ䝞䝑䝣䜯䛸ཷಙ䝞䝑䝣䜯䛿ᚲ䛪䛧䜒␗䛺䛳䛯ྡ⛠䛾㓄ิ䛷䛒䜛ᚲせ䛿䛺䛔䛜䠈ᚲ䛪䜰䝗䝺䝇䛜␗䛺䛳䛶䛔䛺䛡䜜䜀䛺䜙䛺䛔䚹

S2-ref76

MPI_REDUCE䛾౛䠄1/2䠅
call MPI_REDUCE(sendbuf,recvbuf,count,datatype,op,root,comm,ierr)real(kind=8):: X0, X1
call MPI_REDUCE(X0, X1, 1, MPI_DOUBLE_PRECISION, MPI_MAX, 0, <comm>, ierr)real(kind=8):: X0(4), XMAX(4)call MPI_REDUCE(X0, XMAX, 4, MPI_DOUBLE_PRECISION, MPI_MAX, 0, <comm>, ierr)ྛ䝥䝻䝉䝇䛻䛚䛡䜛䠈X0(i)䛾᭱኱್䛜0␒䝥䝻䝉䝇䛾XMAX(i)䛻ධ䜛䠄i=1~4䠅S2-ref77

MPI_REDUCE䛾౛䠄2/2䠅
call MPI_REDUCE(sendbuf,recvbuf,count,datatype,op,root,comm,ierr)real(kind=8):: X0, XSUM
call MPI_REDUCE(X0, XSUM, 1, MPI_DOUBLE_PRECISION, MPI_SUM, 0, <comm>, ierr)ྛ䝥䝻䝉䝇䛻䛚䛡䜛䠈X0䛾⥲࿴䛜0␒PE䛾XSUM䛻ධ䜛䚹
real(kind=8):: X0(4)call MPI_REDUCE(X0(1), X0(3), 2, MPI_DOUBLE_PRECISION, MPI_SUM, 0, <comm>, ierr)ྛ䝥䝻䝉䝇䛻䛚䛡䜛䠈䞉 X0䠄1䠅䛾⥲࿴䛜0␒䝥䝻䝉䝇䛾X0䠄3䠅䛻ධ䜛䚹
䞉 X0䠄2䠅䛾⥲࿴䛜0␒䝥䝻䝉䝇䛾X0䠄4䠅䛻ධ䜛䚹

S2-ref78

MPI_ALLREDUCE

P#0 op.A0-A3 op.B0-B3 op.C0-C3 op.D0-D3

P#0 A0 B0 C0 D0P#1 A1 B1 C1 D1All reduceP#1 op.A0-A3 op.B0-B3 op.C0-C3 op.D0-D3

P#2 A2 B2 C2 D2P#2 op.A0-A3 op.B0-B3 op.C0-C3 op.D0-D3

P#3 A3 B3 C3 D3P#3 op.A0-A3 op.B0-B3 op.C0-C3 op.D0-D3• MPI_REDUCE + MPI_BCAST• ⥲࿴䠈᭱኱್䜢ィ⟬䛧䛯䜙䠈ྛ䝥䝻䝉䝇䛷฼⏝䛧䛯䛔ሙྜ䛜ከ䛔• call MPI_ALLREDUCE(sendbuf,recvbuf,count,datatype,op, comm,ierr)
௵ព
௵ព

I
O

count
ᩚᩘ
datatype ᩚᩘ
op
ᩚᩘ
comm
ᩚᩘ
ierr
ᩚᩘ

I
I
I
I
O

– sendbuf– recvbuf
–
–
–
–
–

㏦ಙ䝞䝑䝣䜯䛾ඛ㢌䜰䝗䝺䝇䠈ཷಙ䝞䝑䝣䜯䛾ඛ㢌䜰䝗䝺䝇䠈䝍䜲䝥䛿䛂datatype䛃䛻䜘䜚Ỵᐃ䝯䝑䝉䞊䝆䛾䝃䜲䝈䝯䝑䝉䞊䝆䛾䝕䞊䝍䝍䜲䝥ィ⟬䛾✀㢮䝁䝭䝳䝙䜿䞊䝍䜢ᣦᐃ䛩䜛᏶஢䝁䞊䝗

S2-ref79

CGἲ䠄1/5䠅&& ^U` ^E` >$@^[LQL`GRQHLE 1(,%3(727GRN H[SRUWBLQGH[ QHLE H[SRUWBLQGH[ QHLE
NN H[SRUWBLWHP N
6(1'EXI N 3+, NN
HQGGR
HQGGR

Compute r(0)= b-[A]x(0)for i= 1, 2, …
solve [M]z(i-1)= r(i-1)
Ui-1= r(i-1) z(i-1)
if i=1GRQHLE 1(,%3(727p(1)= z(0)LV H[SRUWBLQGH[ QHLE OHQBV H[SRUWBLQGH[ QHLE  H[SRUWBLQGH[ QHLE
else
FDOO03,B,VHQG 6(1'EXI LV OHQBV
Ei-1= Ui-1/Ui-2
03,B'28%/(B35(&,6,21
1(,%3( QHLE 03,B&200B:25/' (i)p = z(i-1) + Ei-1 p(i-1)
UHTXHVWBVHQG QHLE LHUU
HQGGR
endif
q(i)= [A]p(i)GRQHLE 1(,%3(727
LU LPSRUWBLQGH[ QHLE Di = Ui-1/p(i)q(i)OHQBU LPSRUWBLQGH[ QHLE  LPSRUWBLQGH[ QHLE
x(i)= x(i-1) + Dip(i)FDOO03,B,UHFY 5(&9EXI LU OHQBU
03,B'28%/(B35(&,6,21
r(i)= r(i-1) - Diq(i)1(,%3( QHLE 03,B&200B:25/'
UHTXHVWBUHFY QHLE LHUU
check convergence |r|
HQGGR
FDOO03,B:DLWDOO 1(,%3(727UHTXHVWBUHFYVWDWBUHFYLHUU end
GRQHLE 1(,%3(727GRN LPSRUWBLQGH[ QHLE LPSRUWBLQGH[ QHLE
NN LPSRUWBLWHP N
3+, NN 5(&9EXI N
HQGGR
HQGGR
FDOO03,B:DLWDOO 1(,%3(727UHTXHVWBVHQGVWDWBVHQGLHUU

S2-ref80

CGἲ䠄2/5䠅
GRL 1: L5  ',$* L 3+, L
GRM ,1'(; L ,1'(; L: L5  : L5 $0$7 M 3+, ,7(0 M
HQGGR
HQGGR
%150 '
GRL 1%150 %1505+6 L  
: L5  5+6 L  : L5
HQGGR
FDOO03,B$OOUHGXFH %150%150
03,B'28%/(B35(&,6,21
03,B68003,B&200B:25/'LHUU
&GRLWHU ,7(5PD[&& ^]` >0LQY@^U`
GRL 1: L= : L''  : L5
HQGGR

&
& 5+2 ^U`^]`

Compute r(0)= b-[A]x(0)for i= 1, 2, …
solve [M]z(i-1)= r(i-1)
Ui-1= r(i-1) z(i-1)
if i=1p(1)= z(0)else
Ei-1= Ui-1/Ui-2p(i)= z(i-1) + Ei-1 p(i-1)
endifq(i)= [A]p(i)Di = Ui-1/p(i)q(i)x(i)= x(i-1) + Dip(i)r(i)= r(i-1) - Diq(i)check convergence |r|end

5+2 G
GRL 1
5+2 5+2: L5 : L=
HQGGR
FDOO03,B$OOUHGXFH 5+25+203,B'28%/(B35(&,6,21
03,B68003,B&200B:25/'LHUU

S2-ref81

CGἲ䠄3/5䠅&& ^S` ^]`LI,7(5 &%(7$ 5+25+2RWKHUZLVH
LI LWHUHT WKHQ
GRL 1: L3 : L=
HQGGR
HOVH
%(7$ 5+25+2
GRL 1: L3 : L= %(7$ : L3
HQGGR
HQGLI
&
& ^T` >$@^S`
GRQHLE 1(,%3(727GRN H[SRUWBLQGH[ QHLE H[SRUWBLQGH[ QHLE
NN H[SRUWBLWHP N
6(1'EXI N : NN3
HQGGR
HQGGR

Compute r(0)= b-[A]x(0)for i= 1, 2, …
solve [M]z(i-1)= r(i-1)
Ui-1= r(i-1) z(i-1)
if i=1p(1)= z(0)else
Ei-1= Ui-1/Ui-2p(i)= z(i-1) + Ei-1 p(i-1)
endifq(i)= [A]p(i)Di = Ui-1/p(i)q(i)x(i)= x(i-1) + Dip(i)r(i)= r(i-1) - Diq(i)check convergence |r|
end

GRQHLE 1(,%3(727
LV H[SRUWBLQGH[ QHLE OHQBV H[SRUWBLQGH[ QHLE  H[SRUWBLQGH[ QHLE
FDOO03,B,VHQG 6(1'EXI LV OHQBV03,B'28%/(B35(&,6,21
1(,%3( QHLE 03,B&200B:25/'
UHTXHVWBVHQG QHLE LHUU
HQGGR

S2-ref82

CGἲ䠄4/5䠅
GRQHLE 1(,%3(727
LU LPSRUWBLQGH[ QHLE (0)= b-[A]x(0)Computer
OHQBU LPSRUWBLQGH[ QHLE  LPSRUWBLQGH[ QHLE
FDOO03,B,UHFY 5(&9EXI LU OHQBU
for i= 1, 2, …
03,B'28%/(B35(&,6,21
(i-1)= r(i-1)
1(,%3( QHLE 03,B&200B:25/' solve [M]zUHTXHVWBUHFY QHLE LHUU
Ui-1= r(i-1) z(i-1)
HQGGR
if i=1
FDOO03,B:DLWDOO 1(,%3(727UHTXHVWBUHFYVWDWBUHFYLHUU

p(1)= z(0)else
Ei-1= Ui-1/Ui-2p(i)= z(i-1) + Ei-1 p(i-1)
endifq(i)= [A]p(i)Di = Ui-1/p(i)q(i)x(i)= x(i-1) + Dip(i)r(i)= r(i-1) - Diq(i)check convergence |r|

GRQHLE 1(,%3(727
GRN LPSRUWBLQGH[ QHLE LPSRUWBLQGH[ QHLE
NN LPSRUWBLWHP N: NN3 5(&9EXI NN
HQGGR
HQGGR
FDOO03,B:DLWDOO 1(,%3(727UHTXHVWBVHQGVWDWBVHQGLHUU
GRL 1: L4  ',$* L : L3
GRM ,1'(; L ,1'(; L: L4  : L4 $0$7 M : ,7(0 M 3
HQGGR
HQGGR
&
& $/3+$ 5+2^S`^T`end& G
GRL 1& &: L3 : L4
HQGGR
FDOO03,B$OOUHGXFH &&03,B'28%/(B35(&,6,2103,B68003,B&200B:25/'LHUU$/3+$ 5+2&S2-ref83

CGἲ䠄5/5䠅&& ^[` ^[`$/3+$ ^S`&^U` ^U` $/3+$ ^T`
GRL 1
3+, L 3+, L $/3+$ : L3: L5 : L5  $/3+$ : L4
HQGGR'150 GRL 1'150 '150: L5
HQGGR



FDOO03,B$OOUHGXFH '150'150
03,B'28%/(B35(&,6,21
03,B68003,B&200B:25/'LHUU
5(6,' GVTUW '150%150
LI P\BUDQNHTDQGPRG LWHU HT WKHQ
ZULWH  LSH LWHU5(6,'
HQGLI
LI 5(6,'OH(36 JRWR
5+2 5+2
HQGGR

Compute r(0)= b-[A]x(0)for i= 1, 2, …
solve [M]z(i-1)= r(i-1)
Ui-1= r(i-1) z(i-1)
if i=1p(1)= z(0)else
Ei-1= Ui-1/Ui-2p(i)= z(i-1) + Ei-1 p(i-1)
endifq(i)= [A]p(i)Di = Ui-1/p(i)q(i)x(i)= x(i-1) + Dip(i)r(i)= r(i-1) - Diq(i)check convergence |r|endS2-ref䝥䝻䜾䝷䝮䠖1d.f䠄11/11䠅
⤖ᯝ᭩䛝ฟ䛧䠖ྛ䝥䝻䝉䝇䛤䛸䛻ᐇ᪋&
& 287387
LI P\BUDQNHT WKHQ
ZULWH   SH
HQGLI

(7LPH67LPH(7LPH(7LPH

ZULWH  D 7(03(5$785(
GRL 1
ZULWH  L SHP\BUDQNL3+, L
HQGGR
FDOO03,B),1$/,=( LHUU
HQGSURJUDPKHDW'S

84

S2-ref• ၥ㢟ࡢᴫせ㸪ᐇ⾜᪉ἲ• ᒁᡤศᩓࢹ࣮ࢱࡢ⪃࠼᪉• ࣉࣟࢢ࣒ࣛࡢㄝ᫂
• ィ⟬౛

85

S2-ref86ィ⟬⤖ᯝ䠄1ḟඖ䠅䠖CGἲ㒊ศ
N=106䛾ሙྜ䛿100ᅇ཯᚟䛻せ䛩䜛᫬㛫1䝁䜰䜢ᇶ‽1䝜䞊䝗䞉16䝁䜰䜢ᇶ‽
197.6

ideal
N=10^4N=10^6

70.5

131.7

N=10^680.7 100.0

95.0

58.3
42.1

Speed-Up

Speed-Up

100.0

159.2

ideal

87.5

17.6

10.0
5.9

7.1

39.9

3.7
16.0

2.0

1.0

10.0

1.0

1

10
Core #

100

10

100
Core #

182.2

S2-ref87⌮᝿್䛛䜙䛾䛪䜜
• MPI㏻ಙ䛭䛾䜒䛾䛻せ䛩䜛᫬㛫– 䝕䞊䝍䜢㏦௜䛧䛶䛔䜛᫬㛫– 䝜䞊䝗㛫䛻䛚䛔䛶䛿㏻ಙ䝞䞁䝗ᖜ䛻䜘䛳䛶Ỵ䜎䜛• Gigabit Ethernet䛷䛿 1Gbit/sec.䠄⌮᝿್䠅– ㏻ಙ᫬㛫䛿㏦ཷಙ䝞䝑䝣䜯䛾䝃䜲䝈䛻ẚ౛• MPI䛾❧䛱ୖ䛜䜚᫬㛫– latency– ㏦ཷಙ䝞䝑䝣䜯䛾䝃䜲䝈䛻䜘䜙䛺䛔• ࿧䜃ฟ䛧ᅇᩘ౫Ꮡ䠈䝥䝻䝉䝇ᩘ䛜ቑຍ䛩䜛䛸ቑຍ䛩䜛ഴྥ– ㏻ᖖ䠈ᩘ䡚ᩘ༑Psec䛾䜸䞊䝎䞊• MPI䛾ྠᮇ䛾䛯䜑䛾᫬㛫– 䝥䝻䝉䝇ᩘ䛜ቑຍ䛩䜛䛸ቑຍ䛩䜛ഴྥ

88⌮᝿್䛛䜙䛾䛪䜜䠄⥆䛝䠅• ィ⟬᫬㛫䛜ᑠ䛥䛔ሙྜ䠄S1-䠏䛷䛿N䛜ᑠ䛥䛔ሙྜ䠅䛿䛣䜜䜙䛾ຠᯝ䜢↓ど䛷䛝䛺䛔䚹– ≉䛻䠈㏦ಙ䝯䝑䝉䞊䝆ᩘ䛜ᑠ䛥䛔ሙྜ䛿䠈䛂Latency䛃䛜ຠ䛟䚹

S2-ref891䝁䜰䡚16䝁䜰䛷䛒䜎䜚ᛶ⬟䛜ฟ䛶䛔䛺䛔௳
1䝁䜰䜢ᇶ‽
ideal
N=10^4N=10^6

Speed-Up

100.0

87.5
70.5
80.7
58.3
42.1
17.6

• 16䝁䜰䛷1䝁䜰䛾7.1ಸ⛬ᗘ䛾ᛶ⬟䛻䛧䛛䛺䛳䛶䛔䛺䛔䛾䛿䠈䝯䝰䝸➇ྜ䛾䛯䜑䚹– STREAM䛾䜿䞊䝇– ㏻ಙ䛜ཎᅉ䛷䛿䛺䛔

10.0
5.9

7.1

3.7
2.0

1.0

Memory1.0

110Core #

100

L1

L1

L1

L1

L1

L1

L1

L2
L1 L1

L1

L1

L1

L1

L1

L1

L1

C

C

C

C

C

C

C

C

C

C

C

C

C

C

C

C

S2-ref90ྎᙧ✚ศ䛷䛿䛒䜎䜚ᙳ㡪䛜↓䛔• 䕺䠖N=106䠈䖃䠖108䠈䕦䠖109䠈䠉䠖⌮᝿್• 1䝁䜰䛻䛚䛡䜛ィ ⤖ᯝ䠄sec.䠅䛛䜙䛭䜜䛮䜜⟬ฟ
ideal
N=10^6
N=10^8N=10^915.0

Speed-Up• ྎᙧ✚ศ䠖䜋䛸䜣䛹䝯䝰䝸䜢౑䜟䛺䛔䠈䝯䝰
䝸䛻㈇ᢸ䛾䛛䛛䜙䛺䛔䜰䝥䝸䜿䞊䝅䝵䞁• 1䝕䞊䝍䠄䝇䜹䝷䞊䠅䜢
Allreduce䛩䜛䛰䛡

20.0

10.0
7.9

5.0

4
³0 1  x 2 dx

4.0

1

S1-3

15.8

0.0

2.0
1.0

0

5

10
Core #

1520

S2-ref91

Strong-Scaling䛻䛚䛡䜛䛂Super-Linear䛃
Speed-Up

ideal
super-linear

actualPE#• ၥ㢟つᶍ䜢ᅛᐃ䛧䛶䠈౑⏝PEᩘ䜢ቑຍ䛥䛫䛶⾜䛳䛯ሙྜ䠈㏻ᖖ䛿㏻ಙ䛾ᙳ㡪䛾䛯䜑䛻䠈ຠ⋡䛿⌮᝿್䠄mಶ䛾
PE䜢౑⏝䛧䛯ሙྜ䠈⌮᝿ⓗ䛻䛿mಸ䛾ᛶ⬟䛻䛺䜛䠅䜘䜚䜒ప䛟䛺䜛䛾䛜ᬑ㏻䛷䛒䜛䚹• 䛧䛛䛧䠈䝇䜹䝷䞊䝥䝻䝉䝑䝃䠄PC➼䠅䛾ሙྜ䠈㏫䛻⌮᝿್䜘䜚䜒䠈㧗䛔ᛶ⬟䛜ฟ䜛ሙྜ䛜䛒䜛䚹䛣䛾䜘䛖䛺⌧㇟䜢䛂Super-Linear䛃䛸࿧䜆䚹– 䝧䜽䝖䝹ィ⟬ᶵ䛷䛿㉳䛣䜙䛺䛔䚹S2-ref92඾ᆺⓗ䛺ᣲື
3.00

1.0E+0240 % of peak

2.00

GFLOPSGFLOPS

2.50

1.50
1.00

1.0E+01

1.0E+00

8 % of peak

0.50
0.00

1.0E-01

1.0E+04

1.0E+05

1.0E+06

1.0E+07

DOF: Problem Size

IBM-SP3:

ၥ㢟ࢧ࢖ࢬࡀᑠࡉ࠸ሙྜࡣ࢟ࣕࢵࢩࣗࡢ
ᙳ㡪ࡢࡓࡵᛶ⬟ࡀⰋ࠸

1.0E+04

1.0E+05

1.0E+06

1.0E+07

DOF: Problem Size

Earth Simulator:኱つᶍ࡞ၥ㢟࡯࡝࣋ࢡࢺࣝ㛗ࡀ㛗ࡃ࡞ࡾ㸪ᛶ⬟ࡀ㧗࠸

S2-ref

93

୪ิィ⟬

Ideal

PE#
IBM-SP3:

PE㸦Processing Element㸧ᩘࡀᑡ࡞࠸
ሙྜࡣ࠸ࢃࡺࡿࢫ࣮ࣃ࣮ࢫ࣮࢝ࣛࠋ
PEᩘࡀቑຍࡍࡿ࡜㏻ಙ࣮࢜ࣂ࣮࣊ࢵࢻ
ࡢࡓࡵᛶ⬟పୗࠋ

Performance

Performance㻿㼠㼞㼛㼚㼓㻌㻿㼏㼍㼘㼕㼚㼓㻌㻔඲యၥ㢟つᶍᅛᐃ㻕

Ideal

PE#
Earth Simulator:

PEᩘࡀቑຍࡍࡿ࡜㸪㏻ಙ࣮࢜ࣂ࣮࣊ࢵࢻ࡟
ຍ࠼㸪PE࠶ࡓࡾࡢၥ㢟つᶍࡀᑠࡉࡃ࡞ࡿࡓ
ࡵᛶ⬟పୗࠋ

S2-ref

94

Super-Linear䛾⏕䛨䜛⌮⏤• 䜻䝱䝑䝅䝳䛾ᙳ㡪• 䝇䜹䝷䞊䝥䝻䝉䝑䝃䛷䛿䠈඲⯡䛻ၥ㢟つᶍ䛜ᑠ䛥䛔䜋䛹ᛶ⬟䛜㧗䛔䚹– 䜻䝱䝑䝅䝳䛾᭷ຠ฼⏝CPU
Register
FAST

Cache
SLOW

Main Memory

S2-ref95䝯䝰䝸䞊䝁䝢䞊䜒ពእ䛻᫬㛫䛛䛛䜛䠄1/2䠅
SENDbuf

neib#1neib#2neib#3neib#4BUFlength_e

BUFlength_e

BUFlength_e

BUFlength_e

export_index(0)+1

export_index(1)+1 export_index(2)+1

export_index(3)+1

export_index(4)do neib= 1, NEIBPETOT
do k= export_index(neib-1)+1, export_index(neib)
kk= export_item(k)SENDbuf(k)= VAL(kk)㏦ಙ䝞䝑䝣䜯䜈䛾௦ධ
enddo
enddo
ᗘ䛺䛹䛾ኚᩘ䜢┤᥋㏦ಙ䠈ཷಙ䛻౑do neib= 1, NEIBPETOTiS_e= export_index(neib-1) + 1
iE_e= export_index(neib )BUFlength_e= iE_e + 1 - iS_e䛖䛾䛷䛿䛺䛟䠈䛣䛾䜘䛖䛺䝞䝑䝣䜯䜈୍ᅇ௦ධ䛧䛶ィ⟬䛩䜛䛣䛸䜢່䜑䜛䚹call MPI_ISEND&&(SENDbuf(iS_e), BUFlength_e, MPI_INTEGER, NEIBPE(neib), 0,&&
MPI_COMM_WORLD, request_send(neib), ierr)enddo
call MPI_WAITALL (NEIBPETOT, request_send, stat_recv, ierr)S2-ref96䝯䝰䝸䞊䝁䝢䞊䜒ពእ䛻᫬㛫䛛䛛䜛䠄2/2䠅
do neib= 1, NEIBPETOTiS_i= import_index(neib-1) + 1
iE_i= import_index(neib )BUFlength_i= iE_i + 1 - iS_i
call MPI_IRECV&&(RECVbuf(iS_i), BUFlength_i, MPI_INTEGER, NEIBPE(neib), 0,&&
MPI_COMM_WORLD, request_recv(neib), ierr)enddo
call MPI_WAITALL (NEIBPETOT, request_recv, stat_recv, ierr)do neib= 1, NEIBPETOT
do k= import_index(neib-1)+1, import_index(neib)
kk= import_item(k)VAL(kk)= RECVbuf(k)
enddo
enddo

RECVbufཷಙ䝞䝑䝣䜯䛛䜙௦ධ

neib#1neib#2neib#3neib#4BUFlength_i

BUFlength_i

BUFlength_i

BUFlength_i

import_index(0)+1

import_index(1)+1 import_index(2)+1

import_index(3)+1

import_index(4)S2-ref97୪ิ᭷㝈せ⣲ἲ䠖䜎䛸䜑• 䛂ᒁᡤศᩓ䝕䞊䝍ᵓ㐀䛾㐺ษ䛺タィ䛃䛻ᑾ䛝䜛• ၥ㢟Ⅼ– ୪ิ䝯䝑䝅䝳⏕ᡂ䠈୪ิྍど໬– ᝏ᮲௳ၥ㢟䛻䛚䛡䜛୪ิ๓ฎ⌮ᡭἲ– ኱つᶍI/O

98୪ิィ⟬ྥ䛡ᒁᡤ䠄ศᩓ䠅䝕䞊䝍ᵓ㐀• ᕪศἲ䠈᭷㝈せ⣲ἲ䠈᭷㝈య✚ἲ➼ಀᩘ䛜␯⾜ิ䛾䜰䝥䝸䜿䞊䝅䝵䞁䛻䛴䛔䛶䛿㡿ᇦ㛫㏻ಙ䛿䛣䛾䜘䛖䛺ᒁᡤ䠄ศᩓ䠅䝕䞊䝍䛻䜘䛳䛶ᐇ᪋ྍ⬟– SPMD– ෆⅬ䡚እⅬ䛾㡰䛻䛂ᒁᡤ䛃␒ྕ௜䛡– ㏻ಙ䝔䞊䝤䝹䠖୍⯡໬䛥䜜䛯㏻ಙ䝔䞊䝤䝹• 㐺ษ䛺䝕䞊䝍ᵓ㐀䛜ᐃ䜑䜙䜜䜜䜀䠈ฎ⌮䛿⡆༢䚹
– ㏦ಙ䝞䝑䝣䜯䛻䛂ቃ⏺Ⅼ䛃䛾್䜢௦ධ
– ㏦ಙ䠈ཷಙ– ཷಙ䝞䝑䝣䜯䛾್䜢䛂እⅬ䛃䛾್䛸䛧䛶᭦᪂