:LVWHULD%'(& ฼⏝஦౛㸦,㸧
:LVWHULD%'(&㸦2G\VVH\㸧࡟࠾ࡅࡿ␯⾜ิゎἲࡢືⓗ࣮ࣝࣉࢫࢣࢪ࣮ࣗࣜࣥࢢ࡟ࡼࡿ㏻ಙ᭱㐺໬୰ ᓥ ◊ ࿃ᮾி኱Ꮫ᝟ሗᇶ┙ࢭࣥࢱ࣮
 ࠕ:LVWHULD%'(& ฼⏝஦౛ࠖ࡟ࡘ࠸࡚
ᮾி኱Ꮫ᝟ሗᇶ┙ࢭࣥࢱ࣮㸦ᮏࢭࣥࢱ࣮㸧࡛ࡣ㸪 ᖺ  ᭶  ᪥࡟ࠕィ⟬࣭ࢹ࣮ࢱ࣭Ꮫ⩦ࠖ⼥ྜࢫ࣮ࣃ࣮ࢥࣥ
ࣆ࣮ࣗࢱࢩࢫࢸ࣒㸦:LVWHULD%'(&㸧ࡢ㐠⏝ࢆ㛤ጞࡋࡓࠝࠞ
ࠋ
ࠕ:LVWHULD%'(&ࠖࡣࢩ࣑࣮ࣗࣞࢩࣙ
ࣥࣀ࣮ࢻ⩌㸦2G\VVH\㸧࡜ࢹ࣮ࢱ࣭Ꮫ⩦ࣀ࣮ࢻ⩌㸦$TXDULXV㸧ࡢ  ࡘࡢィ⟬ࣀ࣮ࢻ⩌ࢆ᭷ࡍࡿ࣊ࢸࣟࢪࢽ࢔ࢫ࡞ࢩࢫࢸ
࣒࡛࠶ࡾ㸪≉࡟ࠕィ⟬࣭ࢹ࣮ࢱ࣭Ꮫ
⩦ࠖ⼥ྜࡀ᥎㐍ࡉࢀ㸪ࢧ࢖ࣂ࣮✵㛫㸦௬᝿㸧࡜ࣇ࢕ࢪ࢝ࣝ✵㛫㸦⌧ᐇ㸧ࢆ㧗ᗘ࡟⼥ྜࡉࡏࡓ 6RFLHW\ ࡢᐇ⌧࡟ᅗ  :LVWHULD%'(& ࡢᴫせࠝࠞ኱ࡁࡃ㈉⊩ࡍࡿࡇ࡜ࡀᮇᚅࡉࢀ࡚࠸ࡿࠋࠕࢫ࣮ࣃ࣮ࢥࣥࣆ࣮ࣗࢸ࢕ࣥࢢࢽ࣮ࣗࢫ࡛ࠖࡣ㸪ࡇࢀ࠿ࡽ  ᖺ⛬ᗘࢆ࠿ࡅ࡚㸪ࠕ:LVWHULD%'(&ࠖࡢᵝࠎ࡞฼⏝஦౛ࢆ⤂௓ࡋ࡚࠸ࡃணᐃ࡛࠶ࡿࠋ௒ᅇࡣ➨  ᅇ࡜ࡋ࡚㸪ࢩ࣑࣮ࣗࣞࢩࣙࣥࣀ࣮ࢻ⩌㸦2G\VVH\㸧࡟࠾ࡅࡿ኱つᶍ୪ิᩘ್ゎἲࡢ㧗㏿໬࡟ࡘ࠸࡚⤂௓ࡍࡿࠋ
ࢩ࣑࣮ࣗࣞࢩࣙࣥࣀ࣮ࢻ⩌㸦2G\VVH\㸧ࡣࠕ)8-,7686XSHUFRPSXWHU35,0(+3&);ࠖ
ࣛࢵࢡ࠿ࡽᵓᡂࡉࢀ㸪ୡ⏺᭱㧗ᛶ⬟ࢆ᭷ࡍࡿࢫ࣮ࣃ࣮ࢥࣥࣆ࣮ࣗࢱࠕᐩᓅࠖ࡜ྠࡌᐩኈ㏻ᰴᘧ఍♫ࡢࠕ)8-,7683URFHVVRU$);ࠖࢆ  ࣀ࣮ࢻ㸦 ࢥ࢔㸧ᦚ㍕ࡋ࡚࠸ࡿࠋࠕ$);ࠖ
ࡣ㸪᭱ඛ➃ࡢ QP ࣉࣟࢭࢫ࡛〇㐀ࡉࢀ㸪 ಶࡢ₇⟬ࢥ࢔࡜  ಶࡲࡓࡣ  ಶࡢ࢔ࢩࢫࢱࣥࢺࢥ࢔
ࢆ᭷ࡋ㸪ಸ⢭ᗘᾋືᑠᩘⅬ₇⟬࡛ 7)/236 ࡢ⌮ㄽࣆ࣮ࢡᛶ⬟ࢆᐇ⌧ࡋ㸪ྜィࣆ࣮ࢡᛶ⬟ࡣ 3)/236 ࡛࠶ࡿࠋྛࣀ࣮ࢻࡣ *L% ࡢ +%0 ࣓ࣔࣜࢆᦚ㍕ࡋ㸪ࢩ࣑࣮ࣗࣞࢩࣙࣥࣀ࣮
ࢻ⩌㸦2G\VVH\㸧ࡢ⥲࣓ࣔࣜᐜ㔞ࡣ 7L%㸪⥲࣓ࣔࣜࣂࣥࢻᖜࡣ 3%㸭⛊࡛࠶ࡿࠋྛࣀ࣮ࢻ
ࡣࣂ࢖ࢭࢡࢩࣙࣥࣂࣥࢻᖜࡀ 7%㸭⛊ࡢࣀ࣮ࢻ㛫┦஫⤖ྜࢿࢵࢺ࣮࣡ࢡ㸦7RIX ࢖ࣥࢱ࣮ࢥ
ࢿࢡࢺ '㸧࡛⤖ྜࡉࢀ࡚࠸ࡿࠋ
㸬ࡣࡌࡵ࡟኱つᶍ࡞୪ิィ⟬ᶵࢆ౑⏝ࡍࡿሙྜ㸪ࣀ࣮ࢻᩘࡢቑຍ࡟ࡼࡗ࡚㏻ಙࡢ࣮࢜ࣂ࣮࣊ࢵࢻ
ࡣቑຍࡍࡿഴྥ࡟࠶ࡿࠋ୪ิィ⟬࡟࠾࠸࡚㏻ಙࡣᚲ㡲ࡢࣉࣟࢭࢫ࡛࠶ࡿࡀ㸪㏻ಙࢆ࡛ࡁ
ࡿ㝈ࡾຠ⋡ⓗ࡟ᐇ᪋ࡋ㸪๐ῶࡍࡿࡇ࡜ࡣ ()/236 ⣭ࢩࢫࢸ࣒࡟࠾࠸࡚㔜せ࡛࠶ࡿࠋ㏻ಙ

スーパーコンピューティングニュース- 49 -

Vol. 23, No.5

2021ࡢ๐ῶࡀ࢔ࣝࢦࣜࢬ࣒࡟୙Ᏻᐃࢆࡶࡓࡽࡍྍ⬟ᛶࡶ࠶ࡿࠋᮏ✏࡛ࡣ㸪ィ⟬⛉Ꮫ㸪ィ⟬ᕤᏛ࡟࠾࠸࡚ᗈࡃ౑⏝ࡉࢀ࡚࠸ࡿ೫ᚤศ᪉⛬ᘧࡢᩘ್ゎἲ࡛࠶ࡿ㸪᭷㝈せ⣲ἲ࠿ࡽᑟฟࡉ
ࢀࡿ␯⾜ิࢆಀᩘ࡜ࡍࡿ኱つᶍ⥺ᙧ᪉⛬ᘧ㸦㐃❧୍ḟ᪉⛬ᘧ㸧ࢆ㸪๓ฎ⌮௜ࡁ୪ิ཯᚟
ἲ㸦Preconditioned Parallel Iterative Methods㸧࡟ࡼࡗ࡚ゎࡃሙྜ࡟ࡘ࠸᳨࡚ウࡍࡿࠋᮏ✏ࡢᡭἲࡣⴭ⪅➼ࡢඛ⾜◊✲ࠝ2ࠞ࡟ᇶ࡙ࡃࡶࡢ࡛࠶ࡿࠋ


2㸬୪ิ᭷㝈せ⣲ἲ࢔ࣉࣜࢣ࣮ࢩࣙࣥࠕGeoFEM/Cubeࠖᮏ✏࡛ᑐ㇟࡜ࡋ࡚࠸ࡿ࢔ࣉࣜࢣ࣮ࢩࣙࣥࡣ㸪GeoFEM ࣉࣟࢪ࢙ࢡࢺࠝ3㸪4࡛ࠞ㛤Ⓨࡉࢀࡓ୪ิ
᭷㝈せ⣲ἲ࢔ࣉࣜࢣ࣮ࢩࣙࣥࢆඖ࡟ᩚഛࡋࡓᛶ⬟ホ౯ࡢࡓࡵࡢ࣋ࣥࢳ࣐࣮ࢡࣉࣟࢢࣛ ࣒
ࠕGeoFEM㸭Cube࡛ࠖ࠶ࡾ㸪ᮏࢭࣥࢱ࣮࡛ࡣ㸪ࢫ࣮ࣃ࣮ࢥࣥࣆ࣮ࣗࢱࡢㄪ㐩᫬ࡢᛶ⬟ホ౯ヨ㦂࡟౑⏝ࡋ࡚࠸ࡿࠋᮏ࣋ࣥࢳ࣐࣮ࢡࡣ㸪ᆒ㉁ሙ࡟࠾ࡅࡿ୕ḟඖᙎᛶ㟼ゎᯒၥ㢟㸦Cube ᆺࣔࢹࣝ㸦ᅗ
2㸧㸧࡟᭷㝈せ⣲ἲࢆ㐺⏝ࡍࡿࡇ࡜࡟ࡼࡾᑟ࠿ࢀࡿ኱つᶍ␯⾜ิࢆಀᩘ⾜ิ࡜ࡍࡿ㐃❧୍ḟ᪉⛬ᘧࢆ୪ิ๓ฎ⌮௜ࡁ཯᚟ἲ㸦ࢡࣜࣟࣇ㒊ศ✵㛫ἲࠝ5ࠞ㸧࡟ࡼࡗ࡚ゎࡃࡶࡢ࡛࠶ࡾ㸪཯᚟ἲࢯࣝࣂ࣮
ࡢᐇ⾜᫬ᛶ⬟ࢆᵝࠎ࡞᮲௳ୗ࡛ィࡍࡿࠋせ⣲ࢱ࢖ࣉࡣ୕ḟඖ୍ḟභ㠃యせ⣲㸦tri-linear㸧࡛࠶ࡾ㸪ྛせ⣲ 8 ࡘࡢ⠇Ⅼࢆ᭷ࡋ࡚࠸ࡿࠋྛせ⣲ࡣ㎶㛗ࡉ㸻1 ࡢ༢఩❧᪉య࡛࠶ࡾ㸪x㸪y㸪㹸ྛ᪉ྥࡢ⠇Ⅼᩘࡣࡑࢀࡒࢀ㸪Nx㸪Ny㸪Nz ࡛࠶ࡿ㸦ᚑࡗ࡚せ⣲ᩘࡣ㸪Nx-1㸪Ny-1㸪Nz-1㸧ࠋz

Uniform Distributed Force inz-direction @ z=Zmax

Uy=0 @ y=Ymin



Ux=0 @ x=Xmin(Nz-1) elements
Nz nodes(Ny-1) elements
Ny nodesy

Uz=0 @ z=Zminx(Nx-1) elements
Nx nodesᅗ 2 GeoFEM/Cube ࡢゎᯒᑐ㇟ࠝ2,3,4ࠞ୕ḟඖᙎᛶၥ㢟࡛ࡣ 1 ⠇Ⅼ࠶ࡓࡾ 3 ࡘࡢ⮬⏤ᗘࡀ࠶ࡿࡓࡵ㸪ࡇࢀࡽࢆ 1 ࡘࡢࣈࣟࢵࢡ࡜ࡋ࡚ྲྀ
ࡾᢅࡗ࡚࠸ࡿ㸦ᅗ 3㸧ࠋಀᩘ⾜ิࡣࡇࡢࣈࣟࢵࢡᆺࡢ≉ᛶࢆ฼⏝ࡋࡓࣈࣟࢵࢡ CRS ᙧᘧ㸦Compressed Row Storage㸧࡟ࡼࡗ࡚᱁⣡ࡉࢀ࡚࠸ࡿࠋ
᭷㝈せ⣲ἲ࡟࠾࠸࡚㸦ཝᐦ࡟ࡣ࣮࢞ࣛ࢟ࣥἲ࡟ᇶ࡙ࡃ᭷㝈せ⣲ἲ࡟࠾࠸࡚㸧㸪୕ḟඖᙎᛶ㟼ゎᯒၥ㢟࡛ࡣಀᩘ⾜ิࡀᑐ⛠ṇᐃ࡞␯⾜ิ࡜࡞ࡿࡇ࡜࠿ࡽ㸪๓ฎ⌮ࢆ᪋ࡋࡓඹᙺ໙㓄ἲ㸦Conjugate
Gradient㸪CG㸧ἲࠝ5ࠞ
㸦ᅗ 4㸧࡟ࡼࡗ࡚㐃❧୍ḟ᪉⛬ᘧࢆゎ࠸࡚࠸ࡿࠋ๓ฎ⌮ᡭἲ࡜ࡋ࡚ࡣ㸪ᵝࠎ࡞ᡭἲࡀ฼⏝ྍ⬟࡛࠶ࡿࡀ㸪ᮏ✏࡛ࡣ㸪ᅗ 3 ࡟♧ࡍᑐゅࣈࣟࢵࢡ࡟ᑐࡋ࡚ࣇࣝ LU ศゎࢆ㐺⏝ࡍࡿᡭἲ㸦Block Diagonal LU Factorization㸧ࠝ2ࠞࢆ㐺⏝ࡋ࡚࠸ࡿࠋࡇࡢᡭἲࡣ㸪ྛ⠇Ⅼ༢఩࡛⊂❧࡟๓ฎ⌮ࢆ㐺⏝࡛ࡁࡿࡓࡵ㸪୪ิ໬ࡀᐜ࡛᫆࠶ࡾ㸪MPI ࡟ࡼࡿ㏻ಙࡶⓎ⏕ࡋ࡞࠸ࠋ
スーパーコンピューティングニュース- 50 -

Vol. 23, No.5

2021

j


(− 1,+1,+1) 8

(− 1,−1,+1) 5i

7 (+ 1,+1,+1)
6 (+ 1,−1,+1)(− 1,+1,−1)


4


1



2

(ξ ,η , ζ ) = (− 1,−1,−1)

3 (+ 1,+1,−1)(+ 1,−1,−1) ªa¬ ie je 31



aie je 13 º»aiie je » (ie , je = 1!8)23
aie je 33 »¼

a ie je12
a ie je22
aie je 32« ie je 11 a« ie je 21 «aᅗ 3 8 ⠇Ⅼභ㠃యせ⣲㸪1 ⠇Ⅼ 3 ⮬⏤ᗘࡢฎ⌮ࠝ2,3,4ࠞ

Compute r(0)= b-[A]x(0)for i= 1, 2, …
せ⣲ࢆ᭷ࡍࡿ⠇Ⅼ࣮࣋ࢫࡢศ๭ᡭἲࠝ2,3,4ࠞࡀᗈࡃ౑
solve [M]z(i-1)= r(i-1)⏝ࡉࢀ࡚࠾ࡾ㸪ᮏ✏࡛ࡶࡑࢀ࡟ᇶ࡙࠸࡚࠸ࡿࠋ᭷㝈せ
ρi-1= r(i-1) z(i-1)
if i=1⣲ἲ࡟࠾࠸࡚ࡣ㸪኱つᶍ㐃❧୍ḟ᪉⛬ᘧࡢồゎ࡟ィ⟬p(1)= z(0)᫬㛫ࡢ኱㒊ศࡀ㈝ࡸࡉࢀࡿࡇ࡜࠿ࡽ㸪୪ิィ⟬᫬࡟ࡣelse⮬⏤ᗘࡀᐃ⩏ࡉࢀࡿ⠇Ⅼ㸦Node㸧ࡀྛࣉࣟࢭࢫ࡛ᆒ➼βi-1= ρi-1/ρi-2p(i)= z(i-1) + βi-1 p(i-1)࡜࡞ࡿࡼ࠺࡟㡿ᇦศ๭ࢆ㐺⏝ࡍࡿࠋGeoFEM/Cube ࡣつ
endif๎ṇࡋ࠸ᙧ≧ࢆᢅ࠺ࡀ㸪ࣉࣟࢢ࣒ࣛෆ࡟୪ิ࣓ࢵࢩࣗ
q(i)= [A]p(i)αi = ρi-1/p(i)q(i)⏕ᡂᶵ⬟ࢆෆⶶࡋ㸪ࣉࣟࢭࢫ㛫㈇Ⲵࣂࣛࣥࢫࢆ⮬ືⓗx(i)= x(i-1) + αip(i)࡟ㄪᩚࡋ࡚࠸ࡿࠋᅗ 5 ࡣ㸪25 ⠇Ⅼ㸪16 せ⣲㸦ᅄゅᙧ㸧
r(i)= r(i-1) - αiq(i)࠿ࡽᵓᡂࡉࢀࡿ஧ḟඖゎᯒᑐ㇟ࢆ 4 ศ๭ࡋࡓ౛࡛࠶
check convergence |r|end
ࡿࠋPE#0㹼PE#3㸦PE=Processing Element㸧ࡢ 4 㡿ᇦ࡟᭷㝈せ⣲ἲࢆ୪ิ໬ࡍࡿሙྜ࡟ࡣ㸪࣮࢜ࣂ࣮ࣛࢵࣉ

ศ๭ࡉࢀ㸪ྛ㡿ᇦ࡟ MPI ࣉࣟࢭࢫࡀ๭ࡾᙜ࡚ࡽࢀࡿࠋ

ᅗ 4 ๓ฎ⌮௜ࡁඹᙺ໙㓄ἲࡢ࢔ࣝࢦࣜࢬ࣒ࠝ5ࠞᅗ 5㸦b㸧ࡣྛࣉࣟࢭࢫ࡛ᢅ࠺ᒁᡤศᩓࢹ࣮ࢱࢆ♧ࡋ࡚࠸ࡿࠋ
(a)

PE#1

PE#0

(b) 4

5

6

12

1

2

3

11

7

8

9

10

15

6

7

13

14

4

5

9

1

2

3

8

10

11

12

PE#0

PE#1
10

11

12

5

6

9

9

10

11

12

3

4

8

8

4

5

6

1

2

7

7

1

2

3

PE#3

PE#3

PE#2

ᅗ 5 ୪ิ᭷㝈せ⣲ἲ࡟࠾ࡅࡿ㡿ᇦศ๭
ࠝ2,3,4ࠞ㸦a㸧⠇Ⅼ࣮࣋ࢫศ๭㸪㸦b㸧ྛࣉࣟࢭࢫࡢᒁᡤศᩓࢹ࣮ࢱᵓ㐀

PE#2



スーパーコンピューティングニュース- 51 -

Vol. 23, No.5

2021

ᅗ 5㸦b㸧࡟♧ࡍ PE#2 ࡣ㸪ᅗ 5㸦a㸧࡟♧ࡍࡼ࠺࡟ᮏ᮶㯤Ⰽ࠸⠇Ⅼࡀ๭ࡾᙜ࡚ࡽࢀ࡚࠸ࡿࡀ㸪ࡑࢀ௨እ࡟㸪㯤Ⰽ࠸⠇Ⅼࢆྵࡴせ⣲㸪ࡑࢀ࡟ྵࡲࢀࡿ௚ࡢ㡿ᇦ࡟๭ࡾᙜ࡚ࡽࢀࡓ⠇Ⅼ㸦㉥㸦PE#3㸧㸪
㟷㸦PE#1㸧㸧ࡀྵࡲࢀ࡚࠸ࡿࠋ୪ิ᭷㝈せ⣲ἲ࡟࠾ࡅࡿᒁᡤศᩓࢹ࣮ࢱᵓ㐀࡟࠾࠸࡚ࡣ㸪⠇Ⅼࢆ
௨ୗࡢ 3 ✀㢮ࡢ࢝ࢸࢦࣜ࡟ศ㢮ࡋ࡚࠸ࡿࠝ2,3,4ࠞࠋy ಼఼ʤInternal Nodesʥʁຌཔͨ͹྘ҮͶׄΕ౲ͱΔΗͪઇ఼ʤਦ 5ʤaʥPE#2 ͹ԭ৯͏ઇ఼ʥy ֐఼ʤExternal Nodesʥʁຌཔҡ͵Ζ྘ҮͶׄΕ౲ͱΔΗͪઇ఼y ‫ڧ‬ֆ఼ʤBoundary Nodesʥʁʰ಼఼ʱ͹͑ͬɾଠ͹྘Ү͹ʰ֐఼ʱͲ͍Ζઇ఼ʤ֐఼ͶંͤΖʥ
ᅗ 5 ࡟♧ࡍࡼ࠺࡞࣮࢜ࣂ࣮ࣛࢵࣉせ⣲㸪እⅬࢆྵࡴᒁᡤࢹ࣮ࢱᵓ㐀ࢆ㐺⏝ࡍࡿࡇ࡜࡟ࡼࡾ㸪せ⣲࣐ࢺࣜࢡࢫ㸪඲య࣐ࢺࣜࢡࢫࡢ⏕ᡂࢆ㏻ಙ↓ࡋ࡟ᐇ᪋ࡍࡿࡇ࡜ࡀ࡛ࡁ㸪ࡲࡓ㸪ඹᙺ໙㓄ἲࡢࡼ࠺࡞཯᚟ἲࢆ୪ิ໬ࡍࡿࡇ࡜ࡶᐜ࡛᫆࠶ࡿࠝ2,3,4ࠞࠋྛᒁᡤศᩓࢹ࣮ࢱ࡟࠾ࡅࡿᒁᡤ⠇Ⅼ␒ྕࡣ㸪ᅗ 5㸦b㸧࡟♧ࡍࠕෆⅬэእⅬࠖࡢࡼ࠺࡟ࡍࡿ࡜㏻ಙ➼ࡀᐜ᫆࡟ᐇ᪋࡛ࡁࡿࠋ࡞࠾㸪OpenMP/MPIࣁ࢖ࣈࣜࢵࢻ୪ิࣉࣟࢢ࣑ࣛࣥࢢࣔࢹࣝࢆ㐺⏝ࡍࡿሙྜ࡟ࡣ㸪ᅗ 5㸦b㸧ࡢྛ MPI ࣉࣟࢭࢫ࡟๭
ࡾᙜ࡚ࡽࢀࡓᒁᡤศᩓࢹ࣮ࢱ࡟ᑐࡋ࡚ OpenMP ࡟ࡼࡿࢫࣞࢵࢻ୪ิ໬ࢆ㐺⏝ࡍࡿࠝ2,3,4ࠞࠋ୪ิ᭷㝈せ⣲ἲ࡛ MPI ࣉࣟࢭࢫ㛫ࡢࢢ࣮ࣟࣂࣝ࡞㏻ಙࡀⓎ⏕ࡍࡿࡢࡣ㸪ᮏ✏࡛ᢅ࠺ࡼ࠺࡞⥺ᙧ㸪ᐃᖖࡢၥ㢟ࡢሙྜ࡟ࡣ㸪୪ิ๓ฎ⌮௜ࡁ཯᚟ἲ࡟ࡼࡿ⥺ᙧ᪉⛬ᘧࡢồゎ㒊ࡢࡳ࡛࠶ࡿࠋᮏ✏࡛ࡣ㸪Block Diagonal LU Factorization ๓ฎ⌮ࢆ㐺⏝ࡋ࡚࠸ࡿࡓࡵ㸪ᅗ 4 ࡟♧ࡍࡼ࠺࡞๓ฎ⌮௜ࡁඹᙺ໙㓄ἲ࡟࠾࠸࡚㸪๓ฎ⌮㐺⏝㒊ศ㸦solve [M]z(i-1)=r(i-1)㸧ࡣ㸪ྛ⠇Ⅼ᏶඲࡟⊂❧࡟ィ⟬ࡋ㸪MPI ࣉࣟࢭࢫ㛫ࡢ㏻ಙࡣⓎ⏕ࡋ࡞࠸ࠋࡋࡓࡀࡗ࡚㸪MPI ࣉࣟࢭࢫ㛫㏻ಙࡀⓎ⏕ࡍࡿࡢࡣ㸪ձෆ✚㸪ղ␯⾜ิ࣋ࢡࢺࣝ✚㸦q(i)= [A]p(i)㸧ࡢࡳ࡛࠶ࡿࠋղ࡛ࡣ㸪ᅗ 5㸦b㸧࡟♧ࡋࡓᒁᡤศᩓ
ࢹ࣮ࢱࡢ≉ᛶࢆ฼⏝ࡍࡿࠋྛ MPI ࣉࣟࢭࢫ࡛ q(i)= [A]p(i)ࢆィ⟬ࡍࡿࡓࡵ࡟ࡣ㸪እⅬ࡟࠾ࡅࡿ
p(i)ࡢ್ࢆồࡵ࡚࠾ࡃᚲせࡀ࠶ࡿࠋࡑࡢࡓࡵ࡟ࡣ MPI ࣉࣟࢭࢫ㛫㏻ಙ࡟ࡼࡗ࡚㸪እⅬࡢ್ࢆ௚ࡢ
MPI ࣉࣟࢭࢫ࠿ࡽணࡵཷࡅྲྀࡗ࡚࠾ࡃᚲせࡀ࠶ࡿࠋGeoFEM/Cube ࡛ࡣ㸪ձቃ⏺Ⅼࡢ್ࢆ
MPI_Isend ࡟ࡼࡗ࡚㏦ಙ㸪ղእⅬࡢ್ࢆ MPI_Irecv ࡟ࡼࡗ࡚ཷಙ㸪ճMPI_Waitall ࡟ࡼࡗ࡚㏻ಙࢆ᏶஢㸪࡜࠸࠺ᡭ㡰࡟ࡼࡗ࡚ᐇ᪋ࡋ࡚࠸ࡿࠝ2,3,4ࠞ
㸦ᅗ 6㸧ࠋእⅬࡢࡇ࡜ࢆࠕHaloࠖ࡜࿧ࡪࡇ࡜ࡀ࠶
ࡾ㸪ᅗ 6 ࡟♧ࡍࡼ࠺࡞㏻ಙࢆࠕHalo ㏻ಙࠖ࡜࿧ࡪࡇ࡜ࡶ࠶ࡿࠋ
DOORFDWH VWDWBVHQG 03,B67$786B6,=( 1(,%3(727
DOORFDWH UHTXHVWBVHQG  1(,%3(727RPS SDUDOOHOGRSULYDWH QHLENNN
GRQHLE 1(,%3(727GRN H[SRUWBLQGH[ QHLE H[SRUWBLQGH[ QHLE
NN H[SRUWBLWHP N
6(1'EXI N : NN3
HQGGR
HQGGR

15

6

7

13

14

4

5

9

1

2

3

8

10

11

12

PE#0

ቃ⏺Ⅼ䠖㏦ಙPE#3

GRQHLE 1(,%3(727
LV H[SRUWBLQGH[ QHLE OHQBV H[SRUWBLQGH[ QHLE  H[SRUWBLQGH[ QHLE
FDOO03,B,VHQG 6(1'EXI LV OHQBV03,B'28%/(B35(&,6,21
1(,%3( QHLE 03,B&200B:25/'
UHTXHVWBVHQG QHLE LHUU
HQGGR

10

11

12

5

6

9

9

10

11

12

3

4

8

8

4

5

6

1

2

7

7

1

2

3

GRQHLE 1(,%3(727
LULPSRUWBLQGH[ QHLE OHQBU LPSRUWBLQGH[ QHLE  LPSRUWBLQGH[ QHLE
FDOO03,B,UHFY : LU13 OHQBU03,B'28%/(B35(&,6,21
1(,%3( QHLE 03,B&200B:25/'
UHTXHVWBVHQG QHLE1(,%3(727 LHUU
HQGGR
FDOO03,B:DLWDOO  1(,%3(727UHTXHVWBVHQGVWDWBVHQGLHUU

15

6

7

13

14

4

5

9

1

2

3

8

10

11

12

PE#0

እⅬ䠖ཷಙ
PE#3

PE#2

10

11

12

5

6

9

9

10

11

12

3

4

8

8

4

5

6

1

2

7

7

1

2

3

PE#2

ᅗ 6 MPI_Isend/MPI_Recv/MPI_Waitall ࡟ࡼࡿቃ⏺Ⅼ㏦ಙ࣭እⅬཷಙ㸦Halo ㏻ಙ㸧ࠝ2,3,4ࠞ


スーパーコンピューティングニュース- 52 -

Vol. 23, No.5

2021

3. ㏻ಙ࡜ィ⟬ࡢ࣮࢜ࣂ࣮ࣛࢵࣉ
ศᩓ࣓ࣔࣜᆺ୪ิィ⟬ᶵୖ࡛୪ิ๓ฎ⌮௜ࡁ཯᚟ἲࢆᐇ᪋ࡍࡿሙྜࡣ㸪ձ␯⾜ิ࣋ࢡࢺࣝ
✚㸪ղෆ✚㸪ճ๓ฎ⌮㸪࡛㏻ಙࡀⓎ⏕ࡍࡿྍ⬟ᛶࡀ࠶ࡿࠋᮏ✏࡟࠾ࡅࡿ㸪౛࡛ࡣ๓ฎ⌮㒊ศ࡛
ࡣ㏻ಙࡣⓎ⏕ࡋ࡞࠸ࠋ␯⾜ิ࣋ࢡࢺࣝ✚࡛ࡣ㞄᥋ࣉࣟࢭࢫ࡜ࡢ 1 ᑐ 1 ㏻ಙ㸦Point-to-Point
Communication㸧㸪ෆ✚࡛ࡣ඲ࣉࣟࢭࢫ࡜ࡢ㞟ྜ㏻ಙ㸦Collective Communication㸧ࡀⓎ⏕ࡍࡿࠋMPI ࢆ౑⏝ࡍࡿሙྜࡣ㸪ձࡣᅗ 6 ࡛♧ࡋࡓࡼ࠺࡞ Halo ㏻ಙࢆ㐺⏝ࡋ㸪MPI_Isend㸪MPI_Irecv ࢆ
MPI_Waitall ࡜⤌ࡳྜࢃࡏ㸪ղ࡟࠾࠸࡚ࡣ MPI_Allreduce ࡢࡼ࠺࡞㛵ᩘࡀ౑⏝ࡉࢀࡿࠋ㏻ಙ࡟ࡼࡿ࣮࢜ࣂ࣮࣊ࢵࢻࢆ๐ῶࡍࡿࡓࡵ࡟㸪㏻ಙᅇ㑊࣭๐ῶᆺ࢔ࣝࢦࣜࢬ࣒
㸦Communication Avoiding/Reducing Algorithm㸧ࡢ◊✲㛤Ⓨࡀ┒ࢇ࡟㐍ࡵࡽࢀ࡚࠸ࡿࠋࣃ࢖ࣉࣛ
࢖ࣥᆺඹᙺ໙㓄ἲ㸦Pipelined CG Method㸧ࠝ6ࠞࡣ㸪₞໬ᘧࡢ㐺⏝࡟ࡼࡗ࡚㸪ᅗ 4 ࡟♧ࡋࡓ࢜ࣜࢪࢼࣝࡢ๓ฎ⌮௜ࡁ CG ἲࡢ࢔ࣝࢦࣜࢬ࣒ࡀኚࢃࡽ࡞࠸ࡼ࠺࡟ィ⟬ࡢ㡰ᗎࢆኚ᭦ࡍࡿᡭἲ࡛࠶
ࡿࠋෆ✚ࡢ┤ᚋ࡟␯⾜ิ࣋ࢡࢺࣝ✚㸪๓ฎ⌮࡞࡝ࡢィ⟬㔞ࡀከࡃ㸪࠿ࡘ┤๓࡛ᐇ᪋ࡋࡓෆ✚ࡢ⤖ᯝࢆ౑ࢃ࡞࠸ࡼ࠺࡞ฎ⌮ࢆᐇ⾜ࡍࡿࡼ࠺࡟ィ⟬㡰ᗎࡀኚ᭦ࡉࢀࡿࠋPipelined ἲ࡛ࡣ㸪ࡇࢀ࡟
MPI-3 ࡛ࢧ࣏࣮ࢺࡉࢀ࡚࠸ࡿ MPI_Iallreduce ➼ࡢ㠀ྠᮇ㞟ᅋ㏻ಙ㸦Asynchronous Collective
Communication㸧ࢆ⤌ࡳྜࢃࡏࡿࡇ࡜࡟ࡼࡗ࡚㸪㞟ᅋ㏻ಙ࡜␯⾜ิ࣋ࢡࢺࣝ✚㸪๓ฎ⌮➼ࡢ₇⟬ࢆ࣮࢜ࣂ࣮ࣛࢵࣉࡍࡿࡇ࡜ࡀ࡛ࡁ㸪㏻ಙ࡟ࡼࡿ࣮࢜ࣂ࣮࣊ࢵࢻࡢ㞃ⶸࡀྍ⬟࡜࡞ࡿࠝ6ࠞࠋᮏ✏࡛ࡣ㸪≉࡟␯⾜ิ࣋ࢡࢺࣝ✚㒊ศࡢ㧗㏿໬࡟ࡘ࠸᳨࡚ウࡍࡿࠋ␯⾜ิ࣋ࢡࢺࣝ✚ࡢィ⟬ࢆᐇ᪋ࡍࡿሙྜ࡟ࡣ㸪ᅗ 6 ࡟♧ࡍࡼ࠺࡞ Halo ㏻ಙ࡟ࡼࡗ࡚㸪ྛ MPI ࣉࣟࢭࢫ㛫࡛㏻ಙࢆᐇ᪋
ࡋ㸪᭱᪂ࡢ p ࣋ࢡࢺࣝࡢ್ࢆᚓࡓࡢࡕ㸪ྛ MPI ࣉࣟࢭࢫ࡛ෆⅬ࡟ᑐࡋ࡚␯⾜ิ࣋ࢡࢺࣝ✚ࡢ₇⟬ࢆᐇ᪋ࡍࡿࠝ2,3,4ࠞࠋෆⅬࡢ࠺ࡕ㸪ቃ⏺ⅬࡣእⅬ࡟᥋ࡋ࡚࠸ࡿࡓࡵ㸪እⅬࡢ᭱᪂ࡢ್ࡀ࡞࠸࡜
␯⾜ิ࣋ࢡࢺࣝ✚ࢆィ⟬࡛ࡁ࡞࠸ࡀ㸪ቃ⏺Ⅼ௨እࡢෆⅬ㸦⣧ෆⅬ㸦Pure Internal Nodes㸧㸧࡛ࡣ㏻ಙࡢ᏶஢ࢆᚅࡓࡎ࡟ィ⟬ࢆ㛤ጞ࡛ࡁࡿࠋࡇࢀ࡟ᇶ࡙ࡁ㸪ᅗ 6 ࡟♧ࡍ MPI ࣉࣟࢭࢫ㛫ࡢ Halo ㏻ಙ࡜㸪ࠕ⣧ෆⅬࠖࡢィ⟬ࢆྠ᫬࡟ᐇ᪋ࡍࡿᡭἲࡀ㸪ྂ඾ⓗ࡞ࠕ㏻ಙ࡜ィ⟬ࡢ࣮࢜ࣂ࣮ࣛࢵࣉࠖ
ࠝ7࡛ࠞ࠶ࡿ㸦ᅗ 7㸧ࠋᅗ 8ࠕStatic࡛ࠖィ⟬ᡭ㡰ࡢᴫせࢆ⤂௓ࡋ࡚࠸ࡿࡼ࠺࡟㸪MPI_Isend㸪
MPI_Irecv ࡟ࡼࡗ࡚ Halo ㏻ಙࢆ㉳ືࡉࡏ㸪⣧ෆⅬࡢィ⟬ࡢᚋ࡟ MPI_Waitall ࢆ࿧ࢇ࡛㸪㏻ಙ࡜
ィ⟬ࢆ࣮࢜ࣂ࣮ࣛࢵࣉࡉࡏ࡚࠸ࡿࠋ⣧ෆⅬ㸪ቃ⏺Ⅼࡢィ⟬ࢆูࠎࡢ࣮ࣝࣉ࡛ᐇ᪋ࡋ࡚࠸ࡿࡇ࡜
࠿ࡽ㸪ᅗ 7 ࡟♧ࡍࡼ࠺࡟㸪ෆⅬ࡟ reordering ࢆ᪋ࡋ㸪ࠕ⣧ෆⅬэቃ⏺Ⅼࠖ࡜࡞ࡿࡼ࠺࡟୪ࡧ᭰࠼
࡚࠸ࡿࠋࡇ࠺ࡍࡿࡇ࡜࡟ࡼࡗ࡚㸪ྛ࣮ࣝࣉෆࡣศᒱࡶ࡞ࡃ㸪࢔ࢡࢭࢫࡶ㐃⥆࡜࡞ࡿࡓࡵ㸪ィ⟬ຠ⋡ࡀྥୖࡍࡿࠝ2,8ࠞࠋ
31

32

33

34

35

36

31

32

33

34

35

36

31

32

33

34

35

36

21

22

23

24

25

30

21

22

23

24

25

30

21

22

23

24

25

30

16

17

18

19

20

29

16

17

18

19

20

29

13

14

15

16

20

29

11

12

13

14

15

28

11

12

13

14

15

28

9

10

11

12

19

28

6

7

8

9

10

27

6

7

8

9

10

27

5

6

7

8

18

27

1

2

3

4

5

26

1

2

3

4

5

26

1

2

3

4

17

26

䖃 Internal Nodes
䖃 Boundary Nodes䕔 Overlapped Elem’s䖃 Pure Internal Nodes
䖃 External Nodes

Reorderingᅗ 7 ␯⾜ิ࣋ࢡࢺࣝ✚࡟࠾ࡅࡿ㏻ಙ࡜ィ⟬ࡢ࣮࢜ࣂ࣮ࣛࢵࣉࠝ2,7,8ࠞ

スーパーコンピューティングニュース- 53 -

Vol. 23, No.5

2021

31

32

33

34

35

36

21

22

23

24

25

30

13

14

15

16

20

29

9

10

11

12

19

28

5

1

6

2

7

3

8

4

18

17

27

26

Dynamic Loop
Scheduling

Static
FDOO03,B,VHQG
FDOO03,B,UHFY
GRL 1LQQ
FRPSXWLQJ
HQGR
FDOO03,B:DLWDOO

Pure Internal
Nodes

GRL 1LQQ 1DOOFRPSXWLQJ
HQGGR

Boundary
Nodes

䖃 Internal Nodes
䖃 Boundary Nodes䕔 Overlapped Elem’s

FDOO03,B,VHQG
FDOO03,B,UHFY
FDOO03,B:DLWDOO

MasterGRL  1LQQ
FRPSXWLQJ
HQGR

Dynamic
Pure Internal
Nodes

GRL 1LQQ 1DOOFRPSXWLQJ
HQGGR

StaticBoundary
Nodes

䖃 Pure Internal Nodes
䖃 External Nodesᅗ 8 ␯⾜ิ࣋ࢡࢺࣝ✚࡟࠾ࡅࡿィ⟬࡜㏻ಙࡢ࣮࢜ࣂ࣮ࣛࢵࣉ㸦Static㸸㏻ಙ࡜⣧ෆⅬ㸦Pure Internal
Nodes㸧ࡢィ⟬ࢆ࣮࢜ࣂ࣮ࣛࢵࣉ㸪Dynamic Loop Scheduling㸸OpenMP ࡢ࣐ࢫࢱ࣮ࢫࣞࢵࢻࡀ㏻ಙ
ࢆᢸᙜࡍࡿ㸧
㏻ಙ࡜ィ⟬ࡢ࣮࢜ࣂ࣮ࣛࢵࣉࢆ᭦࡟ຠ⋡໬ࡍࡿࡓࡵ࡟㸪Halo ㏻ಙ࡟ OpenMP ࡢືⓗ࣮ࣝࣉࢫࢣࢪ࣮ࣗࣜࣥࢢ㸦Dynamic Loop Scheduling㸧ᶵ⬟ࢆ㐺⏝ࡍࡿᡭἲࡀ⪃᱌ࡉࢀ࡚࠸ࡿࠝ2㸪8ࠞࠋࡇࡢᡭἲ࡛ࡣ㸪Halo ㏻ಙࢆ OpenMP ࡢ࣐ࢫࢱ࣮ࢫࣞࢵࢻࡀཷࡅᣢࡕ㸪⣧ෆⅬࡢィ⟬࡜࣮࢜ࣂ࣮ࣛࢵ
ࣉࡉࡏࡿࠋᅗ 8 ࡢࠕDynamic Loop Schedulingࠖࡀࡇࢀ࡟┦ᙜࡋ࡚࠸ࡿࠋࡼࡾヲ⣽࡞ᐇ⿦ࢆᅗ 9 ࡟♧ࡍࠋHalo ㏻ಙࢆ OpenMP ࡢ࣐ࢫࢱ࣮ࢫࣞࢵࢻࡀཷࡅᣢࡕ㸪ṧࡾࡢࢫࣞࢵࢻࢆ౑ࡗ࡚⣧ෆⅬࡢィ⟬ࢆ㸪Halo ㏻ಙ࡜࣮࢜ࣂ࣮ࣛࢵࣉࡉࡏ࡚ᐇ᪋ࡋ࡚࠸ࡿࠋࠕ!$omp doschedule (dynamic,200)ࠖ࡜࠶ࡿࠕ200ࠖࡣࠕࢳࣕࣥࢡࢧ࢖ࢬ㸦Chunk Size㸧࡛ࠖ࠶ࡿࠋࡇࢀࡣ㸪࣐ࢫࢱ࣮ࢫࣞࢵࢻࡀᢸᙜࡋ࡚࠸ࡿ㏻ಙࡀ⤊஢ࡋࡓ࠿࡝࠺࠿ࢆࢳ࢙ࢵࢡࡍࡿࡓࡵࡢ㛫㝸࡛㸪ࡶࡋ㏻ಙࡀ
⤊஢ࡋ࡚࠸ࢀࡤ㸪࣐ࢫࢱ࣮ࢫࣞࢵࢻࡶ⣧ෆⅬࡢィ⟬࡟ཧຍࡍࡿࠋࢳࣕࣥࢡࢧ࢖ࢬࡀᑠࡉࡅࢀࡤ㸪ࡑࢀࡔࡅ⣽࠿࠸㛫㝸࡛ṇ☜࡟㏻ಙࡢ≧ἣࢆࢳ࢙ࢵࢡ࡛ࡁࡿࡢ࡛㸪࣐ࢫࢱ࣮ࢫࣞࢵࢻࡀ⣧ෆⅬࡢィ⟬࡟ࡼࡾ᪩ࡃཧຍ࡛ࡁ㸪ࡑࡢศィ⟬᫬㛫ࡀ▷⦰ࡉࢀࡿྍ⬟ᛶࡀ࠶ࡿࠋࡓࡔ㸪ࡑࡢศ㸪㏻ಙ⤊஢ࢳ࢙ࢵࢡࡢࡓࡵࡢᅇᩘࡀቑຍࡋ㸪࣮࢜ࣂ࣮࣊ࢵࢻࡀ኱ࡁࡃ࡞ࡿྍ⬟ᛶࡶ࠶ࡿࠋ
!$omp parallel private (neib,j,k,i,X1,X2,X3,WVAL1,WVAL2,WVAL3)!$omp&
private (istart,inum,ii,ierr)!$omp masterCommunication is done by the master thread (#0)!C
!C– Send & Recv.
(…)
call MPI_WAITALL (2*NEIBPETOT, req1, sta1, ierr)!$omp end master!C!C-- Pure Internal Nodes

The master thread can join computing of internal
nodes after the completion of communication!$omp do schedule (dynamic,200)
Chunk Size= 200
do j= 1, Ninn
(…)
enddo
!C
!C-- Boundary Nodes
Computing for boundary nodes are by all threads!$omp dodo j= Ninn+1, N
(…)
enddo

default: !$omp do schedule (static)!$omp end parallelᅗ 9 ືⓗ࣮ࣝࣉࢫࢣࢪ࣮ࣗࣜࣥࢢ㸦Dynamic Loop Scheduling㸧ࡢ␯⾜ิ࣋ࢡࢺࣝ✚࡬ࡢ㐺⏝౛

スーパーコンピューティングニュース- 54 -

Vol. 23, No.52021

4. ィ⟬஦౛
3.ࡲ࡛࡟♧ࡋࡓᡭἲ࡟ࡘ࠸࡚㸪Wisteria/BDEC-01㸦Odyssey㸧ࡢ᭱኱ 6,144 ࣀ࣮ࢻࡲ࡛ࢆ౑ࡗ࡚ホ౯ࢆᐇ᪋ࡋ㸪ྠࡌࡃᮏࢭࣥࢱ࣮ࡢ Oakforest-PACS㸦OFP㸧ࠝ9ࠞࢆ᭱኱ 2,048 ࣀ࣮ࢻ౑⏝ࡋࡓሙྜ࡜⤖ᯝࢆẚ㍑ࡋࡓࠋOakforest-PACS㸦OFP㸧࡜ Odyssey ࡢࣀ࣮ࢻ༢యㅖඖࢆ⾲ 1 ࡟♧ࡍࠋᮏ✏࡟♧ࡋࡓࢣ࣮ࢫ࡛ࡣ㸪OFP ࡟ࡘ࠸࡚ࡣ Flat ࣮ࣔࢻࢆ㐺⏝ࡋ㸪MCDRAM ࡢࡳࢆ౑⏝ࡋࡓࠋOFP ࡣྛࣀ࣮ࢻ 68 ࢥ࢔ࢆᦚ㍕ࡋ࡚࠸ࡿࡀ㸪64 ࢥ࢔ࢆ౑⏝ࡋࡓࠋ⾲ 1 Oakforest-PACS㸦OFP㸧࡜ Odyssey ࡢࣀ࣮ࢻ༢యㅖඖ
Oakforet-PACSࠝ9ࠞ
Wisteria/BDEC-01 (Odyssey)ࠝ1ࠞ
OFP
OdysseyIntel Xeon Phi  7250  (Knights
CPU ྡ⛠
Fujitsu A64FX㸦2.2GHz㸧
Landing, KNL)
ࢥ࢔ᩘ
68
48
⌮ㄽ₇⟬ᛶ⬟㸦GFLOPS㸧 3,046
3,379
MCDRAM: 16୺グ᠈ᐜ㔞㸦GB㸧
32
DDR4: 96࣓ࣔࣜᛶ⬟㸦GB/sec㸧
MCDRAM: 490+
840+
STREAM Triadࠝ10ࠞ
DDR4: 84.5
ࢥࣥࣃ࢖ࣛIntel Parallel Studio 2019
Fujitsu FCC
ࢩࢫࢸ࣒ྡ
␎⛠

ᛶ⬟ホ౯ࡣୗグࡢ 3 ✀㢮ࡢࣉࣟࢢ࣒ࣛ࡟ࡘ࠸࡚㸪๓ฎ⌮௜ࡁ CG ἲ୍཯᚟࠶ࡓࡾࡢィ⟬᫬㛫࡟ࡼࡗ࡚ᐇ᪋ࡋࡓ㸸ᶅΨϨζψϩ͹ GeoFEM/CubeᶆᶅͶ௪৶ͳ‫ࢋܯ‬͹Ψʖώʖϧρϕʤਦ 8 ͹ʰStaticʱʥΝన༽ͪ͢ྭʤStaticʥᶇᶆͶಊదϩʖϕηίζϣʖϨϱήʤਦ 8 ͹ʰDynamic Loop SchedulingʱʥΝన༽ͪ͢ྭɾοϡϱέγ΢θΝ࠹୉ 1,000 ΉͲรԿͦͪ͠

 1 ࣀ࣮ࢻᙜࡓࡾࡢၥ㢟つᶍࢆୗグࡢ 3 ✀㢮ࡢࢣ࣮ࢫ࡟タᐃࡋ㸪ࣀ࣮ࢻᩘࢆኚ໬ࡉࡏ࡚ Weak
Scaling ࡜ࡋ࡚ᐇ᪋ࡋࡓ㸸y Sʁ96ʹ96ʹ48 ઇ఼ʤ=1,327,104 DOFʥy Mʁ128ʹ128ʹ64 ઇ఼ʤ=3,145,728 DOFʥy Lʁ200ʹ200ʹ100 ઇ఼ʤ=12,000,000 DOFʥ᭱኱ၥ㢟ࢧ࢖ࢬࡣ 7.37281010DOF ࡛࠶ࡿࠋࡲࡓ୪ิࣉࣟࢢ࣑ࣛࣥࢢࣔࢹࣝ࡜ࡋ࡚㸸y OFPʁHB 8ʹ8ʤ֦όʖχɾ8 ηϪρχ͹ MPI ϕϫιηʹ8 ϕϫιηʥɾHB 16ʹ4ʤ16 ηϪρχ
ʹ4 ϕϫιηʥy OdysseyʁHB 6ʹ8ʤ6 ηϪρχʹ8 ϕϫιηʥɾHB 12ʹ4ʤ12 ηϪρχʹ4 ϕϫιηʥࢆᐇ᪋ࡋࡓࠋᅗ 10㸪ᅗ 11 ࡣ㸪OFP㸪Odyssey ࡟ࡘ࠸࡚㸪128 ࣀ࣮ࢻ㸪1,024 ࣀ࣮ࢻ࡟࠾ࡅࡿ࢜ࣜ
ࢪࢼࣝࡢ GeoFEM/Cube ࡟ᑐࡍࡿᛶ⬟ᨵၿ⋡࡛࠶ࡿࠋ100㹼1,000 ࡢᩘᏐࡣືⓗ࣮ࣝࣉࢫࢣࢪ࣮ࣗ
ࣜࣥࢢࡢࢳࣕࣥࢡࢧ࢖ࢬ࡛࠶ࡿࠋ

スーパーコンピューティングニュース- 55 -

Vol. 23, No.52021

2.0E+01(b) 2.5E+01
s-08x8: 0128
m-08x8: 0128
l-08x8: 0128

Improvement over Original Case (%)

Improvement over Original Case (%)(a) 2.5E+01
s-16x4: 0128
m-16x4: 0128
l-16x4: 0128

1.5E+01
1.0E+01
5.0E+00
0.0E+00-5.0E+00

2.0E+01s-06x8: 0128
m-06x8: 0128
l-06x8: 0128

s-12x4: 0128
m-12x4: 0128
l-12x4: 0128

1.5E+01
1.0E+01
5.0E+00
0.0E+00-5.0E+00

2.0E+01s-08x8: 1024
m-08x8: 1024l-08x8: 1024

s-16x4: 1024
m-16x4: 1024
l-16x4: 1024

Improvement over Original Case (%)

Improvement over Original Case (%)ᅗ 10 ㏻ಙ࡜ィ⟬ࡢ࣮࢜ࣂ࣮ࣛࢵࣉࡢຠᯝ㸸࢜ࣜࢪࢼࣝ GeoFEM/Cube ࡟ᑐࡍࡿ๓ฎ⌮௜ࡁ཯᚟ἲࡢ㏿ᗘྥୖ⋡㸪Static㸸ྂ඾ⓗ࣮࢜ࣂ࣮ࣛࢵࣉ㸪ᩘᏐ㸸ືⓗ࣮ࣝࣉࢫࢣࢪ࣮ࣗࣜࣥࢢ࡟࠾ࡅࡿࢳ
ࣕࣥࢡࢧ࢖ࢬ㸪128 ࣀ࣮ࢻ㸦a㸧OFP㸪㸦b㸧Odyssey(a) 2.5E+01(b) 2.5E+01

1.5E+01
1.0E+01
5.0E+00
0.0E+00-5.0E+00

2.0E+01
1.5E+01
1.0E+01
5.0E+00
0.0E+00

s-06x8: 1024
m-06x8: 1024
l-06x8: 1024

s-12x4: 1024
m-12x4: 1024
l-12x4: 1024

-5.0E+00

25.0
s:08x8m:08x8l:08x8

20.0
15.0
10.0
5.0
0.0
1.E+02

1.E+03
Node #

s:16x4m:16x4l:16x4

Improvement over Original Case (%)

Improvement over Original Case (%)ᅗ 11 ㏻ಙ࡜ィ⟬ࡢ࣮࢜ࣂ࣮ࣛࢵࣉࡢຠᯝ㸸࢜ࣜࢪࢼࣝ GeoFEM/Cube ࡟ᑐࡍࡿ๓ฎ⌮௜ࡁ཯᚟ἲࡢ㏿ᗘྥୖ⋡㸪Static㸸ྂ඾ⓗ࣮࢜ࣂ࣮ࣛࢵࣉ㸪ᩘᏐ㸸ືⓗ࣮ࣝࣉࢫࢣࢪ࣮ࣗࣜࣥࢢ࡟࠾ࡅࡿࢳ
ࣕࣥࢡࢧ࢖ࢬ㸪1,024 ࣀ࣮ࢻ㸦a㸧OFP㸪㸦b㸧Odyssey(a)(b)

1.E+04

25.0
20.0
15.0
10.0
s:06x8m:06x8l:06x85.0
0.0
1.E+02s:12x4m:12x4l:12x4

1.E+03
Node #

1.E+04ᅗ 12 ㏻ಙ࡜ィ⟬ࡢ࣮࢜ࣂ࣮ࣛࢵࣉࡢຠᯝ㸸ྛࣀ࣮ࢻᩘ࡟࠾ࡅࡿ࣋ࢫࢺࢣ࣮ࢫࡢ࢜ࣜࢪࢼࣝ
GeoFEM/Cube ࡟ᑐࡍࡿ๓ฎ⌮௜ࡁ཯᚟ἲࡢ㏿ᗘྥୖ⋡㸪
㸦a㸧OFP㸸HB 88 Original ࡟ᑐࡍࡿẚ㸪᭱኱ 2,048 ࣀ࣮ࢻ㸦b㸧Odyssey㸸HB 68 Original ࡟ᑐࡍࡿẚ㸪᭱኱ 6,144 ࣀ࣮ࢻ
OFP ࡣ᭱኱ 10㸣⛬ᗘࡢ㏿ᗘྥୖ⋡࡛࠶ࡿࡀ㸪Odyssey ࡣ 20㸣⛬ᗘ࡛࠶ࡿࠋOFP ࡣ 128 ࣀ࣮ࢻ࡜ 1,024 ࣀ࣮ࢻ࡛ࡣ㸪ࡸࡸ 1,024 ࣀ࣮ࢻࡢ᪉ࡀ㏿ᗘྥୖ⋡ࡀ㧗࠸ࡀ㸪ᅗ 12 ࡟ࡶ♧ࡍࡼ࠺࡟㸪Odyssey
࡟࠾࠸࡚㏿ᗘྥୖ⋡࡜ࣀ࣮ࢻᩘࡢ㛵ಀࡣᚲࡎࡋࡶ୍ᵝ࡛ࡣ࡞࠸ࠋOFP ࡜ẚ㍑ࡍࡿ࡜ Static ࡟ࡼࡿ㏿ᗘྥୖࡣ Odyssey ࡟࠾࠸࡚ࡼࡾ㢧ⴭ࡛࠶ࡿࠋࡲࡓ Odyssey ࡟࠾࠸࡚ࢳࣕࣥࢡࢧ࢖ࢬࡀᑠࡉ࠸ሙ
ྜ㸦700 ௨ୗ㸧࡛ࡣ㸪ࣀ࣮ࢻᙜࡓࡾၥ㢟ࢧ࢖ࢬࡀ኱ࡁ࠸ሙྜࡢ㏿ᗘྥୖ⋡ࡀẚ㍑ⓗప࠸ࠋOFP㸪
Odyssey ࡜ࡶ࡟ࣀ࣮ࢻᙜࡓࡾၥ㢟ࢧ࢖ࢬࡀᑠࡉ࠸ሙྜࡣ㸪HB 88㸪HB 68 ࡢᛶ⬟ࡀⰋࡃ㸪኱ࡁ

スーパーコンピューティングニュース- 56 -

Vol. 23, No.5

2021࠸ሙྜࡣ㸪HB 164㸪HB 124 ࡢᛶ⬟ࡀⰋࡃ࡞ࡗ࡚࠸ࡿࠋ
ᅗ 12 ࡣ OFP㸪Odyssey㸪ྛࣀ࣮ࢻ࡟࠾ࡅࡿ࢜ࣜࢪࢼࣝࡢ HB 88㸦OFP㸧㸪HB 68㸦Odyssey㸧࡟ᑐࡍࡿ࣋ࢫࢺࢣ࣮ࢫ࡟࠾ࡅࡿ㏿ᗘྥୖ⋡࡛࠶ࡿࠋഴྥࡣᅗ 10㸪ᅗ 11 ࡢሙྜ࡜ྠࡌ࡛࠶ࡿࡀ㸪
Odyssey ࡟ࡘ࠸࡚ࡣࣀ࣮ࢻᩘ࡜ᛶ⬟ྥୖࡢ㛵㐃ࡀ᫂࠿࡛࡞ࡃ㸪᳨ウࡀᚲせ࡛࠶ࡿࠋ ᅗ 13 ࡣ㸪1,024 ࣀ࣮ࢻ࡟࠾ࡅࡿ Odyssey㸦HB 68㸧࡜ OFP㸦HB 88㸧ࡢᛶ⬟ẚ࡛࠶ࡿࠋࣀ࣮
ࢻᙜࡓࡾၥ㢟つᶍࡀᑠࡉ࠸ሙྜ㸪Odyssey ࡣ OFP ࡢ 2 ಸ௨ୖ㸪኱ࡁ࠸ሙྜ࡛ࡶ 1.75 ಸ⛬ᗘࡢィ⟬ᛶ⬟࡛࠶ࡿࡇ࡜ࡀࢃ࠿ࡿࠋ␯⾜ิィ⟬ࡣ memory-bound ࡞ࣉࣟࢭࢫ࡛࠶ࡾ㸪࣓ࣔࣜᛶ⬟ࡢᙳ㡪ࡀ኱ࡁ࠸ࠋ⾲ 1 ࡟♧ࡍࡼ࠺࡟㸪Odyssey ࡜ OFP ࡢ࣓ࣔࣜᛶ⬟ẚࡣ STREAM Triadࠝ10࡛ࠞ 1.71㸸
1.00 ࡛࠶ࡿࡇ࡜ࢆ⪃៖ࡍࡿ࡜㸪ࡇࡢᛶ⬟ẚࡣጇᙜ࡞ᩘᏐ࡛࠶ࡿࠋ
ᅗ 13 ࡣ M ࢧ࢖ࢬ㸪128 ࣀ࣮ࢻ㸪HB 124 ࡟࠾࠸࡚㸪๓ฎ⌮௜ࡁඹᙺ໙㓄ἲ㒊ศࡢィ⟬᫬㛫ࡢෆヂࢆ Odyssey ୖ࡛ヲ⣽ࣉࣟࣇ࢓࢖ࣛࠝ1ࠞࢆ౑⏝ࡋ࡚ ᐃࡋࡓࡶࡢ࡛࠶ࡿࠋືⓗ࣮ࣝࣉࢫࢣࢪ
࣮ࣗࣜࣥࢢ࡟ࡼࡗ࡚㸪࣓ࣔࣜࢫ࣮ࣝࣉࢵࢺࡀ 29.4㸣э37.8%࡜⣙ 30㸣ྥୖࡋ࡚࠾ࡾ㸪ィ⟬᫬㛫࠿ࡽ⟬ᐃࡋࡓ㏿ᗘྥୖ⋡ࡶ 30㸣⛬ᗘ࡛࠶ࡿࠋࡇࢀࡣ㸪࣓ࣔࣜᛶ⬟ࡢ 40㸣⛬ᗘࡋ࠿ά⏝࡛ࡁ࡚࠸࡞
࠸㸪࡜࠸࠺ࡇ࡜࡛ࡶ࠶ࡾ㸪඲యⓗ࡞ᛶ⬟ྥୖࡀᚲせ࡛࠶ࡿࠋ(a)(b)
2WKHULQVWUXFWLRQFRPPLW&\FOH$FFRXQWLQJH[HFXWLRQWLPH V
(

2WKHULQVWUXFWLRQFRPPLW&\FOH$FFRXQWLQJH[HFXWLRQWLPH V

LQVWUXFWLRQFRPPLW

(

LQVWUXFWLRQFRPPLWLQVWUXFWLRQFRPPLWLQVWUXFWLRQFRPPLWLQVWUXFWLRQFRPPLW

(

LQVWUXFWLRQFRPPLW

(

LQVWUXFWLRQFRPPLWLQVWUXFWLRQFRPPLW%DUULHUV\QFKURQL]DWLRQZDLW
(

%DUULHUV\QFKURQL]DWLRQZDLW,QVWUXFWLRQIHWFKZDLW

(

6WRUHSRUWEXV\ZDLW,QVWUXFWLRQIHWFKZDLW
6WRUHSRUWEXV\ZDLW

2WKHUZDLW

(

2WKHUZDLW

(

%UDQFKLQVWUXFWLRQZDLW%UDQFKLQVWUXFWLRQZDLW)ORDWLQJSRLQWRSHUDWLRQZDLW
(

)ORDWLQJSRLQWRSHUDWLRQZDLW,QWHJHURSHUDWLRQZDLW)ORDWLQJSRLQWORDG/'FDFKHDFFHVVZDLW(,QWHJHURSHUDWLRQZDLW)ORDWLQJSRLQWORDG/'FDFKHDFFHVVZDLW)ORDWLQJSRLQWORDG/FDFKHDFFHVVZDLW

(

)ORDWLQJSRLQWORDG/FDFKHDFFHVVZDLW

(,QWHJHUORDG/'FDFKHDFFHVVZDLW,QWHJHUORDG/'FDFKHDFFHVVZDLW,QWHJHUORDG/FDFKHDFFHVVZDLW
(,QWHJHUORDG/FDFKHDFFHVVZDLW)ORDWLQJSRLQWORDGPHPRU\DFFHVVZDLW

(,QWHJHUORDGPHPRU\DFFHVVZDLW)ORDWLQJSRLQWORDGPHPRU\DFFHVVZDLW,QWHJHUORDGPHPRU\DFFHVVZDLW

3UHIHWFKSRUWEXV\ZDLWE\VRIWZDUHSUHIHWFK

(

3UHIHWFKSRUWEXV\ZDLWE\KDUGZDUHSUHIHWFK

3UHIHWFKSRUWEXV\ZDLWE\VRIWZDUHSUHIHWFK

(

3UHIHWFKSRUWEXV\ZDLWE\KDUGZDUHSUHIHWFK0HPRU\EXV\UDWHH[HFXWLRQWLPH

3URFHVV,QFOXGHZDLWWLPHIRULQWHJHU/'FDFKHDFFHVV

7KUHDG

7KUHDG

7KUHDG

7KUHDG

7KUHDG

7KUHDG

7KUHDG/EXV\UDWHH[HFXWLRQWLPH

7KUHDG/EXV\UDWHH[HFXWLRQWLPH

7KUHDG,QWHJHUEXV\UDWHH[HFXWLRQWLPH

7KUHDG(

7KUHDG

)ORDWLQJSRLQWEXV\UDWHH[HFXWLRQWLPH

7KUHDG

3URFHVV

7KUHDG

7KUHDG

7KUHDG

7KUHDG

7KUHDG

7KUHDG

7KUHDG

7KUHDG

7KUHDG

7KUHDG

7KUHDG

7KUHDG
(

)ORDWLQJSRLQWEXV\UDWHH[HFXWLRQWLPH,QWHJHUEXV\UDWHH[HFXWLRQWLPH/EXV\UDWHH[HFXWLRQWLPH/EXV\UDWHH[HFXWLRQWLPH
0HPRU\EXV\UDWHH[HFXWLRQWLPH,QFOXGHZDLWWLPHIRULQWHJHU/'FDFKHDFFHVVᅗ 13 Odyssey ヲ⣽ࣉࣟࣇ࢓࢖ࣛ࡟ࡼࡿ 0 ␒ࣉࣟࢭࢫྛࢫࣞࢵࢻ࡟࠾ࡅࡿィ⟬᫬㛫 ᐃ⤖ᯝ㸦๓ฎ⌮௜ࡁඹᙺ໙㓄ἲ㒊ศ㸪ࢧ࢖ࢬ M㸪128 ࣀ࣮ࢻ㸪HB 124㸧㸪512 ࣉࣟࢭࢫࡢ࠺ࡕ㸡220 ࣉࣟࢭࢫ㸪㸦a㸧࢜ࣜࢪࢼࣝ㸦࣓ࣔࣜࢫ࣮ࣝࣉࢵࢺ㸸29.4%㸧㸪
㸦b㸧ືⓗ࣮ࣝࣉࢫࢣࢪ࣮ࣗࣜࣥࢢ㸦ࢳࣕࣥࢡࢧ࢖ࢬ㸻700㸧㸦࣓ࣔࣜࢫ࣮ࣝࣉࢵࢺ㸸37.8 %㸧

5. ࡲ࡜ࡵ
Wisteria/BDEC-01㸦Oddysey㸧࡟࠾࠸࡚㸪୪ิ᭷㝈せ⣲ἲ࠿ࡽᑟ࠿ࢀࡿ኱つᶍ⥺ᙧ᪉⛬ᘧࡢ㸪୪ิ๓ฎ⌮௜ࡁ཯᚟ἲ࡟ࡼࡿồゎࣉࣟࢭࢫࡢ࠺ࡕ㸪␯⾜ิ࣋ࢡࢺࣝ✚₇⟬㒊ศ࡟㸪㏻ಙ࡜ィ⟬ࡢ࣮࢜ࣂ࣮ࣛࢵࣉ㸪ືⓗ࣮ࣝࣉࢫࢣࢪ࣮ࣗࣜࣥࢢࢆ㐺⏝ࡍࡿࡇ࡜࡟ࡼࡗ࡚㸪᭱኱ 20㸣௨ୖࡢ㏿ᗘྥୖࡀᚓࡽࢀࡿࡇ࡜ࡀࢃ࠿ࡗࡓࠋ௒ᅇࡣ␯⾜ิ᱁⣡ἲࡀ CRS ࡛࠶ࡗࡓࡓࡵ㸪ᛶ⬟ࡣపࡃ㸪࣓ࣔࣜࢫ࣮ࣝࣉࢵࢺࡣ 40㸣ᮍ‶࡛࠶ࡗࡓࠋࡼࡾ㧗ᛶ⬟ࡀᮇᚅࡉࢀࡿ㸪ELL㸪SELL-C-σ➼ࡢᡭἲࢆ㐺⏝ࡋ㸪ホ౯ࡍࡿᚲせࡀ࠶ࡿࠋࡲࡓ㸪ࣀ࣮ࢻᩘࡀ 2,000 ࢆ㉸࠼ࡿሙྜ࡟ࡘ࠸࡚ࡣ㸪ࣀ࣮ࢻ㓄⨨ࡶྵࡵࡓ᭦࡞ࡿヲ⣽࡞᳨ウࡀᚲせ࡛࠶ࡿࠋ
スーパーコンピューティングニュース- 57 -

Vol. 23, No.5

2021

ཧ⪃ᩥ⊩
[1] Wisteria/BDEC-01: https://www.cc.u-tokyo.ac.jp/supercomputer/wisteria/system.php[2] Nakajima, K., Hanawa, T., Communication-Computation Overlapping with Dynamic Loop Schedulingfor Preconditioned Parallel Iterative Solvers on Multicore/Manycore Clusters, IEEE Proceedings of
10th International Workshop on Parallel Programming Models & Systems Software for High-End
Computing (P2S2 2017) in conjunction with the 46th International Conference on Parallel Processing(ICPP 2017), Bristol, UK, 2017[3] Nakajima, K. Okuda, H., Parallel Iterative Solvers for Unstructured Grids using an OpenMP/MPI
Hybrid Programming Model for the GeoFEM Platform on SMP Cluster Architectures, Lecture Notes
in Computer Science 2327, 437-448, 2002
[4] Nakajima, K., Parallel Iterative Solvers of GeoFEM with Selective Blocking Preconditioning for
Nonlinear Contact Problems on the Earth Simulator, ACM/IEEE Proceedings of SC 2003, 2003
[5] Saad, Y., Iterative Methods for Sparse Linear Systems (2nd Edition), SIAM, 2003
[6] Ghysels, P. and W. Vanroose, Hiding global synchronization latency in the preconditioned Conjugate
Gradient algorithm, Parallel Computing 40-7, 224-238, 2014[7] Shimokawabe, T., Aoki, T., Takaki, T., Yamanaka, A., Nukada, A., Endo, T., Maruyama, N., Matsuoka,
S., Peta-scale Phase-Field Simulation for Dendritic Solidification on the TSUBAME 2.0
Supercomputer, ACM/IEEE Proceedings of SC’11, 2011[8] ґ಺୕໷ɾ௉ർ ༠ҲɾҬ‫ރ‬ଞ଻޼ɾτϧϓϫρϕη‫څ‬ϟωʖαΠΠʖΫτέοϡͶ͕͜Ζητϱεϩ‫ࢋܯ‬͹࠹నԿघ๑͹֋൅ɾ৚ๅॴཀྵָճ‫ڂݜ‬ๅࠄʤ2015-HPC-152-10ʥɾೖຌ৚ๅॴཀྵָճ୊ 152 յ HPC ‫ڂݜ‬ճɾ2015[9] Oakforest-PACS: https://www.cc.u-tokyo.ac.jp/supercomputer/ofp/system.php[10] STREAM: https://www.cs.virginia.edu/stream/

スーパーコンピューティングニュース- 58 -

Vol. 23, No.5

2021