5. 各会合での富士通からの情報提供資料
富士通からの情報提供として、以下を行った。
・Open Petascale Libraries (OPL) 活動状況報告
OPL 活動は，アプリケーション起点によるライブラリ開発を目指している．すなわち，各アプリケーション
分野で要求される要素技術を分析し，需要の高い要素技術を高速ライブラリとして提供することを活動目的とする．要素技術の分析結果に基づき，以下の 4 分野について高速ライブラリをスーパーコンピュータ・PC クラスタ向けに開発・移植・高速化することを決定し，活動を続けている．
・密行列計算- PLASMA/DPLASMA
・大規模偏微分方程式によるシミュレーション- PETSc- spBLAS- PRAgMaTIc
・高速フーリエ変換 (FFT)- FFTE- 2DECOMP&FFT
・乱数生成- MRG8 他
・OPL 活動で提供しているライブラリの実アプリケーションへの適用事例紹介
OPL 活動の中間成果として，開発・移植・高速化したライブラリを実際のアプリケーションに適用し，適用性・性能を評価した．1 つは平面波基底による密度汎関数法アプリケーション Quantum ESPRESSO への FFT ライ
ブラリ適用である．FFT を利用する典型的なアプリケーションであり，FFT 計算部の計算負荷が高い．もう 1
つはフラグメント分子軌道法アプリケーション OpenFMO への密行列計算ライブラリ適用である．密行列計算を利用する典型的なアプリケーションであり，特に固有値計算部の計算負荷が高い．両者とも，より高速な計算が望まれている分野であり，高速ライブラリの適用は関心が高い．適用評価は，FFT 計算部を FFTE に，固有値計算部を PLASMA に置き換えることで実施した．それぞれ，ライブラリ固有の課題が見つかり，ライブラリの在り方そのものについても再認識することとなった．
・富士通の数学ライブラリ取組み紹介
富士通におけるライブラリ開発の立場から，現在の製品である SSL II について，その経緯，開発方針，スーパーコンピュータにおける性能，方針などについて紹介した．
また，現在の数値計算ライブラリを取り巻く環境として，OSS のライブラリの現状について調査した結果を紹介した．
次ページ以降に，以下の資料を掲載する．
No

実施日

1

2011/08/23

2
3
4

2011/10/28
2011/10/28
2012/03/01

5
6

2012/07/23
2013/02/22

タイトル
Open Petascale Libraries ： Application
Requirements

情報提供者 (所属は当時)

R. Nobes(欧州富士通研究所)
J. Southern (同上)
R. Saksena (同上)
OPL 状況報告(2011 年 10 月)
金澤 宏幸 (富士通)
FFTE ライブラリ適用事例(Quantum ESPRESSO)
堀田 普介 (富士通)
大規模数値計算における数学ライブラリの取り組み 臼井 徹三 (富士通)についてOpen Petascale Libraries ： Current Status
堀田 普介 (富士通)Application of PLASMA to OpenFMO code
堀田 普介 (富士通)5.1 Open Petascale Libraries:Application Requirements     
   

   !""#$%&'#

 
"( %)  ( &
*

+(         ) 

,( -  " !""#$%&'#

   

 %) 
 %)                      -) ).  ) )   /)/  0  )
  0        & )        0      /    

0   1        

2 0  #!

%  "" !""#$%&'#

㻼㼞㼛㼢㼕㼐㼕㼚㼓㻌㼍㻌㼟㼛㼒㼠㼣㼍㼞㼑㻌㼜㼘㼍㼠㼒㼛㼞㼙㻌㼠㼛㻌㼍㼏㼏㼑㼘㼑㼞㼍㼠㼑㻌㼍㼜㼜㼘㼕㼏㼍㼠㼕㼛㼚㼟㻌㼞㼡㼚㼚㼕㼚㼓㻌㼛㼚㻌㼙㼍㼟㼟㼕㼢㼑㼘㼥㻌㼜㼍㼞㼍㼘㼘㼑㼘㻌㼙㼡㼘㼠㼕㼏㼛㼞㼑㻌㼟㼡㼜㼑㼞㼏㼛㼙㼜㼡㼠㼑㼞㼟" !""#$%&'# 2

 !""#$%&'#"+

 2  #
$#
!"#
#%
& ' "%#%(!



!
"



",

 !""#$%&'#  )/  ($ 5- 
(8 7(95 3 (&
-25)-2   3)  

'%'

)  /415$   2

7  2
 '    6  )"#  $ %& ' !""#$%&'#":

  ;+-      '26- '2< )) )1 $ 5-  >

 ;,

0'-08    =)' 

 3)&' <1%   )4   > '8 ' < )) / +   8 '=0 > 2 &% <)  )    =0 >

 ;:

&   &<-& 5 =& 5 >-2?&<4  7=47>

 ;@ )47 ) 9 2 =4%%"@ !""#$%&'#

 

0-/



 

-/-/  )/   <3 =-  =7=&>A    15)/   
 B;   -/ ;5&  2) 

<1%   )=#/ &5= (> '  

5 

4   C0- D  E"F !""#$%&'#

 



 9   /)    1   ) /  
 
 D  1    )     
      '      )) 5 G 1  )/  3    1   H  6)/ 

"I

 !""#$%&'#'  %               / )1  3  0  E    3  15  %'E"" 9        <$  > '  9     /))30  <$  >   %' )  )15  <#'> '&= ).  -%' )
  5 <>!

 !""#$%&'#

  -  <  > - / <.  > &   <&> 2   2)<22>     '  <   = (> ) / 
 '   ))     - '    8 5=%'15 =2 =3 1 =  !"!(
(3  =3  ;5 =%'E""( !""#$%&'#"

2   -)  <J> 2%=7    =74  2&' 8 '=' &;
 7  9 D %=D&9

 #29=' #=
2#2'

 &   &' =3=&  =
'#4-% ' '% =-%'=3-: ='2%'    2 8'&

 8 5=%'15 =2 =3 1 =  !"! !""#$%&'#

2   -)  <J> &)=7    =7 4='3 ""!,  2&' D %=D&9

 7  

 *%=%)#

 &  

 #29=' #=
2#2'  2
 '% =-%'=-+".=
4- )!=3=&  =+%

 *$$!=7' = ='2%< )    >' 9  1   %' 2= !"" !""#$%&'#+

2   -)  <J> &)=7    =7 4='3 ""!,=&  2&' D %=D&9

 7  

 *%=%)#

 &  

 #29=' #=
2#2'  2
 '% =-%'=-+".=
4- )!=3=&  =+%

 *$$!=7' = ='2%< )    >' (3  =3  ;5 =%'E"", !""#$%&'#

2   -)  <J> &)=7    =7 4='3 ""!,=&  2&' D %=D&9

 7  

 *%=%)#

 &  

 #29=' #=
2#2'  2
 '% =-%'=3-:=
4-='+ )!=-/ =&  =
'#4-% '=;'2=' '

 *$$!=%= ='2%< )    >' '& (=  !"": !""#$%&'#' 
 &   / 1   ))
    /1)  )
 )
 8 '6'   9 &'
 &;

 '=      
 =  / 1/   1        @ !""#$%&'#

-





 ;     )         )
 ) -D     )/ J;   -/ ;5&  2) 

<1%   )=#/ &5= (> '  

5 

4   C0- D  E
K !""#$%&'#

F

 !""#$%&'#

5.2 OPL状況報告 (2011年10月)

OPL⁁ᴫႎ๔ (2011ᐕ10᦬) 䊤䉟䊑䊤䊥ೋ 䈱౏㐿 SC11 BoF䉶䉾䉲䊢䊮0䊤䉟䊑䊤䊥ೋ

Copyright 2011 FUJITSU LIMITED䈱౏㐿䋨Ḱ஻ਛ䋩

 2DECOMP&FFT
a library for 2D pencil decomposition and parallel fast Fourier transform FFTEa package to compute discrete Fourier transforms in 1, 2 and 3 dimensions

 spBLASa threaded level 3 (matrix-matrix) sparse BLAS package

 PLASMAa dense linear algebra package optimized for multicore processors

 PETScdata structures and routines for the solution of partial differential equations

http://www.openpetascale.org/
1

Copyright 2011 FUJITSU LIMITEDSC11 BoF䉶䉾䉲䊢䊮 BoF (Birds of a Feather) 䉶䉾䉲䊢䊮 䉶䉾䉲䊢䊮ฬ䋺Open Petascale Libraries ᣣᤨ䋺11᦬16ᣣ(᳓) 17:30~19:00
 ႐ᚲ䋺TCC LL22

Copyright 2011 FUJITSU LIMITEDSC11 BoF䈱䉝䉳䉢䊮䉻䋨઒䋩
1. Gerard Gorman - Introduction, architectural trends,
programming models (MPI, MPI+threads, PGAS),
arguments for MPI+threads
2. Motoi Okuda - Architectures designed for MPI+threads(K computer and others)
3. Jack Dongarra - Dense linear algebra (PLASMA andDPLASMA)
4. Gerard Gorman - Sparse solvers and adaptive meshing(PETSc and PRAGMATIC)
5. Daisuke Takahashi - Fast Fourier transforms (FFTE)
6. Kenichi Miura - NRGs and Monte Carlo methods
7. Ross Nobes - Structure of OPL Project
8. Discussion
3

Copyright 2010 FUJITSU LIMITED5.3 FFTEライブラリ適用事例 (Quantum ESPRESSO)

FFTE䊤䉟䊑䊤䊥ㆡ↪੐଀䋨Quantum ESPRESSO䋩
ን჻ㅢᩣᑼળ␠⸘▚⑼ቇ䉸䊥䊠䊷䉲䊢䊮⛔᜝ㇱၳ↰ ᥉੺

Copyright 2011 FUJITSU LIMITED

AgendaQE outline
QE cost profile
Application of FFTE to QE
Performance comparison on PFX1
Conclusions & Comments

1

Copyright 2011 FUJITSU LIMITEDWhat is Quantum ESPRESSO ?
 An integrated suite of computer codes for electronic-structure
calculations and materials modeling at the nanoscale Based on density-functional theory, plane waves, andpseudopotentials
 Standing for opEn Source Package for Research in Electronic
Structure, Simulation, and Optimization
 Freely available to researchers around the world under theterms of the GPL (http://www.quantum-espresso.org/) There are two core solvers (PWscf and CP), and several
packages and utilities
 PWscf was ported onto K (PFX1)

2

Copyright 2011 FUJITSU LIMITED

New version of PWscf (4.3.2)
 Program codes (PWscf part only)kind# of files# of lines

Fortran

475

296,176C

12

29,473Header

8

2,261

Total

495

327,910

 Parallelisation
 Basically parallelized by MPI
 More OpenMP codes than the previous version

 FFT
 FFTW V1.2 is implemented and tuned by OpenMP Could be linked with the external FFTW V3 instead of the internal FFTW Modification
 Slightly tuned by directive (!ocl) effective for the hybrid run
3

Copyright 2011 FUJITSU LIMITEDMachine environment
The machine used was as follows.
 PFX1 + Fujitsu compilers
Fujitsu’s proprietary machine, each node has one
SPARC64VIIIfx CPU (SPARC64VIIIfx is 8-core chip whose
clock signal frequency is 2.0GHz)
PWscf runs in parallel with up to 84 nodes (672 cores)

4

Copyright 2011 FUJITSU LIMITED

Input data Name : graphene Calculation : SCF
 # of atoms : 128
 # of electrons : 512
 Pseudopotential : C.pbe-rrkjus.UPF
 # of K points : 1
 FFT dimensions : (240,240,192), (150,150,120)

5Copyright 2011 FUJITSU LIMITEDCost profile Below are the results of profiler.
(All processes and threads are accumulated)
 FFT costs >16.5% (64 processes x 8 threads)*************************************************************************************
Application - procedures*************************************************************************************

Cost

%

Barrier

%

Start

End

---------------------------------------------------------------------3863547

100.0000

64550016.7074----

Application---------------------------------------------------------------------664215

17.191800.0000----mca_btl_tofu_component_progress

39436810.207436676092.9994315347

fft_scalar.cft_2xy._OMP_3_

3749859.705700.0000116

166

dfunct.newq._OMP_1_

3687639.544700.0000----__jwe_phb_sync

257502

6.664900.0000----ptlib_read_mrq

244872

6.338024131698.5478

353

380

fft_scalar.cft_2xy._OMP_4_

2196005.683900.0000----

opal_progress

1455653.767700.0000----dl_gmwn1_base_16x4_

139775

3.617800.0000112

117

addusdens_g._PRL_3_

1110022.873100.0000----dl_gmwn1_general_16x4_

6

Copyright 2011 FUJITSU LIMITEDFFTE performance (FYI)

7

Copyright 2011 FUJITSU LIMITEDComparison of FFTs (FYI)
FFTE

FFTW

Language

FortranC with Fortran Interface

Parallelization

OpenMP + MPI

OpenMP + MPI

Input/Output

complex to complex

complex to complex
real to complex
complex to realRadix

2,3,5 (only 2p3q5r areallowed) (*)

arbitrary

Dimension

1D, 2D, 3D

1D to nD

Precision

doublesingle and double# of plans

1nNormalization

yes (1/N for inverse)

noScaling factornono(*) PWscf looks satisfying this condition fortunately.

8

Copyright 2011 FUJITSU LIMITED

Application of FFTE to QE (1) # of plans
FFTE can deal with only one plan (*)
PWscf needs at least 9 plans (3 plans for x,y,z axes)
Made 9 copies(e.g. zfft1d_x1, fft235_y1 etc).

(*) transform matrix for FFT

9Copyright 2011 FUJITSU LIMITEDApplication of FFTE to QE (2) Normalization factor
FFTE has normalization factors 1 for forward transform and1/N for inverse transform, where N is # of points
FFTW doesn’t have normalization factor(i.e.,)
PWscf claims 1/N for forward and 1 for inverse
FFTE needed modifying.
Coefficient (1/N in this style)and sign (- for forward and +for inverse in this style)depend on style of definition

cf. In some style,(including FFTE)

10

Copyright 2011 FUJITSU LIMITED

Application of FFTE to QE (3) OpenMP overhead
As FFTE is highly customized for FFT with large # of points,
even loops composed of 1 line are parallelized by OpenMP
Loop length (150 – 240 in case of graphene) is too small
Cancelled OpenMP directives
for loops composed of 1 line.

11

Copyright 2011 FUJITSU LIMITEDProposal to FFTE developer
Through the application, I thought some features need
implementing, which make porting programs easier.
 Support of multiple plans
 Support of scaling factor instead of normalization
 Support of better performance even with small # of points

Reply from Dr. Takahashi :The requests you suggested seems to contain important ideas for FFTE.
I will consider supporting your requests in a future release of FFTE.

12

Copyright 2011 FUJITSU LIMITED

Results of FFTE
Using tuned version with graphene input data, comparison up to 84 nodes(672 cores on PFX1).
Comparison of FFTW and FFTE (log-log plot)

Elapsed time (sec)

㪈㪇㪇㪇㪇

㪝㪝㪫㪮
㪝㪝㪫㪜

㪈㪇㪇㪇

㪈㪇㪇

㪈

㪈㪇

㪈㪇㪇

# of nodes
13

Copyright 2011 FUJITSU LIMITEDConclusions & Comments
 Replacement of FFTW by FFTE was done, but with small datasuch as graphene, the less effect was seen than expected
while FFTE is customized for PFX1.
 FFTE is easy-to-use and simple FFT library, but lessapplicable for general applications.

14

Copyright 2011 FUJITSU LIMITED

15

Copyright 2010 FUJITSU LIMITED5.4 大規模数値計算における数学ライブラリの取り組みについてᄢⷙᮨᢙ୯⸘▚䈮䈍䈔䉎ᢙቇ䊤䉟䊑䊤䊥䈱ข䉍⚵䉂䈮䈧䈇䈩ን჻ㅢᩣᑼળ␠ 䊚䊄䊦䉡䉢䉝੐ᬺᧄㇱ䉝䊒䊥䌍䌗੐ᬺㇱ ╙྾㐿⊒ㇱ ᢙቇ䊤䉟䊑䊤䊥  ⎇䊕䉺䉴䉬䊷䊦

  

  

ႎ๔ౝኈ ᒰ␠ᢙቇ䊤䉟䊑䊤䊥⚫੺ ን჻ㅢᢙቇ䊤䉟䊑䊤䊥૕♽  ♽䊤䉟䊑䊤䊥㐿⊒䈱⚻✲䈮䈧䈇䈩 ⃻࿷䈱ข䉍⚵䉂 ᣇ㊎䈮䈧䈇䈩 ᄢⷙᮨᢙ୯⸘▚ะ䈔 䊤䉟䊑䊤䊥᭎ⷰ 䉁䈫䉄⚫੺!"䋺ን჻ㅢᢙቇ䊤䉟䊑䊤䊥૕♽ ㅙᰴ 䊤䉟䊑䊤䊥䋨䉴䊧䉾䊄䉶䊷䊐䋩  #⚂ ⒳䊦䊷䉼䊮䇯 $%䈪⸥ㅀ䇯න䊶୚♖ᐲ䇯  # 䈻䈱䉟䊮䉺䊐䉢䊷䉴䇯ਥ䈮୚♖ᐲ䇯
 &'(')'* #☨࿖䈪㐿⊒䈘䉏+, -䈪౏㐿䈘䉏䈩䈇䉎䊂䊐䉜䉪䊃䉴䉺䊮䉻䊷䊄䈭✢ᒻ⸘▚䊤䉟䊑䊤䊥䇯&'⚂. ⒳䇮')'*⚂/ ⒳䊦䊷䉼䊮䇯ฦ♖ᐲ䈅䉒䈞䈩⚂䊦䊷䉼䊮䇯 䉴䊧䉾䊄ਗ೉ 䊤䉟䊑䊤䊥  䉴䊧䉾䊄ਗ೉ᯏ⢻ #㊀ⷐᯏ⢻⚂. 䊦䊷䉼䊮䇯ᷙ࿷૶↪䈪䈐䉎䉋䈉䈮ㅙᰴ 䈫೎䉟䊮䉺䊐䉢䊷䉴䇯୚♖ᐲ䇯  䉴䊧䉾䊄ਗ೉ᯏ⢻ # 䉴䊧䉾䊄ਗ೉ᯏ⢻䈻䈱䉟䊮䉺䊐䉢䊷䉴䇯 &'(')'* #ㅙᰴะ䈔䈫ห৻䉟䊮䉺䊐䉢䊷䉴䇯')'*ጀ䈪䉴䊧䉾䊄ਗ೉ൻ䈚䈢䊦䊷䉼䊮䉅䈅䉍䇯 )ਗ೉ 䊤䉟䊑䊤䊥  ()
 0$')'*#䋳ᰴరᯏ⢻䊦䊷䉼䊮䇯#)&'䈫&'䊧䉟䊟䊷䉕฽䉃᭴ᚑ䇯⚂䉼䊮䇮ฦ♖ᐲ䈅䉒䈞䈩⚂1 䊦䊷䉼䊮䇯⒳䊦䊷

  

⚫੺! "䋺 ♽䊤䉟䊑䊤䊥䈮䈧䈇䈩
ᯏ⢻৻ⷩ
ᯏ⢻৻ⷩ
✢ᒻ⸘▚
࿕᦭୯໧㗴
㕖✢ᒻᣇ⒟ᑼ
ᭂ୯໧㗴
⵬㑆䊶ㄭૃ
ᄌ឵
ᢙ୯ᓸಽ䊶Ⓧಽ
ᓸಽᣇ⒟ᑼ
․ᱶ㑐ᢙ
ᡆૃੂᢙ

䋺ㅪ┙৻ᰴᣇ⒟ᑼ䇮ㅒⴕ೉䇮ᦨዊੑਸ਼⸃䇮․⇣୯ಽ⸃䋺࿕᦭୯䊶࿕᦭䊔䉪䊃䊦䋺ઍᢙᣇ⒟ᑼ䇮⿥⿧ᣇ⒟ᑼ䇮ㅪ┙㕖✢ᒻᣇ⒟ᑼ
䋺㑐ᢙ䈱ᭂዊൻ䇮✢ᒻ⸘↹໧㗴䇮㕖✢ᒻ⸘↹໧㗴䋺⵬㑆ᑼ䋯⵬㑆୯䇮ㄭૃᑼ䇮ᐔṖൻᑼ䋯ᐔṖ୯䇮⚖ᢙዷ㐿䋺䊐䊷䊥䉣ᄌ឵䇮䊤䊒䊤䉴ᄌ឵䇮䉡䉢䊷䊑䊧䉾䊃ᄌ឵䋺㔌ᢔὐ䋯㑐ᢙ౉ജ䇮᦭㒢඙㑆䋯ή㒢㗔ၞ䇮৻ᰴర䋯ੑᰴర䋺ㅪ┙৻㓏Ᏹᓸಽᣇ⒟ᑼ䇮ㅪ┙৻㓏䉴䊁䉞䊐Ᏹᓸಽᣇ⒟ᑼ
䋺䊔䉾䉶䊦㑐ᢙ䇮ᬦ౞Ⓧಽ䇮ᜰᢙⓍಽ䇮ᱜᒏ䊶૛ᒏⓍಽ䋺ੂᢙ↢ᚑ䋨৻᭽䋯ᱜⷙ䋯ᜰᢙ䋯䊘䉝䉸䊮䋯ੑ㗄䋩䇮ੂᢙᬌቯ

㐿⊒
㐿⊒䬽
㐿⊒䬽ᱧผ

㪪㪤㪧䊙䉲䊮
㪪㪤㪧䊙䉲䊮䈻䈱
䊙䉲䊮䈻䈱䊒䊤䉾䊃䈻䈱䊒䊤䉾䊃
䊐䉤䊷䊛⒖ⴕ
⒖ⴕ㩿㪈㪐㪐㪏
㩿㪈㪐㪐㪏ᐕ㗃
ᐕ㗃㪀㪀
䊐䉤䊷䊛
⒖ⴕ
㩿㪈㪐㪐㪏
ᐕ㗃

㪪㪪㪣㩷㪠㪠㩷
㪪㪪㪣㩷㪠㪠㩷ၮᧄᯏ⢻㪪㪪㪣㩷㪠㪠㩷ၮᧄᯏ⢻䋺
ၮᧄᯏ⢻䋺
࿖ౝ䈱
䈱ᄢቇ䊶
࿖ౝ
ᄢቇ䊶⎇ⓥᯏ㑐䈱⎇ⓥᯏ㑐䈱ኾ㐷ኅ䈮ኾ㐷ኅ䈮දജ䈇䈢䈣䈇䈩දജ䈇䈢䈣䈇䈩౒ห䈇䈢䈣䈇䈩౒ห
㐿⊒䇯㐿⊒䇯㩿㪈㪐㪎㪇ᐕઍ㩿㪈㪐㪎㪇ᐕઍ䌾ᐕઍ䌾㪈㪐㪏㪇ᐕઍ㪈㪐㪏㪇ᐕઍ㪀ᐕઍ㪀㪪㪪㪣㩷㪠㪠㩷
㪪㪪㪣㩷㪠㪠㩷ㅙᰴᯏ⢻㪪㪪㪣㩷㪠㪠㩷ㅙᰴᯏ⢻㪑ㅙᰴᯏ⢻㪑䉴䊧䉾䊄䉶䊷䊐ൻ䉴䊧䉾䊄䉶䊷䊐ൻ䇮䉴䉦䊤ᯏะ䉴䉦䊤ᯏะ䈔
ᯏะ䈔䉼䊠䊷䊆䊮䉫㪪㪪㪣㩷㪠㪠㪆㪭㪧㪪㪪㪣㩷㪠㪠㪆㪭㪧᜛ᒛᯏ⢻㪪㪪㪣㩷㪠㪠㪆㪭㪧᜛ᒛᯏ⢻㪃㩷㪪㪪㪣㩷㪠㪠㪆㪭㪧㪧᜛ᒛᯏ⢻㪃㩷㪪㪪㪣㩷㪠㪠㪆㪭㪧㪧䋺㪃㩷㪪㪪㪣㩷㪠㪠㪆㪭㪧㪧䋺㪘㪥㪬䋨㪘㪥㪬䋨䉥䊷䉴䊃䊤䊥䉝࿖┙ᄢቇ䉥䊷䉴䊃䊤䊥䉝࿖┙ᄢቇ䋩࿖┙ᄢቇ䋩䈫౒ห㐿⊒ 㩿㪈㪐㪐㪇ᐕઍ㩿㪈㪐㪐㪇ᐕઍ㪀ᐕઍ㪀㪪㪪㪣㩷㪠㪠
㪪㪪㪣㩷㪠㪠䉴䊧䉾䊄㪪㪪㪣㩷㪠㪠䉴䊧䉾䊄ਗ೉ᯏ⢻䉴䊧䉾䊄ਗ೉ᯏ⢻䋺ਗ೉ᯏ⢻䋺㊀ⷐᯏ⢻⚂㪏㪇
㊀ⷐᯏ⢻⚂㪏㪇䊦䊷䉼䊮㪏㪇䊦䊷䉼䊮

㪚㪚㪄㪪㪪㪣㪠㪠㪆㪭㪧䋺㪪㪪㪣㪠㪠㪆㪭㪧䋺㪪㪪㪣㩷㪠㪠㪆㪭㪧䈻䈱㪪㪪㪣㩷㪠㪠㪆㪭㪧䈻䈱㪚䈻䈱㪚䉟䊮䉺䊐䉢䊷䉴㩿㪈㪐㪐㪎䉟䊮䉺䊐䉢䊷䉴㩿㪈㪐㪐㪎䌾㩿㪈㪐㪐㪎䌾㪃㩷㪝㪜㪚㪠㪫㪀㪪㪪㪣㩷㪠㪠㪆㪯㪧㪝㪃㩷㪪㪪㪣㩷㪠㪠㪆㪤㪧㪠㪪㪪㪣㩷㪠㪠㪆㪯㪧㪝㪃㩷㪪㪪㪣㩷㪠㪠㪆㪤㪧㪠䋺㪪㪪㪣㩷㪠㪠㪆㪯㪧㪝㪃㩷㪪㪪㪣㩷㪠㪠㪆㪤㪧㪠䋺ಽᢔ䊜䊝䊥ಽᢔ䊜䊝䊥ਗ೉ะ
䊜䊝䊥ਗ೉ะ䈔ਗ೉ะ䈔㪙㪣㪘㪪㪆㪭㪧㪃㩷㪣㪘㪧㪘㪚㪢㪆㪭㪧㪃㩷㪪㪺㪸㪣㪘㪧㪘㪚㪢䊔䉪䊃䊦䊙䉲䊮ะ䈔㪙㪣㪘㪪㪃㩷㪣㪘㪧㪘㪚㪢㪃㩷㪪㪺㪸㪣㪘㪧㪘㪚㪢䉴䉦䊤䊙䉲䊮ะ䈔䉼䊠䊷䊆䊮䉫 䇮䉴䊧䉾䊄ਗ೉ൻ  

⚫੺!"䋺ಽᢔ䊜䊝䊥ဳᢙቇ䊤䉟䊑䊤䊥㐿⊒⚻✲  (2)) 䋨33 䌾333ᐕ䋩 2))  44'+䈱䉰䊑䊦䊷䉼䊮䈫䈚䈩⸥ㅀ䇯 333ᐕᤨὐ䈪5 䊦䊷䉼䊮䇯 '+!'6 $-$% +$ %$- %7,6 "䈫౒ห㐿⊒䇯 ᯏ⢻▸࿐䈲ኒⴕ೉໧㗴䊶䉴䊌䊷䉴ⴕ೉෻ᓳ⸃ᴺ䊶䊶ੂᢙ↢ᚑ䈭䈬䇯 ၮᧄ⊛䈮䈲㈩೉䈱䋱䈧䈱ᰴర䉕╬ಽഀ䈜䉎䊂䊷䉺ಽᢔᣇᑼ䈏ᄙ䈇䇯  (8) 䋨ᐕ㗃䋩  (2))䉕䊔䊷䉴䈮䉟䊮䉺䊷䊐䉢䊷䉴ㇱಽ䉕㐿⊒䇯 /䊦䊷䉼䊮䇯  (9) 䋨ᐕ䌾䋩  (2))䈫ห䈛䉟䊮䉺䊷䊐䉢䊷䉴䇮หᯏ⢻▸࿐䈪ឭଏ䇯 ਗⴕ䈚䈩䇮㊀ⷐᛛⴚ䈲 䉴䊧䉾䊄ਗ೉ᯏ⢻䈮⒖ⴕ䇯  () 䋨5ᐕ䌾䋩 ⃻ᤨὐ䈪䈲ታ䊶ⶄ⚛䈱䋳ᰴర!ゲಽᢔ"䈱 䊦䊷䉼䊮䈱䉂䇯 ᰴ 䈪2-:, 0 !ゲ౒䈮ಽഀ䈚䈩䊂䊷䉺ಽᢔ"䉕ㅊട੍ቯ䇯 䊉䊷䊄ౝ䉴䊧䉾䊄ਗ೉䈱8;ਗ೉䇯 0$')'* ੹ᓟ䉅⛮⛯⊛䈮䉰䊘䊷䊃䈚䈩䈇䈒ᣇ㊎䇯/  

⃻࿷䈱ข䉍⚵䉂  ᭎ⷐ    䉴䊧䉾䊄ਗ೉ᯏ⢻䊤䉟䊑䊤䊥䈱ᯏ⢻䉣䊮䊊䊮䉴 䉴䊌䊷䉴ⴕ೉ㅪ┙৻ᰴᣇ⒟ᑼ䈱⋥ធ⸃ᴺ!ᱜ୯ኻ⒓ⴕ೉< ,= ->%" 䉴䊌䊷䉴ⴕ೉ㅪ┙৻ᰴᣇ⒟ᑼ䈱෻ᓳ⸃ᴺ!4ᴺ< 4ᴺ<ㄭૃㅒⴕ೉೨ಣℂ" 䉴䊌䊷䉴ⴕ೉䈱࿕᦭୯໧㗴⸃ᴺ!$0$7;6%ᴺ" Ᏹᓸಽᣇ⒟ᑼ⸃ᴺ!4''5"  ()䊤䉟䊑䊤䊥䈱ᯏ⢻䉣䊮䊊䊮䉴 ಽᢔਗ೉䋳ᰴర䊦䊷䉼䊮㐿⊒!2-:, 0 ;,0:6 %" 䊙䉲䊮䉝䊷䉨Ფ䈱䉼䊠䊷䊆䊮䉫 ๮઎೉䉴䉬䉳䊠䊷䊥䊮䉫䈱ᡷༀ䊶ᶋേዊᢙὐ䊧䉳䉴䉺䈱᦭ല೑↪
 ๮઎䈱೑↪
? 䉝䉶䊮䊑䊤䉮䊷䊂䉞䊮䉫!ㅙᰴ&'㊀ⷐ䊦䊷䉼䊮"? ๮઎↢ᚑ⚵䉂ㄟ䉂㑐ᢙ೑↪!♽"? 䉮䊮䊌䉟䊤䈮䉋䉎ൻଦㅴ ,=, 0 䈱䉼䊠䊷䊆䊮䉫 䉨䊞䉾䉲䊠᦭ല೑↪
 䉴䊧䉾䊄ਗ೉ൻ䋨 ,%)⸥ㅀ䋩
5  

⃻࿷䈱ข䉍⚵䉂!"  䉴䊌䊷䉴ⴕ೉䈱࿕᦭୯໧㗴 
$0$7;6%ᴺ䈮䉋䉎䉴䊌䊷䉴ⴕ೉࿕᦭୯໧㗴⸃ᴺ䊦䊷䉼䊮䉕ᣂⷙ㐿⊒!䉣䊦䊚䊷䊃䈍䉋䈶㕖ኻ⒓"䉝䊦䉯䊥䉵䊛䈏ⶄ㔀䈪䇮 䈪䈲⹜㛎⊛ታⵝ䈮⇐䉁䈦䈩䈇䉎䉅䈱䈏ᄙ䈇䈖䈫䈎䉌㐿⊒ᔅⷐᕈ᦭䈫್ᢿ䇯ౝㇱ෻ᓳ䈫䈚䈩ㅪ┙৻ᰴᣇ⒟ᑼ෻ᓳ⸃ᴺ䈱ᢙ䉴䊁䉾䊒䉕૶↪䈜䉎!4<& $!"䈮ኻᔕ"䇯:,66,; 4D  $,ᩰ⚊ᣇᑼ䈮ኻᔕ䇯࿕᦭୯ㆬᛯ䈲2$-ᜰቯ<<<4<4<<䈮ኻᔕ䇯 ᷹ቯ䊙䉲䊮䋺 ) A $%;&;, 1 B )㬍.0, 䈪@ਗ೉ ⸳ቯ䊌䊤䊜䉺᧦ઙ䋺 ⛘ኻ୯ᦨᄢ࿕᦭୯䉕 ୘䉁䈪 ෻ᓳ࿁ᢙ਄㒢 ࿁ ౝㇱ෻ᓳ࿁ᢙ਄㒢 ࿁<4૶↪
 ೽ⓨ㑆ᦨᄢᰴరA.
 䊥䉴䉺䊷䊃ᤨ䈱❗ዊᰴరA5

$ C+$:,,    䉣䊦䊚䊷䊃! /5/ 3 15//5@/<
@ /@ 5. ".. 3

 35!13333333..@/5<  1/33 

./.5353/33 
6,0%;@3 @/. 1.@@  1/

/3

 



31. 5/5./. 

.



㕖ኻ⒓ታ

//1@

55/ /

.

㕖ኻ⒓ታ

/.315

 31@.500 6:

 

///



,-,0 :$=-; ;%$:06

,-, :,/ /@3 ⶄ⚛ኻ⒓,-,0 :$
㕖ኻ⒓ታ

@. 3/

%%B,  , %: ,,%1@51@@!ਅਃⷺㇱ"
3 1:, ,$-6

 

:

 1 1@/ . "!51/ 11 133<1 @.1 .5//"!@ //5 51 .@<35@ 35 1"@  

⃻࿷䈱ข䉍⚵䉂!"  ')'*䉴䊧䉾䊄ਗ೉  ⸘᷹䊙䉲䊮
9 !)'4@/9=C ./.8B @䉮䉝" 2 E2 ಽ⸃䈮䉋䉎ㅪ┙৻ᰴᣇ⒟ᑼ

 ኻ⒓ⴕ೉䈱࿕᦭୯໧㗴 రᢙ!+"A రᢙ!+"A 5<< ')'*ጀ䈪ਗ೉ൻ ో࿕᦭୯䊶ో࿕᦭䊔䉪䊃䊦᳞⸃ 䉴䊧䉾䊄ਗ೉ൻലᨐ䉕಴䈚䈮䈒䈇໧㗴㪞㪝㫃㫆㫇㫊
㪈㪏 㪇
㪈㪍 㪇
㪈㪋 㪇
㪈㪉 㪇
㪈㪇 㪇
㪏㪇
㪍㪇
㪋㪇
㪉㪇
㪇

㪎

䉴䉬䊷䊤䊎
䊥䊁䉞

㪍

㪈㪉㪅㪅 㪏㪋⑽
㪈㪉
㪏㪋⑽

㪌
㪋
㪊
㪉
㪈

㪎㪏㪅
㪎㪏 㪅 㪌㪎⑽
㪌㪎⑽

㪇
㪇

㪌

㪈㪇
䉴䊧䉾䊄 ᢙ

㪈㪌

㪇

㪉㪇

.

㪌

㪈㪇
䉴䊧䉾䊄 ᢙ

㪈㪌

㪉㪇  

ን჻ㅢ ᢙቇ䊤䉟䊑䊤䊥ᣇ㊎ 㜞ㅦൻ䋺⛮⛯⊛䈮ข䉍⚵䉂 ๮઎䊧䊔䊦䉼䊠䊷䊆䊮䉫!๮઎䉴䉬䉳䊠䊷䊥䊮䉫ᡷༀ< ൻ< ,=, 0 < #"䇯
 䉨䊞䉾䉲䊠᦭ല೑↪!䊑䊨䉾䉪ൻ"䇯 䉴䊧䉾䊄ਗ೉ൻ䉼䊠䊷䊆䊮䉫䇯 ᯏ⢻⊛䉣䊮䊊䊮䉴䋺䉴䊧䉾䊄ਗ೉ᯏ⢻䉕㊀ὐ⊛䈮 㘈ቴ䉇䊐䉞䊷䊦䊄䈎䉌䈱ⷐᦸ䈮ኻ䈚䈩䇮ఝవ㗅૏䉕᳿䉄䈩ኻᔕ䇯 ᛛⴚേะ⺞ᩏ䈮ၮ䈨䈐⸘↹? ᛛⴚㅴዷ䈮䉋䉍㒸⣣ൻ䈚䈢ᣢሽ䊦䊷䉼䊮ᯏ⢻䈏䈅䉏䈳೚ᣂ䇯? ໧㗴ᄢⷙᮨൻ䈮䉋䉍․᦭䈱䉝䊦䉯䊥䉵䊛䈏ᔅⷐ䈮䈭䉏䈳ข䉍ㄟ䉂䇯 䊒䊨䉶䉴ਗ೉ 䋺⁁ᴫ䉡䉤䉾䉼 䊂䊷䉺ಽᢔᣇᴺ䈏౒ㅢ䈪䈭䈒䊤䉟䊑䊤䊥ൻ䈏㔍䈚䈇䇯 ⚻㛎⊛䈮䇮⁛⥄䉟䊮䉺䊷䊐䉢䊷䉴䈪䈱䊒䊨䉶䉴ਗ೉ᯏ⢻䈲䉝䊒䊥䈎䉌⸵ኈ䈘䉏䈭䈇䇯 ᄙᢙ䊡䊷䉱䉕₪ᓧ䈚䈢ᮡḰ⊛ᣂ઀᭽䈏⃻䉏䈢႐ว䈮䈲ㅊᓥ䈚䈩䈇䈒䇯
3  

  

ႎ๔ౝኈ ᒰ␠ᢙቇ䊤䉟䊑䊤䊥⚫੺ ᄢⷙᮨᢙ୯⸘▚ะ䈔 䊤䉟䊑䊤䊥᭎ⷰ ᄢⷙᮨኒⴕ೉໧㗴 ಽᢔਗ೉ 䉴䊌䊷䉴ⴕ೉♽䈱໧㗴

 䉁䈫䉄


⺞ᩏ!" ᄢⷙᮨኒⴕ೉໧㗴 0$')'* ')'*䈱ಽᢔ䊜䊝䊥ਗ೉ 䊤䉟䊑䊤䊥䇯 ಽᢔਗ೉䈱ኒⴕ೉໧㗴䋨ㅪ┙৻ᰴᣇ⒟ᑼ䊶ᦨዊੑਸ਼⸃䊶࿕᦭୯໧㗴䊶․⇣୯໧㗴䈭䈬䋩䈱䉸䊦䊋䈫䈚䈩䊂䊐䉜䉪䊃䉴䉺䊮䉻䊷䊄䇯 ⴕ೉䉕ੑᰴర䉰䉟䉪䊥䉾䉪ಽഀ䈜䉎䊂䊷䉺ಽᢔᣇᑼ䇯 8)䉟䊮䉺䊐䉢䊷䉴䈱䊒䊨䊃䉺䉟䊒 䉅ሽ࿷䈚䈢䇯 ')'*䈱$%0, 䈫䈚䈩㐿⊒ਛ䈱)''䈮ኻᔕ䈜䉎䉅䈱䈫䈚䈩䇮)''䉅㐿⊒ㅴⴕਛ䇯 ಽᢔ䊜䊝䊥ဳ䈱ᣂਗ೉⸒⺆䈏᥉෸䈜䉎䈢䉄䈮䈲䇮D$,䉕੺䈜䈭䈬䈚䈩
0$')'*䉕೑↪䈪䈐䉎䈱䈏ᦸ䉁䈚䈇䈫ᕁ䉒䉏䉎䇯? 90$-$-,)䈪䈲)ਗ೉䊤䉟䊑䊤䊥䈫䈱䉟䊮䉺䊐䉢䊷䉴ᬌ⸛ਛ䋿? $䈲$$ 7,6%䈱0$')'*䉕⸘↹ਛ䋿  

⺞ᩏ! " ಽᢔਗ೉





1ᧃ䈱ᦨᣂ !2"䈪)ਗ೉ᯏ⢻䉅ട䉒䈦䈢䈏䇮䋱ゲಽᢔ⋧ᒰ䈪䈅䉍䇮ᄢⷙᮨ䈪䈱ਗ೉ലᨐ䈲ᦼᓙ䈪䈐䈭䈇䇯 ⥄േ䉼䊠䊷䊆䊮䉫ᯏ᭴䉕ᜬ䈦䈩䈇䉎䈖䈫䈏․ᓽ⊛䇯 ᥉෸䈜䉎௑ะ䈣䈏䇮৻ᣇ䈪ౝㇱ᜼േ䈏ⶄ㔀䈪ᛒ䈇䈮䈒䈇㕙䉅䈅䉎䇯 9 ะ䈔䈮䈲䇮)'4@/9=C䈱๮઎䉕೑↪䈜䉎䈢䉄䈱ୃᱜᣇᴺḰ஻䇯











㕖Ᏹ䈮㐳䈇㐳䈪䈱)ਗ೉ᰴర䊦䊷䉼䊮䉕ᜬ䈦䈩䈇䉎䈖䈫䈏․ᓽ䇯)ਗ೉ᰴర䈲8)䈱-$-䈪↪䈇䉌䉏䈩䈇䉎䇯᳢↪ᕈ䈮䈲⇼໧䉅䈅䉍䋨૶↪䊉䊷䊄ᢙ䈫㐳䈘䈮㑐䈜䉎೙⚂䈏ᒝ䈇䈭䈬䋩䇯ᄢⷙᮨ䉲䉴䊁䊛䈪䈱ᄙᰴర䈮㑐䈜䉎ᯏ⢻䉇ᕈ⢻䈲ᧂ⺞ᩏ䈣䈏ᦼᓙ⭯䇯 㐳䈘䈲䋲౹䈱䉂䈪ታ䈲ή䇯)ਗ೉䈲䋲ᰴర䈱䉂䇯 4ᚑᨐ‛䇯䈠䈱ᓟ䈱ᵴേ⁁ᴫਇ᣿䇯 )F 䋳ᰴర䈱䋲ゲಽᢔ䋨ᩇ⁁䊂䊷䉺ಽഀ䋩䇯㶎 ᰴర䉦䊷䊈䊦ಣℂ䈣䈔䊤䉟䊑䊤䊥䉕೑↪䈚䈩䇮)ㇱಽ䈲䊡䊷䉱䊷䈏⥄૞䈚䈩䈚䉁䈉䈖䈫䉅ᄙ䈇䉅䉋䈉䇯  

⺞ᩏ!" 䉴䊌䊷䉴ⴕ೉♽䊤䉟䊑䊤䊥 ㅪ┙৻ᰴᣇ⒟ᑼ෻ᓳ⸃ᴺ䈭䈬 )0 -%6 8, 6#ⶄ⚛ᢙኻᔕ䇯⋥ធ⸃ᴺ䈭䈬䉅฽䇯G,0 ᜰะ䇯ⴕ೉ᚑಽ䉕ᩰ⚊䈚䈭䈇$ C=,,ᣇᑼ䉅น⢻䇯#ฦ⒳$0>$,㓸ว䇯G,0 ᜰะ䇯#䊙䊦䉼䉫䊥䉾䊄೨ಣℂ䇯ታⴕ೉䈱䉂䇯#ౝㇱ⊛䈮ᣇᑼ/୚♖ᐲṶ▚᦭䇯4ᚑᨐ‛䇯 ㅪ┙৻ᰴᣇ⒟ᑼ⋥ធ⸃ᴺ ,!H"#4(ᩰ⚊ᣇᑼ䈮ኻᔕ䇯ዋ䈚G,0 ᜰะ⊛䇯 )'*#䊙䊦䉼䊐䊨䊮䉺䊦ᴺ䇯ᩰ⚊ᣇᑼ!ォ⟎䉅น"䇯)ਗ೉ή䇯 )
#䊙䊦䉼䊐䊨䊮䉺䊦ᴺ䇯䉥䊷䉻䊥䊮䉫฽䋨'<I'<'䋩䇯 !)'4 <) #㕖 "㸢 䉴䊌䊷䉴ⴕ೉䉕࿶❗ᩰ⚊ᣇᑼ䈪ᷰ䈜䋨䊂䊷䉺ಽᢔᣇᑼ䈲ᧂ⺞ᩏ䋩䇯 ࿕᦭୯໧㗴 !)"'4)'*? 4!4,7,6, ::%0$ % % ,=$0,"䉕↪䈇䈩䈍䉍䇮ⴕ೉䊔䉪䊃䊦Ⓧ䈲䊡䊷䉱䊷஥䈱ಣℂ䈮䉉䈣䈰䉌䉏䈩䈇䉎䇯᣿䈮ⴕ೉ᚑಽ䉕ᩰ⚊䈜䉎ᔅⷐ䈏䈭䈇ὐ䈪᳢↪⊛䈪䈅䉎䈏䇮;,%)$::% ;,-䈫䈲⸒䈇㔍䈇䇯 ઁᄙᢙ!IJ< &)<&J)'*<''<'%$6$B<)0<#"? ᄢⷙᮨ⸘▚䈮⠴䈋䈭䈇⹜㛎⊛ታⵝ䈮䈫䈬䉁䈦䈩䈇䉎䉅䈱䈏ᄙ䈇ශ⽎䇯  

䉁䈫䉄䋺໧㗴䉺䉟䊒ಽ㘃᩺䈫ಽᢔਗ೉䊤䉟䊑䊤䊥⁁ᴫ ᄢⷙᮨኒⴕ೉䉺䉟䊒㸢 0$')'*䈱೑↪䈏⃻ታ⊛䇯 ⓨ㑆㗔ၞන⚐ಽഀ䉺䉟䊒 䋳ᰴరᯏ⢻㸢 ૶䈇䉇䈜䈇 䊤䉟䊑䊤䊥䈏ᢛ஻䈘䉏䈩䈇䉎䈫䈲⸒䈋䈭䈇⁁ᴫ䇯 ()ᰴ 䈪䋳ᰴరᯏ⢻䈱ᣂ䊦䊷䉼䊮䉕ㅊട䇯 ଀䈋䈳  )䋨,%,$-6,  ,%0- ;,0:6 % -$䋩 ⶄ㔀ಽഀ䈍䉋䈶䉴䊌䊷䉴ⴕ೉䉺䉟䊒㸢 ᷙㅅ⁁ᘒ䇯ᣂ䈚䈇)$::% ;,-䉕೑↪䈚䈢⛔৻⊛ᜰ㊎䈏ᔅⷐ䈎䇯

 䈠䈱ઁ
ੂᢙ↢ᚑ
ⴕ೉ᚑಽ䉥䊷䉻䊥䊮䉫㩿㪧㪸㫉㪤㪜㪫㪠㪪㪀䉫䊤䊐ಽഀ
䊜䉾䉲䊠↢ᚑ䋨㪧㪩㪘㪾㪤㪸㪫㪠㪺䋩
Ⓧಽ⸘▚䈱ಽᢔಣℂ䋨㪪㪪ᴺ䈫䈎䋩䊌䊤䊜䉺䉴䉟䊷䊒ㅪᚑ⸘▚㪚㫆㫄㫄㫆㫅㩷㪛㪸㫋㪸㩷㪝㫆㫉㫄㩿㪥㪼㫋㪚㪛㪝㪃㪟㪛㪝㪌㪀
นⷞൻ
#
/

  

5  

5.5 Open Petascale Libraries:Current Status

Open Petascale Libraries:
Current Status
Technical Computing Research Division
Fujitsu Laboratories of Europe
20 July 2012
FLE-TCR-PSL-12-010

Open Petascale Libraries
Global collaboration to develop advanced
numerical software for supercomputing.
 Dedicated forum to promote the open exchange of ideas andthe collaborative development of general-purpose andapplication-specific numerical libraries.
 Targeted initially at parallel computers built from multi-coreprocessors.
Open Petascale Libraries
Hybrid Programming Model(MPI + Threading)Multi-core CPU

Interconnect1

Copyright 2012 FUJITSU LABORATORIES OF EUROPE

OPL Members

2

Copyright 2012 FUJITSU LABORATORIES OF EUROPE

Open Petascale Libraries
 Six currently supported libraries:
 2DECOMP&FFT, version 1.4.682 (original developers: NAG Ltd).
 FFTE, version 5.0 (University of Tsukuba).
 spBLAS, version 1.01.672 (Fujitsu Laboratories of Europe).
 PLASMA, version 2.4.5 (ICL, University of Tennessee).
 PETSc, version 3.3-p1 (Argonne National Laboratory).
 PRAgMaTIc, version 1.0.2 (Imperial College London).

 www.openpetascale.org/index.php/public/page/download.
 Related libraries with no current OPL release: OpenMP-enabled version of PETSc under development within OPL.
 DPLASMA: distributed memory version of PLASMA.
 Plan for future OPL Random Number Generator (RNG) library.
3

Copyright 2012 FUJITSU LABORATORIES OF EUROPE

Status of Each Supported Library Features
 Status of FX10 Porting and Optimisation

4

2DECOMP&FFT 1.4.682
 2D pencil decomposition and parallel fast Fouriertransforms.
䃂

2D pencil decomposition FFTs.

䃂

3D FFTs.

䃂

Arbitrary input sizes.

䃂Complex-to-complex transforms.

䃂Complex-to-real transforms.

䃂Real-to-complex transforms.

䃂Interface with external FFT libraries.
Legend :䃂 : Ported onto FX10.
䂥 : Optimised for FX10.
No marks : There are features with the original version.
Grey coloured : Under development (for the original version).
5

Copyright 2012 FUJITSU LABORATORIES OF EUROPE

FFTE 5.0
 A package to compute discrete Fourier transforms in 1, 2
and 3 dimensions.
 K computer optimisation of FFTE carried out by University ofTsukuba as part of work towards 2011 HPC Challenge Benchmarks.
䃂䂥

1D, 2D and 3D FFTs.

䃂䂥

Inputs of sizes 2p3q5r.

䃂䂥Complex-to-complex transforms.

䃂䂥Complex-to-real transforms.

䃂䂥

Hybrid parallelism using OpenMP and MPI.

Legend :䃂 : Ported onto FX10.
䂥 : Optimised for FX10.
No marks : There are features with the original version.
Grey coloured : Under development (for the original version).

6

Copyright 2012 FUJITSU LABORATORIES OF EUROPE

spBLAS 1.01
 A threaded level 3 (matrix-matrix) sparse BLAS package.
䃂Compressed Sparse Row Matrices.

䃂

Compressed Sparse Column Matrices.

䃂

Coordinate Format Matrices.

䃂

Block Compressed Sparse Row Matrices.

Legend :䃂 : Ported onto FX10.
䂥 : Optimised for FX10.
No marks : There are features with the original version.
Grey coloured : Under development (for the original version).

7

Copyright 2012 FUJITSU LABORATORIES OF EUROPE

PLASMA 2.4.5
 A dense linear algebra package optimized for multi-coreprocessors.
䃂䃂䃂Symmetric Positive Definite Systems of Linear Equations.
Cholesky factorization (PLASMA_dpotrf).
General Systems of Linear Equations.

䃂

LU factorization (PLASMA_dgetrf).

䃂QR factorization (PLASMA_dgeqrf).

䃂䃂Standard and Generalized Dense Symmetric Eigenproblems.
Compute eigenvalues (PLASMA_dsyev).
Compute eigenvectors (research topic for developers).

䃂䃂

Dense Singular Value Decomposition.
Compute singular values (PLASMA_dgesvd).
Compute singular vectors (research topic for developers).

䃂

Tiled BLAS Level 3.

Legend :䃂 : Ported onto FX10.
䂥 : Optimised for FX10.

No marks : Features exist in original version (not ported).
Grey text : Under development (in the original version).
8

Copyright 2012 FUJITSU LABORATORIES OF EUROPE

PETSc 3.3-p1
 Data structures and routines for the solution of partial
differential equations.
䃂

Vector data structure.

䃂Matrix data structures.

䃂

Dense.

䃂Compressed sparse row (CSR).

䃂Block CSR.

䃂

Symmetric block CSR.

䃂Preconditioners.

䃂Krylov subspace methods (sparse iterative solvers).

䃂

Newton-based nonlinear solvers.

䃂

ODE solvers.

Legend :䃂 : Ported onto FX10.
䂥 : Optimised for FX10.

No marks : Features exist in original version (not ported).
Grey text : Under development (in the original version).
9

Copyright 2012 FUJITSU LABORATORIES OF EUROPE

PRAgMaTIc
 2D and 3D anisotropic mesh adaptivity for meshes ofsimplexes.
 Currently not supported on PRIMEHPC FX10.
 Port scheduled for completion by end of September.
Distributed mesh data structure.
Mesh operations.
Mesh smooth.
Mesh refine.
Mesh coarsen.
Face/edge swapping.
Legend :䃂 : Ported onto FX10.
䂥 : Optimised for FX10.
No marks : There are features with the original version.
Grey coloured : Under development (for the original version).
10

Copyright 2012 FUJITSU LABORATORIES OF EUROPE

Status of Related Libraries
 Features
 Status of FX10 Porting and Optimisation

11

PETSc with OpenMP Extend PETSc by adding OpenMP support (within OPL).
 Code is currently not ready for public release.
䃂

Vector data structure.

䃂Matrix data structures.

䃂

Dense.

䃂Compressed sparse row (CSR).
Block CSR.
Symmetric block CSR.

䃂Preconditioners.

䃂Krylov subspace methods (sparse iterative solvers).
Newton-based nonlinear solvers.
ODE solvers.

Legend :䃂 : Ported onto FX10.
䂥 : Optimised for FX10.

No marks : Features exist in original version (not ported).
Grey text : Under development (in the original version).
12

Copyright 2012 FUJITSU LABORATORIES OF EUROPE

Distributed PLASMA Initial release candidate of DAGuE/DPLASMA available
since December 2011 from University of Tennessee.
 DPLASMA is the multi-node version of PLASMA.
 DAGuE is the framework that distributes the DAG scheduler overmultiple nodes.
 OPL development (e.g. FX10 port) awaiting a full release.
Cholesky (factorization and solve).
QR and LQ (factorization and generation of Q).
LU (factorization and solve).
Matrix-matrix and triangular solve operations.
Legend :䃂 : Ported onto FX10.
䂥 : Optimised for FX10.
No marks : Features exist in original version (not ported).
Grey text : Under development (in the original version).
13

Copyright 2012 FUJITSU LABORATORIES OF EUROPE

PRIMEHPC FX10 Optimisation
 PLASMA PETSc

14

PLASMA Optimisation
 Targets for PRIMEHPC FX10 optimisation:
 Cholesky factorisation.
 LU and QR factorisation.
 Computation of eigenvalues and singular values.

 These have been highlighted by the developers as: Important functionality.
 Benefiting from the Directed Acyclic Graph approach (instead offork-join OpenMP).

 Benchmark performance of optimised version against:
 PLASMA with no optimisations.
 Equivalent SSL II routines.

15

Copyright 2012 FUJITSU LABORATORIES OF EUROPE

PETSc Optimisation
 Optimisation of PETSc is focused on the OpenMP versionof the code.
 Currently unreleased.
 Identical to non-OpenMP version if “-Kopenmp” is not specified.
 Uses SSL II for low-level BLAS routines.

 Initial experiments with using a threaded version of spBLAS
to achieve a hybrid version of PETSc were carried out.
 Good scaling (at least for a small number of threads), but introduced
a performance overhead when running with a single thread.
 PETSc developers at Argonne National Laboratory advised against
this strategy and recommended adding OpenMP to the PETSc
source code instead.
16

Copyright 2012 FUJITSU LABORATORIES OF EUROPE

PETSc Optimisation
 Four initial target matrices for optimisation (all from CFD):
 Backward-facing step (turbulent fluid flow).
 Lock exchange (mixing of two fluids with different densities).
 Salt fingering (mixing of two fluids with different densities).
 Flue pressure (dispersion of a momentum driven flue emission in across wind under neutral atmospheric conditions).

 Between 4 million and 750 million nonzero entries.
 Benchmark performance of optimised version against:
 Standard version of PETSc (no OpenMP, no optimisations).
 Initial OpenMP version of PETSc (no optimisations).
 Pthreads version of PETSc (available from PETSc developers).

17

Copyright 2012 FUJITSU LABORATORIES OF EUROPE

Future Plans

18

Future Plans
 By 30 September 2012: Release initial OpenMP-enabled version of PETSc.
 Complete PRIMEHPC FX10 optimisations for PETSc and PLASMA.
 Complete port of PRAgMaTIc to PRIMEHPC FX10.

 By 31 March 2013:
 Extend functionality of OpenMP-enabled version of PETSc (e.g. toinclude block CSR matrices).
 Identify and provide a supported release for a RNG library.
 PRIMEHPC FX10 optimisations for PRAgMaTIc and spBLAS.
 Consider inclusion of supported DPLASMA release in OPL.

19

Copyright 2012 FUJITSU LABORATORIES OF EUROPE

20

5.6 Application of PLASMA to OpenFMO code

Application of PLASMA to
OpenFMO code
James Southern, Paul Caheny, Michael Li
Fujitsu Laboratories of Europe
20 February 2012
FLE-TCR-PSL-13-004
Copyright 2012 FUJITSU LABORATORIES OF EUROPE

Overview
 Aim: Investigate the use of PLASMA in the OpenFMO
application.
 OpenFMO is a fragment molecular orbital (FMO) software package
for computational chemistry.
 The FMO method requires many eigenvector calculations.
 Typically these calculations use LAPACK.
 PLASMA is the OPL library identified as a replacement for LAPACK
on multi-core systems.
 What is the prospect for PLASMA to improve OpenFMO
performance compared to vendor LAPACK?
• On PRIMEHPC FX10 (compared to Fujitsu SSL II).
• On PRIMERGY cluster (compared to Intel MKL).

1

Copyright 2012 FUJITSU LABORATORIES OF EUROPE

OpenFMO Kyushu University’s implementation of the FMO method.
 The FMO method decomposes the molecule to becomputed into many small fragments.
 Quantum mechanical calculations on fragments and their dimers.
 Coulomb field calculation for entire system.
 Leads to highly efficient parallelization as calculation for individual
fragments can be farmed out to processors.
• Each process constructs a relatively small matrix for its fragment
and then computes its eigenvectors.
• Global broadcast of new energy field at end of iteration.
 A larger molecule or more basis functions means more small
fragments to farm out.
• Matrix size for each fragment remains small.
2

Copyright 2012 FUJITSU LABORATORIES OF EUROPE

PLASMA PLASMA is a linear algebra package aimed for use on
multi-threaded CPUs.
 Replaces LAPACK’s block algorithms withtile algorithms more suitable for finegrained parallelism.
 Uses Directed Acyclic Graph (DAG)
scheduling rather than fork-join parallelism to minimize idle time.
 These methods provide significant benefits for large matrices (lotsof work) on a large number of cores (when the overhead of forkjoin becomes very large).
 For small problems tile algorithms and DAG scheduling do not
perform so well Æ PLASMA is not designed for these problems.

Small matrix sizes in OpenFMO mean that prospects for
improving its performance using PLASMA are not good3

Copyright 2012 FUJITSU LABORATORIES OF EUROPE

OpenFMO Update
 Runtime and compilation problems for PRIMEHPC FX10
reported at last teleconference resolved.
 Compiling with –Xg resolves SIGBUS in ofmo_input() New openfmo tar received Feb 5th resolves problem compilingwithout –DDEBUG.
 OpenFMO profiled with SSL2.
 Small proportion of runtime spent in LAPACK functionality andmatrix sizes are small.
 Issues integrating PLASMA with OpenMP threaded application
codes on PRIMEHPC FX10 Æ PLASMA much slower than SSL II.

 OpenFMO also run on PRIMERGY cluster.
 No issues integrating PLASMA with OpenMP application.
 PLASMA performance with OpenFMO is much better than on
PRIMEHPC FX10 (but still not as good as MKL).
4

Copyright 2012 FUJITSU LABORATORIES OF EUROPE

OpenFMO Profile (FX10)
 Profiled OpenFMO with “Small” problem data set (1ejg-sto3) running on
48 nodes of Varuna GE (48 MPI x 16 OpenMP) with single threaded
SSL2 (runtime 5m15s).

<15% of total
OpenFMO run-time
spent in LAPACK
routines
Limited prospect for
improvement by useof any alternative to
LAPACK
Percentage of Thread 0 runtime spent in SYEV & SYGST per process.
5

Copyright 2012 FUJITSU LABORATORIES OF EUROPE

OpenFMO and Matrix Size SYGST & SYEV not themselves threaded in SSLBLAMP
but make use of multi-threaded BLAS.
 Impact of linking with SSL2BLAMP instead of SSL2 minimal ~ 1% in overall
application runtime.
 Possibly due to small matrix sizes in calls to SYEV & SYGST.

 Average N in SYEV calls from OpenFMO is small. (Ave N ~ 100 – 400
depending on input data set. More detail in following slides.)On SPARC64 IXfx, PLASMA
requires a matrix size over
6000 before it beats SSL II.

40
35
30
25
Gflop/s

PLASMA needs much larger
problem sizes to beat SSL II(due to overhead of DAGs andtile algorithms).

20
15
10

SSLIIdsyev

5

PLASMAdsyev

0
0

2000

4000

6000
8000
MatrixSize

6

10000

12000

14000

Copyright 2012 FUJITSU LABORATORIES OF EUROPE

PLASMA and OpenMP PLASMA must be linked with single threaded BLAS/LAPACK andhave OMP_NUM_THREADS set to 1 when initialised and calledby application code.
This is a problem for multi-threaded interoperability with OpenMP application code.

Current solution (from the PLASMA forum) is:
omp_set_num_threads(1) //setOpenMPnumberofthreadsto1
PLASMA_init(16)

//setPLASMAnumberofthreadsto16

PLASMA_dsyev(....)

//callPLASMAfunctionality

PLASMA_Finalize()

//FinalizePLASMA

omp_set_num_threads(16) //SetOpenMPnumberofthreadsbackto16for//applicationcodethatfollows.

 This workaround does not have a negative impact on performanceon PRIMERGY cluster.
 On FX10, using omp_set_num_threads requires setting
FLIB_FASTOMP=FALSE.
Negative impact of FLIB_FASTOMP=FALSE is significant on performance of
OpenFMO application code.
7

Copyright 2012 FUJITSU LABORATORIES OF EUROPE

OpenFMO on FX10
ExecutionTime(secs)
12000

Total Execution timefor OpenFMO with
1ejg-631s data set.

10000
8000
6000

ExecutionTime
4000
2000
0
LAPACKFASTOMP=TRUE

LAPACKFASTOMP=FALSE

PLASMA
FASTOMP=FALSE

For 1ejg-631s data set
Max N in SYEV = 680,
Min N in SYEV = 81
Avg N in SYEV = 337

PRIMEHPC FX10, 8 MPI x 16 OpenMP Threads, 1ejg-631s data set.

 Running OpenFMO with PLASMA is 1.13x slower than using SSL II.
 Largely due to setting FLIB_FASTOMP=FALSE.
8

Copyright 2012 FUJITSU LABORATORIES OF EUROPE

OpenFMO on PRIMERGY
Total Execution timefor OpenFMO with
1ejg-631s data set.

For 1ejg-631s data set
Max N in SYEV = 680,
Min N in SYEV = 81
Avg N in SYEV = 337
PRIMERGY BX900 Cluster, 8 MPI x 12 OpenMP Threads, 1ejg-631s data set.

 Running OpenFMO with PLASMA is 1.03x slower than using Intel MKL.
 Same as on PRIMEHPC FX10 when FLIB_FASTOMP=FALSE for SSL II.
9

Copyright 2012 FUJITSU LABORATORIES OF EUROPE

Scaling
Performance gap between
using SSL II and PLASMA
becomes larger when more
MPI processes are used
(proportion of execution
time in LAPACK increases).

2 processes are always
allocated as Master and I/Oby OpenFMO. Therefore 4
Nodes = 2 worker processes,
8 Nodes = 6 worker
processes etc.

For 1ejg-sto3 data set
Max N in SYEV = 240,
Min N in SYEV = 28
Ave N in SYEV = 123.4
Total Execution time for OpenFMO with 1ejg-sto3 data set, 16 threads per nodeon PRIMEHPC FX10.
10Copyright 2012 FUJITSU LABORATORIES OF EUROPE

Summary The use of PLASMA as an alternative to SSL II (or Intel
MKL) LAPACK within OpenFMO has been investigated on
PRIMEHPC FX10 and PRIMERGY cluster.
 The FMO method is not well suited for use with PLASMA: thesize of the matrices are too small to justify the overheads incurred
when using tile algorithms and DAGs.
• On PRIMEHPC FX10, 1.13x slowdown when using PLASMA(1.10x is due to setting FLIB_FASTOMP=FALSE).
• On PRIMERGY, FLIB_FASTOMP is not available, so slowdown isless (1.03x) Æ comparable to PRIMEHPC FX10.
 Little prospect of overcoming these issues for an FMO code.
 For a standard quantum chemistry code (that does not use the FMO
method) the size of the matrices will be much larger: PLASMA ismuch more likely to work well with this type of application.
11

Copyright 2012 FUJITSU LABORATORIES OF EUROPE12