PMlibのインストールとテスト	
•
•
•
•
•

ゲスト無線LANの使用について（別資料）	
PMlibパッケージの入手方法	
テストシステムへのログイン	
PMlibのインストール	
exampleプログラムの実行	PMlibパッケージの入手方法	
• PMlibパッケージの入手方法	
– 通常は下記公開リポジトリからDownload	
• http://avr-aics-riken.github.io/PMlib/	
– 本日は下記リポジトリからDownload	
• https://github.com/mikami3heart/PMlib-tutorials	
• PMlibに関する別ドキュメント	– doc/ディレクトリ以下にある	• html/index.html	
– 関数仕様の説明	– doc/ディレクトリでdoxygenコマンドにより自動生成	• How_to_use_PMlib.pdf ：クラスライブラリの説明書	

PMlibのインストール 京コンピュータ(1)	
• PMlibを利用するアプリケーションは計算ノードで実行される。	
• PMlibのインストール作業はログインノードでも計算ノードでも可能。本日はログインノードでPMlibをインストール	
• 公式リポジトリからパッケージをダウンロード	
• Webブラウザで以下ページをアクセス	– http://avr-aics-riken.github.io/PMlib/	• 右端のメニューからdownload tar.gzアイコンを選択	– 自分のPCにパッケージがダウンロードされる	– avr-aics-riken-PMlib-4.1.3-0-gced9279.tar.gz 	
• このファイルを京コンピュータログインノードへ転送する	• （本日京コンピュータ以外のシステムをリモート利用して実習参加の方は、各環境にあわせて以下読み替えて下さい）	

PMlibのインストール 京コンピュータ(2)	
• 京コンピュータの適当なディレクトリにパッケージを転送する	
myPC$ ssh ログイン名@k.aics.riken.jp ls -go
myPC$ ssh ログイン名@k.aics.riken.jp mkdir –p pmlib/tar_balls
myPC$ scp *PMlib*.tar.gz ログイン名:pmlib/tar_balls	• 京コンピュータへログインし、パッケージを展開する	
myPC$ ssh ログイン名@k.aics.riken.jp

K$ cd pmlib/tar_balls
K$ tar –zxf tar_balls/avr-aics-riken-PMlib-4.1.3-0-gced9279.tar.gz
K$ ln –s tar_balls/avr-aics-riken-PMlib-4.1.3-0-gced9279 Pmlib
K$ ls –golrwxrwxrwx 1 28 2015-08-24 PMlib -> avr-aics-riken-PMlib-ced9279
drwxr-xr-x 9 4096 2015-08-24 avr-aics-riken-PMlib-ced9279
drwxr-xr-x 2 4096 2015-08-24 tar_ballsPMlibのインストール 京コンピュータ(3)	
• ログインノード上で以下のコマンドで MPI版をmake する。	
K$ cat x.make-pmlib-K.sh#! /bin/bash
INSTALL_DIR=${HOME}/pmlib/install_dir
SRC_DIR=${HOME}/pmlib/PMlib
BUILD_DIR=${SRC_DIR}/BUILD_DIR
cd $BUILD_DIR; if [ $? != 0 ] ; then echo '@@@ Directory error @@@'; exit; fi# make distclean 2>&1 >/dev/null
CFLAGS="-std=c99 -Kopenmp,fast -Ntl_notrt"
FCFLAGS="-Cpp -Kopenmp,fast -Ntl_notrt"
CXXFLAGS="-Kopenmp,fast -Ntl_notrt "../configure CXX=mpiFCCpx CC=mpifccpx FC=mpifrtpx \CXXFLAGS="${CXXFLAGS}" CFLAGS="${CFLAGS}" FCFLAGS="${FCFLAGS}" \--with-comp=FJ --host=sparc64-unknown-linux-gnu \--with-papi=yes --with-example=yes --prefix=${INSTALL_DIR}
make
K$ ./x.make-pmlib-K.shPMlibのインストール 京コンピュータ(4)	
• configure/make のログ例	
K$ ./x.make-pmlib-K.sh+ ../configure CXX=mpiFCCpx CC=mpifccpx FC=mpifrtpx … ---------------------------------------Running PMlib Configure Script---------------------------------------checking for a BSD-compatible install... /usr/bin/install -cchecking whether build environment is sane... Yes…
--------------------------------------------Finished Running PMlib Configure Script--------------------------------------------+ makemake all-recursive
make[1]: ディレクトリ `/volume2/home/ra000004/a03155/pmlib/PMlib/BUILD_DIR' に入ります… (かなりメッセージが表示されますが、無害なものです）
make[1]: ディレクトリ `/volume2/home/ra000004/a03155/pmlib/PMlib/BUILD_DIR' から出ます	
K$	

PMlibのインストール 京コンピュータ(5)	
• exampleプログラムがmakeされた事を確認する	
K$ cd PMlib/BUILD_DIR/example
K$ ls -go test?/test?
-rwxr-xr-x 1 4451152 2015-08-25 12:39 test1/test1-rwxr-xr-x 1 4451397 2015-08-25 12:39 test2/test2-rwxr-xr-x 1 4456094 2015-08-25 12:39 test3/test3-rwxr-xr-x 1 4454195 2015-08-25 12:39 test4/test4
K$ file test?/test?
test1/test1: ELF 64-bit MSB executable, SPARC V9, total store ordering, version 1(SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.12, not stripped
test2/test2: ELF 64-bit MSB executable, SPARC V9, total store ordering, version 1(SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.12, not stripped
test3/test3: ELF 64-bit MSB executable, SPARC V9, total store ordering, version 1(SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.12, not stripped
test4/test4: ELF 64-bit MSB executable, SPARC V9, total store ordering, version 1(SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.12, not strippedK$PMlib例題プログラムの実行 京コンピュータ(1)	
• makeされたexampleプログラムを計算ノード上で対話的に実行
してみる	
K$ pjsub --interact --rsc-list "elapse=01:00:00" --rsc-list "node=1" --mpi "proc=2"[INFO] PJM 0000 pjsub Job 2955440 submitted.
[INFO] PJM 0081 ....connected.
[INFO] PJM 0082 pjsub Interactive job 2955440 started.
Env_base: K-1.2.0-18
K$ pwd${HOME}/pmlib/PMlib/BUILD_DIR/example
K$ /opt/FJSVXosPA/bin/xospastop
K$ mpiexec -np 2 ./test1/test1

PMlib例題プログラムの実行 京コンピュータ(2)	
• exampleプログラムのバッチジョブ実行例	
K$ cat x.run-test1.sh#!/bin/bash#PJM -N MYTEST1#PJM --rsc-list "elapse=1:00:00"#PJM --rsc-list "node=1"#PJM --mpi "proc=2”#PJM -j#PJM -S# stage io files#PJM --stg-transfiles all#PJM --mpi "use-rankdir"#PJM --stgin-basedir "/home/ra000004/a03155/pmlib/PMlib/BUILD_DIR/example"#PJM --stgin "rank=* test1/test1 %r:./test1”
source /work/system/Env_base/opt/FJSVXosPA/bin/xospastop
export OMP_NUM_THREADS=2 NPROCS=2
mpiexec -n ${NPROCS} ./test1
K$ pjsub x.run-test1.sh

example/test1の実行結果例	
#	 PMlib	 Basic	 Report	 -------------------------------------------------------

Timing	 Statistics	 Report	 from	 PMlib	 version	 4.1.2

Linked	 PMlib	 supports:	 MPI,	 OpenMP,	 HWPC

Host	 name	 :	 v04-005

Date	 	 	 	 	 	 :	 2015/08/25	 :	 15:36:16

Mr.	 Bean

Parallel	 Mode:	 	 	 Hybrid	 (2	 processes	 x	 2	 threads)

Total	 execution	 time	 	 	 	 	 	 	 	 	 	 	 	 =	 2.248993e+00	 [sec]Total	 time	 of	 measured	 sections	 =	 2.237422e+00	 [sec]Exclusive	 sections	 Statistics	 per	 process	 and	 per	 job.

Section	 	 	 	 	 	 	 	 	 |	 	 	 	 call	 	 |	 	 	 	 	 	 	 	 accumulated	 time[sec]	 	 	 	 	 	 	 	 	 	 	 |	 [flop	 counts	 or	 byte	 counts	 ]Label	 	 	 	 	 	 	 	 	 	 	 |	 	 	 	 	 	 	 	 	 	 |	 	 	 	 	 	 avr	 	 	 avr[%]	 	 	 	 	 	 sdv	 	 	 avr/call	 	 |	 	 	 	 	 	 avr	 	 	 	 	 	 	 sdv	 	 	 speed----------------+----------+----------------------------------------+----------------------------First	 location	 	 :	 	 	 	 	 	 	 	 3	 	 	 1.344e+00	 	 60.07	 	 6.02e-03	 	 4.480e-01	 	 	 	 0.000e+00	 	 0.00e+00	 	 	 0.00	 Mflops

Third	 location	 	 :	 	 	 	 	 	 	 	 1	 	 	 4.468e-01	 	 19.97	 	 2.98e-04	 	 4.468e-01	 	 	 	 1.601e+10	 	 0.00e+00	 	 35.83	 GB/sec

Second	 location	 :	 	 	 	 	 	 	 	 1	 	 	 4.466e-01	 	 19.96	 	 3.99e-04	 	 4.466e-01	 	 	 	 4.000e+09	 	 0.00e+00	 	 	 8.96	 Gflops----------------+----------+----------------------------------------+----------------------------

Per	 Process	 flop	 sections	 	 	 	 1.791e+00	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 4.000e+09	 	 	 	 	 	 	 	 	 	 	 	 	 2.23	 Gflops

Per	 Process	 byte	 sections	 	 	 	 4.468e-01	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 1.601e+10	 	 	 	 	 	 	 	 	 	 	 	 35.83	 GB/sec----------------+----------+----------------------------------------+----------------------------

Job	 Total	 flop	 sections	 	 	 	 	 	 1.791e+00	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 8.000e+09	 	 	 	 	 	 	 	 	 	 	 	 	 4.47	 Gflops

Job	 Total	 byte	 sections	 	 	 	 	 	 4.468e-01	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 3.202e+10	 	 	 	 	 	 	 	 	 	 	 	 71.66	 GB/sec

example/test1の実行結果例（続き）	
#	 PMlib	 Process	 Report	 ---	 Elapsed	 time	 for	 individual	 MPI	 ranks	 ------

Label	 	 First	 location

Header	 ID	 	 :	 	 	 	 	 call	 	 	 time[s]	 time[%]	 	 t_wait[s]	 	 t[s]/call	 	 	 flop|msg	 	 	 	 speed	 	 	 	 	 	 	 	 	 	 	 	 	 	 Rank	 	 	 	 	 0	 :	 	 	 	 	 	 	 	 3	 	 1.348e+00	 	 	 60.3	 	 0.000e+00	 	 4.494e-01	 	 0.000e+00	 	 0.000e+00	 Flops

Rank	 	 	 	 	 1	 :	 	 	 	 	 	 	 	 3	 	 1.340e+00	 	 	 59.9	 	 8.512e-03	 	 4.466e-01	 	 0.000e+00	 	 0.000e+00	 Flops

Label	 	 Second	 location

Header	 ID	 	 :	 	 	 	 	 call	 	 	 time[s]	 time[%]	 	 t_wait[s]	 	 t[s]/call	 	 	 flop|msg	 	 	 	 speed	 	 	 	 	 	 	 	 	 	 	 	 	 	 Rank	 	 	 	 	 0	 :	 	 	 	 	 	 	 	 1	 	 4.469e-01	 	 	 20.0	 	 0.000e+00	 	 4.469e-01	 	 4.000e+09	 	 8.950e+09	 Flops

Rank	 	 	 	 	 1	 :	 	 	 	 	 	 	 	 1	 	 4.464e-01	 	 	 20.0	 	 5.641e-04	 	 4.464e-01	 	 4.000e+09	 	 8.961e+09	 Flops

Label	 	 Third	 location

Header	 ID	 	 :	 	 	 	 	 call	 	 	 time[s]	 time[%]	 	 t_wait[s]	 	 t[s]/call	 	 	 flop|msg	 	 	 	 speed	 	 	 	 	 	 	 	 	 	 	 	 	 	 Rank	 	 	 	 	 0	 :	 	 	 	 	 	 	 	 1	 	 4.470e-01	 	 	 20.0	 	 0.000e+00	 	 4.470e-01	 	 1.601e+10	 	 3.581e+10	 Bytes/sec

Rank	 	 	 	 	 1	 :	 	 	 	 	 	 	 	 1	 	 4.466e-01	 	 	 20.0	 	 4.210e-04	 	 4.466e-01	 	 1.601e+10	 	 3.584e+10	 Bytes/sec#	 PMlib	 hardware	 performance	 counter	 (HWPC)	 Report	 -------------------------

HWPC_CHOOSER	 environment	 variable	 was	 not	 given.	 So	 there	 will	 be	 no	 HWPC	 output.

example/test2の実行結果例	
#	 #	 PMlib	 Basic	 Report	 -------------------------------------------------------

Timing	 Statistics	 Report	 from	 PMlib	 version	 4.1.2

Linked	 PMlib	 supports:	 MPI,	 OpenMP,	 HWPC

Host	 name	 :	 v04-005

Date	 	 	 	 	 	 :	 2015/08/25	 :	 16:17:09

Mr.	 Bean

Parallel	 Mode:	 	 	 Hybrid	 (2	 processes	 x	 2	 threads)

Total	 execution	 time	 	 	 	 	 	 	 	 	 	 	 	 =	 1.781796e+00	 [sec]Total	 time	 of	 measured	 sections	 =	 1.774080e+00	 [sec]Exclusive	 sections	 Statistics	 per	 process	 and	 per	 job.

Section	 	 	 	 	 	 	 	 	 |	 	 	 	 call	 	 |	 	 	 	 	 	 	 	 accumulated	 time[sec]	 	 	 	 	 	 	 	 	 	 	 |	 [flop	 counts	 or	 byte	 counts	 ]Label	 	 	 	 	 	 	 	 	 	 	 |	 	 	 	 	 	 	 	 	 	 |	 	 	 	 	 	 avr	 	 	 avr[%]	 	 	 	 	 	 sdv	 	 	 avr/call	 	 |	 	 	 	 	 	 avr	 	 	 	 	 	 	 sdv	 	 	 speed----------------+----------+----------------------------------------+----------------------------First	 location	 	 :	 	 	 	 	 	 	 	 1	 	 	 1.329e+00	 	 74.93	 	 6.96e-04	 	 1.329e+00	 	 	 	 0.000e+00	 	 0.00e+00	 	 	 0.00	 Mflops

Second	 location	 :	 	 	 	 	 	 	 	 1	 	 	 4.448e-01	 	 25.07	 	 3.39e-04	 	 4.448e-01	 	 	 	 0.000e+00	 	 0.00e+00	 	 	 0.00	 Mflops----------------+----------+----------------------------------------+----------------------------

Per	 Process	 flop	 sections	 	 	 	 1.774e+00	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 0.000e+00	 	 	 	 	 	 	 	 	 	 	 	 	 0.00	 Mflops----------------+----------+----------------------------------------+----------------------------

Job	 Total	 flop	 sections	 	 	 	 	 	 1.774e+00	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 0.000e+00	 	 	 	 	 	 	 	 	 	 	 	 	 0.00	 Mflops

example/test2の実行結果例（続き）	
#	 PMlib	 Process	 Report	 ---	 Elapsed	 time	 for	 individual	 MPI	 ranks	 ------

Label	 	 First	 location

Header	 ID	 	 :	 	 	 	 	 call	 	 	 time[s]	 time[%]	 	 t_wait[s]	 	 t[s]/call	 	 	 flop|msg	 	 	 	 speed

Rank	 	 	 	 	 0	 :	 	 	 	 	 	 	 	 1	 	 1.329e+00	 	 	 74.9	 	 9.842e-04	 	 1.329e+00	 	 0.000e+00	 	 0.000e+00	 Flops

Rank	 	 	 	 	 1	 :	 	 	 	 	 	 	 	 1	 	 1.330e+00	 	 	 75.0	 	 0.000e+00	 	 1.330e+00	 	 0.000e+00	 	 0.000e+00	 Flops

Label	 	 Second	 location

Header	 ID	 	 :	 	 	 	 	 call	 	 	 time[s]	 time[%]	 	 t_wait[s]	 	 t[s]/call	 	 	 flop|msg	 	 	 	 speed

Rank	 	 	 	 	 0	 :	 	 	 	 	 	 	 	 1	 	 4.451e-01	 	 	 25.1	 	 0.000e+00	 	 4.451e-01	 	 0.000e+00	 	 0.000e+00	 Flops

Rank	 	 	 	 	 1	 :	 	 	 	 	 	 	 	 1	 	 4.446e-01	 	 	 25.1	 	 4.790e-04	 	 4.446e-01	 	 0.000e+00	 	 0.000e+00	 Flops#	 PMlib	 hardware	 performance	 counter	 (HWPC)	 Report	 -------------------------

	 	 	 HWPC_CHOOSER	 environment	 variable	 was	 not	 given.	 So	 there	 will	 be	 no	 HWPC	 output.

example/test3の実行結果例	
#	 PMlib	 Basic	 Report	 -------------------------------------------------------

Timing	 Statistics	 Report	 from	 PMlib	 version	 4.1.2

Linked	 PMlib	 supports:	 MPI,	 OpenMP,	 HWPC

Host	 name	 :	 v04-005

Date	 	 	 	 	 	 :	 2015/08/25	 :	 16:20:55

Mr.	 Bean

Parallel	 Mode:	 	 	 Hybrid	 (2	 processes	 x	 2	 threads)

Total	 execution	 time	 	 	 	 	 	 	 	 	 	 	 	 =	 1.917597e+00	 [sec]Total	 time	 of	 measured	 sections	 =	 1.959178e+00	 [sec]Exclusive	 sections	 Statistics	 per	 process	 and	 per	 job.

Section	 	 	 	 	 	 |	 	 	 	 call	 	 |	 	 	 	 	 	 	 	 accumulated	 time[sec]	 	 	 	 	 	 	 	 	 	 	 |	 [flop	 counts	 or	 byte	 counts	 ]Label	 	 	 	 	 	 	 	 |	 	 	 	 	 	 	 	 	 	 |	 	 	 	 	 	 avr	 	 	 avr[%]	 	 	 	 	 	 sdv	 	 	 avr/call	 	 |	 	 	 	 	 	 avr	 	 	 	 	 	 	 sdv	 	 	 speed-------------+----------+----------------------------------------+----------------------------First	 block	 	 :	 	 	 	 	 	 	 	 1	 	 	 1.329e+00	 	 67.85	 	 4.00e-03	 	 1.329e+00	 	 	 	 0.000e+00	 	 0.00e+00	 	 	 0.00	 Mflops

Second	 block	 :	 	 	 	 	 	 	 	 1	 	 	 6.298e-01	 	 32.15	 	 7.70e-02	 	 6.298e-01	 	 	 	 0.000e+00	 	 0.00e+00	 	 	 0.00	 Mflops-------------+----------+----------------------------------------+----------------------------

Per	 Process	 flop	 sections	 	 	 1.959e+00	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 0.000e+00	 	 	 	 	 	 	 	 	 	 	 	 	 0.00	 Mflops-------------+----------+----------------------------------------+----------------------------

Job	 Total	 flop	 sections	 	 	 1.959e+00	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 0.000e+00	 	 	 	 	 	 	 	 	 	 	 	 	 0.00	 Mflops

example/test3の実行結果例（続き）	
#	 #	 PMlib	 Process	 Report	 ---	 Elapsed	 time	 for	 individual	 MPI	 ranks	 ------

Label	 	 First	 block

Header	 ID	 	 :	 	 	 	 	 call	 	 	 time[s]	 time[%]	 	 t_wait[s]	 	 t[s]/call	 	 	 flop|msg	 	 	 	 speed

Rank	 	 	 	 	 0	 :	 	 	 	 	 	 	 	 1	 	 1.332e+00	 	 	 68.0	 	 0.000e+00	 	 1.332e+00	 	 0.000e+00	 	 0.000e+00	 Flops

Rank	 	 	 	 	 1	 :	 	 	 	 	 	 	 	 1	 	 1.327e+00	 	 	 67.7	 	 5.660e-03	 	 1.327e+00	 	 0.000e+00	 	 0.000e+00	 Flops

Label	 	 Second	 block

Header	 ID	 	 :	 	 	 	 	 call	 	 	 time[s]	 time[%]	 	 t_wait[s]	 	 t[s]/call	 	 	 flop|msg	 	 	 	 speed

Rank	 	 	 	 	 0	 :	 	 	 	 	 	 	 	 1	 	 5.753e-01	 	 	 29.4	 	 1.089e-01	 	 5.753e-01	 	 0.000e+00	 	 0.000e+00	 Flops

Rank	 	 	 	 	 1	 :	 	 	 	 	 	 	 	 1	 	 6.842e-01	 	 	 34.9	 	 0.000e+00	 	 6.842e-01	 	 0.000e+00	 	 0.000e+00	 Flops#	 PMlib	 hardware	 performance	 counter	 (HWPC)	 Report	 -------------------------

	 	 	 HWPC_CHOOSER	 environment	 variable	 was	 not	 given.	 So	 there	 will	 be	 no	 HWPC	 output.

#	 PMlib	 Process	 Group	 [	 	 	 77]	 Elapsed	 time	 for	 individual	 MPI	 ranks	 --------

Label	 	 First	 block

Header	 ID	 	 :	 	 	 	 	 call	 	 	 time[s]	 time[%]	 	 t_wait[s]	 	 t[s]/call	 	 	 flop|msg	 	 	 	 speed

Rank	 	 	 	 	 0	 :	 	 	 	 	 	 	 	 1	 	 1.332e+00	 	 	 68.0	 	 0.000e+00	 	 1.332e+00	 	 0.000e+00	 	 0.000e+00	 Flops

Label	 	 Second	 block

Header	 ID	 	 :	 	 	 	 	 call	 	 	 time[s]	 time[%]	 	 t_wait[s]	 	 t[s]/call	 	 	 flop|msg	 	 	 	 speed

Rank	 	 	 	 	 0	 :	 	 	 	 	 	 	 	 1	 	 5.753e-01	 	 	 29.4	 	 0.000e+00	 	 5.753e-01	 	 0.000e+00	 	 0.000e+00	 Flops#	 PMlib	 Process	 Group	 [	 	 	 77]	 hardware	 performance	 counter	 (HWPC)	 Report	 ---

	 	 	 HWPC_CHOOSER	 environment	 variable	 was	 not	 given.	 So	 there	 will	 be	 no	 HWPC	 output.

example/test3の実行結果例	
#	 #	 PMlib	 Process	 Report	 ---	 Elapsed	 time	 for	 individual	 MPI	 ranks	 ------

Label	 	 First	 block

Header	 ID	 	 :	 	 	 	 	 call	 	 	 time[s]	 time[%]	 	 t_wait[s]	 	 t[s]/call	 	 	 flop|msg	 	 	 	 speed

Rank	 	 	 	 	 0	 :	 	 	 	 	 	 	 	 1	 	 1.332e+00	 	 	 68.0	 	 0.000e+00	 	 1.332e+00	 	 0.000e+00	 	 0.000e+00	 Flops

Rank	 	 	 	 	 1	 :	 	 	 	 	 	 	 	 1	 	 1.327e+00	 	 	 67.7	 	 5.660e-03	 	 1.327e+00	 	 0.000e+00	 	 0.000e+00	 Flops

Label	 	 Second	 block

Header	 ID	 	 :	 	 	 	 	 call	 	 	 time[s]	 time[%]	 	 t_wait[s]	 	 t[s]/call	 	 	 flop|msg	 	 	 	 speed

Rank	 	 	 	 	 0	 :	 	 	 	 	 	 	 	 1	 	 5.753e-01	 	 	 29.4	 	 1.089e-01	 	 5.753e-01	 	 0.000e+00	 	 0.000e+00	 Flops

Rank	 	 	 	 	 1	 :	 	 	 	 	 	 	 	 1	 	 6.842e-01	 	 	 34.9	 	 0.000e+00	 	 6.842e-01	 	 0.000e+00	 	 0.000e+00	 Flops#	 PMlib	 hardware	 performance	 counter	 (HWPC)	 Report	 -------------------------

	 	 	 HWPC_CHOOSER	 environment	 variable	 was	 not	 given.	 So	 there	 will	 be	 no	 HWPC	 output.

#	 PMlib	 Process	 Group	 [	 	 	 77]	 Elapsed	 time	 for	 individual	 MPI	 ranks	 --------

Label	 	 First	 block

Header	 ID	 	 :	 	 	 	 	 call	 	 	 time[s]	 time[%]	 	 t_wait[s]	 	 t[s]/call	 	 	 flop|msg	 	 	 	 speed

Rank	 	 	 	 	 0	 :	 	 	 	 	 	 	 	 1	 	 1.332e+00	 	 	 68.0	 	 0.000e+00	 	 1.332e+00	 	 0.000e+00	 	 0.000e+00	 Flops

Label	 	 Second	 block

Header	 ID	 	 :	 	 	 	 	 call	 	 	 time[s]	 time[%]	 	 t_wait[s]	 	 t[s]/call	 	 	 flop|msg	 	 	 	 speed

Rank	 	 	 	 	 0	 :	 	 	 	 	 	 	 	 1	 	 5.753e-01	 	 	 29.4	 	 0.000e+00	 	 5.753e-01	 	 0.000e+00	 	 0.000e+00	 Flops#	 PMlib	 Process	 Group	 [	 	 	 77]	 hardware	 performance	 counter	 (HWPC)	 Report	 ---

	 	 	 HWPC_CHOOSER	 environment	 variable	 was	 not	 given.	 So	 there	 will	 be	 no	 HWPC	 output.

example/test3の実行結果例(8MPIの場合)	
#	 PMlib	 Process	 Report	 ---	 Elapsed	 time	 for	 individual	 MPI	 ranks	 ------

Label	 	 First	 block

Header	 ID	 	 :	 	 	 	 	 call	 	 	 time[s]	 time[%]	 	 t_wait[s]	 	 t[s]/call	 	 	 flop|msg	 	 	 	 speed

Rank	 	 	 	 	 0	 :	 	 	 	 	 	 	 	 1	 	 3.671e+00	 	 	 60.3	 	 2.049e-03	 	 3.671e+00	 	 0.000e+00	 	 0.000e+00	 Flops

Rank	 	 	 	 	 1	 :	 	 	 	 	 	 	 	 1	 	 3.672e+00	 	 	 60.3	 	 1.986e-03	 	 3.672e+00	 	 0.000e+00	 	 0.000e+00	 Flops

Rank	 	 	 	 	 2	 :	 	 	 	 	 	 	 	 1	 	 3.673e+00	 	 	 60.3	 	 8.080e-04	 	 3.673e+00	 	 0.000e+00	 	 0.000e+00	 Flops

Rank	 	 	 	 	 3	 :	 	 	 	 	 	 	 	 1	 	 3.674e+00	 	 	 60.3	 	 1.502e-05	 	 3.674e+00	 	 0.000e+00	 	 0.000e+00	 Flops

Rank	 	 	 	 	 4	 :	 	 	 	 	 	 	 	 1	 	 3.674e+00	 	 	 60.3	 	 0.000e+00	 	 3.674e+00	 	 0.000e+00	 	 0.000e+00	 Flops

Rank	 	 	 	 	 5	 :	 	 	 	 	 	 	 	 1	 	 3.673e+00	 	 	 60.3	 	 2.699e-04	 	 3.673e+00	 	 0.000e+00	 	 0.000e+00	 Flops

Rank	 	 	 	 	 6	 :	 	 	 	 	 	 	 	 1	 	 3.673e+00	 	 	 60.3	 	 7.451e-04	 	 3.673e+00	 	 0.000e+00	 	 0.000e+00	 Flops

Rank	 	 	 	 	 7	 :	 	 	 	 	 	 	 	 1	 	 3.673e+00	 	 	 60.3	 	 5.660e-04	 	 3.673e+00	 	 0.000e+00	 	 0.000e+00	 Flops

Label	 	 Second	 block

Header	 ID	 	 :	 	 	 	 	 call	 	 	 time[s]	 time[%]	 	 t_wait[s]	 	 t[s]/call	 	 	 flop|msg	 	 	 	 speed

Rank	 	 	 	 	 0	 :	 	 	 	 	 	 	 	 1	 	 2.048e+00	 	 	 33.6	 	 7.445e-01	 	 2.048e+00	 	 0.000e+00	 	 0.000e+00	 Flops

Rank	 	 	 	 	 1	 :	 	 	 	 	 	 	 	 1	 	 2.048e+00	 	 	 33.6	 	 7.446e-01	 	 2.048e+00	 	 0.000e+00	 	 0.000e+00	 Flops

Rank	 	 	 	 	 2	 :	 	 	 	 	 	 	 	 1	 	 2.048e+00	 	 	 33.6	 	 7.445e-01	 	 2.048e+00	 	 0.000e+00	 	 0.000e+00	 Flops

Rank	 	 	 	 	 3	 :	 	 	 	 	 	 	 	 1	 	 2.048e+00	 	 	 33.6	 	 7.444e-01	 	 2.048e+00	 	 0.000e+00	 	 0.000e+00	 Flops

Rank	 	 	 	 	 4	 :	 	 	 	 	 	 	 	 1	 	 2.793e+00	 	 	 45.8	 	 1.788e-05	 	 2.793e+00	 	 0.000e+00	 	 0.000e+00	 Flops

Rank	 	 	 	 	 5	 :	 	 	 	 	 	 	 	 1	 	 2.793e+00	 	 	 45.8	 	 0.000e+00	 	 2.793e+00	 	 0.000e+00	 	 0.000e+00	 Flops

Rank	 	 	 	 	 6	 :	 	 	 	 	 	 	 	 1	 	 2.793e+00	 	 	 45.8	 	 4.792e-05	 	 2.793e+00	 	 0.000e+00	 	 0.000e+00	 Flops

Rank	 	 	 	 	 7	 :	 	 	 	 	 	 	 	 1	 	 2.793e+00	 	 	 45.8	 	 3.695e-05	 	 2.793e+00	 	 0.000e+00	 	 0.000e+00	 Flops#	 PMlib	 Process	 Group	 [	 	 	 77]	 Elapsed	 time	 for	 individual	 MPI	 ranks	 --------

Label	 	 First	 block

Header	 ID	 	 :	 	 	 	 	 call	 	 	 time[s]	 time[%]	 	 t_wait[s]	 	 t[s]/call	 	 	 flop|msg	 	 	 	 speed

Rank	 	 	 	 	 0	 :	 	 	 	 	 	 	 	 1	 	 3.671e+00	 	 	 60.3	 	 2.034e-03	 	 3.671e+00	 	 0.000e+00	 	 0.000e+00	 Flops

Rank	 	 	 	 	 1	 :	 	 	 	 	 	 	 	 1	 	 3.672e+00	 	 	 60.3	 	 1.971e-03	 	 3.672e+00	 	 0.000e+00	 	 0.000e+00	 Flops

Rank	 	 	 	 	 2	 :	 	 	 	 	 	 	 	 1	 	 3.673e+00	 	 	 60.3	 	 7.930e-04	 	 3.673e+00	 	 0.000e+00	 	 0.000e+00	 Flops

Rank	 	 	 	 	 3	 :	 	 	 	 	 	 	 	 1	 	 3.674e+00	 	 	 60.3	 	 0.000e+00	 	 3.674e+00	 	 0.000e+00	 	 0.000e+00	 Flops

Label	 	 Second	 block

Header	 ID	 	 :	 	 	 	 	 call	 	 	 time[s]	 time[%]	 	 t_wait[s]	 	 t[s]/call	 	 	 flop|msg	 	 	 	 speed

Rank	 	 	 	 	 0	 :	 	 	 	 	 	 	 	 1	 	 2.048e+00	 	 	 33.6	 	 1.597e-05	 	 2.048e+00	 	 0.000e+00	 	 0.000e+00	 Flops

Rank	 	 	 	 	 1	 :	 	 	 	 	 	 	 	 1	 	 2.048e+00	 	 	 33.6	 	 1.390e-04	 	 2.048e+00	 	 0.000e+00	 	 0.000e+00	 Flops

Rank	 	 	 	 	 2	 :	 	 	 	 	 	 	 	 1	 	 2.048e+00	 	 	 33.6	 	 1.311e-05	 	 2.048e+00	 	 0.000e+00	 	 0.000e+00	 Flops

Rank	 	 	 	 	 3	 :	 	 	 	 	 	 	 	 1	 	 2.048e+00	 	 	 33.6	 	 0.000e+00	 	 2.048e+00	 	 0.000e+00	 	 0.000e+00	 Flops

example/test4の実行結果例	
#	 PMlib	 Basic	 Report	 -------------------------------------------------------

Timing	 Statistics	 Report	 from	 PMlib	 version	 4.1.2

Linked	 PMlib	 supports:	 MPI,	 OpenMP,	 HWPC

Host	 name	 :	 v04-008

Date	 	 	 	 	 	 :	 2015/08/25	 :	 16:33:16

user

Parallel	 Mode:	 	 	 Hybrid	 (2	 processes	 x	 2	 threads)

Total	 execution	 time	 	 	 	 	 	 	 	 	 	 	 	 =	 1.666007e+01	 [sec]Total	 time	 of	 measured	 sections	 =	 1.665253e+01	 [sec]Exclusive	 sections	 Statistics	 per	 process	 and	 per	 job.

Section	 	 	 |	 	 	 	 call	 	 |	 	 	 	 	 	 	 	 accumulated	 time[sec]	 	 	 	 	 	 	 	 	 	 	 |	 [flop	 counts	 or	 byte	 counts	 ]Label	 	 	 	 	 |	 	 	 	 	 	 	 	 	 	 |	 	 	 	 	 	 avr	 	 	 avr[%]	 	 	 	 	 	 sdv	 	 	 avr/call	 	 |	 	 	 	 	 	 avr	 	 	 	 	 	 	 sdv	 	 	 speed----------+----------+----------------------------------------+----------------------------

2-submtxm	 :	 	 	 	 	 	 	 	 3	 	 	 1.665e+01	 	 99.97	 	 4.45e-03	 	 5.549e+00	 	 	 	 0.000e+00	 	 0.00e+00	 	 	 0.00	 MB/sec

1-subinit	 :	 	 	 	 	 	 	 	 1	 	 	 5.305e-03	 	 	 0.03	 	 6.91e-06	 	 5.305e-03	 	 	 	 0.000e+00	 	 0.00e+00	 	 	 0.00	 MB/sec----------+----------+----------------------------------------+----------------------------

Per	 Process	 byte	 sections	 	 	 1.665e+01	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 0.000e+00	 	 	 	 	 	 	 	 	 	 	 	 	 0.00	 MB/sec----------+----------+----------------------------------------+----------------------------

Job	 Total	 byte	 sections	 	 	 1.665e+01	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 0.000e+00	 	 	 	 	 	 	 	 	 	 	 	 	 0.00	 MB/sec

example/test4の実行結果例（続き）	
#	 PMlib	 Process	 Report	 ---	 Elapsed	 time	 for	 individual	 MPI	 ranks	 ------

Label	 	 1-subinit

Header	 ID	 	 :	 	 	 	 	 call	 	 	 time[s]	 time[%]	 	 t_wait[s]	 	 t[s]/call	 	 	 flop|msg	 	 	 	 speed

Rank	 	 	 	 	 0	 :	 	 	 	 	 	 	 	 1	 	 5.310e-03	 	 	 	 0.0	 	 0.000e+00	 	 5.310e-03	 	 0.000e+00	 	 0.000e+00	 Bytes/sec

Rank	 	 	 	 	 1	 :	 	 	 	 	 	 	 	 1	 	 5.300e-03	 	 	 	 0.0	 	 9.775e-06	 	 5.300e-03	 	 0.000e+00	 	 0.000e+00	 Bytes/sec

Label	 	 2-submtxm

Header	 ID	 	 :	 	 	 	 	 call	 	 	 time[s]	 time[%]	 	 t_wait[s]	 	 t[s]/call	 	 	 flop|msg	 	 	 	 speed

Rank	 	 	 	 	 0	 :	 	 	 	 	 	 	 	 3	 	 1.665e+01	 	 100.0	 	 0.000e+00	 	 5.550e+00	 	 0.000e+00	 	 0.000e+00	 Bytes/sec

Rank	 	 	 	 	 1	 :	 	 	 	 	 	 	 	 3	 	 1.664e+01	 	 	 99.9	 	 6.287e-03	 	 5.548e+00	 	 0.000e+00	 	 0.000e+00	 Bytes/sec#	 PMlib	 hardware	 performance	 counter	 (HWPC)	 Report	 -------------------------

	 	 	 HWPC_CHOOSER	 environment	 variable	 was	 not	 given.	 So	 there	 will	 be	 no	 HWPC	 output.

例題プログラムでHWPCを利用 京コンピュータ(1)	
• ジョブスクリプト x.run-test1.shを編集（一行追加）	
K$ cat x.run-test1.sh#!/bin/bash#PJM -N MYTEST1#PJM --rsc-list "elapse=1:00:00"#PJM --rsc-list "node=1"#PJM --mpi "proc=2”#PJM -j#PJM -S# stage io files#PJM --stg-transfiles all#PJM --mpi "use-rankdir"#PJM --stgin-basedir "/home/ra000004/a03155/pmlib/PMlib/BUILD_DIR/example"#PJM --stgin "rank=* test1/test1 %r:./test1”
source /work/system/Env_base/opt/FJSVXosPA/bin/xospastop
export OMP_NUM_THREADS=2 NPROCS=2
export HWPC_CHOOSER=FLOPS
mpiexec -n ${NPROCS} ./test1
K$ pjsub x.run-test1.sh

example/test1のHWPC利用結果例(1)	
#	 PMlib	 Basic	 Report	 -------------------------------------------------------

Timing	 Statistics	 Report	 from	 PMlib	 version	 4.1.2

Linked	 PMlib	 supports:	 MPI,	 OpenMP,	 HWPC

Host	 name	 :	 e05-008

Date	 	 	 	 	 	 :	 2015/08/26	 :	 01:49:07

Mr.	 Bean

Parallel	 Mode:	 	 	 Hybrid	 (2	 processes	 x	 2	 threads)

Total	 execution	 time	 	 	 	 	 	 	 	 	 	 	 	 =	 2.240497e+00	 [sec]Total	 time	 of	 measured	 sections	 =	 2.233066e+00	 [sec]Exclusive	 sections	 Statistics	 per	 process	 and	 per	 job.

Section	 	 	 	 	 	 	 	 	 |	 	 	 	 call	 	 |	 	 	 	 	 	 	 	 accumulated	 time[sec]	 	 	 	 	 	 	 	 	 	 	 |	 [flop	 counts	 or	 byte	 counts	 ]Label	 	 	 	 	 	 	 	 	 	 	 |	 	 	 	 	 	 	 	 	 	 |	 	 	 	 	 	 avr	 	 	 avr[%]	 	 	 	 	 	 sdv	 	 	 avr/call	 	 |	 	 	 	 	 	 avr	 	 	 	 	 	 	 sdv	 	 	 speed----------------+----------+----------------------------------------+----------------------------First	 location	 	 :	 	 	 	 	 	 	 	 3	 	 	 1.339e+00	 	 59.97	 	 1.58e-03	 	 4.464e-01	 	 	 	 1.216e+10	 	 4.24e+00	 	 	 9.08	 Gflops

Second	 location	 :	 	 	 	 	 	 	 	 1	 	 	 4.472e-01	 	 20.03	 	 4.22e-04	 	 4.472e-01	 	 	 	 4.033e+09	 	 6.74e-07	 	 	 9.02	 Gflops

Third	 location	 	 :	 	 	 	 	 	 	 	 1	 	 	 4.468e-01	 	 20.01	 	 6.49e-04	 	 4.468e-01	 	 	 	 4.033e+09	 	 7.07e-01	 	 	 9.03	 Gflops----------------+----------+----------------------------------------+----------------------------

Per	 Process	 flop	 sections	 	 	 	 2.233e+00	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 2.022e+10	 	 	 	 	 	 	 	 	 	 	 	 	 9.06	 Gflops----------------+----------+----------------------------------------+----------------------------

Job	 Total	 flop	 sections	 	 	 	 	 	 2.233e+00	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 4.044e+10	 	 	 	 	 	 	 	 	 	 	 	 18.11	 Gflops

example/test1のHWPC利用結果例(1)	
#	 PMlib	 Process	 Report	 ---	 Elapsed	 time	 for	 individual	 MPI	 ranks	 ------

Label	 	 First	 location

Header	 ID	 	 :	 	 	 	 	 call	 	 	 time[s]	 time[%]	 	 t_wait[s]	 	 t[s]/call	 	 	 flop|msg	 	 	 	 speed

Rank	 	 	 	 	 0	 :	 	 	 	 	 	 	 	 3	 	 1.340e+00	 	 	 60.0	 	 0.000e+00	 	 4.467e-01	 	 1.216e+10	 	 9.070e+09	 Flops	 (HWPC)

Rank	 	 	 	 	 1	 :	 	 	 	 	 	 	 	 3	 	 1.338e+00	 	 	 59.9	 	 2.240e-03	 	 4.460e-01	 	 1.216e+10	 	 9.086e+09	 Flops	 (HWPC)

Label	 	 Second	 location

Header	 ID	 	 :	 	 	 	 	 call	 	 	 time[s]	 time[%]	 	 t_wait[s]	 	 t[s]/call	 	 	 flop|msg	 	 	 	 speed

Rank	 	 	 	 	 0	 :	 	 	 	 	 	 	 	 1	 	 4.475e-01	 	 	 20.0	 	 0.000e+00	 	 4.475e-01	 	 4.033e+09	 	 9.012e+09	 Flops	 (HWPC)

Rank	 	 	 	 	 1	 :	 	 	 	 	 	 	 	 1	 	 4.469e-01	 	 	 20.0	 	 5.963e-04	 	 4.469e-01	 	 4.033e+09	 	 9.024e+09	 Flops	 (HWPC)

Label	 	 Third	 location

Header	 ID	 	 :	 	 	 	 	 call	 	 	 time[s]	 time[%]	 	 t_wait[s]	 	 t[s]/call	 	 	 flop|msg	 	 	 	 speed

Rank	 	 	 	 	 0	 :	 	 	 	 	 	 	 	 1	 	 4.472e-01	 	 	 20.0	 	 0.000e+00	 	 4.472e-01	 	 4.033e+09	 	 9.018e+09	 Flops	 (HWPC)

Rank	 	 	 	 	 1	 :	 	 	 	 	 	 	 	 1	 	 4.463e-01	 	 	 20.0	 	 9.172e-04	 	 4.463e-01	 	 4.033e+09	 	 9.036e+09	 Flops	 (HWPC)#	 PMlib	 hardware	 performance	 counter	 (HWPC)	 Report	 -------------------------

	 	 	 HWPC_CHOOSER=FLOPS	 statistics	 were	 collected.

	 	 	 The	 values	 of	 each	 process	 are	 the	 sum	 of	 threads.

Label	 	 	 First	 location

Header	 	 ID	 :	 	 	 	 	 FP_OPS	 	 	 	 [Flops]

Rank	 	 	 	 	 0	 :	 	 1.216e+10	 	 9.070e+09

Rank	 	 	 	 	 1	 :	 	 1.216e+10	 	 9.086e+09

Label	 	 	 Second	 location

Header	 	 ID	 :	 	 	 	 	 FP_OPS	 	 	 	 [Flops]

Rank	 	 	 	 	 0	 :	 	 4.033e+09	 	 9.012e+09

Rank	 	 	 	 	 1	 :	 	 4.033e+09	 	 9.024e+09

Label	 	 	 Third	 location

Header	 	 ID	 :	 	 	 	 	 FP_OPS	 	 	 	 [Flops]例題プログラムでHWPCを利用 京コンピュータ(2)	
• ジョブスクリプト x.run-test1.shを編集	
K$ cat x.run-test1.sh#!/bin/bash#PJM -N MYTEST1#PJM --rsc-list "elapse=1:00:00"#PJM --rsc-list "node=1"#PJM --mpi "proc=2”#PJM -j#PJM -S# stage io files#PJM --stg-transfiles all#PJM --mpi "use-rankdir"#PJM --stgin-basedir "/home/ra000004/a03155/pmlib/PMlib/BUILD_DIR/example"#PJM --stgin "rank=* test1/test1 %r:./test1"#source /work/system/Env_base/opt/FJSVXosPA/bin/xospastop
export OMP_NUM_THREADS=2 NPROCS=2
export HWPC_CHOOSER=BANDWIDTH
mpiexec -n ${NPROCS} ./test1
K$ pjsub x.run-test1.sh

example/test1のHWPC利用結果例(2)	
#	 PMlib	 Basic	 Report	 -------------------------------------------------------

Timing	 Statistics	 Report	 from	 PMlib	 version	 4.1.2

Linked	 PMlib	 supports:	 MPI,	 OpenMP,	 HWPC

Host	 name	 :	 e05-008

Date	 	 	 	 	 	 :	 2015/08/26	 :	 02:02:09

Mr.	 Bean

Parallel	 Mode:	 	 	 Hybrid	 (2	 processes	 x	 2	 threads)

Total	 execution	 time	 	 	 	 	 	 	 	 	 	 	 	 =	 2.240130e+00	 [sec]Total	 time	 of	 measured	 sections	 =	 2.232404e+00	 [sec]Exclusive	 sections	 Statistics	 per	 process	 and	 per	 job.

Section	 	 	 	 	 	 	 	 	 |	 	 	 	 call	 	 |	 	 	 	 	 	 	 	 accumulated	 time[sec]	 	 	 	 	 	 	 	 	 	 	 |	 [flop	 counts	 or	 byte	 counts	 ]Label	 	 	 	 	 	 	 	 	 	 	 |	 	 	 	 	 	 	 	 	 	 |	 	 	 	 	 	 avr	 	 	 avr[%]	 	 	 	 	 	 sdv	 	 	 avr/call	 	 |	 	 	 	 	 	 avr	 	 	 	 	 	 	 sdv	 	 	 speed----------------+----------+----------------------------------------+----------------------------First	 location	 	 :	 	 	 	 	 	 	 	 3	 	 	 1.340e+00	 	 60.02	 	 1.90e-03	 	 4.466e-01	 	 	 	 2.072e+10	 	 2.11e+09	 	 15.46	 GB/sec

Third	 location	 	 :	 	 	 	 	 	 	 	 1	 	 	 4.466e-01	 	 20.00	 	 6.90e-04	 	 4.466e-01	 	 	 	 6.885e+09	 	 9.21e+08	 	 15.42	 GB/sec

Second	 location	 :	 	 	 	 	 	 	 	 1	 	 	 4.459e-01	 	 19.97	 	 5.09e-04	 	 4.459e-01	 	 	 	 6.405e+09	 	 1.03e+09	 	 14.36	 GB/sec----------------+----------+----------------------------------------+----------------------------

Per	 Process	 byte	 sections	 	 	 	 2.232e+00	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 3.401e+10	 	 	 	 	 	 	 	 	 	 	 	 15.23	 GB/sec----------------+----------+----------------------------------------+----------------------------

Job	 Total	 byte	 sections	 	 	 	 	 	 2.232e+00	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 6.802e+10	 	 	 	 	 	 	 	 	 	 	 	 30.47	 GB/sec

example/test1のHWPC利用結果例(2)	
#	 PMlib	 Process	 Report	 ---	 Elapsed	 time	 for	 individual	 MPI	 ranks	 ------

Label	 	 First	 location

Header	 ID	 	 :	 	 	 	 	 call	 	 	 time[s]	 time[%]	 	 t_wait[s]	 	 t[s]/call	 	 	 flop|msg	 	 	 	 speed

Rank	 	 	 	 	 0	 :	 	 	 	 	 	 	 	 3	 	 1.341e+00	 	 	 60.1	 	 0.000e+00	 	 4.471e-01	 	 2.221e+10	 	 1.656e+10	 Bytes/s	 (HWPC)

Rank	 	 	 	 	 1	 :	 	 	 	 	 	 	 	 3	 	 1.339e+00	 	 	 60.0	 	 2.689e-03	 	 4.462e-01	 	 1.922e+10	 	 1.436e+10	 Bytes/s	 (HWPC)

Label	 	 Second	 location

Header	 ID	 	 :	 	 	 	 	 call	 	 	 time[s]	 time[%]	 	 t_wait[s]	 	 t[s]/call	 	 	 flop|msg	 	 	 	 speed

Rank	 	 	 	 	 0	 :	 	 	 	 	 	 	 	 1	 	 4.463e-01	 	 	 20.0	 	 0.000e+00	 	 4.463e-01	 	 7.132e+09	 	 1.598e+10	 Bytes/s	 (HWPC)

Rank	 	 	 	 	 1	 :	 	 	 	 	 	 	 	 1	 	 4.455e-01	 	 	 20.0	 	 7.200e-04	 	 4.455e-01	 	 5.679e+09	 	 1.275e+10	 Bytes/s	 (HWPC)

Label	 	 Third	 location

Header	 ID	 	 :	 	 	 	 	 call	 	 	 time[s]	 time[%]	 	 t_wait[s]	 	 t[s]/call	 	 	 flop|msg	 	 	 	 speed

Rank	 	 	 	 	 0	 :	 	 	 	 	 	 	 	 1	 	 4.470e-01	 	 	 20.0	 	 0.000e+00	 	 4.470e-01	 	 7.537e+09	 	 1.686e+10	 Bytes/s	 (HWPC)

Rank	 	 	 	 	 1	 :	 	 	 	 	 	 	 	 1	 	 4.461e-01	 	 	 20.0	 	 9.751e-04	 	 4.461e-01	 	 6.233e+09	 	 1.397e+10	 Bytes/s	 (HWPC)#	 PMlib	 hardware	 performance	 counter	 (HWPC)	 Report	 -------------------------

	 	 	 HWPC_CHOOSER=BANDWIDTH	 statistics	 were	 collected.

	 	 	 The	 values	 of	 each	 process	 are	 the	 sum	 of	 threads.

Label	 	 	 First	 location

Header	 	 ID	 :	 	 	 	 	 L2_TCM	 	 	 L2_WB_DM	 	 	 L2_WB_PF	 	 	 [HW	 B/s]

Rank	 	 	 	 	 0	 :	 	 1.732e+08	 	 6.850e+02	 	 3.898e+05	 	 1.656e+10

Rank	 	 	 	 	 1	 :	 	 1.500e+08	 	 2.080e+02	 	 1.527e+05	 	 1.436e+10

Label	 	 	 Second	 location

Header	 	 ID	 :	 	 	 	 	 L2_TCM	 	 	 L2_WB_DM	 	 	 L2_WB_PF	 	 	 [HW	 B/s]

Rank	 	 	 	 	 0	 :	 	 5.567e+07	 	 2.800e+01	 	 4.718e+04	 	 1.598e+10

Rank	 	 	 	 	 1	 :	 	 4.433e+07	 	 2.700e+01	 	 3.832e+04	 	 1.275e+10

Label	 	 	 Third	 location

Header	 	 ID	 :	 	 	 	 	 L2_TCM	 	 	 L2_WB_DM	 	 	 L2_WB_PF	 	 	 [HW	 B/s]

例題プログラムでHWPCを利用 京コンピュータ(3)	
• ジョブスクリプト x.run-test1.shを編集	
K$ cat x.run-test1.sh#!/bin/bash#PJM -N MYTEST1#PJM --rsc-list "elapse=1:00:00"#PJM --rsc-list "node=1"#PJM --mpi "proc=2”#PJM -j#PJM -S# stage io files#PJM --stg-transfiles all#PJM --mpi "use-rankdir"#PJM --stgin-basedir "/home/ra000004/a03155/pmlib/PMlib/BUILD_DIR/example"#PJM --stgin "rank=* test1/test1 %r:./test1"#source /work/system/Env_base/opt/FJSVXosPA/bin/xospastop
export OMP_NUM_THREADS=2 NPROCS=2
export HWPC_CHOOSER=VECTOR
mpiexec -n ${NPROCS} ./test1
K$ pjsub x.run-test1.sh

example/test1のHWPC利用結果例(3)	
	 PMlib	 Basic	 Report	 -------------------------------------------------------

Timing	 Statistics	 Report	 from	 PMlib	 version	 4.1.2

Linked	 PMlib	 supports:	 MPI,	 OpenMP,	 HWPC

Host	 name	 :	 g05-036

Date	 	 	 	 	 	 :	 2015/08/26	 :	 02:11:00

Mr.	 Bean

Parallel	 Mode:	 	 	 Hybrid	 (2	 processes	 x	 2	 threads)

Total	 execution	 time	 	 	 	 	 	 	 	 	 	 	 	 =	 2.241645e+00	 [sec]Total	 time	 of	 measured	 sections	 =	 2.234203e+00	 [sec]Exclusive	 sections	 Statistics	 per	 process	 and	 per	 job.

Section	 	 	 	 	 	 	 	 	 |	 	 	 	 call	 	 |	 	 	 	 	 	 	 	 accumulated	 time[sec]	 	 	 Label	 	 	 	 	 	 	 	 	 	 	 |	 	 	 	 	 	 	 	 	 	 |	 	 	 	 	 	 avr	 	 	 avr[%]	 	 	 	 	 	 sdv	 	 	 avr/call	 	 ----------------+----------+----------------------------------------First	 location	 	 :	 	 	 	 	 	 	 	 3	 	 	 1.341e+00	 	 60.02	 	 1.79e-03	 	 4.470e-01

Third	 location	 	 :	 	 	 	 	 	 	 	 1	 	 	 4.471e-01	 	 20.01	 	 5.45e-04	 	 4.471e-01

Second	 location	 :	 	 	 	 	 	 	 	 1	 	 	 4.461e-01	 	 19.97	 	 3.57e-04	 	 4.461e-01----------------+----------+--------------------------------------------------------+----------+----------------------------------------

example/test1のHWPC利用結果例(3)	
#	 PMlib	 Process	 Report	 ---	 Elapsed	 time	 for	 individual	 MPI	 ranks	 ------

Label	 	 First	 location

Header	 ID	 	 :	 	 	 	 	 call	 	 	 time[s]	 time[%]	 	 t_wait[s]	 	 t[s]/call

Rank	 	 	 	 	 0	 :	 	 	 	 	 	 	 	 3	 	 1.342e+00	 	 	 60.1	 	 0.000e+00	 	 4.474e-01

Rank	 	 	 	 	 1	 :	 	 	 	 	 	 	 	 3	 	 1.340e+00	 	 	 60.0	 	 2.531e-03	 	 4.466e-01

Label	 	 Second	 location

Header	 ID	 	 :	 	 	 	 	 call	 	 	 time[s]	 time[%]	 	 t_wait[s]	 	 t[s]/call

Rank	 	 	 	 	 0	 :	 	 	 	 	 	 	 	 1	 	 4.464e-01	 	 	 20.0	 	 0.000e+00	 	 4.464e-01

Rank	 	 	 	 	 1	 :	 	 	 	 	 	 	 	 1	 	 4.459e-01	 	 	 20.0	 	 5.050e-04	 	 4.459e-01

Label	 	 Third	 location

Header	 ID	 	 :	 	 	 	 	 call	 	 	 time[s]	 time[%]	 	 t_wait[s]	 	 t[s]/call

Rank	 	 	 	 	 0	 :	 	 	 	 	 	 	 	 1	 	 4.475e-01	 	 	 20.0	 	 0.000e+00	 	 4.475e-01

Rank	 	 	 	 	 1	 :	 	 	 	 	 	 	 	 1	 	 4.468e-01	 	 	 20.0	 	 7.710e-04	 	 4.468e-01#	 PMlib	 hardware	 performance	 counter	 (HWPC)	 Report	 -------------------------

	 	 	 HWPC_CHOOSER=VECTOR	 statistics	 were	 collected.

	 	 	 The	 values	 of	 each	 process	 are	 the	 sum	 of	 threads.

Label	 	 	 First	 location

Header	 	 ID	 :	 	 	 	 VEC_INS	 	 	 	 FMA_INS

Rank	 	 	 	 	 0	 :	 	 4.761e+09	 	 1.316e+09

Rank	 	 	 	 	 1	 :	 	 4.761e+09	 	 1.316e+09

Label	 	 	 Second	 location

Header	 	 ID	 :	 	 	 	 VEC_INS	 	 	 	 FMA_INS

Rank	 	 	 	 	 0	 :	 	 1.581e+09	 	 4.350e+08

Rank	 	 	 	 	 1	 :	 	 1.581e+09	 	 4.350e+08

Label	 	 	 Third	 location

Header	 	 ID	 :	 	 	 	 VEC_INS	 	 	 	 FMA_INSexample/test1のHWPC利用結果例(others)	
#	 PMlib	 hardware	 performance	 counter	 (HWPC)	 Report	 -------------------------

	 	 	 HWPC_CHOOSER=CYCLE,INSTRUCTION	 statistics	 were	 collected.

Label	 	 	 First	 location

Header	 	 ID	 :	 	 	 	 TOT_CYC	 	 	 	 TOT_INS	 	 	 	 	 LD_INS	 	 	 	 	 SR_INS

Rank	 	 	 	 	 0	 :	 	 5.363e+09	 	 1.194e+10	 	 6.007e+09	 	 1.803e+07

Rank	 	 	 	 	 1	 :	 	 5.355e+09	 	 1.194e+10	 	 6.007e+09	 	 1.803e+07

Label	 	 	 Second	 location#	 PMlib	 hardware	 performance	 counter	 (HWPC)	 Report	 -------------------------

	 	 	 HWPC_CHOOSER=CACHE	 statistics	 were	 collected.

Label	 	 	 First	 location

Header	 	 ID	 :	 	 	 	 	 L1_TCM	 	 	 	 	 L2_TCM

Rank	 	 	 	 	 0	 :	 	 2.431e+08	 	 1.668e+08

Rank	 	 	 	 	 1	 :	 	 2.431e+08	 	 1.490e+08

Label	 	 	 Second	 locationクイズ：APIへの引数再確認	
• 出力レポートを別ファイルに出力するには？

• HWPCのlegendをプリントするには？

• 計測区間を追加するには？

本日はお疲れさまでした