AWSのEPYCインスタンスをHPCで使えるか検証してみた

# EPYCインスタンスが来たぞー

というわけで、「AWSでEPYCでHPCがしたい！」という全世界のユーザーは歓喜したとかなんとか(ほんとか？
なので、EPYCインスタンスのベンチマークを取ることにします。

https://qiita.com/telmin_orca/items/bea941314b8051793db2
の続き。Ryzenを3世代にわたって適当にベンチ取った記録。

## ハードウェア

/proc/cpuinfo

```
processor       : 0
vendor_id       : AuthenticAMD
cpu family      : 23
model           : 1
model name      : AMD EPYC 7571
stepping        : 2
microcode       : 0x8001227
cpu MHz         : 2598.604
cache size      : 512 KB
physical id     : 0
siblings        : 48
core id         : 0
cpu cores       : 24
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid amd_dcm aperfmperf pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch topoext perfctr_core vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr arat npt nrip_save
bugs            : sysret_ss_attrs null_seg spectre_v1 spectre_v2 spec_store_bypass
bogomips        : 4399.32
TLB size        : 2560 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 48 bits physical, 48 bits virtual
power management:

```

EPYCはMCMですね！というわけで、パッケージ内もNUMAというなんかもーややこしいなぁという構造をしているわけで…はぁ

```
$ numactl -H
available: 3 nodes (0-2)
node 0 cpus: 0 1 2 3 4 5 6 7 24 25 26 27 28 29 30 31
node 0 size: 63310 MB
node 0 free: 62715 MB
node 1 cpus: 8 9 10 11 12 13 14 15 32 33 34 35 36 37 38 39
node 1 size: 63374 MB
node 1 free: 62707 MB
node 2 cpus: 16 17 18 19 20 21 22 23 40 41 42 43 44 45 46 47
node 2 size: 63352 MB
node 2 free: 62892 MB
node distances:
node   0   1   2
  0:  10  16  16
  1:  16  10  16
  2:  16  16  10

```

3クラスタなのかい。なんかきもちわるい

## ベンチマーク
### STREAM

なんもコントロールしない版

```
$ ./stream_cxx.out -s 1G
-------------------------------------------------------------
STREAM version $Revision : 5.10 $
-------------------------------------------------------------
This system uses 8 bytes per array element.
-------------------------------------------------------------
Array Size = 1073741824 (elements), Offset = 0 (elements)
Memory per array = 8192 MiB (= 8 GiB).
Total Memory required = 24576 MiB (= 24 GiB).
Each kernel will be executed 10 times.
Function    Best Rate MB/s  Avg time     Min time     Max time
Copy:           30452.7     0.737589     0.564149     1.488491
Scale:          31323.2     0.691516     0.548470     1.135745
Add:            32674.3     0.912753     0.788686     1.135196
Triad:          30949.2     0.919351     0.832648     1.123737
-------------------------------------------------------------
```

numactlでlocalallocした版

```
$ numactl --localalloc ./stream_cxx.out -s 1G
-------------------------------------------------------------
STREAM version $Revision : 5.10 $
-------------------------------------------------------------
This system uses 8 bytes per array element.
-------------------------------------------------------------
Array Size = 1073741824 (elements), Offset = 0 (elements)
Memory per array = 8192 MiB (= 8 GiB).
Total Memory required = 24576 MiB (= 24 GiB).
Each kernel will be executed 10 times.
Function    Best Rate MB/s  Avg time     Min time     Max time
Copy:           35016.2     0.907369     0.490627     3.395209
Scale:          34924.1     1.116415     0.491921     5.284777
Add:            36964.8     0.988101     0.697144     1.739983
Triad:          40281.4     0.772501     0.639745     1.186238
-------------------------------------------------------------
```

numactlでinterleave=allした版

```
$ numactl --interleave=all ./stream_cxx.out -s 1G
-------------------------------------------------------------
STREAM version $Revision : 5.10 $
-------------------------------------------------------------
This system uses 8 bytes per array element.
-------------------------------------------------------------
Array Size = 1073741824 (elements), Offset = 0 (elements)
Memory per array = 8192 MiB (= 8 GiB).
Total Memory required = 24576 MiB (= 24 GiB).
Each kernel will be executed 10 times.
Function    Best Rate MB/s  Avg time     Min time     Max time
Copy:           51429.9     0.336871     0.334044     0.343149
Scale:          51399.8     0.335059     0.334240     0.339794
Add:            55332.8     0.468382     0.465724     0.474934
Triad:          55190.6     0.469062     0.466924     0.475884
-------------------------------------------------------------
```

おお、すげえ伸びた

24スレッドのみ(=1物理コアに1スレッド割り当て) で、numactlで適当にバインドしてみると？

```
$ OMP_NUM_THREADS=24 numactl --cpubind=0,1,2 --membind=0,1,2 ./stream_cxx.out -s 1G
-------------------------------------------------------------
STREAM version $Revision : 5.10 $
-------------------------------------------------------------
This system uses 8 bytes per array element.
-------------------------------------------------------------
Array Size = 1073741824 (elements), Offset = 0 (elements)
Memory per array = 8192 MiB (= 8 GiB).
Total Memory required = 24576 MiB (= 24 GiB).
Each kernel will be executed 10 times.
Function    Best Rate MB/s  Avg time     Min time     Max time
Copy:           18458.0     0.935805     0.930755     0.942491
Scale:          18457.4     0.937502     0.930783     0.945430
Add:            19904.0     1.302825     1.294708     1.312583
Triad:          19975.5     1.305319     1.290069     1.317308
-------------------------------------------------------------
```

なんか遅くなった。アフィニティちゃんと設定しないとダメか？

24スレッドでinterleave=allしてみると…

```
$ OMP_NUM_THREADS=24 numactl --interleave=all ./stream_cxx.out -s 1G
-------------------------------------------------------------
STREAM version $Revision : 5.10 $
-------------------------------------------------------------
This system uses 8 bytes per array element.
-------------------------------------------------------------
Array Size = 1073741824 (elements), Offset = 0 (elements)
Memory per array = 8192 MiB (= 8 GiB).
Total Memory required = 24576 MiB (= 24 GiB).
Each kernel will be executed 10 times.
Function    Best Rate MB/s  Avg time     Min time     Max time
Copy:           51883.6     0.334359     0.331124     0.352611
Scale:          50945.1     0.342290     0.337223     0.363494
Add:            55125.5     0.469205     0.467475     0.473028
Triad:          55026.9     0.472796     0.468313     0.504610
-------------------------------------------------------------
```

大して変わらん。
ということは、ちゃんと引けているんだな…なるほど…

### FFT
3次元FFT。

何もしない版

```
$ ./fftw
3-D FFT: 128 x 128 x 128
On-board: 0.005765 msec, 38.195858 GFLOPS.
On-board: 0.003017 msec, 72.975774 GFLOPS.
3-D FFT: 256 x 256 x 256
On-board: 0.064630 msec, 31.150834 GFLOPS.
On-board: 0.061369 msec, 32.806107 GFLOPS.
```

なんかクソ速いな

24スレッド版

```
$ OMP_NUM_THREADS=24 ./fftw
3-D FFT: 128 x 128 x 128
On-board: 0.004932 msec, 44.643677 GFLOPS.
On-board: 0.003317 msec, 66.378631 GFLOPS.
3-D FFT: 256 x 256 x 256
On-board: 0.099970 msec, 20.138779 GFLOPS.
On-board: 0.086103 msec, 23.381948 GFLOPS.
```

ちょっと落ちた

numactl --interleave=all入れてみた

```
$ numactl --interleave=all ./fftw
3-D FFT: 128 x 128 x 128
On-board: 0.005450 msec, 40.403388 GFLOPS.
On-board: 0.003117 msec, 70.648559 GFLOPS.
3-D FFT: 256 x 256 x 256
On-board: 0.064362 msec, 31.280534 GFLOPS.
On-board: 0.065973 msec, 30.516550 GFLOPS.
```

うーん？

```
$ OMP_NUM_THREADS=24 numactl --interleave=all ./fftw
3-D FFT: 128 x 128 x 128
On-board: 0.005006 msec, 43.985406 GFLOPS.
On-board: 0.003303 msec, 66.665421 GFLOPS.
3-D FFT: 256 x 256 x 256
On-board: 0.085201 msec, 23.629514 GFLOPS.
On-board: 0.070042 msec, 28.743652 GFLOPS.
```

あんまり変わらん…

一応念のため

```
$ numactl --localalloc ./fftw
3-D FFT: 128 x 128 x 128
On-board: 0.005704 msec, 38.601561 GFLOPS.
On-board: 0.003038 msec, 72.470349 GFLOPS.
3-D FFT: 256 x 256 x 256
On-board: 0.063662 msec, 31.624237 GFLOPS.
On-board: 0.063053 msec, 31.929787 GFLOPS.
```

特に有意な差っていうのはないなぁ…そんなもんか

### DGEMM, SGEMM

GEMMはどうなるだろう。
きっと、ThreadRipperの傾向をみるに、32スレッドでは性能が結構落ちるのだろう。まぁ、最内ループは命令ほぼ埋まってて、論理スレッドは意味なくなるはずだからそれは正しいのだが。

まずはなにもしないで

```
$ ./gemms
M 10240 N 10240 K 10240 al -1 b 1
Dgemm start
memory use 2.34375 GB
1:20:32 initialize done.
238.255 GFlops, 89.1692
Sgemm start
1:21:7 initialize done.
727.395 GFlops, 99.035
```

24スレッドにしてみて。

```
$ OMP_NUM_THREADS=24 ./gemms
M 10240 N 10240 K 10240 al -1 b 1
Dgemm start
memory use 2.34375 GB
1:22:28 initialize done.
201.881 GFlops, 22.8572
Sgemm start
1:23:2 initialize done.
693.773 GFlops, 6.97528
```

いや下がっとるやないかーい！(輝夜月風)

numactlでinterleave=allかけてみて

```
$ OMP_NUM_THREADS=24 numactl --interleave=all ./gemms
M 10240 N 10240 K 10240 al -1 b 1
Dgemm start
memory use 2.34375 GB
1:24:54 initialize done.
338.489 GFlops, 0.915681
Sgemm start
1:25:28 initialize done.
706.782 GFlops, 0.809519
```

結構マシになった。あと、分散も大幅に減ったので、やはりnuma制御は必須か…ううむ。めんどくさい

あと、まさかとは思うけど…まさかとは思うけど…48スレッドでも確認してみよう。

```
$ numactl --interleave=all ./gemms
M 10240 N 10240 K 10240 al -1 b 1
Dgemm start
memory use 2.34375 GB
1:26:35 initialize done.
391.78 GFlops, 1.15513
Sgemm start
1:27:10 initialize done.
814.027 GFlops, 5.30401
```

えぇ…(ドン引き


### HPL
```
$ ./lu2 -n 32768
main top omp_max_threads=48 procs=48
optchar = n optarg=32768
N=32768 Seed=1 NB=2048 usehuge=0
Board id=0 # boards=1
N=32768 Seed=1 NB=2048 usehuge=0
Board id=0 # boards=1
read/set mat end
copy mat end
before lumcolumn omp_max_threads=48 procs=48
Emax=  1.618e-05
Nswap=0 cpsec =  11626.1 wsec=407.86 57.5105 Gflops
swaprows    time=     5.35257e+09 ops/cycle=0.100302
scalerow    time=      1.5302e+08 ops/cycle=3.5085
trans rtoc  time=     9.99468e+10 ops/cycle=0.00537157
trans ctor  time=     7.00587e+10 ops/cycle=0.00766316
trans mmul  time=     4.07757e+11 ops/cycle=0.0197496
tr nr  cdec time=     4.99821e+09 ops/cycle=0.107413
trans vvmul time=     1.90071e+09 ops/cycle=0.282458
trans findp time=     3.06734e+09 ops/cycle=0.175028
solve tri u time=     1.55983e+10 ops/cycle=2.10074e-06
solve tri   time=     2.93913e+10 ops/cycle=74.819
matmul nk8  time=               0 ops/cycle=inf
matmul snk  time=     4.19544e+07 ops/cycle=204.745
trans mmul8 time=     1.51567e+11 ops/cycle=0.028337
trans mmul4 time=      2.5248e+11 ops/cycle=0.00850557
trans mmul2 time=     3.70726e+09 ops/cycle=0.289632
DGEMM2K     time=     1.90254e+11 ops/cycle=117.51
DGEMM1K     time=     1.09285e+10 ops/cycle=50.3048
DGEMM512    time=     7.84843e+09 ops/cycle=35.0233
DGEMMrest   time=     8.24044e+10 ops/cycle=3.33572
col dec t   time=     5.83185e+11 ops/cycle=0.0294587
Total       time=     8.93463e+11 ops/cycle=26.2532
```

50て。

```
$ numactl --interleave=all ./lu2 -n 32768
main top omp_max_threads=48 procs=48
optchar = n optarg=32768
N=32768 Seed=1 NB=2048 usehuge=0
Board id=0 # boards=1
N=32768 Seed=1 NB=2048 usehuge=0
Board id=0 # boards=1
read/set mat end
copy mat end
before lumcolumn omp_max_threads=48 procs=48
Emax=  1.618e-05
Nswap=0 cpsec =  10791.6 wsec=346.712 67.6534 Gflops
swaprows    time=     1.42583e+09 ops/cycle=0.376533
scalerow    time=     1.82128e+08 ops/cycle=2.94777
trans rtoc  time=     9.52548e+10 ops/cycle=0.00563616
trans ctor  time=     5.13747e+10 ops/cycle=0.0104501
trans mmul  time=     3.89029e+11 ops/cycle=0.0207004
tr nr  cdec time=     5.81692e+09 ops/cycle=0.0922947
trans vvmul time=     2.36237e+09 ops/cycle=0.227259
trans findp time=     3.43785e+09 ops/cycle=0.156165
solve tri u time=     8.44437e+09 ops/cycle=3.88045e-06
solve tri   time=     2.34857e+10 ops/cycle=93.6326
matmul nk8  time=               0 ops/cycle=inf
matmul snk  time=     1.19951e+06 ops/cycle=7161.23
trans mmul8 time=     1.44905e+11 ops/cycle=0.02964
trans mmul4 time=     2.40618e+11 ops/cycle=0.00892485
trans mmul2 time=      3.5034e+09 ops/cycle=0.306486
DGEMM2K     time=     1.27428e+11 ops/cycle=175.446
DGEMM1K     time=     7.71364e+09 ops/cycle=71.2706
DGEMM512    time=      6.1085e+09 ops/cycle=44.9992
DGEMMrest   time=      7.1944e+10 ops/cycle=3.82072
col dec t   time=     5.41751e+11 ops/cycle=0.0317118
Total       time=     7.61303e+11 ops/cycle=30.8107
```

いや、60てｗ

```
$ OMP_NUM_THREADS=24 numactl --interleave=all ./lu2 -n 32768
main top omp_max_threads=24 procs=48
optchar = n optarg=32768
N=32768 Seed=1 NB=2048 usehuge=0
Board id=0 # boards=1
N=32768 Seed=1 NB=2048 usehuge=0
Board id=0 # boards=1
read/set mat end
copy mat end
before lumcolumn omp_max_threads=24 procs=48
Emax=  6.855e-06
Nswap=0 cpsec =  1752.89 wsec=78.1302 300.22 Gflops
swaprows    time=      1.2731e+09 ops/cycle=0.421703
scalerow    time=     1.54877e+08 ops/cycle=3.46642
trans rtoc  time=     1.15648e+09 ops/cycle=0.46423
trans ctor  time=     7.43654e+08 ops/cycle=0.721936
trans mmul  time=     1.97772e+09 ops/cycle=4.07189
tr nr  cdec time=     1.02064e+09 ops/cycle=0.526014
trans vvmul time=     3.40386e+08 ops/cycle=1.57724
trans findp time=     6.78659e+08 ops/cycle=0.791077
solve tri u time=     3.49471e+09 ops/cycle=9.37646e-06
solve tri   time=     1.67429e+10 ops/cycle=131.341
matmul nk8  time=               0 ops/cycle=inf
matmul snk  time=          618574 ops/cycle=13886.7
trans mmul8 time=     7.82076e+08 ops/cycle=5.49175
trans mmul4 time=      6.9772e+08 ops/cycle=3.07786
trans mmul2 time=      4.9601e+08 ops/cycle=2.16476
DGEMM2K     time=     1.38579e+11 ops/cycle=161.328
DGEMM1K     time=     5.73259e+09 ops/cycle=95.9
DGEMM512    time=     3.96182e+09 ops/cycle=69.3817
DGEMMrest   time=     1.19867e+10 ops/cycle=22.9319
col dec t   time=      4.9462e+09 ops/cycle=3.47335
Total       time=     1.70473e+11 ops/cycle=137.595
```

いきなり300になるなよｗｗ

```
$ OMP_NUM_THREADS=24 numactl --interleave=all ./lu2 -n 65536
main top omp_max_threads=24 procs=48
optchar = n optarg=65536
N=65536 Seed=1 NB=2048 usehuge=0
Board id=0 # boards=1
N=65536 Seed=1 NB=2048 usehuge=0
Board id=0 # boards=1
read/set mat end
copy mat end
before lumcolumn omp_max_threads=24 procs=48
Emax=  1.712e-03
Nswap=0 cpsec =  12019.3 wsec=555.36 337.889 Gflops
swaprows    time=     3.90339e+09 ops/cycle=0.550159
scalerow    time=     3.09347e+08 ops/cycle=6.94199
trans rtoc  time=     4.34846e+09 ops/cycle=0.493849
trans ctor  time=     3.12601e+09 ops/cycle=0.686973
trans mmul  time=     6.34419e+09 ops/cycle=5.07745
tr nr  cdec time=     4.45475e+09 ops/cycle=0.482066
trans vvmul time=     1.32924e+09 ops/cycle=1.61558
trans findp time=       3.121e+09 ops/cycle=0.688075
solve tri u time=     6.51899e+09 ops/cycle=1.00531e-05
solve tri   time=     6.40876e+10 ops/cycle=137.251
matmul nk8  time=               0 ops/cycle=inf
matmul snk  time=          607926 ops/cycle=56519.6
trans mmul8 time=     2.63849e+09 ops/cycle=6.51124
trans mmul4 time=     1.78724e+09 ops/cycle=4.80627
trans mmul2 time=     1.91419e+09 ops/cycle=2.24375
DGEMM2K     time=     1.10841e+12 ops/cycle=165.329
DGEMM1K     time=     1.93481e+10 ops/cycle=113.655
DGEMM512    time=     1.39368e+10 ops/cycle=78.893
DGEMMrest   time=     4.25798e+10 ops/cycle=25.8224
col dec t   time=     1.83769e+10 ops/cycle=3.73944
Total       time=     1.21635e+12 ops/cycle=154.273
```

行列を大きくすると結果が伸びる。ということは、もう少し大きくするともっと良くなる？
まぁHPLってそういうもんだけど、無限に時間がかかる=俺の財布が死ぬのでやりたくない


## まとめ

さて、珍しく途中で計算してなかったので、ここでピーク性能を計算すると、~~~AWSのところに表示されていたクロックは2GHzだったので~~~
** 2018/11/09修正。これ、僕の目が腐ってただけで、公式に2.5GHzが正しいです。なので、下の式をみてください **

```
24(Core) * 2(clock) * 2(SIMD) * 2(FMA) * 2(port) = 384GFlops/DP
```


```
24(Core) * 2.5(clock) * 2(SIMD) * 2(FMA) * 2(port) = 480GFlops/DP
```


仮に480だとすると、DGEMM, SGEMMの実効効率は、それぞれ

`391.78 / 480 = 81.6%`
`814.027 / 960 = 84.7%`

と、なんかまずまずな数字に見えます。

HPLでは

`337.8 / 480 = 70.3%`

になってしまうので、なんかもう少し欲しいなぁという感じでしょうか。
ほかの実装ならもう少し効率がいいとかあるのかもしれません。他のも試してみたいところですね。ただまぁ、HPLってめっちゃ時間かかるので、ほんと財布にやさしくない…

メモリ帯域は、AWSで使っているSKUの情報があんまりないのであれですが、似たようなSKUから推察するに、DDR4の8chっぽいですね。
https://en.wikipedia.org/wiki/Epyc

だとすると、6chのSkylakeに負けているような…気が…しないでも…ないような…気がする…

次はSkylakeあたりでちゃんとベンチとるべきかもしれませんね…
いや、その前に別のEPYC環境で追試しとくべきか。