Pythonプログラム(BenchmarkerやPyBenchなど)をAARCH64とx86で動かしてみる。

# 1.はじめに
PythonのBenchmarkerのサンプルコードをちょっと流してみた。(AWS `m6g.large`/`r4.large`)(2コア)(AARCH64は、AWS Graviton 2)
後半では、別のベンチマーク(Pythonのループ処理)も取った。
OSは、Ubuntu 20.04(aarch64/intel), Amazon Linux 2 20200520(aarch64)で行った。Amazon Linux 2は、CentOS8に近い構成なのでその比較として行った。なお、Graviton 2(m6g)がもとにしているNeoverseN1用のコード出力は、GCC9以降から対応している。このため、Amazon Linux 2は、GCC7であり、あまり性能が期待できない。



## 1.1.手順

```bash
sudo apt update
sudo apt install python3-pip
sudo pip3 install Benchmarker
python3 example.py -o result.json
```
なお、example.pyは、Benchmarkerの頁に記載されているもの

```python
from benchmarker import Benchmarker

## specify number of loop
with Benchmarker(1000*1000, width=20) as bench:
    s1, s2, s3, s4, s5 = "Haruhi", "Mikuru", "Yuki", "Itsuki", "Kyon"

    @bench(None)                ## empty loop
    def _(bm):
        for i in bm:
            pass

    @bench("join")
    def _(bm):
        for i in bm:
            sos = ''.join((s1, s2, s3, s4, s5))

    @bench("concat")
    def _(bm):
        for i in bm:
            sos = s1 + s2 + s3 + s4 + s5

    @bench("format")
    def _(bm):
        for i in bm:
            sos = '%s%s%s%s%s' % (s1, s2, s3, s4, s5)
```


## 1.2.参考資料
- [Benchmarker 4.0.1](https://pypi.org/project/Benchmarker/4.0.1/)


## 1.3.AARCH64環境

m6g.large(AARCH64) Ubuntu 20.04で確認した。

```bash
$ python3 example.py -o result.json
## benchmarker:         release 4.0.1 (for python)
## python version:      3.8.2
## python compiler:     GCC 9.3.0
## python platform:     Linux-5.4.0-1009-aws-aarch64-with-glibc2.29
## python executable:   /usr/bin/python3
## cpu model:           -
## parameters:          loop=1000000, cycle=1, extra=0

##                        real    (total    = user    + sys)
(Empty)                 0.0237    0.0200    0.0200    0.0000
join                    0.1849    0.1900    0.1900    0.0000
concat                  0.2015    0.2100    0.2100    0.0000
format                  0.2611    0.2600    0.2600    0.0000

## Ranking                real
join                    0.1849  (100.0) ********************
concat                  0.2015  ( 91.8) ******************
format                  0.2611  ( 70.8) **************

## Matrix                 real    [01]    [02]    [03]
[01] join               0.1849   100.0   109.0   141.2
[02] concat             0.2015    91.8   100.0   129.6
[03] format             0.2611    70.8    77.2   100.0
```
### Amazon Linux 2 20200520だと以下の通りであった。

```bash
## benchmarker:         release 4.0.1 (for python)
## python version:      3.7.6
## python compiler:     GCC 7.3.1 20180712 (Red Hat 7.3.1-6)
## python platform:     Linux-4.14.177-139.254.amzn2.aarch64-aarch64-with-glibc2.17
## python executable:   /usr/bin/python3
## cpu model:           -
## parameters:          loop=1000000, cycle=1, extra=0

##                        real    (total    = user    + sys)
(Empty)                 0.0291    0.0300    0.0300    0.0000
join                    0.2316    0.2300    0.2300    0.0000
concat                  0.2481    0.2500    0.2500    0.0000
format                  0.3155    0.3100    0.3100    0.0000

## Ranking                real
join                    0.2316  (100.0) ********************
concat                  0.2481  ( 93.4) *******************
format                  0.3155  ( 73.4) ***************

## Matrix                 real    [01]    [02]    [03]
[01] join               0.2316   100.0   107.1   136.2
[02] concat             0.2481    93.4   100.0   127.2
[03] format             0.3155    73.4    78.6   100.0
```

### A1.mediumだと以下の感じ

```bash
## benchmarker:         release 4.0.1 (for python)
## python version:      3.8.2
## python compiler:     GCC 9.3.0
## python platform:     Linux-5.4.0-1009-aws-aarch64-with-glibc2.29
## python executable:   /usr/bin/python3
## cpu model:           -
## parameters:          loop=1000000, cycle=1, extra=0

##                        real    (total    = user    + sys)
(Empty)                 0.0370    0.0300    0.0300    0.0000
join                    0.3585    0.3700    0.3700    0.0000
concat                  0.3990    0.4100    0.4100    0.0000
format                  0.4804    0.4800    0.4800    0.0000

## Ranking                real
join                    0.3585  (100.0) ********************
concat                  0.3990  ( 89.9) ******************
format                  0.4804  ( 74.6) ***************

## Matrix                 real    [01]    [02]    [03]
[01] join               0.3585   100.0   111.3   134.0
[02] concat             0.3990    89.9   100.0   120.4
[03] format             0.4804    74.6    83.1   100.0
```





## 1.4.インテル版
r4.large Ubuntu 20.04で確認した。


```bash
$ python3 example.py -o result.json
## benchmarker:         release 4.0.1 (for python)
## python version:      3.8.2
## python compiler:     GCC 9.3.0
## python platform:     Linux-5.4.0-1009-aws-x86_64-with-glibc2.29
## python executable:   /usr/bin/python3
## cpu model:           Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz  # 2706.243 MHz
## parameters:          loop=1000000, cycle=1, extra=0

##                        real    (total    = user    + sys)
(Empty)                 0.0189    0.0100    0.0100    0.0000
join                    0.1930    0.2100    0.2100    0.0000
concat                  0.2347    0.2400    0.2400    0.0000
format                  0.2531    0.2600    0.2600    0.0000

## Ranking                real
join                    0.1930  (100.0) ********************
concat                  0.2347  ( 82.2) ****************
format                  0.2531  ( 76.3) ***************

## Matrix                 real    [01]    [02]    [03]
[01] join               0.1930   100.0   121.6   131.1
[02] concat             0.2347    82.2   100.0   107.8
[03] format             0.2531    76.3    92.7   100.0
```

## 1.5.感想
- 空ループ(Empty)/Formatだと、Intelのほうが早い。Join/Concatは、AARCH64のほうが早い。
- Benchmarkerで、ARMもCPU名を記載してほしい。


# 2. Pythonループのテスト
## 2.1. テスト内容
### 2.1.1. デフォルトの性能比較
Python 3.8.2　(Ubuntu 20.04)で以下のコードをテストしてみる。4コアでやってみた。

```python
def main():
   total = 0.0
   for i, _ in enumerate(range(10000)):
      for j, _ in enumerate(range(1, 10000)):
         total += (i * j)
   print("total=",total)

main()
```
また、掛け算の性能差も見てみる。具体的には以下と書き変えて見る。

```python
         total += i # (i * j)
```
グローバル変数の性能影響については、以下で確認してみる。(`total`を関数外で定義)

```Python
total = 0.0
for i, _ in enumerate(range(10000)):
    for j, _ in enumerate(range(1, 10000)):
        total += (i * j)
print("total=",total)
```

### 2.1.2. numbaによる高速化での比較
[numbaを使いjitコンパイルすると高速化する。](https://numba.pydata.org/numba-doc/latest/user/5minguide.html)それを試してみた。
インストール手順

```bash
sudo apt update
sudo apt install python3-pip
sudo pip3 install numba
```

aarch64の場合なぜか以下の手順が追加で必要であった.(ubuntu 20.04 LTSもどきなので修正が必要だったのかも

```bash
sudo pip3 install clang-9
sudo ln -s /usr/bin/llvm-config-9 /usr/bin/llvm-config
sudo pip3 install numba
```

実行コード

```python
from numba import jit

@jit(cache=True)
def main():
   total = 0.0
   for i, _ in enumerate(range(10000)):
      for j, _ in enumerate(range(1, 10000)):
         total += (i * j)
   print("total=",total)

main()
```



## 2.2. AARCH64
`m6g.xlarge`の場合

```bash
total= 2499000124995000.0

real    0m11.507s
user    0m11.507s
sys     0m0.000s
```
掛け算をやめた場合　2.6秒早くなる

```bash
total= 499900005000.0

real    0m8.916s
user    0m8.916s
sys     0m0.000s
```
グローバル変数で操作した場合、1.7倍程度になる。

```
total= 2499000124995000.0

real    0m20.470s
user    0m20.465s
sys     0m0.004s
```

参考までに、numbaでコンパイルした場合、C++の2倍程度となる。
1回目(コンパイルするため若干遅い)

```bash
total= 2499000124995000.0

real    0m0.719s
user    0m0.831s
sys     0m0.074s
```

2回目(バイナリコードを実行するのみ)

```bash
total= 2499000124995000.0

real    0m0.599s
user    0m0.754s
sys     0m0.052s
```


### 2.2.1. AARCH64 with Amazon Linux 2 20200520
ubuntu 20.04に比べ15%ぐらい遅い。

```bash
total= 2499000124995000.0

real    0m13.597s
user    0m13.597s
sys     0m0.000s
```

```bash
total= 499900005000.0

real    0m10.271s
user    0m10.271s
sys     0m0.000s
```
グローバル変数にすると1.7倍ぐらい遅くなる。

```bash
total= 2499000124995000.0

real    0m23.265s
user    0m23.265s
sys     0m0.000s
```


### 2.2.2 AWS A1
a1.medium

```bash
total= 2499000124995000.0

real    0m19.772s
user    0m19.761s
sys     0m0.008s
```

```bash
total= 499900005000.0

real    0m15.026s
user    0m15.024s
sys     0m0.000s
```

```bash
total= 2499000124995000.0

real    0m33.144s
user    0m33.045s
sys     0m0.004s
```

## 2.3. Intel
`r4.xlarge`の場合

```bash
total= 2499000124995000.0

real    0m10.867s
user    0m10.858s
sys     0m0.008s
```
掛け算をやめた場合 3.1秒早くなる。

```bash
total= 499900005000.0

real    0m7.695s
user    0m7.691s
sys     0m0.004s
```
グローバル変数で実行した場合　1.7倍ぐらいになる。

```
total= 2499000124995000.0

real    0m17.823s
user    0m17.824s
sys     0m0.000s
```

参考までに、numbaでコンパイルした場合、C++の2倍程度となる。
1回目(コンパイルするため若干遅い)

```bash
total= 2499000124995000.0

real    0m0.917s
user    0m0.847s
sys     0m0.167s
```

2回目(バイナリコードを実行するのみ)

```bash
total= 2499000124995000.0

real    0m0.778s
user    0m0.731s
sys     0m0.145s
```

## 2.4. 感想
AARCH64とIntelを比較して、演算性能は10％以内で変わらない。

# 3. PyBenchでやってみる。
PyBenchは、[openbenchmarking.org](https://openbenchmarking.org/innhold/f28e7b617e3c4614a76336ed4097fcf6ce94918b)にあるPyBench 1.13をダウンロードして使った。

## 3.1. 手順

```bash
curl http://www.phoronix-test-suite.com/benchmark-files/pybench-2018-02-16.tar.gz --output pybench.tgz
tar xzf pybench.tgz
cd pybench-2018-02-16/
python3 pybench.py Arithmetic.py
python3 pybench.py Strings.py
```
Intelに比べてAARCH64は20%程度遅い

|アーキ|Arithmetics|Strings|
|:--:|:--:|:--:|
|aarch64|1709ms|1718ms|
|intel|1411ms|1397ms|


## 3.2. AARCH64
m6g.large (2コア)の結果

Arithmetic.py(数値演算)の結果

```bash
-------------------------------------------------------------------------------
Benchmark: 2020-06-17 11:10:40
-------------------------------------------------------------------------------

    Rounds: 10
    Warp:   10
    Timer:  time.time

    Machine Details:
       Platform ID:    Linux-5.4.0-1009-aws-aarch64-with-glibc2.29
       Processor:      aarch64

    Python:
       Implementation: CPython
       Executable:     /usr/bin/python3
       Version:        3.8.2
       Compiler:       GCC 9.2.1 20200306
       Bits:           64bit
       Build:          Mar 13 2020 10:14:16 (#default)
       Unicode:        UCS4


Test                             minimum  average  operation  overhead
-------------------------------------------------------------------------------
          BuiltinFunctionCalls:     48ms     49ms    0.10us    0.139ms
           BuiltinMethodLookup:     35ms     35ms    0.03us    0.162ms
                 CompareFloats:     26ms     26ms    0.02us    0.187ms
         CompareFloatsIntegers:     37ms     37ms    0.04us    0.139ms
               CompareIntegers:     40ms     40ms    0.02us    0.281ms
        CompareInternedStrings:     32ms     32ms    0.02us    0.703ms
                  CompareLongs:     23ms     23ms    0.02us    0.163ms
                CompareStrings:     28ms     28ms    0.03us    0.474ms
    ComplexPythonFunctionCalls:     37ms     37ms    0.18us    0.233ms
                 ConcatStrings:     26ms     26ms    0.05us    0.265ms
               CreateInstances:     32ms     33ms    0.29us    0.204ms
            CreateNewInstances:     24ms     25ms    0.30us    0.158ms
       CreateStringsWithConcat:     46ms     46ms    0.05us    0.469ms
                  DictCreation:     23ms     24ms    0.06us    0.186ms
             DictWithFloatKeys:     37ms     37ms    0.04us    0.352ms
           DictWithIntegerKeys:     36ms     36ms    0.03us    0.469ms
            DictWithStringKeys:     34ms     34ms    0.03us    0.473ms
                      ForLoops:     29ms     29ms    1.17us    0.037ms
                    IfThenElse:     33ms     33ms    0.02us    0.351ms
                   ListSlicing:     37ms     37ms    2.65us    0.028ms
                NestedForLoops:     31ms     31ms    0.02us    0.017ms
      NestedListComprehensions:     44ms     44ms    3.70us    0.045ms
          NormalClassAttribute:     44ms     44ms    0.04us    0.239ms
       NormalInstanceAttribute:     39ms     39ms    0.03us    0.239ms
           PythonFunctionCalls:     34ms     34ms    0.10us    0.138ms
             PythonMethodCalls:     38ms     38ms    0.17us    0.075ms
                     Recursion:     53ms     53ms    1.07us    0.234ms
                  SecondImport:      9ms      9ms    0.09us    0.091ms
           SecondPackageImport:     10ms     10ms    0.10us    0.092ms
         SecondSubmoduleImport:     22ms     22ms    0.22us    0.092ms
       SimpleComplexArithmetic:     31ms     31ms    0.04us    0.186ms
        SimpleDictManipulation:     65ms     65ms    0.05us    0.233ms
         SimpleFloatArithmetic:     28ms     28ms    0.02us    0.285ms
      SimpleIntFloatArithmetic:     28ms     28ms    0.02us    0.280ms
       SimpleIntegerArithmetic:     28ms     28ms    0.02us    0.281ms
      SimpleListComprehensions:     37ms     37ms    3.10us    0.045ms
        SimpleListManipulation:     27ms     28ms    0.02us    0.304ms
          SimpleLongArithmetic:     19ms     19ms    0.03us    0.139ms
                    SmallLists:     31ms     31ms    0.05us    0.186ms
                   SmallTuples:     35ms     35ms    0.07us    0.209ms
         SpecialClassAttribute:     46ms     46ms    0.04us    0.239ms
      SpecialInstanceAttribute:     38ms     39ms    0.03us    0.239ms
                StringMappings:     73ms     73ms    0.29us    0.183ms
              StringPredicates:     37ms     37ms    0.05us    0.610ms
                 StringSlicing:     37ms     37ms    0.07us    0.387ms
                     TryExcept:     18ms     18ms    0.01us    0.348ms
                    TryFinally:     24ms     24ms    0.15us    0.184ms
                TryRaiseExcept:     10ms     10ms    0.16us    0.184ms
                  TupleSlicing:     48ms     48ms    0.18us    0.018ms
                   WithFinally:     27ms     27ms    0.17us    0.184ms
               WithRaiseExcept:     26ms     26ms    0.32us    0.231ms
-------------------------------------------------------------------------------
Totals:                           1701ms   1709ms
```

Strings.py(文字列演算)の結果

```bash
-------------------------------------------------------------------------------
Benchmark: 2020-06-17 11:11:10
-------------------------------------------------------------------------------

    Rounds: 10
    Warp:   10
    Timer:  time.time

    Machine Details:
       Platform ID:    Linux-5.4.0-1009-aws-aarch64-with-glibc2.29
       Processor:      aarch64

    Python:
       Implementation: CPython
       Executable:     /usr/bin/python3
       Version:        3.8.2
       Compiler:       GCC 9.2.1 20200306
       Bits:           64bit
       Build:          Mar 13 2020 10:14:16 (#default)
       Unicode:        UCS4


Test                             minimum  average  operation  overhead
-------------------------------------------------------------------------------
          BuiltinFunctionCalls:     50ms     50ms    0.10us    0.141ms
           BuiltinMethodLookup:     36ms     36ms    0.03us    0.165ms
                 CompareFloats:     27ms     27ms    0.02us    0.189ms
         CompareFloatsIntegers:     37ms     37ms    0.04us    0.141ms
               CompareIntegers:     40ms     41ms    0.02us    0.285ms
        CompareInternedStrings:     32ms     32ms    0.02us    0.725ms
                  CompareLongs:     24ms     24ms    0.02us    0.165ms
                CompareStrings:     28ms     28ms    0.03us    0.480ms
    ComplexPythonFunctionCalls:     37ms     37ms    0.18us    0.238ms
                 ConcatStrings:     26ms     26ms    0.05us    0.268ms
               CreateInstances:     33ms     33ms    0.30us    0.206ms
            CreateNewInstances:     25ms     25ms    0.30us    0.159ms
       CreateStringsWithConcat:     46ms     46ms    0.05us    0.471ms
                  DictCreation:     24ms     24ms    0.06us    0.187ms
             DictWithFloatKeys:     37ms     37ms    0.04us    0.353ms
           DictWithIntegerKeys:     36ms     36ms    0.03us    0.471ms
            DictWithStringKeys:     34ms     34ms    0.03us    0.471ms
                      ForLoops:     29ms     29ms    1.17us    0.037ms
                    IfThenElse:     33ms     33ms    0.02us    0.352ms
                   ListSlicing:     37ms     37ms    2.66us    0.028ms
                NestedForLoops:     31ms     31ms    0.02us    0.017ms
      NestedListComprehensions:     44ms     44ms    3.67us    0.046ms
          NormalClassAttribute:     44ms     44ms    0.04us    0.241ms
       NormalInstanceAttribute:     40ms     40ms    0.03us    0.242ms
           PythonFunctionCalls:     33ms     34ms    0.10us    0.143ms
             PythonMethodCalls:     38ms     38ms    0.17us    0.076ms
                     Recursion:     54ms     54ms    1.07us    0.237ms
                  SecondImport:      9ms      9ms    0.09us    0.093ms
           SecondPackageImport:     10ms     10ms    0.10us    0.093ms
         SecondSubmoduleImport:     22ms     22ms    0.22us    0.093ms
       SimpleComplexArithmetic:     32ms     32ms    0.04us    0.190ms
        SimpleDictManipulation:     65ms     65ms    0.05us    0.236ms
         SimpleFloatArithmetic:     29ms     29ms    0.02us    0.285ms
      SimpleIntFloatArithmetic:     28ms     28ms    0.02us    0.284ms
       SimpleIntegerArithmetic:     28ms     28ms    0.02us    0.285ms
      SimpleListComprehensions:     37ms     38ms    3.14us    0.046ms
        SimpleListManipulation:     28ms     28ms    0.02us    0.308ms
          SimpleLongArithmetic:     20ms     20ms    0.03us    0.141ms
                    SmallLists:     30ms     31ms    0.04us    0.189ms
                   SmallTuples:     35ms     36ms    0.07us    0.213ms
         SpecialClassAttribute:     45ms     45ms    0.04us    0.241ms
      SpecialInstanceAttribute:     39ms     39ms    0.03us    0.242ms
                StringMappings:     73ms     73ms    0.29us    0.189ms
              StringPredicates:     37ms     37ms    0.05us    0.619ms
                 StringSlicing:     38ms     38ms    0.07us    0.401ms
                     TryExcept:     18ms     18ms    0.01us    0.361ms
                    TryFinally:     24ms     24ms    0.15us    0.189ms
                TryRaiseExcept:     10ms     10ms    0.16us    0.191ms
                  TupleSlicing:     48ms     48ms    0.18us    0.018ms
                   WithFinally:     27ms     27ms    0.17us    0.189ms
               WithRaiseExcept:     26ms     26ms    0.32us    0.237ms
-------------------------------------------------------------------------------
Totals:                           1711ms   1718ms
```
## 3.3. Intel
m5.largeでやってみた。

Arithmetics.pyのテスト結果は以下の通りである。

```bash
-------------------------------------------------------------------------------
Benchmark: 2020-06-17 11:34:47
-------------------------------------------------------------------------------

    Rounds: 10
    Warp:   10
    Timer:  time.time

    Machine Details:
       Platform ID:    Linux-5.4.0-1009-aws-x86_64-with-glibc2.29
       Processor:      x86_64

    Python:
       Implementation: CPython
       Executable:     /usr/bin/python3
       Version:        3.8.2
       Compiler:       GCC 9.3.0
       Bits:           64bit
       Build:          Mar 13 2020 10:14:16 (#default)
       Unicode:        UCS4


Test                             minimum  average  operation  overhead
-------------------------------------------------------------------------------
          BuiltinFunctionCalls:     41ms     41ms    0.08us    0.104ms
           BuiltinMethodLookup:     26ms     26ms    0.02us    0.123ms
                 CompareFloats:     20ms     20ms    0.02us    0.139ms
         CompareFloatsIntegers:     30ms     31ms    0.03us    0.101ms
               CompareIntegers:     32ms     32ms    0.02us    0.210ms
        CompareInternedStrings:     29ms     29ms    0.02us    0.521ms
                  CompareLongs:     19ms     19ms    0.02us    0.122ms
                CompareStrings:     26ms     28ms    0.03us    0.350ms
    ComplexPythonFunctionCalls:     31ms     31ms    0.16us    0.177ms
                 ConcatStrings:     21ms     21ms    0.04us    0.199ms
               CreateInstances:     25ms     25ms    0.23us    0.157ms
            CreateNewInstances:     19ms     19ms    0.23us    0.121ms
       CreateStringsWithConcat:     43ms     44ms    0.04us    0.350ms
                  DictCreation:     20ms     20ms    0.05us    0.140ms
             DictWithFloatKeys:     32ms     32ms    0.04us    0.269ms
           DictWithIntegerKeys:     27ms     27ms    0.02us    0.355ms
            DictWithStringKeys:     27ms     27ms    0.02us    0.353ms
                      ForLoops:     22ms     22ms    0.89us    0.027ms
                    IfThenElse:     27ms     27ms    0.02us    0.255ms
                   ListSlicing:     28ms     29ms    2.08us    0.022ms
                NestedForLoops:     24ms     24ms    0.02us    0.012ms
      NestedListComprehensions:     35ms     35ms    2.95us    0.035ms
          NormalClassAttribute:     35ms     35ms    0.03us    0.181ms
       NormalInstanceAttribute:     29ms     30ms    0.02us    0.185ms
           PythonFunctionCalls:     25ms     25ms    0.08us    0.107ms
             PythonMethodCalls:     29ms     29ms    0.13us    0.058ms
                     Recursion:     49ms     49ms    0.98us    0.176ms
                  SecondImport:      7ms      7ms    0.07us    0.066ms
           SecondPackageImport:      8ms      8ms    0.08us    0.067ms
         SecondSubmoduleImport:     17ms     17ms    0.17us    0.066ms
       SimpleComplexArithmetic:     24ms     24ms    0.03us    0.133ms
        SimpleDictManipulation:     50ms     51ms    0.04us    0.175ms
         SimpleFloatArithmetic:     26ms     26ms    0.02us    0.209ms
      SimpleIntFloatArithmetic:     26ms     26ms    0.02us    0.210ms
       SimpleIntegerArithmetic:     26ms     26ms    0.02us    0.211ms
      SimpleListComprehensions:     30ms     30ms    2.50us    0.034ms
        SimpleListManipulation:     24ms     25ms    0.02us    0.231ms
          SimpleLongArithmetic:     17ms     17ms    0.03us    0.101ms
                    SmallLists:     26ms     26ms    0.04us    0.134ms
                   SmallTuples:     29ms     29ms    0.05us    0.151ms
         SpecialClassAttribute:     34ms     35ms    0.03us    0.173ms
      SpecialInstanceAttribute:     29ms     29ms    0.02us    0.183ms
                StringMappings:     55ms     55ms    0.22us    0.138ms
              StringPredicates:     28ms     28ms    0.04us    0.526ms
                 StringSlicing:     31ms     31ms    0.06us    0.289ms
                     TryExcept:     18ms     18ms    0.01us    0.254ms
                    TryFinally:     18ms     18ms    0.11us    0.140ms
                TryRaiseExcept:      9ms      9ms    0.14us    0.140ms
                  TupleSlicing:     41ms     56ms    0.21us    0.014ms
                   WithFinally:     20ms     21ms    0.13us    0.139ms
               WithRaiseExcept:     21ms     21ms    0.27us    0.175ms
-------------------------------------------------------------------------------
Totals:                           1384ms   1411ms
```



Strings.pyのテスト結果は以下の通りである。

```bash
-------------------------------------------------------------------------------
Benchmark: 2020-06-17 11:35:05
-------------------------------------------------------------------------------

    Rounds: 10
    Warp:   10
    Timer:  time.time

    Machine Details:
       Platform ID:    Linux-5.4.0-1009-aws-x86_64-with-glibc2.29
       Processor:      x86_64

    Python:
       Implementation: CPython
       Executable:     /usr/bin/python3
       Version:        3.8.2
       Compiler:       GCC 9.3.0
       Bits:           64bit
       Build:          Mar 13 2020 10:14:16 (#default)
       Unicode:        UCS4


Test                             minimum  average  operation  overhead
-------------------------------------------------------------------------------
          BuiltinFunctionCalls:     41ms     42ms    0.08us    0.103ms
           BuiltinMethodLookup:     26ms     26ms    0.03us    0.123ms
                 CompareFloats:     20ms     20ms    0.02us    0.139ms
         CompareFloatsIntegers:     31ms     31ms    0.03us    0.103ms
               CompareIntegers:     32ms     32ms    0.02us    0.212ms
        CompareInternedStrings:     29ms     29ms    0.02us    0.530ms
                  CompareLongs:     19ms     19ms    0.02us    0.118ms
                CompareStrings:     26ms     26ms    0.03us    0.356ms
    ComplexPythonFunctionCalls:     31ms     31ms    0.16us    0.179ms
                 ConcatStrings:     21ms     21ms    0.04us    0.193ms
               CreateInstances:     25ms     26ms    0.23us    0.157ms
            CreateNewInstances:     19ms     19ms    0.23us    0.122ms
       CreateStringsWithConcat:     43ms     43ms    0.04us    0.337ms
                  DictCreation:     20ms     20ms    0.05us    0.141ms
             DictWithFloatKeys:     32ms     32ms    0.04us    0.262ms
           DictWithIntegerKeys:     26ms     27ms    0.02us    0.354ms
            DictWithStringKeys:     25ms     26ms    0.02us    0.350ms
                      ForLoops:     22ms     22ms    0.89us    0.027ms
                    IfThenElse:     27ms     27ms    0.02us    0.255ms
                   ListSlicing:     29ms     29ms    2.09us    0.022ms
                NestedForLoops:     24ms     24ms    0.02us    0.012ms
      NestedListComprehensions:     35ms     36ms    2.97us    0.034ms
          NormalClassAttribute:     35ms     35ms    0.03us    0.185ms
       NormalInstanceAttribute:     29ms     29ms    0.02us    0.183ms
           PythonFunctionCalls:     26ms     26ms    0.08us    0.104ms
             PythonMethodCalls:     30ms     30ms    0.13us    0.058ms
                     Recursion:     49ms     49ms    0.98us    0.176ms
                  SecondImport:      7ms      7ms    0.07us    0.070ms
           SecondPackageImport:      8ms      8ms    0.08us    0.069ms
         SecondSubmoduleImport:     18ms     18ms    0.18us    0.068ms
       SimpleComplexArithmetic:     25ms     25ms    0.03us    0.141ms
        SimpleDictManipulation:     50ms     50ms    0.04us    0.175ms
         SimpleFloatArithmetic:     26ms     26ms    0.02us    0.209ms
      SimpleIntFloatArithmetic:     26ms     26ms    0.02us    0.208ms
       SimpleIntegerArithmetic:     26ms     26ms    0.02us    0.207ms
      SimpleListComprehensions:     30ms     30ms    2.51us    0.034ms
        SimpleListManipulation:     25ms     25ms    0.02us    0.222ms
          SimpleLongArithmetic:     17ms     17ms    0.03us    0.103ms
                    SmallLists:     26ms     27ms    0.04us    0.139ms
                   SmallTuples:     29ms     29ms    0.05us    0.157ms
         SpecialClassAttribute:     34ms     34ms    0.03us    0.183ms
      SpecialInstanceAttribute:     29ms     29ms    0.02us    0.184ms
                StringMappings:     55ms     55ms    0.22us    0.139ms
              StringPredicates:     28ms     28ms    0.04us    0.528ms
                 StringSlicing:     31ms     31ms    0.05us    0.295ms
                     TryExcept:     18ms     18ms    0.01us    0.261ms
                    TryFinally:     18ms     18ms    0.11us    0.141ms
                TryRaiseExcept:     10ms     10ms    0.15us    0.138ms
                  TupleSlicing:     40ms     40ms    0.15us    0.014ms
                   WithFinally:     21ms     21ms    0.13us    0.140ms
               WithRaiseExcept:     22ms     22ms    0.28us    0.178ms
-------------------------------------------------------------------------------
Totals:                           1389ms   1397ms
```

# 4. C++のループで比較
以下のコードをg++ (9.3.0)でコンパイルしてみた。なお、性能差はほぼなし。

```C++
#include <iostream>

int main() {
    float total = 0;
    for (int i = 0; i < 10000; i++) {
        for (int j = 0; j < 10000; j++) {
           total += (float)(i*j);
        }
    }
    std::cout << "total = " << total << std::endl;
    return 0;
}
```

## 4.1 AARCH64 
`m6g.large`の場合

```bash
total = 1.94611e+15

real    0m0.332s
user    0m0.332s
sys     0m0.000s
```

## 4.2 Intel
`t3.medium`の場合

```bash
total = 1.94611e+15

real    0m0.325s
user    0m0.325s
sys     0m0.000s
```

# 5. scikit-learn
## 5.1. aarch64
m6g.xlarge

```
python3 bench_hist_gradient_boosting_higgsboson.py
________________________________________________________________________________
[Memory] Calling __main__--home-ubuntu-scikit-learn-benchmarks-bench_hist_gradient_boosting_higgsboson.load_data...
load_data()
Downloading https://archive.ics.uci.edu/ml/machine-learning-databases/00280/HIGGS.csv.gz to HIGGS.csv.gz (2.6 GB)...
done.
Parsing HIGGS.csv.gz...
Loaded 1.276 GB in 111.853s
______________________________________________________load_data - 355.2s, 5.9min
Training set with 8800000 records with 28 features.
Fitting a sklearn model...
Binning 1.971 GB of training data: 4.977 s
Fitting gradient boosted rounds:
[1/10] 1 tree, 31 leaves, max depth = 7, in 0.649s
[2/10] 1 tree, 31 leaves, max depth = 9, in 0.662s
[3/10] 1 tree, 31 leaves, max depth = 8, in 0.626s
[4/10] 1 tree, 31 leaves, max depth = 10, in 0.672s
[5/10] 1 tree, 31 leaves, max depth = 8, in 0.637s
[6/10] 1 tree, 31 leaves, max depth = 8, in 0.662s
[7/10] 1 tree, 31 leaves, max depth = 11, in 0.666s
[8/10] 1 tree, 31 leaves, max depth = 11, in 0.670s
[9/10] 1 tree, 31 leaves, max depth = 10, in 0.608s
[10/10] 1 tree, 31 leaves, max depth = 10, in 0.659s
Fit 10 trees in 13.241 s, (310 total leaves)
Time spent computing histograms: 3.541s
Time spent finding best splits:  0.020s
Time spent applying splits:      0.609s
Time spent predicting:           0.229s
fitted in 13.249s
predicted in 0.893s, ROC AUC: 0.7951, ACC: 0.7182
```
## 5.2. Intel
r5.xlarge

```
python3 bench_hist_gradient_boosting_higgsboson.py
________________________________________________________________________________
[Memory] Calling __main__--home-ubuntu-scikit-learn-benchmarks-bench_hist_gradient_boosting_higgsboson.load_data...
load_data()
Downloading https://archive.ics.uci.edu/ml/machine-learning-databases/00280/HIGGS.csv.gz to HIGGS.csv.gz (2.6 GB)...
done.
Parsing HIGGS.csv.gz...
Loaded 1.276 GB in 106.308s
______________________________________________________load_data - 347.2s, 5.8min
Training set with 8800000 records with 28 features.
Fitting a sklearn model...
Binning 1.971 GB of training data: 5.313 s
Fitting gradient boosted rounds:
[1/10] 1 tree, 31 leaves, max depth = 7, in 0.920s
[2/10] 1 tree, 31 leaves, max depth = 9, in 0.938s
[3/10] 1 tree, 31 leaves, max depth = 8, in 0.893s
[4/10] 1 tree, 31 leaves, max depth = 10, in 0.924s
[5/10] 1 tree, 31 leaves, max depth = 8, in 0.903s
[6/10] 1 tree, 31 leaves, max depth = 8, in 0.939s
[7/10] 1 tree, 31 leaves, max depth = 11, in 0.943s
[8/10] 1 tree, 31 leaves, max depth = 11, in 0.906s
[9/10] 1 tree, 31 leaves, max depth = 10, in 0.852s
[10/10] 1 tree, 31 leaves, max depth = 10, in 0.914s
Fit 10 trees in 16.254 s, (310 total leaves)
Time spent computing histograms: 4.698s
Time spent finding best splits:  0.023s
Time spent applying splits:      1.160s
Time spent predicting:           0.364s
fitted in 16.262s
predicted in 1.324s, ROC AUC: 0.7951, ACC: 0.7182
```



# A. 参考資料
- [Blog Perspectives Graviton 2](https://perspectives.mvdirona.com/2020/01/aws-graviton2/)
- [7nmプロセスで独自設計の次世代ARMプロセッサ「Graviton 2」をAmazon Web Servicesが発表](https://gigazine.net/news/20191204-aws-graviton-2/)
- [Benchmarks for Amazon's Graviton2 Arm Processor](https://www.infoq.com/news/2020/03/Graviton2-benchmarks/)

## A.1. Benchmark
- [(anandtech)Amazon's Arm-Based Graviton 2 Againist AMD and Intel: Comparing Cloud Compute](https://www.anandtech.com/show/15578/cloud-clash-amazon-graviton2-arm-against-intel-and-amd)
- [(phornix)Benchmarking Amazon's Graviton2 Performance With 64 Neoverse N1 Cores Against Intel Xeon, AMD EPYC](https://www.phoronix.com/scan.php?page=article&item=amazon-graviton2-benchmarks)
    - [PyBench etc](https://www.phoronix.com/scan.php?page=article&item=amazon-graviton2-benchmarks&num=10)
- [(phornix)Benchmarks Of 2nd Gen AMD EPYC On Amazon EC2 Against Intel Xeon, Graviton2](https://www.phoronix.com/scan.php?page=article&item=epyc-ec2-c5a&num=1)
- [(keydb)Benchmarking the AWS Graviton2 with KeyDB – M6g up to 65% faster](https://docs.keydb.dev/blog/2020/03/02/blog-post/)
- [Benchmarking the new Arm-powered M6g instances from AWS](https://extricate.org/2020/05/13/benchmarking-the-new-arm-powered-m6g-instances-from-aws/)
- [(hpc)M6g 触ってみた (1) ーARM64の広がる世界ー](https://www.hpc.co.jp/tech-blog/2020/05/27/m6g-1-hpl/)
- [(hpc)M6g 触ってみた (2) -OpenFOAM ベンチマーク結果-](https://www.hpc.co.jp/tech-blog/2020/05/27/m6g-2-openfoam/)

## A.2. Neoverse N1
- [(OpenBLAS)Add Neoverse-N1 core](https://github.com/xianyi/OpenBLAS/commit/c623a965f95b83bd4340a5ebe1a370b1a900545d)
   - 0.3.9 (2020/03/02)から、Neoverse N1対応
- [GCC9 Changes](https://gcc.gnu.org/gcc-9/changes.html) 
    - Neoverse N1は、GCC9から対応

## A.3. Amazon Linux 2 20200520の場合
参考までにメモ

```bash
glibc.aarch64                           2.26-34.amzn2                 installed
gcc.aarch64                             7.3.1-6.amzn2.0.4             amzn2-core
kernel.aarch64                          4.14.177-139.254.amzn2        installed
```
## A.4 Ubuntu
perfツールの使い方

```bash
sudo apt install linux-tools-common
sudo apt install linux-tools-5.4.0-1009-aws linux-cloud-tools-5.4.0-1009-aws
```

```bash
  Children      Self  Command  Shared Object      Symbol
+   99.98%     0.00%  python3  python3.8          [.] _start
+   99.98%     0.00%  python3  libc-2.31.so       [.] __libc_start_main        a
+   99.98%     0.00%  python3  python3.8          [.] Py_BytesMain             a
+   99.95%    36.77%  python3  python3.8          [.] _PyEval_EvalFrameDefault a
+   99.95%     0.00%  python3  python3.8          [.] _PyFunction_Vectorcall   a
+   99.95%     0.00%  python3  python3.8          [.] _PyEval_EvalCodeWithName a
+   99.93%     0.00%  python3  python3.8          [.] PyEval_EvalCode          a
+   99.89%     0.00%  python3  python3.8          [.] 0x000000000064b454       a
+   99.89%     0.00%  python3  python3.8          [.] 0x000000000064b388       a
+   99.87%     0.00%  python3  python3.8          [.] Py_RunMain               a
+   99.84%     0.00%  python3  python3.8          [.] PyRun_SimpleFileExFlags  a
+   99.84%     0.00%  python3  python3.8          [.] PyRun_FileExFlags        a
+   14.35%     4.65%  python3  python3.8          [.] PyNumber_InPlaceAdd      a
+   14.16%     3.67%  python3  python3.8          [.] PyNumber_Multiply        a
+   11.24%    11.23%  python3  python3.8          [.] PyObject_Malloc          a
+    7.05%     0.00%  python3  python3.8          [.] 0x0000000000488758       a
+    6.69%     6.69%  python3  python3.8          [.] _PyLong_New              a
+    6.42%     0.00%  python3  python3.8          [.] 0x0000000000488820       a
+    5.66%     0.00%  python3  python3.8          [.] 0x0000000000576ba8       a
+    4.71%     4.71%  python3  python3.8          [.] PyType_IsSubtype         a
+    4.21%     0.00%  python3  python3.8          [.] 0x000000000054b784       a

```

## A.4. numbaのインストレーション失敗について
Ubuntu 20.04で、AARCH64の場合、llvmliteの問題でインストールできない。
この場合、`/usr/bin/llvm-config-9`をシンボリックリンクを張って`/usr/bin/llvm-config`とやると動く。なお、llvmliteとの問題で、llvm-10で動かずllvm-9にしておく必要がある。


```bash
Processing /root/.cache/pip/wheels/d7/77/7a/386f626dfe913d2869da6a1dc83bf496b7314c3273f5ab04bb/numba-0.50.0-cp38-cp38-linux_aarch64.whl
Collecting llvmlite<0.34,>=0.33.0.dev0
  Using cached llvmlite-0.33.0.tar.gz (104 kB)
Requirement already satisfied: numpy>=1.15 in /usr/local/lib/python3.8/dist-packages (from numba) (1.19.0)
Requirement already satisfied: setuptools in /usr/lib/python3/dist-packages (from numba) (45.2.0)
Building wheels for collected packages: llvmlite
  Building wheel for llvmlite (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-hbxxlrnb/llvmlite/setup.py'"'"'; __file__='"'"'/tmp/pip-install-hbxxlrnb/llvmlite/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-disjz20c
       cwd: /tmp/pip-install-hbxxlrnb/llvmlite/
  Complete output (26 lines):
  running bdist_wheel
  /usr/bin/python3 /tmp/pip-install-hbxxlrnb/llvmlite/ffi/build.py
  LLVM version... Traceback (most recent call last):
    File "/tmp/pip-install-hbxxlrnb/llvmlite/ffi/build.py", line 105, in main_posix
      out = subprocess.check_output([llvm_config, '--version'])
    File "/usr/lib/python3.8/subprocess.py", line 411, in check_output
      return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
    File "/usr/lib/python3.8/subprocess.py", line 489, in run
      with Popen(*popenargs, **kwargs) as process:
    File "/usr/lib/python3.8/subprocess.py", line 854, in __init__
      self._execute_child(args, executable, preexec_fn, close_fds,
    File "/usr/lib/python3.8/subprocess.py", line 1702, in _execute_child
      raise child_exception_type(errno_num, err_msg, err_filename)
  FileNotFoundError: [Errno 2] No such file or directory: 'llvm-config'

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/tmp/pip-install-hbxxlrnb/llvmlite/ffi/build.py", line 191, in <module>
      main()
    File "/tmp/pip-install-hbxxlrnb/llvmlite/ffi/build.py", line 181, in main
      main_posix('linux', '.so')
    File "/tmp/pip-install-hbxxlrnb/llvmlite/ffi/build.py", line 107, in main_posix
      raise RuntimeError("%s failed executing, please point LLVM_CONFIG "
  RuntimeError: llvm-config failed executing, please point LLVM_CONFIG to the path for llvm-config
  error: command '/usr/bin/python3' failed with exit status 1
  ----------------------------------------
  ERROR: Failed building wheel for llvmlite
  Running setup.py clean for llvmlite
Failed to build llvmlite
Installing collected packages: llvmlite, numba
    Running setup.py install for llvmlite ... error
    ERROR: Command errored out with exit status 1:
     command: /usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-hbxxlrnb/llvmlite/setup.py'"'"'; __file__='"'"'/tmp/pip-install-hbxxlrnb/llvmlite/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-d14nmr8w/install-record.txt --single-version-externally-managed --compile --install-headers /usr/local/include/python3.8/llvmlite
         cwd: /tmp/pip-install-hbxxlrnb/llvmlite/
    Complete output (29 lines):
    running install
    running build
    got version from file /tmp/pip-install-hbxxlrnb/llvmlite/llvmlite/_version.py {'version': '0.33.0', 'full': '76c8ecba36e23fa8b6878505a1afc8a3d7a2c8a1'}
    running build_ext
    /usr/bin/python3 /tmp/pip-install-hbxxlrnb/llvmlite/ffi/build.py
    LLVM version... Traceback (most recent call last):
      File "/tmp/pip-install-hbxxlrnb/llvmlite/ffi/build.py", line 105, in main_posix
        out = subprocess.check_output([llvm_config, '--version'])
      File "/usr/lib/python3.8/subprocess.py", line 411, in check_output
        return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
      File "/usr/lib/python3.8/subprocess.py", line 489, in run
        with Popen(*popenargs, **kwargs) as process:
      File "/usr/lib/python3.8/subprocess.py", line 854, in __init__
        self._execute_child(args, executable, preexec_fn, close_fds,
      File "/usr/lib/python3.8/subprocess.py", line 1702, in _execute_child
        raise child_exception_type(errno_num, err_msg, err_filename)
    FileNotFoundError: [Errno 2] No such file or directory: 'llvm-config'

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/tmp/pip-install-hbxxlrnb/llvmlite/ffi/build.py", line 191, in <module>
        main()
      File "/tmp/pip-install-hbxxlrnb/llvmlite/ffi/build.py", line 181, in main
        main_posix('linux', '.so')
      File "/tmp/pip-install-hbxxlrnb/llvmlite/ffi/build.py", line 107, in main_posix
        raise RuntimeError("%s failed executing, please point LLVM_CONFIG "
    RuntimeError: llvm-config failed executing, please point LLVM_CONFIG to the path for llvm-config
    error: command '/usr/bin/python3' failed with exit status 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-hbxxlrnb/llvmlite/setup.py'"'"'; __file__='"'"'/tmp/pip-install-hbxxlrnb/llvmlite/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-d14nmr8w/install-record.txt --single-version-externally-managed --compile --install-headers /usr/local/include/python3.8/llvmlite Check the logs for full command output.
```

## A.5. scikit-learnのインストール
aarch64のscikit-learnでは、gfortranがインストールする必要がある。また、そのベースのscipyがOpenBLASがインストールされている必要がある。

```bash
sudo apt update
sudo apt install python3-pip
sudo apt install libopenblas-dev
sudo apt install gfortran
sudo pip3 install scipy
sudo pip3 install scikit-learn
```