BenchmarkDotNetのサマリについて

#趣旨
シャッフル処理の速度検証を進める中で、発生した疑問(仮定)を検証する。
処理速度の計測にはBenchmarkDotNet(v0.11.0)を用いる。
~~計測中にBenchmarkDotNetのバグらしきものを発見:frowning2:~~
→と思ってたら、何もおかしくない、普通の結果でした！
 結果をよく確認しようねって自分への戒め。

##結論
~~BenchmarkDotNetの計測対象に5行以下の処理があると、サマリだけ実測値に*1000された値が表示される？~~
→サマリは一番速い計測結果の単位にまとめられるため。
　例えば、マイクロ秒(us)とナノ秒(ns)が混在するとナノ秒にまとめられる。

##仮定→検証
###乱数の幅は速度に影響するか→乱数の幅は結果に影響しない

```fsharp
[<Benchmark>]
member this.randBench() = 
    Math.Next(7)

[<Benchmark>]
member this.randBench2() = 
    Math.Next(28)
```
     Method |     Mean |     Error |    StdDev |
----------- |---------:|----------:|----------:|
  randBench | 40.16 ns | 0.5381 ns | 0.5033 ns |
 randBench2 | 41.28 ns | 1.3240 ns | 1.7675 ns |

###結果を束縛するかどうかで速度に影響はあるか→束縛すると速くなる

```fsharp

[<Benchmark>]
member this.randBench() = 
    Math.Next(7)

[<Benchmark>]
member this.randBench2() = 
    let rand = Math.Next(7)
    rand |> ignore
```

           Method |        Mean |      Error |     StdDev |
----------------- |------------:|-----------:|-----------:|
        randBench |    40.84 ns |  0.6457 ns |  0.6040 ns |
       randBench2 |    27.82 ns |  0.1573 ns |  0.1395 ns |

###！謎の計測結果が発生！

```fsharp
[<Benchmark>]
member this.randBench2() = 
    let rand = Math.Next(7)
    rand |> ignore

[<Benchmark>]
member this.patternShuffled2() = 
    let mutable cards = []
    for i in 1..7 do
        cards <- List.append [{cardNumber = i; cardCount = i}] cards

    let mutable shuffledCards = []

    while shuffledCards.Length < 28 do
        let rand = Math.Next(7)
        if cards.[rand].cardCount > 0 then
            shuffledCards <- List.append shuffledCards [cards.[rand].cardNumber]
            useCard(cards.[rand])

```
単体なら速いが

           Method |     Mean |     Error |    StdDev |
----------------- |---------:|----------:|----------:|
 patternShuffled2 | 7.478 us | 0.0892 us | 0.0835 us |

randBenchの後に呼ぶと猛烈に遅い？
→※nsにまとめられているだけ:sweat:

           Method |        Mean |      Error |     StdDev |
----------------- |------------:|-----------:|-----------:|
        randBench |    40.18 ns |  0.5177 ns |  0.4589 ns |
       randBench2 |    27.56 ns |  0.4559 ns |  0.4264 ns |
 patternShuffled2 | 7,427.43 ns | 72.0899 ns | 67.4329 ns |

しかし、別の処理の後だと単体時と同様の実行速度が出る

```fsharp

[<Benchmark>]
member this.patternShuffled() = 
    let mutable cards = [1;2;3;4;5;6;7]
    while cards.Length < 28 do
        let randCard = Math.Next(8)
        List.countBy (fun elem -> if (elem = randCard) then 1 else 0) cards
        |> List.filter (fun tpl -> fst(tpl) = 1 && snd(tpl) < randCard)
        |> List.iter (fun x -> cards <- List.append [randCard] cards)
        |> ignore

    //showCards("Shuffled", cards) |> ignore

[<Benchmark>]
member this.patternShuffled2() = 
    let mutable cards = []
    for i in 1..7 do
        cards <- List.append [{cardNumber = i; cardCount = i}] cards

    let mutable shuffledCards = []

    while shuffledCards.Length < 28 do
        let rand = Math.Next(7)
        if cards.[rand].cardCount > 0 then
            shuffledCards <- List.append shuffledCards [cards.[rand].cardNumber]
            useCard(cards.[rand])

```

           Method |      Mean |     Error |    StdDev |
----------------- |----------:|----------:|----------:|
  patternShuffled | 45.486 us | 0.8616 us | 0.7637 us |
 patternShuffled2 |  7.597 us | 0.1484 us | 0.1588 us |
※どちらもusなので、揃って見えている:sweat:

この実行順を保ったまま前の処理を足すと…

```fsharp
[<Benchmark>]
member this.randBench2() = 
    let rand = Math.Next(7)
    rand |> ignore

[<Benchmark>]
member this.patternShuffled() = 
    let mutable cards = [1;2;3;4;5;6;7]
    while cards.Length < 28 do
        let randCard = Math.Next(8)
        List.countBy (fun elem -> if (elem = randCard) then 1 else 0) cards
        |> List.filter (fun tpl -> fst(tpl) = 1 && snd(tpl) < randCard)
        |> List.iter (fun x -> cards <- List.append [randCard] cards)
        |> ignore

    //showCards("Shuffled", cards) |> ignore

[<Benchmark>]
member this.patternShuffled2() = 
    let mutable cards = []
    for i in 1..7 do
        cards <- List.append [{cardNumber = i; cardCount = i}] cards

    let mutable shuffledCards = []

    while shuffledCards.Length < 28 do
        let rand = Math.Next(7)
        if cards.[rand].cardCount > 0 then
            shuffledCards <- List.append shuffledCards [cards.[rand].cardNumber]
            useCard(cards.[rand])
```

後続がすべて猛烈に遅くなる？
※nsにまとめられて、遅くなったように見えるだけ:sweat:

           Method |         Mean |       Error |      StdDev |
----------------- |-------------:|------------:|------------:|
       randBench2 |     27.84 ns |   0.4091 ns |   0.3627 ns |
  patternShuffled | 46,328.89 ns | 754.1993 ns | 668.5780 ns |
 patternShuffled2 |  7,681.59 ns | 147.5512 ns | 138.0195 ns |

###乱数処理の有無が後続処理に影響を与えているか→関係ない

```fsharp
[<Benchmark>]
member this.Bench() = 
    let x = 3
    x

[<Benchmark>]
member this.patternShuffled2() = 
    let mutable cards = []
    for i in 1..7 do
        cards <- List.append [{cardNumber = i; cardCount = i}] cards

    let mutable shuffledCards = []

    while shuffledCards.Length < 28 do
        let rand = Math.Next(7)
        if cards.[rand].cardCount > 0 then
            shuffledCards <- List.append shuffledCards [cards.[rand].cardNumber]
            useCard(cards.[rand])

```

           Method |          Mean |      Error |     StdDev |
----------------- |--------------:|-----------:|-----------:|
            Bench |     0.0040 ns |  0.0072 ns |  0.0067 ns |
 patternShuffled2 | 7,450.4946 ns | 87.6111 ns | 81.9514 ns |

###結果表示に誤りはないか→サマリだけ実測に*1000された値が表示されている？

```fsharp
[<Benchmark>]
member this.Bench() = 
    let list = [3;4;5;6;7;8;9]
    let result = List.reduce(fun acc x -> x * x) list
    result

[<Benchmark>]
member this.patternShuffled2() = 
    let mutable cards = []
    for i in 1..7 do
        cards <- List.append [{cardNumber = i; cardCount = i}] cards

    let mutable shuffledCards = []

    while shuffledCards.Length < 28 do
        let rand = Math.Next(7)
        if cards.[rand].cardCount > 0 then
            shuffledCards <- List.append shuffledCards [cards.[rand].cardNumber]
            useCard(cards.[rand])

```

// * Detailed results *
Benchmarks.Bench: DefaultJob
Runtime = .NET Framework 4.7.2 (CLR 4.0.30319.42000), 32bit LegacyJIT-v4.7.3132.0; GC = Concurrent Workstation
**Mean = 59.7232 ns, StdErr = 0.1317 ns (0.22%); N = 14, StdDev = 0.4926 ns**
Min = 59.1001 ns, Q1 = 59.2893 ns, Median = 59.6948 ns, Q3 = 60.0225 ns, Max = 60.6935 ns
IQR = 0.7332 ns, LowerFence = 58.1895 ns, UpperFence = 61.1223 ns
ConfidenceInterval = [59.1674 ns; 60.2789 ns] (CI 99.9%), Margin = 0.5557 ns (0.93% of Mean)
Skewness = 0.38, Kurtosis = 1.9, MValue = 2
'-------------------- Histogram --------------------
[58.921 ns ; 60.872 ns) | @@@@@@@@@@@@@@
'---------------------------------------------------

Benchmarks.patternShuffled2: DefaultJob
Runtime = .NET Framework 4.7.2 (CLR 4.0.30319.42000), 32bit LegacyJIT-v4.7.3132.0; GC = Concurrent Workstation
**Mean = 7.5061 us, StdErr = 0.0171 us (0.23%); N = 15, StdDev = 0.0664 us**
Min = 7.4140 us, Q1 = 7.4543 us, Median = 7.4845 us, Q3 = 7.5430 us, Max = 7.6229 us
IQR = 0.0886 us, LowerFence = 7.3214 us, UpperFence = 7.6759 us
ConfidenceInterval = [7.4351 us; 7.5771 us] (CI 99.9%), Margin = 0.0710 us (0.95% of Mean)
Skewness = 0.38, Kurtosis = 1.7, MValue = 2
'-------------------- Histogram --------------------
[7.390 us ; 7.646 us) | @@@@@@@@@@@@@@@
'---------------------------------------------------

// * Summary *

BenchmarkDotNet=v0.11.0, OS=Windows 10.0.17134.228 (1803/April2018Update/Redstone4)
Intel Core i7-6700 CPU 3.40GHz (Max: 3.41GHz) (Skylake), 1 CPU, 8 logical and 4 physical cores
Frequency=3328124 Hz, Resolution=300.4696 ns, Timer=TSC
  [Host]     : .NET Framework 4.7.2 (CLR 4.0.30319.42000), 32bit LegacyJIT-v4.7.3132.0
  DefaultJob : .NET Framework 4.7.2 (CLR 4.0.30319.42000), 32bit LegacyJIT-v4.7.3132.0


           Method |        Mean |      Error |     StdDev |
----------------- |------------:|-----------:|-----------:|
            Bench |    59.72 ns |  0.5557 ns |  0.4926 ns |
 patternShuffled2 | 7,506.15 ns | 71.0002 ns | 66.4136 ns |

一体どういうことなの...
※nsにまとめられただけでした！:sweat:

###直前の処理の長さが影響しているか→影響する

```fsharp
[<Benchmark>]
member this.xxxxxxxxxxxxBench() = 
    let mutable list = [3;4;5;6;7;8;9]
    for i in 1..20 do
        list
        |> List.filter(fun f -> f % 3 = 0)
        |> List.filter(fun f -> f % 6 = 0)
        |> List.reduce(fun acc x -> x * x)
        |> ignore
        list <- List.append [i] list


[<Benchmark>]
member this.patternShuffled2() = 
    let mutable cards = []
    for i in 1..7 do
        cards <- List.append [{cardNumber = i; cardCount = i}] cards

    let mutable shuffledCards = []

    while shuffledCards.Length < 28 do
        let rand = Math.Next(7)
        if cards.[rand].cardCount > 0 then
            shuffledCards <- List.append shuffledCards [cards.[rand].cardNumber]
            useCard(cards.[rand])
```

// * Detailed results *
Benchmarks.xxxxxxxxxxxxBench: DefaultJob
Runtime = .NET Framework 4.7.2 (CLR 4.0.30319.42000), 32bit LegacyJIT-v4.7.3132.0; GC = Concurrent Workstation
Mean = 3.3612 us, StdErr = 0.0156 us (0.46%); N = 16, StdDev = 0.0623 us
Min = 3.2898 us, Q1 = 3.3133 us, Median = 3.3342 us, Q3 = 3.4082 us, Max = 3.4746 us
IQR = 0.0948 us, LowerFence = 3.1711 us, UpperFence = 3.5504 us
ConfidenceInterval = [3.2977 us; 3.4246 us] (CI 99.9%), Margin = 0.0634 us (1.89% of Mean)
Skewness = 0.59, Kurtosis = 1.76, MValue = 2
'-------------------- Histogram --------------------
[3.283 us ; 3.368 us) | @@@@@@@@@@
[3.368 us ; 3.496 us) | @@@@@@
'---------------------------------------------------

Benchmarks.patternShuffled2: DefaultJob
Runtime = .NET Framework 4.7.2 (CLR 4.0.30319.42000), 32bit LegacyJIT-v4.7.3132.0; GC = Concurrent Workstation
Mean = 7.6634 us, StdErr = 0.0391 us (0.51%); N = 20, StdDev = 0.1747 us
Min = 7.4249 us, Q1 = 7.5325 us, Median = 7.6369 us, Q3 = 7.7952 us, Max = 8.0300 us
IQR = 0.2627 us, LowerFence = 7.1384 us, UpperFence = 8.1893 us
ConfidenceInterval = [7.5117 us; 7.8151 us] (CI 99.9%), Margin = 0.1517 us (1.98% of Mean)
Skewness = 0.45, Kurtosis = 2.13, MValue = 2
'-------------------- Histogram --------------------
[7.401 us ; 7.561 us) | @@@@@@
[7.561 us ; 7.825 us) | @@@@@@@@@
[7.825 us ; 8.086 us) | @@@@@
'---------------------------------------------------

// * Summary *

BenchmarkDotNet=v0.11.0, OS=Windows 10.0.17134.228 (1803/April2018Update/Redstone4)
Intel Core i7-6700 CPU 3.40GHz (Max: 3.41GHz) (Skylake), 1 CPU, 8 logical and 4 physical cores
Frequency=3328124 Hz, Resolution=300.4696 ns, Timer=TSC
  [Host]     : .NET Framework 4.7.2 (CLR 4.0.30319.42000), 32bit LegacyJIT-v4.7.3132.0
  DefaultJob : .NET Framework 4.7.2 (CLR 4.0.30319.42000), 32bit LegacyJIT-v4.7.3132.0


            Method |     Mean |     Error |    StdDev |
------------------ |---------:|----------:|----------:|
 xxxxxxxxxxxxBench | 3.361 us | 0.0634 us | 0.0623 us |
  patternShuffled2 | 7.663 us | 0.1517 us | 0.1747 us |

ちゃんと結果が出ている
※処理を増やした結果、どちらもusになっただけ:sweat:

###異常発生の境界値はどこか？→5行

6行→問題なし

```fsharp
[<Benchmark>]
member this.yyyyyyyyyyBench() = 
    let mutable list = [3;4;5;6;7;8;9]
    for i in 1..20 do
        list
        |> List.filter(fun f -> f % 3 = 0)
        |> List.reduce(fun acc x -> x * x)
        |> ignore

[<Benchmark>]
member this.patternShuffled2() = 
    let mutable cards = []
    for i in 1..7 do
        cards <- List.append [{cardNumber = i; cardCount = i}] cards

    let mutable shuffledCards = []

    while shuffledCards.Length < 28 do
        let rand = Math.Next(7)
        if cards.[rand].cardCount > 0 then
            shuffledCards <- List.append shuffledCards [cards.[rand].cardNumber]
            useCard(cards.[rand])
```

// * Detailed results *
Benchmarks.yyyyyyyyyyBench: DefaultJob
Runtime = .NET Framework 4.7.2 (CLR 4.0.30319.42000), 32bit LegacyJIT-v4.7.3132.0; GC = Concurrent Workstation
Mean = 1.1542 us, StdErr = 0.0067 us (0.58%); N = 65, StdDev = 0.0544 us
Min = 1.0743 us, Q1 = 1.1191 us, Median = 1.1385 us, Q3 = 1.1749 us, Max = 1.2949 us
IQR = 0.0557 us, LowerFence = 1.0355 us, UpperFence = 1.2585 us
ConfidenceInterval = [1.1309 us; 1.1774 us] (CI 99.9%), Margin = 0.0233 us (2.02% of Mean)
Skewness = 1.05, Kurtosis = 3.42, MValue = 2
'-------------------- Histogram --------------------
[1.069 us ; 1.092 us) | @@@@
[1.092 us ; 1.118 us) | @@@@@@@@@@
[1.118 us ; 1.141 us) | @@@@@@@@@@@@@@@@@@@@
[1.141 us ; 1.167 us) | @@@@@@@@@@@
[1.167 us ; 1.194 us) | @@@@@@@
[1.194 us ; 1.241 us) | @@@@@@
[1.241 us ; 1.271 us) | @@@
[1.271 us ; 1.301 us) | @@@@
'---------------------------------------------------

Benchmarks.patternShuffled2: DefaultJob
Runtime = .NET Framework 4.7.2 (CLR 4.0.30319.42000), 32bit LegacyJIT-v4.7.3132.0; GC = Concurrent Workstation
Mean = 7.6973 us, StdErr = 0.0404 us (0.53%); N = 27, StdDev = 0.2100 us
Min = 7.3632 us, Q1 = 7.5211 us, Median = 7.6920 us, Q3 = 7.8339 us, Max = 8.0726 us
IQR = 0.3128 us, LowerFence = 7.0520 us, UpperFence = 8.3030 us
ConfidenceInterval = [7.5475 us; 7.8470 us] (CI 99.9%), Margin = 0.1498 us (1.95% of Mean)
Skewness = 0.2, Kurtosis = 1.82, MValue = 2
'-------------------- Histogram --------------------
[7.302 us ; 7.555 us) | @@@@@@@@@
[7.555 us ; 7.824 us) | @@@@@@@@@@
[7.824 us ; 8.098 us) | @@@@@@@@
'---------------------------------------------------

// * Summary *

BenchmarkDotNet=v0.11.0, OS=Windows 10.0.17134.228 (1803/April2018Update/Redstone4)
Intel Core i7-6700 CPU 3.40GHz (Max: 3.41GHz) (Skylake), 1 CPU, 8 logical and 4 physical cores
Frequency=3328124 Hz, Resolution=300.4696 ns, Timer=TSC
  [Host]     : .NET Framework 4.7.2 (CLR 4.0.30319.42000), 32bit LegacyJIT-v4.7.3132.0
  DefaultJob : .NET Framework 4.7.2 (CLR 4.0.30319.42000), 32bit LegacyJIT-v4.7.3132.0


           Method |     Mean |     Error |    StdDev |
----------------- |---------:|----------:|----------:|
  yyyyyyyyyyBench | 1.154 us | 0.0233 us | 0.0544 us |
 patternShuffled2 | 7.697 us | 0.1498 us | 0.2100 us |

5行→異常発生

```fsharp
[<Benchmark>]
member this.yyyyyyyyyyBench() = 
    let mutable list = [3;4;5;6;7;8;9]
    for i in 1..20 do
        list
        |> List.reduce(fun acc x -> x * x)
        |> ignore

[<Benchmark>]
member this.patternShuffled2() = 
    let mutable cards = []
    for i in 1..7 do
        cards <- List.append [{cardNumber = i; cardCount = i}] cards

    let mutable shuffledCards = []

    while shuffledCards.Length < 28 do
        let rand = Math.Next(7)
        if cards.[rand].cardCount > 0 then
            shuffledCards <- List.append shuffledCards [cards.[rand].cardNumber]
            useCard(cards.[rand])
```

// * Detailed results *
Benchmarks.yyyyyyyyyyBench: DefaultJob
Runtime = .NET Framework 4.7.2 (CLR 4.0.30319.42000), 32bit LegacyJIT-v4.7.3132.0; GC = Concurrent Workstation
Mean = 431.9756 ns, StdErr = 1.8186 ns (0.42%); N = 14, StdDev = 6.8046 ns
Min = 420.2765 ns, Q1 = 426.6926 ns, Median = 431.8563 ns, Q3 = 434.5630 ns, Max = 444.2776 ns
IQR = 7.8704 ns, LowerFence = 414.8870 ns, UpperFence = 446.3686 ns
ConfidenceInterval = [424.2996 ns; 439.6517 ns] (CI 99.9%), Margin = 7.6760 ns (1.78% of Mean)
Skewness = 0.26, Kurtosis = 2.09, MValue = 2
'-------------------- Histogram --------------------
[417.806 ns ; 428.498 ns) | @@@@
[428.498 ns ; 446.748 ns) | @@@@@@@@@@
'---------------------------------------------------

Benchmarks.patternShuffled2: DefaultJob
Runtime = .NET Framework 4.7.2 (CLR 4.0.30319.42000), 32bit LegacyJIT-v4.7.3132.0; GC = Concurrent Workstation
Mean = 7.4652 us, StdErr = 0.0243 us (0.33%); N = 15, StdDev = 0.0942 us
Min = 7.3506 us, Q1 = 7.3925 us, Median = 7.4452 us, Q3 = 7.5384 us, Max = 7.6639 us
IQR = 0.1460 us, LowerFence = 7.1735 us, UpperFence = 7.7574 us
ConfidenceInterval = [7.3645 us; 7.5659 us] (CI 99.9%), Margin = 0.1007 us (1.35% of Mean)
Skewness = 0.52, Kurtosis = 1.98, MValue = 2
'-------------------- Histogram --------------------
[7.317 us ; 7.697 us) | @@@@@@@@@@@@@@@
'---------------------------------------------------

// * Summary *

BenchmarkDotNet=v0.11.0, OS=Windows 10.0.17134.228 (1803/April2018Update/Redstone4)
Intel Core i7-6700 CPU 3.40GHz (Max: 3.41GHz) (Skylake), 1 CPU, 8 logical and 4 physical cores
Frequency=3328124 Hz, Resolution=300.4696 ns, Timer=TSC
  [Host]     : .NET Framework 4.7.2 (CLR 4.0.30319.42000), 32bit LegacyJIT-v4.7.3132.0
  DefaultJob : .NET Framework 4.7.2 (CLR 4.0.30319.42000), 32bit LegacyJIT-v4.7.3132.0


           Method |       Mean |      Error |    StdDev |
----------------- |-----------:|-----------:|----------:|
  yyyyyyyyyyBench |   432.0 ns |   7.676 ns |  6.805 ns |
 patternShuffled2 | 7,465.2 ns | 100.657 ns | 94.154 ns |

どうやら5行以下の処理があると、異常が発生するようである。
※行数はたぶん関係なくて、nsレベルにコストが減ったのでnsに統一されただけ:sweat:

##おわりに
F#の速度検証だったはずが、いつのまにやらBenchmarkDotNetとの闘いに...
~~githubのopenなissue見てみたけど、それらしいのは無さそう。~~
~~BenchmarkDotNetのソースを解読したわけじゃないので100%バグとも言い切れないのがなんともなところ。~~
~~まさか仕様ではないとは思うんだけど...報告したほうがええんやろうか:confounded:~~

~~報告するにしても、google翻訳先生頼りになるので、きちんとした英語で伝えられるかどうかがまったくの謎。~~
~~この記事へのリンク貼り付けたら、対応してもらえるかしら？？~~

最初に出した計測結果がusだったから、nsに統一されて表示された時に違和感を感じただけだったという話。
１人で開発や検証してるときこそ、思い込みに気を付けないと:tired_face:

BenchmarkDotNetの開発者の方々、バグとか疑ってすみませんでした:bow: