BBR Congestion Control algorithm - Bandwidth faster with UEKR5 

# BBR : Bottleneck Bandwidth and Round-trip propagation time


## iperf x 10 times faster - result

![image.png](https://qiita-image-store.s3.amazonaws.com/0/114634/787d6f8b-ce19-cf88-b316-81122a6f556c.png)




## References
 * Neal Cardwell, Yuchung Cheng, C. Stephen Gunn, Soheil Hassas Yeganeh, Van Jacobson
 “BBR: Congestion Based Congestion Control,” Communications of the ACM, Vol. 60 No. 2, Pages 58-66
 https://queue.acm.org/detail.cfm?id=3022184

![vanjacobson8[1].png](https://qiita-image-store.s3.amazonaws.com/0/114634/71da2bbd-ac2d-7322-495d-1ecbfafd1488.png)

 * Congestion Control algorithms in UEK5 preview - try out BBR
https://blogs.oracle.com/wim/done-cancel-v6


 * パケットロスに基づかない新しい輻輳制御の仕組み ― BBR
    * https://www.school.ctc-g.co.jp/columns/nakai2/nakai222.html
    * https://www.school.ctc-g.co.jp/columns/nakai2/nakai223.html
 * centOS7 
    * https://www.vultr.com/docs/how-to-deploy-google-bbr-on-centos-7


### enable Oracle Linux **UEKR5**

**enabled=1**

```
[ol7_developer_UEKR5]
name=Oracle Linux $releasever Unbreakable Enterprise Kernel Release 5  Packages for Development and test ($basearch)
baseurl=http://yum.oracle.com/repo/OracleLinux/OL7/developer_UEKR5/$basearch/
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-oracle
gpgcheck=1
enabled=1
```

## Let's Try BBR

- before BBR

```
iperf -c iperf.xxx -i 2 -t 30
```

```
------------------------------------------------------------
Client connecting to iperf.xxx, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  3] local xxx.xxx.xxx.xxx port 60808 connected with xxx.xxx.xxx.xxx port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 2.0 sec   192 KBytes   788 Kbits/sec
[  3]  2.0- 4.0 sec   177 KBytes   724 Kbits/sec
[  3]  4.0- 6.0 sec   256 KBytes  1.05 Mbits/sec
[  3]  6.0- 8.0 sec   768 KBytes  3.15 Mbits/sec
[  3]  8.0-10.0 sec  1.88 MBytes  7.86 Mbits/sec
[  3] 10.0-12.0 sec  2.00 MBytes  8.39 Mbits/sec
[  3] 12.0-14.0 sec  1.46 MBytes  6.14 Mbits/sec
[  3] 14.0-16.0 sec  1.36 MBytes  5.72 Mbits/sec
[  3] 16.0-18.0 sec  1.24 MBytes  5.21 Mbits/sec
[  3] 18.0-20.0 sec  1.24 MBytes  5.21 Mbits/sec
[  3] 20.0-22.0 sec  1.06 MBytes  4.43 Mbits/sec
[  3] 22.0-24.0 sec   764 KBytes  3.13 Mbits/sec
[  3] 24.0-26.0 sec   827 KBytes  3.39 Mbits/sec
[  3] 26.0-28.0 sec   891 KBytes  3.65 Mbits/sec
[  3] 28.0-30.0 sec   827 KBytes  3.39 Mbits/sec
[  3]  0.0-30.1 sec  14.8 MBytes  4.14 Mbits/sec
```



- set BBR

```
# sysctl -w net.ipv4.tcp_congestion_control=bbr
net.ipv4.tcp_congestion_control = bbr

# sysctl -w net.core.default_qdisc=fq
net.core.default_qdisc = fq
```

 cf. with **UEKR4**, you cannot set BBR

```
# sysctl -w net.ipv4.tcp_congestion_control=bbr

sysctl: setting key "net.ipv4.tcp_congestion_control": No such file or directory
net.ipv4.tcp_congestion_control = bbr
```

- (set permanently)

```
echo 'net.core.default_qdisc=fq' | sudo tee -a /etc/sysctl.conf
echo 'net.ipv4.tcp_congestion_control=bbr' | sudo tee -a /etc/sysctl.conf
sudo sysctl -p
```

- after

```
iperf -c iperf.xxx -i 2 -t 30
```

```
------------------------------------------------------------
Client connecting to iperf.xxx, TCP port 5001
TCP window size:  128 KByte (default)
------------------------------------------------------------
[  3] local xxx.xxx.xxx.xxx port 33064 connected with xxx.xxx.xxx.xxx port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 2.0 sec  2.38 MBytes  9.96 Mbits/sec
[  3]  2.0- 4.0 sec  4.00 MBytes  16.8 Mbits/sec
[  3]  4.0- 6.0 sec  5.25 MBytes  22.0 Mbits/sec
[  3]  6.0- 8.0 sec  8.00 MBytes  33.6 Mbits/sec
[  3]  8.0-10.0 sec  8.96 MBytes  37.6 Mbits/sec
[  3] 10.0-12.0 sec  3.17 MBytes  13.3 Mbits/sec
[  3] 12.0-14.0 sec  5.00 MBytes  21.0 Mbits/sec
[  3] 14.0-16.0 sec  7.75 MBytes  32.5 Mbits/sec
[  3] 16.0-18.0 sec  7.62 MBytes  32.0 Mbits/sec
[  3] 18.0-20.0 sec  10.2 MBytes  43.0 Mbits/sec
[  3] 20.0-22.0 sec  10.2 MBytes  43.0 Mbits/sec
[  3] 22.0-24.0 sec  6.25 MBytes  26.2 Mbits/sec
[  3] 24.0-26.0 sec  10.9 MBytes  45.6 Mbits/sec
[  3] 26.0-28.0 sec  10.6 MBytes  44.6 Mbits/sec
[  3] 28.0-30.0 sec  10.8 MBytes  45.1 Mbits/sec
[  3]  0.0-30.4 sec   111 MBytes  30.7 Mbits/sec
```


:smile: 


- follow up

```
iperf -c  www -p 80 -f M -i 1 -t 30 -e
------------------------------------------------------------
Write buffer size: 0.12 MByte
TCP window size: 0.08 MByte (default)
------------------------------------------------------------
[ ID] Interval        Transfer    Bandwidth       Write/Err  Rtry    Cwnd/RTT
[  3] 0.00-1.00 sec  0.62 MBytes  0.62 MBytes/sec  5/0          1       29K/38437 us
[  3] 1.00-2.00 sec  0.88 MBytes  0.88 MBytes/sec  7/0          5       33K/39418 us
[  3] 2.00-3.00 sec  1.00 MBytes  1.00 MBytes/sec  8/0          0       50K/37639 us
[  3] 3.00-4.00 sec  1.00 MBytes  1.00 MBytes/sec  8/0          1       49K/38945 us
[  3] 4.00-5.00 sec  1.62 MBytes  1.62 MBytes/sec  13/0          0       69K/37844 us
[  3] 5.00-6.00 sec  1.75 MBytes  1.75 MBytes/sec  14/0          4       63K/38286 us
[  3] 6.00-7.00 sec  1.25 MBytes  1.25 MBytes/sec  10/0          3       60K/40307 us
[  3] 7.00-8.00 sec  1.38 MBytes  1.38 MBytes/sec  11/0          3       57K/38726 us
[  3] 8.00-9.00 sec  1.75 MBytes  1.75 MBytes/sec  14/0         13       53K/37995 us
[  3] 9.00-10.00 sec  1.50 MBytes  1.50 MBytes/sec  12/0          1       49K/39413 us
[  3] 10.00-11.00 sec  1.38 MBytes  1.38 MBytes/sec  11/0          0       65K/40179 us
[  3] 11.00-12.00 sec  1.75 MBytes  1.75 MBytes/sec  14/0          7       59K/38352 us
[  3] 12.00-13.00 sec  1.25 MBytes  1.25 MBytes/sec  10/0          6       56K/38859 us
[  3] 13.00-14.00 sec  1.12 MBytes  1.12 MBytes/sec  9/0          4       53K/37793 us
[  3] 14.00-15.00 sec  1.12 MBytes  1.12 MBytes/sec  9/0          3       53K/40159 us
[  3] 15.00-16.00 sec  1.50 MBytes  1.50 MBytes/sec  12/0          0       72K/37953 us
[  3] 16.00-17.00 sec  2.00 MBytes  2.00 MBytes/sec  16/0          0       90K/37801 us
[  3] 17.00-18.00 sec  1.75 MBytes  1.75 MBytes/sec  14/0         24       56K/40686 us
[  3] 18.00-19.00 sec  1.38 MBytes  1.38 MBytes/sec  11/0         11       53K/38617 us
[  3] 19.00-20.00 sec  1.50 MBytes  1.50 MBytes/sec  12/0          0       70K/37634 us
[  3] 20.00-21.00 sec  1.50 MBytes  1.50 MBytes/sec  12/0          1       65K/38342 us
[  3] 21.00-22.00 sec  1.88 MBytes  1.88 MBytes/sec  15/0          0       83K/37827 us
[  3] 22.00-23.00 sec  1.75 MBytes  1.75 MBytes/sec  14/0          2       74K/38250 us
[  3] 23.00-24.00 sec  2.00 MBytes  2.00 MBytes/sec  16/0          3       69K/41142 us
[  3] 24.00-25.00 sec  0.88 MBytes  0.88 MBytes/sec  7/0         19       33K/38047 us
[  3] 25.00-26.00 sec  1.00 MBytes  1.00 MBytes/sec  8/0          0       53K/40102 us
[  3] 26.00-27.00 sec  1.50 MBytes  1.50 MBytes/sec  12/0          3       50K/37855 us
[  3] 27.00-28.00 sec  1.25 MBytes  1.25 MBytes/sec  10/0          0       66K/38692 us
[  3] 28.00-29.00 sec  2.00 MBytes  2.00 MBytes/sec  16/0          0       86K/38185 us
[  3] 29.00-30.00 sec  1.62 MBytes  1.62 MBytes/sec  13/0         11       76K/38041 us

[  3] 0.00-30.03 sec  42.9 MBytes  1.43 MBytes/sec  343/0        125       76K/38041 us
```

```
sysctl -w net.ipv4.tcp_congestion_control=bbr
net.ipv4.tcp_congestion_control = bbr

sysctl -w net.core.default_qdisc=fq
net.core.default_qdisc = fq
```

```
iperf -c  www -p 80 -f M -i 1 -t 30 -e
------------------------------------------------------------
Write buffer size: 0.12 MByte
TCP window size: 0.12 MByte (default)
------------------------------------------------------------
[ ID] Interval        Transfer    Bandwidth       Write/Err  Rtry    Cwnd/RTT
[  3] 0.00-1.00 sec  23.9 MBytes  23.9 MBytes/sec  191/0        895     2856K/33006 us
[  3] 1.00-2.00 sec  50.8 MBytes  50.8 MBytes/sec  406/0        273     4595K/36386 us
[  3] 2.00-3.00 sec  42.5 MBytes  42.5 MBytes/sec  340/0        562     3967K/37662 us
[  3] 3.00-4.00 sec  41.1 MBytes  41.1 MBytes/sec  329/0        734     4457K/40878 us
[  3] 4.00-5.00 sec  50.6 MBytes  50.6 MBytes/sec  405/0        606     3044K/32899 us
[  3] 5.00-6.00 sec  49.0 MBytes  49.0 MBytes/sec  392/0        278     4669K/42670 us
[  3] 6.00-7.00 sec  50.9 MBytes  50.9 MBytes/sec  407/0        239     4409K/45483 us
[  3] 7.00-8.00 sec  43.4 MBytes  43.4 MBytes/sec  347/0        770     2638K/33185 us
[  3] 8.00-9.00 sec  54.1 MBytes  54.1 MBytes/sec  433/0          6     4072K/43616 us
[  3] 9.00-10.00 sec  43.8 MBytes  43.8 MBytes/sec  350/0        362     2757K/40692 us
[  3] 10.00-11.00 sec  51.9 MBytes  51.9 MBytes/sec  415/0        177     4929K/34683 us
[  3] 11.00-12.00 sec  50.6 MBytes  50.6 MBytes/sec  405/0        607     4686K/36481 us
[  3] 12.00-13.00 sec  54.2 MBytes  54.2 MBytes/sec  434/0        368     5413K/34346 us
[  3] 13.00-14.00 sec  58.6 MBytes  58.6 MBytes/sec  469/0        716     2918K/34584 us
[  3] 14.00-15.00 sec  49.5 MBytes  49.5 MBytes/sec  396/0        632     5438K/40706 us
[  3] 15.00-16.00 sec  47.4 MBytes  47.4 MBytes/sec  379/0        516     4626K/40840 us
[  3] 16.00-17.00 sec  53.1 MBytes  53.1 MBytes/sec  425/0        477     2743K/49570 us
[  3] 17.00-18.00 sec  54.9 MBytes  54.9 MBytes/sec  439/0        240     4827K/37155 us
[  3] 18.00-19.00 sec  45.6 MBytes  45.6 MBytes/sec  365/0        442     4892K/46708 us
[  3] 19.00-20.00 sec  48.5 MBytes  48.5 MBytes/sec  388/0        333     3085K/33415 us
[  3] 20.00-21.00 sec  53.9 MBytes  53.9 MBytes/sec  431/0        352     4776K/39637 us
[  3] 21.00-22.00 sec  48.6 MBytes  48.6 MBytes/sec  389/0        850     5127K/44432 us
[  3] 22.00-23.00 sec  62.2 MBytes  62.2 MBytes/sec  498/0        400     5483K/41042 us
[  3] 23.00-24.00 sec  59.8 MBytes  59.8 MBytes/sec  478/0        396     5376K/35238 us
[  3] 24.00-25.00 sec  57.6 MBytes  57.6 MBytes/sec  461/0        384     4875K/45756 us
[  3] 25.00-26.00 sec  54.2 MBytes  54.2 MBytes/sec  434/0        221     2927K/33351 us
[  3] 26.00-27.00 sec  48.4 MBytes  48.4 MBytes/sec  387/0        878     5096K/34626 us
[  3] 27.00-28.00 sec  47.6 MBytes  47.6 MBytes/sec  381/0        600     5331K/37356 us
[  3] 28.00-29.00 sec  56.5 MBytes  56.5 MBytes/sec  452/0        606     2508K/33246 us
[  3] 29.00-30.00 sec  54.6 MBytes  54.6 MBytes/sec  437/0        525     5215K/34458 us

[  3] 0.00-30.03 sec  1508 MBytes  50.2 MBytes/sec  12063/0      14445     5215K/34458 us
```