Real-time Packet Observation Tool (RPOT) を ubuntu 18.04 で動かす

# Real-time Packet Observation Tool をインストール

## 概要

[Real-time Packet Observation Tool (RPOT) の README.md](https://github.com/super-a1ice/rpot/blob/master/README.md) をみつつインストールします。本家は適当に端折られていますが、下記はインストール過程に使うコマンドをすべて記録しています（作業途中の出力については適宜端折っています）。

## 環境

解析する pcap ファイルの規模によると思いますが、超絶重いアプリケーションです。

* ubuntu 18.04
* core-i5 メモリ 8G

## インストール

1. docker-compose と必要なパッケージをインストール

    ```shell-session:apt_install_docker-compose
[yamachan@ubuntu ~]$ sudo su
[sudo] yamachan のパスワード:
[root@ubuntu yamachan]# apt update
- 略 -
[root@ubuntu yamachan]# apt upgrade
- 略 -
続行しますか? [Y/n] y
- 略 -
[root@ubuntu yamachan]# apt install docker-compose
パッケージリストを読み込んでいます... 完了
依存関係ツリーを作成しています
状態情報を読み取っています... 完了
以下の追加パッケージがインストールされます:
  bridge-utils cgroupfs-mount docker.io golang-docker-credential-helpers pigz python-asn1crypto python-backports.ssl-match-hostname python-cached-property
  python-certifi python-cffi-backend python-chardet python-cryptography python-docker python-dockerpty python-dockerpycreds python-docopt python-enum34
  python-funcsigs python-functools32 python-idna python-ipaddress python-jsonschema python-mock python-openssl python-pbr python-pkg-resources python-requests
  python-six python-texttable python-urllib3 python-websocket python-yaml ubuntu-fan
提案パッケージ:
  aufs-tools btrfs-progs debootstrap docker-doc rinse python-cryptography-doc python-cryptography-vectors python-enum34-doc python-funcsigs-doc python-mock-doc
  python-openssl-doc python-openssl-dbg python-setuptools python-socks python-ntlm
以下のパッケージが新たにインストールされます:
  bridge-utils cgroupfs-mount docker-compose docker.io golang-docker-credential-helpers pigz python-asn1crypto python-backports.ssl-match-hostname
  python-cached-property python-certifi python-cffi-backend python-chardet python-cryptography python-docker python-dockerpty python-dockerpycreds python-docopt
  python-enum34 python-funcsigs python-functools32 python-idna python-ipaddress python-jsonschema python-mock python-openssl python-pbr python-pkg-resources
  python-requests python-six python-texttable python-urllib3 python-websocket python-yaml ubuntu-fan
アップグレード: 0 個、新規インストール: 34 個、削除: 0 個、保留: 0 個。
42.3 MB のアーカイブを取得する必要があります。
この操作後に追加で 207 MB のディスク容量が消費されます。
続行しますか? [Y/n] y
略
[root@ubuntu yamachan]# exit
exit
```
1. [README.md](https://github.com/super-a1ice/rpot/blob/master/README.md) のとおりスレッドの上限をあげる
 * 詳しくは [Linux におけるスレッド数の上限](http://www.yunabe.jp/docs/maximum_number_of_threads.html) などを参照

    ```shell-session:max_map_count
[yamachan@ubuntu ~]$ cat /proc/sys/vm/max_map_count
65530
[yamachan@ubuntu ~]$ echo 'vm.max_map_count = 262144' | sudo tee -a /etc/sysctl.conf
[sudo] yamachan のパスワード:
vm.max_map_count = 262144
[yamachan@ubuntu ~]$ sudo sysctl -p
vm.max_map_count = 262144
[yamachan@ubuntu ~]$ cat /proc/sys/vm/max_map_count
262144
```
1. ここで docker-compose pull を実行しても失敗します。あたりまえか。

    ```shell-session:git_clone
[yamachan@ubuntu ~]$ docker-compose pull
ERROR:
        Can't find a suitable configuration file in this directory or any
        parent. Are you in the right directory?

        Supported filenames: docker-compose.yml, docker-compose.yaml
```
1. git レポジトリより、rpot を clone します。
2. clone したら、docker-compose pull を実行するも失敗、なぜだ。

    ```shell-session:git_clone
[yamachan@ubuntu ~]$ git clone https://github.com/super-a1ice/rpot.git
Cloning into 'rpot'...
remote: Enumerating objects: 308, done.
remote: Total 308 (delta 0), reused 0 (delta 0), pack-reused 308
Receiving objects: 100% (308/308), 25.44 MiB | 1.92 MiB/s, done.
Resolving deltas: 100% (129/129), done.
　
[yamachan@ubuntu ~]$ cd rpot/
[yamachan@ubuntu rpot]$ docker-compose pull
Pulling zookeeper (wurstmeister/zookeeper:latest)...
ERROR: Couldn't connect to Docker daemon at http+docker://localunixsocket - is it running?
　
If it's at a non-standard location, specify the URL with the DOCKER_HOST environment variable.
```

1. この問題の解決には、単に docker グループに自身を加えればいいという記事が多い中 [If you faced an issue like “Couldn’t connect to Docker daemon at http+docker://localunixsocket — is it running?”…](https://medium.com/@ibrahimgunduz34/if-you-faced-an-issue-like-couldnt-connect-to-docker-daemon-at-http-docker-localunixsocket-is-27b35f17d09d) が親切に説明しているのでその通りに調べてみる
 * まずは docker daemon が動いているかどうか、どうやらまともに動いているっぽい

    ```shell-session:Check_docker_operation
[yamachan@ubuntu rpot]$ sudo service docker status
● docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; disabled; vendor preset: enabled)
   Active: active (running) since Thu 2019-01-10 10:05:33 JST; 19min ago
     Docs: https://docs.docker.com
 Main PID: 26791 (dockerd)
    Tasks: 30
   CGroup: /system.slice/docker.service
           tq26791 /usr/bin/dockerd -H fd://
           mq26815 docker-containerd --config /var/run/docker/containerd/containerd.toml
　
 1月 10 10:05:31 ubuntu dockerd[26791]: time="2019-01-10T10:05:31.803620849+09:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc42023
 1月 10 10:05:31 ubuntu dockerd[26791]: time="2019-01-10T10:05:31.803871119+09:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc42023
 1月 10 10:05:31 ubuntu dockerd[26791]: time="2019-01-10T10:05:31.803955441+09:00" level=info msg="Loading containers: start."
 1月 10 10:05:32 ubuntu dockerd[26791]: time="2019-01-10T10:05:32.523214514+09:00" level=info msg="Default bridge (docker0) is assigned with an IP addre
 1月 10 10:05:32 ubuntu dockerd[26791]: time="2019-01-10T10:05:32.920223182+09:00" level=info msg="Loading containers: done."
 1月 10 10:05:32 ubuntu dockerd[26791]: time="2019-01-10T10:05:32.985440077+09:00" level=warning msg="failed to retrieve docker-runc version: unknown ou
 1月 10 10:05:32 ubuntu dockerd[26791]: time="2019-01-10T10:05:32.989692423+09:00" level=info msg="Docker daemon" commit=e68fc7a graphdriver(s)=zfs vers
 1月 10 10:05:32 ubuntu dockerd[26791]: time="2019-01-10T10:05:32.992411718+09:00" level=info msg="Daemon has completed initialization"
 1月 10 10:05:33 ubuntu dockerd[26791]: time="2019-01-10T10:05:33.160966540+09:00" level=info msg="API listen on /var/run/docker.sock"
 1月 10 10:05:33 ubuntu systemd[1]: Started Docker Application Container Engine.
```
 * 次に socket の権限、自身（yamachan）に権限がないので加える、グループを有効にするためシェルから抜けます

    ```shell-session:Add_to_docker_group
[yamachan@ubuntu rpot]$ sudo ls -la /var/run/docker.sock
srw-rw---- 1 root docker 0  1月 10 10:05 /var/run/docker.sock
[yamachan@ubuntu rpot]$ sudo usermod -aG docker ${USER}
[yamachan@ubuntu rpot]$ exit
exit
```
1.  再びログインして、docker グループに自身（yamachan）が加わっていることを確認して、
1. docker-compose pull （数百メガ引っ張ってきますので待ちます）

    ```shell-session:docker-compose_pull
[yamachan@ubuntu ~]$ id
uid=1000(yamachan) gid=1000(yamachan) groups=1000(yamachan),4(adm),20(dialout),24(cdrom),27(sudo),30(dip),46(plugdev),116(lpadmin),126(sambashare),127(vboxusers),133(docker),10000(admin),10001(public)
[yamachan@ubuntu ~]$ cd rpot/
[yamachan@ubuntu rpot]$ docker-compose pull
Pulling zookeeper (wurstmeister/zookeeper:latest)...
latest: Pulling from wurstmeister/zookeeper
略
Digest: sha256:f8122897f0a30b314234151789cc4b69dc579762ee9a380faa83b67a4b5bad99
Status: Downloaded newer image for tatsui/bro:latest
```

1. docker-compose up manager （立ち上がるまで 90 秒くらいかかります）

    ```shell-session:docker-compose_up_manager
[yamachan@ubuntu rpot]$ docker-compose up manager
elasticsearch is up-to-date
rpot_zookeeper_1 is up-to-date
rpot_kafka_1 is up-to-date
rpot_logstash_1 is up-to-date
Creating rpot_kibana_1 ...
Creating rpot_kibana_1 ... done
Creating rpot_manager_1 ...
Creating rpot_manager_1 ... done
Attaching to rpot_manager_1
略
manager_1        | {"acknowledged":true}rpot_manager_1 exited with code 0
[yamachan@ubuntu rpot]$
```

1. docker-compose up bro ここでは解析対象 pcap ファイルがないと怒られます。
 * そういえば [README.md](https://github.com/super-a1ice/rpot/blob/master/README.md) に **step 1 copy or mount pcap file directory** とかいてありましたね... しかしコピー先の pcap ディレクトリが git clone のときはなかったけど

    ```shell-session:docker-compose_up_bro
[yamachan@ubuntu rpot]$ docker-compose up bro
elasticsearch is up-to-date
rpot_zookeeper_1 is up-to-date
rpot_kafka_1 is up-to-date
rpot_logstash_1 is up-to-date
Creating rpot_bro_1 ...
Creating rpot_bro_1 ... done
Attaching to rpot_bro_1
bro_1            | ls: cannot access '*.pcap': No such file or directory
rpot_bro_1 exited with code 0
```
1. 見てみると、docker-compose up bro 実行時に pcap ディレクトリが作られている

    ```shell-session:Put_a_pcap_file_1
[yamachan@ubuntu rpot]$ ls -la
合計 114
drwxrwxr-x 13 yamachan yamachan    20  1月 10 11:24 .
drwxr-xr-x 51 yamachan yamachan    77  1月 10 11:15 ..
drwxrwxr-x  8 yamachan yamachan    13  1月 10 10:15 .git
-rw-rw-r--  1 yamachan yamachan    64  1月 10 10:15 .gitignore
-rw-rw-r--  1 yamachan yamachan    87  1月 10 10:15 .gitmodules
-rw-rw-r--  1 yamachan yamachan 11337  1月 10 10:15 LICENSE
-rw-rw-r--  1 yamachan yamachan  3129  1月 10 10:15 README.md
drwxrwxr-x 11 yamachan yamachan    11  1月 10 10:15 antivirus
drwxrwxr-x  3 yamachan yamachan     5  1月 10 10:15 bro
drwxrwxr-x  3 yamachan yamachan     3  1月 10 10:15 doc
-rw-rw-r--  1 yamachan yamachan  2034  1月 10 10:15 docker-compose-hunting.yml
-rw-rw-r--  1 yamachan yamachan  4161  1月 10 10:15 docker-compose-scale.yml
-rw-rw-r--  1 yamachan yamachan  2507  1月 10 10:15 docker-compose.yml
drwxr-xr-x  2 root     root         2  1月 10 11:24 extract_files
drwxrwxr-x  3 yamachan yamachan     9  1月 10 10:15 logstash
drwxrwxr-x  4 yamachan yamachan     7  1月 10 10:15 manager
drwxr-xr-x  2 root     root         2  1月 10 11:24 pcap
drwxrwxr-x  4 yamachan yamachan     8  1月 10 10:15 suricata
drwxrwxr-x  3 yamachan yamachan     5  1月 10 10:15 yara
drwxrwxr-x  2 yamachan yamachan     5  1月 10 10:15 yara-gen
```

1. pcap ディレクトリに権限がないので変更し、
2. あらかじめ用意していた解析用ファイル（example.pcap）を pcap ディレクトリに入れ、
3. docker-compose up bro で起動します

    ```shell-session:Put_a_pcap_file_2
[yamachan@ubuntu rpot]$ sudo chown -R yamachan:yamachan pcap extract_files
[yamachan@ubuntu rpot]$ mv ~/example.pcap pcap
[yamachan@ubuntu rpot]$ docker-compose up bro
rpot_zookeeper_1 is up-to-date
elasticsearch is up-to-date
rpot_kafka_1 is up-to-date
rpot_logstash_1 is up-to-date
Starting rpot_bro_1 ...
Starting rpot_bro_1 ... done
Attaching to rpot_bro_1
bro_1            | scan example.pcap standard mode
bro_1            | packet_filter/Log::WRITER_KAFKAWRITER: Debug is turned off.
bro_1            | loaded_scripts/Log::WRITER_KAFKAWRITER: Debug is turned off.
bro_1            | 1544593781.729090 reporter/Log::WRITER_KAFKAWRITER: Debug is turned off.
bro_1            | 1544593781.729090 stats/Log::WRITER_KAFKAWRITER: Debug is turned off.
bro_1            | 1544593784.054183 weird/Log::WRITER_KAFKAWRITER: Debug is turned off.
bro_1            | 1544593787.078383 conn/Log::WRITER_KAFKAWRITER: Debug is turned off.
bro_1            | 1544593791.991236 dns/Log::WRITER_KAFKAWRITER: Debug is turned off.
bro_1            | 1544593828.634234 files/Log::WRITER_KAFKAWRITER: Debug is turned off.
bro_1            | 1544593828.634234 http/Log::WRITER_KAFKAWRITER: Debug is turned off.
bro_1            | 1544593838.324624 capture_loss/Log::WRITER_KAFKAWRITER: Debug is turned off.
bro_1            | WARNING: No Site::local_nets have been defined.  It's usually a good idea to define your local networks.
rpot_bro_1 exited with code 0
```

1. [README.md](https://github.com/super-a1ice/rpot/blob/master/README.md) にはlocalhost をブラウザで見ろ（ttp://localhost:5601）とありますが localhost 以外でも見れないかどう確認します
 * どこからでも OK なようです。

    ```shell-session:Check_access_permission
[yamachan@ubuntu rpot]$ sudo lsof -i | grep 5601
docker-pr 26590            root    4u  IPv6  66894      0t0  TCP *:5601 (LISTEN)
```

1. 現在の IP アドレスを調べてブラウザから ttp://192.168.1.199:5601（私の場合）にアクセスします

    ```
[yamachan@ubuntu rpot]$ ifconfig | grep -1 eno1
　
eno1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.1.199  netmask 255.255.255.0  broadcast 192.168.1.255
```

3. kibana の画面が出てきます
4. とめます

    ````shell-session:docker-compose_down
[yamachan@ubuntu rpot]$ docker-compose down -v
Stopping rpot_logstash_1  ... done
Stopping rpot_kafka_1     ... done
Stopping rpot_kibana_1    ... done
Stopping rpot_zookeeper_1 ... done
Stopping elasticsearch    ... done
Removing rpot_bro_1       ... done
Removing rpot_manager_1   ... done
Removing rpot_logstash_1  ... done
Removing rpot_kafka_1     ... done
Removing rpot_kibana_1    ... done
Removing rpot_zookeeper_1 ... done
Removing elasticsearch    ... done
Removing network rpot_frontend
Removing network rpot_backend
Removing volume rpot_rules-data
Removing volume rpot_json-data
Removing volume rpot_es-data
```