Lagopus Routerで iOS/Android VPN接続できる IPsecサーバーを構築する

#イメージ図
![lagopus-router-ipsec-network.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/134927/15cc3677-03f4-215a-2ffa-52d2d8d5b274.png)

#目的
Lagopus Routerに IPsec終端機能が実装されているので、スマホ端末を接続できるような IPsec/XAUTH PSKサーバーを構築してみました。

#Lagopus Routerとは
[O3プロジェクト](http://o3project.github.io/)で開発され、OSS公開されている高速パケット処理が可能なソフトウェアルーター。 
I/OにDPDKを採用し、スペックシートでは IPsecスループット 10Gbps(!)、 2,048セッション(!!) と記載されている。 

詳しくは、リンク先を参照ください。[[\*1]](http://www.lagopus.org/)[[\*2]](https://www.slideshare.net/tomoyahibi/lagopus-router-v19071)


#セットアップ環境
使用したOS, ソフトウェア

- OS / HWスペック
 - Ubuntu 16.04.3 LTS
 - CPU 4コア, Memory 4GB, NIC * 2

- Software
 - Lagopus Router v19.07.1
 - DPDK 18.11.5
 - Go v1.10.4
 - vsw (Dataplane of Lagopus Router)
 - openconfigd
 - iked
 - strongSwan
 - Zebra/VRF library
 
#インストール手順
インストール手順は [Lagopus software router book [\*3]](http://www.lagopus.org/lagopus-book/en/router/html/) を参考におこないます。


##1, DPDKのインストール
vsw (Dataplane of Lagopus router)は DPDK 18.11.1以上が必要のため、現時点のDPDK 18系最新の DPDK 18.11.5をインストールする。

```
$ wget http://static.dpdk.org/rel/dpdk-18.11.5.tar.xz
$ tar xvf dpdk-18.11.5.tar.xz
$ cd dpdk-stable-18.11.5/
~dpdk-stable-18.11.5$ cp config/common_base config/common_base.original
~dpdk-stable-18.11.5$ vi config/common_base
CONFIG_RTE_BUILD_SHARED_LIB=y
CONFIG_RTE_IXGBE_INC_VECTOR=n
CONFIG_RTE_LIBRTE_PMD_BOND=n
CONFIG_RTE_LIBRTE_PMD_OPENSSL=y

~dpdk-stable-18.11.5$ make T=x86_64-native-linuxapp-gcc config
~dpdk-stable-18.11.5$ make
~dpdk-stable-18.11.5$ sudo make instlal
```

##2, Goのインストール
```
$ sudo apt install golang-go go-dep
$ vi ~/.bashrc
export GOPATH=$HOME/go
export PATH=$GOPATH/bin:$PATH

~$ mkdir -p go/src
~$ mkdir -p go/pkg
~$ mkdir -p go/bin
```

##3, openconfigd, cliのインストール
openconfigd/README.md を参考にすすめる

```
$ go get github.com/coreswitch/openconfigd/openconfigd
$ go get github.com/coreswitch/openconfigd/cli_command
$ cd $GOPATH/src/github.com/coreswitch/openconfigd/cli
$ ./configure
$ make
$ sudo make install
$ cd $GOPATH/src/github.com/coreswitch/openconfigd/bash_completion.d
$ sudo cp cli /etc/bash_completion.d/
```

##4, vsw(Dataplane of Lagopus router)のインストール
lagopus/vsw/README.md を参考にしてすすめる

```
$ mkdir -p go/src/github.com/lagopus
$ cd go/src/github.com/lagopus
$ ~/go/src/github.com/lagopus$ git clone https://github.com/lagopus/vsw.git

% git clone https://github.com/lagopus/utils ~/utils
% cd ~/utils/
% ./configure
% make
% sudo make install

~/go/src/github.com/lagopus/vsw$ make 
~/go/src/github.com/lagopus/vsw$ sudo make install 
~/go/src/github.com/lagopus/vsw$ go install
```

##5, Lagopus Routerのインストール

```
$ cd ~/go/src/github.com/lagopus/
~/go/src/github.com/lagopus$ git clone https://github.com/lagopus/lagopus-router.git
```

##6, Lagopus iked/strongSwanのインストール
```
必要ディレクトリの作成
~$ sudo mkdir -p /usr/local/ipsec/etc/ipsec.d/run
~$ sudo mkdir -p /usr/local/ipsec/etc/ipsec.d/cacerts
~$ sudo mkdir -p /usr/local/ipsec/etc/ipsec.d/certs
~$ sudo mkdir -p /usr/local/ipsec/etc/ipsec.d/private

~$ git clone https://github.com/lagopus/iked.git
~$ cd iked/strongswan
~iked/strongswan$ ./autogen.sh
~iked/strongswan$ ./configure --enable-openssl --enable-lagopus-pfkey --enable-xauth-eap --enable-eap-radius --sysconfdir=/usr/local/ipsec/etc --with-piddir=/usr/local/ipsec/etc/ipsec.d/run
~iked/strongswan$ make, sudo make install
```

##7, Lagopus iked/ipsec_vrfのインストール
```
~$ cd iked/ipsec_vrf
~iked/ipsec_vrf$ sudo make install
```

##8, ZebraのVRF Library(vrf_socket.so)のインストール
```
~$ git clone https://github.com/coreswitch/zebra
~$ cd zebra/tools/vrf
~zebra/tools/vrf$ make
~zebra/tools/vrf$ sudo mkdir -p /usr/local/ipsec/lib
~zebra/tools/vrf$ sudo cp vrf_socket.so /usr/local/ipsec/lib
```


#設定方法
設定ファイルをそれぞれ以下のパスに作成します。

- openconfigd
 - /usr/local/etc/openconfigd.conf
- vsw
 - /usr/local/etc/vsw.conf
- iked/strongswan
 - /usr/local/ipsec/etc/strongswan.conf
 - /usr/local/ipsec/etc/strongswan.d/charon/socket-default.conf
- ipsec
 - /usr/local/ipsec/etc/ipsec.conf
 - /usr/local/ipsec/etc/ipsec.secrets
- vrf (vrf1を作成する場合)
 - /usr/local/ipsec/vrf1/strongswan.d/charon.conf
 - /usr/local/ipsec/vrf1/strongswan.d/charon/lagopus-pfkey.conf

##1, openconfigd 設定
事前に設定ファイルの作成は必要なし。後述の openconfigdを起動後 cliから設定していく。

##2, vsw 設定
以下の設定ファイルを作成編集する。

- /usr/local/etc/vsw.conf

デフォルト設定から以下の部分を変更する。 (デフォルト設定は $GOPATH/src/github.com/lagopus/vsw/vsw.conf)
- [dpdk] core_mask をCPUコア数に合わせて変更
- [dpdk] vdevs=["crypto_openssl"], [agents] enable=[ "ipsec" ], [router] rr_process_mode = "enable" を追加。

```/usr/local/etc/vsw.conf
# VSW configuration file
[agents]
enable = [ "netlink", "config", "debugsh", "ipsec" ]

# Logging settings
[logging]
log = "stdout" # "file", "stdout", or "syslog"

# DPDK configuration section
[dpdk]
core_mask = 0x0e
# core_list = "1,2,3,4,5,6,7"
memory_channel = 2
pmd_path = "/usr/local/lib"
num_elements = 131072
cache_size = 256
vdevs=["crypto_openssl"] 

# Openconfigd section
[openconfig]
server_host = "localhost"       # Openconfigd server host
server_port = 2650              # Openconfigd server port
listen_port = 2653              # Port to listen for show command

# ethdev configuration section
[ethdev]
rx_core = 2     # Slave core to use for RX
tx_core = 3     # Slave core to use for TX

# Set force_linearize to true, if the link device doesn't
# support multi-segment mbuf.
force_linearize = false

# bridge configuration section
[bridge]
core = 2
# RIF configuration section
[rif]
core = 3

# tunnel configuration section
[tunnel]
  # IPsec tunnel
  [tunnel.ipsec]
  # Enable/Disable CPU core binding for IPsec.
  core_bind = false
  # Set CPU core mask.
  # It is better not to overlap with dpdk.core_mask.
  inbound_core_mask = 0x10  # core ID: 4
  outbound_core_mask = 0x20 # core ID: 5

  # IP in IP tunnel
  [tunnel.ipip]
  inbound_core = 2
  outbound_core = 3

  # GRE tunnel
  [tunnel.gre]
  inbound_core = 2
  outbound_core = 3

  # L2GRE tunnel
  [tunnel.l2gre]
  inbound_core = 2
  outbound_core = 3

  # VXLAN tunnel
  [tunnel.vxlan]
  inbound_core = 2
  outbound_core = 3
  aging_time = 300

# router configuration section
#
# rr_process_mode is processing mode for IPv4 options that require
# to record a route. The following modes are available.
#   * enable: process and forward packets as per RFC791
#   * disable: drops packets
#   * ignore: forward packets as if they did not contain the options
[router]
core = 3
rr_process_mode = "enable"
```

##3, strongSwan 設定
以下の設定ファイルを作成編集する。

- /usr/local/ipsec/strongswan.conf
- /usr/local/ipsec/etc/strongswan.d/charon/socket-default.conf

```/usr/local/ipsec/strongswan.conf
# strongswan.conf - strongSwan configuration file
#
# Refer to the strongswan.conf(5) manpage for details
#
# Configuration changes should be made in the included files

charon {
        load_modular = yes
        plugins {
                include strongswan.d/charon/*.conf
        }
}

include strongswan.d/*.conf
```
```/usr/local/ipsec/etc/strongswan.d/charon/socket-default.conf
socket-default {

    # Firewall mark to set on outbound packets.
    # fwmark =

    # Whether to load the plugin. Can also be an integer to increase the
    # priority of this plugin.
    load = yes

    # Set source address on outbound packets, if possible.
    # set_source = yes

    # Force sending interface on outbound packets, if possible.
    set_sourceif = yes    # ← yesに変更

    # Listen on IPv4, if possible.
    # use_ipv4 = yes

    # Listen on IPv6, if possible.
    # use_ipv6 = yes

}
```

##4, vrf(vrf1)向け strongSwan/ipsec 設定
/usr/local/ipsec/etc を /usr/local/ipsec/vrf1 へコピーしてから、以下の設定ファイルを作成編集する。

- /usr/local/ipsec/vrf1/ipsec.conf
- /usr/local/ipsec/vrf1/ipsec.secrets
- /usr/local/ipsec/vrf1/strongswan.d/charon.conf
- /usr/local/ipsec/vrf1/strongswan.d/charon/lagopus-pfkey.conf

```/usr/local/ipsec/vrf1/ipsec.conf
#ipsec.conf - strongSwan IPsec configuration file
config setup
#         charondebug="cfg 2, dmn 2, ike 2, net 2"

conn ipsec-xauth
    auto=add
    dpdaction=clear
    keyexchange=ikev1
    ike=aes128-sha1-modp1024
    esp=aes128-sha1
    left=%any
    leftauth=psk
    leftsubnet=0.0.0.0/0
    leftid=203.0.113.1
    right=%any
    rightsourceip=192.0.2.0/24
    rightdns=8.8.8.8
    dpdaction=clear
    rightauth=psk
    rightauth2=xauth

```
```/usr/local/ipsec/vrf1/ipsec.secrets
: PSK "lagopus"
test1 : XAUTH "test1-lagopus"
test2 : XAUTH "test2-lagopus"
```

```/usr/local/ipsec/vrf1/strongswan.d/charon.conf
..
    # A comma-separated list of network interfaces that should be used by
    # charon. All other interfaces are ignored.
    # interfaces_use =
    interfaces_use = if0-0    # 鍵交換に使うインターフェースを追加
..
```
```/usr/local/ipsec/vrf1/strongswan.d/charon/lagopus-pfkey.conf
lagopus-pfkey {

    # Size of the receive buffer for the event socket (0 for default size).
    # events_buffer_size = 0

    # Whether to load the plugin. Can also be an integer to increase the
    # priority of this plugin.
    load = yes

    # UNIX domain socket path for lagopus
    socket_path = /var/tmp/lagopus-vrf1.sock # ← 変更

}
```


#起動方法
##事前準備
DPDKのセットアップ、NICバインドをおこなう。

###1, DPDKのセットアップ
DPDKを使うためにはHugepagesの有効化をおこなう。

Hugepagesの設定 & reboot

```
$ sudo vi /etc/sysctl.conf
vm.nr_hugepages = 1024
$ sudo mkdir -p /mnt/huge
$ sudo vi /etc/fstab
nodev /mnt/huge hugetlbfs defaults 0 0
$ sudo reboot
..
```
Hugepagesの有効化確認

```
$ grep -i "HugePages" /proc/meminfo
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
HugePages_Total:    1024
HugePages_Free:     1024
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB

$ mount | grep huge
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
nodev on /mnt/huge type hugetlbfs (rw,relatime,pagesize=2M)
```

###2, NICを DPDKへバインド
- kernel moduleの有効化 (uio, DPDKのigp_uio.ko)
- if downさせてからバインドする

```
~dpdk-stable-18.11.5$ sudo -s
~dpdk-stable-18.11.5# modprobe uio
~dpdk-stable-18.11.5# insmod build/kmod/igb_uio.ko
~dpdk-stable-18.11.5# lsmod | grep uio
igb_uio                16384  2
uio                    20480  5 igb_uio

~dpdk-stable-18.11.5# ip link set ens160 down
~dpdk-stable-18.11.5# ip link set ens192 down
~dpdk-stable-18.11.5# ./usertools/dpdk-devbind.py --bind=igb_uio ens160 ens192
```

###3, NICがDPDKに適用されたかを確認
```
~dpdk-stable-18.11.5# ./usertools/dpdk-devbind.py -s

Network devices using DPDK-compatible driver
============================================
0000:03:00.0 'VMXNET3 Ethernet Controller 07b0' drv=igb_uio unused=vmxnet3
0000:0b:00.0 'VMXNET3 Ethernet Controller 07b0' drv=igb_uio unused=vmxnet3

Network devices using kernel driver
===================================
0000:13:00.0 'VMXNET3 Ethernet Controller 07b0' if=ens224 drv=vmxnet3 unused=igb_uio *Active*
```

##起動
openconfigdの起動 → vswの起動 → cliでinterface, vrf設定 → ipsec daemonの順番で起動する。
注意事項は、
- vsw起動の前に Openconfigdが動いていないと設定が読み込めない。
- ipsec daemonを起動する前に、interface, vrf設定がされていないと、起動できないためです。

###1, openconfigdの起動

```
$ cd ~/go/src/github.com/lagopus/lagopus-router/yang
~/go/src/github.com/lagopus/lagopus-router/yang$ sudo openconfigd -y modules:modules/policy:modules/bgp:modules/interfaces:modules/local-routing:modules/vlan:modules/rib:modules/network-instance:modules/types lagopus-router.yang
```

###2, vswの起動

```
$ sudo -s
# DPDKDIR=~/dpdk-stable-18.11.5
 env LD_LIBRARY_PATH=$DPDKDIR/build/lib vsw
```

###3, openconfigd cliから interface, vrf設定
commitすると設定が適用され、設定ファイル /usr/local/etc/openconfigd.conf に保存される

```
$ cli
> config
# set interfaces interface if0
# set interfaces interface if0 config device "0000:03:00.0"
# set interfaces interface if0 config driver dpdk
# set interfaces interface if0 config enabled true
# set interfaces interface if0 config type ethernetCsmacd
# set interfaces interface if0 subinterfaces subinterface 0
# set interfaces interface if0 subinterfaces subinterface 0 config enabled true
# set interfaces interface if0 subinterfaces subinterface 0 ipv4 addresses address 203.0.113.1
# set interfaces interface if0 subinterfaces subinterface 0 ipv4 addresses address 203.0.113.1 config prefix-length 24
# set interfaces interface if1
# set interfaces interface if1 config device "0000:0b:00.0"
# set interfaces interface if1 config driver dpdk
# set interfaces interface if1 config enabled true
# set interfaces interface if1 config type ethernetCsmacd
# set interfaces interface if1 subinterfaces subinterface 0
# set interfaces interface if1 subinterfaces subinterface 0 config enabled true
# set interfaces interface if1 subinterfaces subinterface 0 ipv4 addresses address 198.51.100.1
# set interfaces interface if1 subinterfaces subinterface 0 ipv4 addresses address 198.51.100.1 config prefix-length 24
# set network-instances network-instance vrf1
# set network-instances network-instance vrf1 interfaces interface if0
# set network-instances network-instance vrf1 interfaces interface if0 subinterface 0
# set network-instances network-instance vrf1 interfaces interface if1
# set network-instances network-instance vrf1 interfaces interface if1 subinterface 0
# set network-instances network-instance vrf1 config enabled true
# set network-instances network-instance vrf1 config enabled-address-families IPV4
# set network-instances network-instance vrf1 config type L3VRF
# commit
```


###4, ipsec daemonの起動

```
$ sudo /usr/local/ipsec/bin/ipsec_vrf start
$ sudo /usr/local/ipsec/bin/ipsec_vrf statusall
vrf1
Status of IKE charon daemon (strongSwan 5.6.2, Linux 4.15.0-74-generic, x86_64):
  uptime: 1 days, since Jan 16 16:11:34 2020
  malloc: sbrk 2686976, mmap 0, used 913488, free 1773488
  worker threads: 11 of 16 idle, 5/0/0/0 working, job queue: 0/0/0/0, scheduled: 20
  loaded plugins: charon aes des rc2 sha2 sha1 md5 mgf1 random nonce x509 revocation constraints pubkey pkcs1 pkcs7 pkcs8 pkcs12 pgp dnskey sshkey pem openssl fips-prf gmp curve25519 xcbc cmac hmac attr kernel-netlink resolve socket-default stroke vici updown eap-radius xauth-generic xauth-eap counters
Virtual IP pools (size/online/offline):
  192.0.2.0/24: 254/0/1
Listening IP addresses:
  203.0.113.1
Connections:
 ipsec-xauth:  %any...%any  IKEv1, dpddelay=3s
 ipsec-xauth:   local:  [203.0.113.1] uses pre-shared key authentication
 ipsec-xauth:   remote: uses pre-shared key authentication
 ipsec-xauth:   remote: uses XAuth authentication: any
 ipsec-xauth:   child:  0.0.0.0/0 === dynamic TUNNEL, dpdaction=clear
Security Associations (0 up, 0 connecting):
  none
```


##その他設定
###1, 追加設定
- Policy Based Routingは現状ARP解決の機能がないため、Static ARPを設定。
- vrfのrouting設定を追加。

```
$ sudo arp -s 203.0.113.2 00:00:5e:00:53:00 -i if0-0
$ sudo arp -s 198.51.100.2 00:00:5e:00:53:ff -i if1-0
$ arp -na

$ sudo ip rule add iif vrf1 from 192.0.2.0/24 table 2
$ sudo ip route add table 1 default via 203.0.113.2 dev if0-0
$ sudo ip route add table 2 default via 198.51.100.2 dev if1-0
$ ip rule
0:      from all lookup local 
219:    from 192.0.2.0/24 iif vrf1 lookup 2 
220:    from all lookup 220 
1000:   from all lookup [l3mdev-table] 
1000:   from all iif vrf1 lookup 1 
1000:   from all oif vrf1 lookup 1 
32766:  from all lookup main 
32767:  from all lookup default
$ routel
```

#スマホ設定
- iOS : 設定 → VPN
- Android : 設定 → ネットワークとインタネット → 詳細設定 → VPN
![device-ipsec-config.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/134927/fd44174d-54c6-4dc1-7976-238e67975f24.png)


# 完成
接続できたところで満足して、細かいテストなどはまだおこなっておりません。。
ご興味がある方は、色々と試してみてください。
コメントなど歓迎します。

# 参考資料
*1 [Lagopus Switch and Router - http://www.lagopus.org/](http://www.lagopus.org/)
*2 [Lagopus Router v19.07.1 SlideShare - https://www.slideshare.net/tomoyahibi/lagopus-router-v19071](https://www.slideshare.net/tomoyahibi/lagopus-router-v19071)
*3 [Lagopus software router book - http://www.lagopus.org/lagopus-book/en/router/html/](http://www.lagopus.org/lagopus-book/en/router/html/)
[](
*4 [FD.io VPP/IPSec and IKEv2](https://wiki.fd.io/view/VPP/IPSec_and_IKEv2)
*5 [TNSR - TNSR Secure Networking Software Platform - Router VPN Firewall](https://www.tnsr.com/)
)