Open5GS 5GCとUERANSIM UE/RANでDPDK対応VPP-UPFを試す

本記事では、ローカル5G SAを想定して、Open5GS 5GCとVPP-UPFを連携動作させるための簡単な構成について説明します。 DPDK対応VPP-UPFのインストールと設定については、[こちら](https://qiita.com/s5uishida/items/fc5f4e1c394c9f5dd181)を参照してください。

---

## Open5GS 5GCのシミュレーション携帯網の概要

今回、Open5GS 5GCのC-Plane、VPP-UPF、およびData Network Gatewayの簡単な構成について説明します。**この構成はVirtualbox VMを使用して実装されることに注意して下さい。**
このシミュレーション環境は以下の条件を満たす最小構成になります。

- 1つのUPFとData Network Gateway
- 1つのUEと1つのDNN

構築したシミュレーション環境は以下の通りです。

<img width="800" alt="network-overview.png" src="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/457466/801a2935-6d7e-9395-11e6-7cf9cb96fe9f.png">

5GC、VPP-UPFとUE/RANに使用しているOSSは以下の通りです。

- 5GC - Open5GS v2.6.4 (2023.06.14) - https://github.com/open5gs/open5gs
- VPP-UPF - OpenAir CN 5G for UPF v1.5.1 (2023.06.14) - https://gitlab.eurecom.fr/oai/cn5g/oai-cn5g-upf-vpp
- UE / RAN - UERANSIM v3.2.6 (2023.06.14) - https://github.com/aligungr/UERANSIM

各VMは以下の通りです。

| VM | SW & Role | IP address | OS | CPU<br>(Min) | Memory<br>(Min) | HDD<br>(Min) |
| --- | --- | --- | --- | --- | --- | --- |
| VM1 | Open5GS 5GC<br>C-Plane | 192.168.0.111/24 | Ubuntu<br>22.04 | 1 | 1GB | 20GB |
| VM-UP | OpenAir CN 5G<br>for UPF | 192.168.0.151/24 | Ubuntu<br>22.04 | 2 | 8GB | 20GB |
| VM-DN | Data Network<br>Gateway  | 192.168.0.152/24 | Ubuntu<br>22.04 | 1 | 1GB | 10GB |
| VM2 | UERANSIM RAN<br>(gNodeB) | 192.168.0.131/24 | Ubuntu<br>22.04 | 1 | 1GB | 10GB |
| VM3 | UERANSIM UE | 192.168.0.132/24 | Ubuntu<br>22.04 | 1 | 1GB | 10GB |

各VMのネットワークインターフェースは以下の通りですが、**この内、DPDKの制御下に入るデバイスを有効(UP)しないで下さい。これらのインターフェースについてはVPP-UPFの`init.conf`ファイルにIPアドレスと合わせて設定しており、VPP-UPF起動時にこれらのデバイスは有効になります。**

| VM | Device | Network<br>Adapter | IP address | Interface | Under<br>DPDK |
| --- | --- | --- | --- | --- | --- |
| VM1 | enp0s3 | NAT(default) | 10.0.2.15/24 | (VM default NW) | -- |
| | enp0s8 | Bridged Adapter | 192.168.0.111/24 | (Mgmt NW) | -- |
| | enp0s9 | NAT Network | 192.168.14.111/24 | N4 | -- |
| VM-UP | enp0s3 | NAT(default) | 10.0.2.15/24 | (VM default NW) | -- |
| | enp0s8 | Bridged Adapter | 192.168.0.151/24 | (Mgmt NW) | -- |
| | enp0s9 | NAT Network | 192.168.13.151/24 | N3 | x |
| | enp0s10 | NAT Network | 192.168.14.151/24 | N4 | x |
| | enp0s16 | NAT Network | 192.168.16.151/24 | N6 | x |
| VM-DN | enp0s3 | NAT(default) | 10.0.2.15/24 | (VM default NW) | -- |
| | enp0s8 | Bridged Adapter | 192.168.0.152/24 | (Mgmt NW) | -- |
| | enp0s9 | NAT Network | 192.168.16.152/24 | N6 | -- |
| VM2 | enp0s3 | NAT(default) | 10.0.2.15/24 | (VM default NW) | -- |
| | enp0s8 | Bridged Adapter | 192.168.0.131/24 | (Mgmt NW) | -- |
| | enp0s9 | NAT Network | 192.168.13.131/24 | N3 | -- |
| VM3 | enp0s3 | NAT(default) | 10.0.2.15/24 | (VM default NW) | -- |
| | enp0s8 | Bridged Adapter | 192.168.0.132/24 | (Mgmt NW) | -- |

5GのNF間のインターフェースの内、N3、N4、N6については、VirtualboxのNATネットワークで以下の通りに設定しました。

| Network Name | Network CIDR |
| --- | --- |
| N3 | 192.168.13.0/24 |
| N4 | 192.168.14.0/24 |
| N6 | 192.168.16.0/24 |

network instanceを`internet`に設定しています。

| Network Instance |
| --- |
| internet |

加入者情報は以下の通りです。(その他の項目はデフォルト値)
**UERANSIMのUEの設定に合わせて、OPあるいはOPcを選択してください。**

| UE | IMSI | DNN | OP/OPc |
| --- | --- | --- | --- |
| UE | 001010000000000 | internet | OPc |

これらの情報の登録には、Open5GSが提供するWebUIを使用しました。なお、3GPP認証とキー生成機能(MILENAGE)の試験データとして、[3GPP TS 35.208](https://www.3gpp.org/DynaReport/35208.htm) "4.3 Test Sets"が公開されています。

DNは以下の通りです。

| DN | DNN | TUNnel interface of UE |
| --- | --- | --- |
| 10.45.0.0/16 | internet | uesimtun0 |

## Open5GS 5GC、VPP-UPFとUERANSIM UE/RANの設定ファイルの変更点

Open5GS、VPP-UPFとUERANSIMのビルド方法は以下を参照して下さい。

- Open5GS v2.6.4 (2023.06.14) - https://open5gs.org/open5gs/docs/guide/02-building-open5gs-from-sources/
- OpenAir CN 5G for UPF v1.5.1 (2023.09.05) - https://qiita.com/s5uishida/items/fc5f4e1c394c9f5dd181
- UERANSIM v3.2.6 (2023.06.14) - https://github.com/aligungr/UERANSIM/wiki/Installation

### Open5GS 5GC C-Planeの設定ファイルの変更点

Open5GSでは、PFCPの接続先としてUPFを選択するロジックに使用可能なパラメータは以下の通りです。PFCP(Packet Forwarding Control Protocol)とは、5GCでCU分離のために導入されたプロトコルです。

- DNN
- TAC (Tracking Area Code)
- nr_CellID

なお、TACとnr_CellIDはRANの設定項目です。話を簡単にするために、今回はDNNのみ使用しています。Open5GSでUPFを選択するロジックについては、[こちら](https://github.com/open5gs/open5gs/pull/560#issue-483001043)を参照して下さい。

- `open5gs/install/etc/open5gs/amf.yaml`

```diff
--- amf.yaml.orig       2023-06-15 21:34:19.913862330 +0900
+++ amf.yaml    2023-06-15 21:36:01.887014131 +0900
@@ -474,26 +474,26 @@
       - addr: 127.0.0.5
         port: 7777
     ngap:
-      - addr: 127.0.0.5
+      - addr: 192.168.0.111
     metrics:
       - addr: 127.0.0.5
         port: 9090
     guami:
       - plmn_id:
-          mcc: 999
-          mnc: 70
+          mcc: 001
+          mnc: 01
         amf_id:
           region: 2
           set: 1
     tai:
       - plmn_id:
-          mcc: 999
-          mnc: 70
+          mcc: 001
+          mnc: 01
         tac: 1
     plmn_support:
       - plmn_id:
-          mcc: 999
-          mnc: 70
+          mcc: 001
+          mnc: 01
         s_nssai:
           - sst: 1
     security:
```

- `open5gs/install/etc/open5gs/smf.yaml`

```diff
--- smf.yaml.orig       2023-06-15 21:34:19.886859373 +0900
+++ smf.yaml    2023-06-15 21:36:14.354599217 +0900
@@ -602,25 +602,20 @@
       - addr: 127.0.0.4
         port: 7777
     pfcp:
-      - addr: 127.0.0.4
-      - addr: ::1
+      - addr: 192.168.14.111
     gtpc:
       - addr: 127.0.0.4
-      - addr: ::1
     gtpu:
-      - addr: 127.0.0.4
-      - addr: ::1
+      - addr: 192.168.14.111
     metrics:
       - addr: 127.0.0.4
         port: 9090
     subnet:
       - addr: 10.45.0.1/16
-      - addr: 2001:db8:cafe::1/48
+        dnn: internet
     dns:
       - 8.8.8.8
       - 8.8.4.4
-      - 2001:4860:4860::8888
-      - 2001:4860:4860::8844
     mtu: 1400
     ctf:
       enabled: auto
@@ -808,7 +803,8 @@
 #
 upf:
     pfcp:
-      - addr: 127.0.0.7
+      - addr: 192.168.14.151
+        dnn: internet
 
 #
 #  o Disable use of IPv4 addresses (only IPv6)
```

### VPP-UPFの設定ファイルの変更点

元の設定ファイルは、[こちら](https://qiita.com/s5uishida/items/fc5f4e1c394c9f5dd181#%E8%A8%AD%E5%AE%9A%E3%83%95%E3%82%A1%E3%82%A4%E3%83%AB%E3%82%92%E4%BD%9C%E6%88%90)を参照して下さい。

- `openair-upf/startup.conf`  
元ファイルからの変更はありません。

- `openair-upf/init.conf`  
元ファイルからの変更はありません。

### UERANSIM UE/RANの設定ファイルの変更点

#### RANの設定ファイルの変更点

- `UERANSIM/config/open5gs-gnb.yaml`

```diff
--- open5gs-gnb.yaml.orig       2022-07-03 13:06:44.000000000 +0900
+++ open5gs-gnb.yaml    2023-06-18 21:25:34.307164598 +0900
@@ -1,17 +1,17 @@
-mcc: '999'          # Mobile Country Code value
-mnc: '70'           # Mobile Network Code value (2 or 3 digits)
+mcc: '001'          # Mobile Country Code value
+mnc: '01'           # Mobile Network Code value (2 or 3 digits)
 
 nci: '0x000000010'  # NR Cell Identity (36-bit)
 idLength: 32        # NR gNB ID length in bits [22...32]
 tac: 1              # Tracking Area Code
 
-linkIp: 127.0.0.1   # gNB's local IP address for Radio Link Simulation (Usually same with local IP)
-ngapIp: 127.0.0.1   # gNB's local IP address for N2 Interface (Usually same with local IP)
-gtpIp: 127.0.0.1    # gNB's local IP address for N3 Interface (Usually same with local IP)
+linkIp: 192.168.0.131   # gNB's local IP address for Radio Link Simulation (Usually same with local IP)
+ngapIp: 192.168.0.131   # gNB's local IP address for N2 Interface (Usually same with local IP)
+gtpIp: 192.168.13.131    # gNB's local IP address for N3 Interface (Usually same with local IP)
 
 # List of AMF address information
 amfConfigs:
-  - address: 127.0.0.5
+  - address: 192.168.0.111
     port: 38412
 
 # List of supported S-NSSAIs by this gNB
```

#### UE(IMSI-001010000000000)の設定ファイルの変更点

- `UERANSIM/config/open5gs-ue.yaml`

```diff
--- open5gs-ue.yaml.orig        2023-05-10 19:00:38.000000000 +0900
+++ open5gs-ue.yaml     2023-06-15 21:42:14.363706123 +0900
@@ -1,9 +1,9 @@
 # IMSI number of the UE. IMSI = [MCC|MNC|MSISDN] (In total 15 digits)
-supi: 'imsi-999700000000001'
+supi: 'imsi-001010000000000'
 # Mobile Country Code value of HPLMN
-mcc: '999'
+mcc: '001'
 # Mobile Network Code value of HPLMN (2 or 3 digits)
-mnc: '70'
+mnc: '01'
 # SUCI Protection Scheme : 0 for Null-scheme, 1 for Profile A and 2 for Profile B
 protectionScheme: 0
 # Home Network Public Key for protecting with SUCI Profile A
@@ -28,7 +28,7 @@
 
 # List of gNB IP addresses for Radio Link Simulation
 gnbSearchList:
-  - 127.0.0.1
+  - 192.168.0.131
 
 # UAC Access Identities Configuration
 uacAic:
```

## Open5GS 5GC、VPP-UPFとUERANSIM UE/RANのネットワーク設定

### VPP-UPFとData Network Gatewayのネットワーク設定

[こちら](https://qiita.com/s5uishida/items/fc5f4e1c394c9f5dd181#vm-up%E3%81%ABdpdk%E5%AF%BE%E5%BF%9Cvpp-upf%E3%82%92%E3%82%BB%E3%83%83%E3%83%88%E3%82%A2%E3%83%83%E3%83%97)と[こちら](https://qiita.com/s5uishida/items/fc5f4e1c394c9f5dd181#vm-dn%E3%81%A7data-network-gateway%E3%82%92%E8%A8%AD%E5%AE%9A)を参照して下さい。

## Open5GS、VPP-UPFとUERANSIMのビルド

Open5GS、VPP-UPFとUERANSIMのビルド方法は以下を参考にして下さい。

- Open5GS v2.6.4 (2023.06.14) - https://open5gs.org/open5gs/docs/guide/02-building-open5gs-from-sources/
- OpenAir CN 5G for UPF v1.5.1 (2023.09.05) - https://qiita.com/s5uishida/items/fc5f4e1c394c9f5dd181
- UERANSIM v3.2.6 (2023.06.14) - https://github.com/aligungr/UERANSIM/wiki/Installation

Open5GS 5GC C-PlaneマシンにMongoDBをインストールして下さい。なお、MongoDBのデータベースを簡単に確認するには、[MongoDB Compass](https://www.mongodb.com/products/compass)が便利です。

## Open5GS 5GC、VPP-UPFとUERANSIM UE/RANを実行

最初にVPP-UPFを起動し、次にOpen5GS 5GC、UERANSIM(UEとRANのシミュレータ)を起動します。

### VPP-UPFを実行

[こちら](https://qiita.com/s5uishida/items/fc5f4e1c394c9f5dd181#vm-up%E3%81%A7dpdk%E5%AF%BE%E5%BF%9Cvpp-upf%E3%82%92%E5%AE%9F%E8%A1%8C)を参考にして下さい。

### Open5GS 5GC C-Planeを実行

```
./install/bin/open5gs-nrfd &
sleep 2
./install/bin/open5gs-scpd &
sleep 2
./install/bin/open5gs-amfd &
sleep 2
./install/bin/open5gs-smfd &
./install/bin/open5gs-ausfd &
./install/bin/open5gs-udmd &
./install/bin/open5gs-udrd &
./install/bin/open5gs-pcfd &
./install/bin/open5gs-nssfd &
./install/bin/open5gs-bsfd &
```

VPP-UPFとOpen5GS SMFのPFCP associationのステータスは次のように表示されます。

```
vpp# show upf association 
Node: 192.168.14.111
  Recovery Time Stamp: 2023/06/18 22:26:01:000
  Sessions: 0
vpp#
```

### UERANSIMを実行

ここでは、UE(IMSI-001010000000000)とRANを実行する場合を説明します。まず、gNodeBと5GCの間でNG-Setupし、5GCにUEを登録し、PDUセッションを確立します。

UERANSIMの使い方は以下を参照して下さい。

- https://github.com/aligungr/UERANSIM/wiki/Usage

#### gNBを起動

次のようにgNBを起動します。

```
# ./nr-gnb -c ../config/open5gs-gnb.yaml
UERANSIM v3.2.6
[2023-06-18 22:27:02.521] [sctp] [info] Trying to establish SCTP connection... (192.168.0.111:38412)
[2023-06-18 22:27:02.524] [sctp] [info] SCTP connection established (192.168.0.111:38412)
[2023-06-18 22:27:02.524] [sctp] [debug] SCTP association setup ascId[14]
[2023-06-18 22:27:02.524] [ngap] [debug] Sending NG Setup Request
[2023-06-18 22:27:02.525] [ngap] [debug] NG Setup Response received
[2023-06-18 22:27:02.525] [ngap] [info] NG Setup procedure is successful
```

Open5GS C-Planeのログは以下の通りです。

```
06/18 22:27:02.506: [amf] INFO: gNB-N2 accepted[192.168.0.131]:55952 in ng-path module (../src/amf/ngap-sctp.c:113)
06/18 22:27:02.507: [amf] INFO: gNB-N2 accepted[192.168.0.131] in master_sm module (../src/amf/amf-sm.c:741)
06/18 22:27:02.507: [amf] INFO: [Added] Number of gNBs is now 1 (../src/amf/context.c:1178)
06/18 22:27:02.507: [amf] INFO: gNB-N2[192.168.0.131] max_num_of_ostreams : 10 (../src/amf/amf-sm.c:780)
```

#### UEを起動

次のようにUEを起動します。これにより、5GCにUEが登録され、PDUセッションが確立します。

```
# ./nr-ue -c ../config/open5gs-ue.yaml
UERANSIM v3.2.6
[2023-06-18 22:29:40.222] [nas] [info] UE switches to state [MM-DEREGISTERED/PLMN-SEARCH]
[2023-06-18 22:29:40.223] [rrc] [debug] New signal detected for cell[1], total [1] cells in coverage
[2023-06-18 22:29:40.223] [nas] [info] Selected plmn[001/01]
[2023-06-18 22:29:40.224] [rrc] [info] Selected cell plmn[001/01] tac[1] category[SUITABLE]
[2023-06-18 22:29:40.224] [nas] [info] UE switches to state [MM-DEREGISTERED/PS]
[2023-06-18 22:29:40.224] [nas] [info] UE switches to state [MM-DEREGISTERED/NORMAL-SERVICE]
[2023-06-18 22:29:40.224] [nas] [debug] Initial registration required due to [MM-DEREG-NORMAL-SERVICE]
[2023-06-18 22:29:40.226] [nas] [debug] UAC access attempt is allowed for identity[0], category[MO_sig]
[2023-06-18 22:29:40.227] [nas] [debug] Sending Initial Registration
[2023-06-18 22:29:40.227] [rrc] [debug] Sending RRC Setup Request
[2023-06-18 22:29:40.227] [nas] [info] UE switches to state [MM-REGISTER-INITIATED]
[2023-06-18 22:29:40.227] [rrc] [info] RRC connection established
[2023-06-18 22:29:40.228] [rrc] [info] UE switches to state [RRC-CONNECTED]
[2023-06-18 22:29:40.228] [nas] [info] UE switches to state [CM-CONNECTED]
[2023-06-18 22:29:40.235] [nas] [debug] Authentication Request received
[2023-06-18 22:29:40.240] [nas] [debug] Security Mode Command received
[2023-06-18 22:29:40.240] [nas] [debug] Selected integrity[2] ciphering[0]
[2023-06-18 22:29:40.254] [nas] [debug] Registration accept received
[2023-06-18 22:29:40.254] [nas] [info] UE switches to state [MM-REGISTERED/NORMAL-SERVICE]
[2023-06-18 22:29:40.254] [nas] [debug] Sending Registration Complete
[2023-06-18 22:29:40.254] [nas] [info] Initial Registration is successful
[2023-06-18 22:29:40.254] [nas] [debug] Sending PDU Session Establishment Request
[2023-06-18 22:29:40.254] [nas] [debug] UAC access attempt is allowed for identity[0], category[MO_sig]
[2023-06-18 22:29:40.458] [nas] [debug] Configuration Update Command received
[2023-06-18 22:29:40.479] [nas] [debug] PDU Session Establishment Accept received
[2023-06-18 22:29:40.484] [nas] [info] PDU Session establishment is successful PSI[1]
[2023-06-18 22:29:40.508] [app] [info] Connection setup for PDU session[1] is successful, TUN interface[uesimtun0, 10.45.0.2] is up.
```

Open5GS C-Planeのログは以下の通りです。

```
06/18 22:29:40.192: [amf] INFO: InitialUEMessage (../src/amf/ngap-handler.c:372)
06/18 22:29:40.192: [amf] INFO: [Added] Number of gNB-UEs is now 1 (../src/amf/context.c:2502)
06/18 22:29:40.192: [amf] INFO:     RAN_UE_NGAP_ID[1] AMF_UE_NGAP_ID[1] TAC[1] CellID[0x10] (../src/amf/ngap-handler.c:533)
06/18 22:29:40.192: [amf] INFO: [suci-0-001-01-0000-0-0-0000000000] Unknown UE by SUCI (../src/amf/context.c:1776)
06/18 22:29:40.192: [amf] INFO: [Added] Number of AMF-UEs is now 1 (../src/amf/context.c:1563)
06/18 22:29:40.192: [gmm] INFO: Registration request (../src/amf/gmm-sm.c:985)
06/18 22:29:40.192: [gmm] INFO: [suci-0-001-01-0000-0-0-0000000000]    SUCI (../src/amf/gmm-handler.c:152)
06/18 22:29:40.194: [sbi] WARNING: [aad249e2-0ddb-41ee-970a-612fd8d093f3] (NF-discover) NF has already been added (../lib/sbi/nnrf-handler.c:833)
06/18 22:29:40.195: [sbi] WARNING: NF EndPoint updated [127.0.0.12:80] (../lib/sbi/context.c:1623)
06/18 22:29:40.195: [sbi] WARNING: NF EndPoint updated [127.0.0.12:7777] (../lib/sbi/context.c:1532)
06/18 22:29:40.195: [sbi] WARNING: NF EndPoint updated [127.0.0.12:7777] (../lib/sbi/context.c:1532)
06/18 22:29:40.195: [sbi] WARNING: NF EndPoint updated [127.0.0.12:7777] (../lib/sbi/context.c:1532)
06/18 22:29:40.195: [sbi] INFO: [aad249e2-0ddb-41ee-970a-612fd8d093f3] (NF-discover) NF Profile updated (../lib/sbi/nnrf-handler.c:856)
06/18 22:29:40.214: [sbi] WARNING: [aad9122c-0ddb-41ee-8a34-472da38afdc3] (NF-discover) NF has already been added (../lib/sbi/nnrf-handler.c:833)
06/18 22:29:40.215: [sbi] WARNING: NF EndPoint updated [127.0.0.20:80] (../lib/sbi/context.c:1623)
06/18 22:29:40.215: [sbi] WARNING: NF EndPoint updated [127.0.0.20:7777] (../lib/sbi/context.c:1532)
06/18 22:29:40.215: [sbi] INFO: [aad9122c-0ddb-41ee-8a34-472da38afdc3] (NF-discover) NF Profile updated (../lib/sbi/nnrf-handler.c:856)
06/18 22:29:40.421: [gmm] INFO: [imsi-001010000000000] Registration complete (../src/amf/gmm-sm.c:1917)
06/18 22:29:40.422: [amf] INFO: [imsi-001010000000000] Configuration update command (../src/amf/nas-path.c:612)
06/18 22:29:40.422: [gmm] INFO:     UTC [2023-06-18T13:29:40] Timezone[0]/DST[0] (../src/amf/gmm-build.c:546)
06/18 22:29:40.422: [gmm] INFO:     LOCAL [2023-06-18T22:29:40] Timezone[32400]/DST[0] (../src/amf/gmm-build.c:551)
06/18 22:29:40.422: [amf] INFO: [Added] Number of AMF-Sessions is now 1 (../src/amf/context.c:2523)
06/18 22:29:40.422: [gmm] INFO: UE SUPI[imsi-001010000000000] DNN[internet] S_NSSAI[SST:1 SD:0xffffff] (../src/amf/gmm-handler.c:1236)
06/18 22:29:40.424: [smf] INFO: [Added] Number of SMF-UEs is now 1 (../src/smf/context.c:1010)
06/18 22:29:40.424: [smf] INFO: [Added] Number of SMF-Sessions is now 1 (../src/smf/context.c:3051)
06/18 22:29:40.425: [sbi] WARNING: [aad249e2-0ddb-41ee-970a-612fd8d093f3] (NF-discover) NF has already been added (../lib/sbi/nnrf-handler.c:833)
06/18 22:29:40.425: [sbi] WARNING: NF EndPoint updated [127.0.0.12:80] (../lib/sbi/context.c:1623)
06/18 22:29:40.425: [sbi] WARNING: NF EndPoint updated [127.0.0.12:7777] (../lib/sbi/context.c:1532)
06/18 22:29:40.425: [sbi] WARNING: NF EndPoint updated [127.0.0.12:7777] (../lib/sbi/context.c:1532)
06/18 22:29:40.426: [sbi] WARNING: NF EndPoint updated [127.0.0.12:7777] (../lib/sbi/context.c:1532)
06/18 22:29:40.426: [sbi] INFO: [aad249e2-0ddb-41ee-970a-612fd8d093f3] (NF-discover) NF Profile updated (../lib/sbi/nnrf-handler.c:856)
06/18 22:29:40.429: [sbi] WARNING: [aad9a926-0ddb-41ee-b14b-93c544a1ff65] (NF-discover) NF has already been added (../lib/sbi/nnrf-handler.c:833)
06/18 22:29:40.430: [sbi] WARNING: NF EndPoint updated [127.0.0.13:80] (../lib/sbi/context.c:1623)
06/18 22:29:40.430: [sbi] WARNING: NF EndPoint updated [127.0.0.13:7777] (../lib/sbi/context.c:1532)
06/18 22:29:40.430: [sbi] WARNING: NF EndPoint updated [127.0.0.13:7777] (../lib/sbi/context.c:1532)
06/18 22:29:40.430: [sbi] WARNING: NF EndPoint updated [127.0.0.13:7777] (../lib/sbi/context.c:1532)
06/18 22:29:40.430: [sbi] INFO: [aad9a926-0ddb-41ee-b14b-93c544a1ff65] (NF-discover) NF Profile updated (../lib/sbi/nnrf-handler.c:856)
06/18 22:29:40.432: [sbi] WARNING: [aad9122c-0ddb-41ee-8a34-472da38afdc3] (NF-discover) NF has already been added (../lib/sbi/nnrf-handler.c:833)
06/18 22:29:40.432: [sbi] WARNING: NF EndPoint updated [127.0.0.20:80] (../lib/sbi/context.c:1623)
06/18 22:29:40.432: [sbi] WARNING: NF EndPoint updated [127.0.0.20:7777] (../lib/sbi/context.c:1532)
06/18 22:29:40.432: [sbi] INFO: [aad9122c-0ddb-41ee-8a34-472da38afdc3] (NF-discover) NF Profile updated (../lib/sbi/nnrf-handler.c:856)
06/18 22:29:40.435: [sbi] WARNING: [aad17de6-0ddb-41ee-9200-cb54cf262ce4] (NF-discover) NF has already been added (../lib/sbi/nnrf-handler.c:833)
06/18 22:29:40.436: [sbi] WARNING: NF EndPoint updated [127.0.0.15:80] (../lib/sbi/context.c:1623)
06/18 22:29:40.436: [sbi] WARNING: NF EndPoint updated [127.0.0.15:7777] (../lib/sbi/context.c:1532)
06/18 22:29:40.436: [sbi] INFO: [aad17de6-0ddb-41ee-9200-cb54cf262ce4] (NF-discover) NF Profile updated (../lib/sbi/nnrf-handler.c:856)
06/18 22:29:40.438: [smf] INFO: UE SUPI[imsi-001010000000000] DNN[internet] IPv4[10.45.0.2] IPv6[] (../src/smf/npcf-handler.c:546)
06/18 22:29:40.439: [gtp] INFO: gtp_connect() [192.168.13.151]:2152 (../lib/gtp/path.c:60)
06/18 22:29:40.449: [amf] INFO: [imsi-001010000000000:1:11][0:0:NULL] /nsmf-pdusession/v1/sm-contexts/{smContextRef}/modify (../src/amf/nsmf-handler.c:744)
```

VPP-UPFのPDUセッション確立状況は次のように表示されます。

```
vpp# show upf session 
CP F-SEID: 0x000000000000028d (653) @ 192.168.14.111
UP F-SEID: 0x000000000000028d (653) @ 192.168.14.151
  PFCP Association: 0
  TEID assignment per choose ID
PDR: 1 @ 0x7ff41c4eaed0
  Precedence: 255
  PDI:
    Fields: 0000000c
    Source Interface: Core
    Network Instance: internet
    UE IP address (destination):
      IPv4 address: 10.45.0.2
    SDF Filter [1]:
      permit out ip from any to assigned 
  Outer Header Removal: no
  FAR Id: 1
  URR Ids: [1] @ 0x7ff3e7b77d70
  QER Ids: [1] @ 0x7ff3e7b77da0
PDR: 2 @ 0x7ff41c4eaf50
  Precedence: 255
  PDI:
    Fields: 0000000d
    Source Interface: Access
    Network Instance: internet
    Local F-TEID: 459247801 (0x1b5f90b9)
            IPv4: 192.168.13.151
    UE IP address (source):
      IPv4 address: 10.45.0.2
    SDF Filter [1]:
      permit out ip from any to assigned 
  Outer Header Removal: GTP-U/UDP/IPv4
  FAR Id: 2
  URR Ids: [] @ 0x0
  QER Ids: [1] @ 0x7ff41c4e5380
PDR: 3 @ 0x7ff41c4eafd0
  Precedence: 1000
  PDI:
    Fields: 00000001
    Source Interface: CP-function
    Network Instance: internet
    Local F-TEID: 369361604 (0x160402c4)
            IPv4: 192.168.13.151
  Outer Header Removal: GTP-U/UDP/IPv4
  FAR Id: 1
  URR Ids: [] @ 0x0
  QER Ids: [1] @ 0x7ff3e85622e0
PDR: 4 @ 0x7ff41c4eb050
  Precedence: 1
  PDI:
    Fields: 00000009
    Source Interface: Access
    Network Instance: internet
    Local F-TEID: 459247801 (0x1b5f90b9)
            IPv4: 192.168.13.151
    SDF Filter [1]:
      permit out 58 from ff02::2 to assigned 
  Outer Header Removal: GTP-U/UDP/IPv4
  FAR Id: 3
  URR Ids: [] @ 0x0
  QER Ids: [] @ 0x0
FAR: 1
  Apply Action: 00000002 == [FORWARD]
  Forward:
    Network Instance: internet
    Destination Interface: 0
    Outer Header Creation: [GTP-U/UDP/IPv4],TEID:00000001,IP:192.168.13.131
FAR: 2
  Apply Action: 00000002 == [FORWARD]
  Forward:
    Network Instance: internet
    Destination Interface: 1
FAR: 3
  Apply Action: 00000002 == [FORWARD]
  Forward:
    Network Instance: internet
    Destination Interface: 3
    Outer Header Creation: [GTP-U/UDP/IPv4],TEID:00000001,IP:192.168.14.111
URR: 1
  Measurement Method: 0002 == [VOLUME]
  Reporting Triggers: 0002 == [VOLUME THRESHOLD]
  Status: 0 == []
  Start Time: 2023/06/18 22:29:40:480
  vTime of First Usage:       0.0000 
  vTime of Last Usage:        0.0000 
  Volume
    Up:    Measured:                    0, Theshold:                    0, Pkts:          0
           Consumed:                    0, Quota:                       0
    Down:  Measured:                    0, Theshold:                    0, Pkts:          0
           Consumed:                    0, Quota:                       0
    Total: Measured:                    0, Theshold:            104857600, Pkts:          0
           Consumed:                    0, Quota:                       0
vpp#
```

`nr-ue`コマンドのコンソールログから、Open5GS 5GCからIPアドレス`10.45.0.2`がUEに割り当てられたことが分かります。

```
[2023-06-18 22:29:40.508] [app] [info] Connection setup for PDU session[1] is successful, TUN interface[uesimtun0, 10.45.0.2] is up.
```

念のため、UEのTUNインターフェースに割り当てられたIPアドレスと一致していることを確認して下さい。

```
# ip addr show
...
10: uesimtun0: <POINTOPOINT,PROMISC,NOTRAILERS,UP,LOWER_UP> mtu 1400 qdisc fq_codel state UNKNOWN group default qlen 500
    link/none 
    inet 10.45.0.2/32 scope global uesimtun0
       valid_lft forever preferred_lft forever
    inet6 fe80::3307:3808:e44a:8bac/64 scope link stable-privacy 
       valid_lft forever preferred_lft forever
...
```

## google.comにpingする

UEのTUNインターフェースを指定して、`ping`してみます。

TUNインターフェースの使い方は以下を参照して下さい。

- https://github.com/aligungr/UERANSIM/wiki/Usage

### DN=10.45.0.0/16を経由する場合

VM-DNで`tcpdump`を実行し、パケットがN6インターフェース`enp0s9`を通過することを確認します。

- VM3(UE)で`ping google.com`

```
# ping google.com -I uesimtun0 -n
PING google.com (216.58.220.142) from 10.45.0.2 uesimtun0: 56(84) bytes of data.
64 bytes from 216.58.220.142: icmp_seq=2 ttl=59 time=29.7 ms
64 bytes from 216.58.220.142: icmp_seq=3 ttl=59 time=18.9 ms
64 bytes from 216.58.220.142: icmp_seq=4 ttl=59 time=16.9 ms
```

- VM-DNで`tcpdump`を実行

```
# tcpdump -i enp0s9 -n
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on enp0s9, link-type EN10MB (Ethernet), snapshot length 262144 bytes
22:32:39.054526 IP 10.45.0.2 > 216.58.220.142: ICMP echo request, id 10, seq 2, length 64
22:32:39.083079 IP 216.58.220.142 > 10.45.0.2: ICMP echo reply, id 10, seq 2, length 64
22:32:40.055326 IP 10.45.0.2 > 216.58.220.142: ICMP echo request, id 10, seq 3, length 64
22:32:40.073334 IP 216.58.220.142 > 10.45.0.2: ICMP echo reply, id 10, seq 3, length 64
22:32:41.055397 IP 10.45.0.2 > 216.58.220.142: ICMP echo request, id 10, seq 4, length 64
22:32:41.071228 IP 216.58.220.142 > 10.45.0.2: ICMP echo reply, id 10, seq 4, length 64
```

また、以下のように`nr-binder`ツールにおいて、TUNインターフェースに割り当てられたIPアドレスを指定して、大半のアプリケーションを使用することができます。

- VM3(UE)で`curl google.com`を実行

```
# sh nr-binder 10.45.0.2 curl google.com
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="http://www.google.com/">here</A>.
</BODY></HTML>
```

- VM-DNで`tcpdump`を実行

```
22:33:41.691618 IP 10.45.0.2.60435 > 216.58.220.142.80: Flags [S], seq 29076856, win 65280, options [mss 1360,sackOK,TS val 1470382296 ecr 0,nop,wscale 7], length 0
22:33:41.707297 IP 216.58.220.142.80 > 10.45.0.2.60435: Flags [S.], seq 4928001, ack 29076857, win 65535, options [mss 1460], length 0
22:33:41.708039 IP 10.45.0.2.60435 > 216.58.220.142.80: Flags [.], ack 1, win 65280, length 0
22:33:41.708147 IP 10.45.0.2.60435 > 216.58.220.142.80: Flags [P.], seq 1:75, ack 1, win 65280, length 74: HTTP: GET / HTTP/1.1
22:33:41.708248 IP 216.58.220.142.80 > 10.45.0.2.60435: Flags [.], ack 75, win 65535, length 0
22:33:41.762838 IP 216.58.220.142.80 > 10.45.0.2.60435: Flags [P.], seq 1:774, ack 75, win 65535, length 773: HTTP: HTTP/1.1 301 Moved Permanently
22:33:41.764008 IP 10.45.0.2.60435 > 216.58.220.142.80: Flags [.], ack 774, win 64507, length 0
22:33:41.766355 IP 10.45.0.2.60435 > 216.58.220.142.80: Flags [F.], seq 75, ack 774, win 64507, length 0
22:33:41.766501 IP 216.58.220.142.80 > 10.45.0.2.60435: Flags [.], ack 76, win 65535, length 0
22:33:41.783432 IP 216.58.220.142.80 > 10.45.0.2.60435: Flags [F.], seq 774, ack 76, win 65535, length 0
22:33:41.784262 IP 10.45.0.2.60435 > 216.58.220.142.80: Flags [.], ack 775, win 64507, length 0
```

なお、`nr-binder`は`ping`では機能しません。理由は[こちら](https://github.com/aligungr/UERANSIM/issues/186#issuecomment-729534464)を参照して下さい。

以上で、DPDK対応VPP-UPFのDNに接続し、DNに任意のパケットを流せるようになり、SA構成のローカル5Gを想定した5GCのシミュレーション携帯網として、MECを試す環境を構築できました。

## 最後に

DPDK対応VPP-UPFは、U-Planeのデータネットワークを高速化する技術と期待しています。本記事では、Virtualbox VM上の設定例を書いてみましたが、高性能な物理NICを搭載したサーバを設定する際のご参考になれれば幸いです。

最後に、元記事は[Github](https://github.com/s5uishida/open5gs_5gc_ueransim_vpp_upf_dpdk_sample_config)に書いたものです。

## 主な変更履歴

- [2023.09.06] 初版。