Open5GS EPCとZeroMQ対応srsRAN 4G UE/RANでeUPF(eBPF/XDP UPF)を試す

本記事では、プライベートLTEを想定して、Open5GS EPCとeBPF/XDP対応UPFのeUPFを連携動作させるための簡単な構成について説明します。 eUPFのインストールと設定については、[こちら](https://qiita.com/s5uishida/items/8890fb67cb8d71da6080)を参照してください。

## Open5GS CU分離EPCのシミュレーション携帯網の概要

今回、Open5GS EPCのC-Plane、eUPF、およびData Network Gatewayの簡単な構成について説明します。**この構成はVirtualbox VMを使用して実装されることに注意して下さい。**
このシミュレーション環境は以下の条件を満たす最小構成になります。

- 1つのSGW-U/UPF(PGW-U)とData Network Gateway
- 1つのUEと1つのAPN

構築したシミュレーション環境は以下の通りです。

<img width="800" alt="network-overview.png" src="https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/457466/1a4858a4-1cde-4197-4467-fa30836c15fc.png">

EPC、eUPFとUE/RANに使用しているOSSは以下の通りです。

- EPC - Open5GS v2.7.0 (2023.12.04) - https://github.com/open5gs/open5gs
- eBPF/XDP PGW-U - eUPF v0.6.0 (2023.12.04) - https://github.com/edgecomllc/eupf
- UE / RAN - srsRAN 4G (2023.11.23) - https://github.com/srsran/srsRAN_4G

各VMは以下の通りです。

| VM | SW & Role | IP address | OS | CPU<br>(Min) | Memory<br>(Min) | HDD<br>(Min) |
| --- | --- | --- | --- | --- | --- | --- |
| VM1 | Open5GS EPC<br>C-Plane | 192.168.0.111/24 | Ubuntu<br>22.04 | 1 | 1GB | 20GB |
| VM2 | Open5GS EPC<br>U-Plane(SGW-U) | 192.168.0.112/24 | Ubuntu<br>22.04 | 1 | 1GB | 20GB |
| VM-UP | eUPF<br>U-Plane(PGW-U) | 192.168.0.151/24 | Ubuntu 22.04 | 1 | 2GB | 20GB |
| VM-DN | Data Network<br>Gateway  | 192.168.0.152/24 | Ubuntu<br>22.04 | 1 | 1GB | 10GB |
| VM3 | srsRAN 4G<br>ZMQ RAN (eNodeB) | 192.168.0.121/24 | Ubuntu<br>22.04 | 1 | 2GB | 10GB |
| VM4 | srsRAN 4G<br>ZMQ UE | 192.168.0.122/24 | Ubuntu<br>22.04 | 1 | 2GB | 10GB |

各VMのネットワークインターフェースは以下の通りです。

| VM | Device | Network Adapter | IP address | Interface | XDP |
| --- | --- | --- | --- | --- | --- |
| VM1 | enp0s3 | NAT(default) | 10.0.2.15/24 | (VM default NW) | -- |
| | enp0s8 | Bridged Adapter | 192.168.0.111/24 | (Mgmt NW) | -- |
| | enp0s9 | NAT Network | 192.168.14.111/24 | Sxb (N4 for 5GC) | -- |
| VM2 | enp0s3 | NAT(default) | 10.0.2.15/24 | (VM default NW) | -- |
| | enp0s8 | Bridged Adapter | 192.168.0.112/24 | (Mgmt NW) | -- |
| | enp0s9 | NAT Network | 192.168.13.112/24 | S1-U,S5u (N3 for 5GC) | -- |
| VM-UP | ~~enp0s3~~ | ~~NAT(default)~~ | ~~10.0.2.15/24~~ | ~~(VM default NW)~~<br>***down*** | -- |
| | enp0s8 | Bridged Adapter | 192.168.0.151/24 | (Mgmt NW) | -- |
| | enp0s9 | NAT Network | 192.168.13.151/24 | S5u (N3 for 5GC) | x |
| | enp0s10 | NAT Network | 192.168.14.151/24 | Sxb (N4 for 5GC) | -- |
| | enp0s16 | NAT Network | 192.168.16.151/24 | SGi (N6 for 5GC) | x |
| VM-DN | enp0s3 | NAT(default) | 10.0.2.15/24 | (VM default NW) | -- |
| | enp0s8 | Bridged Adapter | 192.168.0.152/24 | (Mgmt NW) | -- |
| | enp0s9 | NAT Network | 192.168.16.152/24 | N6,<br>***default GW<br>for VM-UP*** | -- |
| VM3 | enp0s3 | NAT(default) | 10.0.2.15/24 | (VM default NW) | -- |
| | enp0s8 | Bridged Adapter | 192.168.0.121/24 | (Mgmt NW) | -- |
| | enp0s9 | NAT Network | 192.168.13.121/24 | S1-U (N3 for 5GC) | -- |
| VM4 | enp0s3 | NAT(default) | 10.0.2.15/24 | (VM default NW) | -- |
| | enp0s8 | Bridged Adapter | 192.168.0.122/24 | (Mgmt NW) | -- |

EPCのNF間のインターフェースの内、S1-U、S5u、Sxb、SGiについては、VirtualboxのNATネットワークで以下の通りに設定しました。

| Network Name | Network CIDR | Note |
| --- | --- | --- |
| N3 | 192.168.13.0/24 | S1-U,S5u for EPC |
| N4 | 192.168.14.0/24 | Sxb for EPC |
| N6 | 192.168.16.0/24 | SGi for EPC |

加入者情報は以下の通りです。(その他の項目はデフォルト値)
**srsRAN 4GのUEの設定に合わせて、OPあるいはOPcを選択してください。**

| UE | IMSI | APN | OP/OPc |
| --- | --- | --- | --- |
| UE | 001010000000100 | internet | OPc |

これらの情報の登録には、Open5GSが提供するWebUIを使用しました。なお、3GPP認証とキー生成機能(MILENAGE)の試験データとして、[3GPP TS 35.208](https://www.3gpp.org/DynaReport/35208.htm) "4.3 Test Sets"が公開されています。

PDNは以下の通りです。

| PDN | APN | TUNnel interface of UE |
| --- | --- | --- |
| 10.45.0.0/16 | internet | tun_srsue |

eNodeBの主な情報は以下の通りです。

| MCC | MNC | TAC | eNodeB ID | Cell ID | E-UTRAN Cell ID |
| --- | --- | --- | --- | --- | --- |
| 001 | 01 | 1 | 0x19b | 0x01 | 0x19b01 |

## Open5GS EPC、eUPFとZMQ対応srsRAN 4G UE/RANの設定ファイルの変更点

Open5GS、eUPFとZMQ対応srsRAN 4G UE/RANのビルド方法は以下を参照して下さい。

- Open5GS v2.7.0 (2023.12.04) - https://open5gs.org/open5gs/docs/guide/02-building-open5gs-from-sources/
- eUPF v0.6.0 (2023.12.04) - https://qiita.com/s5uishida/items/8890fb67cb8d71da6080
- srsRAN 4G (2023.11.23) - https://github.com/s5uishida/build_srsran_4g_zmq_disable_rf_plugins

### Open5GS EPC C-Planeの設定ファイルの変更点

Open5GSでは、PFCPの接続先としてSGW-Uを選択するロジックに使用可能なパラメータは以下の通りです。PFCP(Packet Forwarding Control Protocol)とは、5GCでCU分離のために導入されたプロトコルです。

- APN
- TAC (Tracking Area Code)
- e_CellID

なお、TACとe_CellIDはRANの設定項目です。話を簡単にするために、今回はAPNのみ使用しています。Open5GSでSGW-Uを選択するロジックについては、[こちら](https://github.com/open5gs/open5gs/pull/560#issue-483001043)を参照して下さい。

- `open5gs/install/etc/open5gs/mme.yaml`

```diff
--- mme.yaml.orig       2023-12-05 03:49:16.000000000 +0900
+++ mme.yaml    2023-12-05 04:32:50.429030682 +0900
@@ -11,7 +11,7 @@
   freeDiameter: /root/open5gs/install/etc/freeDiameter/mme.conf
   s1ap:
     server:
-      - address: 127.0.0.2
+      - address: 192.168.0.111
   gtpc:
     server:
       - address: 127.0.0.2
@@ -26,14 +26,14 @@
         port: 9090
   gummei:
     - plmn_id:
-        mcc: 999
-        mnc: 70
+        mcc: 001
+        mnc: 01
       mme_gid: 2
       mme_code: 1
   tai:
     - plmn_id:
-        mcc: 999
-        mnc: 70
+        mcc: 001
+        mnc: 01
       tac: 1
   security:
     integrity_order : [ EIA2, EIA1, EIA0 ]
```

- `open5gs/install/etc/open5gs/sgwc.yaml`

```diff
--- sgwc.yaml.orig      2023-12-05 03:49:16.000000000 +0900
+++ sgwc.yaml   2023-12-05 04:34:51.785067121 +0900
@@ -13,10 +13,11 @@
       - address: 127.0.0.3
   pfcp:
     server:
-      - address: 127.0.0.3
+      - address: 192.168.0.111
     client:
       sgwu:
-        - address: 127.0.0.6
+        - address: 192.168.0.112
+          apn: internet
 
 ################################################################################
 # GTP-C Server
```

- `open5gs/install/etc/open5gs/smf.yaml`

```diff
--- smf.yaml.orig       2023-12-05 03:49:16.000000000 +0900
+++ smf.yaml    2023-12-05 04:45:38.004295776 +0900
@@ -8,39 +8,29 @@
 #    peer: 64
 
 smf:
-  sbi:
-    server:
-      - address: 127.0.0.4
-        port: 7777
-    client:
-#      nrf:
-#        - uri: http://127.0.0.10:7777
-      scp:
-        - uri: http://127.0.0.200:7777
   pfcp:
     server:
-      - address: 127.0.0.4
+      - address: 192.168.14.111
     client:
       upf:
-        - address: 127.0.0.7
+        - address: 192.168.14.151
+          dnn: internet
   gtpc:
     server:
       - address: 127.0.0.4
   gtpu:
     server:
-      - address: 127.0.0.4
+      - address: 192.168.14.111
   metrics:
     server:
       - address: 127.0.0.4
         port: 9090
   session:
     - subnet: 10.45.0.1/16
-    - subnet: 2001:db8:cafe::1/48
+      dnn: internet
   dns:
     - 8.8.8.8
     - 8.8.4.4
-    - 2001:4860:4860::8888
-    - 2001:4860:4860::8844
   mtu: 1400
 #  p-cscf:
 #    - 127.0.0.1
```

### Open5GS EPC U-Planeの設定ファイルの変更点

- `open5gs/install/etc/open5gs/sgwu.yaml`
```diff
--- sgwu.yaml.orig      2023-12-05 03:49:16.000000000 +0900
+++ sgwu.yaml   2023-12-05 04:55:03.425607103 +0900
@@ -10,13 +10,13 @@
 sgwu:
   pfcp:
     server:
-      - address: 127.0.0.6
+      - address: 192.168.0.112
     client:
 #      sgwc:    # SGW-U PFCP Client try to associate SGW-C PFCP Server
 #        - address: 127.0.0.3
   gtpu:
     server:
-      - address: 127.0.0.6
+      - address: 192.168.13.112
 
 ################################################################################
 # PFCP Server
```

### eUPFの設定ファイルの変更点

元の設定ファイルは、[こちら](https://qiita.com/s5uishida/items/8890fb67cb8d71da6080#%E8%A8%AD%E5%AE%9A%E3%83%95%E3%82%A1%E3%82%A4%E3%83%AB%E3%82%92%E4%BD%9C%E6%88%90)を参照して下さい。

- `eupf/config.yml`  
元ファイルからの変更はありません。

### ZMQ対応srsRAN 4G UE/RANの設定ファイルの変更点

#### RAN(eNodeB)の設定ファイルの変更点

- `srsRAN_4G/build/srsenb/enb.conf`

```diff
--- enb.conf.example    2023-12-02 14:18:34.000000000 +0900
+++ enb.conf    2023-12-02 15:06:53.357631639 +0900
@@ -22,9 +22,9 @@
 enb_id = 0x19B
 mcc = 001
 mnc = 01
-mme_addr = 127.0.1.100
-gtp_bind_addr = 127.0.1.1
-s1c_bind_addr = 127.0.1.1
+mme_addr = 192.168.0.111
+gtp_bind_addr = 192.168.13.121
+s1c_bind_addr = 192.168.0.121
 s1c_bind_port = 0
 n_prb = 50
 #tm = 4
@@ -80,8 +80,8 @@
 #time_adv_nsamples = auto
 
 # Example for ZMQ-based operation with TCP transport for I/Q samples
-#device_name = zmq
-#device_args = fail_on_disconnect=true,tx_port=tcp://*:2000,rx_port=tcp://localhost:2001,id=enb,base_srate=23.04e6
+device_name = zmq
+device_args = fail_on_disconnect=true,tx_port=tcp://192.168.0.121:2000,rx_port=tcp://192.168.0.122:2001,id=enb,base_srate=23.04e6
 
 #####################################################################
 # Packet capture configuration
```

- `srsRAN_4G/build/srsenb/rr.conf`

```diff
--- rr.conf.example     2023-12-02 14:18:34.000000000 +0900
+++ rr.conf     2023-05-02 11:52:54.000000000 +0900
@@ -55,7 +55,7 @@
   {
     // rf_port = 0;
     cell_id = 0x01;
-    tac = 0x0007;
+    tac = 0x0001;
     pci = 1;
     // root_seq_idx = 204;
     dl_earfcn = 3350;
```

#### UE(IMSI-001010000000100)の設定ファイルの変更点

- `srsRAN_4G/build/srsue/ue.conf`

```diff
--- ue.conf.example     2023-05-02 10:51:20.000000000 +0900
+++ ue.conf     2023-05-02 12:01:28.000000000 +0900
@@ -42,8 +42,8 @@
 #continuous_tx     = auto
 
 # Example for ZMQ-based operation with TCP transport for I/Q samples
-#device_name = zmq
-#device_args = tx_port=tcp://*:2001,rx_port=tcp://localhost:2000,id=ue,base_srate=23.04e6
+device_name = zmq
+device_args = tx_port=tcp://192.168.0.122:2001,rx_port=tcp://192.168.0.121:2000,id=ue,base_srate=23.04e6
 
 #####################################################################
 # EUTRA RAT configuration
@@ -139,9 +139,9 @@
 [usim]
 mode = soft
 algo = milenage
-opc  = 63BFA50EE6523365FF14C1F45F88737D
-k    = 00112233445566778899aabbccddeeff
-imsi = 001010123456780
+opc  = E8ED289DEBA952E4283B54E88E6183CA
+k    = 465B5CE8B199B49FAA5F0A2EE238A6BC
+imsi = 001010000000100
 imei = 353490069873319
 #reader =
 #pin  = 1234
@@ -180,8 +180,8 @@
 #                      Supported: 0 - NULL, 1 - Snow3G, 2 - AES, 3 - ZUC
 #####################################################################
 [nas]
-#apn = internetinternet
-#apn_protocol = ipv4
+apn = internet
+apn_protocol = ipv4
 #user = srsuser
 #pass = srspass
 #force_imsi_attach = false
```

## Open5GS EPC、eUPFとZMQ対応srsRAN 4G UE/RANのネットワーク設定

### eUPFとData Network Gatewayのネットワーク設定

[こちら](https://qiita.com/s5uishida/items/8890fb67cb8d71da6080#vm-up%E3%81%ABeupf%E3%82%92%E3%82%BB%E3%83%83%E3%83%88%E3%82%A2%E3%83%83%E3%83%97)と[こちら](https://qiita.com/s5uishida/items/8890fb67cb8d71da6080#vm-dn%E3%81%A7data-network-gateway%E3%82%92%E8%A8%AD%E5%AE%9A)を参照して下さい。

## Open5GS、eUPFとUERANSIMのビルド

Open5GS、eUPFとUERANSIMのビルド方法は以下を参考にして下さい。

- Open5GS v2.7.0 (2023.12.04) - https://open5gs.org/open5gs/docs/guide/02-building-open5gs-from-sources/
- eUPF v0.6.0 (2023.12.04) - https://qiita.com/s5uishida/items/8890fb67cb8d71da6080
- srsRAN 4G (2023.11.23) - https://github.com/s5uishida/build_srsran_4g_zmq_disable_rf_plugins

Open5GS 5GC C-PlaneマシンにMongoDBをインストールして下さい。なお、MongoDBのデータベースを簡単に確認するには、[MongoDB Compass](https://www.mongodb.com/products/compass)が便利です。

## Open5GS EPC、eUPFとZMQ対応srsRAN 4G UE/RANを実行

最初にeUPFを起動し、次にOpen5GS EPC U-Plane(SGW-U)とC-Plane、ZMQ対応srsRAN 4G RANとUEを起動します。

### eUPFを実行

[こちら](https://qiita.com/s5uishida/items/8890fb67cb8d71da6080#vm-up%E3%81%A7eupf%E3%82%92%E5%AE%9F%E8%A1%8C)を参考にして下さい。

### Open5GS EPC U-Planeを実行

次に、Open5GS EPC U-Planeを起動します。

```
./install/bin/open5gs-sgwud &
```

### Open5GS EPC C-Planeを実行

次に、Open5GS EPC C-Planeを起動します。

```
./install/bin/open5gs-mmed &
./install/bin/open5gs-sgwcd &
./install/bin/open5gs-smfd &
./install/bin/open5gs-hssd &
./install/bin/open5gs-pcrfd &
```

eUPFとOpen5GS SMF間のPFCP Associationログは以下の通りです。

```
2023/12/05 20:21:41 INF Got Association Setup Request from: 192.168.14.111. 

2023/12/05 20:21:41 INF 
Association Setup Request:
  Node ID: 192.168.14.111
  Recovery Time: 2023-12-05 20:21:40 +0900 JST

2023/12/05 20:21:41 INF Saving new association: &{ID:192.168.14.111 Addr:192.168.14.111 NextSessionID:1 NextSequenceID:1 Sessions:map[] HeartbeatRetries:0 cancelRetries:<nil>}
```

#### ZMQ対応srsRAN 4G RAN(eNodeB)を起動

ZMQ対応srsRAN 4G RAN(eNodeB)を起動し、Open5GS EPCに接続します。

```
# cd srsRAN_4G/build/srsenb
# ./src/srsenb enb.conf
---  Software Radio Systems LTE eNodeB  ---

Reading configuration file enb.conf...

Built in Release mode using commit eea87b1d8 on branch master.

Opening 1 channels in RF device=zmq with args=fail_on_disconnect=true,tx_port=tcp://192.168.0.121:2000,rx_port=tcp://192.168.0.122:2001,id=enb,base_srate=23.04e6
Supported RF device list: zmq file
CHx base_srate=23.04e6
CHx id=enb
Current sample rate is 1.92 MHz with a base rate of 23.04 MHz (x12 decimation)
CH0 rx_port=tcp://192.168.0.122:2001
CH0 tx_port=tcp://192.168.0.121:2000
CH0 fail_on_disconnect=true

==== eNodeB started ===
Type <t> to view trace
Current sample rate is 11.52 MHz with a base rate of 23.04 MHz (x2 decimation)
Current sample rate is 11.52 MHz with a base rate of 23.04 MHz (x2 decimation)
Setting frequency: DL=2680.0 Mhz, UL=2560.0 MHz for cc_idx=0 nof_prb=50
```
The Open5GS C-Plane log when executed is as follows.
```
12/05 20:22:10.188: [mme] INFO: eNB-S1 accepted[192.168.0.121]:47130 in s1_path module (../src/mme/s1ap-sctp.c:114)
12/05 20:22:10.188: [mme] INFO: eNB-S1 accepted[192.168.0.121] in master_sm module (../src/mme/mme-sm.c:108)
12/05 20:22:10.188: [mme] INFO: [Added] Number of eNBs is now 1 (../src/mme/mme-context.c:2808)
12/05 20:22:10.188: [mme] INFO: eNB-S1[192.168.0.121] max_num_of_ostreams : 30 (../src/mme/mme-sm.c:150)
```

Open5GS C-Planeのログは以下の通りです。

```
12/05 20:22:10.188: [mme] INFO: eNB-S1 accepted[192.168.0.121]:47130 in s1_path module (../src/mme/s1ap-sctp.c:114)
12/05 20:22:10.188: [mme] INFO: eNB-S1 accepted[192.168.0.121] in master_sm module (../src/mme/mme-sm.c:108)
12/05 20:22:10.188: [mme] INFO: [Added] Number of eNBs is now 1 (../src/mme/mme-context.c:2808)
12/05 20:22:10.188: [mme] INFO: eNB-S1[192.168.0.121] max_num_of_ostreams : 30 (../src/mme/mme-sm.c:150)
```

#### ZMQ対応srsRAN 4G UEを起動

ZMQ対応srsRAN 4G UEを起動し、eNodeB、Open5GS EPCに接続します。

```
# cd srsRAN_4G/build/srsue
# ./src/srsue ue.conf
Reading configuration file ue.conf...

Built in Release mode using commit eea87b1d8 on branch master.

Opening 1 channels in RF device=zmq with args=tx_port=tcp://192.168.0.122:2001,rx_port=tcp://192.168.0.121:2000,id=ue,base_srate=23.04e6
Supported RF device list: zmq file
CHx base_srate=23.04e6
CHx id=ue
Current sample rate is 1.92 MHz with a base rate of 23.04 MHz (x12 decimation)
CH0 rx_port=tcp://192.168.0.121:2000
CH0 tx_port=tcp://192.168.0.122:2001
Waiting PHY to initialize ... done!
Attaching UE...
Current sample rate is 1.92 MHz with a base rate of 23.04 MHz (x12 decimation)
Current sample rate is 1.92 MHz with a base rate of 23.04 MHz (x12 decimation)
.
Found Cell:  Mode=FDD, PCI=1, PRB=50, Ports=1, CP=Normal, CFO=-0.2 KHz
Current sample rate is 11.52 MHz with a base rate of 23.04 MHz (x2 decimation)
Current sample rate is 11.52 MHz with a base rate of 23.04 MHz (x2 decimation)
Found PLMN:  Id=00101, TAC=1
Random Access Transmission: seq=38, tti=341, ra-rnti=0x2
RRC Connected
Random Access Complete.     c-rnti=0x46, ta=0
Network attach successful. IP: 10.45.0.2
 nTp) ((t) 5/12/2023 11:23:6 TZ:99
```

Open5GS C-Planeのログは以下の通りです。

```
12/05 20:23:05.989: [mme] INFO: InitialUEMessage (../src/mme/s1ap-handler.c:406)
12/05 20:23:05.989: [mme] INFO: [Added] Number of eNB-UEs is now 1 (../src/mme/mme-context.c:4684)
12/05 20:23:05.989: [mme] INFO: Unknown UE by S_TMSI[G:2,C:1,M_TMSI:0xc0000672] (../src/mme/s1ap-handler.c:486)
12/05 20:23:05.989: [mme] INFO:     ENB_UE_S1AP_ID[1] MME_UE_S1AP_ID[1] TAC[1] CellID[0x19b01] (../src/mme/s1ap-handler.c:582)
12/05 20:23:05.989: [mme] INFO: Unknown UE by GUTI[G:2,C:1,M_TMSI:0xc0000672] (../src/mme/mme-context.c:3538)
12/05 20:23:05.989: [mme] INFO: [Added] Number of MME-UEs is now 1 (../src/mme/mme-context.c:3340)
12/05 20:23:05.989: [emm] INFO: [] Attach request (../src/mme/emm-sm.c:412)
12/05 20:23:05.990: [emm] INFO:     GUTI[G:2,C:1,M_TMSI:0xc0000672] IMSI[Unknown IMSI] (../src/mme/emm-handler.c:236)
12/05 20:23:06.028: [emm] INFO: Identity response (../src/mme/emm-sm.c:382)
12/05 20:23:06.028: [emm] INFO:     IMSI[001010000000100] (../src/mme/emm-handler.c:425)
12/05 20:23:06.144: [mme] INFO: [Added] Number of MME-Sessions is now 1 (../src/mme/mme-context.c:4698)
12/05 20:23:06.223: [sgwc] INFO: [Added] Number of SGWC-UEs is now 1 (../src/sgwc/context.c:239)
12/05 20:23:06.223: [sgwc] INFO: [Added] Number of SGWC-Sessions is now 1 (../src/sgwc/context.c:881)
12/05 20:23:06.223: [sgwc] INFO: UE IMSI[001010000000100] APN[internet] (../src/sgwc/s11-handler.c:237)
12/05 20:23:06.224: [gtp] INFO: gtp_connect() [127.0.0.4]:2123 (../lib/gtp/path.c:60)
12/05 20:23:06.225: [smf] INFO: [Added] Number of SMF-UEs is now 1 (../src/smf/context.c:1019)
12/05 20:23:06.225: [smf] INFO: [Added] Number of SMF-Sessions is now 1 (../src/smf/context.c:3068)
12/05 20:23:06.225: [smf] INFO: UE IMSI[001010000000100] APN[internet] IPv4[10.45.0.2] IPv6[] (../src/smf/s5c-handler.c:275)
12/05 20:23:06.230: [gtp] INFO: gtp_connect() [192.168.13.151]:2152 (../lib/gtp/path.c:60)
12/05 20:23:06.540: [emm] INFO: [001010000000100] Attach complete (../src/mme/emm-sm.c:1298)
12/05 20:23:06.540: [emm] INFO:     IMSI[001010000000100] (../src/mme/emm-handler.c:274)
12/05 20:23:06.540: [emm] INFO:     UTC [2023-12-05T11:23:06] Timezone[0]/DST[0] (../src/mme/emm-handler.c:281)
12/05 20:23:06.540: [emm] INFO:     LOCAL [2023-12-05T20:23:06] Timezone[32400]/DST[0] (../src/mme/emm-handler.c:285)
```

Open5GS U-Planeのログは以下の通りです。

```
12/05 20:23:06.187: [sgwu] INFO: UE F-SEID[UP:0x970 CP:0x2a7] (../src/sgwu/context.c:171)
12/05 20:23:06.187: [sgwu] INFO: [Added] Number of SGWU-Sessions is now 1 (../src/sgwu/context.c:176)
12/05 20:23:06.194: [gtp] INFO: gtp_connect() [192.168.13.151]:2152 (../lib/gtp/path.c:60)
12/05 20:23:06.504: [gtp] INFO: gtp_connect() [192.168.13.121]:2152 (../lib/gtp/path.c:60)
```

eUPFとOpen5GS SMF(PGW-C)のPFCP associationのステータスは次のように表示されます。

```
2023/12/05 20:23:06 INF Got Session Establishment Request from: 192.168.14.111.
2023/12/05 20:23:06 INF 
Session Establishment Request:
  CreatePDR ID: 1 
    FAR ID: 1 
    QER ID: 1 
    Source Interface: 1 
    UE IPv4 Address: 10.45.0.2 
  CreatePDR ID: 2 
    Outer Header Removal: 0 
    FAR ID: 2 
    QER ID: 1 
    Source Interface: 0 
    TEID: 0 
    Ipv4: <nil> 
    Ipv6: <nil> 
    UE IPv4 Address: 10.45.0.2 
  CreatePDR ID: 3 
    Outer Header Removal: 0 
    FAR ID: 1 
    Source Interface: 3 
    TEID: 0 
    Ipv4: <nil> 
    Ipv6: <nil> 
  CreatePDR ID: 4 
    Outer Header Removal: 0 
    FAR ID: 3 
    Source Interface: 0 
    TEID: 0 
    Ipv4: <nil> 
    Ipv6: <nil> 
    SDF Filter: permit out 58 from ff02::2/128 to assigned 
  CreateFAR ID: 1 
    Apply Action: [2 0] 
    Forwarding Parameters:
      Network Instance:internet 
      Outer Header Creation: &{OuterHeaderCreationDescription:256 TEID:60524 IPv4Address:192.168.13.112 IPv6Address:<nil> PortNumber:0 CTag:0 STag:0} 
  CreateFAR ID: 2 
    Apply Action: [2 0] 
    Forwarding Parameters:
      Network Instance:internet 
  CreateFAR ID: 3 
    Apply Action: [2 0] 
    Forwarding Parameters:
      Network Instance:internet 
      Outer Header Creation: &{OuterHeaderCreationDescription:256 TEID:1 IPv4Address:192.168.14.111 IPv6Address:<nil> PortNumber:0 CTag:0 STag:0} 
  CreateQER ID: 1 
    Gate Status DL: 0 
    Gate Status UL: 0 
    Max Bitrate DL: 1000000 
    Max Bitrate UL: 1000000 
  CreateBAR ID: 1

2023/12/05 20:23:06 INF Saving FAR info to session: 1, {Action:2 OuterHeaderCreation:1 Teid:60524 RemoteIP:1879943360 LocalIP:2534254784 TransportLevelMarking:0}
2023/12/05 20:23:06 INF WARN: No OuterHeaderCreation
2023/12/05 20:23:06 INF Saving FAR info to session: 2, {Action:2 OuterHeaderCreation:0 Teid:0 RemoteIP:0 LocalIP:2534254784 TransportLevelMarking:0}
2023/12/05 20:23:06 INF Saving FAR info to session: 3, {Action:2 OuterHeaderCreation:1 Teid:1 RemoteIP:1863231680 LocalIP:2534254784 TransportLevelMarking:0}
2023/12/05 20:23:06 INF Saving QER info to session: 1, {GateStatusUL:0 GateStatusDL:0 Qfi:0 MaxBitrateUL:1000000000 MaxBitrateDL:1000000000 StartUL:0 StartDL:0}
2023/12/05 20:23:06 Matched groups: [permit out 58 from ff02::2/128 to assigned 58 ff02::2 128  assigned  ]
2023/12/05 20:23:06 INF Session Establishment Request from 192.168.14.111 accepted.
```

VM4(UE)での`ip addr show`は以下の通りです。TUNnelインターフェース`tun_srsue`は次のように作成され、IPアドレス`10.45.0.2`がUEに割り当てられたことが分かります。

```
5: tun_srsue: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 500
    link/none 
    inet 10.45.0.2/24 scope global tun_srsue
       valid_lft forever preferred_lft forever
```

## google.comにpingする

UEのTUNインターフェースを指定して、`ping`してみます。

### DN=10.45.0.0/16を経由する場合

VM-DNで`tcpdump`を実行し、パケットがSGiインターフェース`enp0s9`を通過することを確認します。

- VM4(UE)で`ping google.com`

```
# ping google.com -I tun_srsue -n
PING google.com (142.251.222.14) from 10.45.0.2 tun_srsue: 56(84) bytes of data.
64 bytes from 142.251.222.14: icmp_seq=1 ttl=61 time=101 ms
64 bytes from 142.251.222.14: icmp_seq=2 ttl=61 time=73.8 ms
64 bytes from 142.251.222.14: icmp_seq=3 ttl=61 time=58.0 ms
```

- VM-DNで`tcpdump`を実行

```
# tcpdump -i enp0s9 -n
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on enp0s9, link-type EN10MB (Ethernet), snapshot length 262144 bytes
20:25:50.944272 IP 10.45.0.2 > 142.251.222.14: ICMP echo request, id 3, seq 1, length 64
20:25:50.973107 IP 142.251.222.14 > 10.45.0.2: ICMP echo reply, id 3, seq 1, length 64
20:25:51.933226 IP 10.45.0.2 > 142.251.222.14: ICMP echo request, id 3, seq 2, length 64
20:25:51.955657 IP 142.251.222.14 > 10.45.0.2: ICMP echo reply, id 3, seq 2, length 64
20:25:52.919980 IP 10.45.0.2 > 142.251.222.14: ICMP echo request, id 3, seq 3, length 64
20:25:52.938727 IP 142.251.222.14 > 10.45.0.2: ICMP echo reply, id 3, seq 3, length 64
```

- VM-UPで`/sys/kernel/debug/tracing/trace_pipe`を確認

```
# cat /sys/kernel/debug/tracing/trace_pipe
...
          <idle>-0       [000] d.s31  3184.066978: bpf_trace_printk: upf: gtp-u received
          <idle>-0       [000] d.s31  3184.066982: bpf_trace_printk: SDF: filter protocol: 4
          <idle>-0       [000] d.s31  3184.066985: bpf_trace_printk: SDF: filter source ip: 0.0.0.2, destination ip: 0.0.0.0
          <idle>-0       [000] d.s31  3184.066987: bpf_trace_printk: SDF: filter source ip mask: 255.255.255.255, destination ip mask: 0.0.0.0
          <idle>-0       [000] d.s31  3184.066988: bpf_trace_printk: SDF: filter source port lower bound: 0, source port upper bound: 65535
          <idle>-0       [000] d.s31  3184.066989: bpf_trace_printk: SDF: filter destination port lower bound: 0, destination port upper bound: 65535
          <idle>-0       [000] d.s31  3184.066989: bpf_trace_printk: SDF: packet protocol: 0
          <idle>-0       [000] d.s31  3184.066990: bpf_trace_printk: SDF: packet source ip: 10.45.0.2, destination ip: 142.251.222.14
          <idle>-0       [000] d.s31  3184.066991: bpf_trace_printk: SDF: packet source port: 0, destination port: 0
          <idle>-0       [000] d.s31  3184.066992: bpf_trace_printk: upf: sdf filter doesn't match teid:1
          <idle>-0       [000] d.s31  3184.066993: bpf_trace_printk: upf: far:1 action:2 outer_header_creation:0
          <idle>-0       [000] d.s31  3184.066994: bpf_trace_printk: upf: qer:0 gate_status:0 mbr:1000000000
          <idle>-0       [000] d.s31  3184.066995: bpf_trace_printk: upf: session for teid:1 far:1 outer_header_removal:0
          <idle>-0       [000] d.s31  3184.067003: bpf_trace_printk: upf: bpf_fib_lookup 10.45.0.2 -> 142.251.222.14: nexthop: 192.168.16.152
          <idle>-0       [000] d.s31  3184.096132: bpf_trace_printk: upf: downlink session for ip:10.45.0.2  far:0 action:2
          <idle>-0       [000] d.s31  3184.096135: bpf_trace_printk: upf: qer:0 gate_status:0 mbr:1000000000
          <idle>-0       [000] d.s31  3184.096136: bpf_trace_printk: upf: use mapping 10.45.0.2 -> TEID:60524
          <idle>-0       [000] d.s31  3184.096138: bpf_trace_printk: upf: send gtp pdu 192.168.13.151 -> 192.168.13.112
          <idle>-0       [000] d.s31  3184.096146: bpf_trace_printk: upf: bpf_fib_lookup 192.168.13.151 -> 192.168.13.112: nexthop: 192.168.13.112
          <idle>-0       [000] d.s31  3185.055908: bpf_trace_printk: upf: gtp-u received
          <idle>-0       [000] d.s31  3185.055912: bpf_trace_printk: SDF: filter protocol: 4
          <idle>-0       [000] d.s31  3185.055915: bpf_trace_printk: SDF: filter source ip: 0.0.0.2, destination ip: 0.0.0.0
          <idle>-0       [000] d.s31  3185.055916: bpf_trace_printk: SDF: filter source ip mask: 255.255.255.255, destination ip mask: 0.0.0.0
          <idle>-0       [000] d.s31  3185.055917: bpf_trace_printk: SDF: filter source port lower bound: 0, source port upper bound: 65535
          <idle>-0       [000] d.s31  3185.055918: bpf_trace_printk: SDF: filter destination port lower bound: 0, destination port upper bound: 65535
          <idle>-0       [000] d.s31  3185.055918: bpf_trace_printk: SDF: packet protocol: 0
          <idle>-0       [000] d.s31  3185.055938: bpf_trace_printk: SDF: packet source ip: 10.45.0.2, destination ip: 142.251.222.14
          <idle>-0       [000] d.s31  3185.055939: bpf_trace_printk: SDF: packet source port: 0, destination port: 0
          <idle>-0       [000] d.s31  3185.055940: bpf_trace_printk: upf: sdf filter doesn't match teid:1
          <idle>-0       [000] d.s31  3185.055941: bpf_trace_printk: upf: far:1 action:2 outer_header_creation:0
          <idle>-0       [000] d.s31  3185.055942: bpf_trace_printk: upf: qer:0 gate_status:0 mbr:1000000000
          <idle>-0       [000] d.s31  3185.055943: bpf_trace_printk: upf: session for teid:1 far:1 outer_header_removal:0
          <idle>-0       [000] d.s31  3185.055950: bpf_trace_printk: upf: bpf_fib_lookup 10.45.0.2 -> 142.251.222.14: nexthop: 192.168.16.152
          <idle>-0       [000] d.s31  3185.078676: bpf_trace_printk: upf: downlink session for ip:10.45.0.2  far:0 action:2
          <idle>-0       [000] d.s31  3185.078680: bpf_trace_printk: upf: qer:0 gate_status:0 mbr:1000000000
          <idle>-0       [000] d.s31  3185.078681: bpf_trace_printk: upf: use mapping 10.45.0.2 -> TEID:60524
          <idle>-0       [000] d.s31  3185.078683: bpf_trace_printk: upf: send gtp pdu 192.168.13.151 -> 192.168.13.112
          <idle>-0       [000] d.s31  3185.078691: bpf_trace_printk: upf: bpf_fib_lookup 192.168.13.151 -> 192.168.13.112: nexthop: 192.168.13.112
          <idle>-0       [000] d.s31  3186.042681: bpf_trace_printk: upf: gtp-u received
          <idle>-0       [000] d.s31  3186.042685: bpf_trace_printk: SDF: filter protocol: 4
          <idle>-0       [000] d.s31  3186.042688: bpf_trace_printk: SDF: filter source ip: 0.0.0.2, destination ip: 0.0.0.0
          <idle>-0       [000] d.s31  3186.042690: bpf_trace_printk: SDF: filter source ip mask: 255.255.255.255, destination ip mask: 0.0.0.0
          <idle>-0       [000] d.s31  3186.042691: bpf_trace_printk: SDF: filter source port lower bound: 0, source port upper bound: 65535
          <idle>-0       [000] d.s31  3186.042692: bpf_trace_printk: SDF: filter destination port lower bound: 0, destination port upper bound: 65535
          <idle>-0       [000] d.s31  3186.042692: bpf_trace_printk: SDF: packet protocol: 0
          <idle>-0       [000] d.s31  3186.042693: bpf_trace_printk: SDF: packet source ip: 10.45.0.2, destination ip: 142.251.222.14
          <idle>-0       [000] d.s31  3186.042694: bpf_trace_printk: SDF: packet source port: 0, destination port: 0
          <idle>-0       [000] d.s31  3186.042695: bpf_trace_printk: upf: sdf filter doesn't match teid:1
          <idle>-0       [000] d.s31  3186.042696: bpf_trace_printk: upf: far:1 action:2 outer_header_creation:0
          <idle>-0       [000] d.s31  3186.042697: bpf_trace_printk: upf: qer:0 gate_status:0 mbr:1000000000
          <idle>-0       [000] d.s31  3186.042698: bpf_trace_printk: upf: session for teid:1 far:1 outer_header_removal:0
          <idle>-0       [000] d.s31  3186.042705: bpf_trace_printk: upf: bpf_fib_lookup 10.45.0.2 -> 142.251.222.14: nexthop: 192.168.16.152
          <idle>-0       [000] d.s31  3186.061671: bpf_trace_printk: upf: downlink session for ip:10.45.0.2  far:0 action:2
          <idle>-0       [000] d.s31  3186.061674: bpf_trace_printk: upf: qer:0 gate_status:0 mbr:1000000000
          <idle>-0       [000] d.s31  3186.061676: bpf_trace_printk: upf: use mapping 10.45.0.2 -> TEID:60524
          <idle>-0       [000] d.s31  3186.061677: bpf_trace_printk: upf: send gtp pdu 192.168.13.151 -> 192.168.13.112
          <idle>-0       [000] d.s31  3186.061684: bpf_trace_printk: upf: bpf_fib_lookup 192.168.13.151 -> 192.168.13.112: nexthop: 192.168.13.112
...
```

`ping`以外にも、以下のように`curl`でTUNnelインターフェースを指定してWebにアクセスしてみることもできます。

- `curl google.com` on VM4 (UE)

```
# curl --interface tun_srsue google.com
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="http://www.google.com/">here</A>.
</BODY></HTML>
```

- VM-DNで`tcpdump`を実行

```
20:27:46.955340 IP 10.45.0.2.54964 > 172.217.161.78.80: Flags [S], seq 718929327, win 64240, options [mss 1460,sackOK,TS val 1478275073 ecr 0,nop,wscale 7], length 0
20:27:46.971398 IP 172.217.161.78.80 > 10.45.0.2.54964: Flags [S.], seq 192001, ack 718929328, win 65535, options [mss 1460], length 0
20:27:47.095296 IP 10.45.0.2.54964 > 172.217.161.78.80: Flags [.], ack 1, win 64240, length 0
20:27:47.095296 IP 10.45.0.2.54964 > 172.217.161.78.80: Flags [P.], seq 1:75, ack 1, win 64240, length 74: HTTP: GET / HTTP/1.1
20:27:47.095530 IP 172.217.161.78.80 > 10.45.0.2.54964: Flags [.], ack 75, win 65535, length 0
20:27:47.173318 IP 172.217.161.78.80 > 10.45.0.2.54964: Flags [P.], seq 1:774, ack 75, win 65535, length 773: HTTP: HTTP/1.1 301 Moved Permanently
20:27:47.234283 IP 10.45.0.2.54964 > 172.217.161.78.80: Flags [.], ack 774, win 63467, length 0
20:27:47.234283 IP 10.45.0.2.54964 > 172.217.161.78.80: Flags [F.], seq 75, ack 774, win 63467, length 0
20:27:47.234527 IP 172.217.161.78.80 > 10.45.0.2.54964: Flags [.], ack 76, win 65535, length 0
20:27:47.264013 IP 172.217.161.78.80 > 10.45.0.2.54964: Flags [F.], seq 774, ack 76, win 65535, length 0
20:27:47.332814 IP 10.45.0.2.54964 > 172.217.161.78.80: Flags [.], ack 775, win 63467, length 0
```

以上で、eBPF/XDP対応UPFのeUPFのPDNに接続し、PDNに任意のパケットを流せるようになり、プライベートLTEを想定したEPCのシミュレーション携帯網として、MECを試す環境を構築できました。

## 最後に

eBPF/XDP対応UPFのeUPFは、U-Planeのデータネットワークを高速化する技術と期待しています。本記事では、Virtualbox VM上の設定例を書いてみましたが、高性能な物理NICを搭載したサーバを設定する際のご参考になれれば幸いです。

最後に、元記事は[Github](https://github.com/s5uishida/open5gs_epc_srsran_eupf_sample_config)に書いたものです。

## 主な変更履歴

- [2024.02.22] 初版。