kubernetes cluster上にfree5gcを構築してみる(後編) free5gc コンテナ版のDeploy編


前編↓ではKubernetesクラスターを構築してFree5GCコンテナ版のDeployのための下準備が整いました。今回は後編としてFree5GCのコンテナ版をDeployしていきます。

https://qiita.com/wzm/items/0dbe928ed891d82380d6

# 1. Worker Node側の作業
# 1-1. GTPモジュールのInstall
worker NodeでUPF向けのGTPモジュールのコンパイル
makeとgccが入ってない場合はapt intallで入れます。
githubレポジトリからgtp5gをcloneしてコンパイルしてInstallします。
```
root@kubeworker:~# git clone https://github.com/PrinzOwO/gtp5g.git
Cloning into 'gtp5g'...
remote: Enumerating objects: 175, done.
remote: Counting objects: 100% (86/86), done.
remote: Compressing objects: 100% (66/66), done.
remote: Total 175 (delta 44), reused 49 (delta 20), pack-reused 89
Receiving objects: 100% (175/175), 103.70 KiB | 5.76 MiB/s, done.
Resolving deltas: 100% (88/88), done.
root@kubeworker:~# cd gtp5g/
root@kubeworker:~/gtp5g# make
make -C /usr/src/linux-headers-5.4.0-88-generic/ M=/root/gtp5g  modules
make[1]: Entering directory '/usr/src/linux-headers-5.4.0-88-generic'
  CC [M]  /root/gtp5g/gtp5g.o
  Building modules, stage 2.
  MODPOST 1 modules
  CC [M]  /root/gtp5g/gtp5g.mod.o
  LD [M]  /root/gtp5g/gtp5g.ko
make[1]: Leaving directory '/usr/src/linux-headers-5.4.0-88-generic'
root@kubeworker:~/gtp5g# sudo make install
modprobe udp_tunnel
cp gtp5g.ko /lib/modules/`uname -r`/kernel/drivers/net
depmod -a
modprobe gtp5g
echo "gtp5g" >> /etc/modules
root@kubeworker:~/gtp5g# lsmod|grep -i gtp
gtp5g                  94208  0
udp_tunnel             16384  2 gtp5g,vxlan

```
# 1-2. PVむけのフォルダ作成
あとwoker nodeではMongoDB用のPersistent Volume向けのフォルダを作っておきます
```
q14537@kubeworker1:~$ pwd
/home/q14537
q14537@kubeworker1:~$ mkdir kubedata
```

# 2. Master Node側の作業
# 2-1. PVの準備
MongoDB用のPV(Persistent Volume)の作成。以下の様にpv.yamlを準備。

```yaml
root@kubemaster:~# cat pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: example-local-pv
  labels:
    project: free5gc
spec:
  capacity:
    storage: 8Gi
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  local:
    path: /home/q14537/kubedata
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - kubeworker1
```

free5gcというNamespaceを作成してpv.yamlをapply。そうするとexample-local-pvが出来上がる。

```
root@kubemaster:~# kubectl create ns free5gc
namespace/free5gc created
root@kubemaster:~# kubectl get pv
No resources found
root@kubemaster:~# kubectl get pv -n free5gc
No resources found
root@kubemaster:~# kubectl apply -f pv.yaml
persistentvolume/example-local-pv created
root@kubemaster:~# kubectl get pv -n free5gc
NAME               CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   REASON   AGE
example-local-pv   8Gi        RWO            Retain           Available                                   3s
```
# 2-2. Helmレポジトリの準備
helmでfree5gc用のレポジトリを追加

```
root@kubemaster:~# helm repo add towards5gs 'https://raw.githubusercontent.com/Orange-OpenSource/towards5gs-helm/main/repo/'
"towards5gs" has been added to your repositories
root@kubemaster:~# helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "towards5gs" chart repository
Update Complete. ⎈Happy Helming!⎈
root@kubemaster:~# helm search repo
NAME                            CHART VERSION   APP VERSION     DESCRIPTION
towards5gs/free5gc              1.1.3           v3.2.0          A Helm chart to deploy Free5gc
towards5gs/free5gc-amf          0.2.3           v3.2.0          A Helm chart to deploy the Free5GC AMF
towards5gs/free5gc-ausf         0.2.3           v3.2.0          A Helm chart to deploy the Free5GC AUSF
towards5gs/free5gc-n3iwf        0.2.3           v3.2.0          A Helm chart to deploy the Free5GC N3IWF
towards5gs/free5gc-nrf          0.2.3           v3.2.0          A Helm chart to deploy the Free5GC NRF
towards5gs/free5gc-nssf         0.2.3           v3.2.0          A Helm chart to deploy the Free5GC NSSF
towards5gs/free5gc-pcf          0.2.3           v3.2.0          A Helm chart to deploy the Free5GC PCF
towards5gs/free5gc-smf          0.2.3           v3.2.0          A Helm chart to deploy the Free5GC SMF
towards5gs/free5gc-udm          0.2.3           v3.2.0          A Helm chart to deploy the Free5GC UDM
towards5gs/free5gc-udr          0.2.3           v3.2.0          A Helm chart to deploy the Free5GC UDR
towards5gs/free5gc-upf          0.2.3           v3.2.0          A Helm chart to deploy the Free5GC User Plane
towards5gs/free5gc-webui        0.1.3           v3.2.0          A Helm chart to deploy the Free5GC WEBUI
towards5gs/free5gcControlPlane  0.1.2           v3.0.5          DEPRECATED - A Helm chart to deploy the control...
towards5gs/free5gcN3iwf         0.1.2           v3.0.5          DEPRECATED - A Helm chart to deploy the n3iwf f...
towards5gs/free5gcUserPlane     0.1.2           v3.0.5          DEPRECATED - Helm chart to deploy the user plan...
towards5gs/networks5g           0.1.2           0.1.2           DEPRECATED - A Helm chart to deploy the user pl...
towards5gs/ueransim             2.0.14          v3.2.6          A Helm chart to deploy UERANSIM

```
# 2-3. コンテナ間通信用にMulturs CNIをInstall
Multus CNI PluginをInstall

```
root@kubemaster:~# git clone https://github.com/k8snetworkplumbingwg/multus-cni.git && cd multus-cni
Cloning into 'multus-cni'...
remote: Enumerating objects: 39541, done.
remote: Total 39541 (delta 0), reused 0 (delta 0), pack-reused 39541
Receiving objects: 100% (39541/39541), 49.08 MiB | 3.51 MiB/s, done.
Resolving deltas: 100% (18232/18232), done.
root@kubemaster:~/multus-cni# ls
cmd  CODE_OF_CONDUCT.md  CONTRIBUTING.md  deployments  docs  e2e  examples  go.mod  go.sum  hack  images  LICENSE  NOTICE  pkg  README.md  vendor
root@kubemaster:~/multus-cni# cat ./deployments/multus-daemonset.yml |kubectl apply -f -
customresourcedefinition.apiextensions.k8s.io/network-attachment-definitions.k8s.cni.cncf.io created
clusterrole.rbac.authorization.k8s.io/multus created
clusterrolebinding.rbac.authorization.k8s.io/multus created
serviceaccount/multus created
configmap/multus-cni-config created
daemonset.apps/kube-multus-ds created

root@kubemaster:~/multus-cni# kubectl get pods --all-namespaces | grep -i multus
kube-system    kube-multus-ds-8s6nw                 0/1     Init:0/1   0          17s
kube-system    kube-multus-ds-zsvpq                 0/1     Init:0/1   0          17s

```
# 3. Free5GCとUERANSIMのDeploy
# 3-1. まずはHelmを使ってFree5GCのDeploy

ますはLocal PCにHelmファイルをダウンロード
```
helm pull towards5gs/free5gc
```
tgz形式でダウンロードされるので解凍する。
```
root@kubemaster:~# tar -zxvf  free5gc-1.1.3.tgz
```
helmチャート内のvalues.yamlファイル内に定義されているinterface名をVirtualBox向けに編集。eth0,eth1になっている部分をenp0s8に書き換えます。
```
root@kubemaster:~# vi free5gc/values.yaml
```
```diff_yaml
  #Global network parametes
  n2network:
    name: n2network
-    masterIf: eth0
+    masterIf: enp0s8
    subnetIP: 10.100.50.248
    cidr: 29
    gatewayIP: 10.100.50.254
    excludeIP: 10.100.50.254
  n3network:
    name: n3network
-    masterIf: eth0
+    masterIf: enp0s8
    subnetIP: 10.100.50.232
    cidr: 29
    gatewayIP: 10.100.50.238
    excludeIP: 10.100.50.238
  n4network:
    name: n4network
-    masterIf: eth0
+    masterIf: enp0s8
    subnetIP: 10.100.50.240
    cidr: 29
    gatewayIP: 10.100.50.246
    excludeIP: 10.100.50.246
  n6network:
    name: n6network
-    masterIf: eth1
+    masterIf: enp0s8
    subnetIP: 10.100.100.0
    cidr: 24
    gatewayIP: 10.100.100.1
    excludeIP: 10.100.100.254
  n9network:
    name: n9network
-    masterIf: eth0
+    masterIf: enp0s8
    subnetIP: 10.100.50.224
    cidr: 29
    gatewayIP: 10.100.50.230
    excludeIP: 10.100.50.230
```
helmを使ってfree5gcのコンテナ達をDeploy
```
root@kubemaster:~# helm -n free5gc install free5gc-v1 ./free5gc
NAME: free5gc-v1
LAST DEPLOYED: Mon Jan  9 22:54:30 2023
NAMESPACE: free5gc
STATUS: deployed
REVISION: 1
NOTES:
#
# Software Name : towards5gs-helm
# SPDX-FileCopyrightText: Copyright (c) 2021 Orange
# SPDX-License-Identifier: Apache-2.0
#
# This software is distributed under the Apache License 2.0,
# the text of which is available at https://github.com/Orange-OpenSource/towards5gs-helm/blob/main/LICENSE
# or see the "LICENSE" file for more details.
#
# Author: Abderaouf KHICHANE, Ilhem FAJJARI
# Software description: An open-source project providing Helm charts to deploy 5G components (Core + RAN) on top of Kubernetes
#
#
# Visit the project at https://github.com/Orange-OpenSource/towards5gs-helm
#

1. Get the list of created Pods by running:
  kubectl get pods --namespace free5gc -l "project="
```
コンテナ状態のチェック。mongodbだけは直ぐに立ち上がると思います。

```
root@kubemaster:~# kubectl get pods -n free5gc
NAME                                              READY   STATUS              RESTARTS   AGE
free5gc-v1-free5gc-amf-amf-5578495746-f8h5k       0/1     Init:0/1            0          67s
free5gc-v1-free5gc-ausf-ausf-7fdc67bc7b-tkn29     0/1     Init:0/1            0          67s
free5gc-v1-free5gc-nrf-nrf-5fcd95cff7-zkwkw       0/1     Init:0/1            0          66s
free5gc-v1-free5gc-nssf-nssf-5b5dbc6c8f-rssmk     0/1     Init:0/1            0          67s
free5gc-v1-free5gc-pcf-pcf-584dcc844-tr5lz        0/1     Init:0/1            0          66s
free5gc-v1-free5gc-smf-smf-59b89bd7df-gj9qw       0/1     Init:0/1            0          66s
free5gc-v1-free5gc-udm-udm-7cc9d9c4-t6642         0/1     Init:0/1            0          66s
free5gc-v1-free5gc-udr-udr-688d9cd66d-6cdh7       0/1     Init:0/1            0          67s
free5gc-v1-free5gc-upf-upf-cf6f64c78-gzswg        0/1     ContainerCreating   0          67s
free5gc-v1-free5gc-webui-webui-5c9dd574df-mh5zn   0/1     Init:0/1            0          67s
mongodb-0                                         0/1     ContainerCreating   0          66s
```
そのままだとなぜか5GC NF達のPodが立ち上がらない事があるので以下のワークアラウンドの適用する
参考:
https://githubhelp.com/orange-opensource/towards5gs-helm/issues/4
```
root@kubemaster:~# kubectl -n kube-system rollout restart deployment coredns
deployment.apps/coredns restarted
```
しばらく待って再度ステータスチェックすると、全Pod立ち上がりました！
```
root@kubemaster:~# kubectl get pods -n free5gc
NAME                                              READY   STATUS    RESTARTS   AGE
free5gc-v1-free5gc-amf-amf-5578495746-g5fhz       1/1     Running   0          2m55s
free5gc-v1-free5gc-ausf-ausf-7fdc67bc7b-n7nbr     1/1     Running   0          2m55s
free5gc-v1-free5gc-nrf-nrf-5fcd95cff7-cw57z       1/1     Running   0          2m55s
free5gc-v1-free5gc-nssf-nssf-5b5dbc6c8f-6fwgm     1/1     Running   0          2m55s
free5gc-v1-free5gc-pcf-pcf-584dcc844-fcg7b        1/1     Running   0          2m55s
free5gc-v1-free5gc-smf-smf-59b89bd7df-xfhxt       1/1     Running   0          2m55s
free5gc-v1-free5gc-udm-udm-7cc9d9c4-vs8qh         1/1     Running   0          2m55s
free5gc-v1-free5gc-udr-udr-688d9cd66d-b6hpp       1/1     Running   0          2m55s
free5gc-v1-free5gc-upf-upf-cf6f64c78-5vqb5        1/1     Running   0          2m55s
free5gc-v1-free5gc-webui-webui-5c9dd574df-x8lhw   1/1     Running   0          2m55s
mongodb-0                                         1/1     Running   0          2m55s
```
# 3-2. 次はWebUIを使ってコール試験用のUE登録をします
以下の様にport 30500番でWebUIサービスが立ち上がってますのでホスト側のLocalPCから接続してみます
```
root@kubemaster:~#  kubectl get svc -n free5gc
NAME            TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
amf-namf        ClusterIP   10.99.46.15      <none>        80/TCP           46m
ausf-nausf      ClusterIP   10.110.129.146   <none>        80/TCP           46m
gnb-service     ClusterIP   10.98.232.34     <none>        4997/UDP         40m
mongodb         ClusterIP   10.101.229.35    <none>        27017/TCP        46m
nrf-nnrf        ClusterIP   10.111.51.185    <none>        8000/TCP         46m
nssf-nnssf      ClusterIP   10.109.244.6     <none>        80/TCP           46m
pcf-npcf        ClusterIP   10.102.192.231   <none>        80/TCP           46m
smf-nsmf        ClusterIP   10.101.37.83     <none>        80/TCP           46m
udm-nudm        ClusterIP   10.108.118.199   <none>        80/TCP           46m
udr-nudr        ClusterIP   10.104.238.37    <none>        80/TCP           46m
webui-service   NodePort    10.98.76.101     <none>        5000:30500/TCP   46m
```
LocalPC側でSSH転送設定
DOS窓で以下のコマンドを投入
```
C:\Program Files\Oracle\VirtualBox>ssh q14537@192.168.56.103 -L 5000:localhost:30500
```
LocalPCのWebブラウザからWebUIにアクセスできると思います。

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/241184/c4893063-49eb-9010-a545-89d8d103a92e.png)

コール試験用の加入者データをNew Subscriberボタンをクリックして追加します。
各種値はデフォルトのままでOK

![image.png](https://qiita-image-store.s3.ap-northeast-1.amazonaws.com/0/241184/8848d3bd-7b35-bcd1-b1cf-52ead08f9590.png)

# 3-3. Helmを使ってUERANSIMのDeploy
ここまで準備が出来たら後はUERANSIM使ってコール試験をしてみます。
```
root@kubemaster:~# helm pull towards5gs/ueransim
root@kubemaster:~# ls
free5gc-1.1.3.tgz  get-helm-3  multus-cni  pv.yaml  snap  towards5gs  towards5gs-helm  ueransim-2.0.14.tgz
root@kubemaster:~# tar -zxvf ueransim-2.0.14.tgz
ueransim/Chart.yaml
ueransim/values.yaml
ueransim/templates/NOTES.txt
ueransim/templates/_helpers.tpl
ueransim/templates/gnb/gnb-configmap.yaml
ueransim/templates/gnb/gnb-deployment.yaml
ueransim/templates/gnb/gnb-n2-nad.yaml
ueransim/templates/gnb/gnb-n3-nad.yaml
ueransim/templates/gnb/gnb-service.yaml
ueransim/templates/tests/connectivity-test.yaml
ueransim/templates/ue/ue-configmap.yaml
ueransim/templates/ue/ue-deployment.yaml
ueransim/README.md
ueransim/multicluster-enabled-values.yaml
ueransim/open5gs-values.yaml
ueransim/services-enabled-values.yaml
```
Free5GCと同じくvi ueransim/values.yamlでvalue.yamlのinterface名変更
```diff_yaml
global:
  multiCluster: false
  #Global network parametes
  n2network:
    name: n2network
-    masterIf: eth0
+    masterIf: enp0s8
    subnetIP: 10.100.50.248
    cidr: 29
    gatewayIP: 10.100.50.254
    excludeIP: 10.100.50.254
  n3network:
    name: n3network
-    masterIf: eth0
+    masterIf: enp0s8
    subnetIP: 10.100.50.232
    cidr: 29
    gatewayIP: 10.100.50.238
    excludeIP: 10.100.50.238
```
でDeploy
```
root@kubemaster:~# helm -n free5gc install ueransim-v1 ./ueransim/
NAME: ueransim-v1
LAST DEPLOYED: Wed Jan 11 22:57:51 2023
NAMESPACE: free5gc
STATUS: deployed
REVISION: 1
NOTES:
#
# Software Name : towards5gs-helm
# SPDX-FileCopyrightText: Copyright (c) 2021 Orange
# SPDX-License-Identifier: Apache-2.0
#
# This software is distributed under the Apache License 2.0,
# the text of which is available at https://github.com/Orange-OpenSource/towards5gs-helm/blob/main/LICENSE
# or see the "LICENSE" file for more details.
#
# Author: Abderaouf KHICHANE, Ilhem FAJJARI
# Software description: An open-source project providing Helm charts to deploy 5G components (Core + RAN) on top of Kubernetes
#
#
# Visit the project at https://github.com/Orange-OpenSource/towards5gs-helm
#

1. Run UE connectivity test by running these commands:
  helm --namespace free5gc test ueransim-v1

If you want to run connectivity tests manually, follow:

1. Get the UE Pod name by running:
  export POD_NAME=$(kubectl get pods --namespace free5gc -l "component=ue" -o jsonpath="{.items[0].metadata.name}")

2. Check that uesimtun0 interface has been created by running these commands:
  kubectl --namespace free5gc logs $POD_NAME
  kubectl --namespace free5gc exec -it $POD_NAME -- ip address

3. Try to access internet from the UE by running:
  kubectl --namespace free5gc exec -it $POD_NAME -- ping -I uesimtun0 www.google.com
  kubectl --namespace free5gc exec -it $POD_NAME -- curl --interface uesimtun0 www.google.com
  kubectl --namespace free5gc exec -it $POD_NAME -- traceroute -i uesimtun0 www.google.com
```
ステータスチェックするとueransimのPod達が立ち上がってます
```
root@kubemaster:~# kubectl get pods -n free5gc -o wide
NAME                                              READY   STATUS    RESTARTS   AGE   IP             NODE          NOMINATED NODE   READINESS GATES
free5gc-v1-free5gc-amf-amf-5578495746-hbmkm       1/1     Running   0          31m   10.244.1.226   kubeworker1   <none>           <none>
free5gc-v1-free5gc-ausf-ausf-7fdc67bc7b-xjxcd     1/1     Running   0          31m   10.244.1.228   kubeworker1   <none>           <none>
free5gc-v1-free5gc-nrf-nrf-5fcd95cff7-2v26m       1/1     Running   0          31m   10.244.1.222   kubeworker1   <none>           <none>
free5gc-v1-free5gc-nssf-nssf-5b5dbc6c8f-k4qdm     1/1     Running   0          31m   10.244.1.227   kubeworker1   <none>           <none>
free5gc-v1-free5gc-pcf-pcf-584dcc844-gvbks        1/1     Running   0          31m   10.244.1.223   kubeworker1   <none>           <none>
free5gc-v1-free5gc-smf-smf-59b89bd7df-clxtf       1/1     Running   0          31m   10.244.1.225   kubeworker1   <none>           <none>
free5gc-v1-free5gc-udm-udm-7cc9d9c4-wh86g         1/1     Running   0          31m   10.244.1.230   kubeworker1   <none>           <none>
free5gc-v1-free5gc-udr-udr-688d9cd66d-zlsgc       1/1     Running   0          31m   10.244.1.232   kubeworker1   <none>           <none>
free5gc-v1-free5gc-upf-upf-cf6f64c78-ct46v        1/1     Running   0          31m   10.244.1.229   kubeworker1   <none>           <none>
free5gc-v1-free5gc-webui-webui-5c9dd574df-62z2c   1/1     Running   0          31m   10.244.1.224   kubeworker1   <none>           <none>
mongodb-0                                         1/1     Running   0          31m   10.244.1.231   kubeworker1   <none>           <none>
ueransim-v1-gnb-6cc6674bbc-5m24w                  1/1     Running   0          26m   10.244.1.236   kubeworker1   <none>           <none>
ueransim-v1-ue-7c56b95cd-lzrg2                    1/1     Running   0          26m   10.244.1.235   kubeworker1   <none>           <none>
```
Deployした後の出てくる説明通りにのLogを確認してみます。
```
root@kubemaster:~# export POD_NAME=$(kubectl get pods --namespace free5gc -l "component=ue" -o jsonpath="{.items[0].metadata.name}")
root@kubemaster:~# kubectl --namespace free5gc logs $POD_NAME
UERANSIM v3.2.6
[2023-01-11 13:57:54.129] [nas] [info] UE switches to state [MM-DEREGISTERED/PLMN-SEARCH]
[2023-01-11 13:57:56.145] [rrc] [debug] New signal detected for cell[1], total [1] cells in coverage
[2023-01-11 13:57:56.146] [nas] [error] PLMN selection failure, no cells in coverage
[2023-01-11 13:57:56.146] [nas] [info] Selected plmn[208/93]
[2023-01-11 13:57:56.146] [rrc] [info] Selected cell plmn[208/93] tac[1] category[SUITABLE]
[2023-01-11 13:57:56.147] [nas] [info] UE switches to state [MM-DEREGISTERED/PS]
[2023-01-11 13:57:56.147] [nas] [info] UE switches to state [MM-DEREGISTERED/NORMAL-SERVICE]
[2023-01-11 13:57:56.147] [nas] [debug] Initial registration required due to [MM-DEREG-NORMAL-SERVICE]
[2023-01-11 13:57:56.147] [nas] [debug] UAC access attempt is allowed for identity[0], category[MO_sig]
[2023-01-11 13:57:56.147] [nas] [debug] Sending Initial Registration
[2023-01-11 13:57:56.147] [nas] [info] UE switches to state [MM-REGISTER-INITIATED]
[2023-01-11 13:57:56.147] [rrc] [debug] Sending RRC Setup Request
[2023-01-11 13:57:56.152] [rrc] [info] RRC connection established
[2023-01-11 13:57:56.152] [rrc] [info] UE switches to state [RRC-CONNECTED]
[2023-01-11 13:57:56.152] [nas] [info] UE switches to state [CM-CONNECTED]
[2023-01-11 13:57:56.637] [nas] [debug] Authentication Request received
[2023-01-11 13:57:56.638] [nas] [debug] Sending Authentication Failure due to SQN out of range
[2023-01-11 13:57:57.044] [nas] [debug] Authentication Request received
[2023-01-11 13:57:57.261] [nas] [debug] Security Mode Command received
[2023-01-11 13:57:57.261] [nas] [debug] Selected integrity[2] ciphering[0]
[2023-01-11 13:57:59.042] [nas] [debug] Registration accept received
[2023-01-11 13:57:59.042] [nas] [info] UE switches to state [MM-REGISTERED/NORMAL-SERVICE]
[2023-01-11 13:57:59.042] [nas] [debug] Sending Registration Complete
[2023-01-11 13:57:59.042] [nas] [info] Initial Registration is successful
[2023-01-11 13:57:59.042] [nas] [debug] Sending PDU Session Establishment Request
[2023-01-11 13:57:59.042] [nas] [debug] UAC access attempt is allowed for identity[0], category[MO_sig]
[2023-01-11 13:58:01.040] [nas] [debug] PDU Session Establishment Accept received
[2023-01-11 13:58:01.040] [nas] [info] PDU Session establishment is successful PSI[1]
[2023-01-11 13:58:01.093] [app] [info] Connection setup for PDU session[1] is successful, TUN interface[uesimtun0, 10.1.0.1] is up.
```
UERANSIMはDeployすると勝手にRegitrationからPDU Session確立までのシーケンスが実行される様で、ログの最後に"Connection setup for PDU session[1] is successful, TUN interface[uesimtun0, 10.1.0.1] is up."と出力されておるPDU Session確立が成功した事が分かります。

AMFのログとかも見てみます。 Handle PDU Session Resource Setup Responseと出ていて成功です。
```
root@kubemaster:~# kubectl -n free5gc logs free5gc-v1-free5gc-amf-amf-5578495746-hbmkm
Defaulted container "amf" out of: amf, wait-nrf (init)
2023-01-11T13:55:02Z [INFO][AMF][CFG] config version [1.0.3]
2023-01-11T13:55:02Z [INFO][AMF][Init] AMF Log level is set to [info] level
2023-01-11T13:55:02Z [INFO][LIB][NAS] set log level : info
2023-01-11T13:55:02Z [INFO][LIB][NAS] set report call : false
2023-01-11T13:55:02Z [INFO][LIB][NGAP] set log level : info
2023-01-11T13:55:02Z [INFO][LIB][NGAP] set report call : false
2023-01-11T13:55:02Z [INFO][LIB][FSM] set log level : info
2023-01-11T13:55:02Z [INFO][LIB][FSM] set report call : false
2023-01-11T13:55:02Z [INFO][LIB][Aper] set log level : info
2023-01-11T13:55:02Z [INFO][LIB][Aper] set report call : false
2023-01-11T13:55:02Z [INFO][AMF][App] amf
2023-01-11T13:55:02Z [INFO][AMF][App] AMF version:
        free5GC version: v3.2.0
        build time:      2022-08-15T14:13:37Z
        commit hash:     a3bd5358
        commit time:     2022-05-01T14:58:26Z
        go version:      go1.14.4 linux/amd64
2023-01-11T13:55:02Z [INFO][AMF][Init] Server started
2023-01-11T13:55:02Z [INFO][AMF][Util] amfconfig Info: Version[1.0.3] Description[AMF initial local configuration]
2023-01-11T13:55:02Z [INFO][AMF][NGAP] Listen on 10.100.50.249:38412
2023-01-11T13:57:54Z [INFO][AMF][NGAP] [AMF] SCTP Accept from: 10.100.50.250:52896
2023-01-11T13:57:54Z [INFO][AMF][NGAP] Create a new NG connection for: 10.100.50.250:52896
2023-01-11T13:57:54Z [INFO][AMF][NGAP][10.100.50.250:52896] Handle NG Setup request
2023-01-11T13:57:54Z [INFO][AMF][NGAP][10.100.50.250:52896] Send NG-Setup response
2023-01-11T13:57:56Z [INFO][AMF][NGAP][10.100.50.250:52896] Handle Initial UE Message
2023-01-11T13:57:56Z [INFO][LIB][FSM] Handle event[Gmm Message], transition from [Deregistered] to [Deregistered]
2023-01-11T13:57:56Z [INFO][AMF][GMM][AMF_UE_NGAP_ID:1] Handle Registration Request
2023-01-11T13:57:56Z [INFO][LIB][FSM] Handle event[Start Authentication], transition from [Deregistered] to [Authentication]
2023-01-11T13:57:56Z [INFO][AMF][GMM][AMF_UE_NGAP_ID:1] Authentication procedure
2023-01-11T13:57:56Z [INFO][AMF][GMM][AMF_UE_NGAP_ID:1] Send Authentication Request
2023-01-11T13:57:56Z [INFO][AMF][NGAP][10.100.50.250:52896][AMF_UE_NGAP_ID:1] Send Downlink Nas Transport
2023-01-11T13:57:56Z [INFO][AMF][NGAP][10.100.50.250:52896][AMF_UE_NGAP_ID:1] Uplink NAS Transport (RAN UE NGAP ID: 1)
2023-01-11T13:57:56Z [INFO][LIB][FSM] Handle event[Gmm Message], transition from [Authentication] to [Authentication]
2023-01-11T13:57:56Z [INFO][AMF][GMM][AMF_UE_NGAP_ID:1] Handle Authentication Failure
2023-01-11T13:57:56Z [WARN][AMF][GMM][AMF_UE_NGAP_ID:1] Authentication Failure 5GMM Cause: Synch Failure
2023-01-11T13:57:57Z [INFO][AMF][GMM][AMF_UE_NGAP_ID:1] Send Authentication Request
2023-01-11T13:57:57Z [INFO][AMF][NGAP][10.100.50.250:52896][AMF_UE_NGAP_ID:1] Send Downlink Nas Transport
2023-01-11T13:57:57Z [INFO][AMF][NGAP][10.100.50.250:52896][AMF_UE_NGAP_ID:1] Uplink NAS Transport (RAN UE NGAP ID: 1)
2023-01-11T13:57:57Z [INFO][LIB][FSM] Handle event[Gmm Message], transition from [Authentication] to [Authentication]
2023-01-11T13:57:57Z [INFO][AMF][GMM][AMF_UE_NGAP_ID:1] Handle Authentication Response
2023-01-11T13:57:57Z [INFO][LIB][FSM] Handle event[Authentication Success], transition from [Authentication] to [SecurityMode]
2023-01-11T13:57:57Z [INFO][AMF][GMM][AMF_UE_NGAP_ID:1][SUPI:imsi-208930000000003] Send Security Mode Command
2023-01-11T13:57:57Z [INFO][AMF][NGAP][10.100.50.250:52896][AMF_UE_NGAP_ID:1] Send Downlink Nas Transport
2023-01-11T13:57:57Z [INFO][AMF][NGAP][10.100.50.250:52896][AMF_UE_NGAP_ID:1] Uplink NAS Transport (RAN UE NGAP ID: 1)
2023-01-11T13:57:57Z [INFO][LIB][FSM] Handle event[Gmm Message], transition from [SecurityMode] to [SecurityMode]
2023-01-11T13:57:57Z [INFO][AMF][GMM][AMF_UE_NGAP_ID:1][SUPI:imsi-208930000000003] Handle Security Mode Complete
2023-01-11T13:57:57Z [INFO][LIB][FSM] Handle event[SecurityMode Success], transition from [SecurityMode] to [ContextSetup]
2023-01-11T13:57:57Z [INFO][AMF][GMM][AMF_UE_NGAP_ID:1][SUPI:imsi-208930000000003] Handle InitialRegistration
2023-01-11T13:57:57Z [INFO][AMF][GMM][AMF_UE_NGAP_ID:1][SUPI:imsi-208930000000003] RequestedNssai - ServingSnssai: &{Sst:1 Sd:010203}, HomeSnssai: <nil>
2023-01-11T13:57:59Z [INFO][AMF][Comm] Handle AMF Status Change Subscribe Request
2023-01-11T13:57:59Z [INFO][AMF][Comm] new AMF Status Subscription[1]
2023-01-11T13:57:59Z [INFO][AMF][GIN] | 201 |    10.244.1.223 | POST    | /namf-comm/v1/subscriptions |
2023-01-11T13:57:59Z [INFO][AMF][GMM][AMF_UE_NGAP_ID:1][SUPI:imsi-208930000000003] Send Registration Accept
2023-01-11T13:57:59Z [INFO][AMF][NGAP][10.100.50.250:52896][AMF_UE_NGAP_ID:1] Send Initial Context Setup Request
2023-01-11T13:57:59Z [INFO][AMF][NGAP][10.100.50.250:52896][AMF_UE_NGAP_ID:1] Handle Initial Context Setup Response
2023-01-11T13:57:59Z [INFO][AMF][NGAP][10.100.50.250:52896][AMF_UE_NGAP_ID:1] Uplink NAS Transport (RAN UE NGAP ID: 1)
2023-01-11T13:57:59Z [INFO][LIB][FSM] Handle event[Gmm Message], transition from [ContextSetup] to [ContextSetup]
2023-01-11T13:57:59Z [INFO][AMF][GMM][AMF_UE_NGAP_ID:1][SUPI:imsi-208930000000003] Handle Registration Complete
2023-01-11T13:57:59Z [INFO][LIB][FSM] Handle event[ContextSetup Success], transition from [ContextSetup] to [Registered]
2023-01-11T13:57:59Z [INFO][AMF][NGAP][10.100.50.250:52896][AMF_UE_NGAP_ID:1] Uplink NAS Transport (RAN UE NGAP ID: 1)
2023-01-11T13:57:59Z [INFO][LIB][FSM] Handle event[Gmm Message], transition from [Registered] to [Registered]
2023-01-11T13:57:59Z [INFO][AMF][GMM][AMF_UE_NGAP_ID:1][SUPI:imsi-208930000000003] Handle UL NAS Transport
2023-01-11T13:57:59Z [INFO][AMF][GMM][AMF_UE_NGAP_ID:1][SUPI:imsi-208930000000003] Transport 5GSM Message to SMF
2023-01-11T13:57:59Z [INFO][AMF][GMM][AMF_UE_NGAP_ID:1][SUPI:imsi-208930000000003] Select SMF [snssai: {Sst:1 Sd:010203}, dnn: internet]
2023-01-11T13:58:00Z [INFO][AMF][GMM][AMF_UE_NGAP_ID:1][SUPI:imsi-208930000000003] create smContext[pduSessionID: 1] Success
2023-01-11T13:58:01Z [INFO][AMF][Producer] Handle N1N2 Message Transfer Request
2023-01-11T13:58:01Z [INFO][AMF][NGAP][10.100.50.250:52896][AMF_UE_NGAP_ID:1] Send PDU Session Resource Setup Request
2023-01-11T13:58:01Z [INFO][AMF][GIN] | 200 |    10.244.1.225 | POST    | /namf-comm/v1/ue-contexts/imsi-208930000000003/n1-n2-messages |
2023-01-11T13:58:01Z [INFO][AMF][NGAP][10.100.50.250:52896][AMF_UE_NGAP_ID:1] Handle PDU Session Resource Setup Response
```
ここまで来れば本来ならば外部へのpingもGTP経由で通るはずなのですが、そこまではちょっと動作確認できていません。以下の様にpingの返事が返って来ません。。何かのInterface系の設定が間違っているのだと思いますが、まぁここまで出来たのでヨシ！トラブルシューティングする気力がないので、今回はここまで！中途半端ですみません…
```
root@kubemaster:~#   kubectl --namespace free5gc exec -it $POD_NAME -- ping -I uesimtun0 www.google.com
PING www.google.com (216.58.220.100) from 10.1.0.1 uesimtun0: 56(84) bytes of data.
^X^C
--- www.google.com ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4093ms

command terminated with exit code 1
```


# 4. その他
実験中は色々試行錯誤するのでコンテナのDeployやDestroyを繰り返す事になると思います。
なのでご参考。

DeployしたFree5GCを削除
```
root@kubemaster:~# helm -n free5gc uninstall free5gc-v1
```
DeployしたUERANSIMを削除
```
root@kubemaster:~# helm -n free5gc uninstall ueransim-v1
````
Kubernetes自体を一度作り直し
resetしてからinitし直します
```
root@kubemaster:~# kubeadm reset
````

# 5. 参考リンク

https://kubernetes.io/ja/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/
https://medium.com/rahasak/deploying-5g-core-network-with-free5gc-kubernets-and-helm-charts-29741cea3922
https://github.com/Orange-OpenSource
https://www.server-world.info/query?os=Ubuntu_20.04&p=kubernetes&f=4