opened
JamesLovie
username_0
I've been Googling and going through logs to try and solve this, but I can't seem to get microk8s to work on my Raspberry Pi, running Ubuntu 20.10

Microk8s 1.21 just fails to start the node at all. Gets stuck on "kube-proxy  Starting kube-proxy." when running describe node command.

I then snap install v1.15 which at least starts the node to running state.
sudo snap install microk8s --classic --channel=1.15/stable

I can confirm that microk8s.status returns that it is running.
kubectl get nodes --namespace kube-system returns:
```
NAME         STATUS   ROLES    AGE   VERSION
mydnsname   Ready    <none>   22h   v1.15.11 
```

I have installed the following enabled: dashboard
I had dashboard dns ingress - installed but the same issue was present.

kubectl get all --all-namespaces returns
```
NAMESPACE     NAME                                                  READY   STATUS              RESTARTS   AGE
kube-system   pod/heapster-v1.5.2-c4dcfd9f8-kl5g5                   0/4     ContainerCreating   0          68s
kube-system   pod/kubernetes-dashboard-c775bf88b-pmhz6              0/1     ContainerCreating   0          68s
kube-system   pod/monitoring-influxdb-grafana-v4-6f74479987-rd5ck   0/2     ContainerCreating   0          68s


NAMESPACE     NAME                           TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
default       service/kubernetes             ClusterIP   10.152.183.1     <none>        443/TCP             18h
kube-system   service/heapster               ClusterIP   10.152.183.195   <none>        80/TCP              68s
kube-system   service/kubernetes-dashboard   ClusterIP   10.152.183.171   <none>        443/TCP             68s
kube-system   service/monitoring-grafana     ClusterIP   10.152.183.120   <none>        80/TCP              68s
kube-system   service/monitoring-influxdb    ClusterIP   10.152.183.175   <none>        8083/TCP,8086/TCP   68s


NAMESPACE     NAME                                             READY   UP-TO-DATE   AVAILABLE   AGE
kube-system   deployment.apps/heapster-v1.5.2                  0/1     1            0           68s
kube-system   deployment.apps/kubernetes-dashboard             0/1     1            0           68s
kube-system   deployment.apps/monitoring-influxdb-grafana-v4   0/1     1            0           68s

NAMESPACE     NAME                                                        DESIRED   CURRENT   READY   AGE
kube-system   replicaset.apps/heapster-v1.5.2-c4dcfd9f8                   1         1         0       68s
kube-system   replicaset.apps/kubernetes-dashboard-c775bf88b              1         1         0       68s
kube-system   replicaset.apps/monitoring-influxdb-grafana-v4-6f74479987   1         1         0       68s
```

kubectl describe pods kubernetes-dashboard-c775bf88b-pmhz6 --namespace kube-system returns
```
Warning  FailedCreatePodSandBox  2m26s (x3 over 2m54s)   kubelet, mydnsname  (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to start sandbox container: failed to create containerd task: failed to mount rootfs component &{overlay overlay [workdir=/var/snap/microk8s/common/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/1386/work upperdir=/var/snap/microk8s/common/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/1386/fs lowerdir=/var/snap/microk8s/common/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/2/fs]}: invalid argument: unknown
  Warning  MissingClusterDNS       2m14s (x13 over 4m58s)  kubelet, mydnsname  pod: "kubernetes-dashboard-c775bf88b-pmhz6_kube-system(777df7aa-d4de-40f0-8275-70b36d2e5029)". kubelet does not have ClusterDNS IP configured and cannot create Pod using "ClusterFirst" policy. Falling back to "Default" policy.
```

I have checked and I am not using ZFS as my filesystem for my main partition, I'm using overlayfs
df -Th
```
Filesystem     Type     Size  Used Avail Use% Mounted on
none           **overlay**   59G   21G   36G  37% /
tmpfs          tmpfs    1.9G     0  1.9G   0% /dev/shm
tmpfs          tmpfs    743M  1.2M  742M   1% /run
tmpfs          tmpfs    5.0M  4.0K  5.0M   1% /run/lock
tmpfs          tmpfs    4.0M     0  4.0M   0% /sys/fs/cgroup
tmpfs          tmpfs    372M  104K  372M   1% /run/user/1000
```
[inspection-report-20210507_181312.tar.gz](https://github.com/ubuntu/microk8s/files/6440095/inspection-report-20210507_181312.tar.gz)

Therefore I'm not sure why containerd can't mount or has an issue, considering that containerd doesn't have any issues with overlayfs.

Have tried microk8s.reset and uninstalling and reinstalling, and tried a few different versions and still to no avail.

Any ideas or assistance is greatly appreciated as I'm keen to get a local kubernetes setup for local dev!

Thanks in advance!
MicroK8s 1.15 containers unable to start. All pods stuck on "ContainerCreating" state. Ubuntu 20.10 RP4
issue
created
balchua
username_1
I will recommend that we fix your issue with MicroK8s v 1.21 atleast.  1.15 is no longer active.

So when you install microk8s v1.21 can you run the inspect command and upload the tarball?
Thanks.
comment
created
JamesLovie
username_0
Thanks @username_1 for getting back to me.

I have just:
```
sudo microk8s.reset
sudo microk8s stop && sudo snap remove microk8s
sudo snap install microk8s --classic --channel=1.21 
```

Then ran microk8s.inspect and uploaded tarball.

Thanks
[inspection-report-20210507_214053.tar.gz](https://github.com/ubuntu/microk8s/files/6441245/inspection-report-20210507_214053.tar.gz)
comment
created
balchua
username_1
Suggest to do 
`sudo snap remove microk8s --purge`. This will remove any remnants of microk8s.

Then you can do `sudo snap install microk8s --channel 1.21/stable`

May i know the rpi 4 specs?  Just so you know if ur running on a 1GB RAM, the kernel might kill the apiserver due to OOM.
comment
created
balchua
username_1
Hi @username_0 i just looked at the last tarball you uploaded.

I wonder if you have storage in place.  Microk8s need to have a good amount of storage in `/var/snap/microk8s`. Make sure you have that.

In the inspect tarball `df -h` yields nothing.
Can you do a `df -h|?
comment
created
JamesLovie
username_0
Hi @username_1 I think it's working now!

You were correct in that Microk8s, specifically containerd, didn't work with a squashfs with overlay.

Reason being, BerryBoot uses SquashFS images and the lowerdir I think was SquashFS for root.

I reinstalled Ubuntu using PINN, which makes root ext4 filesystem.

After enabling cgroups, I was able to start microk8s 1.21 and see the node in running state.

After running kubectl get all --namespace kube-system
I get all nodes returning running state now!

Thanks for your help.

James
comment
created
balchua
username_1
Glad that it's working!! Im going to close this issue! Thanks for using MicroK8s! 👍
comment
closed
balchua
username_1
issue