Repository: canonical/microk8s

Issue:
microk8s.reset does not remove resources
<!--
   Thank you for submitting an issue. Please fill in the template below
   information about the bug you encountered.
-->

#### Summary
<!-- Please explain the bug in a few short sentences -->

#### What Should Happen Instead?
<!-- Please explain what the expected behavior is -->
All k8s resources should be remove, no exception!

#### System info
```
archlinux up-to-date
snap version
snap    2.56-1
snapd   2.56-1
series  16
arch    -
kernel  5.18.3-arch1-1
```

#### Reproduction Steps
<!-- Are you able to consistently reproduce the issue? Please add a list of steps that lead to the bug. -->

1. sudo snap install microk8s --classic
2. microk8s enable dns
```
2022-06-14T14:25:12+07:00 INFO Waiting for automatic snapd restart...
microk8s (1.24/stable) v1.24.0 from Canonical✓ installed
```
3. microk8s.kubectl get all -A
```
NAMESPACE     NAME                                           READY   STATUS              RESTARTS   AGE
kube-system   pod/calico-kube-controllers-5c54889544-9h5fk   0/1     ContainerCreating   0          58s
kube-system   pod/calico-node-5wtlb                          1/1     Running             0          59s

NAMESPACE   NAME                 TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
default     service/kubernetes   ClusterIP   10.152.183.1   <none>        443/TCP   66s

NAMESPACE     NAME                         DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
kube-system   daemonset.apps/calico-node   1         1         1       1            1           kubernetes.io/os=linux   64s

NAMESPACE     NAME                                      READY   UP-TO-DATE   AVAILABLE   AGE
kube-system   deployment.apps/calico-kube-controllers   0/1     1            0           64s

NAMESPACE     NAME                                                 DESIRED   CURRENT   READY   AGE
kube-system   replicaset.apps/calico-kube-controllers-5c54889544   1         1         0       58s
```
4. sudo microk8s.reset
```
Disabling all addons
Disabling addon : core/dashboard
Disabling addon : core/dns
Disabling addon : core/gpu
Disabling addon : core/helm
Disabling addon : core/helm3
Disabling addon : core/host-access
Disabling addon : core/hostpath-storage
Disabling addon : core/ingress
Disabling addon : core/mayastor
Disabling addon : core/metallb
Disabling addon : core/metrics-server
Disabling addon : core/prometheus
Disabling addon : core/rbac
Disabling addon : core/registry
Disabling addon : core/storage
All addons are disabled.
Deleting the CNI
Cleaning resources in namespace kube-system
Cleaning resources in namespace kube-public
Cleaning resources in namespace kube-node-lease
Cleaning resources in namespace default
Removing CRDs
Removing PriorityClasses
Removing StorageClasses
Restarting cluster
Stopped.
Started.
sudo: symbol lookup error: /snap/core18/current/lib/x86_64-linux-gnu/libpthread.so.0: undefined symbol: __libc_vfork, version GLIBC_PRIVATE
Setting up the CNI
```
5. microk8s.kubectl get all -A
```
NAMESPACE     NAME                                          READY   STATUS    RESTARTS   AGE
kube-system   pod/calico-node-m9jxn                         1/1     Running   0          2m26s
kube-system   pod/calico-kube-controllers-9969d55bb-b7t76   1/1     Running   0          2m25s

NAMESPACE   NAME                 TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
default     service/kubernetes   ClusterIP   10.152.183.1   <none>        443/TCP   5m21s

NAMESPACE     NAME                         DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
kube-system   daemonset.apps/calico-node   1         1         1       1            1           kubernetes.io/os=linux   3m28s

NAMESPACE     NAME                                      READY   UP-TO-DATE   AVAILABLE   AGE
kube-system   deployment.apps/calico-kube-controllers   1/1     1            1           3m28s

NAMESPACE     NAME                                                DESIRED   CURRENT   READY   AGE
kube-system   replicaset.apps/calico-kube-controllers-9969d55bb   1         1         1       2m25s
```
So as you can see above steps, resources weren't removed after `microk8s.reset` command.


I tried to do a clean refresh and re-installed snapd like below:
```
sudo systemctl disable snapd.socket
sudo rm -rf /var/lib/snapd
sudo pacman -Rns snapd
sudo shutdown -r now #reboot the pc
paru -S snapd #re-install snapd
sudo systemctl enable --now snapd.socket
```
Surprisingly is that even I've not enabled `dns` yet, after above command and then I've just issued the command `microk8s.kubectl get all -A`, all calico pods and all of the resources still there (even I removed /var/lib/snapd and reboot the pc)

So why resources weren't removed and how to have a clean reset microk8s?

Full `microk8s inspect` logs [here](https://drive.google.com/file/d/1B4A_cwIks64ZbNnKqKrjalpTj1u2JClj/view?usp=sharing)


Discussion:
Hi @tmpm697 

Thanks for reaching out. 

`microk8s reset` command is the correct way to have a clean reset. The command first clears and stops everything including the CNI, and then it starts up the required services and sets up the CNI.
You can see that the `replicaset` names and `pod` names are different between the logs you've provided, which can indicate that the CNI is reset as well. In addition I am not seeing any `dns` related workloads on your logs. To me it seems like the `reset` command works as intended. 

If this is not the case, could you clarify the issue a bit more? Thanks.


yes, i did enable `dns` for a while and then i manually kill some `dns`'s related resource like pods/daemonset/deployments.

that when the mess becomes.

eventually `calico` resources always keep spawned even after i removed `/var/lib/snapd` as i mentioned in the first post.

I recently tried with v1.23 and same issue but with below output when `microk8s.reset`:
```
Disabling all addons.
Disabling addon : ambassador
Disabling addon : cilium
Disabling addon : dashboard
Disabling addon : dashboard-ingress
Disabling addon : dns
Disabling addon : fluentd
Disabling addon : gpu
Disabling addon : helm
Disabling addon : helm3
Disabling addon : host-access
Disabling addon : inaccel
Disabling addon : ingress
Disabling addon : istio
Disabling addon : jaeger
Disabling addon : juju
Disabling addon : kata
Disabling addon : keda
Disabling addon : knative
Disabling addon : kubeflow
Disabling addon : linkerd
Disabling addon : metallb
Disabling addon : metrics-server
Disabling addon : multus
Disabling addon : openebs
Disabling addon : openfaas
Disabling addon : portainer
Disabling addon : prometheus
Disabling addon : rbac
Disabling addon : registry
Disabling addon : registry-help
Disabling addon : storage
Disabling addon : traefik
Waiting for kubernetes resources to be released
All addons disabled.
Deleting CNI
configmap "calico-config" deleted
customresourcedefinition.apiextensions.k8s.io "bgpconfigurations.crd.projectcalico.org" deleted
customresourcedefinition.apiextensions.k8s.io "bgppeers.crd.projectcalico.org" deleted
customresourcedefinition.apiextensions.k8s.io "blockaffinities.crd.projectcalico.org" deleted
customresourcedefinition.apiextensions.k8s.io "caliconodestatuses.crd.projectcalico.org" deleted
customresourcedefinition.apiextensions.k8s.io "clusterinformations.crd.projectcalico.org" deleted
customresourcedefinition.apiextensions.k8s.io "felixconfigurations.crd.projectcalico.org" deleted
customresourcedefinition.apiextensions.k8s.io "globalnetworkpolicies.crd.projectcalico.org" deleted
customresourcedefinition.apiextensions.k8s.io "globalnetworksets.crd.projectcalico.org" deleted
customresourcedefinition.apiextensions.k8s.io "hostendpoints.crd.projectcalico.org" deleted
customresourcedefinition.apiextensions.k8s.io "ipamblocks.crd.projectcalico.org" deleted
customresourcedefinition.apiextensions.k8s.io "ipamconfigs.crd.projectcalico.org" deleted
customresourcedefinition.apiextensions.k8s.io "ipamhandles.crd.projectcalico.org" deleted
customresourcedefinition.apiextensions.k8s.io "ippools.crd.projectcalico.org" deleted
customresourcedefinition.apiextensions.k8s.io "ipreservations.crd.projectcalico.org" deleted
customresourcedefinition.apiextensions.k8s.io "kubecontrollersconfigurations.crd.projectcalico.org" deleted
customresourcedefinition.apiextensions.k8s.io "networkpolicies.crd.projectcalico.org" deleted
customresourcedefinition.apiextensions.k8s.io "networksets.crd.projectcalico.org" deleted
clusterrole.rbac.authorization.k8s.io "calico-kube-controllers" deleted
clusterrolebinding.rbac.authorization.k8s.io "calico-kube-controllers" deleted
clusterrole.rbac.authorization.k8s.io "calico-node" deleted
clusterrolebinding.rbac.authorization.k8s.io "calico-node" deleted
daemonset.apps "calico-node" deleted
serviceaccount "calico-node" deleted
deployment.apps "calico-kube-controllers" deleted
serviceaccount "calico-kube-controllers" deleted
Warning: policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
poddisruptionbudget.policy "calico-kube-controllers" deleted
Calling clean_cluster
Cleaning resources in namespace kube-system
No resources found
No resources found
No resources found
configmap "extension-apiserver-authentication" deleted
configmap "kube-root-ca.crt" deleted
event "calico-kube-controllers-85b5b5888d-hzll7.16f87bf888c9cf20" deleted
event "calico-node-8bsv9.16f882511ec9579e" deleted
pod "calico-kube-controllers-85b5b5888d-hzll7" deleted
secret "service-account-controller-token-lvht2" deleted
secret "default-token-jfjrc" deleted
secret "resourcequota-controller-token-nj7g9" deleted
secret "generic-garbage-collector-token-jljg6" deleted
secret "root-ca-cert-publisher-token-s2pwt" deleted
secret "endpoint-controller-token-n4z85" deleted
secret "deployment-controller-token-kds5x" deleted
secret "certificate-controller-token-jzgmw" deleted
secret "service-controller-token-8jvj4" deleted
secret "endpointslicemirroring-controller-token-6xrnk" deleted
secret "node-controller-token-n5tsb" deleted
secret "cronjob-controller-token-sljbj" deleted
secret "ephemeral-volume-controller-token-mckh8" deleted
secret "statefulset-controller-token-8jd7z" deleted
secret "replication-controller-token-wnbzg" deleted
secret "ttl-controller-token-2rk96" deleted
secret "expand-controller-token-6fjgn" deleted
secret "job-controller-token-dtz4k" deleted
secret "replicaset-controller-token-trm5j" deleted
secret "attachdetach-controller-token-z5vz2" deleted
secret "clusterrole-aggregation-controller-token-xtc6d" deleted
secret "pvc-protection-controller-token-l6mx6" deleted
secret "pod-garbage-collector-token-m8c2s" deleted
secret "namespace-controller-token-djqqz" deleted
secret "disruption-controller-token-rqvrp" deleted
secret "pv-protection-controller-token-qmr28" deleted
secret "ttl-after-finished-controller-token-dhx8r" deleted
secret "endpointslice-controller-token-kdbj9" deleted
secret "daemon-set-controller-token-ns4k9" deleted
secret "horizontal-pod-autoscaler-token-9p248" deleted
secret "persistent-volume-binder-token-h5wvx" deleted
serviceaccount "service-controller" deleted
serviceaccount "endpointslicemirroring-controller" deleted
serviceaccount "node-controller" deleted
serviceaccount "cronjob-controller" deleted
serviceaccount "ephemeral-volume-controller" deleted
serviceaccount "statefulset-controller" deleted
serviceaccount "replication-controller" deleted
serviceaccount "ttl-controller" deleted
serviceaccount "expand-controller" deleted
serviceaccount "job-controller" deleted
serviceaccount "replicaset-controller" deleted
serviceaccount "attachdetach-controller" deleted
serviceaccount "clusterrole-aggregation-controller" deleted
serviceaccount "pvc-protection-controller" deleted
serviceaccount "pod-garbage-collector" deleted
serviceaccount "namespace-controller" deleted
serviceaccount "disruption-controller" deleted
serviceaccount "pv-protection-controller" deleted
serviceaccount "ttl-after-finished-controller" deleted
serviceaccount "endpointslice-controller" deleted
serviceaccount "daemon-set-controller" deleted
serviceaccount "horizontal-pod-autoscaler" deleted
serviceaccount "persistent-volume-binder" deleted
serviceaccount "service-account-controller" deleted
serviceaccount "default" deleted
serviceaccount "resourcequota-controller" deleted
serviceaccount "generic-garbage-collector" deleted
serviceaccount "deployment-controller" deleted
serviceaccount "root-ca-cert-publisher" deleted
serviceaccount "certificate-controller" deleted
serviceaccount "endpoint-controller" deleted
lease.coordination.k8s.io "kube-scheduler" deleted
lease.coordination.k8s.io "kube-controller-manager" deleted
error: timed out waiting for the condition on pods/calico-kube-controllers-85b5b5888d-hzll7
Cleaning resources in namespace kube-public
No resources found
No resources found
No resources found
configmap "kube-root-ca.crt" deleted
configmap "local-registry-hosting" deleted
secret "default-token-t5ztp" deleted
serviceaccount "default" deleted
Cleaning resources in namespace kube-node-lease
No resources found
No resources found
No resources found
configmap "kube-root-ca.crt" deleted
secret "default-token-zd7z4" deleted
serviceaccount "default" deleted
lease.coordination.k8s.io "localhost" deleted
Cleaning resources in namespace default
No resources found
No resources found
No resources found
configmap "kube-root-ca.crt" deleted
endpoints "kubernetes" deleted
event "localhost.16f87bf888c92161" deleted
secret "default-token-gpwfv" deleted
serviceaccount "default" deleted
service "kubernetes" deleted
endpointslice.discovery.k8s.io "kubernetes" deleted
Removing CRDs
No resources found
Removing PriorityClasses
Removing StorageClasses
Stopped.
Started.
Waiting for kubernetes resources to be released
Cleaning up addons client binaries..
Setting up the CNI
configmap/calico-config created
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/caliconodestatuses.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipreservations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers created
clusterrole.rbac.authorization.k8s.io/calico-node created
clusterrolebinding.rbac.authorization.k8s.io/calico-node created
daemonset.apps/calico-node created
serviceaccount/calico-node created
deployment.apps/calico-kube-controllers created
serviceaccount/calico-kube-controllers created
Warning: policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
poddisruptionbudget.policy/calico-kube-controllers created
```

Then `microk8s.kubectl get all -A` shows:
```
NAMESPACE     NAME                                           READY   STATUS    RESTARTS   AGE
kube-system   pod/calico-node-bgspf                          1/1     Running   0          49s
kube-system   pod/calico-kube-controllers-85b5b5888d-jn95r   1/1     Running   0          49s

NAMESPACE   NAME                 TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
default     service/kubernetes   ClusterIP   10.152.183.1   <none>        443/TCP   115s

NAMESPACE     NAME                         DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
kube-system   daemonset.apps/calico-node   1         1         1       1            1           kubernetes.io/os=linux   85s

NAMESPACE     NAME                                      READY   UP-TO-DATE   AVAILABLE   AGE
kube-system   deployment.apps/calico-kube-controllers   1/1     1            1           85s

NAMESPACE     NAME                                                 DESIRED   CURRENT   READY   AGE
kube-system   replicaset.apps/calico-kube-controllers-85b5b5888d   1         1         1       49s
```
My question is simple: why `reset` doesn't clean up all resources? is that expected behavior?
Calico is responsible for the networking of pods. It is expected that reset will give you back a clean and working cluster, therefore calico should be running after the reset command finishes. 
so what is default service will be enabled by default when you have a fresh microk8s? because from what i can recall before the issue, `microk8s.kubectl get all -A` would give empty resources.

only when you `microk8s enable dns` would spawn `calico` resources. is that correct?
Only the calico CNI and its services are available after a fresh install (or after a `microk8s.reset`).
> Only the calico CNI and its services are available after a fresh install (or after a `microk8s.reset`).

Are those pods, replicaset, deployment, daemonset are default when you have a fresh microk8s? like I mentioned above:


```
> NAMESPACE     NAME                                           READY   STATUS    RESTARTS   AGE
> kube-system   pod/calico-node-bgspf                          1/1     Running   0          49s
> kube-system   pod/calico-kube-controllers-85b5b5888d-jn95r   1/1     Running   0          49s
> 
> NAMESPACE   NAME                 TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
> default     service/kubernetes   ClusterIP   10.152.183.1   <none>        443/TCP   115s
> 
> NAMESPACE     NAME                         DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
> kube-system   daemonset.apps/calico-node   1         1         1       1            1           kubernetes.io/os=linux   85s
> 
> NAMESPACE     NAME                                      READY   UP-TO-DATE   AVAILABLE   AGE
> kube-system   deployment.apps/calico-kube-controllers   1/1     1            1           85s
> 
> NAMESPACE     NAME                                                 DESIRED   CURRENT   READY   AGE
> kube-system   replicaset.apps/calico-kube-controllers-85b5b5888d   1         1         1       49s
```


Yes, this is the calico CNI.
I thought a fresh microk8s would have none resources in it? How to remove all resources?
You can remove the CNI with:
```
microk8s kubectl delete -f  /var/snap/microk8s/current/args/cni-network/cni.yaml
```
I saw some yaml files under `/var/snap/microk8s/current`, do they represent what will be in fresh state of microk8s?
```
args/cni-network/cni.yaml
args/traefik/provider-template.yaml
args/traefik/traefik-template.yaml
inspection-report/dqlite/cluster.yaml
inspection-report/dqlite/info.yaml
inspection-report/dqlite/localnode.yaml
var/kubernetes/backend/cluster.yaml
var/kubernetes/backend/info.yaml
var/kubernetes/backend/localnode.yaml
```

> do they represent what will be in fresh state of microk8s?

No. 

Only the calico CNI is available in a fresh install (or after a microk8s.reset).
Hi, I have a similar issue. I had a cluster with some deployments, among others, vault, prometheus, and cert-manager. I wanted to tear down the cluster by using `sudo microk8s reset`. Most of the deployed resources were removed, however, all services are still shown when calling `microk8s get all --all-namespaces`. From what I understand from the above comments, the `pod`, `demonset`, `deployment`, and `replicaset`resources are expected, but is that true for the services?

``` bash
admin@node1:~$ sudo microk8s reset
Disabling all addons
Disabling addon : core/cert-manager
Disabling addon : core/dashboard
Disabling addon : core/dns
Disabling addon : core/gpu
Disabling addon : core/helm
Disabling addon : core/helm3
Disabling addon : core/host-access
Disabling addon : core/hostpath-storage
Disabling addon : core/ingress
Disabling addon : core/kube-ovn
Disabling addon : core/mayastor
Disabling addon : core/metallb
Disabling addon : core/metrics-server
Disabling addon : core/observability
Disabling addon : core/prometheus
Disabling addon : core/rbac
Disabling addon : core/registry
Disabling addon : core/storage
All addons are disabled.
Deleting the CNI
Cleaning resources in namespace kube-system
Cleaning resources in namespace kube-public
Cleaning resources in namespace kube-node-lease
Cleaning resources in namespace default
Removing CRDs
Removing PriorityClasses
Removing StorageClasses
Restarting cluster
Stopped.
Setting up the CNI
admin@node1:~$ kubectl get all --all-namespaces
NAMESPACE     NAME                                          READY   STATUS    RESTARTS   AGE
kube-system   pod/calico-node-bnjn4                         1/1     Running   0          61s
kube-system   pod/calico-kube-controllers-d8b9b6478-6pdwg   1/1     Running   0          60s

NAMESPACE     NAME                                                         TYPE           CLUSTER-IP       EXTERNAL-IP    PORT(S)                                        AGE
default       service/kubernetes                                           ClusterIP      10.152.183.1     <none>         443/TCP                                        10d
kube-system   service/kube-dns                                             ClusterIP      10.152.183.10    <none>         53/UDP,53/TCP,9153/TCP                         10d
kube-system   service/prometheus-stack-kube-prom-kubelet                   ClusterIP      None             <none>         10250/TCP,10255/TCP,4194/TCP                   9d
default       service/cert-manager                                         ClusterIP      10.152.183.62    <none>         9402/TCP                                       7d9h
default       service/cert-manager-webhook                                 ClusterIP      10.152.183.179   <none>         443/TCP                                        7d9h
kube-system   service/prometheus-stack-kube-prom-kube-scheduler            ClusterIP      None             <none>         10259/TCP                                      2d20h
kube-system   service/prometheus-stack-kube-prom-kube-etcd                 ClusterIP      None             <none>         2381/TCP                                       2d20h
kube-system   service/prometheus-stack-kube-prom-kube-controller-manager   ClusterIP      None             <none>         10257/TCP                                      2d20h
kube-system   service/prometheus-stack-kube-prom-kube-proxy                ClusterIP      None             <none>         10249/TCP                                      2d20h

... # more services

NAMESPACE     NAME                         DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
kube-system   daemonset.apps/calico-node   1         1         1       1            1           kubernetes.io/os=linux   2m3s

NAMESPACE     NAME                                      READY   UP-TO-DATE   AVAILABLE   AGE
kube-system   deployment.apps/calico-kube-controllers   1/1     1            1           2m3s

NAMESPACE     NAME                                                DESIRED   CURRENT   READY   AGE
kube-system   replicaset.apps/calico-kube-controllers-d8b9b6478   1         1         1       60s


```
Hi @m4rc3l-h3,

Looks like we in fact do not clean up any of the services in the cluster https://github.com/ubuntu/microk8s/blob/master/scripts/wrappers/reset.py#L121

Looks like a simple fix.
Update:
Also identified that `ValidatingWebhookConfiguration` was not removed from a previous MetalLB deployment, which caused a `webhook not found issue` as described and solved in [https://github.com/metallb/metallb/issues/1823](https://github.com/metallb/metallb/issues/1823).


Hi @neoaggelos 

Thanks for your feedback, much appreciated.
Might that also be true for `secrets` and `configMaps` or are they of different nature?

``` bash
admin@node1:~$ sudo microk8s reset
[sudo] password for admin: 
Disabling all addons
Disabling addon : core/cert-manager
Disabling addon : core/dashboard
Disabling addon : core/dns
Disabling addon : core/gpu
Disabling addon : core/helm
Disabling addon : core/helm3
Disabling addon : core/host-access
Disabling addon : core/hostpath-storage
Disabling addon : core/ingress
Disabling addon : core/kube-ovn
Disabling addon : core/mayastor
Disabling addon : core/metallb
Disabling addon : core/metrics-server
Disabling addon : core/observability
Disabling addon : core/prometheus
Disabling addon : core/rbac
Disabling addon : core/registry
Disabling addon : core/storage
All addons are disabled.
Deleting the CNI
Cleaning resources in namespace kube-system
Cleaning resources in namespace kube-public
Cleaning resources in namespace kube-node-lease
Cleaning resources in namespace default
Cleaning resources in namespace tigera-operator
Removing CRDs
Removing PriorityClasses
Removing StorageClasses
Removing namespace/tigera-operator
Restarting cluster
Stopped.
Setting up the CNI
admin@node1:~$ kubectl get configmaps
NAME                              DATA   AGE
kube-root-ca.crt                  1      10d
cert-manager-webhook              0      8d
configmap-ca-certificates-vault   1      47h
release-name-vault-config         1      47h
configmap-vault-agent-prom        1      47h

admin@node1:~$ kubectl get secrets
NAME                                   TYPE                 DATA   AGE
local-ca-root-cert                     Opaque               2      10d
sh.helm.release.v1.reloader.v1         helm.sh/release.v1   1      10d
cert-manager-webhook-ca                Opaque               3      8d
sh.helm.release.v1.cert-manager.v1     helm.sh/release.v1   1      8d
prometheus-stack-kube-prom-admission   Opaque               6      8d
prometheus-stack-server-tls            kubernetes.io/tls    3      3d15h
vault-server-tls                       kubernetes.io/tls    3      47h
vault-client-tls                       kubernetes.io/tls    3      47h
cert-manager-vault-approle             Opaque               1      47h
example-com-tls                        kubernetes.io/tls    3      6d3h

```
I did some digging and I think I found the issue in line [235](https://github.com/canonical/microk8s/blob/master/scripts/wrappers/reset.py#L235) of the `reset.py` script. In its current version, the `cmd` does not take the `rs` into account, i.e., if I execute the command in the command line interface, it fails with `error: at least one resource must be specified to use a selector`, which might not be catched by `subprocess` call.

``` python
def remove_extra_resources(ns_name):
    # Remove all resource types except the standard k8s apiservices themselves
    cmd = [KUBECTL, "api-resources", "-o", "name", "--verbs=delete", "--namespaced=true"]
    res = run_silently(cmd)
    if not res:
        return
    extra_resources = res.split()
    for rs in extra_resources:
        if rs.startswith("apiservices"):
            continue
        cmd = [KUBECTL, "delete", "--all", "-n", ns_name, "--timeout=60s"]
        subprocess.run(cmd, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)

```

I would change the respective line to
``` python
cmd = [KUBECTL, "delete", rs, "--all", "-n", ns_name, "--timeout=60s"]
```

As I am new to this, would you please advise on what to do next? Would that be a good starting point to contribute to the project? Would be glad to do so.
Hi @m4rc3l-h3, sorry for the late reply. Indeed, nice catch, this should be a good starting point.

Feel free to create a PR with a fix. See the [CONTRIBUTING](https://github.com/ubuntu/microk8s/blob/master/CONTRIBUTING.md) for details, and feel free to ask here if something is not clear. Thanks a lot!
Hi @neoaggelos, no problem and thanks for the reply and the offering to ask if something is unclear. I will keep you updated here. Thank you for your support! 

Resolving patch:
diff --git a/scripts/wrappers/reset.py b/scripts/wrappers/reset.py
--- a/scripts/wrappers/reset.py
+++ b/scripts/wrappers/reset.py
@@ -232,7 +232,7 @@ def remove_extra_resources(ns_name):
     for rs in extra_resources:
         if rs.startswith("apiservices"):
             continue
-        cmd = [KUBECTL, "delete", "--all", "-n", ns_name, "--timeout=60s"]
+        cmd = [KUBECTL, "delete", "--all", rs, "-n", ns_name, "--timeout=3s"]
         subprocess.run(cmd, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
 
 


Test patch: