Repository: sosreport/sos

Issue:
[RFE] collect kubernetes informations for kubeadm installs
Currently `RedHatKubernetes` is specific to OpenShift (only older versions ?)
I would like to have it run for classic kubeadm installs on top of RHEL family OS (also Ubuntu as a bonus),
but I don't have access to OpenShift to move the OpenShift bits in openshift.py and to make sure either openshift plugin runs or kubernetes plugin runs, but not both (a bit of a waste to collect kubelet logs twice)

To collect kubeadm installs we would need
```
files = (
    "/etc/kubernetes/admin.conf",
)
services = (
    "kubelet",
)
```

Would something like
```
def check_enabled(self):
    return not self.path_exists('/etc/origin/') and super().check_enabled()
```
be ok ?



Discussion:
> Currently `RedHatKubernetes` is specific to OpenShift (only older versions ?) 

It isn't - that's what the `openshift` plugin is for. The `kubernetes` plugin is for community k8s, regardless of the distro. I believe, however, that the RH bits in that plugin were last updated back in the days of RHEL Atomic Host, when RH still supported "base" kubernetes installs, and we needed to account for both setups. When the focus from RH turned entirely to Openshift (and RHEL Atomic Host was discontinued), the `openshift` plugin was updated to allow the sos project to serve both use cases without having to juggle a bunch of conflicting logic and configuration setups in a single plugin. The lingering OCP bits in the kubernetes plugin would only be relevant for _extremely_ old versions of OCP, but wouldn't affect installs of community k8s.

>I would like to have it run for classic kubeadm installs on top of RHEL family OS (also Ubuntu as a bonus), but I don't have access to OpenShift to move the OpenShift bits in openshift.py and to make sure either openshift plugin runs or kubernetes plugin runs, but not both (a bit of a waste to collect kubelet logs twice)
> 

The plugins shouldn't be both activating at once today anyway. They have different enablement triggers, e.g.:

```python
class RedHatKubernetes(Kubernetes, RedHatPlugin):

    # OpenShift Container Platform uses the atomic-openshift-master package
    # to provide kubernetes
    packages = ('kubernetes', 'kubernetes-master', 'atomic-openshift-master')

    files = (
        '/etc/origin/master/admin.kubeconfig',
        '/etc/origin/node/pods/master-config.yaml',
    )
```

vs.

```python
class Openshift(Plugin, RedHatPlugin):
[...]
    plugin_name = "openshift"
    plugin_timeout = 900
    profiles = ('openshift',)
    packages = ('openshift-hyperkube',)
```

The note in the `kubernetes` plugin is from a long-ago version, but at present , AFAIK, OCP doesn't install the `kuberentes` package at all. If this is not the case, then we simply need to override `check_enabled()` in the kubernetes plugin to avoid enabling is the `openshift-hyperkube` plugin is installed.

>> Currently RedHatKubernetes is specific to OpenShift (only older versions ?)

> It isn't

- Fedora has a `kubernetes` packages, but the upstream repo pkgs.k8s.io doesn't, and some projects like kubespray just install the binaries + unit files without using packages at all.
- `check_is_master()` look at `self.files`, which currently is openshift specific for RedHatKubernetes.
- finally, RedHatKubernetes unconditionally adds openshift specific ressources

I'll put together a draft PR shortly

Resolving patch:
diff --git a/sos/report/plugins/kubernetes.py b/sos/report/plugins/kubernetes.py
--- a/sos/report/plugins/kubernetes.py
+++ b/sos/report/plugins/kubernetes.py
@@ -12,8 +12,28 @@
 from fnmatch import translate
 import re
 import json
+import os
 from sos.report.plugins import (Plugin, RedHatPlugin, DebianPlugin,
                                 UbuntuPlugin, PluginOpt)
+from sos.utilities import is_executable
+
+
+KUBE_PACKAGES = (
+    'kubelet',
+    'kubernetes',
+)
+
+KUBE_SVCS = (
+    'kubelet',
+    'kube-apiserver',
+    'kube-proxy',
+    'kube-scheduler',
+    'kube-controller-manager',
+)
+
+KUBECONFIGS = (
+    '/etc/kubernetes/admin.conf',
+)
 
 
 class Kubernetes(Plugin):
@@ -45,7 +65,9 @@ class Kubernetes(Plugin):
         'jobs',
         'cronjobs',
         'clusterroles',
-        'clusterrolebindings'
+        'clusterrolebindings',
+        'limitranges',
+        'resourcequotas',
     ]
 
     # these are not namespaced, must pull separately.
@@ -53,7 +75,7 @@ class Kubernetes(Plugin):
         'sc',
         'pv',
         'roles',
-        'rolebindings'
+        'rolebindings',
     ]
 
     option_list = [
@@ -69,6 +91,14 @@ class Kubernetes(Plugin):
 
     kube_cmd = "kubectl"
 
+    def set_kubeconfig(self):
+        if os.environ.get('KUBECONFIG'):
+            return
+        for _kconf in self.files:
+            if self.path_exists(_kconf):
+                self.kube_cmd += f" --kubeconfig={_kconf}"
+                break
+
     def check_is_master(self):
         """ Check if this is the master node """
         return any(self.path_exists(f) for f in self.files)
@@ -80,25 +110,9 @@ def setup(self):
             'KUBECONFIG',
             'KUBERNETES_HTTP_PROXY',
             'KUBERNETES_HTTPS_PROXY',
-            'KUBERNETES_NO_PROXY'
+            'KUBERNETES_NO_PROXY',
         ])
 
-        svcs = [
-            'kubelet',
-            'kube-apiserver',
-            'kube-proxy',
-            'kube-scheduler',
-            'kube-controller-manager',
-            'snap.kubelet.daemon',
-            'snap.kube-apiserver.daemon',
-            'snap.kube-proxy.daemon',
-            'snap.kube-scheduler.daemon',
-            'snap.kube-controller-manager.daemon'
-        ]
-
-        for svc in svcs:
-            self.add_journal(units=svc)
-
         # We can only grab kubectl output from the master
         if not self.check_is_master():
             return
@@ -250,69 +264,81 @@ def postproc(self):
         # output that is not hit by the previous iteration.
         self.do_cmd_private_sub(self.kube_cmd)
 
+        pathexp = fr'^({"|".join(self.config_files)})'
+        self.do_file_private_sub(pathexp)
+
+        # clear base64 encoded PEM from kubeconfigs files
+        regexp = r'LS0tLS1CRUdJ[A-Za-z0-9+/=]+'
+        subst = '***** SCRUBBED BASE64 PEM *****'
+        pathexp = fr'^({"|".join(list(self.files)+self.config_files)})'
+        self.do_path_regex_sub(pathexp, regexp, subst)
+
 
 class RedHatKubernetes(Kubernetes, RedHatPlugin):
 
-    # OpenShift Container Platform uses the atomic-openshift-master package
-    # to provide kubernetes
-    packages = ('kubernetes', 'kubernetes-master', 'atomic-openshift-master')
+    packages = KUBE_PACKAGES + (
+        'kubernetes-master',
+        'atomic-openshift-master',
+    )
 
-    files = (
+    files = KUBECONFIGS + (
         '/etc/origin/master/admin.kubeconfig',
         '/etc/origin/node/pods/master-config.yaml',
     )
 
-    kube_cmd = "kubectl"
+    services = KUBE_SVCS
+
+    def check_enabled(self):
+        # do not run at the same time as the openshift plugin
+        if self.is_installed("openshift-hyperkube"):
+            return False
+        return super().check_enabled()
 
     def setup(self):
-        # Rather than loading the config file, use the OCP command directly
-        # that wraps kubectl, so we don't have to manually account for any
-        # other changes the `oc` binary may implement
-        if self.path_exists('/etc/origin/master/admin.kubeconfig'):
+        self.set_kubeconfig()
+
+        # if present, use `oc` command and add some OCP specific ressources
+        if is_executable('oc'):
             self.kube_cmd = 'oc'
-        self.resources.extend([
-            'limitranges',
-            'policies',
-            'resourcequotas',
-            'routes'
-        ])
-        self.global_resources.extend([
-            'projects',
-            'pvs'
-        ])
+            self.resources.extend([
+                'policies',
+                'routes'
+            ])
+            self.global_resources.extend([
+                'projects',
+                'pvs'
+            ])
         super().setup()
 
 
 class UbuntuKubernetes(Kubernetes, UbuntuPlugin, DebianPlugin):
 
-    packages = ('kubernetes',)
+    packages = KUBE_PACKAGES
 
-    files = (
+    files = KUBECONFIGS + (
         '/root/cdk/cdk_addons_kubectl_config',
-        '/etc/kubernetes/admin.conf',
         '/var/snap/microk8s/current/credentials/client.config',
     )
 
-    services = (
+    services = KUBE_SVCS + (
+        'snap.kubelet.daemon',
+        'snap.kube-apiserver.daemon',
+        'snap.kube-proxy.daemon',
+        'snap.kube-scheduler.daemon',
+        'snap.kube-controller-manager.daemon',
         # CDK
         'cdk.master.auth-webhook',
     )
 
     def setup(self):
-        for _kconf in self.files:
-            if self.path_exists(_kconf):
-                self.kube_cmd += f" --kubeconfig={_kconf}"
-                break
-
-        for svc in self.services:
-            self.add_journal(units=svc)
+        self.set_kubeconfig()
 
         if self.is_installed('microk8s'):
             self.kube_cmd = 'microk8s kubectl'
 
         self.config_files.extend([
             '/root/cdk/kubelet/config.yaml',
-            '/root/cdk/audit/audit-policy.yaml'
+            '/root/cdk/audit/audit-policy.yaml',
         ])
         super().setup()
 


Test patch: