OpenShift - oc command Tips - debug node

OpenShift の様々な操作で使用する `oc` コマンドの `debug node` をご紹介します。

### debug node
`debug node` により Node 上で処理を実行することが出来ます。

https://github.com/kubernetes/enhancements/blob/master/keps/sig-cli/1441-kubectl-debug/README.md

https://kubernetes.io/docs/tasks/debug/debug-cluster/kubectl-node-debug/

https://kubernetes.io/docs/tasks/debug/debug-cluster/#example-debugging-a-down-unreachable-node

https://kubernetes.io/docs/reference/kubectl/generated/kubectl_debug/

```shell
$ oc debug --help
Launch a command shell to debug a running application.

 When debugging images and setup problems, it's useful to get an exact copy of a running pod configuration and
troubleshoot with a shell. Since a pod that is failing may not be started and not accessible to 'rsh' or 'exec', the
'debug' command makes it easy to create a carbon copy of that setup.

 The default mode is to start a shell inside of the first container of the referenced pod. The started pod will be a
copy of your source pod, with labels stripped, the command changed to '/bin/sh' for Linux containers or 'cmd.exe' for
Windows containers, and readiness and liveness checks disabled. If you just want to run a command, add '--' and a
command to run. Passing a command will not create a TTY or send STDIN by default. Other flags are supported for altering
the container or pod in common ways.

 A common problem running containers is a security policy that prohibits you from running as a root user on the cluster.
You can use this command to test running a pod as non-root (with --as-user) or to run a non-root pod as root (with
--as-root).

 You may invoke other types of objects besides pods - any controller resource that creates a pod (like a deployment,
build, or job), objects that can host pods (like nodes), or resources that can be used to create pods (such as image
stream tags), or simply pass '--image=IMAGE' to start a simple shell session in an image with a shell program

 The debug pod is deleted when the remote command completes or the user interrupts the shell.

Examples:
  # Start a shell session into a pod using the OpenShift tools image
  oc debug

  # Debug a currently running deployment by creating a new pod
  oc debug deploy/test

  # Debug a node as an administrator
  oc debug node/master-1

  # Launch a shell in a pod using the provided image stream tag
  oc debug istag/mysql:latest -n openshift

  # Test running a job as a non-root user
  oc debug job/test --as-user=1000000

  # Debug a specific failing container by running the env command in the 'second' container
  oc debug daemonset/test -c second -- /bin/env

  # See the pod that would be created to debug
  oc debug mypod-9xbc -o yaml

  # Debug a resource but launch the debug pod in another namespace
  # Note: Not all resources can be debugged using --to-namespace without modification. For example,
  # volumes and service accounts are namespace-dependent. Add '-o yaml' to output the debug pod definition
  # to disk.  If necessary, edit the definition then run 'oc debug -f -' or run without --to-namespace
  oc debug mypod-9xbc --to-namespace testns

Options:
    --allow-missing-template-keys=true:
        If true, ignore any errors in templates when a field or map key is missing in the template. Only applies to
        golang and jsonpath output formats.

    --as-root=false:
        If true, try to run the container as the root user

    --as-user=-1:
        Try to run the container as a specific user UID (note: admins may limit your ability to use this flag)

    -c, --container='':
        Container name; defaults to first container

    --dry-run='none':
        Must be "none", "server", or "client". If client strategy, only print the object that would be sent, without
        sending it. If server strategy, submit server-side request without persisting the resource.

    -f, --filename=[]:
        Filename, directory, or URL to files to read a template

    --image='':
        Override the image used by the targeted container.

    --image-stream='':
        Specify an image stream (namespace/name:tag) containing a debug image to run.

    --keep-annotations=false:
        If true, keep the original pod annotations

    --keep-init-containers=true:
        Run the init containers for the pod. Defaults to true.

    --keep-labels=false:
        If true, keep the original pod labels

    --keep-liveness=false:
        If true, keep the original pod liveness probes

    --keep-readiness=false:
        If true, keep the original pod readiness probes

    --keep-startup=false:
        If true, keep the original startup probes

    -k, --kustomize='':
        Process the kustomization directory. This flag can't be used together with -f or -R.

    -I, --no-stdin=false:
        Bypasses passing STDIN to the container, defaults to true if no command specified

    -T, --no-tty=false:
        Disable pseudo-terminal allocation

    --node-name='':
        Set a specific node to run on - by default the pod will run on any valid node

    --one-container=false:
        If true, run only the selected container, remove all others

    -o, --output='':
        Output format. One of: (json, yaml, name, go-template, go-template-file, template, templatefile, jsonpath,
        jsonpath-as-json, jsonpath-file).

    --preserve-pod=false:
        If true, the pod will not be deleted after the debug command exits.

    -q, --quiet=false:
        No informational messages will be printed.

    -R, --recursive=false:
        Process the directory used in -f, --filename recursively. Useful when you want to manage related manifests
        organized within the same directory.

    --show-labels=false:
        When printing, show all labels as the last column (default hide labels column)

    --show-managed-fields=false:
        If true, keep the managedFields when printing objects in JSON or YAML format.

    --template='':
        Template string or path to template file to use when -o=go-template, -o=go-template-file. The template format
        is golang templates [http://golang.org/pkg/text/template/#pkg-overview].

    --to-namespace='':
        Override the namespace to create the pod into (instead of using --namespace).

    -t, --tty=false:
        Force a pseudo-terminal to be allocated

Usage:
  oc debug RESOURCE/NAME [ENV1=VAL1 ...] [-c CONTAINER] [flags] [-- COMMAND] [options]

Use "oc options" for a list of global command-line options (applies to all commands).
```

以下の要領で `debug node` を使用します。

```shell
$ oc get node
NAME           STATUS                     ROLES           AGE    VERSION
my-worker-01   Ready,SchedulingDisabled   master,worker   51d    v1.27.16+03a907c
my-worker-02   Ready                      master,worker   133d   v1.27.15+6147456
my-worker-z    Ready                      master,worker   133d   v1.27.15+6147456

$ oc debug node/my-worker-z
Starting pod/10244645-debug-tgllz ...
To use host binaries, run `chroot /host`
Pod IP: my-worker-z
If you don't see a command prompt, try pressing enter.
sh-4.4# pwd
/
sh-4.4# whoami
root
sh-4.4# chroot /host
sh-4.4# ps
   PID TTY          TIME CMD
105602 ?        00:00:00 sh
106081 ?        00:00:00 sh
106451 ?        00:00:00 ps
sh-4.4# exit
exit
sh-4.4# exit
exit

Removing debug pod ...
```

`debug node` を実行すると、Current Project に以下のような Pod が実行され、この Pod を介して Node にアクセスします。`hostPath` で、指定した Node (= Pod が実行されている Node) の `/ (root)` を Pod 内 container-00 Container の `/host` にマウントしていることが分かります。
```shell
$ oc project -q
default

$ oc get pod -o wide
NAME                   READY   STATUS    RESTARTS   AGE   IP            NODE          NOMINATED NODE   READINESS GATES
10244645-debug-lvn2k   1/1     Running   0          34s   11.222.33.4   my-worker-z   <none>           <none>

$ oc -o yaml get pod 10244645-debug-lvn2k
apiVersion: v1
kind: Pod
metadata:
  annotations:
    debug.openshift.io/source-container: container-00
    debug.openshift.io/source-resource: /v1, Resource=nodes/my-worker-z
  name: 10244645-debug-lvn2k
  namespace: default
spec:
  containers:
  - command:
    - /bin/sh
    env:
    - name: TMOUT
      value: "900"
    image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ee9b4cbef856130e029b9c2a4b5b16d805d347579271743ac823f79dd40c02ef
    imagePullPolicy: IfNotPresent
    name: container-00
    resources: {}
    securityContext:
      privileged: true
      runAsUser: 0
    stdin: true
    stdinOnce: true
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    tty: true
    volumeMounts:
    - mountPath: /host
      name: host
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-jmnlg
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  hostIPC: true
  hostNetwork: true
  hostPID: true
  imagePullSecrets:
  - name: default-dockercfg-pvql5
  - name: all-icr-io
  nodeName: my-worker-z
  preemptionPolicy: PreemptLowerPriority
  priority: 1000000000
  priorityClassName: openshift-user-critical
  restartPolicy: Never
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - hostPath:
      path: /
      type: Directory
    name: host
  - name: kube-api-access-jmnlg
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
      - configMap:
          items:
          - key: service-ca.crt
            path: service-ca.crt
          name: openshift-service-ca.crt
status:
～省略～
```

なお、`debu node` の実行先 User は root である必要があります。
```shell
$ oc debug --as-user=65534 node/my-worker-z
error: cannot debug my-worker-z: can't debug nodes without running as the root user
$ echo $?
1
```
デフォルトの `sh` ではなく `bash` を指定することもできます。
```shell
$ oc debug -t node/my-worker-z -- bash
Starting pod/10244645-debug-bjjqd ...
To use host binaries, run `chroot /host`
Pod IP: my-worker-z
If you don't see a command prompt, try pressing enter.
[root@openshift-1023-node-worker-spacename-9990-cons-1111111 /]# chroot /host bash
[root@openshift-1023-node-worker-spacename-9990-cons-1111111 /]# ps
   PID TTY          TIME CMD
107593 ?        00:00:00 bash
108501 ?        00:00:00 bash
108581 ?        00:00:00 ps
```
特定のコマンドを実行することも可能です。
```shell
$ oc debug -t node/my-worker-z -- cat /etc/hosts
Starting pod/10244645-debug-zn242 ...
To use host binaries, run `chroot /host`
# Kubernetes-managed hosts file (host network).
# Your system has configured 'manage_etc_hosts' as True.
# As a result, if you wish for changes to this file to persist
# then you will need to either
# a.) make changes to the master file in /etc/cloud/templates/hosts.redhat.tmpl
# b.) change or remove the value of 'manage_etc_hosts' in
#     /etc/cloud/cloud.cfg or cloud-config from user-data
#
# The following lines are desirable for IPv4 capable hosts
127.0.0.1 openshift-1023-node-worker-spacename-9990-cons-1111111 openshift-1023-node-worker-spacename-9990-cons-1111111
127.0.0.1 localhost.localdomain localhost
127.0.0.1 localhost4.localdomain4 localhost4
～省略～
```