Project: Inspektor Gadget (Observability and Analysis / Observability)
File: ig.md

title: ig weight: 80 description: > Description of the ig tool. Inspektor Gadget relies on the Kubernetes API server to work. However, there are where it is necessary, or preferred, to trace containers without passing through Kubernetes. In such scenarios, you can use the `ig` tool, as it allows you to collect insights from the nodes to debug your Kubernetes containers without relying on Kubernetes itself, but on the container runtimes. It is important to remark that `ig` can also be used to trace containers that were not created via Kubernetes. Some characteristics of `ig`: It uses eBPF as its underlying core technology. Enriches the collected data with the Kubernetes metadata. Easy to install as it is a single binary (statically linked). The architecture of `ig` is described in the main document. In a Kubernetes environment, when the Kubernetes API server is not working properly, we cannot deploy Inspektor Gadget. Therefore, we still need a way to debug the containers of the cluster. In some cases, you might have root SSH access to the Kubernetes nodes of a cluster, but not to the `kubeconfig`. If you don't want to install `kubectl-gadget` on your machine, you can run `ig` in a Kubernetes pod and read the output directly. If you are implementing an application that needs to get insights from the Kubernetes node, you could include the `ig` binary in your container image, and your app simply execs it. In such a case, it is suggested to use the JSON output format to ease the parsing. Outside a Kubernetes environment, for observing and debugging standalone containers. The instruction to install `ig` are available in the main guide. Currently, `ig` can trace containers managed by Docker regardless of whether they were created via Kubernetes or not. In case of containerd, we are using containerd API directly and containerd namespace (default `k8s.io`) can be configured using `--containerd-namespace` flag. It uses the CRI to trace containers managed by CRI-O. Similarly, it uses the to trace podman containers. By default, `ig` will try to communicate with all the supported container runtimes (docker, containerd, CRI-O, podman): ```bash $ docker run -d --name myContainer nginx:1.21 95b814bb82b9e30dd935b03d04a7b00b6978ce018a6f55d6a9c7a824b31ec6b5 $ sudo ig list-containers RUNTIME.RUNTIMENAME RUNTIME.CONTAINERID RUNTIME.CONTAINERNAME containerd c7dfa4c92fec235626157417bf45745969006bd3bfd2607e87fdd0a176547603 konnectivity-agent containerd cd8ce885c115adbc87da5243630b90935e5bf1c2af96b00154ec475fd9b393b0 nsenter containerd b436a9886ee6e59ac7d38d1b76f8a306e2efeb3f1b6679ea1a58028edb198db3 azure-ip-masq-agent containerd 642fc58b15fbaf578340f4bd656b427db51be63a94a7b6eb663388486e73d855 azuredisk containerd 466a9d7e7b2087966621eacd792cc492f48f08f741f9dc82d88ef62a9d7d3e0f liveness-probe containerd f79db0f2ea6518869c89e1d0a0892221047b23e04e4dab59dc7e42d6808e2530 azurefile containerd 85d74aeb6d29aaa38b282f3e51202bb648f7ba16a681d0d39dda684e724bb8a3 node-driver-registrar containerd a126df15fba5713f57f1abad9c484cb75569e9f48f1169bd9710f63bb8af0e46 kube-proxy containerd 428c933882f1e4459c397da20bd89bbe7df7d437880a254472879d33b125b4da node-driver-registrar containerd cd75a08ea2e69756cd7b1de5935c49f5ba08ba7495b0589567dcd9493193d712 cloud-node-manager containerd d4bdf83ba71c7b22ee339ae5bb6fa7359f8a6bc7cd2f35ccd5681c728869cd39 liveness-probe docker b72558e589cb95e835c4840de19f0306d4081091c34045246d62b6efed3549f4 myContainer ``` To check which paths `ig` is using, you can use the `--help` flag: ```bash $ sudo ig list-containers --help List all containers Usage: ig list-containers [flags] Flags: ... --containerd-namespace string Namespace used by containerd (default "k8s.io") --containerd-socketpath string containerd CRI Unix socket path (default "/run/containerd/containerd.sock") --crio-socketpath string CRI-O CRI Unix socket path (default "/run/crio/crio.sock") --docker-socketpath string Docker Engine API Unix socket path (default "/run/docker.sock") --podman-socketpath string Podman Unix socket path (default "/run/podman/podman.sock") ... -r, --runtimes string Container runtimes to be used separated by comma. Supported values are: docker, containerd, cri-o, podman (default "docker,containerd,cri-o,podman") -w, --watch After listing the containers, watch for new containers

``` If needed, we can also specify the runtimes to be used and their UNIX socket path: ```bash $ sudo ig list-containers --runtimes docker --docker-socketpath /some/path/docker.sock RUNTIME.RUNTIMENAME RUNTIME.CONTAINERID RUNTIME.CONTAINERNAME docker b72558e589cb95e835c4840de19f0306d4081091c34045246d62b6efed3549f4 myContainer ``` Notice that most of the commands support the following features even if, for simplicity, they are not demonstrated in each command guide: JSON format and `columns` output mode are supported through the `--output` flag. It is possible to filter events by container name using the `--containername` flag. It is possible to trace events from all the running processes, even though they were not generated from containers, using the `--host` flag. For instance, for the `list-containers` command: ```bash $ sudo ig list-containers -o json --containername kube-proxy [ { "runtime": { "runtimeName": "containerd", "containerId": "a126df15fba5713f57f1abad9c484cb75569e9f48f1169bd9710f63bb8af0e46", "containerName": "kube-proxy" }, "k8s": { "namespace": "kube-system", "podName": "kube-proxy-tcbn4", "containerName": "kube-proxy", "podUID": "87c52d60-fefd-45a9-a420-895256fc03b5" }, "pid": 454674, "mntns": 4026532232, "netns": 4026531840, "hostNetwork": true, "cgroupPath": "/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87c52d60fefd45a9a420895256fc03b5.slice/cri-containerd-a126df15fba5713f57f1abad9c484cb75569e9f48f1169bd9710f63bb8af0e46.scope", "cgroupID": 41286, "cgroupV2": "/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87c52d60fefd45a9a420895256fc03b5.slice/cri-containerd-a126df15fba5713f57f1abad9c484cb75569e9f48f1169bd9710f63bb8af0e46.scope" } ] ``` For example, with `--host`, you can get the following output: ```bash $ sudo ig trace exec --host RUNTIME.CONTAINERNAME PID PPID COMM RET ARGS $ cat /dev/null $ docker run --name test-host --rm -t debian sh -c 'ls > /dev/null' RUNTIME.CONTAINERNAME PID PPID COMM RET ARGS 3326022 308789 cat 0 /usr/bin/cat /dev/null test-host 3326093 3326070 sh 0 /usr/bin/sh -c ls > /dev/null test-host 3326123 3326093 ls 0 /usr/bin/ls ``` Events generated from containers have their container field set, while events which are generated from the host do not. The "kubectl debug node" command is documented in . Examples of commands: ```bash $ kubectl debug node/minikube-docker -ti --image=ghcr.io/inspektor-gadget/ig -- ig --auto-sd-unit-restart trace exec Creating debugging pod node-debugger-minikube-docker-c2wfw with container debugger on node minikube-docker. If you don't see a command prompt, try pressing enter. RUNTIME.CONTAINERNAME PID PPID COMM RET ARGS k8sshellshelldefaultb4ebb 3186934 3186270 cat 0 /bin/cat file ``` ```bash $ kubectl debug node/minikube-docker -ti --image=ghcr.io/inspektor-gadget/ig -- ig --auto-sd-unit-restart list-containers -o json ``` As of today, the `kubectl debug` command does not have a way to give enough privileges to the debugging pod to be able to use `ig`. This might change in the future: the Kubernetes Enhancement Proposal 1441 () suggests to implement Debugging Profiles (`--profile=`) to be able to give the necessary privileges. kubectl v1.27 implements some of those profiles but not yet the "sysadmin" profile, so it is not possible to use `--profile=` yet. Meanwhile, `ig` provides the `--auto-sd-unit-restart` flag. The flag is `false` by default. When it is set to `true`, `ig` will detect if it does not have enough privileges and it can transparently re-execute itself in a privileged systemd unit if necessary. This is possible because the "kubectl debug node" gives access to the systemd socket (`/run/systemd/private`) via the /host volume. `ig` can also be run as a daemon. You can then use `gadgetctl` to connect to it as an unprivileged user. We will be using a new group called `ig` to allow controlling `ig` in this example. ```bash $ sudo addgroup ig $ sudo adduser $USER ig ``` Logout and login to make sure the new group membership is applied to your session. This assumes you installed `ig` in `/usr/local/bin` and are using

Create a new service file at `/etc/systemd/system/ig.service` with the following content: ```ini [Unit] Description=Inspektor Gadget [Service] User=root Restart=on-failure RestartSec=30 WorkingDirectory=/usr/local/bin ExecStart=/usr/local/bin/ig daemon --group ig [Install] WantedBy=multi-user.target ``` If you want to use another group than `ig`, make sure to adjust the `--group` parameter on the "ExecStart" line. Enable and start the new service: ```bash $ sudo systemctl enable ig.service $ sudo systemctl start ig.service ``` If all went well, you can now run `gadgetctl` with your favorite gadgets! ```bash $ gadgetctl trace open RUNTIME.CONTAINERNAME PID COMM FD ERR PATH minikube-docker 3293 cri-dockerd 11 0 /etc/cni/net.d minikube-docker 3293 cri-dockerd 11 0 /etc/cni/net.d/1-k8s.conflist minikube-docker 2180730 bridge 3 0 /etc/ld.so.cache ... ``` This is not yet a recommended way of working with ig, as the connection is not secure. Please only use it on otherwise secured and/or trusted networks. We will be adding secure connections later on. Modify the `ig.service` file to something like this: ``` ... ExecStart=/usr/local/bin/ig daemon -H tcp://127.0.0.1:9999 ... ``` Restart the service: ```bash $ sudo systemctl restart ig.service ``` Use `gadgetctl` like this: ```bash $ gadgetctl trace open --remote-address tcp://127.0.0.1:9999 ``` In case anything is not working, you can look at the logs: ```bash $ journalctl -u ig.service ``` To increase the verbosity of `ig`, you can edit the service file and add the verbose flag (-v) like so: ```ini ... ExecStart=/usr/local/bin/ig daemon -v --group ig ... ``` The flag can also be used with `gadgetctl` like so: ```bash $ gadgetctl trace open -v ``` Example of command: ```bash $ docker run -ti --rm \ --privileged \ -v /run:/run \ -v /:/host \ --pid=host \ ghcr.io/inspektor-gadget/ig \ trace exec RUNTIME.CONTAINERNAME PID PPID COMM RET ARGS heuristic_yonath 3329233 3329211 ls 0 /bin/ls ``` List of flags: `--privileged` gives all capabilities such as `CAPSYSADMIN`. It is required to run eBPF programs. `-v /run:/run` gives access to the container runtimes sockets (docker, containerd, CRI-O). `-v /:/host` gives access to the host filesystem. This is used to access the host processes via /host/proc, and access container runtime hooks (rootfs and config.json). `--pid=host` runs in the host PID namespace. Optional on Linux. This is necessary on Docker Desktop on Windows because /host/proc does not give access to the host processes. In order to run `ig` in a Kubernetes pod use . ```bash $ kubectl apply -f docs/examples/pod-ig.yaml $ kubectl logs ig RUNTIME.CONTAINERNAME RUNTIME.CONTAIN PID PPID COMM RET ARGS kube-proxy k8s.gcr.io/kube 3985376 3024961 ip6tables 0 /usr/sbin/ip6tables -w 5 -W 100000 -S K ``` In order to add `ig` in your own container image, you can take example on the following Dockerfile: ```Dockerfile ARG BASE_IMAGE=ghcr.io/inspektor-gadget/ig:latest FROM ${BASE_IMAGE} as ig FROM alpine:3.17 COPY --from=ig /usr/bin/ig /usr/bin/ig ENV HOST_ROOT=/host ``` The `ghcr.io/inspektor-gadget/ig` image supports amd64 and arm64. Your own image can also support both architectures if you use the appropriate `--platforms` flag of `docker buildx build` (see ). You can then run your image locally or in a Kubernetes pod. Here is an example using a Kubernetes DaemonSet: : ```bash $ kubectl apply -f docs/examples/ds-ig.yaml $ kubectl exec -ti $(kubectl get pod -o name -l name=example-ig | head -1) -- sh / # ig trace exec ```