Chapter 21. Monitoring containers Use Podman commands to manage a Podman environment. With that, you can determine the health of the container, by displaying system and pod information, and monitoring Podman events. 21.1. Using a health check on a container You can use the health check to determine the health or readiness of the process running inside the container. If the health check succeeds, the container is marked as "healthy"; otherwise, it is "unhealthy". You can compare a health check with running the podman exec command and examining the exit code. The zero exit value means that the container is "healthy". Health checks can be set when building an image using the HEALTHCHECK instruction in the Containerfile or when creating the container on the command line. You can display the health-check status of a container using the podman inspect or podman ps commands. A health check consists of six basic components: Command Retries Interval Start-period Timeout Container recovery The description of health check components follows: Command ( --health-cmd option) Podman executes the command inside the target container and waits for the exit code. The other five components are related to the scheduling of the health check and they are optional. Retries ( --health-retries option) Defines the number of consecutive failed health checks that need to occur before the container is marked as "unhealthy". A successful health check resets the retry counter. Interval ( --health-interval option) Describes the time between running the health check command. Note that small intervals cause your system to spend a lot of time running health checks. The large intervals cause struggles with catching time outs. Start-period ( --health-start-period option) Describes the time between when the container starts and when you want to ignore health check failures. Timeout ( --health-timeout option) Describes the period of time the health check must complete before being considered unsuccessful. Note The values of the Retries, Interval, and Start-period components are time durations, for example "30s" or "1h15m". Valid time units are "ns," "us," or "ms", "ms," "s," "m," and "h". Container recovery ( --health-on-failure option) Determines which actions to perform when the status of a container is unhealthy. When the application fails, Podman restarts it automatically to provide robustness. The --health-on-failure option supports four actions: none : Take no action, this is the default action. kill : Kill the container. restart : Restart the container. stop : Stop the container. Note The --health-on-failure option is available in Podman version 4.2 and later. Warning Do not combine the restart action with the --restart option. When running inside of a systemd unit, consider using the kill or stop action instead, to make use of systemd restart policy. Health checks run inside the container. Health checks only make sense if you know what the health state of the service is and can differentiate between a successful and unsuccessful health check. Additional resources podman-healthcheck and podman-run man pages on your system Podman at the edge: Keeping services alive with custom healthcheck actions Monitoring container vitality and availability with Podman 21.2. Performing a health check using the command line You can set a health check when creating the container on the command line. Prerequisites The container-tools module is installed. Procedure Define a health check: The --health-cmd option sets a health check command for the container. The --health-interval=0 option with 0 value indicates that you want to run the health check manually. Check the health status of the hc-container container: Using the podman inspect command: Using the podman ps command: Using the podman healthcheck run command: Additional resources podman-healthcheck and podman-run man pages on your system Podman at the edge: Keeping services alive with custom healthcheck actions Monitoring container vitality and availability with Podman 21.3. Performing a health check using a Containerfile You can set a health check by using the HEALTHCHECK instruction in the Containerfile . Prerequisites The container-tools module is installed. Procedure Create a Containerfile : Note The HEALTHCHECK instruction is supported only for the docker image format. For the oci image format, the instruction is ignored. Build the container and add an image name: Run the container: Check the health status of the hc-container container: Using the podman inspect command: Using the podman ps command: Using the podman healthcheck run command: Additional resources podman-healthcheck and podman-run man pages on your system Podman at the edge: Keeping services alive with custom healthcheck actions Monitoring container vitality and availability with Podman 21.4. Displaying Podman system information The podman system command enables you to manage the Podman systems by displaying system information. Prerequisites The container-tools module is installed. Procedure Display Podman system information: To show Podman disk usage, enter: To show detailed information about space usage, enter: To display information about the host, current storage stats, and build of Podman, enter: To remove all unused containers, images and volume data, enter: The podman system prune command removes all unused containers (both dangling and unreferenced), pods and optionally, volumes from local storage. Use the --all option to delete all unused images. Unused images are dangling images and any image that does not have any containers based on it. Use the --volume option to prune volumes. By default, volumes are not removed to prevent important data from being deleted if there is currently no container using the volume. Additional resources podman-system-df , podman-system-info , and podman-system-prune man pages on your system 21.5. Podman event types You can monitor events that occur in Podman. Several event types exist and each event type reports different statuses. The container event type reports the following statuses: attach checkpoint cleanup commit create exec export import init kill mount pause prune remove restart restore start stop sync unmount unpause The pod event type reports the following statuses: create kill pause remove start stop unpause The image event type reports the following statuses: prune push pull save remove tag untag The system type reports the following statuses: refresh renumber The volume type reports the following statuses: create prune remove Additional resources podman-events man page on your system 21.6. Monitoring Podman events You can monitor and print events that occur in Podman using the podman events command. Each event will include a timestamp, a type, a status, name, if applicable, and image, if applicable. Prerequisites The container-tools module is installed. Procedure Run the myubi container: Display the Podman events: To display all Podman events, enter: The --stream=false option ensures that the podman events command exits when reading the last known event. You can see several events that happened when you enter the podman run command: container create when creating a new container. image pull when pulling an image if the container image is not present in the local storage. container init when initializing the container in the runtime and setting a network. container start when starting the container. container attach when attaching to the terminal of a container. That is because the container runs in the foreground. container died is emitted when the container exits. container remove because the --rm flag was used to remove the container after it exits. You can also use the journalctl command to display Podman events: To show only Podman create events, enter: You can also use the journalctl command to display Podman create events: Additional resources podman-events man page on your system Container Events and Auditing 21.7. Using Podman events for auditing Previously, the events had to be connected to an event to interpret them correctly. For example, the container-create event had to be linked with an image-pull event to know which image had been used. The container-create event also did not include all data, for example, the security settings, volumes, mounts, and so on. Beginning with Podman v4.4, you can gather all relevant information about a container directly from a single event and journald entry. The data is in JSON format, the same as from the podman container inspect command and includes all configuration and security settings of a container. You can configure Podman to attach the container-inspect data for auditing purposes. Prerequisites The container-tools module is installed. Procedure Modify the ~/.config/containers/containers.conf file and add the events_container_create_inspect_data=true option to the [engine] section: For the system-wide configuration, modify the /etc/containers/containers.conf or /usr/share/container/containers.conf file. Create the container: Display the Podman events: Using the podman events command: The --format "{{.ContainerInspectData}}" option displays the inspect data. The jq ".Config.CreateCommand" transforms the JSON data into a more readable format and displays the parameters for the podman create command. Using the journalctl command: The output data for the podman events and journalctl commands are the same. Additional resources podman-events and containers.conf man pages on your system Container Events and Auditing

Commands:
podman run -dt --name=hc-container -p 8080:8080 --health-cmd='curl http://localhost:8080 || exit 1' --health-interval=0 registry.access.redhat.com/ubi8/httpd-24
podman inspect --format='{{json .State.Health.Status}}' hc-container healthy
podman ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES a680c6919fe localhost/hc-container:latest /usr/bin/run-http... 2 minutes ago Up 2 minutes (healthy) hc-container
podman healthcheck run hc-container healthy
cat Containerfile FROM registry.access.redhat.com/ubi8/httpd-24 EXPOSE 8080 HEALTHCHECK CMD curl http://localhost:8080 || exit 1
podman build --format=docker -t hc-container . STEP 1/3: FROM registry.access.redhat.com/ubi8/httpd-24 STEP 2/3: EXPOSE 8080 --> 5aea97430fd STEP 3/3: HEALTHCHECK CMD curl http://localhost:8080 || exit 1 COMMIT health-check Successfully tagged localhost/health-check:latest a680c6919fe6bf1a79219a1b3d6216550d5a8f83570c36d0dadfee1bb74b924e
podman run -dt --name=hc-container localhost/hc-container
podman inspect --format='{{json .State.Health.Status}}' hc-container healthy
podman ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES a680c6919fe localhost/hc-container:latest /usr/bin/run-http... 2 minutes ago Up 2 minutes (healthy) hc-container
podman healthcheck run hc-container healthy
podman system df TYPE TOTAL ACTIVE SIZE RECLAIMABLE Images 3 2 1.085GB 233.4MB (0%) Containers 2 0 28.17kB 28.17kB (100%) Local Volumes 3 0 0B 0B (0%)
podman system df -v Images space usage: REPOSITORY TAG IMAGE ID CREATED SIZE SHARED SIZE UNIQUE SIZE CONTAINERS registry.access.redhat.com/ubi8 latest b1e63aaae5cf 13 days 233.4MB 233.4MB 0B 0 registry.access.redhat.com/ubi8/httpd-24 latest 0d04740850e8 13 days 461.5MB 0B 461.5MB 1 registry.redhat.io/rhel8/podman latest dce10f591a2d 13 days 390.6MB 233.4MB 157.2MB 1 Containers space usage: CONTAINER ID IMAGE COMMAND LOCAL VOLUMES SIZE CREATED STATUS NAMES 311180ab99fb 0d04740850e8 /usr/bin/run-httpd 0 28.17kB 16 hours exited hc1 bedb6c287ed6 dce10f591a2d podman run ubi8 echo hello 0 0B 11 hours configured dazzling_tu Local Volumes space usage: VOLUME NAME LINKS SIZE 76de0efa83a3dae1a388b9e9e67161d28187e093955df185ea228ad0b3e435d0 0 0B 8a1b4658aecc9ff38711a2c7f2da6de192c5b1e753bb7e3b25e9bf3bb7da8b13 0 0B d9cab4f6ccbcf2ac3cd750d2efff9d2b0f29411d430a119210dd242e8be20e26 0 0B
podman system info host: arch: amd64 buildahVersion: 1.22.3 cgroupControllers: [] cgroupManager: cgroupfs cgroupVersion: v1 conmon: package: conmon-2.0.29-1.module+el8.5.0+12381+e822eb26.x86_64 path: /usr/bin/conmon version: 'conmon version 2.0.29, commit: 7d0fa63455025991c2fc641da85922fde889c91b' cpus: 2 distribution: distribution: '"rhel"' version: "8.5" eventLogger: file hostname: localhost.localdomain idMappings: gidmap: - container_id: 0 host_id: 1000 size: 1 - container_id: 1 host_id: 100000 size: 65536 uidmap: - container_id: 0 host_id: 1000 size: 1 - container_id: 1 host_id: 100000 size: 65536 kernel: 4.18.0-323.el8.x86_64 linkmode: dynamic memFree: 352288768 memTotal: 2819129344 ociRuntime: name: runc package: runc-1.0.2-1.module+el8.5.0+12381+e822eb26.x86_64 path: /usr/bin/runc version: |- runc version 1.0.2 spec: 1.0.2-dev go: go1.16.7 libseccomp: 2.5.1 os: linux remoteSocket: path: /run/user/1000/podman/podman.sock security: apparmorEnabled: false capabilities: CAP_NET_RAW,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT rootless: true seccompEnabled: true seccompProfilePath: /usr/share/containers/seccomp.json selinuxEnabled: true serviceIsRemote: false slirp4netns: executable: /usr/bin/slirp4netns package: slirp4netns-1.1.8-1.module+el8.5.0+12381+e822eb26.x86_64 version: |- slirp4netns version 1.1.8 commit: d361001f495417b880f20329121e3aa431a8f90f libslirp: 4.4.0 SLIRP_CONFIG_VERSION_MAX: 3 libseccomp: 2.5.1 swapFree: 3113668608 swapTotal: 3124752384 uptime: 11h 24m 12.52s (Approximately 0.46 days) registries: search: - registry.fedoraproject.org - registry.access.redhat.com - registry.centos.org - docker.io store: configFile: /home/user/.config/containers/storage.conf containerStore: number: 2 paused: 0 running: 0 stopped: 2 graphDriverName: overlay graphOptions: overlay.mount_program: Executable: /usr/bin/fuse-overlayfs Package: fuse-overlayfs-1.7.1-1.module+el8.5.0+12381+e822eb26.x86_64 Version: |- fusermount3 version: 3.2.1 fuse-overlayfs: version 1.7.1 FUSE library version 3.2.1 using FUSE kernel interface version 7.26 graphRoot: /home/user/.local/share/containers/storage graphStatus: Backing Filesystem: xfs Native Overlay Diff: "false" Supports d_type: "true" Using metacopy: "false" imageStore: number: 3 runRoot: /run/user/1000/containers volumePath: /home/user/.local/share/containers/storage/volumes version: APIVersion: 3.3.1 Built: 1630360721 BuiltTime: Mon Aug 30 23:58:41 2021 GitCommit: "" GoVersion: go1.16.7 OsArch: linux/amd64 Version: 3.3.1
podman system prune WARNING! This will remove: - all stopped containers - all stopped pods - all dangling images - all build cache Are you sure you want to continue? [y/N] y
podman run -q --rm --name=myubi registry.access.redhat.com/ubi8/ubi:latest
now=USD(date --iso-8601=seconds) podman events --since=now --stream=false 2023-03-08 14:27:20.696167362 +0100 CET container create d4748226a2bcd271b1bc4b9f88b54e8271c13ffea9b30529968291c62d72fe09 (image=registry.access.redhat.com/ubi8/ubi:latest, name=myubi,...) 2023-03-08 14:27:20.652325082 +0100 CET image pull registry.access.redhat.com/ubi8/ubi:latest 2023-03-08 14:27:20.795695396 +0100 CET container init d4748226a2bcd271b1bc4b9f88b54e8271c13ffea9b30529968291c62d72fe09 (image=registry.access.redhat.com/ubi8/ubi:latest, name=myubi...) 2023-03-08 14:27:20.809205161 +0100 CET container start d4748226a2bcd271b1bc4b9f88b54e8271c13ffea9b30529968291c62d72fe09 (image=registry.access.redhat.com/ubi8/ubi:latest, name=myubi...) 2023-03-08 14:27:20.809903022 +0100 CET container attach d4748226a2bcd271b1bc4b9f88b54e8271c13ffea9b30529968291c62d72fe09 (image=registry.access.redhat.com/ubi8/ubi:latest, name=myubi...) 2023-03-08 14:27:20.831710446 +0100 CET container died d4748226a2bcd271b1bc4b9f88b54e8271c13ffea9b30529968291c62d72fe09 (image=registry.access.redhat.com/ubi8/ubi:latest, name=myubi...) 2023-03-08 14:27:20.913786892 +0100 CET container remove d4748226a2bcd271b1bc4b9f88b54e8271c13ffea9b30529968291c62d72fe09 (image=registry.access.redhat.com/ubi8/ubi:latest, name=myubi...)
journalctl --user -r SYSLOG_IDENTIFIER=podman Mar 08 14:27:20 fedora podman[129324]: 2023-03-08 14:27:20.913786892 +0100 CET m=+0.066920979 container remove Mar 08 14:27:20 fedora podman[129289]: 2023-03-08 14:27:20.696167362 +0100 CET m=+0.079089208 container create d4748226a2bcd271b1bc4b9f88b54e8271c13ffea9b30529968291c62d72f>
podman events --filter event=create 2023-03-08 14:27:20.696167362 +0100 CET container create d4748226a2bcd271b1bc4b9f88b54e8271c13ffea9b30529968291c62d72fe09 (image=registry.access.redhat.com/ubi8/ubi:latest, name=myubi,...)
journalctl --user -r PODMAN_EVENT=create Mar 08 14:27:20 fedora podman[129289]: 2023-03-08 14:27:20.696167362 +0100 CET m=+0.079089208 container create d4748226a2bcd271b1bc4b9f88b54e8271c13ffea9b30529968291c62d72f>
cat ~/.config/containers/containers.conf [engine] events_container_create_inspect_data=true
podman create registry.access.redhat.com/ubi8/ubi:latest 19524fe3c145df32d4f0c9af83e7964e4fb79fc4c397c514192d9d7620a36cd3
now=USD(date --iso-8601=seconds) podman events --since USDnow --stream=false --format "{{.ContainerInspectData}}" | jq ".Config.CreateCommand" [ "/usr/bin/podman", "create", "registry.access.redhat.com/ubi8" ]
journalctl --user -r PODMAN_EVENT=create --all -o json | jq ".PODMAN_CONTAINER_INSPECT_DATA | fromjson" | jq ".Config.CreateCommand" [ "/usr/bin/podman", "create", "registry.access.redhat.com/ubi8" ]