Chapter 23. Diagnosing virtual machine problems When working with virtual machines (VMs), you may encounter problems with varying levels of severity. Some problems may have a quick and easy fix, while for others, you may have to capture VM-related data and logs to report or diagnose the problems. The following sections provide detailed information about generating logs and diagnosing some common VM problems, as well as about reporting these problems. 23.1. Generating libvirt debug logs To diagnose virtual machine (VM) problems, it is helpful to generate and review libvirt debug logs. Attaching debug logs is also useful when asking for support to resolve VM-related problems. The following sections explain what debug logs are , how you can set them to be persistent , enable them during runtime , and attach them when reporting problems. 23.1.1. Understanding libvirt debug logs Debug logs are text files that contain data about events that occur during virtual machine (VM) runtime. The logs provide information about fundamental server-side functionalities, such as host libraries and the libvirt daemon. The log files also contain the standard error output ( stderr ) of all running VMs. Debug logging is not enabled by default and has to be enabled when libvirt starts. You can enable logging for a single session or persistently . You can also enable logging when a libvirt daemon session is already running by modifying the daemon run-time settings . Attaching the libvirt debug logs is also useful when requesting support with a VM problem. 23.1.2. Enabling persistent settings for libvirt debug logs You can configure libvirt debug logging to be automatically enabled whenever libvirt starts. By default, virtqemud is the main libvirt daemon in RHEL 9. To make persistent changes in the libvirt configuration, you must edit the virtqemud.conf file, located in the /etc/libvirt directory. Note In some cases, for example when you upgrade from RHEL 8, libvirtd might still be the enabled libvirt daemon. In that case, you must edit the libvirtd.conf file instead. Procedure Open the virtqemud.conf file in an editor. Replace or set the filters according to your requirements. Table 23.1. Debugging filter values 1 logs all messages generated by libvirt. 2 logs all non-debugging information. 3 logs all warning and error messages. This is the default value. 4 logs only error messages. Example 23.1. Sample daemon settings for logging filters The following settings: Log all error and warning messages from the remote , util.json , and rpc layers Log only error messages from the event layer. Save the filtered logs to /var/log/libvirt/libvirt.log Save and exit. Restart the libvirt daemon. 23.1.3. Enabling libvirt debug logs during runtime You can modify the libvirt daemon's runtime settings to enable debug logs and save them to an output file. This is useful when restarting the libvirt daemon is not possible because restarting fixes the problem, or because there is another process, such as migration or backup, running at the same time. Modifying runtime settings is also useful if you want to try a command without editing the configuration files or restarting the daemon. Prerequisites Make sure the libvirt-admin package is installed. Procedure Optional: Back up the active set of log filters. Note It is recommended that you back up the active set of filters so that you can restore them after generating the logs. If you do not restore the filters, the messages will continue to be logged which may affect system performance. Use the virt-admin utility to enable debugging and set the filters according to your requirements. Table 23.2. Debugging filter values 1 logs all messages generated by libvirt. 2 logs all non-debugging information. 3 logs all warning and error messages. This is the default value. 4 logs only error messages. Example 23.2. Sample virt-admin setting for logging filters The following command: Logs all error and warning messages from the remote , util.json , and rpc layers Logs only error messages from the event layer. Use the virt-admin utility to save the logs to a specific file or directory. For example, the following command saves the log output to the libvirt.log file in the /var/log/libvirt/ directory. Optional: You can also remove the filters to generate a log file that contains all VM-related information. However, it is not recommended since this file may contain a large amount of redundant information produced by libvirt's modules. Use the virt-admin utility to specify an empty set of filters. Optional: Restore the filters to their original state using the backup file. Perform the second step with the saved values to restore the filters. 23.1.4. Attaching libvirt debug logs to support requests You may have to request additional support to diagnose and resolve virtual machine (VM) problems. Attaching the debug logs to the support request is highly recommended to ensure that the support team has access to all the information they need to provide a quick resolution of the VM-related problem. Procedure To report a problem and request support, open a support case . Based on the encountered problems, attach the following logs along with your report: For problems with the libvirt service, attach the /var/log/libvirt/libvirt.log file from the host. For problems with a specific VM, attach its respective log file. For example, for the testguest1 VM, attach the testguest1.log file, which can be found at /var/log/libvirt/qemu/testguest1.log . Additional resources How to provide log files to Red Hat Support? (Red Hat Knowledgebase) 23.2. Dumping a virtual machine core To analyze why a virtual machine (VM) crashed or malfunctioned, you can dump the VM core to a file on disk for later analysis and diagnostics. This section provides a brief introduction to core dumping and explains how you can dump a VM core to a specific file. 23.2.1. How virtual machine core dumping works A virtual machine (VM) requires numerous running processes to work accurately and efficiently. In some cases, a running VM may terminate unexpectedly or malfunction while you are using it. Restarting the VM may cause the data to be reset or lost, which makes it difficult to diagnose the exact problem that caused the VM to crash. In such cases, you can use the virsh dump utility to save (or dump ) the core of a VM to a file before you reboot the VM. The core dump file contains a raw physical memory image of the VM which contains detailed information about the VM. This information can be used to diagnose VM problems, either manually, or by using a tool such as the crash utility. Additional resources crash man page on your system The crash Github repository 23.2.2. Creating a virtual machine core dump file A virtual machine (VM) core dump contains detailed information about the state of a VM at any given time. This information, which is similar to a snapshot of the VM, can help you detect problems if a VM malfunctions or shuts down suddenly. Prerequisites Make sure you have sufficient disk space to save the file. Note that the space occupied by the VM depends on the amount of RAM allocated to the VM. Procedure Use the virsh dump utility. For example, the following command dumps the lander1 VM's cores, its memory and the CPU common register file to gargantua.file in the /core/file directory. Important The crash utility no longer supports the default file format of the virsh dump command. To analyze a core dump file by using crash , you must create the file with the --memory-only option. Additionally, you must use the --memory-only option when creating a core dump file to attach to a Red Hat Support Case. Troubleshooting If the virsh dump command fails with a System is deadlocked on memory error, ensure you are assigning sufficient memory for the core dump file. To do so, use the following crashkernel option value. Alternatively, do not use crashkernel at all, which assigns core dump memory automatically. Additional resources virsh dump --help command virsh man page on your system Opening a Support Case 23.3. Backtracing virtual machine processes When a process related to a virtual machine (VM) malfunctions, you can use the gstack command along with the process identifier (PID) to generate an execution stack trace of the malfunctioning process. If the process is a part of a thread group then all the threads are traced as well. Prerequisites Ensure that the GDB package is installed. For details about installing GDB and the available components, see Installing the GNU Debugger . Make sure you know the PID of the processes that you want to backtrace. You can find the PID by using the pgrep command followed by the name of the process. For example: Procedure Use the gstack utility followed by the PID of the process you wish to backtrace. For example, the following command backtraces the libvirt process with the PID 22014. Additional resources gstack man page on your system GNU Debugger (GDB) Additional resources for reporting virtual machine problems and providing logs To request additional help and support, you can: Raise a service request by using the redhat-support-tool command line option, the Red Hat Portal UI, or several methods of FTP. To report problems and request support, see Open a Support Case . Upload the SOS Report and the log files when you submit a service request. This ensures that the Red Hat support engineer has all the necessary diagnostic information for reference. For more information about SOS reports, see the Red Hat Knowledgebase solution What is an SOS Report and how to create one in Red Hat Enterprise Linux? For information about attaching log files, see the Red Hat Knowledgebase solution How to provide files to Red Hat Support?

Commands:
log_filters="3:remote 4:event 3:util.json 3:rpc" log_outputs="1:file:/var/log/libvirt/libvirt.log"
systemctl restart virtqemud.service
virt-admin -c virtqemud:///system daemon-log-filters >> virt-filters-backup
virt-admin -c virtqemud:///system daemon-log-filters "3:remote 4:event 3:util.json 3:rpc"
virt-admin -c virtqemud:///system daemon-log-outputs "1:file:/var/log/libvirt/libvirt.log"
virt-admin -c virtqemud:///system daemon-log-filters Logging filters:
virsh dump lander1 /core/file/gargantua.file --memory-only Domain 'lander1' dumped to /core/file/gargantua.file
crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M
pgrep libvirt 22014 22025
gstack 22014 Thread 3 (Thread 0x7f33edaf7700 (LWP 22017)): #0 0x00007f33f81aef21 in poll () from /lib64/libc.so.6 #1 0x00007f33f89059b6 in g_main_context_iterate.isra () from /lib64/libglib-2.0.so.0 #2 0x00007f33f8905d72 in g_main_loop_run () from /lib64/libglib-2.0.so.0