# Analyzing GPU Performance in Virtualized Environments: A Case Study
## 1 Introduction
GVT-g is open source and has been integrated into the Linux mainline kernel, which makes it a desirable option due to its accessibility and potential for broader integration. On the other hand, performance analysis tools for GPUs are important for debugging performance issues in GPU-accelerated applications [8]. These tools help understand how the GPU resources are allocated and consumed, and they facilitate the diagnosis of potential performance bottlenecks. They are particularly crucial in virtualized environments, where resource sharing in vGPUs and its impact on performance need to be better understood. However, developing practical tools for monitoring and debugging vGPUs remains challenging. This is because virtualized environments often present many layers, encompassing hardware, middleware, and host and guest operating systems, which increase the isolation and abstraction of GPU resources, making it difficult to pinpoint the causes of performance issues. The landscape of GPU performance analysis features diverse tools, with academic contributions tailored to specific GPU programming models or diagnosing particular GPU-related performance issues. For instance, the works in [9; 10] leveraged library interposition to capture runtime events from OpenCL- and HSA-based programs, respectively. While they enabled the analysis of GPU kernel execution and CPU-GPU interaction, they presented high overhead and faced challenges for broader analysis and portability. On the other hand, vendor-provided options, such as vTune Profiler [11] and Nsight Systems [12], leverage dynamic instrumentation and hardware counters to gather performance data and unveil application behavior in various aspects. These tools offer interesting analyses covering kernel execution, memory access patterns, and GPU API call paths. Nevertheless, they are often limited in terms of openness and cross-architecture applicability. In summary, while each of these tools has its strengths and weaknesses, a common shortfall is their lack of support for vGPUs--with the exception of Nsight Systems, which, however, is proprietary and closed-source software. This paper presents a novel performance analysis framework dedicated to GPUs virtualized with the GVT-g technology. Our framework uses host-based tracing to gather performance data efficiently and with minimal overhead. Tracing is a proven technique for collecting detailed performance data from complex systems. A key advantage of our approach is that it does not necessitate the installation of any agents or tools within the VMs, as tracing is confined to the host operating system's kernel space. The benefits of this methodology include ease of deployment, preservation of VM owner privacy, and cost-effectiveness. This study's contributions are as follows:  
1. Efficient instrumentation of KVMGT, the open-source implementation of GVT-g for KVM [13], and integrated modules within the Linux Trace Toolkit Next Generation (LTTng) [14] to collect performance data from both the virtualization solution and the Intel GPU driver. 2. Development of a unified, stateful model capable of filtering and organizing the collected trace data. This model simplifies data processing and supports our analysis in computing many performance metrics. 3. Implementation of various analyses within Trace Compass [15], an open-source performance analyzer, and a series of synchronized graphical views. These views aim to assist practitioners in understanding GVT-g's internal mechanisms and in diagnosing many performance issues related to vGPU usage. The remainder of this paper is organized as follows: In Section 2, we conduct a comprehensive literature review of the methodologies and techniques applied in GPU virtualization. Section 3 introduces prominent GPU profiling and performance analysis tools. Section 4 details the design of our proposed framework and outlines several pertinent performance metrics. Next, Section 5 presents three practical scenarios that demonstrate the effectiveness of our framework.