Investigating the Difference Between Emulated and Paravirtual Network I/O: The Strange, Untold Story

In virtual setups, guest virtual machines (VMs) perform their I/O through virtual I/O devices that are implemented by the hypervisor in software. The hypervisor commonly implements virtual devices using an indirection layer, which intercepts all the I/O requests of VMs and fulfills them using the corresponding physical devices. The indirection layer has two flavors. The first is “emulation’’, which provides an interface identical to that of some preexisting physical I/O device, thus allowing the operating system (OS) inside the VM to use the original driver of the device, as is, unaware that it is in fact virtual (implemented in software). The second software indirection layer flavor is “paravirtualization’’, which makes the VM aware that it is being virtualized. The purpose of paravirtual devices is to leverage this awareness to reduce the number of virtualization exits (context switches between guest and hypervisor) and thus improve performance. That is, paravirtualization defines a new software-based device specifically designed to minimize the number of exits. The downside of paravirtualization is that it is typically hypervisor-specific, it might not be available for all guest OSes, and it requires users to install hypervisor “tools’’, which is sometimes problematic. Whereas emulated I/O is slower, it is always available and is easier for clients to use and for hypervisor developers to maintain. Focusing on networking, we observe that the performance of emulation is so much worse than that of paravirtualization, that it significantly reduces the utility of emulation and effectively prevents performance-sensitive users from enjoying the benefits. We hypothesize that the poor performance of emulation is unjustified and can be rectified. Our hypothesis is based on a simple model that accounts for the additional virtualization exits incurred by emulation relative to paravirtualization. The model indicates that the poor performance exhibited by emulation could be due to issues different than exits, suggesting that these issues may be resolvable. We systematically analyze the implementation of emulation as compared to paravirtualization. We uncover the problems underlying the performance difference and find that, indeed, they can largely be resolved. The fixes that we apply to emulation significantly shorten its latency: from 1.50–3.59x longer than paravirtualization to only 1.14–1.31x. Counterintuitively, our fixed emulation usually outperforms paravirtualization in terms of throughput, increasing it from as low as 0.07–0.23x the throughput of paravir1 Technion Computer Science Department M.Sc. Thesis MSC-2021-19 2021 tualization to 0.95–3.63x. This result initially seems impossible, because paravirtualization is supposed to be more efficient than emulation by design. Upon investigation, we find that the implementation of paravirtualization is suboptimal relative to emulation in one respect: it delivers many more virtual interrupts than necessary, consequently needlessly incurring many wasteful virtualization exits. Surprisingly, fixing this issue still does not resolve the mystery, as emulation still frequently outperforms paravirtualization, delivering 1.23–2.08x higher throughput for ≤8KB message sizes. We conduct a deeper analysis and discover the root cause of the mystery: the nature of the TCP/IP network stack is such that it batches the transmitted data more aggressively when the underlying (virtual) network device is slower—as is the case for emulation. When experimentally neutralizing this batching behavior (an act that is impractical on production setups but that allows us to compare “apples to apples”), paravirtualization finally outperforms emulation for all message sizes, as one would normally expect.