Change:
Virtualbox corrupts guest filesystems by randomly flipping bits when the guest writes data to the virtual hard disks. virtualbox on the host with a new virtual machine for linux 2.6/3.x/4.x with default settings (1GiB RAM, 8GiB harddisk). "Insert" a linux live medium into the virtual CD drive. The virtual hard disk should be placed on a filesystem on a physical hard disk (the bug cannot be triggered if the virtual hard disk is placed on a tmpfs, see below). the machine and open a root shell after booting. these commands inside the guest's root shell to write random data onto the hard disk, then read it back and compare it: Caution: This will overwrite anything on the guest's virtual hard disk The shown hashes should match, but in 90% of test runs, they won't. This bug makes virtualbox quite unreliable: It is still possible to install NixOS inside a virtualbox, but during installation, Nix will report a couple of checksum failures, and sometimes the installation is to be tried more then once. After that, random failures of random programs occur. If btrfs is used, this will also complain about checksum errors. After observing this on my production machine with NixOS 22.05, I tried narrowing down the offending commit. To this end, I adapted the nixpkgs minimal live cd build recipe to build an image with virtualbox installed on it, and built this agains different nixpkgs versions. So most of my tests have been performed on a test machine booted from a usb flash drive. The offending commit seems to be . I guess that commit brings in some kernel patch, which is incompatible with the way virtualbox handles disk images. The bug occurs when the host's filesystem containing the disk image is or . In both cases, I tried with freshly created filesystems. The bug does not occur if the disk image is on (or it occurs with very low probablity). I haven't tried any other filesystems on the host. Just to be sure there isn't a general problem with virtualization: I failed to reproduce the bug with qemu and a qcow2 disk image. So it seems to be a virtualbox-only problem. I observed the bug on three different machines (two desktop machines with NixOS 22.05 and NixOS 21.11, and one laptop with NixOS 21.11). I was also able to reproduce the bug with a linux live cd inside the guest. So the bug can probably be triggered by any linux in the guest. I haven't, however, tried with anything else (e.g. Windows). I was also able to reproduce the bug with a live iso based on (currently ), i.e., with virtualbox 6.1.30 (as opposed to virtualbox 6.1.34 from , currently ). By default, the virtual disk controller has it's checkmark "Use Host I/O cache" turned off in the vm config dialog. Turning it on doesn't help (). I cannot reproduce the bug with these linux kernel versions (all checked with current ) (4.19.247) (5.4.198) (5.18.4) So users of these kernel branches are safe ... until the breaking kernel patch also reaches these branches. <details<summaryFor the record: This is the build recipe I'm using for the live iso</summary<p</p</detailsMaybe I can try to bisect inside the kernel sources, but the compile-reboot-test cycle is quite slow on my machine. Strangely, I cannot find any reports about similar behaviour. The closest report might be . It would be interesting for me if this bug can be reproduced/verified by someone else. I'm not sure if this should be considered a kernel bug (the kernel should probably not change the way disk writes are handled), or a virtualbox bug (at the moment I don't know of any other software affected). kernel maintainer team: virtualbox maintainer: (by the way: I'm surprised there's only one maintainer for virtualbox. Thanks for your good work, ; I'm a heavy user!) This is my production system, which also shows that bug. But I did most of the tests to reproduce the bug and narrow down the cause with live iso images based on current ().
I bisected the linux kernel tree to find as the cause for the guest data corruption. Googling the commit's hash leads to this issue in the VirtualBox bug tracker. According to the information therein, the bug only occurs on some CPU models. The issue there also contains two patches: from This one fixes the problem on linux 5.18. It got already included in nixpkgs' master branch with . It is on it's way to being backported to NixOS 22.05 with . Note that this patch contains several clauses and does not fix the problem on earlier LTS kernel versions. from This one fixes the problem on linux 5.18+, but also on LTS kernel versions. I tested the following combinations with (currently on , where VirtualBox 6.1.34 with and linux 5.15.51 are used by default): VirtualBox version & patch 5.18.8 -----------------------------:------: 6.1.34 + OK 6.1.34 + OK I also tested the following combinations with (currently on , where VirtualBox 6.1.30 and linux 5.15.49 are used by default): VirtualBox version & patch 5.18.6 -----------------------------:------------: 6.1.30 won't build 6.1.34 + OK 6.1.34 + OK To replace VirtualBox 6.1.30 with 6.1.34, I cherry-picked . All tests have been performed on an Intel Core i5-4460 as host, by repeatedly writing/reading 4GiB of random data onto/from a virtual hard disk. Based on these tests, I suggest to use VirtualBox 6.1.34 with the in NixOS 22.05 and in NixOS unstable, as currently, both channels will exhibit the bug in their default configurations (i.e., with kernel 5.15). I will submit a pull request for master shortly.

Discussion:
, libpng, glib, lvm2, libXrandr, libXinerama, libopus, qtbase, qtx11extras , qttools, qtsvg, qtwayland, pkg-config, which, docbook_xsl, docbook_xml_dtd_43 , alsa-lib, curl, libvpx, nettools, dbus, substituteAll, gsoap, zlib <ins> , fetchpatch </ins> # If open-watcom-bin is not passed, VirtualBox will fall back to use # the shipped alternative sources (assembly). , open-watcom-bin

# https://github.com/NixOS/nixpkgs/issues/123851 ./fix-audio-driver-loading.patch # NOTE: both patches below should be removed when updating to 6.1.35 <del> # https://www.virtualbox.org/ticket/20914#comment:6 ./linux518.patch </del> <ins> # https://www.virtualbox.org/ticket/20914#comment:15 (fetchpatch { url = "https://www.virtualbox.org/raw-attachment/ticket/20914/vbox-linux-5.19.patch"; hash = "sha512-NNiMf8kUuM/PimrQCOacYLkrf7UFPh6ZdPsXKyLlsqWfWQXkG92Fv3qZXvg8weE1Z/SBsFTuHICEI4b4l1wZFw=="; extraPrefix = "/"; }) </ins> # https://www.virtualbox.org/ticket/20904#comment:22 ./ffreestanding.patch ];

<del> Index: include/VBox/sup.h =================================================================== --- trunk/include/VBox/sup.h	(revision 151556) +++ trunk/include/VBox/sup.h	(working copy) @@ -2142,6 +2142,26 @@ */ SUPR0DECL(uint32_t) SUPR0GetKernelFeatures(void); +/** + * Notification from R0 VMM prior to loading the guest-FPU register state. + * + * @returns Whether the host-FPU register state has been saved by the host kernel. + * @param   fCtxHook    Whether thread-context hooks are enabled. + * + * @remarks Called with preemption disabled. + */ +SUPR0DECL(bool) SUPR0FpuBegin(bool fCtxHook); + +/** + * Notification from R0 VMM prior to saving the guest-FPU register state (and + * potentially restoring the host-FPU register state) in ring-0. + * + * @param   fCtxHook    Whether thread-context hooks are enabled. + * + * @remarks Called with preemption disabled. + */ +SUPR0DECL(void) SUPR0FpuEnd(bool fCtxHook); + /** @copydoc RTLogGetDefaultInstanceEx * @remarks To allow overriding RTLogGetDefaultInstanceEx locally. */ SUPR0DECL(struct RTLOGGER *) SUPR0GetDefaultLogInstanceEx(uint32_t fFlagsAndGroup); Index: src/VBox/Additions/linux/sharedfolders/regops.c =================================================================== --- trunk/src/VBox/Additions/linux/sharedfolders/regops.c	(revision 151556) +++ trunk/src/VBox/Additions/linux/sharedfolders/regops.c	(working copy) @@ -3823,7 +3823,9 @@ .readpage       = vbsf_readpage, .writepage      = vbsf_writepage, /** @todo Need .writepages if we want msync performance...  */ -#if RTLNX_VER_MIN(2,5,12) +#if RTLNX_VER_MIN(5,18,0) +    .dirty_folio = filemap_dirty_folio, +#elif RTLNX_VER_MIN(2,5,12) .set_page_dirty = __set_page_dirty_buffers, #endif #if RTLNX_VER_MIN(5,14,0) Index: src/VBox/Additions =================================================================== --- trunk/src/VBox/Additions	(revision 151556) +++ trunk/src/VBox/Additions	(working copy) Property changes on: src/VBox/Additions ___________________________________________________________________ Modified: svn:mergeinfo ## -0,0 +0,1 ## Merged /trunk/src/VBox/Additions:r150844 Index: src/VBox/HostDrivers/Support/SUPDrv.cpp =================================================================== --- trunk/src/VBox/HostDrivers/Support/SUPDrv.cpp	(revision 151556) +++ trunk/src/VBox/HostDrivers/Support/SUPDrv.cpp	(working copy) @@ -98,6 +98,18 @@ # endif #endif +#if defined(RT_OS_LINUX) && !defined(__NO_FORTIFY) && defined(__OPTIMIZE__) && defined(CONFIG_FORTIFY_SOURCE) +/* In Linux 5.18-rc1, memcpy became a wrapper which does fortify checks + * before triggering __underlying_memcpy() call. We do not pass these checks here, + * so bypass them for now.  */ +# if RTLNX_VER_MIN(5,18,0) +#  define SUPDRV_MEMCPY __underlying_memcpy +# else +# define SUPDRV_MEMCPY  memcpy +# endif +#else +# define SUPDRV_MEMCPY  memcpy +#endif /* * Logging assignments: @@ -266,6 +278,8 @@ SUPEXP_STK_BACK(    2,  SUPR0ContFree), SUPEXP_STK_BACK(    2,  SUPR0ChangeCR4), SUPEXP_STK_BACK(    1,  SUPR0EnableVTx), +    SUPEXP_STK_OKAY(    1,  SUPR0FpuBegin), +    SUPEXP_STK_OKAY(    1,  SUPR0FpuEnd), SUPEXP_STK_BACK(    0,  SUPR0SuspendVTxOnCpu), SUPEXP_STK_BACK(    1,  SUPR0ResumeVTxOnCpu), SUPEXP_STK_OKAY(    1,  SUPR0GetCurrentGdtRw), @@ -1742,7 +1756,7 @@ /* execute */ pReq->u.Out.cFunctions = RT_ELEMENTS(g_aFunctions); -            memcpy(&pReq->u.Out.aFunctions[0], g_aFunctions, sizeof(g_aFunctions)); +            SUPDRV_MEMCPY(&pReq->u.Out.aFunctions[0], g_aFunctions, sizeof(g_aFunctions)); pReq->Hdr.rc = VINF_SUCCESS; return 0; } Index: src/VBox/HostDrivers/Support/darwin/SUPDrv-darwin.cpp =================================================================== --- trunk/src/VBox/HostDrivers/Support/darwin/SUPDrv-darwin.cpp	(revision 151556) +++ trunk/src/VBox/HostDrivers/Support/darwin/SUPDrv-darwin.cpp	(working copy) @@ -2002,6 +2002,18 @@ } +SUPR0DECL(bool) SUPR0FpuBegin(bool fCtxHook) +{ +    RT_NOREF(fCtxHook); +    return false; +} + + +SUPR0DECL(void) SUPR0FpuEnd(bool fCtxHook) +{ +    RT_NOREF(fCtxHook); +} + /* * * org_virtualbox_SupDrv Index: src/VBox/HostDrivers/Support/freebsd/SUPDrv-freebsd.c =================================================================== --- trunk/src/VBox/HostDrivers/Support/freebsd/SUPDrv-freebsd.c	(revision 151556) +++ trunk/src/VBox/HostDrivers/Support/freebsd/SUPDrv-freebsd.c	(working copy) @@ -640,3 +640,16 @@ return 0; } + +SUPR0DECL(bool) SUPR0FpuBegin(bool fCtxHook) +{ +    RT_NOREF(fCtxHook); +    return false; +} + + +SUPR0DECL(void) SUPR0FpuEnd(bool fCtxHook) +{ +    RT_NOREF(fCtxHook); +} + Index: src/VBox/HostDrivers/Support/linux/SUPDrv-linux.c =================================================================== --- trunk/src/VBox/HostDrivers/Support/linux/SUPDrv-linux.c	(revision 151556) +++ trunk/src/VBox/HostDrivers/Support/linux/SUPDrv-linux.c	(working copy) @@ -1454,6 +1454,31 @@ } +SUPR0DECL(bool) SUPR0FpuBegin(bool fCtxHook) +{ +    RT_NOREF(fCtxHook); +#if RTLNX_VER_MIN(5,18,0) +    kernel_fpu_begin(); +    /* if (fCtxHook) */ +        preempt_enable();  /* HACK ALERT! undo the implicit preempt_disable() in kernel_fpu_begin(). */ +    return true; +#else +    return false; +#endif +} + + +SUPR0DECL(void) SUPR0FpuEnd(bool fCtxHook) +{ +    RT_NOREF(fCtxHook); +#if RTLNX_VER_MIN(5,18,0) +    /* if (fCtxHook) */ +        preempt_disable();  /* HACK ALERT! undo the implicit preempt_enable() in SUPR0FpuBegin(). */ +    kernel_fpu_end(); +#endif +} + + int VBOXCALL    supdrvOSGetCurrentGdtRw(RTHCUINTPTR *pGdtRw) { #if RTLNX_VER_MIN(4,12,0) Index: src/VBox/HostDrivers/Support/os2/SUPDrv-os2.cpp =================================================================== --- trunk/src/VBox/HostDrivers/Support/os2/SUPDrv-os2.cpp	(revision 151556) +++ trunk/src/VBox/HostDrivers/Support/os2/SUPDrv-os2.cpp	(working copy) @@ -541,3 +541,16 @@ return 0; } + +SUPR0DECL(bool) SUPR0FpuBegin(bool fCtxHook) +{ +    RT_NOREF(fCtxHook); +    return false; +} + + +SUPR0DECL(void) SUPR0FpuEnd(bool fCtxHook) +{ +    RT_NOREF(fCtxHook); +} + Index: src/VBox/HostDrivers/Support/solaris/SUPDrv-solaris.c =================================================================== --- trunk/src/VBox/HostDrivers/Support/solaris/SUPDrv-solaris.c	(revision 151556) +++ trunk/src/VBox/HostDrivers/Support/solaris/SUPDrv-solaris.c	(working copy) @@ -1309,3 +1309,16 @@ return 0; } + +SUPR0DECL(bool) SUPR0FpuBegin(bool fCtxHook) +{ +    RT_NOREF(fCtxHook); +    return false; +} + + +SUPR0DECL(void) SUPR0FpuEnd(bool fCtxHook) +{ +    RT_NOREF(fCtxHook); +} + Index: src/VBox/HostDrivers/Support/win/SUPDrv-win.cpp =================================================================== --- trunk/src/VBox/HostDrivers/Support/win/SUPDrv-win.cpp	(revision 151556) +++ trunk/src/VBox/HostDrivers/Support/win/SUPDrv-win.cpp	(working copy) @@ -2704,6 +2704,19 @@ } +SUPR0DECL(bool) SUPR0FpuBegin(bool fCtxHook) +{ +    RT_NOREF(fCtxHook); +    return false; +} + + +SUPR0DECL(void) SUPR0FpuEnd(bool fCtxHook) +{ +    RT_NOREF(fCtxHook); +} + + SUPR0DECL(int) SUPR0IoCtlSetupForHandle(PSUPDRVSESSION pSession, intptr_t hHandle, uint32_t fFlags, PSUPR0IOCTLCTX *ppCtx) { /* Index: src/VBox/HostDrivers/VBoxNetFlt/linux/VBoxNetFlt-linux.c =================================================================== --- trunk/src/VBox/HostDrivers/VBoxNetFlt/linux/VBoxNetFlt-linux.c	(revision 151556) +++ trunk/src/VBox/HostDrivers/VBoxNetFlt/linux/VBoxNetFlt-linux.c	(working copy) @@ -2311,7 +2311,13 @@ vboxNetFltDumpPacket(pSG, true, "host", (fDst & INTNETTRUNKDIR_WIRE) ? 0 : 1); Log6(("vboxNetFltPortOsXmit: pBuf->cb dump:n%.*Rhxdn", sizeof(pBuf->cb), pBuf->cb)); Log6(("vboxNetFltPortOsXmit: netif_rx_ni(%p)n", pBuf)); +#if RTLNX_VER_MIN(5,18,0) +                local_bh_disable(); +                err = netif_rx(pBuf); +                local_bh_enable(); +#else err = netif_rx_ni(pBuf); +#endif if (err) rc = RTErrConvertFromErrno(err); } Index: src/VBox/VMM/VMMR0/CPUMR0.cpp =================================================================== --- trunk/src/VBox/VMM/VMMR0/CPUMR0.cpp	(revision 151556) +++ trunk/src/VBox/VMM/VMMR0/CPUMR0.cpp	(working copy) @@ -440,6 +440,9 @@ Assert(!(pVCpu->cpum.s.fUseFlags & CPUM_USED_FPU_GUEST)); Assert(!(pVCpu->cpum.s.fUseFlags & CPUM_SYNC_FPU_STATE)); +    /* Notify the support driver prior to loading the guest-FPU register state. */ +    SUPR0FpuBegin(false /* unused */); + if (!pVM->cpum.s.HostFeatures.fLeakyFxSR) { Assert(!(pVCpu->cpum.s.fUseFlags & CPUM_USED_MANUAL_XMM_RESTORE)); @@ -484,6 +487,9 @@ Assert(ASMGetCR4() & X86_CR4_OSFXSR); if (pVCpu->cpum.s.fUseFlags & (CPUM_USED_FPU_GUEST | CPUM_USED_FPU_HOST)) { +        /* Notify the support driver prior to loading the host-FPU register state. */ +        SUPR0FpuEnd(false /* unused */); + fSavedGuest = RT_BOOL(pVCpu->cpum.s.fUseFlags & CPUM_USED_FPU_GUEST); if (!(pVCpu->cpum.s.fUseFlags & CPUM_USED_MANUAL_XMM_RESTORE)) cpumR0SaveGuestRestoreHostFPUState(&pVCpu->cpum.s); </del>