Change:
sha256 = "sha256-gTRf9XENAfbFB3asYCXnw4OV4Af6VE1W56K2xpYDhgM="; revert = true; }) <ins> # make nixos tests that boot from USB more stable # https://lists.nongnu.org/archive/html/qemu-devel/2022-05/msg01484.html (fetchpatch { url = "https://gitlab.com/raboof/qemu/-/commit/3fb5e8fe4434130b1167a995b2a01c077cca2cd5.patch"; sha256 = "sha256-evzrN3i4ntc/AFG0C0rezQpQbWcnx74nXO+5DLErX8o="; }) </ins> ] ++ lib.optional nixosTestRunner ./force-uid0-on-9p.patch;

Discussion:
Specifically: For comparison, a successful run loads bzImage successfully:
Might this be related to ?
Possibly, the certainly overlaps, and it seems is also USB-related. I guess the question now is which is the cause and which is the effect?
Even in the past weeks the job has been failing occasionally. I usually don't even look into the log anymore unless it failed twice. Locally I don't get an error on the same commit. Once Hydra's queue runner gets fixed, we'll see if it succeeds.
Yes, I completely agree. Let's keep this ticket to track the instability, though. Ok, I now think the is the cause and the is the effect: AFAICT what happens is: ISOLINUX asks the BIOS to read bzImage (presumably with or somesuch) the BIOS (SeaBIOS) uses USB to get this file from qemu qemu and aborts the USB transfer the BIOS presumably picks up the error and So it seems like there is something wonky in the USB communication between SeaBIOS and qemu. I had a bit of a look but the implementations look reasonable on both sides at first glance. I wonder if we could run this test with SeaBIOS debugging enabled, but using a different SeaBIOS was not as simple as adding to the test ;). Anyone more well-versed in qemu?
Thanks for looking into this! I found this bug report
Great find! Indeed with that ISO it seems easier to reproduce (while loading the kernel - if you pass '-nographics' you can hit 'enter' and select '2' to load from the console) - perhaps it's just bigger? I also found it sometimes just hangs instead of hitting the cpage error, which might be what causes . Adding some additional shows: Indeed that's an invalid USB packet: you can fit bytes in there, but only if you start at offset 0, since there's only 4 pages and each page is 4096 bytes. Adding a bunch of logging to qemu, seeing the following pattern: It seems that qemu(?) 'writes back' the new offset (785+31=816), and somehow that value can 'leak' into the qTD structure for the next request for transferring bytes (which should start at offset 0, not 816).

I can confirm I can reproduce the problem with seabios and so far not with 1.13.0, though tbh nothing jumps out of me looking at the commits in there...
Would you be able to bisect between and 1.13.0?
I could, though it's rather slow work, since the problem doesn't occur every time. Also it's not entirely clear it would help much: I noticed that when I add some debugging statements to seabios in just the right/wrong places, I can no longer reproduce the problem either... so even if we find the first commit that reproduces the problem, that might not be the commit that actually introduced the bug.
After a painful bisect I found the same commit as the person in the bug report: . I can say quite confidently that commenting out (from that commit) or makes the problem go away. Also, sometimes instead of the cpage error I get . This all seems to point at a concurrency/timing problem. Command used to test the issue: hit Enter and wait a few seconds after "Loading initrd...".
aha interesting... so on my machine this updates from 1194 (~1 MHz - seems kinda weird, not sure where that came from?) to (~2.4 GHz, indeed my host machine cpu speed), and this value is then used when performs calls to or .
Also it updates from 0x40 to 0, and commenting out also seems to fix the issue.
So before it would use the 3.5 MHz PM TIMER, and since then it uses the 2.4GHz TSC (Time-Stamp Counter) timer. When I comment out the problem indeed goes away. When I replace it with , which switches to TSC, I can reproduce the problem again. I don't see any code that obviously would be impacted by a different clock: where the clock is used in EHCI it's usually in timeouts, and AFAICS we're not hitting any timeouts in this scenario in either the 'happy' or the 'problematic' change. So preliminary it seems the different timer causes a timing difference that triggers the bug, but I don't see strong evidence yet that the timer itself is really 'wrong'. The qemu controller uses DMA calls (/) to read and write what I think is called the 'overlay area' to/from main memory.

It looks like that is where things go wrong: 'usually' the 'writeback' writes "0 bytes, offset X" to the 'overlay area' and immediately after that reads " bytes, offset 0" from the same main memory address. However, in the problematic scenario, it writes "0 bytes, offset X" and then reads " bytes, offset X" (same X). So it looks like either seabios writes the 'new' 0 offset too early (before qemu writes the 'old' offset X) or too late (after qemu reads the new offset, which should be 0 but in the problematic case is X). I changed qemu to retry fetching the offset when the values don't make sense (), and with that change I can still reproduce the problem: OK, not very scientific, but that kinda suggests qemu does the writeback after seabios writes the '0' offset. If that's true, then the question becomes: is seabios writing too early, or is qemu writing too late? I haven't figured out yet how EHCI is supposed to guard against such race conditions, but is making me slightly nervous ;)
That's bone-chilling, but on a more practical note: should we just patch out of seabios in nixpkgs? Is using the TSC timer supposed to make a difference? I haven't noticed a speed-up, certainly not on a factor of 1000.
:laughing: That seems a bit heavy-handed, but I guess we could use a build with . I agree I don't think it's supposed to change the speed at which things run, it's just another way to keep the time
Right. Actually we'd need to change the seabios shipped with qemu, not the one in nixpkgs.
I think you can override the one shipped with qemy by passing a to the test - just have to use (and ) while building seabios
Opened
This is great, but I think I figured out the 'real' problem now!
Neat!
Thanks a lot for finding that post by Lin Ma and bouncing ideas here, without that I'd surely have given up much earlier ;)

Describe the bug Build failure on hydra release-20.09 () To Reproduce Expected behavior Screenshots Additional context Notify maintainers Metadata Please run and paste the result. Maintainer information:
Seems like an unreliable test. Removed channel blocker severity.
again...
Please keep this label only for when its currently blocking, as these items are shown on
Aand it's back, on 20.09
This is a flaky test, but not a channel blocker. advanced a day ago: !
Some differences in the log : bad log from previous comment : successful log from local build This error seems closest to the root cause. A subsequent error seems to be about a fallback that doesn't work. In the successful log, no ramdisk is "detected", presumably because the virtual usb drive became the root filesystem. <details<summaryprobably less relevant log</summary</details<details<summarydiff command</summarycanon(){ sed -e 's/ [0-9][.][0-9]//g'; }; diff -U3 --color=always <(canon <~/Downloads/6m5q895jfljzd662wz9yiz66vsc3kxmy-vm-test-run-boot-bios-) <(canon </tmp/nri21z7knvx3zkdxzip88376m178gy6m-vm-test-run-boot-bios-) | less -R </details
Again. This issue may show selection bias because I only check 20.09.
Duplicate of
I marked this as stale due to inactivity. &rarr;

Note: error 0c00 = unsupported track or invalid media () reaches the syslinux menu, then hangs loading the initrd: Haven't been able to reproduce these locally, so it might be triggered by high load on the host. This may be a QEMU USB emulation issue. It doesn't appear to affect the USB UEFI boot test though.
I'm not sure what to do about this. I can imagine syslinux being a bit pickier about the read-times, so it might timeout earlier than Linux itself. Maybe setting io limits on the host system can reproduce this problem. I'll give that a try later. However, this doesn't help the build much. If an IO timeout in syslinux is actually the problem, patching syslinux might solve the problem, but that seems like something you'd want to avoid for the boot test. Is there some way to do this boot test not parallel to other tests?
For now the test has been removed from release blockers:
we could set . opinions?
Just FYI, this is still happening on 16.09
Would removing/disabling this test be the better option? I'm not sure what "big-parallel" does for this test nor whether it will help in this case?
This still occurs quite often. I've gathered logs in . I don't watch hydra closely but in my experience this is the most frequent flaky "blocker".
Should we just disable it? I'm not sure how to solve this other than not using the usb emulation of qemu that we're using right now.
Your earlier comment may be worth considering This means we don't test one failure mode, but we still test many other usb-specific failure modes. I don't think this timeout is a problem in practice anyway, or we could just ship a syslinux with an increased timeout. Are you familiar with ?
Not really. Only a bit as a user when I initially worked on USB boot support of NixOS. That said, I did do a bit of spelunking in the syslinux code and did find that it tries to do this a number of times (retry count): There is does call interrupt 13, which is used to read from disks. The error that mentioned in the description is elsewhere: It seems that error is also triggered when interrupt 13 is tried too many times.

It has its own int13 call, so it also handles its own retry-count. The retry-count of this function is here: There doesn't seem to be a way to change this using configuration, so we would have to patch the code to increase the timeout. This is all because interrupt 13 (IO) on the build server inside qemu is failing 6 times... or so it seems from what I'm reading in the syslinux code. I'm not sure we should be fixing syslinux, as there might be many other low-level tests that might be failing this way. Maybe there is some configuration within qemu to increase disk timeout?
This might be failing inside SeaBIOS; my hypothesis is that we're hitting SeaBIOS' timeout on USB I/O (which is ) Given that this seems to succeed when it's retried, presumably because of lower overhead on whatever machine's getting used for the build: can we do something to force the USB image to remain resident in the disk cache throughout the test, e.g. by using vmtouch?
I tried implementing what you suggested: But that fails because vmtouch can't lock that much memory. Is there a way to raise the ulimits for a test?
Some new developments in
Proposed fix: