Change:
Specifically: For comparison, a successful run loads bzImage successfully:
Might this be related to ?
Possibly, the certainly overlaps, and it seems is also USB-related. I guess the question now is which is the cause and which is the effect?
Even in the past weeks the job has been failing occasionally. I usually don't even look into the log anymore unless it failed twice. Locally I don't get an error on the same commit. Once Hydra's queue runner gets fixed, we'll see if it succeeds.
Yes, I completely agree. Let's keep this ticket to track the instability, though. Ok, I now think the is the cause and the is the effect: AFAICT what happens is: ISOLINUX asks the BIOS to read bzImage (presumably with or somesuch) the BIOS (SeaBIOS) uses USB to get this file from qemu qemu and aborts the USB transfer the BIOS presumably picks up the error and So it seems like there is something wonky in the USB communication between SeaBIOS and qemu. I had a bit of a look but the implementations look reasonable on both sides at first glance. I wonder if we could run this test with SeaBIOS debugging enabled, but using a different SeaBIOS was not as simple as adding to the test ;). Anyone more well-versed in qemu?
Thanks for looking into this! I found this bug report
Great find! Indeed with that ISO it seems easier to reproduce (while loading the kernel - if you pass '-nographics' you can hit 'enter' and select '2' to load from the console) - perhaps it's just bigger? I also found it sometimes just hangs instead of hitting the cpage error, which might be what causes . Adding some additional shows: Indeed that's an invalid USB packet: you can fit bytes in there, but only if you start at offset 0, since there's only 4 pages and each page is 4096 bytes. Adding a bunch of logging to qemu, seeing the following pattern: It seems that qemu(?) 'writes back' the new offset (785+31=816), and somehow that value can 'leak' into the qTD structure for the next request for transferring bytes (which should start at offset 0, not 816). I can confirm I can reproduce the problem with seabios and so far not with 1.13.0, though tbh nothing jumps out of me looking at the commits in there...
Would you be able to bisect between and 1.13.0?
I could, though it's rather slow work, since the problem doesn't occur every time. Also it's not entirely clear it would help much: I noticed that when I add some debugging statements to seabios in just the right/wrong places, I can no longer reproduce the problem either... so even if we find the first commit that reproduces the problem, that might not be the commit that actually introduced the bug.
After a painful bisect I found the same commit as the person in the bug report: . I can say quite confidently that commenting out (from that commit) or makes the problem go away. Also, sometimes instead of the cpage error I get . This all seems to point at a concurrency/timing problem. Command used to test the issue: hit Enter and wait a few seconds after "Loading initrd...".
aha interesting... so on my machine this updates from 1194 (~1 MHz - seems kinda weird, not sure where that came from?) to (~2.4 GHz, indeed my host machine cpu speed), and this value is then used when performs calls to or .
Also it updates from 0x40 to 0, and commenting out also seems to fix the issue.
So before it would use the 3.5 MHz PM TIMER, and since then it uses the 2.4GHz TSC (Time-Stamp Counter) timer. When I comment out the problem indeed goes away. When I replace it with , which switches to TSC, I can reproduce the problem again. I don't see any code that obviously would be impacted by a different clock: where the clock is used in EHCI it's usually in timeouts, and AFAICS we're not hitting any timeouts in this scenario in either the 'happy' or the 'problematic' change. So preliminary it seems the different timer causes a timing difference that triggers the bug, but I don't see strong evidence yet that the timer itself is really 'wrong'. The qemu controller uses DMA calls (/) to read and write what I think is called the 'overlay area' to/from main memory. It looks like that is where things go wrong: 'usually' the 'writeback' writes "0 bytes, offset X" to the 'overlay area' and immediately after that reads " bytes, offset 0" from the same main memory address. However, in the problematic scenario, it writes "0 bytes, offset X" and then reads " bytes, offset X" (same X). So it looks like either seabios writes the 'new' 0 offset too early (before qemu writes the 'old' offset X) or too late (after qemu reads the new offset, which should be 0 but in the problematic case is X). I changed qemu to retry fetching the offset when the values don't make sense (), and with that change I can still reproduce the problem: OK, not very scientific, but that kinda suggests qemu does the writeback after seabios writes the '0' offset. If that's true, then the question becomes: is seabios writing too early, or is qemu writing too late? I haven't figured out yet how EHCI is supposed to guard against such race conditions, but is making me slightly nervous ;)
That's bone-chilling, but on a more practical note: should we just patch out of seabios in nixpkgs? Is using the TSC timer supposed to make a difference? I haven't noticed a speed-up, certainly not on a factor of 1000.
:laughing: That seems a bit heavy-handed, but I guess we could use a build with . I agree I don't think it's supposed to change the speed at which things run, it's just another way to keep the time
Right. Actually we'd need to change the seabios shipped with qemu, not the one in nixpkgs.
I think you can override the one shipped with qemy by passing a to the test - just have to use (and ) while building seabios
Opened
This is great, but I think I figured out the 'real' problem now!
Neat!
Thanks a lot for finding that post by Lin Ma and bouncing ideas here, without that I'd surely have given up much earlier ;)

Discussion:
sha256 = "sha256-gTRf9XENAfbFB3asYCXnw4OV4Af6VE1W56K2xpYDhgM="; revert = true; }) <ins> # make nixos tests that boot from USB more stable # https://lists.nongnu.org/archive/html/qemu-devel/2022-05/msg01484.html (fetchpatch { url = "https://gitlab.com/raboof/qemu/-/commit/3fb5e8fe4434130b1167a995b2a01c077cca2cd5.patch"; sha256 = "sha256-evzrN3i4ntc/AFG0C0rezQpQbWcnx74nXO+5DLErX8o="; }) </ins> ] ++ lib.optional nixosTestRunner ./force-uid0-on-9p.patch;