Code change:
I'm not sure when exactly it happens, but it reproduces roughly like this: Rebase my working branch on upstream/master. Run , the build starts from or later and succeeds. Run again, should be a trivial build, but the build starts from or again and succeeds. Run again, trivial build happens. The target is x86_64-pc-windows-gnu. The issue exists for several weeks, so it's a regression. cc
Reproduction script: You might have to run it multiple times to cause the spurious rebuild in the last command.
For some reason is getting recompiled:
cc
I think was hitting this at some point as well, though we weren't able to determine quite why.
I haven't encountered these rebuilds before, but recently (since December?) they happen quite often, like every second or third build, and that's pretty inconvenient given the long builds. Perhaps something changed in itself?
I don't think we've changed backtrace itself recently. It would perhaps be handy to check if this still happens before (which changed how/when we clear out std's target directory). However, I don't expect it would have, as that would cause all of std to be rebuilt, not just backtrace. Maybe backtrace is unique in being a dependency of both std and the compiler (for example?). I'm not sure what could be causing this odd behavior :/
I'm pretty sure the rebuilds appeared before (and I hoped that PR would fix them, but it didn't).
I suspect to be the cause, at least based on timing.
I've been hitting similar (this?) issue with x86_64-pc-windows-gnu when doing builds. Somehow it disappeared whenever I ran before so I kept doing that as the workaround.
Hm, so in theory at least wouldn't have affected backtrace specifically, I think, though there may have been knock on effects. One thing that might be helpful is getting a log with on the spurious rebuild (to know why Cargo thinks it needs to rebuild it).
Update: this reproduces on too (which runs much quicker and involves less moving parts), just got a rebuild after a clean build this way. I'll try .
Reproduced on () with :
I was able to reproduce this once, but I'm having trouble reproducing it again. When I did reproduce it, I noticed that the included . That is not normal, and I think points to a race condition with the libcore rlib getting emitted at just the wrong time. I suspect it is due to pipelining and maybe . I'm still trying to figure that out.
binary-dep-depinfo should not include backtrace into libcore, I think, since we don't pass to libcore (nor is there an implicit "sysroot" dep from libcore; core depends on nothing). I don't know why we're including dep-lib-backtrace in the libcore dep file though, that does sound like a bug. I guess this is then likely an upstream bug in some sense? I think leaving it here for tracking at least for now makes sense though.
It's the other way around. 's dep-info file noted a dependency on , which is normal. Normally it should only list the file, but when I saw it, it also included the file. That normally shouldn't happen because the libcore codegen step takes such a long time (normally finishes after backtrace is finished). I'm not too familiar with the compiler, so it's a little slow for me to figure out. I seem to recall being told that writing the file is not done in an atomic fashion, is that correct? Perhaps writing it can take a long time, and backtrace started after the was created and finished running before the was done? I also seem to recall that there are protections in rustc to deal with partially written rlib files, but I'm not sure how that is accomplished (are there checksums?).
We currently emit a dependency on both the rmeta and rlib files (presuming they were found during building, I guess?) with . That might be in error! I did so as the most pessimistic option when initially adding that feature, but perhaps I was too eager in pulling in dependencies. If I'm understanding correctly, that may mean that if libcore's codegen completes before backtrace is built, we'll pull in the libcore rlib as a dependency of backtrace, and then when backtrace is checked for freshness in the future, we'll notice that it's actually been built before the rlib (which we finished writing after libbacktrace started). Is there ever a case where there's a dependency on rlibs directly? Maybe we should only emit rmeta/dylib filepaths in binary-dep-depinfo?
For reference, here's what I have in that file (with NUL bytes removed):
Hm, so it looks like in practice we that we find -- so in some sense, we don't actually depend on all three, but rather rlib/rmeta/dylib, in that order, when we're loading up dependency information. I guess there's not really any tracking for which one we've used metadata from, but in practice, we should probably only emit one of these (probably rmeta/rlib/dylib, in that order) that we have set as the CrateSource.
Hm, so it looks like in practice we read metadata from the first of the three that we find -- so in some sense, we don't actually depend on all three, but rather rlib/rmeta/dylib, in that order, when we're loading up dependency information. This is actually something that I wanted to try to change () and read from the rmeta first.
Even if is used for reading metadata, can still be used for reading object code, in that case both should indeed be in dependencies? Perhaps cargo just timestamps the link time dependencies at the wrong time? (Earlier than necessary, together with .)
I was able to record exactly what happens. The timeline is: T1. emitted T2. backtrace build starts T3. emitted T4. backtrace loads T5. backtrace writes dep-info with and T6. backtrace finishes T7. Cargo reverses backtrace dep-info timestamp to T2 () (this is intended to deal with files that are modified in the middle of the build) The result is that 's timestamp is T3 and backtrace dep-info timestamp is T2, and since T3 T2, the next time Cargo runs it thinks backtrace needs to be rebuilt. I'm not sure how to fix this. One solution would be to change the priority to load rmeta over rlib, and have only emit one file. Or Cargo could hash the files :wink:. Maybe has ideas?
I like the idea of inverting rustc's preference, using only the rmeta rather than both the rmeta and rlib if they're available. Other than that though I don't know of a great way to solve this :( I may be missing something though, is this rustc-specific? This seems like a bug that everyone should be hitting naively which makes it quite serious?
It is only with . Also, the timing has to be just right for the rlib to be created in a very small window (the time between T2 and T4 is extremely small).
Ah I see, makes sense! Here's another possible question and/or solution. When Cargo builds libbacktrace it's presumably only passing , so I think it's rustc's automatic logic to pick up . Is that correct? If so it may be more correct to, if is passed, disable all other auto-loading?
Backtrace only sees via . is loaded as a dependency of that. (Via I think?)
I think this is our best bet -- furthermore, I wonder if there's some wins to be had in terms of compile times. It seems like we are currently loading libcore's rmeta/rlib for all crates out there, which presumably is not great (more IO, etc.). I can pursue a PR to that effect soon.
aha ok that makes sense as well. Then yeah I think your suggestion is indeed the only thing I can think of as well!
Hm, actually, in going to do this, it sounds like we can't quite just emit one of them, per "Even if rmeta is used for reading metadata, rlib can still be used for reading object code, in that case both should indeed be in dependencies?" -- right?
When building an rlib we never use contents of upstream rlibs. When building dynamically linked artifacts, though (or executables) we do use upstream object code. (this is basically pipelining)
Aha, okay, so it sounds like a partial fix would be to: If we're producing rmeta/rlib: emit just rmeta dependency if we're producing dylib/cydlib/binary: emit just rlib dependency Does that sound right? Basically, we emit "worst case" for either situation which I think might mean that for dylibs this bug persists -- since they'll start compilation before the rlib is finished -- but I don't know how much we can do about it under the current model.
That should be right yeah, and I think the dylib bug won't be an issue because Cargo won't start libstd until all subsequent rlib compilations are finished (e.g. it doesn't pipeline it)
Great! Okay, then I'll try to get that impl up. (I presume the "dylib bug won't be an issue" also applies to dylibs in the wild, right?)
That's the theory at least!

Related discussion:
text: dylibs: FxHashMap<PathBuf, PathKind>, ) -> Option<(Svh, Library)> { let mut slot = None; <ins> // Order here matters, rmeta should come first. See comment in // `extract_one` below. </ins> let source = CrateSource { <del> rlib: self.extract_one(rlibs, CrateFlavor::Rlib, &mut slot), </del> rmeta: self.extract_one(rmetas, CrateFlavor::Rmeta, &mut slot), <ins> rlib: self.extract_one(rlibs, CrateFlavor::Rlib, &mut slot), </ins> dylib: self.extract_one(dylibs, CrateFlavor::Dylib, &mut slot), }; slot.map(|(svh, metadata)| (svh, Library { source, metadata })) } <ins> fn needs_crate_flavor(&self, flavor: CrateFlavor) -> bool { if flavor == CrateFlavor::Dylib && self.is_proc_macro == Some(true) { return true; } // The all loop is because `--crate-type=rlib --crate-type=rlib` is // legal and produces both inside this type. let is_rlib = self.sess.crate_types.borrow().iter().all(|c| *c == config::CrateType::Rlib); let needs_object_code = self.sess.opts.output_types.should_codegen(); // If we're producing an rlib, then we don't need object code. // Or, if we're not producing object code, then we don't need it either // (e.g., if we're a cdylib but emitting just metadata). if is_rlib || !needs_object_code { flavor == CrateFlavor::Rmeta } else { // we need all flavors (perhaps not true, but what we do for now) true } } </ins> // Attempts to extract *one* library from the set `m`. If the set has no // elements, `None` is returned. If the set has more than one element, then // the errors and notes are emitted about the set of libraries.