Change:
for (module_index, module_name) in shared.module_names.iter().enumerate() { let module_name = module_name_to_str(module_name); <del> // If the module hasn't changed and none of the modules it imports // from has changed, we can re-use the post-ThinLTO version of the // module. if green_modules.contains_key(module_name) { let imports_all_green = import_map.modules_imported_by(module_name) </del> <ins> // If (1.) the module hasn't changed, and (2.) none of the modules // it imports from has changed, *and* (3.) the import-set itself has // not changed from the previous compile when it was last // ThinLTO'ed, then we can re-use the post-ThinLTO version of the // module. Otherwise, freshly perform LTO optimization. // // This strategy means we can always save the computed imports as // canon: when we reuse the post-ThinLTO version, condition (3.) // ensures that the curent import set is the same as the previous // one. (And of course, when we don't reuse the post-ThinLTO // version, the current import set *is* the correct one, since we // are doing the ThinLTO in this current compilation cycle.) // // See rust-lang/rust#59535. if let (Some(prev_import_map), true) = (prev_import_map.as_ref(), green_modules.contains_key(module_name)) { assert!(cgcx.incr_comp_session_dir.is_some()); let prev_imports = prev_import_map.modules_imported_by(module_name); let curr_imports = curr_import_map.modules_imported_by(module_name); let imports_all_green = curr_imports </ins> .iter() .all(|imported_module| green_modules.contains_key(imported_module)); <del> if imports_all_green { </del> <ins> if imports_all_green && equivalent_as_sets(prev_imports, curr_imports) { </ins> let work_product = green_modules[module_name].clone(); copy_jobs.push(work_product); info!(" - {}: re-used", module_name); <ins> assert!(cgcx.incr_comp_session_dir.is_some()); </ins> cgcx.cgu_reuse_tracker.set_actual_reuse(module_name, CguReuse::PostLto); continue

Discussion:
See: Build the first commit (jonas-schievink/rubble then the second (jonas-schievink/rubble using , and you should get a linker error like this: Note that this needs the target, so you might need to run beforehand. This is on . Also reproduces on .
I minimized this to ~60 lines in
looks like it reproduces this in a non-Cargo environment. Some observations from that issue:
pre-triage: P-high; leaving nominated in hopes it increases chance that we discuss it today
discussed at T-compiler meeting. Tentatively assigning to for initial investigation. if you do not expect to be able to do much investigation, let me know (and unassign yourself)
Is this specific to the thumbv7em-none-eabi target?
Not specific to thumbv7em-none-eabi. told me happened when building for x86_64-unknown-linux-gnu. We don't yet have a reliable repro on x86 ("build this commit followed by this commit") and the project in is not using Cargo so minimizing from there may be more complicated.
triage: Reassigning from to self.
spent a little while trying to reproduce this today, and failed. Of note: even just trying to do atop , I am getting I tried to work around this (by adding my own fake definition for ), but that just led linking problems for the basic case: (I assume there's some cross-compilation system library support that I'm missing.)
Okay, I managed to now replicate it atop 's reduction from jonas-schievink/rubble (I was not installing the thumb target correctly before this point; I was probably not doing it with or something to synchronize it with my version.)
Here is a repo with my (further-reduced) reduction: I got the whole thing down to a chain of three crates: a [13-line crate][cortex] at the root, a [65-line crate][rubble] at the leaf, and a .... [1643-line crate][hal] in the middle... Unfortunately the nature of the bug makes it quite difficult to remove any of those 1643 lines "Fortunately" the majority of the lines (over 1000 of them) are . Which is ... interesting... [cortex]: [rubble]: [hal]: It still requires the thumbv7em-none-eabi target.

At this point I think I've gotten it at small as I can with my limited knowledge of embedded rust. So now I'll switch to trying to understand what's going on with the code-generation here.
(for my narrowed test case, passing does seem to make the problem go away, which could be interpreted as contradicting the last bullet in an . but given the heisenbug nature of this issue, that could just be masking over the problem rather than showing something fundamental here...)
Embedded Rust is pretty much regular Rust sans magic done by the operating system and libc in a regular application. The reduced 1.6k middle crate is just an abstraction over fixed bits at fixed memory addresses to talk to built-in peripherals (data busses, timers, I/O pins, radios...) generated from a chip manufacturer provided register description. Having multiple s on the same in the same module sounds quite wrong, is that even allowed? I could imagine that this confuses the heck out of the name mangling.
Its absolutely allowed. See:
Yes, I have learned that last weekend in Barcelona and makes sense to me. Should have updated my comment to not waste your time, sorry about that. It's still kind of weird that Rust accepts multiple identical s. That sounds like something which should possibly be at least a lint.
Current status: using and , I have been inspecting the contents of the distinct codegen units that generates. As a reminder, the undefined symbol is: , (referenced by in ). So searching for in the disassembled bitcode files () seemed like a reasonable step to take. From there, I discovered that has a defintion for the aforementioned symbol: but is missing that definition. So I think that means the definition is getting eliminated by llvm optimization passes. One thing that stood out to me here: The function definition in the llvm bitcode says . But this definition is referenced by another codegen unit. Surely that should not be ? (Maybe this is why the LLVM optimization passes are able to justify removing it?)

Still, seems odd that this only crops up during this combination of factors (namely, the fact that incremental compilation seems to be involved...) Ah: The first clean run, that didn't have incremental build products, does not mark the function definition as ; that definition is merely marked
Well, multiple identical non-trivial blocks is impossible: You get "duplicate definition" errors. And multiple non-identical blocks is useful, e.g. for code-grouping. As mentioned in a very recent tweet: <blockquote class="twitter-tweet"<p lang="en" dir="ltr"rust documentation code layout idea: group functions together in separate impl blocks, even if the blocks have the same trait bounds or are in the same file<br<brthen apply doc comments to the impl blocks that rustdoc can show to create &quot;sections&quot;</p&mdash; a quiet misdreavus ( <a href=""November 11, 2019</a</blockquoteSo I think the only case here that one might want to lint for is trivial (i.e. empty) blocks. (But even there I would want to exercise care to not lint cases where all the items in the block were removed due to attributes.)
More data, after adding some instrumentation: The module has , which tries to mark symbols internal that are only referenced by the current codegen unit. The first (i.e. clean) run does not mark as internal, because it has these two accessors: , The second run with incremental build produces and an empty definition does mark as internal, while noting that it has this one accessor: The logic of the relevant code is here: (The debug print outs above do not include the codegen unit placements for the accessors, so I do not actually know for sure how much sense all of this makes. But it certainly makes sense that the aforementioned change to would cause it to be removed from the list, so there is no smoking gun here yet...)
More data after adding instrumentation of the codegen units involved: The codegen unit that contains the unresolved reference is named . It has a single fn item (one of the fns, specifically ), and that fn item accesses . Note that (which belongs to codegen unit ) has previously been found to be an accessor of (whcih belongs to the same codegen unit).

The latter item is the item that being marked as Internal and subsequently has its definition optimized away by LLVM. In short: it looks like we have a call chain of methods: A calls B calls C, A is in one code-gen unit, B and C are in another, and we make the decision to mark internal (since it is only reachable from B, in the same codegen-unit), but independently, the compiler (presumably LLVM) has inlined the body of B into A, and so we now have a call from A to an internal method (C) in a distinct codegen-unit. Is there some sort of inlining happening here that implies we need to do a deeper traversal of the accessor-chain before we commit to marking a reachable item as internal? From looking at the pre- and post-optimized versions of , I am thinking that this seems like strong possibility. Another thing to note: it seems to me like the inlining in question is happening for both the first (clean) build and also for the the second (reusing) build. (Is it possible that the second build is reusing the object code for the codegen unit from the first build? Is that somehow relevant, or a red herring?) In particular: Using the simplified A,B,C model I summarized above, I would have imagined that LLVM could not inline the body of B into A since it is making a call to the internal function C. But maybe the issue is that we are reusing the results of a previous compile where it was legal to inline B into A (because during that compile, C was not marked internal), and independently we are now deciding to recategorize C as internal. Note however that I also would have thought that LLVM could not inline the body of B into A because they belong to distinct codegen units. So there are multiple points of personal confusion here for me to resolve in my theories above. Further update: it seems like the inlining of B into A is happening during link-time optimization (specifically ThinLTO), which does seem like it would jibe with the rules for link-time optimization...
Okay, further investigation: itself controls whether we can reuse the post-ThinLTO version of a module (or if we have to go back and reoptimize it).

And, thank goodness, we already had debug instrumentation in place to emit information about such reuse: In other words: This lends credence to my theory that we are reusing an artifact from a previous incremental compile that should have been invalidated by the changes to what symbols were marked internal by The code that dictates whether a post-ThinLTO object file for a codegen unit can be reused is here: I'm going to read over this more carefully and see if I can figure out how to make it aware of the impact of ....
and I are continuing to investigate this. () My most as to what the bug is (transcribed here): Specifically: is relying on this: which reflects the whatever the current LTO imports are computed, which may differ from the set of imports that were used to drive the original LTO optimization that we are reusing. In particular, they may differ by being a subset of the original set of imports, which means we can overlook hidden dependencies.