Change:
See: Build the first commit (jonas-schievink/rubble then the second (jonas-schievink/rubble using , and you should get a linker error like this: Note that this needs the target, so you might need to run beforehand. This is on . Also reproduces on .
I minimized this to ~60 lines in
looks like it reproduces this in a non-Cargo environment. Some observations from that issue:
pre-triage: P-high; leaving nominated in hopes it increases chance that we discuss it today
discussed at T-compiler meeting. Tentatively assigning to for initial investigation. if you do not expect to be able to do much investigation, let me know (and unassign yourself)
Is this specific to the thumbv7em-none-eabi target?
Not specific to thumbv7em-none-eabi. told me happened when building for x86_64-unknown-linux-gnu. We don't yet have a reliable repro on x86 ("build this commit followed by this commit") and the project in is not using Cargo so minimizing from there may be more complicated.
triage: Reassigning from to self.
spent a little while trying to reproduce this today, and failed. Of note: even just trying to do atop , I am getting I tried to work around this (by adding my own fake definition for ), but that just led linking problems for the basic case: (I assume there's some cross-compilation system library support that I'm missing.)
Okay, I managed to now replicate it atop 's reduction from jonas-schievink/rubble (I was not installing the thumb target correctly before this point; I was probably not doing it with or something to synchronize it with my version.)
Here is a repo with my (further-reduced) reduction: I got the whole thing down to a chain of three crates: a [13-line crate][cortex] at the root, a [65-line crate][rubble] at the leaf, and a .... [1643-line crate][hal] in the middle... Unfortunately the nature of the bug makes it quite difficult to remove any of those 1643 lines "Fortunately" the majority of the lines (over 1000 of them) are . Which is ... interesting... [cortex]: [rubble]: [hal]: It still requires the thumbv7em-none-eabi target. At this point I think I've gotten it at small as I can with my limited knowledge of embedded rust. So now I'll switch to trying to understand what's going on with the code-generation here.
(for my narrowed test case, passing does seem to make the problem go away, which could be interpreted as contradicting the last bullet in an . but given the heisenbug nature of this issue, that could just be masking over the problem rather than showing something fundamental here...)
Embedded Rust is pretty much regular Rust sans magic done by the operating system and libc in a regular application. The reduced 1.6k middle crate is just an abstraction over fixed bits at fixed memory addresses to talk to built-in peripherals (data busses, timers, I/O pins, radios...) generated from a chip manufacturer provided register description. Having multiple s on the same in the same module sounds quite wrong, is that even allowed? I could imagine that this confuses the heck out of the name mangling.
Its absolutely allowed. See:
Yes, I have learned that last weekend in Barcelona and makes sense to me. Should have updated my comment to not waste your time, sorry about that. It's still kind of weird that Rust accepts multiple identical s. That sounds like something which should possibly be at least a lint.
Current status: using and , I have been inspecting the contents of the distinct codegen units that generates. As a reminder, the undefined symbol is: , (referenced by in ). So searching for in the disassembled bitcode files () seemed like a reasonable step to take. From there, I discovered that has a defintion for the aforementioned symbol: but is missing that definition. So I think that means the definition is getting eliminated by llvm optimization passes. One thing that stood out to me here: The function definition in the llvm bitcode says . But this definition is referenced by another codegen unit. Surely that should not be ? (Maybe this is why the LLVM optimization passes are able to justify removing it?) Still, seems odd that this only crops up during this combination of factors (namely, the fact that incremental compilation seems to be involved...) Ah: The first clean run, that didn't have incremental build products, does not mark the function definition as ; that definition is merely marked
Well, multiple identical non-trivial blocks is impossible: You get "duplicate definition" errors. And multiple non-identical blocks is useful, e.g. for code-grouping. As mentioned in a very recent tweet: <blockquote class="twitter-tweet"<p lang="en" dir="ltr"rust documentation code layout idea: group functions together in separate impl blocks, even if the blocks have the same trait bounds or are in the same file<br<brthen apply doc comments to the impl blocks that rustdoc can show to create &quot;sections&quot;</p&mdash; a quiet misdreavus ( <a href=""November 11, 2019</a</blockquoteSo I think the only case here that one might want to lint for is trivial (i.e. empty) blocks. (But even there I would want to exercise care to not lint cases where all the items in the block were removed due to attributes.)
More data, after adding some instrumentation: The module has , which tries to mark symbols internal that are only referenced by the current codegen unit. The first (i.e. clean) run does not mark as internal, because it has these two accessors: , The second run with incremental build produces and an empty definition does mark as internal, while noting that it has this one accessor: The logic of the relevant code is here: (The debug print outs above do not include the codegen unit placements for the accessors, so I do not actually know for sure how much sense all of this makes. But it certainly makes sense that the aforementioned change to would cause it to be removed from the list, so there is no smoking gun here yet...)
More data after adding instrumentation of the codegen units involved: The codegen unit that contains the unresolved reference is named . It has a single fn item (one of the fns, specifically ), and that fn item accesses . Note that (which belongs to codegen unit ) has previously been found to be an accessor of (whcih belongs to the same codegen unit). The latter item is the item that being marked as Internal and subsequently has its definition optimized away by LLVM. In short: it looks like we have a call chain of methods: A calls B calls C, A is in one code-gen unit, B and C are in another, and we make the decision to mark internal (since it is only reachable from B, in the same codegen-unit), but independently, the compiler (presumably LLVM) has inlined the body of B into A, and so we now have a call from A to an internal method (C) in a distinct codegen-unit. Is there some sort of inlining happening here that implies we need to do a deeper traversal of the accessor-chain before we commit to marking a reachable item as internal? From looking at the pre- and post-optimized versions of , I am thinking that this seems like strong possibility. Another thing to note: it seems to me like the inlining in question is happening for both the first (clean) build and also for the the second (reusing) build. (Is it possible that the second build is reusing the object code for the codegen unit from the first build? Is that somehow relevant, or a red herring?) In particular: Using the simplified A,B,C model I summarized above, I would have imagined that LLVM could not inline the body of B into A since it is making a call to the internal function C. But maybe the issue is that we are reusing the results of a previous compile where it was legal to inline B into A (because during that compile, C was not marked internal), and independently we are now deciding to recategorize C as internal. Note however that I also would have thought that LLVM could not inline the body of B into A because they belong to distinct codegen units. So there are multiple points of personal confusion here for me to resolve in my theories above. Further update: it seems like the inlining of B into A is happening during link-time optimization (specifically ThinLTO), which does seem like it would jibe with the rules for link-time optimization...
Okay, further investigation: itself controls whether we can reuse the post-ThinLTO version of a module (or if we have to go back and reoptimize it). And, thank goodness, we already had debug instrumentation in place to emit information about such reuse: In other words: This lends credence to my theory that we are reusing an artifact from a previous incremental compile that should have been invalidated by the changes to what symbols were marked internal by The code that dictates whether a post-ThinLTO object file for a codegen unit can be reused is here: I'm going to read over this more carefully and see if I can figure out how to make it aware of the impact of ....
and I are continuing to investigate this. () My most as to what the bug is (transcribed here): Specifically: is relying on this: which reflects the whatever the current LTO imports are computed, which may differ from the set of imports that were used to drive the original LTO optimization that we are reusing. In particular, they may differ by being a subset of the original set of imports, which means we can overlook hidden dependencies.

Discussion:
use rustc::middle::exported_symbols::SymbolExportLevel; use rustc::session::config::{self, Lto}; use rustc::util::common::time_ext; <del> use rustc_data_structures::fx::FxHashMap; </del> <ins> use rustc_data_structures::fx::{FxHashSet, FxHashMap}; </ins> use rustc_codegen_ssa::{RLIB_BYTECODE_EXTENSION, ModuleCodegen, ModuleKind}; use log::{info, debug}; use std::ffi::{CStr, CString}; <ins> use std::fs::File; use std::io; use std::mem; use std::path::Path; </ins> use std::ptr; use std::slice; use std::sync::Arc; <ins> /// We keep track of past LTO imports that were used to produce the current set /// of compiled object files that we might choose to reuse during this /// compilation session. pub const THIN_LTO_IMPORTS_INCR_COMP_FILE_NAME: &str = "thin-lto-past-imports.bin"; </ins> pub fn crate_type_allows_lto(crate_type: config::CrateType) -> bool { match crate_type { config::CrateType::Executable |

info!("thin LTO data created"); <del> let import_map = if cgcx.incr_comp_session_dir.is_some() { ThinLTOImports::from_thin_lto_data(data) </del> <ins> let (import_map_path, prev_import_map, curr_import_map) = if let Some(ref incr_comp_session_dir) = cgcx.incr_comp_session_dir { let path = incr_comp_session_dir.join(THIN_LTO_IMPORTS_INCR_COMP_FILE_NAME); // If previous imports have been deleted, or we get an IO error // reading the file storing them, then we'll just use `None` as the // prev_import_map, which will force the code to be recompiled. let prev = if path.exists() { ThinLTOImports::load_from_file(&path).ok() } else { None }; let curr = ThinLTOImports::from_thin_lto_data(data); (Some(path), prev, curr) </ins> } else { // If we don't compile incrementally, we don't need to load the // import data from LLVM. assert!(green_modules.is_empty()); <del> ThinLTOImports::default() </del> <ins> let curr = ThinLTOImports::default(); (None, None, curr) </ins> }; info!("thin LTO import map loaded");

for (module_index, module_name) in shared.module_names.iter().enumerate() { let module_name = module_name_to_str(module_name); <del> // If the module hasn't changed and none of the modules it imports // from has changed, we can re-use the post-ThinLTO version of the // module. if green_modules.contains_key(module_name) { let imports_all_green = import_map.modules_imported_by(module_name) </del> <ins> // If (1.) the module hasn't changed, and (2.) none of the modules // it imports from has changed, *and* (3.) the import-set itself has // not changed from the previous compile when it was last // ThinLTO'ed, then we can re-use the post-ThinLTO version of the // module. Otherwise, freshly perform LTO optimization. // // This strategy means we can always save the computed imports as // canon: when we reuse the post-ThinLTO version, condition (3.) // ensures that the curent import set is the same as the previous // one. (And of course, when we don't reuse the post-ThinLTO // version, the current import set *is* the correct one, since we // are doing the ThinLTO in this current compilation cycle.) // // See rust-lang/rust#59535. if let (Some(prev_import_map), true) = (prev_import_map.as_ref(), green_modules.contains_key(module_name)) { assert!(cgcx.incr_comp_session_dir.is_some()); let prev_imports = prev_import_map.modules_imported_by(module_name); let curr_imports = curr_import_map.modules_imported_by(module_name); let imports_all_green = curr_imports </ins> .iter() .all(|imported_module| green_modules.contains_key(imported_module)); <del> if imports_all_green { </del> <ins> if imports_all_green && equivalent_as_sets(prev_imports, curr_imports) { </ins> let work_product = green_modules[module_name].clone(); copy_jobs.push(work_product); info!(" - {}: re-used", module_name); <ins> assert!(cgcx.incr_comp_session_dir.is_some()); </ins> cgcx.cgu_reuse_tracker.set_actual_reuse(module_name, CguReuse::PostLto); continue

})); } <ins> // Save the curent ThinLTO import information for the next compilation // session, overwriting the previous serialized imports (if any). if let Some(path) = import_map_path { if let Err(err) = curr_import_map.save_to_file(&path) { let msg = format!("Error while writing ThinLTO import data: {}", err); return Err(write::llvm_err(&diag_handler, &msg)); } } </ins> Ok((opt_jobs, copy_jobs)) } } <ins> /// Given two slices, each with no repeat elements. returns true if and only if /// the two slices have the same contents when considered as sets (i.e. when /// element order is disregarded). fn equivalent_as_sets(a: &[String], b: &[String]) -> bool { // cheap path: unequal lengths means cannot possibly be set equivalent. if a.len() != b.len() { return false; } // fast path: before building new things, check if inputs are equivalent as is. if a == b { return true; } // slow path: general set comparison. let a: FxHashSet<&str> = a.iter().map(|s| s.as_str()).collect(); let b: FxHashSet<&str> = b.iter().map(|s| s.as_str()).collect(); a == b } </ins> pub(crate) fn run_pass_manager(cgcx: &CodegenContext<LlvmCodegenBackend>, module: &ModuleCodegen<ModuleLlvm>, config: &ModuleConfig,

self.imports.get(llvm_module_name).map(|v| &v[..]).unwrap_or(&[]) } <ins> fn save_to_file(&self, path: &Path) -> io::Result<()> { use std::io::Write; let file = File::create(path)?; let mut writer = io::BufWriter::new(file); for (importing_module_name, imported_modules) in &self.imports { writeln!(writer, "{}", importing_module_name)?; for imported_module in imported_modules { writeln!(writer, " {}", imported_module)?; } writeln!(writer)?; } Ok(()) } fn load_from_file(path: &Path) -> io::Result<ThinLTOImports> { use std::io::BufRead; let mut imports = FxHashMap::default(); let mut current_module = None; let mut current_imports = vec![]; let file = File::open(path)?; for line in io::BufReader::new(file).lines() { let line = line?; if line.is_empty() { let importing_module = current_module .take() .expect("Importing module not set"); imports.insert(importing_module, mem::replace(&mut current_imports, vec![])); } else if line.starts_with(" ") { // Space marks an imported module assert_ne!(current_module, None); current_imports.push(line.trim().to_string()); } else { // Otherwise, beginning of a new module (must be start or follow empty line) assert_eq!(current_module, None); current_module = Some(line.trim().to_string()); } } Ok(ThinLTOImports { imports }) } </ins> /// Loads the ThinLTO import map from ThinLTOData. unsafe fn from_thin_lto_data(data: *const llvm::ThinLTOData) -> ThinLTOImports { unsafe extern "C" fn imported_module_callback(payload: *mut libc::c_void,

<ins> // revisions: cfail1 cfail2 // compile-flags: -O -Zhuman-readable-cgu-names -Cllvm-args=-import-instr-limit=10 // build-pass // rust-lang/rust#59535: // // This is analgous to cgu_invalidated_when_import_removed.rs, but it covers // the other direction: // // We start with a call-graph like `[A] -> [B -> D] [C]` (where the letters are // functions and the modules are enclosed in `[]`), and add a new call `D <- C`, // yielding the new call-graph: `[A] -> [B -> D] <- [C]` // // The effect of this is that the compiler previously classfied `D` as internal // and the import-set of `[A]` to be just `B`. But after adding the `D <- C` call, // `D` is no longer classified as internal, and the import-set of `[A]` becomes // both `B` and `D`. // // We check this case because an early proposed pull request included an // assertion that the import-sets monotonically decreased over time, a claim // which this test case proves to be false. fn main() { foo::foo(); bar::baz(); } mod foo { // In cfail1, ThinLTO decides that foo() does not get inlined into main, and // instead bar() gets inlined into foo(). // In cfail2, foo() gets inlined into main. pub fn foo(){ bar() } // This function needs to be big so that it does not get inlined by ThinLTO // but *does* get inlined into foo() when it is declared `internal` in // cfail1 (alone). pub fn bar(){ println!("quux1"); println!("quux2"); println!("quux3"); println!("quux4"); println!("quux5"); println!("quux6"); println!("quux7"); println!("quux8"); println!("quux9"); } } mod bar { #[inline(never)] pub fn baz() { #[cfg(cfail2)] { crate::foo::bar(); } } } </ins>

<ins> // revisions: cfail1 cfail2 // compile-flags: -O -Zhuman-readable-cgu-names -Cllvm-args=-import-instr-limit=10 // build-pass // rust-lang/rust#59535: // // Consider a call-graph like `[A] -> [B -> D] <- [C]` (where the letters are // functions and the modules are enclosed in `[]`) // // In our specific instance, the earlier compilations were inlining the call // to`B` into `A`; thus `A` ended up with a external reference to the symbol `D` // in its object code, to be resolved at subsequent link time. The LTO import // information provided by LLVM for those runs reflected that information: it // explicitly says during those runs, `B` definition and `D` declaration were // imported into `[A]`. // // The change between incremental builds was that the call `D <- C` was removed. // // That change, coupled with other decisions within `rustc`, made the compiler // decide to make `D` an internal symbol (since it was no longer accessed from // other codegen units, this makes sense locally). And then the definition of // `D` was inlined into `B` and `D` itself was eliminated entirely. // // The current LTO import information reported that `B` alone is imported into // `[A]` for the *current compilation*. So when the Rust compiler surveyed the // dependence graph, it determined that nothing `[A]` imports changed since the // last build (and `[A]` itself has not changed either), so it chooses to reuse // the object code generated during the previous compilation. // // But that previous object code has an unresolved reference to `D`, and that // causes a link time failure! fn main() { foo::foo(); bar::baz(); } mod foo { // In cfail1, foo() gets inlined into main. // In cfail2, ThinLTO decides that foo() does not get inlined into main, and // instead bar() gets inlined into foo(). But faulty logic in our incr. // ThinLTO implementation thought that `main()` is unchanged and thus reused // the object file still containing a call to the now non-existant bar(). pub fn foo(){ bar() } // This function needs to be big so that it does not get inlined by ThinLTO // but *does* get inlined into foo() once it is declared `internal` in // cfail2. pub fn bar(){ println!("quux1"); println!("quux2"); println!("quux3"); println!("quux4"); println!("quux5"); println!("quux6"); println!("quux7"); println!("quux8"); println!("quux9"); } } mod bar { #[inline(never)] pub fn baz() { #[cfg(cfail1)] { crate::foo::bar(); } } } </ins>