Change:
<!-- Thank you for filing a bug report! Please provide a short summary of the bug, along with any information you feel relevant to replicating the bug. --Disclaimer: I tried to create a testcase from scratch, but for some reason I wasn't able to find a way to trigger the use of instead of . STR: Clone - Actual output: Expected output: Something like: This doesn't happen without LTO. The unfortunate consequence is that when the resulting static library is linked with C or C++ code compiled with clang with (that latter flag is now default in clang trunk), which also uses /, ODR violation detection kick in complaining about globals defined multiple times, because both the clang-side asan constructor and the rust asan constructor register all the globals. Normally, what happens is that they both use the same (thus it normally being ), and set its value, so that only one constructor registers the globals. With the LTOed staticlib, what happens is that there are two distinct , so both constructors go through. : (Edit: fixed typos, changed the peculiar setup with and to the more normal LTO, which shows the problem too)
is tracking this for Chromium as it causes our asan bots to fail.
Here is the code which creates the global: It's interesting that it uses linkage ( as you said). Common linkage should work, but to me it is a surprising choice. I would've expected this global to use linkage and be marked , which is what you would get for a C++17 inline global. I don't know how Rust is producing LTOed static libraries, but maybe somewhere along the way there is a bug in how LTO is handling common linkage globals. One possible summary of this issue is that the ODR violation detector is suffering from an ODR violation, we have two flags when we should have one.
I like this take :)
I consider myself quite familiar with asan, LTO, Clang Driver, but I know very little about Rust. gives me this command (after removing ): Most defined symbols are localized. I do not know why the 4 symbols are special and rustc lto doesn't localize them. I don't know how to rerun the rustc with a locally built ( with containing ). I suspect that preventing from being localized will fix this bug.
is consider bad nowadays but the COMMON symbol use case is fine. It is like a COMDAT group containing just a variable. COMMON is more size efficient than using a COMDAT group (there is a size overhead due to a 64 byte header). If we use a COMDAT for but rustc still localizes the symbol, then we'd still have this bug.
Let me try to help move this bug forward by attempting to answer some of the questions above. I have limited experience with linkers, asan, and compilers, so please shout if there are any mistakes below. RE: I don't know how to rerun the rustc with a locally built ( build) I tweaked the original cmdline (from the first comment/report on this bug) by adding and adding (and other than that I've rerun the original repro steps + debugging steps from your earlier comment at ): (Note that this changes the path where the build artifacts are - e.g. instead of ) RE: I do not know why the 4 symbols are special and rustc lto doesn't localize them. I am not sure if these are the reasons, but this is what I've found for some of the symbols emitted by : I see that which is marked asI see that and is declared as which AFAIU tells to export the function.FWIW, I also see that special-cases "rustehpersonality" (see ), but changing this to cover names with "asan" substring didn't seem to have an effect on the output) RE: I suspect that preventing asanglobalsregistered from being localized will fix this bug. Can you please elaborate on that? From your , it seems that you are saying that a change in is needed - did I get that right? OTOH, it seems that comes from outside of sources - it comes from  and therefore it seems that maybe we need to change how it is marked for linking by LLVM? Interestingly the comment there mentions "a local symbol" - not sure if this is relevant:
/cc who AFAICT the comment above in
/cc fyi. Although this isn't fully understood and it may well be that the fix will be on the rust side, this is a rustc/clang interop issue that's new with the upcoming clang 17 release, so perhaps there should at least be a "known issues" release note.
cc
Probably the symbol needs to be to the symbol export list, see existing handling for llvmprofile and msan symbols in
- thanks for the pointer! After adding the following to : I am now seeing in the output of (which I assume means that the problem is fixed based on ): Repro steps for completeness: I'll try to put together a PR with the changes above. I think one of the first steps is to figure out if/how to add tests for this. I'll try to see if I can cargo cult something from either (Add codegen test that makes sure PGO instrumentation is emitted as expected), (librustctrans: Mark some profiler symbols as exported to avoid LTO removing them), (Mark msankeepgoing as an exported symbol for LTO), or (Mark msantrackorigins as an exported symbol for LTO by - thanks again for the code pointer!), FWIW, I think it would be nice to consolidate the code from the 3 similar cases: , , and . I plan to have a separate commit that refactors gathering all symbol names into a single vector and then uses a single/shared . Finally, I am a bit surprised that needs to be aware of symbol names used in a far-away LLVM land where AFAIU ASAN and MSAN are implemented. This arrangement seems fragile. OTOH, it seems that this does fix the issue at hand, so it seems like a reasonable way to proceed.
This is because Rust assumes that all code provided to (non-plugin) LTO comes from Rust, so it knows about all symbols that are involved. This doesn't hold up for symbols that get injected by LLVM, so they need to be special-cased.
TL;DR: I would appreciate help with creating a regression test for this. I have trouble creating a regression test for this. Based on , FileCheck tests under run with and it seems that exactly the same LLVM IR is present in before and after the changes from In other words, the following test (say, in ) passes before and after the change: AFAICT the test above replicates the following cmdline flags from , . It doesn't replicate , but I haven't pursued this direction further, because necessitates and no longer includes in the LLVM IR. The presence or absence of doesn't seem to change the result here. I notice that (and my repro steps in ) reported seeing rather than . So maybe my changes only have effect on stages after LLVM-IR generation? I dunno... Not sure if this hypothesis makes sense...
When a module doesn't exports any symbols, returns an empty string and instrumentation no longer uses : You can export an arbitrary symbol. For example mark as and then use .
- thanks! I have a working test now :-).
I have a patch lying around to try to simplify this logic, but I haven't prioritized it because this issue is still outstanding and I don't want to complicate matters while we look for a solution.
I may need some extra help with the tests please. The latest version of the PR fails on (see ) and it seems that the FileCheck's input file (i.e. LLVM-IR) doesn't contain any mentions of . I am not sure how to handle the test failure above. One (hacky?) way would be to skip the test on some platforms. FWIW I see that there are no existing tests under Rust repo with as part of a FileCheck expectation (i.e. the test from the PR would be the first such test), but there are some tests under and some of them say things like .OTOH, maybe the test (or the product change) in the PR is not sufficiently versatile and doesn't quite work on some platforms?
since the instrumentation strategy is target specific, limiting exercised targets seems appropriate. Since we don't have a option to run tests only on ELF targets (which would use code path and ), maybe use header (it will be sufficient to exercise test on CI).
Thanks I've (plus an explanatory comment) to the new test. Feel free to ask bors to commit if this LGTY. I note that, since we are saying that is ELF-specific, then maybe we should consider only applying the product code changes to ELF targets. Still, maybe recognizing the symbol on other targets won't hurt (and it definitely simplifies the code). FWIW, I see that  has some -dependent behavior, but I am not sure if this should also be used in the PR.
The latest test failure (see ) on is: This seems to indicate that the product code changes should not unconditionally expect the symbols whenever ASAN is used, but instead the presence of the symbol should depend on the target architecture. This is a little bit tricky - I see that refers to from: So, based on the above should maybe be modified to only recognize for ELF and MachO targets.
I don't know how to detect ELF or MachO targets based on (this would be required if we wanted to pursue the direction outlined at the end of ). There is (), but it's unclear to me how to map this into ELF-or-MachO decision. This hurdle also reinforces the feeling that I've expressed in it feels fragile that we have to keep aware of how instrumentation symbols are used in a far-away LLVM land. I would hope that the information about these symbols can be communicated by LLVM (instead of having to duplicate/replicate this information in sources). OTOH, I have very little experience working with linkers, so I don't know if a viable alternative exists. and/or - could you please comment on the above? Do we really want/need to replicate the knowledge from the (far-away land of) that the symbol exists on ELF and MachO targets and that should handle it with specific ?
RE: Rust assumes that all code provided to (non-plugin) LTO comes from Rust Would it be possible for to detect symbols injected by LLVM (like , , or ) more generically (without knowing about specific symbol names)? If there was a way to see the list of all symbols (in ? in ?) then maybe these LLVM-origin symbols could be detected by special-casing their prefix (, , , or ) rather than their exact names? I tried staring for a while at and I don't currently see a way to go from its input (i.e. ) to the list of all symbols (including ones injected by LLVM). FWIW, I also don't understand how symbols injected by LLVM (e.g. ) present themselves at the API level (I guess they are not "items" in sense and don't have a ).
With an understanding that we want to work together and find a practical solution, let me first represent the perspective of the instrumentation tool writer. From the PoV of one writing instrumentation passes, it is surprising that Rust assumes that all symbols originate from Rust, and that instrumentation passes cannot add symbols and rely on the standard linkage rules. From the ASan PoV, all we are doing is inventing a common symbol and assuming that linking will proceed as normal, meaning the common symbol will be deduplicated, and there will be one per DSO. I would argue that this Rust pass is breaking fairly reasonable assumptions. But I'm more interested in practical solutions, so yes, I think using those prefixes would be a good place to start. I'm not sure that list is complete, however. PGO, for examples, creates lots of symbols using a prefix. I see the check for already exists in I could imagine that sanitizers may start moving some shared sanitizer functionality from the prefix space to something like , and there's some fragility there, but that's hypothetical. There is also the sanitizer coverage mode, which uses . We could go the direction of having LLVM instrumentation passes annotate symbols somehow, but it would be difficult to enforce that as a requirement for instrumentation passes. Anyone writing a new instrumentation pass is likely to discover the annotation requirement when they first try to instrument Rust code, which doesn't reduce the cost of this issue, it just distributes the fixes from to each instrumentation pass. You may be able to put together some heuristic by looking at global variables with weak linkages (common, linkonceodr, weak, weak_odr, etc). If Rust never or rarely produces weak symbols, you could teach the exporter to leave those alone. That would be the simplest.
For the staticlib and cdylib crate types we need to hide all symbols except those marked with to prevent conflicts between multiple copies of rust crates (like the standard library itself) that are statically linked into them.[^1] We are using the same list of symbols to be exported for LTO as these symbols definitively need to be exported and not exporting the rest as far as LTO is concerned will allow for the most optimizations. The staticlib and cdylib crate types use a closed world model: All rust code is contained in them and only an explicitly specified C interface is exported. simply extends the C interface it exports. [^1]: For staticlibs we currently don't do this as there is no way to do this without editing the object files themself it seems. (hidden visibility wouldn't work as applying hidden visibility to the standard library would break dynamic linking) This currently makes it to be impossible to link two rust staticlibs into the same executable except when using a linker which allows multiple identical definitions of a symbol and even then multiple versions of a crate would break.
AFAICT the initial fix attempt at influences whether from contains in the that get returned as the first tuple item. are then passed to in and used in . So, let me try a different fix approach and recognize ASAN/MSAN/etc-related symbols in the predicate (rather than in as in the original fix attempt). See:
I guess one could argue about pushing the knowledge about , , etc. prefixes even further down - e.g. into the implementation of in LLVM. and/or - any strong opinions on continuing with the PR at VS authoring an LLVM PR instead?
Putting this list of symbols in LLVM sounds practical to me. Is it possible to emulate what Rust is doing using ? If so, we could make a small regression test out of this, and put it in the asan test suite.
Hmmm... I now think that it still makes sense to review and attempt to land the PR at even if we (eventually) preserve these ASAN symbols via a separate LLVM PR: At least part of the PR at is desirable in the long-term (i.e. even once LLVM changes are made). Specifically, we want to land the changes that make unaware of , , , names (i.e. keep the removal of code in ). It seems desirable for to work correctly even if built with older LLVM versions We can consider opening a follow-up issue against the llvm-project (and maybe adding a TODO in the PR?) RE: Is it possible to emulate what Rust is doing using ? Maybe. I don't know how to check. Sorry. FWIW, I see that LLVM-level tests that use take LLVM-IR as input. This probably means that would have to be hardcoded/simulated in the test input (rather than generated by the real ASAN). Maybe this is ok. RE: Putting this list of symbols in LLVM sounds practical to me. The source file has quite elaborate code for preserving some symbols. I wonder if ASAN / MSAN / etc symbols should be protected by any of the existing mechanisms: Would we consider adding , , etc. into (e.g. ) Or maybe the already-existing should kick-in (but for some reason doesn't work for ASAN/MSAN/etc)? Or maybe the already existing checks in  should kick-in (but for some readon don't work for ASAN/MSAN/etc). Or maybe these checks need to be extended in a generic way (rather than teaching about and/or prefixes).
Status update / summary (I edited this comment on 2023-08-25 at 10:47 PST to add one other potential next step at the very end of the comment): Impact of the bug (sorry for only listing the impact on Chromium, but this is what I am personally aware of): Blocks further Chromium adoption of which is enabled on all platforms except ChromeOS (where this bug is the only known blocker). Triggered "temporary" disabling of ODR violation detection on Fuchsia (see ) Reproability / testability status: We know how to detect the problem caused by this bug: At LLVM-IR level ( vs ). This translates nicely into a new test. At level (see ). This was checked manually before and after the fix in We don't have a minimized end-to-end repro (e.g. invocations + maybe invocations =incorrect ASAN report about an ODR violation) In I am tentatively suggesting that loading dynamic libraries/plugins might be a required component of the repro has trouble in with creating a testcase from scratch I think that both and suggest that mixing C++ and Rust is a required component of the repro We considered two fix options: Option 1 (rejected for now; see PR at ): Tweaking This mimics , , and This doesn't work as well here, because it requires replicating LLVM's checks to only expect for ELF and MachO targets (see , , and ) Option 2 (currently being pursued; see PR at ): Removing ASAN/MSAN/PGO/LTO-related code from and instead controlling LTO internalization in via lambda (using symbol name prefixes like instead of specific names like ). In the code review adding ThinLTO tests In we concluded that ThinLTO cannot be tested via If the repro indeed requires mixing C++ and Rust toolchains (see the bullet above about a minimized end-to-end repro) then it seems that (, ) cannot provide test coverage (AFAIU can only invoke `rustc). It may be possible to invoke and via , but this requires actually having an end-to-end repro. I spent some time trying to create a repro yesterday but failed. I note that the previous fixes (for MSAN) were not required to add ThinLTO-related test coverage. The new test at mimics and covers similar scenarios as (already existing) and I note that does indeed call only from . In other words, Option 2 stops special-handling , , , and during ThinLTO. OTOH, the problem with these symbols was internalization which happens during fat LTO, but I am not sure if happens during ThinLTO (definitely not the cross-module kind). Can we discuss the next steps?: Can we discuss whether ThinLTO tests are a hard requirement to land the fix? Maybe we can land the fix at in its current shape (with a LLVM-IR-level regression test covering ASAN + no LTO, and ASAN + fat LTO). If ThinLTO or end-to-end tests are required, then can somebody please help with creating a minimal repro? It seems that either or might be good role models. (Let me ask in if anyone else from Chromium might be able to help). Maybe we should also look if the fix can be made in LLVM instead (as suggested in ). It seems that the next step (pointed out by in ) would be figuring out if that bug at hand can be reproted/simulated using (whatever that means in the world of LLVM tests :-)).
in you've suggested that there might be issue with Option 2 of the fix. Let me partially reply here, because I have some questions that are not related to Option 2. Do you think we should proceed with (or is there another approach that you'd suggest)? If so, then I assume that you agree that to avoid we should only inject in when the symbol is actually present (i.e. when and/or inject the symbol). Could you please help me understand how to tweak to do this? How can detect ELF and/or MachO targets? Do you think it would be okay to copy-and-paste ? What would you suggest for detecting MachO targets? (I don't see code for detecting MachO targets outside of  which I assume can't be used outside of cranelist.)
I think so, but I may be misunderstanding what exactly address sanitizer needs in terms of linkage. Like this: Maybe extracting this into a function to ensure it is kept in sync makes sense.
Thanks! That helps :-). Not sure how I managed to miss this piece of code when greepping the source code for ... :-/ FWIW I've applied the changes suggested above to the newly pushed version of (e.g. see )

Discussion:
extern "C" void LLVMRustRunRestrictionPass(LLVMModuleRef M, char **Symbols, size_t Len) { auto PreserveFunctions = [=](const GlobalValue &GV) { <ins> // Preserve LLVM-injected, ASAN-related symbols. // See also https://github.com/rust-lang/rust/issues/113404. if (GV.getName() == "___asan_globals_registered") { return true; } // Preserve symbols exported from Rust modules. </ins> for (size_t I = 0; I < Len; I++) { if (GV.getName() == Symbols[I]) { return true;

<ins> // Verifies that AddressSanitizer symbols show up as expected in LLVM IR with `-Zsanitizer`. // This is a regression test for https://github.com/rust-lang/rust/issues/113404 // // Notes about the `compile-flags` below: // // * The original issue only reproed with LTO - this is why this angle has //   extra test coverage via different `revisions` // * To observe the failure/repro at LLVM-IR level we need to use `staticlib` //   which necessitates `-C prefer-dynamic=false` - without the latter flag, //   we would have run into "cannot prefer dynamic linking when performing LTO". // // The test is restricted to `only-linux`, because the sanitizer-related instrumentation is target // specific.  In particular, `___asan_globals_registered` is only used in the // `InstrumentGlobalsELF` and `InstrumentGlobalsMachO` code paths.  The `only-linux` filter is // narrower than really needed (i.e. narrower than ELF-or-MachO), but this seems ok - having a // linux-only regression test should be sufficient here. // // needs-sanitizer-address // only-linux // // revisions:ASAN ASAN-FAT-LTO //[ASAN]          compile-flags: -Zsanitizer=address //[ASAN-FAT-LTO]  compile-flags: -Zsanitizer=address -Cprefer-dynamic=false -Clto=fat #![crate_type="staticlib"] // The test below mimics `CACHED_POW10` from `library/core/src/num/flt2dec/strategy/grisu.rs` which // (because of incorrect handling of `___asan_globals_registered` during LTO) was incorrectly // reported as an ODR violation in https://crbug.com/1459233#c1.  Before this bug was fixed, // `___asan_globals_registered` would show up as `internal global i64` rather than `common hidden // global i64`.  (The test expectations ignore the exact type because on `arm-android` the type // is `i32` rather than `i64`.) // // CHECK: @___asan_globals_registered = common hidden global // CHECK: @__start_asan_globals = extern_weak hidden global // CHECK: @__stop_asan_globals = extern_weak hidden global #[no_mangle] pub static CACHED_POW10: [(u64, i16, i16); 4] = [ (0xe61acf033d1a45df, -1087, -308), (0xab70fe17c79ac6ca, -1060, -300), (0xff77b1fcbebcdc4f, -1034, -292), (0xbe5691ef416bd60c, -1007, -284), ]; </ins>