jiebi/CodeConvo huggingface dataset:
Code change:
cargo.arg("--release"); } <del> if mode != Mode::Libstd && // FIXME(#45320) mode != Mode::Libtest && // FIXME(#45511) self.config.rust_codegen_units.is_none() && </del> <ins> if self.config.rust_codegen_units.is_none() && </ins> self.build.is_rust_llvm(compiler.host) { cargo.env("RUSTC_THINLTO", "1");

Related discussion:
text: nightly-2017-10-24 contains a libtest with bad DWARF: I can reproduce this if I recompile libtest locally: The problem can be fixed by disabling thinlto (and keeping the codegen units is ok). I noticed this with nightly-2017-10-24 because it is the first nightly compiled with thinlto, but the problem also occurs if I compile libtest with thinlto enabled using earlier compilers (eg nightly-2017-10-17). Inspecting the DWARF, a DIE at that local ref offset does exist in another codegen unit, so the problem seems to be that we should be using a global ref instead.
cc'ing since he's been poking around in LLVM's DWARF generation code recently.
Sorta smells like an LLVM bug -- but hard to say. Who should investigate?
triage: P-medium if you are able to investigate as part of ThinLTO work would be great.
It's likely that this is the cause of , which even if the underlying segfault were fixed would likely prevent backtraces from happening on OSX.
This is unfortunately not "automagically fixed" if we upgrade to LLVM 5
Making progress. First up, minimized Rust source code via creduce to get: Next up let's see if we can reduce it to one object file: Ok looks like only one object file is invalid! Let's take a look the LLVM IR: Ok let's see what's happening by minimizing the LLVM IR: yielding which I . I checked out LLVM trunk, , and looks like the error is still on LLVM trunk.
I'm not sure at this point if this is an LLVM bug, a rustc bug, or a ThinLTO bug. I've managed to extract what I believe is a for this which reproduces the error. I'm hoping fixing that IR will fix all other related issues to this at this point in time. For those of you more familiar with debug information (probably everyone other than me!) mind taking a look at that IR? Does it look "obviously wrong"? If it's "obviously wrong" I can try to work backwards to figure out what piece of the pipeline introduced such a wrong piece, but if it looks correct I can open an LLVM bug saying that we think it may be broken (and maybe they'll say it's "obviously wrong"!)
Code change:
cargo.arg("--release"); } <del> if mode != Mode::Libstd && // FIXME(#45320) mode != Mode::Libtest && // FIXME(#45511) self.config.rust_codegen_units.is_none() && </del> <ins> if self.config.rust_codegen_units.is_none() && </ins> self.build.is_rust_llvm(compiler.host) { cargo.env("RUSTC_THINLTO", "1");

Related discussion:
text: cc
Er, I went ahead and to go ahead and broaden the search.
The DWARF is definitely badly formed: it's using a unit-relative offset to reference the parameter type, when it should be using a section-relative offset because the type is in a different unit. The bad offset is occurring for a parameter to subprogram in . The problem is that subprograms and share the same , but have different units ( and ). Based on some print debugging, it appears that LLVM doesn't handle this case. So and are constructed in unit , but when is constructed, it reuses the existing DIE as its parent, and so it also ends up in unit . So my guess is that when the parameter is constructed, it uses the unit that was specified to determine if the parameter and type are in the same unit, and thus gets it wrong.
thanks for the info! I know basically nothing about DWARF though and malformed whatnots, so the IR here is heavily edited by both machines (creduce/bugpoint) and me (to make it more readable). My assumption is that if LLVM accepts IR and generates an object file it shouldn't yield such a dwarfdump error, but are you thinking that given possibly valid LLVM IR the dwarfdump error is possible? If so that may make reduction far more difficult... Or do you think that this looks like upstream LLVM needs to get updated?
I don't think the editing of the IR caused the problem. I'll check later, but I think it's quite likely that the same problem occurs for the unedited IR. I don't know if the LLVM developers intend for this to be valid IR, or if they are missing a validation check somewhere. I think they may have never encountered this possibility. clang divides code into units based on files, so I don't think it ever splits code in the same scope across multiple units, whereas I think rust splits code into units quite differently. That's a bit of conjecture on my part though, it might be better to ask someone who knows more (I only know about DWARF, not LLVM or rustc). Reading the LLVM code a bit more, I think it will be possible to fix there by stopping it from sharing the scope (the scope creation is initiated and the sharing check is ). It's also worth reconsidering how rust splits code into units though.
Code change:
cargo.arg("--release"); } <del> if mode != Mode::Libstd && // FIXME(#45320) mode != Mode::Libtest && // FIXME(#45511) self.config.rust_codegen_units.is_none() && </del> <ins> if self.config.rust_codegen_units.is_none() && </ins> self.build.is_rust_llvm(compiler.host) { cargo.env("RUSTC_THINLTO", "1");

Related discussion:
text: It might make sense to ensure all code for a given type is kept in the same unit.
Ah ok that all makes sense. Note though that rustc is very much like clang when it comes to debuginfo/codegen units. The major difference here is that LLVM is doing inlining across codegen units via ThinLTO, and that's likely where these new situations are arising from.
I see, yep my understanding of how rust partitions units was wrong. So when using ThinLTO, the additional units by ThinLTO only contain information needed for the additional inlined functions? Do these inlined functions always share scopes/types with unit , or can we determine when they do? If so, it'd be nice to add them to unit instead of creating additonal units. I think this would get us DWARF that looks the same as that generated for . eg dwarfdump is happy if I run:
interesting! I'd have to look into the internals of ThinLTO to know for sure though. It seems like LLVM isn't ready (maybe?) for multiple codegen units in debug info in one LLVM module (a codegen unit), but it makes sense to me what you're thinking about having all inlined debug info share the same codegen unit information. Now where that change would go in LLVM I'm not entirely sure...
Does firefox use fat LTO? There's been a couple of bugs that have some similarities to this ( and maybe ). Also note that absence of dwarfdump errors doesn't mean the offsets are all okay. dwarfdump is only complaining if the offset is larger than the unit length, but even if it's less it may still incorrectly be a local ref. So there may be more instances that are unnoticed.
cc ^
I can reproduce using normal LTO. Library crate is: Binary crate is: Also here's a thin LTO version that gives smaller IR than the previous Vec test case:
Thanks It definitely sounds like at this point this is an existing bug we've just never hit before. My inclination would be to paper over this if we can. Unfortunately though I don't know how to fix this at the LLVM IR layer. My naive idea would be to just point everything at one codegen unit for each LLVM module (debug-info-wise) but I don't know how to change existing debuginfo of a module.
Code change:
cargo.arg("--release"); } <del> if mode != Mode::Libstd && // FIXME(#45320) mode != Mode::Libtest && // FIXME(#45511) self.config.rust_codegen_units.is_none() && </del> <ins> if self.config.rust_codegen_units.is_none() && </ins> self.build.is_rust_llvm(compiler.host) { cargo.env("RUSTC_THINLTO", "1");

Related discussion:
text: I'm not able to access currently, so not sure exactly what Adrian's suggestions were, but here's a that fixes LLVM to use a global ref when needed. It's not as simple as comparing parent DIEs, because the variable is initially , so the check . I don't have a lot of confidence in the patch, but maybe it's a useful starting point for someone who knows more about LLVM.
awesome thanks! I think LLVM's bugzilla is back now so indicating such.
I've confirmed that applying your patch to LLVM fixes having malformed debug information. Next up I'm confirming that it
Sure enough it fixes the issue! would you be interested in sending the patch upstream?
I prefer if someone who knows more about LLVM could upstream the patch.
Would this be something you could do, You can evaluate the changes to the generated DWARF best, I think.
We do use for Firefox builds, yes.
Sure, I will do this. I think the issue is more an LLVM thing than a DWARF thing, though; not certain I know enough there to say whether the patch is good, but I will at least do the shepherding. Also I wanted to point out that knows quite a bit about DWARF; he's one of the principals on
Actually maybe there's nothing to do here except merge this into rust-llvm?
Yes, I think so. I'll do that.
Cool, thanks!
oh no worries! I've tested it locally now and will submit a PR soon (with another change or two)
Ok, thanks
I've opened up which I think should close this out (and also re-enables ThinLTO in a few locations)
Hi, I get a Dwarf Error with GDB 8.0.1 on one of the files of Rust 1.23.0 that was built was Rust 1.23.0 on Fedora 27, in a build enabled with debuginfo. Is it related to this issue or should I open a new one? Relevant build flags:
yeah I believe that's this issue, it should be fixed in 1.24 though!
Code change:
} impl Attribute { <ins> /// Returns `true` if the attribute's path matches the argument. If it matches, then the /// attribute is marked as used. /// /// To check the attribute name without marking it used, use the `path` field directly. </ins> pub fn check_name(&self, name: &str) -> bool { let matches = self.path == name; if matches {

Related discussion:
text: Look for example to It is deprecated, and in source code there is hint what should be used instead: but in generated html documentation there is no hint about
isn't the correct syntax for the attribute. It should be . You can see it work: . This isn't a rustdoc issue. It's an issue with rustc that it doesn't reject .
Code change:
#![feature(const_fn)] #![feature(const_fn_union)] #![feature(const_generics)] <ins> #![cfg_attr(not(bootstrap), feature(const_ptr_offset_from))] #![cfg_attr(not(bootstrap), feature(const_type_name))] </ins> #![feature(custom_inner_attributes)] #![feature(decl_macro)] #![feature(doc_cfg)]

Related discussion:
text: As shown in const-qualification can ignore some s, in the context of libcore / rustc crates: This should not compile without "miri unleashed"; const-qualification does not see this function as const since the unstable feature is not enabled, and thus does not check it. It outside of libcore, nor . cc More discussion is also available . I have a fix and will post a PR shortly. There are a couple of existing cases in where this matters (for and ), and I'll fix those at the same time.
Code change:
fn visit_foreign_item(&mut self, fi: &'tcx hir::ForeignItem) { if self.should_warn_about_foreign_item(fi) { <del> self.warn_dead_code(fi.id, fi.span, fi.name, fi.node.descriptive_variant()); </del> <ins> self.warn_dead_code(fi.id, fi.span, fi.name, fi.node.descriptive_variant(), "used"); </ins> } intravisit::walk_foreign_item(self, fi); } fn visit_struct_field(&mut self, field: &'tcx hir::StructField) { if self.should_warn_about_field(&field) { <del> self.warn_dead_code(field.id, field.span, field.name, "field"); </del> <ins> self.warn_dead_code(field.id, field.span, field.name, "field", "used"); </ins> } intravisit::walk_struct_field(self, field); }

Related discussion:
text: I was changing some code from integer constants to a proper enum and was greeted by the following warning: The warning sounded weird to me because I was pattern-matching on that damn variant in various places of the code; but I was pointed out to the fact that I wasn't actually creating that value. Maybe the warning should be rewritten to explicitly mention value creation? Cc
cc that looks like your thing
duplicate of
Yup! Closing in favor of that.
Code change:
} _ => { let tt = TokenTree::Token(self.token.take()); <del> let is_joint = self.bump(); </del> <ins> let mut is_joint = self.bump(); if !self.token.is_op() { is_joint = NonJoint; } </ins> Ok((tt, is_joint)) } }

Related discussion:
text: Between nightly-2020-09-05 and nightly-2020-09-06, libproc_macro stopped preserving the jointness of a trailing joint punctuation token in a TokenStream (likely ; This breaks some real-world code such as the following, though only with old versions of some libraries as far as I have found. I don't know which behavior above is correct but I would like it to be intentionally decided so that we know what behavior to implement in proc-macro2. fails to compile since nightly-2020-09-06:
This has also broken 's webrender-wrench, which relied on this behavior in one of the dependencies. -- do you think a revert makes sense perhaps? Is there a patch expected soon-ish?
Marking this as P-critical based on the
Code change:
version = "0.6.0" dependencies = [ "clippy_lints", <del> "env_logger 0.6.2", </del> <ins> "env_logger 0.7.0", </ins> "failure", "futures", "log", <del> "rand 0.6.1", </del> <ins> "rand 0.7.0", </ins> "rls-data", "rls-ipc", "serde",

Related discussion:
text: Hello, this is your friendly neighborhood mergebot. After merging PR rust-lang/rust, I observed that the tool rls no longer builds. A follow-up PR to the repository is needed to fix the fallout. cc do you think you would have time to do the follow-up work? If so, that would be great! cc the PR reviewer, and nominating for compiler team prioritization.
Code change:
use std::env; use std::ffi::OsString; use std::fmt::Write as _; <del> use std::fs; </del> <ins> use std::fs::{self, File}; </ins> use std::io::{self, IsTerminal, Read, Write}; <del> use std::panic::{self, catch_unwind}; </del> <ins> use std::panic::{self, catch_unwind, PanicInfo}; </ins> use std::path::PathBuf; use std::process::{self, Command, Stdio}; use std::str;

Related discussion:
text: slightly increased default binary size (checked on x8664-pc-windows-msvc), the reason is now code unconditionally includes File::open (~ 2kb out of 130) and friends. can be tracked back by new imports of , for where is Originally posted by in
cc
cc -- I'm surprised that didn't include any libs review in the first place, having modified the default panic hook. Maybe ought to be registering its own panic hook instead?
Size changed back to 123kb.
Code change:
/// } /// ``` #[stable(feature = "io_error_inner", since = "1.3.0")] <ins> #[inline] </ins> pub fn get_mut(&mut self) -> Option<&mut (dyn error::Error + Send + Sync + 'static)> { match self.repr { Repr::Os(..) => None,

Related discussion:
text: I am facing a problem while reading different kinds of binary formats. If you are going to read primitives like too often, then performance degrades. This is mainly due to the fact that returns box. Also it does boxing TWICE! doc link: I wrote simple bench to show pref problem. Use to get it We can clearly see that performance is degrading. The more calls it takes to read, the worse the results will be. The first very popular crate that comes to mind is . It mainly depends on the data format. But what if you need to receive data over the network? The attacker may abuse wrong data to make ton of pointless memory allocations. stdlib is also use often enough. Let's run a simple bash to make sure of this In many cases, you will see that it is used . stdlib sometimes create with macro, however it doesn't happen very often. Consider adding a new static method such as and new internal member of that's enum. How about using something like , but with static lifetime? : Update 07.03.2021: Benchmarks have been improved as per the comment. This should show the problem better.
The "std_err" bench is still pretty slow though
Yes, I probably made a mistake while copying the code for the benchmark of . It should be . However, this shows that not zero-cost.
This is fixed by
Turns out it was three times, if you also count the that the is copied into.
Running your benchmark (thanks!): Before: After: Looks like it isn't completely solved yet. Adding to now... :)
That fixed it.
Code change:
} } <ins> /// If a tool does not pass its tests, don't ship it. /// Right now, we do this only for Miri. fn check_toolstate(&mut self) { let toolstates: Option<HashMap<String, String>> = File::open(self.input.join("toolstates-linux.json")).ok() .and_then(|f| serde_json::from_reader(&f).ok()); let toolstates = toolstates.unwrap_or_else(|| { println!("WARNING: `toolstates-linux.json` missing/malformed;  assuming all tools failed"); HashMap::default() // Use empty map if anything went wrong. }); // Mark some tools as missing based on toolstate. if toolstates.get("miri").map(|s| &*s as &str) != Some("test-pass") { println!("Miri tests are not passing, removing component"); self.miri_version = None; self.miri_git_commit_hash = None; } } </ins> /// Hash all files, compute their signatures, and collect the hashes in `self.digests`. fn digest_and_sign(&mut self) { for file in t!(self.input.read_dir()).map(|e| t!(e).path()) {

Related discussion:
text: Looking at https://rust-, it seems like Miri is getting shipped as a rustup component even though the tests fail. Unfortunately, Miri is currently entirely usable, so it'd probably be better if it did not get shipped in this state. Is that something we can do? Cc
maybe we can make the miri toolstate always go to even if it technically is just
what would be the best place to change this? I think I'd prefer Miri not to be shipped at all when the tests fail; I'd rather people stick to an old-but-working Miri than to a new-but-broken one. I found but that does not seem to decide if a component gets built for distribution or not.
seems to suggest that it is supposed to work as you describe that it should be working. Since it obviously doesn't and I can't see how that comment could be true from looking at the source, I'm guessing this is some refactoring fallout. I'll dig a little into git blame
I found it. The comment was for rls in but then the code making the comment true was removed later
See for why this is hard to fix: the builders don't even run the tool tests, so they cannot know if the tests pass. :/ Also, that's probably for the better actually, as the "tools" builder sets --cfg miri to be more Miri-friendly and we don't want that in the release artifacts.
One (crazy?) thing we could so is have the dist builders query the toolstate repo to figure out if the tests are passing...
That will either retrieve an outdated toolstate or will need to block until the toolstate is uploaded ...
Without blocking it would lag behind by 1 merged PR, yes.
would it be possible to add some kind of build assert so Miri fails on builders when Rust internals change too much?
Well builders already know when Miri fails to build because internals change too much. The issue is things that make Miri still compile but break the test suite.
would it be possible to at least change the logic that opens issues such that for Miri, it opens an issue like when the tests fail? I could also try to implement that if you point me to the right place. So far we have had zero cases of Miri tests spuriously failing.
Code change:
} } <ins> /// If a tool does not pass its tests, don't ship it. /// Right now, we do this only for Miri. fn check_toolstate(&mut self) { let toolstates: Option<HashMap<String, String>> = File::open(self.input.join("toolstates-linux.json")).ok() .and_then(|f| serde_json::from_reader(&f).ok()); let toolstates = toolstates.unwrap_or_else(|| { println!("WARNING: `toolstates-linux.json` missing/malformed;  assuming all tools failed"); HashMap::default() // Use empty map if anything went wrong. }); // Mark some tools as missing based on toolstate. if toolstates.get("miri").map(|s| &*s as &str) != Some("test-pass") { println!("Miri tests are not passing, removing component"); self.miri_version = None; self.miri_git_commit_hash = None; } } </ins> /// Hash all files, compute their signatures, and collect the hashes in `self.digests`. fn digest_and_sign(&mut self) { for file in t!(self.input.read_dir()).map(|e| t!(e).path()) {

Related discussion:
text: That should be easy to do:
Gave it a shot at
had an interesting idea: can't we do the "don't ship broken tools" as part of a post-processing step? We anyway do some little amount of work when a commit hits (as opposed to ), such as actually updating the toolstate repo. So maybe that could also either delete the broken components from the S3 bucket, or else generate a file to be put into the S3 bucket that says which of the components are really there so that the broken ones can be considered "not there"? do you think something like that would be possible?
Thinking about this again, probably the most natural place to do this is in the . That could "just" also check the toolstate database for the commit it is promoting, and then (possibly only for certain tools) not add them to the nightly if their toolstate is not green.
This is biting us again right now, where CI knows that Miri is broken but we ship the component anyway, leading to CI failures in downstream projects that have Miri in their CI.
For another case where this issue got quite painful, see
My PR is because I was incorrectly assuming that all commits have their toolstate recorded in the toolstate repo. Reopening.
FTR, for testing , I downloaded some artifacts: then I edited my : and finally I ran This produces stuff in .
So looks like we have logic in that specifically does not commit to the toolstate repo if nothing changed: Unfortunately, this means that we cannot easily fetch the toolstate of the to-be-produced release in build-manifest, which caused my to fail. Would it be okay for you to change this logic to record the toolstate of all commits in the toolstate repo? Or would you prefer if I try to find another approach for solving this issue? Right now I am somewhat out of ideas though if we cannot even use the toolstate repo to communicate this information.
Wouldn't it be simpler to save list of "good" components to something like and upload it alongside tarballs? Running would show what you need.
I guess we could upload the two JSON files from the two tool runners somehow? Not sure which would be the right places to hook for that, though.
Status: blocked on which is blocked on
This seems to be finally fixed!
Code change:
} } <ins> /// If a tool does not pass its tests, don't ship it. /// Right now, we do this only for Miri. fn check_toolstate(&mut self) { let toolstates: Option<HashMap<String, String>> = File::open(self.input.join("toolstates-linux.json")).ok() .and_then(|f| serde_json::from_reader(&f).ok()); let toolstates = toolstates.unwrap_or_else(|| { println!("WARNING: `toolstates-linux.json` missing/malformed;  assuming all tools failed"); HashMap::default() // Use empty map if anything went wrong. }); // Mark some tools as missing based on toolstate. if toolstates.get("miri").map(|s| &*s as &str) != Some("test-pass") { println!("Miri tests are not passing, removing component"); self.miri_version = None; self.miri_git_commit_hash = None; } } </ins> /// Hash all files, compute their signatures, and collect the hashes in `self.digests`. fn digest_and_sign(&mut self) { for file in t!(self.input.read_dir()).map(|e| t!(e).path()) {

Related discussion:
text: Today nightly was shipped with , and everything looks normal :tada:
Miri is "test-fail" now, so with the next nightly we should be able to see if indeed it gets removed from rustup as it should.
Confirmed it's working! On miri is missing.