jiebi/CodeConvo huggingface dataset:
In investigating I was generating some source code, and using the shell script listed in we can see: That's a lot of time to expand a crate that doesn't have any macros in it!
cc you may be interested in this!
A looks like almost all the time is spent in In which case, cc
Apparently most time is spent
called ~38k times
Since we need to build the module graph and resolve imports during expansion for macros 2.0, we must construct node ids and def ids throughout expansion (i.e. interleaved arbitrarily many times until we hit fix-point). Ideally we'd keep running totals of the time spent assigning node ids, assigning def ids, etc. for more accurate profiling information within the expansion step.
in theory though isn't the fixed point for this crate the crate itself? (e.g. there's no macros here). Or are the def ids persisted for the rest of compilation, so "blaming" expansion isn't quite correct here?
Yeah, the def ids are persisted for the rest of compilation. We could move the "first round" of def id collection (in which we collect the raw, unexpanded source) outside of expansion, but that would introduce more code complexity and wouldn't count the time we spend collecting def ids for AST that comes from macro expansions.
Ah ok, nah it seems fine! I think there's a number of cases where is attributing time in the wrong place nowadays... Still it seems odd that 30-50k invocations of this function would take so long to execute?
Do I understand this correctly that all DefIds generated are actually used, right? There are no "temporary" DefIds? Each of these invocations creates a Blake2 hasher. I'd be interested how the performance looks with other hashing algorithms. I'd like to switch to something faster anyway:
Are you sure this isn't the other quadratic case fixed in ?
It may very well be! Feel free to close this w/ that PR landing, I'll verify to be sure but I'd imagine you're correct in that it's fixed by
er, didn't mean to close just yet
A of Servo's crate shows a huge amount of time (46 seconds!) in name resolution. It turns out that is calling , specifically . That's a lot of unused imports! Sure enough if we create a synthetic file locally with this script: we find: That's a lot of time spent processing unused imports! The slowdown here isn't quite the same as the style crate, unfortunately. The style crate only has ~7k unused imports, which apparently takes it 48s in that profile. In any case the intent here is clear, lots of unused imports seem to slow down compilation, and the crate looks to have a lot of unused imports!
cc
We could address this quickly by just emitting the first 10 unused imports warnings and then warning that there are more unused imports not shown.
I can never deal with more than 10 lints at a time anyway myself, so seems like a reasonable solution to me!
Dumping out a ton of warnings is useful when you're hooked into an IDE and you can see them as e.g. green squiggles under the offending text as opposed to 10k lines on a terminal.
I think the reason is slow is because it calls , which does this: I don't think we want to read source files multiple times like that.
Thank you for looking into Servo compile times Alex, and great find here! Could you CC me or someone from the Servo team for Servo-related issues? Thanks.
Can do!
In general I've always wanted the lint system to avoid even running in blocks. Currently the lint system runs all lints in parallel on the tree. This is great since it means we only traverse it once, and the cost of the traversal is consolidated. We don't need to reduce that cost itself (which is nice because then we don't have to worry about inside ). Most lints do not maintain state. The lint framework has the capability to do so, and we use this a couple times in Clippy, but most lints are , so they won't notice if or whatever isn't called in a region of . The "allowness" of a lint is tied to where in the tree the lint traversal is, not the span emitted by the lint.
It's perfectly possible to a block of code and have the lint still be emit there because a different block of code (where it is ) hit the lint but emitted a span it got from elsewhere. We've had trouble with this before in clipyp iirc. All of this enables this proposal: If we had a specialized trait for this, then when we have an we can turn off calling lint methods entirely, and when there's a turn it on again. It doesn't even have to be a trait, really, it could just be a defaulted fake-static method (until happens) on LateLintPass. I suspect this will get significant speedups for crates like Style. I may_ be able to take a crack at this if y'all think this is a good approach.
It occurs to me that a boolean check per lint each tree node might actually end up being more expensive? I'll have look at this code closer. Might be possible to make this fast by having a second array of trait objects which only contains the active lint passes.
It seems like we do a large enough amount of work that the whole "vector of active lints" stuff won't really be necessary. However, I just realized that this bug is not during the lint pass but during the resolution phase (which defers linting to the actual lint pass). This won't be able to help, though we can fix it by doing the spantosnippet in the actual lint and then making this optimization. So in that case I suspect this optimization is unnecessary. In general we've rarely had performance issues with lints in clippy (which has way more lints than rustc), and we've been pretty careful to measure impact there. Still, if folks think this optimization can help, let me know.
We need some way to interpret a as UTF-8, using the replacement character for invalid sequences instead of conditions. This would ideally be provided in the form of an .
Could we remove the current condition API in favour of this? It doesn't really seem to offer anything but failing, and we already have an API returning an .
I would be quite pleased if we could do that. Using the condition is rather awkward, and I would rather just have replacement chars myself.
This is a duplicate of , no?
And also the e-mail thread here I think is relevant:
This is an implementation of a mostly complete → UTF-8 decoder
It's only a duplicate if the intended resolution of is to use the replacement char instead of conditions, but it's unclear from the discussion if that's the plan.
I guess my thinking was that a maximally expressive condition-based solution would let one express a replacement char approach using conditions. But I don't actually favor doing that at this point. Or at least, I don't favor making that the only way to accomplish this, since I suspect a more specialized approach will be much nicer to use (both in terms of programmer convenience and in terms of efficiency). So okay, this is not a duplicate of .
Rust needs to support both fatal and replacement error handling modes, it should follow the WHATWG spec simonsapin has said (edit by is WHATWG spec.)
the fatal mode is already handled by the API
well, the current functions in str don't handle decoding buffers in chunks like the proposed encodings API. But yes, it does handle one-off decoding with 'fatal' error handling.
I think we need to improve/extend our API along the lines suggested here for 1.0. Nominating for P-backcompat-lang. (Arguably we could avoid the backwards compatibility hazard by offering a fail-only method for 1.0 and then adding an alternative entry point that provides the more flexible API supporting replacement characters in post 1.0. But I think this case is important enough that we should try to get the primary API method to provide both choices up front. That, or put dynamic fluid support in for representing the state of that choice.)
I submitted a PR for this last night, although I forgot there was an open issue too so I didn't link them. The PR is
I tried to create a minimum example from an issue we've seen in our project (). I tried running: On the following crate: : I expected to see this happen: The command should succeed. Instead, this happened: It crashed with an ICE <!-- If you're using the stable version of the compiler, you should also check if the bug also exists in the beta or nightly versions. --: <!-- Include a backtrace in the code block by setting in your environment. E.g. . --<details<summaryBacktrace</summary<p</p</details
This issue may be related to , however I can only reproduce it on the nightly build. On the stable channel, rustdoc succeeds.
cc
does this reproduce on beta?
I know I'm not celinval, but I've been able to reduce this to a test case, and yeah, it reproduces on beta.
That was quick. Thank you!
Reopening to track beta breakage.
I was silly and nominated even though the PR has already been nominated, and that's really the thing that's supposed to be. Please ignore, and sorry for the noise.
nightly-2017-10-24 contains a libtest with bad DWARF: I can reproduce this if I recompile libtest locally: The problem can be fixed by disabling thinlto (and keeping the codegen units is ok). I noticed this with nightly-2017-10-24 because it is the first nightly compiled with thinlto, but the problem also occurs if I compile libtest with thinlto enabled using earlier compilers (eg nightly-2017-10-17). Inspecting the DWARF, a DIE at that local ref offset does exist in another codegen unit, so the problem seems to be that we should be using a global ref instead.
cc'ing since he's been poking around in LLVM's DWARF generation code recently.
Sorta smells like an LLVM bug -- but hard to say. Who should investigate?
triage: P-medium if you are able to investigate as part of ThinLTO work would be great.
It's likely that this is the cause of , which even if the underlying segfault were fixed would likely prevent backtraces from happening on OSX.
This is unfortunately not "automagically fixed" if we upgrade to LLVM 5
Making progress. First up, minimized Rust source code via creduce to get: Next up let's see if we can reduce it to one object file: Ok looks like only one object file is invalid! Let's take a look the LLVM IR: Ok let's see what's happening by minimizing the LLVM IR: yielding which I . I checked out LLVM trunk, , and looks like the error is still on LLVM trunk.
I'm not sure at this point if this is an LLVM bug, a rustc bug, or a ThinLTO bug. I've managed to extract what I believe is a for this which reproduces the error. I'm hoping fixing that IR will fix all other related issues to this at this point in time. For those of you more familiar with debug information (probably everyone other than me!) mind taking a look at that IR? Does it look "obviously wrong"? If it's "obviously wrong" I can try to work backwards to figure out what piece of the pipeline introduced such a wrong piece, but if it looks correct I can open an LLVM bug saying that we think it may be broken (and maybe they'll say it's "obviously wrong"!)
cc
Er, I went ahead and to go ahead and broaden the search.
The DWARF is definitely badly formed: it's using a unit-relative offset to reference the parameter type, when it should be using a section-relative offset because the type is in a different unit. The bad offset is occurring for a parameter to subprogram in . The problem is that subprograms and share the same , but have different units ( and ). Based on some print debugging, it appears that LLVM doesn't handle this case. So and are constructed in unit , but when is constructed, it reuses the existing DIE as its parent, and so it also ends up in unit . So my guess is that when the parameter is constructed, it uses the unit that was specified to determine if the parameter and type are in the same unit, and thus gets it wrong.
thanks for the info! I know basically nothing about DWARF though and malformed whatnots, so the IR here is heavily edited by both machines (creduce/bugpoint) and me (to make it more readable). My assumption is that if LLVM accepts IR and generates an object file it shouldn't yield such a dwarfdump error, but are you thinking that given possibly valid LLVM IR the dwarfdump error is possible? If so that may make reduction far more difficult... Or do you think that this looks like upstream LLVM needs to get updated?
I don't think the editing of the IR caused the problem. I'll check later, but I think it's quite likely that the same problem occurs for the unedited IR. I don't know if the LLVM developers intend for this to be valid IR, or if they are missing a validation check somewhere. I think they may have never encountered this possibility. clang divides code into units based on files, so I don't think it ever splits code in the same scope across multiple units, whereas I think rust splits code into units quite differently. That's a bit of conjecture on my part though, it might be better to ask someone who knows more (I only know about DWARF, not LLVM or rustc). Reading the LLVM code a bit more, I think it will be possible to fix there by stopping it from sharing the scope (the scope creation is initiated and the sharing check is ). It's also worth reconsidering how rust splits code into units though.
It might make sense to ensure all code for a given type is kept in the same unit.
Ah ok that all makes sense. Note though that rustc is very much like clang when it comes to debuginfo/codegen units. The major difference here is that LLVM is doing inlining across codegen units via ThinLTO, and that's likely where these new situations are arising from.
I see, yep my understanding of how rust partitions units was wrong. So when using ThinLTO, the additional units by ThinLTO only contain information needed for the additional inlined functions? Do these inlined functions always share scopes/types with unit , or can we determine when they do? If so, it'd be nice to add them to unit instead of creating additonal units. I think this would get us DWARF that looks the same as that generated for . eg dwarfdump is happy if I run:
interesting! I'd have to look into the internals of ThinLTO to know for sure though. It seems like LLVM isn't ready (maybe?) for multiple codegen units in debug info in one LLVM module (a codegen unit), but it makes sense to me what you're thinking about having all inlined debug info share the same codegen unit information. Now where that change would go in LLVM I'm not entirely sure...
Does firefox use fat LTO? There's been a couple of bugs that have some similarities to this ( and maybe ). Also note that absence of dwarfdump errors doesn't mean the offsets are all okay. dwarfdump is only complaining if the offset is larger than the unit length, but even if it's less it may still incorrectly be a local ref. So there may be more instances that are unnoticed.
cc ^
I can reproduce using normal LTO. Library crate is: Binary crate is: Also here's a thin LTO version that gives smaller IR than the previous Vec test case:
Thanks It definitely sounds like at this point this is an existing bug we've just never hit before. My inclination would be to paper over this if we can. Unfortunately though I don't know how to fix this at the LLVM IR layer. My naive idea would be to just point everything at one codegen unit for each LLVM module (debug-info-wise) but I don't know how to change existing debuginfo of a module.
I'm not able to access currently, so not sure exactly what Adrian's suggestions were, but here's a that fixes LLVM to use a global ref when needed. It's not as simple as comparing parent DIEs, because the variable is initially , so the check . I don't have a lot of confidence in the patch, but maybe it's a useful starting point for someone who knows more about LLVM.
awesome thanks! I think LLVM's bugzilla is back now so indicating such.
I've confirmed that applying your patch to LLVM fixes having malformed debug information. Next up I'm confirming that it
Sure enough it fixes the issue! would you be interested in sending the patch upstream?
I prefer if someone who knows more about LLVM could upstream the patch.
Would this be something you could do, You can evaluate the changes to the generated DWARF best, I think.
We do use for Firefox builds, yes.
Sure, I will do this. I think the issue is more an LLVM thing than a DWARF thing, though; not certain I know enough there to say whether the patch is good, but I will at least do the shepherding. Also I wanted to point out that knows quite a bit about DWARF; he's one of the principals on
Actually maybe there's nothing to do here except merge this into rust-llvm?
Yes, I think so. I'll do that.
Cool, thanks!
oh no worries! I've tested it locally now and will submit a PR soon (with another change or two)
Ok, thanks
I've opened up which I think should close this out (and also re-enables ThinLTO in a few locations)
Hi, I get a Dwarf Error with GDB 8.0.1 on one of the files of Rust 1.23.0 that was built was Rust 1.23.0 on Fedora 27, in a build enabled with debuginfo. Is it related to this issue or should I open a new one? Relevant build flags:
yeah I believe that's this issue, it should be fixed in 1.24 though!
As shown in const-qualification can ignore some s, in the context of libcore / rustc crates: This should not compile without "miri unleashed"; const-qualification does not see this function as const since the unstable feature is not enabled, and thus does not check it. It outside of libcore, nor . cc More discussion is also available . I have a fix and will post a PR shortly. There are a couple of existing cases in where this matters (for and ), and I'll fix those at the same time.
Between nightly-2020-09-05 and nightly-2020-09-06, libproc_macro stopped preserving the jointness of a trailing joint punctuation token in a TokenStream (likely ; This breaks some real-world code such as the following, though only with old versions of some libraries as far as I have found. I don't know which behavior above is correct but I would like it to be intentionally decided so that we know what behavior to implement in proc-macro2. fails to compile since nightly-2020-09-06:
This has also broken 's webrender-wrench, which relied on this behavior in one of the dependencies. -- do you think a revert makes sense perhaps? Is there a patch expected soon-ish?
Marking this as P-critical based on the
I am facing a problem while reading different kinds of binary formats. If you are going to read primitives like too often, then performance degrades. This is mainly due to the fact that returns box. Also it does boxing TWICE! doc link: I wrote simple bench to show pref problem. Use to get it We can clearly see that performance is degrading. The more calls it takes to read, the worse the results will be. The first very popular crate that comes to mind is . It mainly depends on the data format. But what if you need to receive data over the network? The attacker may abuse wrong data to make ton of pointless memory allocations. stdlib is also use often enough. Let's run a simple bash to make sure of this In many cases, you will see that it is used . stdlib sometimes create with macro, however it doesn't happen very often. Consider adding a new static method such as and new internal member of that's enum. How about using something like , but with static lifetime? : Update 07.03.2021: Benchmarks have been improved as per the comment. This should show the problem better.
The "std_err" bench is still pretty slow though
Yes, I probably made a mistake while copying the code for the benchmark of . It should be . However, this shows that not zero-cost.
This is fixed by
Turns out it was three times, if you also count the that the is copied into.
Running your benchmark (thanks!): Before: After: Looks like it isn't completely solved yet. Adding to now... :)
That fixed it.
Looking at https://rust-, it seems like Miri is getting shipped as a rustup component even though the tests fail. Unfortunately, Miri is currently entirely usable, so it'd probably be better if it did not get shipped in this state. Is that something we can do? Cc
maybe we can make the miri toolstate always go to even if it technically is just
what would be the best place to change this? I think I'd prefer Miri not to be shipped at all when the tests fail; I'd rather people stick to an old-but-working Miri than to a new-but-broken one. I found but that does not seem to decide if a component gets built for distribution or not.
seems to suggest that it is supposed to work as you describe that it should be working. Since it obviously doesn't and I can't see how that comment could be true from looking at the source, I'm guessing this is some refactoring fallout. I'll dig a little into git blame
I found it. The comment was for rls in but then the code making the comment true was removed later
See for why this is hard to fix: the builders don't even run the tool tests, so they cannot know if the tests pass. :/ Also, that's probably for the better actually, as the "tools" builder sets --cfg miri to be more Miri-friendly and we don't want that in the release artifacts.
One (crazy?) thing we could so is have the dist builders query the toolstate repo to figure out if the tests are passing...
That will either retrieve an outdated toolstate or will need to block until the toolstate is uploaded ...
Without blocking it would lag behind by 1 merged PR, yes.
would it be possible to add some kind of build assert so Miri fails on builders when Rust internals change too much?
Well builders already know when Miri fails to build because internals change too much. The issue is things that make Miri still compile but break the test suite.
would it be possible to at least change the logic that opens issues such that for Miri, it opens an issue like when the tests fail? I could also try to implement that if you point me to the right place. So far we have had zero cases of Miri tests spuriously failing.
That should be easy to do:
Gave it a shot at
had an interesting idea: can't we do the "don't ship broken tools" as part of a post-processing step? We anyway do some little amount of work when a commit hits (as opposed to ), such as actually updating the toolstate repo. So maybe that could also either delete the broken components from the S3 bucket, or else generate a file to be put into the S3 bucket that says which of the components are really there so that the broken ones can be considered "not there"? do you think something like that would be possible?
Thinking about this again, probably the most natural place to do this is in the . That could "just" also check the toolstate database for the commit it is promoting, and then (possibly only for certain tools) not add them to the nightly if their toolstate is not green.
This is biting us again right now, where CI knows that Miri is broken but we ship the component anyway, leading to CI failures in downstream projects that have Miri in their CI.
For another case where this issue got quite painful, see
My PR is because I was incorrectly assuming that all commits have their toolstate recorded in the toolstate repo. Reopening.
FTR, for testing , I downloaded some artifacts: then I edited my : and finally I ran This produces stuff in .
So looks like we have logic in that specifically does not commit to the toolstate repo if nothing changed: Unfortunately, this means that we cannot easily fetch the toolstate of the to-be-produced release in build-manifest, which caused my to fail. Would it be okay for you to change this logic to record the toolstate of all commits in the toolstate repo? Or would you prefer if I try to find another approach for solving this issue? Right now I am somewhat out of ideas though if we cannot even use the toolstate repo to communicate this information.
Wouldn't it be simpler to save list of "good" components to something like and upload it alongside tarballs? Running would show what you need.
I guess we could upload the two JSON files from the two tool runners somehow? Not sure which would be the right places to hook for that, though.
Status: blocked on which is blocked on
This seems to be finally fixed!