Change:
<!-- Thank you for finding an Internal Compiler Error! If possible, try to provide a minimal verifiable example. You can read "Rust Bug Minimization Patterns" for how to create smaller examples. -- I'm trying to debug an issue with extensive compile times in diesel. While running the compiler on an example with RUSTFLAGS="-Z self-profile -Z self-profile-events=default,args" I've got the following ICE, which prompt me to report it here. : <!-- Include a backtrace in the code block by setting in your environment. E.g. . --<details<summary<strongBacktrace</strong</summary<p</p</details
This is unfortunate but somewhat known for bigger crates, there's more than a , e.g. or Maybe we should switch to a u64 . There's no wg-self-profile GH team so I'll just cc the team members directly in case they have a preference on how to fix this, and I'd look at doing that.
I've not fully grasped the code yet, but it looks like there's a constant that's significantly less than u32::MAX -- 100 million. Is that perhaps the limit we're actually hitting? It seems a little unlikely to me that we're actually seeing u32::MAX unique(?) string IDs in a compilation that will succeed -- presumably, each of those strings is at least several bytes long, which quickly adds up to several dozen gigabytes of memory usage, right? I guess that makes some sense for very slow compilations.... If bumping to u64 isn't a large memory/performance regression, that seems an OK out here, but I'll also suggest that we could consider just giving up on allocating new strings but not panicking? I'm still a little surprised that we run up against a u32::MAX limit...
I haven't grasped the code either, and don't know yet what this 100 million cutoff between "virtual string ids" and "real string ids" means, or why this addition overflows after adding that value. It could for sure be a different issue than there being so many unique strings. Maybe that 100 million base is somehow multiple times through a different codepath than the expected checked_sub. I'll try to investigate.
Ok that makes more sense: IIUC some of the s (the regular string ids) are addressing a data stream (the ) with a u32, and the overflow would then happen if that stream would get bigger than 4GB worth of strings ?
Yes, that sounds right. Increasing the size of will also increase the size of (which is the main data structure that self-profiling emits) from 24 bytes to 32 bytes. That's probably acceptable. Any optimizations to allow for a bigger string address space while keeping at 4 bytes are probably not worth the additional implementation complexity.
Another way to (temporarily) go at this could be to reduce the size of the strings that rustc records: maybe some of the args' debug output is recorded (and/or some of the impls , looking more like impls) and those outliers push this case over the limit. Maybe we can make these more compact, while retaining the key information (of course, until another bigger crate then overflows this compact output and we have to go to usizes anyways). It could also make these args more readable for people. Here's an example of such a slightly verbose case, from : !
IMO we should probably be forcing a pow2 size, and aligning things to the start of a page, to not end up e.g. straddling cache lines randomly and whatnot - though the impact is probably minimal either way.
I made an attempt at fixing this issue in , and would appreciate feedback!
We merged fix, extending the StringId size to 64 bits. We'll need to update version of measureme before updating the compiler.

Discussion:
[[package]] name = "measureme" <del> version = "10.1.2" </del> <ins> version = "11.0.1" </ins> source = "registry+https://github.com/rust-lang/crates.io-index" <del> checksum = "45e381dcdad44c3c435f8052b08c5c4a1449c48ab56f312345eae12d7a693dbe" </del> <ins> checksum = "dfa4a40f09af7aa6faef38285402a78847d0d72bf8827006cd2a332e1e6e4a8d" </ins> dependencies = [ "log", "memmap2",

bitflags = "2.4.1" itertools = "0.11" libc = "0.2" <del> measureme = "10.0.0" </del> <ins> measureme = "11" </ins> object = { version = "0.32.0", default-features = false, features = ["std", "read"] } rustc-demangle = "0.1.21" rustc_ast = { path = "../rustc_ast" }

itertools = "0.11" jobserver_crate = { version = "0.1.27", package = "jobserver" } libc = "0.2" <del> measureme = "10.0.0" </del> <ins> measureme = "11" </ins> rustc-hash = "1.1.0" rustc-rayon = { version = "0.5.0", optional = true } rustc-rayon-core = { version = "0.5.0", optional = true }

either = "1.5.0" field-offset = "0.3.5" gsgdt = "0.1.2" <del> measureme = "10.0.0" </del> <ins> measureme = "11" </ins> polonius-engine = "0.13.0" rustc-rayon = { version = "0.5.0", optional = true } rustc-rayon-core = { version = "0.5.0", optional = true }

[dependencies] # tidy-alphabetical-start field-offset = "0.3.5" <del> measureme = "10.0.0" </del> <ins> measureme = "11" </ins> rustc-rayon-core = { version = "0.5.0", optional = true } rustc_data_structures = { path = "../rustc_data_structures" } rustc_errors = { path = "../rustc_errors" }

rand = "0.8" smallvec = "1.7" aes = { version = "0.8.3", features = ["hazmat"] } <del> measureme = "10.0.0" </del> <ins> measureme = "11" </ins> ctrlc = "3.2.5" # Copied from `compiler/rustc/Cargo.toml`.