Change:
It seems that memory usage during compilation has increased between 1.60.0 and 1.61.0, enough to trigger an OOM kill on CircleCI when trying to build my project. Running on a (3 vCPUs, 6 GB RAM). Command used: Example run: Same on nightly () The same task runs just fine with Rust 1.60.0 I don't yet have a smaller example to showcase this, nor exact memory measurments. I'm willing to explore this further if you let me know how to best record memory usage.
try reduce parallel job
That ultimately does make it succeed, but lengthens build time quite a bit
FWIW, I'm having this issue too. 1.61.0 seems to use substantially more memory during compilation (specifically, we are running into the issue with ), and it's causing CI builds to time out. I'd be happy to provide more detailed information if there's a good way to measure memory usage. Using 1.60.0 solves the issue.
I think it's a insolvable problem, thus maybe we could at least be aware of memory increase consomption by rustc when we run crater for a new rustc release, with crater we could see rustc used more or less memory since last time, maybe... but in finite memory is always a hard problem, if you don't have enough memory to build your project the only solution is add memory, or reduce concurrent job, having cpu / 2 job for example. I don't think rustc will have enough contributor to guarantee stable memory usage, that a non goal I think. labels: +regression-from-stable-to-stable -regression-untriaged (add label thus I'm not very used to this)
Assigning priority as discussed in the of the Prioritization Working Group. Ping to WG-compiler-performance for possible insights/suggestions to diagnose the source of the regression label -I-prioritize +P-high +wg-compiler-performance
Can you confirm that you have 3 CPUs? In the error message in your log, I see 9 process errors which implies that more than 3 processes were running concurrently. Usually, if Cargo is limited to building 3 jobs in parallel, and one of them fails, it will halt immediately and not try any more. 1.61 changed to use the standard library , which might behaving differently in your setup. There have also been some changes to how it is computed (). I'm not sure what the best way is to check. Perhaps you can compile a small program and run it to test: Or if you want to get cargo to tell you directly, the only way I can think of is to create an empty project and get the timing data:
So: 36 cores visible, 70G of memory visible. But this is a CircleCI instance, so capped at 3 vCPUs / 6G RAM, according to the docs. I don't know the details how they implement it. I would assume means something like and the 6G being a cap for the process group/container/whatever-it-uses. Now on that same machine gives me still with Rust 1.60.0, but the cargo invocation with 1.60.0 sees . So that would definitely explain why it keeps within limits on that older version.
OK. My suspicion is that CircleCI's Docker uses cgroups v1 (as does my local system), and doesn't help. is it feasible to support cgroups v1 in ? If not, I'm inclined to revert cc
Should be feasible, yes. But taking a look at the implementation it could end up being more complex because cgroupv1 has a more configurable directory structure that might require scanning to find the place where cpu controller is mounted, which is something wanted to avoid for performance reasons.
The comments above make it sound like the memory problems are not the compiler's fault per se, but rather changes in how many compiler processes Cargo is spawning. is a quick sanity check, comparing the max-rss measurement for the development compiler from Jan 1st (definitely before the 1.61 cutoff) and June 6th. Between those dates we upgraded most of the "primary" benchmarks so we don't have many results in that section, but we do have helloworld and lots of secondary benchmarks, which should be enough to give a rough idea. It shows that 201 benchmark runs saw max-rss reductions, some quite large, while only 2 runs saw increases. So this is pretty strong evidence that the compiler's memory usage has only gone down lately, adding weight to the theory that it's all about how the compiler is being invoked by Cargo. (BTW, I'm not dismissing the significance of this issue, just clarifying the likely cause.)
Supporting cgroupv1 should still be possible without scanning /proc/mountinfo, by looking in the standard location where it's generally mounted. Could you post the output of on CircleCI?
FWIW, people like me who work on compiler performance certainly do care about memory usage. It's and , and we're always happy to hear about regressions.
Explicitly limiting cargo to 6 jobs () seems to give us back about the same run time, while not running into memory or cpu exhaustion, so for now I'll probably go with that.
But hat's what I mean, there is not one standard location. The manpage says Thus, one might mount the cpu controller as follows: mount -t cgroup -o cpu none /sys/fs/cgroup/cpu It is possible to comount multiple controllers against the same hierarchy. For example, here the cpu and cpuacct controllers are comounted against a single hierarchy: mount -t cgroup -o cpu,cpuacct none /sys/fs/cgroup/cpu,cpuacct Those are examples and cgroupv1 is meant to be flexible that one can arbitrarily combine the different controller hierarchies or given them different names. We could try a fixed list but then wouldn't we have to survey distros what they use as their default cgroup v1 setup? A single sample doesn't seem sufficient.
We talked about this in today's meeting. This isn't a libs regression (since libs never supported cgroupv1); it's a cargo regression. Three possibilities: We could consider (or other means of limiting the CPUs cargo uses) a sufficient workaround, and close this. Cargo could (temporarily or permanently) switch back to num_cpus. We could add cgroupv1 support in libs.
Regarding groupv1 support in libs: Since this is not the common case, and on most systems processes aren't in a cgroupv1, I think we should try to minimize the amount of time this takes, even if that means it's best-effort-only. I don't think we should make all the systems where processes aren't in a cgroup go through a (potentially very expensive) mountinfo scan.
We can check first and if the process is in the root cgroup or there is no cgroup support compiled into the kernel then we don't need to scan mountinfo either. We could check cgroupv2 first, then fixed cgroupv1 list and then fallback to mountinfo scan. So we could have a thorough and fast for most configurations implementation, at the price of complexity. But yeah, I guess I'll start with the fixed list and if someone still complains about cgroupv1 later with some non-standard configuration that can be another PR.
I'm not sure I agree that most systems are using cgroups v2. My own system does not, and support has only recently been . I posted with a revert on beta. If we can get (however basic) cgroups v1 support in 1.63, that would make me feel a lot more comfortable with using the std implementation.
I am seeing the same symptom (builds failing due to enormously increased memory usage in 1.61.0) but in a non-Cargo environment. I am still investigating but I notice that rustc calls too, not just Cargo. Cargo's use has been reverted to in but it's possible that will also need to be reverted.
cgroupv1 PR:
landed, so next nightly will have support for cgroupv1 quotas too.

Discussion:
#![feature(hasher_prefixfree_extras)] #![feature(hashmap_internals)] #![feature(int_error_internals)] <ins> #![feature(is_some_with)] </ins> #![feature(maybe_uninit_slice)] #![feature(maybe_uninit_write_slice)] #![feature(mixed_integer_ops)]

))] { #[cfg(any(target_os = "android", target_os = "linux"))] { <del> let quota = cgroup2_quota().max(1); </del> <ins> let quota = cgroups::quota().max(1); </ins> let mut set: libc::cpu_set_t = unsafe { mem::zeroed() }; unsafe { if libc::sched_getaffinity(0, mem::size_of::<libc::cpu_set_t>(), &mut set) == 0 {

} } <del> /// Returns cgroup CPU quota in core-equivalents, rounded down, or usize::MAX if the quota cannot /// be determined or is not set. </del> #[cfg(any(target_os = "android", target_os = "linux"))] <del> fn cgroup2_quota() -> usize { </del> <ins> mod cgroups { //! Currently not covered //! * cgroup v2 in non-standard mountpoints //! * paths containing control characters or spaces, since those would be escaped in procfs //!   output and we don't unescape use crate::borrow::Cow; </ins> use crate::ffi::OsString; use crate::fs::{try_exists, File}; use crate::io::Read; <ins> use crate::io::{BufRead, BufReader}; </ins> use crate::os::unix::ffi::OsStringExt; <ins> use crate::path::Path; </ins> use crate::path::PathBuf; <ins> use crate::str::from_utf8; </ins> <del> let mut quota = usize::MAX; if cfg!(miri) { // Attempting to open a file fails under default flags due to isolation. // And Miri does not have parallelism anyway. return quota; } let _: Option<()> = try { let mut buf = Vec::with_capacity(128); // find our place in the cgroup hierarchy File::open("/proc/self/cgroup").ok()?.read_to_end(&mut buf).ok()?; let cgroup_path = buf .split(|&c| c == b'n') .filter_map(|line| { let mut fields = line.splitn(3, |&c| c == b':'); // expect cgroupv2 which has an empty 2nd field if fields.nth(1) != Some(b"") { return None; } let path = fields.last()?; // skip leading slash Some(path[1..].to_owned()) }) .next()?; let cgroup_path = PathBuf::from(OsString::from_vec(cgroup_path)); </del> <ins> #[derive(PartialEq)] enum Cgroup { V1, V2, } /// Returns cgroup CPU quota in core-equivalents, rounded down or usize::MAX if the quota cannot /// be determined or is not set. pub(super) fn quota() -> usize { let mut quota = usize::MAX; if cfg!(miri) { // Attempting to open a file fails under default flags due to isolation. // And Miri does not have parallelism anyway. return quota; } let _: Option<()> = try { let mut buf = Vec::with_capacity(128); // find our place in the cgroup hierarchy File::open("/proc/self/cgroup").ok()?.read_to_end(&mut buf).ok()?; let (cgroup_path, version) = buf.split(|&c| c == b'n').fold(None, |previous, line| { let mut fields = line.splitn(3, |&c| c == b':'); // 2nd field is a list of controllers for v1 or empty for v2 let version = match fields.nth(1) { Some(b"") => Cgroup::V2, Some(controllers) if from_utf8(controllers) .is_ok_and(|c| c.split(",").any(|c| c == "cpu")) => { Cgroup::V1 } _ => return previous, }; // already-found v1 trumps v2 since it explicitly specifies its controllers if previous.is_some() && version == Cgroup::V2 { return previous; } let path = fields.last()?; // skip leading slash Some((path[1..].to_owned(), version)) })?; let cgroup_path = PathBuf::from(OsString::from_vec(cgroup_path)); quota = match version { Cgroup::V1 => quota_v1(cgroup_path), Cgroup::V2 => quota_v2(cgroup_path), }; }; quota } fn quota_v2(group_path: PathBuf) -> usize { let mut quota = usize::MAX; </ins> let mut path = PathBuf::with_capacity(128); let mut read_buf = String::with_capacity(20); <ins> // standard mount location defined in file-hierarchy(7) manpage </ins> let cgroup_mount = "/sys/fs/cgroup"; path.push(cgroup_mount); <del> path.push(&cgroup_path); </del> <ins> path.push(&group_path); </ins> path.push("cgroup.controllers");

path.pop(); <del> while path.starts_with(cgroup_mount) { path.push("cpu.max"); </del> <ins> let _: Option<()> = try { while path.starts_with(cgroup_mount) { path.push("cpu.max"); read_buf.clear(); if File::open(&path).and_then(|mut f| f.read_to_string(&mut read_buf)).is_ok() { let raw_quota = read_buf.lines().next()?; let mut raw_quota = raw_quota.split(' '); let limit = raw_quota.next()?; let period = raw_quota.next()?; match (limit.parse::<usize>(), period.parse::<usize>()) { (Ok(limit), Ok(period)) => { quota = quota.min(limit / period); } _ => {} } } </ins> <del> read_buf.clear(); </del> <ins> path.pop(); // pop filename path.pop(); // pop dir } }; </ins> <del> if File::open(&path).and_then(|mut f| f.read_to_string(&mut read_buf)).is_ok() { let raw_quota = read_buf.lines().next()?; let mut raw_quota = raw_quota.split(' '); let limit = raw_quota.next()?; let period = raw_quota.next()?; match (limit.parse::<usize>(), period.parse::<usize>()) { (Ok(limit), Ok(period)) => { quota = quota.min(limit / period); } </del> <ins> quota } fn quota_v1(group_path: PathBuf) -> usize { let mut quota = usize::MAX; let mut path = PathBuf::with_capacity(128); let mut read_buf = String::with_capacity(20); // Hardcode commonly used locations mentioned in the cgroups(7) manpage // if that doesn't work scan mountinfo and adjust `group_path` for bind-mounts let mounts: &[fn(&Path) -> Option<(_, &Path)>] = &[ |p| Some((Cow::Borrowed("/sys/fs/cgroup/cpu"), p)), |p| Some((Cow::Borrowed("/sys/fs/cgroup/cpu,cpuacct"), p)), // this can be expensive on systems with tons of mountpoints // but we only get to this point when /proc/self/cgroups explicitly indicated // this process belongs to a cpu-controller cgroup v1 and the defaults didn't work find_mountpoint, ]; for mount in mounts { let Some((mount, group_path)) = mount(&group_path) else { continue }; path.clear(); path.push(mount.as_ref()); path.push(&group_path); // skip if we guessed the mount incorrectly if matches!(try_exists(&path), Err(_) | Ok(false)) { continue; } while path.starts_with(mount.as_ref()) { let mut parse_file = |name| { path.push(name); read_buf.clear(); let f = File::open(&path); path.pop(); // restore buffer before any early returns f.ok()?.read_to_string(&mut read_buf).ok()?; let parsed = read_buf.trim().parse::<usize>().ok()?; Some(parsed) }; let limit = parse_file("cpu.cfs_quota_us"); let period = parse_file("cpu.cfs_period_us"); match (limit, period) { (Some(limit), Some(period)) => quota = quota.min(limit / period), </ins> _ => {} } <ins> path.pop(); </ins> } <del> path.pop(); // pop filename path.pop(); // pop dir </del> <ins> // we passed the try_exists above so we should have traversed the correct hierarchy // when reaching this line break; </ins> } <del> }; quota </del> <ins> quota } /// Scan mountinfo for cgroup v1 mountpoint with a cpu controller /// /// If the cgroupfs is a bind mount then `group_path` is adjusted to skip /// over the already-included prefix fn find_mountpoint(group_path: &Path) -> Option<(Cow<'static, str>, &Path)> { let mut reader = BufReader::new(File::open("/proc/self/mountinfo").ok()?); let mut line = String::with_capacity(256); loop { line.clear(); if reader.read_line(&mut line).ok()? == 0 { break; } let line = line.trim(); let mut items = line.split(' '); let sub_path = items.nth(3)?; let mount_point = items.next()?; let mount_opts = items.next_back()?; let filesystem_type = items.nth_back(1)?; if filesystem_type != "cgroup" || !mount_opts.split(',').any(|opt| opt == "cpu") { // not a cgroup / not a cpu-controller continue; } let sub_path = Path::new(sub_path).strip_prefix("/").ok()?; if !group_path.starts_with(sub_path) { // this is a bind-mount and the bound subdirectory // does not contain the cgroup this process belongs to continue; } let trimmed_group_path = group_path.strip_prefix(sub_path).ok()?; return Some((Cow::Owned(mount_point.to_owned()), trimmed_group_path)); } None } </ins> } #[cfg(all(

/// /// On Linux: /// - It may overcount the amount of parallelism available when limited by a <del> ///   process-wide affinity mask or cgroup quotas and cgroup2 fs or `sched_getaffinity()` can't be </del> <ins> ///   process-wide affinity mask or cgroup quotas and `sched_getaffinity()` or cgroup fs can't be </ins> ///   queried, e.g. due to sandboxing. /// - It may undercount the amount of parallelism if the current thread's affinity mask ///   does not reflect the process' cpuset, e.g. due to pinned threads. <ins> /// - If the process is in a cgroup v1 cpu controller, this may need to ///   scan mountpoints to find the corresponding cgroup v1 controller, ///   which may take time on systems with large numbers of mountpoints. ///   (This does not apply to cgroup v2, or to processes not in a ///   cgroup.) </ins> /// /// On all targets: /// - It may overcount the amount of parallelism available when running in a VM