Repository: rust-lang/rust
Pull request #139502: fix "still mutable" ice while metrics are enabled
Author: login: yaahc
Labels: name: T-compiler
description: Relevant to the compiler team, which will review and decide on the PR/issue.
name: S-waiting-on-bors
description: Status: Waiting on bors to run and complete tests. Bors will change the label on completion.
name: A-run-make
description: Area: port run-make Makefiles to rmake.rs
Created: 2025-04-07T22:45:57Z | Merged: 2025-04-10T20:17:01Z

<!-- homu-ignore:start -->
<!--
If this PR is related to an unstable feature or an otherwise tracked effort,
please link to the relevant tracking issue here. If you don't know of a related
tracking issue or there are none, feel free to ignore this.

This PR will get automatically assigned to a reviewer. In case you would like
a specific user to review your work, you can assign it to them by using


-->
<!-- homu-ignore:end -->

Resolves "still mutable" ICE discovered by @matthiaskrgr here: [#t-docs-rs > metrics intitiative @ 💬](https://rust-lang.zulipchat.com/#narrow/channel/356853-t-docs-rs/topic/metrics.20intitiative/near/510490790)

This was caused by invoking `crate_hash` before the `definitions` struct was frozen here: https://github.com/rust-lang/rust/blob/e643f59f6da3a84f43e75dea99afaa5b041ea6bf/compiler/rustc_interface/src/passes.rs#L951

resolved by moving metrics dumping to occur after `analysis` freezes the definitions

I'm guessing we didn't discover this in CI because the problem only occurs when you try to calculate the crash hash with incremental compilation enabled when it tries to freeze the definitions here: https://github.com/rust-lang/rust/blob/e643f59f6da3a84f43e75dea99afaa5b041ea6bf/compiler/rustc_middle/src/hir/map.rs#L1172

my understanding is that this causes us to freeze the definitions too early in compilation, then we subsequently try to mutate them, likely during `analysis`, and this causes the ICE.
    
r? @bjorn3 


Diff:
diff --git a/compiler/rustc_data_structures/src/sync/freeze.rs b/compiler/rustc_data_structures/src/sync/freeze.rs
index 9720b22ea7d1b..6338afb92c34b 100644
--- a/compiler/rustc_data_structures/src/sync/freeze.rs
+++ b/compiler/rustc_data_structures/src/sync/freeze.rs
@@ -88,7 +88,7 @@ impl<T> FreezeLock<T> {
     #[inline]
     #[track_caller]
     pub fn write(&self) -> FreezeWriteGuard<'_, T> {
-        self.try_write().expect("still mutable")
+        self.try_write().expect("data should not be frozen if we're still attempting to mutate it")
     }
 
     #[inline]
diff --git a/compiler/rustc_driver_impl/src/lib.rs b/compiler/rustc_driver_impl/src/lib.rs
index 595ac1edd289a..bbc18c371ea0e 100644
--- a/compiler/rustc_driver_impl/src/lib.rs
+++ b/compiler/rustc_driver_impl/src/lib.rs
@@ -348,10 +348,6 @@ pub fn run_compiler(at_args: &[String], callbacks: &mut (dyn Callbacks + Send))
             // Make sure name resolution and macro expansion is run.
             let _ = tcx.resolver_for_lowering();
 
-            if let Some(metrics_dir) = &sess.opts.unstable_opts.metrics_dir {
-                dump_feature_usage_metrics(tcx, metrics_dir);
-            }
-
             if callbacks.after_expansion(compiler, tcx) == Compilation::Stop {
                 return early_exit();
             }
@@ -370,6 +366,10 @@ pub fn run_compiler(at_args: &[String], callbacks: &mut (dyn Callbacks + Send))
 
             tcx.ensure_ok().analysis(());
 
+            if let Some(metrics_dir) = &sess.opts.unstable_opts.metrics_dir {
+                dump_feature_usage_metrics(tcx, metrics_dir);
+            }
+
             if callbacks.after_analysis(compiler, tcx) == Compilation::Stop {
                 return early_exit();
             }
diff --git a/compiler/rustc_middle/src/query/mod.rs b/compiler/rustc_middle/src/query/mod.rs
index 0d5fba3cc69b4..dfd877c0b8600 100644
--- a/compiler/rustc_middle/src/query/mod.rs
+++ b/compiler/rustc_middle/src/query/mod.rs
@@ -1894,6 +1894,11 @@ rustc_queries! {
 
     // The macro which defines `rustc_metadata::provide_extern` depends on this query's name.
     // Changing the name should cause a compiler error, but in case that changes, be aware.
+    //
+    // The hash should not be calculated before the `analysis` pass is complete, specifically
+    // until `tcx.untracked().definitions.freeze()` has been called, otherwise if incremental
+    // compilation is enabled calculating this hash can freeze this structure too early in
+    // compilation and cause subsequent crashes when attempting to write to `definitions`
     query crate_hash(_: CrateNum) -> Svh {
         eval_always
         desc { "looking up the hash a crate" }
diff --git a/tests/run-make/unstable-feature-usage-metrics-incremental/main.rs b/tests/run-make/unstable-feature-usage-metrics-incremental/main.rs
new file mode 100644
index 0000000000000..f970d395b2cf2
--- /dev/null
+++ b/tests/run-make/unstable-feature-usage-metrics-incremental/main.rs
@@ -0,0 +1,16 @@
+#![feature(ascii_char)] // random lib feature
+#![feature(box_patterns)] // random lang feature
+
+// picked arbitrary unstable features, just need a random lib and lang feature, ideally ones that
+// won't be stabilized any time soon so we don't have to update this test
+fn main() {
+    for s in quix("foo/bar") {
+        print!("{s}");
+    }
+    println!();
+}
+
+// need a latebound var to trigger the incremental compilation ICE
+fn quix(foo: &str) -> impl Iterator<Item = &'_ str> + '_ {
+    foo.split('/')
+}
diff --git a/tests/run-make/unstable-feature-usage-metrics-incremental/rmake.rs b/tests/run-make/unstable-feature-usage-metrics-incremental/rmake.rs
new file mode 100644
index 0000000000000..7e070d80c7933
--- /dev/null
+++ b/tests/run-make/unstable-feature-usage-metrics-incremental/rmake.rs
@@ -0,0 +1,94 @@
+//! This test checks if unstable feature usage metric dump files `unstable-feature-usage*.json` work
+//! as expected.
+//!
+//! - Basic sanity checks on a default ICE dump.
+//!
+//! See <https://github.com/rust-lang/rust/issues/129485>.
+//!
+//! # Test history
+//!
+//! - forked from dump-ice-to-disk test, which has flakeyness issues on i686-mingw, I'm assuming
+//! those will be present in this test as well on the same platform
+
+//@ ignore-windows
+//FIXME(#128911): still flakey on i686-mingw.
+
+use std::path::{Path, PathBuf};
+
+use run_make_support::rfs::create_dir_all;
+use run_make_support::{
+    cwd, filename_contains, has_extension, rfs, run_in_tmpdir, rustc, serde_json,
+    shallow_find_files,
+};
+
+fn find_feature_usage_metrics<P: AsRef<Path>>(dir: P) -> Vec<PathBuf> {
+    shallow_find_files(dir, |path| {
+        if filename_contains(path, "unstable_feature_usage") && has_extension(path, "json") {
+            true
+        } else {
+            dbg!(path);
+            false
+        }
+    })
+}
+
+fn main() {
+    test_metrics_dump();
+    test_metrics_errors();
+}
+
+#[track_caller]
+fn test_metrics_dump() {
+    run_in_tmpdir(|| {
+        let metrics_dir = cwd().join("metrics");
+        create_dir_all(&metrics_dir);
+        rustc()
+            .input("main.rs")
+            .incremental("incremental")
+            .env("RUST_BACKTRACE", "short")
+            .arg(format!("-Zmetrics-dir={}", metrics_dir.display()))
+            .run();
+        let mut metrics = find_feature_usage_metrics(&metrics_dir);
+        let json_path =
+            metrics.pop().expect("there should be one metrics file in the output directory");
+
+        // After the `pop` above, there should be no files left.
+        assert!(
+            metrics.is_empty(),
+            "there should be no more than one metrics file in the output directory"
+        );
+
+        let message = rfs::read_to_string(json_path);
+        let mut parsed: serde_json::Value =
+            serde_json::from_str(&message).expect("metrics should be dumped as json");
+        // remove timestamps
+        assert!(parsed["lib_features"][0]["timestamp"].is_number());
+        assert!(parsed["lang_features"][0]["timestamp"].is_number());
+        parsed["lib_features"][0]["timestamp"] = serde_json::json!(null);
+        parsed["lang_features"][0]["timestamp"] = serde_json::json!(null);
+        let expected = serde_json::json!(
+            {
+                "lib_features":[{"symbol":"ascii_char", "timestamp":null}],
+                "lang_features":[{"symbol":"box_patterns","since":null, "timestamp":null}]
+            }
+        );
+
+        assert_eq!(expected, parsed);
+    });
+}
+
+#[track_caller]
+fn test_metrics_errors() {
+    run_in_tmpdir(|| {
+        rustc()
+            .input("main.rs")
+            .incremental("incremental")
+            .env("RUST_BACKTRACE", "short")
+            .arg("-Zmetrics-dir=invaliddirectorythatdefinitelydoesntexist")
+            .run_fail()
+            .assert_stderr_contains(
+                "error: cannot dump feature usage metrics: No such file or directory",
+            )
+            .assert_stdout_not_contains("internal compiler error");
+    });
+}


Reviews:
user: login: compiler-errors
body: Are there any run-make tests for these metrics flags? It would be nice to have a test for this to actually show that this flag works without ICEing in incremental.
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-07T22:47:55Z
user: login: bjorn3
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-08T09:03:15Z
user: login: yaahc
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-08T17:14:03Z

Review comments:
diff_hunk: @@ -370,6 +366,10 @@ pub fn run_compiler(at_args: &[String], callbacks: &mut (dyn Callbacks + Send))
 
             tcx.ensure_ok().analysis(());
 
+            if let Some(metrics_dir) = &sess.opts.unstable_opts.metrics_dir {
path: compiler/rustc_driver_impl/src/lib.rs
user: login: bjorn3
body: Maybe add a comment that no metrics may be written before `analysis` to avoid ICE?
created_at: 2025-04-08T09:03:15Z
author_association: MEMBER
diff_hunk: @@ -370,6 +366,10 @@ pub fn run_compiler(at_args: &[String], callbacks: &mut (dyn Callbacks + Send))
 
             tcx.ensure_ok().analysis(());
 
+            if let Some(metrics_dir) = &sess.opts.unstable_opts.metrics_dir {
path: compiler/rustc_driver_impl/src/lib.rs
user: login: yaahc
body: I am not confident that we will always use the same naming scheme involving the SVH for all metrics going forward so I would prefer to put the comment on the `crate_hash` query or next to my call to `crate_hash`
created_at: 2025-04-08T17:14:03Z
author_association: MEMBER

PR comments:
user: login: yaahc
created_at: 2025-04-07T22:50:49Z
body: > Are there any run-make tests for these metrics flags? It would be nice to have a test for this to actually show that this flag works without ICEing in incremental.

we have a run-make test but I doubt it enables incremental compilation. I can enable that, question is do I just do that on the same test, do I create a second otherwise identical test that uses incremental, or is there some way to make the test run twice, once with and once without incremental compilation enabled?
author_association: MEMBER
user: login: compiler-errors
created_at: 2025-04-07T22:52:50Z
body: A second test is best, minimized to whatever you need to repro the breakage. Also probably best to check that it faithfully reproduces the ICE without the change in this PR.
author_association: CONTRIBUTOR
user: login: matthiaskrgr
created_at: 2025-04-08T04:03:24Z
body: maybe we could also change "still mutable" to be slightly more descriptive? 😅
author_association: MEMBER
user: login: yaahc
created_at: 2025-04-08T17:14:46Z
body: > maybe we could also change "still mutable" to be slightly more descriptive? 😅

probably something like "freezelock should not be frozen since we're still trying to write to it"
author_association: MEMBER
user: login: rustbot
created_at: 2025-04-08T21:11:38Z
body: This PR modifies `run-make` tests.

cc @jieyouxu
author_association: COLLABORATOR
user: login: yaahc
created_at: 2025-04-08T21:12:59Z
body: Alright so I implemented all the changes but I'm not yet getting the repro of the ICE on the run-make test. I'm guessing this is because my example is too simplistic and isn't exercising the part of the type system that needed the struct to not yet be frozen, investigating.
author_association: MEMBER
user: login: yaahc
created_at: 2025-04-09T15:59:45Z
body: I got it btw, I needed a late bound parameter which seemed to be the common factor in both of the repros I encountered

https://github.com/rust-lang/rust/pull/139502/files#diff-c576e89e24b5bfa584fdb04e336f6768068b3abc216621232b1dcdc18a3e5315R14
author_association: MEMBER
user: login: bjorn3
created_at: 2025-04-10T11:35:33Z
body: @bors r+
author_association: MEMBER
user: login: bors
created_at: 2025-04-10T11:35:37Z
body: :pushpin: Commit 6f5501583557d2d918cd42fab3f99ad153152afd has been approved by `bjorn3`

It is now in the [queue](https://bors.rust-lang.org/homu/queue/rust) for this repository.

<!-- @bors r=bjorn3 6f5501583557d2d918cd42fab3f99ad153152afd -->
<!-- homu: {"type":"Approved","sha":"6f5501583557d2d918cd42fab3f99ad153152afd","approver":"bjorn3","queue":"https://bors.rust-lang.org/homu/queue/rust"} -->
author_association: COLLABORATOR