Repository: bpftrace/bpftrace
Pull request #3934: Adapt to the renaming of nr_running to nr_queued in `struct cfs_rq`
Author: login: shunghsiyu
Created: 2025-03-25T02:23:22Z | Merged: 2025-03-27T06:54:36Z

With ["sched/fair: Fix statistics with delayed dequeue" series](https://lore.kernel.org/all/20241202174606.4074512-1-vincent.guittot@linaro.org/) (more specifically torvalds/linux@736c55a02c47) several fields in "struct cfs_rq" is renamed, leading to runqlen.bt to fail at runtime with the following error:

```
  ERROR: Struct/union of type 'struct cfs_rq' does not contain a field named 'nr_running'
      $len = $my_q->nr_running;
```

Move existing runqlen script to tools/old/, and create a copy of the original with with nr_running changed to nr_queued in its place.

While at it, also touch up the comment regarding BTF.

This address issue report by @Avinesh in #3898.


Diff:
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index f50a5045edcf..a59f876d5583 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -41,26 +41,33 @@ jobs:
         - NAME: LLVM 16
           CMAKE_BUILD_TYPE: Debug
           NIX_TARGET: .#bpftrace-llvm16
+          TOOLS_TEST_OLDVERSION: runqlen.bt
         - NAME: LLVM 17
           CMAKE_BUILD_TYPE: Debug
           NIX_TARGET: .#bpftrace-llvm17
+          TOOLS_TEST_OLDVERSION: runqlen.bt
         - NAME: LLVM 18
           CMAKE_BUILD_TYPE: Debug
           NIX_TARGET: .#bpftrace-llvm18
+          TOOLS_TEST_OLDVERSION: runqlen.bt
         - NAME: LLVM 19
           CMAKE_BUILD_TYPE: Debug
           NIX_TARGET: .#bpftrace-llvm19
+          TOOLS_TEST_OLDVERSION: runqlen.bt
         - NAME: LLVM 20 Release
           CMAKE_BUILD_TYPE: Release
           NIX_TARGET: .#bpftrace-llvm20
+          TOOLS_TEST_OLDVERSION: runqlen.bt
         - NAME: LLVM 20 Debug
           CMAKE_BUILD_TYPE: Debug
           NIX_TARGET: .#bpftrace-llvm20
+          TOOLS_TEST_OLDVERSION: runqlen.bt
         - NAME: LLVM 20 Clang Debug
           CMAKE_BUILD_TYPE: Debug
           NIX_TARGET: .#bpftrace-llvm20
           CC: clang
           CXX: clang++
+          TOOLS_TEST_OLDVERSION: runqlen.bt
         - NAME: Fuzzing
           CMAKE_BUILD_TYPE: Debug
           NIX_TARGET: .#bpftrace-fuzz
@@ -70,6 +77,7 @@ jobs:
           CMAKE_BUILD_TYPE: Debug
           NIX_TARGET: .#bpftrace-llvm20
           NIX_TARGET_KERNEL: .#kernel-6_12
+          TOOLS_TEST_OLDVERSION: runqlen.bt # Need 6.14 kernel
         - NAME: AOT (LLVM 20 Debug)
           CMAKE_BUILD_TYPE: Debug
           NIX_TARGET: .#bpftrace-llvm20
diff --git a/tools/old/runqlen.bt b/tools/old/runqlen.bt
new file mode 100755
index 000000000000..23a7fb853cfd
--- /dev/null
+++ b/tools/old/runqlen.bt
@@ -0,0 +1,41 @@
+#!/usr/bin/env bpftrace
+/*
+ * runqlen.bt	CPU scheduler run queue length as a histogram.
+ *		For Linux, uses bpftrace, eBPF.
+ *
+ * This is a bpftrace version of the bcc tool of the same name.
+ *
+ * For Linux < 6.14.
+ *
+ * Copyright 2018 Netflix, Inc.
+ * Licensed under the Apache License, Version 2.0 (the "License")
+ *
+ * 07-Oct-2018	Brendan Gregg	Created this.
+ */
+
+#ifndef BPFTRACE_HAVE_BTF
+#include <linux/sched.h>
+
+// Without BTF, we'll need to declare some of this struct manually,
+// since it isn't available to be #included. This will need maintenance to match
+// your kernel version. It is from kernel/sched/sched.h:
+struct cfs_rq {
+	struct load_weight load;
+	unsigned int nr_running;
+	unsigned int h_nr_running;
+};
+#endif
+
+BEGIN
+{
+	printf("Sampling run queue length at 99 Hertz... Hit Ctrl-C to end.\n");
+}
+
+profile:hz:99
+{
+	$task = (struct task_struct *)curtask;
+	$my_q = (struct cfs_rq *)$task->se.cfs_rq;
+	$len = (uint64)$my_q->nr_running;
+	$len = $len > 0 ? $len - 1 : 0;	// subtract currently running task
+	@runqlen = lhist($len, 0, 100, 1);
+}
diff --git a/tools/runqlen.bt b/tools/runqlen.bt
index 5d8730a13ac0..394a7b3d24c4 100755
--- a/tools/runqlen.bt
+++ b/tools/runqlen.bt
@@ -14,13 +14,13 @@
 #ifndef BPFTRACE_HAVE_BTF
 #include <linux/sched.h>
 
-// Until BTF is available, we'll need to declare some of this struct manually,
+// Without BTF, we'll need to declare some of this struct manually,
 // since it isn't available to be #included. This will need maintenance to match
 // your kernel version. It is from kernel/sched/sched.h:
 struct cfs_rq {
 	struct load_weight load;
-	unsigned int nr_running;
-	unsigned int h_nr_running;
+	unsigned int nr_queued;
+	unsigned int h_nr_queued;
 };
 #endif
 
@@ -33,7 +33,7 @@ profile:hz:99
 {
 	$task = (struct task_struct *)curtask;
 	$my_q = (struct cfs_rq *)$task->se.cfs_rq;
-	$len = (uint64)$my_q->nr_running;
+	$len = (uint64)$my_q->nr_queued;
 	$len = $len > 0 ? $len - 1 : 0;	// subtract currently running task
 	@runqlen = lhist($len, 0, 100, 1);
 }


Reviews:
user: login: viktormalik
state: CHANGES_REQUESTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-25T06:56:20Z
user: login: shunghsiyu
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-27T05:56:42Z
user: login: viktormalik
body: LGTM, thanks!

The old field is still used in the tests and the manual, right? Would you mind updating them within this PR? I think just changing to a different field from `runqueues` would suffice.

But I'm not going to block on this and we can do it in a follow-up PR.
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-27T06:49:50Z
user: login: shunghsiyu
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-27T06:50:50Z

Review comments:
diff_hunk: @@ -661,13 +661,13 @@ PROG BEGIN { if (*percpu_kaddr("process_counts") == *percpu_kaddr("process_count
 EXPECT SUCCESS
 
 NAME percpu_kaddr field access
-PROG BEGIN { $runq = ((struct rq *)percpu_kaddr("runqueues", 0)); if ($runq != 0) { printf("nr_running: %d\n", $runq->nr_running) } exit() }
-EXPECT_REGEX nr_running: [0-9]+
+PROG BEGIN { $runq = ((struct rq *)percpu_kaddr("runqueues", 0)); if ($runq != 0) { printf("nr_queued: %d\n", $runq->nr_queued) } exit() }
path: tests/runtime/call
user: login: viktormalik
body: GHA runners certainly don't have kernels with this change at the moment. Ideally, we should change this test to use some different field of `runqueues` which is stable across kernels.
created_at: 2025-03-25T06:53:29Z
author_association: CONTRIBUTOR
diff_hunk: @@ -661,13 +661,13 @@ PROG BEGIN { if (*percpu_kaddr("process_counts") == *percpu_kaddr("process_count
 EXPECT SUCCESS
 
 NAME percpu_kaddr field access
-PROG BEGIN { $runq = ((struct rq *)percpu_kaddr("runqueues", 0)); if ($runq != 0) { printf("nr_running: %d\n", $runq->nr_running) } exit() }
-EXPECT_REGEX nr_running: [0-9]+
+PROG BEGIN { $runq = ((struct rq *)percpu_kaddr("runqueues", 0)); if ($runq != 0) { printf("nr_queued: %d\n", $runq->nr_queued) } exit() }
+EXPECT_REGEX nr_queued: [0-9]+
 
 NAME percpu_kaddr field access no NULL check
-PROG BEGIN { $runq = ((struct rq *)percpu_kaddr("runqueues", 0)); printf("nr_running: %d\n", $runq->nr_running); exit() }
+PROG BEGIN { $runq = ((struct rq *)percpu_kaddr("runqueues", 0)); printf("nr_queued: %d\n", $runq->nr_queued); exit() }
path: tests/runtime/call
user: login: viktormalik
body: Same as above.
created_at: 2025-03-25T06:53:39Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,41 @@
+#!/usr/bin/env bpftrace
+/*
+ * runqlen.bt	CPU scheduler run queue length as a histogram.
+ *		For Linux, uses bpftrace, eBPF.
+ *
+ * This is a bpftrace version of the bcc tool of the same name.
+ *
+ * For Linux <= 6.14.
path: tools/old/runqlen-pre-6.14.bt
user: login: viktormalik
body: Shouldn't be?
```suggestion
 * For Linux < 6.14.
```
The change appears in 6.14.
created_at: 2025-03-25T06:54:46Z
author_association: CONTRIBUTOR
diff_hunk: @@ -14,13 +14,13 @@
 #ifndef BPFTRACE_HAVE_BTF
 #include <linux/sched.h>
 
-// Until BTF is available, we'll need to declare some of this struct manually,
+// Without BTF, we'll need to declare some of this struct manually,
path: tools/runqlen.bt
user: login: viktormalik
body: Nit: we could also do this change in the old version.
created_at: 2025-03-25T06:55:16Z
author_association: CONTRIBUTOR
diff_hunk: @@ -2610,7 +2610,7 @@ be rejected.
 interval:s:1 {
   $runqueues = (struct rq *)percpu_kaddr("runqueues", 0);
   if ($runqueues != 0) {         // The check is mandatory here
-    print($runqueues->nr_running);
+    print($runqueues->nr_queued);
path: man/adoc/bpftrace.adoc
user: login: viktormalik
body: Same as for the tests - it would be good to use a different field here, if possible, so that the example works across different kernels.
created_at: 2025-03-25T06:56:02Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,41 @@
+#!/usr/bin/env bpftrace
+/*
+ * runqlen.bt	CPU scheduler run queue length as a histogram.
+ *		For Linux, uses bpftrace, eBPF.
+ *
+ * This is a bpftrace version of the bcc tool of the same name.
+ *
+ * For Linux <= 6.14.
path: tools/old/runqlen-pre-6.14.bt
user: login: shunghsiyu
body: Absolutely, thanks for spotting that.
created_at: 2025-03-27T05:56:42Z
author_association: CONTRIBUTOR
diff_hunk: @@ -661,13 +661,13 @@ PROG BEGIN { if (*percpu_kaddr("process_counts") == *percpu_kaddr("process_count
 EXPECT SUCCESS
 
 NAME percpu_kaddr field access
-PROG BEGIN { $runq = ((struct rq *)percpu_kaddr("runqueues", 0)); if ($runq != 0) { printf("nr_running: %d\n", $runq->nr_running) } exit() }
-EXPECT_REGEX nr_running: [0-9]+
+PROG BEGIN { $runq = ((struct rq *)percpu_kaddr("runqueues", 0)); if ($runq != 0) { printf("nr_queued: %d\n", $runq->nr_queued) } exit() }
path: tests/runtime/call
user: login: shunghsiyu
body: My bad, I confused `struct cfs_rq` with `struct rq` (my lack of scheduler knowledge is evident here), the latter do still have the `nr_running`.

Those can be left as-is.
created_at: 2025-03-27T06:50:50Z
author_association: CONTRIBUTOR

PR comments:
user: login: shunghsiyu
created_at: 2025-03-25T04:07:42Z
body: Looks like I might need to add `TOOLS_TEST_OLDVERSION` for runqlen, not sure if I need to drop the `pre-6.14` postfix to get it work though.
author_association: CONTRIBUTOR
user: login: viktormalik
created_at: 2025-03-25T06:51:57Z
body: > Looks like I might need to add `TOOLS_TEST_OLDVERSION` for runqlen, not sure if I need to drop the `pre-6.14` postfix to get it work though.

Yeah, you need to drop the suffix, it must be called `tools/old/runqlen.bt` for the script to pick it up.
author_association: CONTRIBUTOR
user: login: shunghsiyu
created_at: 2025-03-27T06:53:00Z
body: > The old field is still used in the tests and the manual, right? Would you mind updating them within this PR? I think just changing to a different field from runqueues would suffice.

That was actually a mistake on my part, see [this comment](https://github.com/bpftrace/bpftrace/pull/3934#discussion_r2015773998)

> My bad, I confused struct cfs_rq with struct rq (my lack of scheduler knowledge is evident here), the latter do still have the nr_running.
> 
>  Those can be left as-is.

So I think this PR is good to merge now.

author_association: CONTRIBUTOR
user: login: viktormalik
created_at: 2025-05-22T08:51:47Z
body: @jordalgo since this patch is not in any release, most distros with the latest kernel (e.g. Fedora) currently ship a broken version of runqlen. Should we craft a new patch release to fix that?
author_association: CONTRIBUTOR
user: login: jordalgo
created_at: 2025-05-22T10:17:06Z
body: @viktormalik We can. Are there any other fixes/patches we can include while we're at it? 
Maybe these:
https://github.com/bpftrace/bpftrace/issues/4131
https://github.com/bpftrace/bpftrace/issues/4111
https://github.com/bpftrace/bpftrace/issues/4104
https://github.com/bpftrace/bpftrace/pull/4100
https://github.com/bpftrace/bpftrace/pull/4074
author_association: CONTRIBUTOR
user: login: viktormalik
created_at: 2025-05-22T10:38:38Z
body: I would certainly include #4100 and #4074 as those fix bugs which started appearing on 0.23.0.

The rest look like the bugs have been there for a long time so it's IMHO ok to postpone them for the next release.
author_association: CONTRIBUTOR
user: login: jordalgo
created_at: 2025-05-22T11:21:27Z
body: > The rest look like the bugs have been there for a long time so it's IMHO ok to postpone them for the next release.

They're pretty small fixes. Even if they've been there for a while, still seems like an easy thing to do to improve on a latest release. They feel pretty low risk.
author_association: CONTRIBUTOR
user: login: viktormalik
created_at: 2025-05-22T11:44:35Z
body: Sure, no problem, let's take them in :+1: 
author_association: CONTRIBUTOR
user: login: jordalgo
created_at: 2025-05-22T11:47:34Z
body: @viktormalik Cool. I'll draft a PR for the release branch
author_association: CONTRIBUTOR
user: login: viktormalik
created_at: 2025-05-22T11:55:28Z
body: Thanks! I think we can do a single PR for all the backported commits and the release commit.
author_association: CONTRIBUTOR
user: login: jordalgo
created_at: 2025-05-22T14:12:30Z
body: https://github.com/bpftrace/bpftrace/pull/4177
author_association: CONTRIBUTOR