Repository: NordicHPC/sonar (language: python)

Issue:
Unlikely peak CPU usage for jobs with --batchless and subjobs
Here's a funny one.  The fifth field (26268) is peak CPU usage, corresponding to 262 cores going at 100%:
```
957985	joachipo  0d 0h35m  3284 	26268 	1    	1     	0    	0     	0       	0        	ml3   bash
```
This is on a system with 64 cores, so it's clearly wrong: the max peak is 6400.

The reason this happens is an artifact of how `--batchless` jobs are handled by sonar.  In this case, processes below a session leader are taken to be root processes for individual jobs (which could be trees of processes); typically the shell is a session leader.  So if somebody runs five (say) heavy python processes (or process trees) in sequence from the shell, each of those is a job, as desired.  Because of how we collect the `cputime_sec` field as the sum of the process's own CPU time and that of all terminated children, however, the CPU time for all those jobs is eventually accumulated in the shell.  It can thus happen that when a long-running job terminates the shell receives a tremendous increase in accumulated CPU time out of nowhere within the time window between two samples.  Since the analysis code computes cpu utilization as (delta cpu usage)/(delta time), the usage can become very high, leading to the artifact shown above.

This is not a big deal at this time, as it pertains only to the shell and its relation to jobs below it, and really we don't care about the shells for analysis, so we can just sweep these artifacts under the carpet.  But it really comes back to how our aggregating data collection leads us into tight corners.  We chose to use the bsdtime field of the ps output along with the `--cumulative` switch to collect self+children time, and this is right for individual jobs, but it breaks down in this weird cross-job situation.

There are a couple of possible fixes, none of them urgent (details need to be worked out in both cases):

- for processes that are session leaders, collect only self time, not self+children
- do not aggregate data the way we do but instead present it in a way that lets the consumer aggregate according to need, for example, by a sonar record containing a list of processes with the self cpu time

Unlikely peak CPU usage for jobs with --batchless and subjobs
Here's a funny one.  The fifth field (26268) is peak CPU usage, corresponding to 262 cores going at 100%:
```
957985	joachipo  0d 0h35m  3284 	26268 	1    	1     	0    	0     	0       	0        	ml3   bash
```
This is on a system with 64 cores, so it's clearly wrong: the max peak is 6400.

The reason this happens is an artifact of how `--batchless` jobs are handled by sonar.  In this case, processes below a session leader are taken to be root processes for individual jobs (which could be trees of processes); typically the shell is a session leader.  So if somebody runs five (say) heavy python processes (or process trees) in sequence from the shell, each of those is a job, as desired.  Because of how we collect the `cputime_sec` field as the sum of the process's own CPU time and that of all terminated children, however, the CPU time for all those jobs is eventually accumulated in the shell.  It can thus happen that when a long-running job terminates the shell receives a tremendous increase in accumulated CPU time out of nowhere within the time window between two samples.  Since the analysis code computes cpu utilization as (delta cpu usage)/(delta time), the usage can become very high, leading to the artifact shown above.

This is not a big deal at this time, as it pertains only to the shell and its relation to jobs below it, and really we don't care about the shells for analysis, so we can just sweep these artifacts under the carpet.  But it really comes back to how our aggregating data collection leads us into tight corners.  We chose to use the bsdtime field of the ps output along with the `--cumulative` switch to collect self+children time, and this is right for individual jobs, but it breaks down in this weird cross-job situation.

There are a couple of possible fixes, none of them urgent (details need to be worked out in both cases):

- for processes that are session leaders, collect only self time, not self+children
- do not aggregate data the way we do but instead present it in a way that lets the consumer aggregate according to need, for example, by a sonar record containing a list of processes with the self cpu time



Discussion / hints:
Digging into some logs to try to substantiate this.  The following is from the case cited on https://github.com/NAICNO/Jobanalyzer/issues/77.

User karths on ml8 on Oct 9 2023, this is `sonalyze load --none --user karths`, fields are date/time, cpu% mem% gpu% gpumem% (the defaults).
```
2023-10-09  07:45  9406     35   0    0
2023-10-09  07:50  2049519  53   0    0
```
Something weird there, max CPU is 9600.

Looking further, job 2169952 makes a sudden jump, more than doubling its CPU, this is `sonalyze parse --user karths, fields are cputime_sec, localtime, user, pid, job, cmd`
```
804693       2023-10-09 07:35  karths  2169952  2169952  jupyter-lab
804699       2023-10-09 07:40  karths  2169952  2169952  jupyter-lab
804699       2023-10-09 07:45  karths  2169952  2169952  jupyter-lab
6931011      2023-10-09 07:50  karths  2169952  2169952  jupyter-lab
6931011      2023-10-09 07:55  karths  2169952  2169952  jupyter-lab
```
800K CPU seconds = 13333 CPU minutes, but available in that interval is 96 cores * 5 minutes = 480 CPU minutes.

With a little more context:
```
6012346      2023-10-09 07:25  karths  2994535  2994535  python
804693       2023-10-09 07:25  karths  2169952  2169952  jupyter-lab
804693       2023-10-09 07:30  karths  2169952  2169952  jupyter-lab
6040575      2023-10-09 07:30  karths  2994535  2994535  python
804693       2023-10-09 07:35  karths  2169952  2169952  jupyter-lab
6068812      2023-10-09 07:35  karths  2994535  2994535  python
6097050      2023-10-09 07:40  karths  2994535  2994535  python
804699       2023-10-09 07:40  karths  2169952  2169952  jupyter-lab
6125268      2023-10-09 07:45  karths  2994535  2994535  python
804699       2023-10-09 07:45  karths  2169952  2169952  jupyter-lab
19576        2023-10-09 07:50  karths  2345804  2345804  python
6931011      2023-10-09 07:50  karths  2169952  2169952  jupyter-lab
47821        2023-10-09 07:55  karths  2345804  2345804  python
6931011      2023-10-09 07:55  karths  2169952  2169952  jupyter-lab
76451        2023-10-09 08:00  karths  2345804  2345804  python
6931011      2023-10-09 08:00  karths  2169952  2169952  jupyter-lab
```
Observe how, if job 2994535 - which ends right at the critical moment - is rolled into job 2169952, we get almost the exact right number for the latter one.  So that's probably what's going on here, something that was seen as an independent job and had been running for a long time is accounted for in the parent.  This is consistent with the hypothesis on this bug, but with a little more complexity.  
And finally, a simple test case.

First run sonar in the background:
```
while true; do
    target/debug/sonar ps --batchless --exclude-system-jobs \
        --exclude-commands kworker,gsd,Isolated,snap,xdg,Utility,evol,gvfs,at-,gnome,Xw,systemd,ibus,pipewire,firefox,WebEx,tracker,dbus,RDD,gdm,Socket,goa,keepass,ssh,pipewire,\(sd,fork,gjs,pulse,Privileged,dconf,update \
        >> sonar-out.txt; sleep 60;
done
```

Then run a program that sucks up a lot of CPU info, here a matrix multiply that takes about 100s on my laptop, run six times:
```
for i in 1 2 3 4 5 6 ; do ../uio-sandbox/mmul/mmul ; done
```

Then look at the sonar output, here pid 8766 is the parent shell of the latter loop above:
```
v=0.10.0,time=2024-04-10T13:54:06+0200,host=bling,user=larstha,cmd=bash,job=8766,pid=8766,cpukib=2996,rssanonkib=2688,cputime_sec=145
v=0.10.0,time=2024-04-10T13:55:06+0200,host=bling,user=larstha,cmd=bash,job=8766,pid=8766,cpukib=2996,rssanonkib=2688,cputime_sec=145
v=0.10.0,time=2024-04-10T13:56:06+0200,host=bling,user=larstha,cmd=bash,job=8766,pid=8766,cpukib=2996,rssanonkib=2688,cputime_sec=145
v=0.10.0,time=2024-04-10T13:57:06+0200,host=bling,user=larstha,cmd=bash,job=8766,pid=8766,cpukib=2996,rssanonkib=2688,cputime_sec=238
v=0.10.0,time=2024-04-10T13:58:06+0200,host=bling,user=larstha,cmd=bash,job=8766,pid=8766,cpukib=2996,rssanonkib=2688,cputime_sec=333
v=0.10.0,time=2024-04-10T13:59:06+0200,host=bling,user=larstha,cmd=bash,job=8766,pid=8766,cpukib=2996,rssanonkib=2688,cputime_sec=333
v=0.10.0,time=2024-04-10T14:00:06+0200,host=bling,user=larstha,cmd=bash,job=8766,pid=8766,cpukib=2996,rssanonkib=2688,cputime_sec=427
v=0.10.0,time=2024-04-10T14:01:06+0200,host=bling,user=larstha,cmd=bash,job=8766,pid=8766,cpukib=2996,rssanonkib=2688,cputime_sec=522
v=0.10.0,time=2024-04-10T14:02:06+0200,host=bling,user=larstha,cmd=bash,job=8766,pid=8766,cpukib=2996,rssanonkib=2688,cputime_sec=522
v=0.10.0,time=2024-04-10T14:03:06+0200,host=bling,user=larstha,cmd=bash,job=8766,pid=8766,cpukib=2996,rssanonkib=2688,cputime_sec=629
v=0.10.0,time=2024-04-10T14:04:06+0200,host=bling,user=larstha,cmd=bash,job=8766,pid=8766,cpukib=2996,rssanonkib=2688,cputime_sec=629
v=0.10.0,time=2024-04-10T14:05:06+0200,host=bling,user=larstha,cmd=bash,job=8766,pid=8766,cpukib=2996,rssanonkib=2688,cputime_sec=741
```
Even though nothing was going on in that shell while it was running the loop, its CPU time as recorded by Sonar increases by about 100s every time one of the matrix multiplies completes.

This is more complicated than I thought.  We have both *process groups* (collections of processes) and *sessions* (collections of process groups).  A session leader is also - at least normally - a process group leader.  But there will be multiple process groups under a session in many cases, and I think the concept we're looking for with `--batchless` is related to process groups, not sessions.  A new process group is created for a new pipeline created from the shell.  The root of the job may more appropriately be the process group leader if that is not also a session leader.  But this is still somewhat mysterious.  And it may actually be not quite clean - after all we're trying to reconstruct an idea (job ID) from a setting in which it does not exist.

Reading a bunch of background documentation, going back esp to 4.3BSD, it is really the case that a "process group" is the same as a "job".  Indeed [this](https://en.wikipedia.org/wiki/Job_control_(Unix)) cites the POSIX definition of a "job" as "A set of processes, comprising a shell pipeline, and any processes descended from it, that are all in the same process group."

What this means is probably that sonar should be tweaked a little to not use "session" but "process group".  The changes to handle this will be very localized and only affect `--batchless` runs anyway.
Digging into some logs to try to substantiate this.  The following is from the case cited on https://github.com/NAICNO/Jobanalyzer/issues/77.

User karths on ml8 on Oct 9 2023, this is `sonalyze load --none --user karths`, fields are date/time, cpu% mem% gpu% gpumem% (the defaults).
```
2023-10-09  07:45  9406     35   0    0
2023-10-09  07:50  2049519  53   0    0
```
Something weird there, max CPU is 9600.

Looking further, job 2169952 makes a sudden jump, more than doubling its CPU, this is `sonalyze parse --user karths, fields are cputime_sec, localtime, user, pid, job, cmd`
```
804693       2023-10-09 07:35  karths  2169952  2169952  jupyter-lab
804699       2023-10-09 07:40  karths  2169952  2169952  jupyter-lab
804699       2023-10-09 07:45  karths  2169952  2169952  jupyter-lab
6931011      2023-10-09 07:50  karths  2169952  2169952  jupyter-lab
6931011      2023-10-09 07:55  karths  2169952  2169952  jupyter-lab
```
800K CPU seconds = 13333 CPU minutes, but available in that interval is 96 cores * 5 minutes = 480 CPU minutes.

With a little more context:
```
6012346      2023-10-09 07:25  karths  2994535  2994535  python
804693       2023-10-09 07:25  karths  2169952  2169952  jupyter-lab
804693       2023-10-09 07:30  karths  2169952  2169952  jupyter-lab
6040575      2023-10-09 07:30  karths  2994535  2994535  python
804693       2023-10-09 07:35  karths  2169952  2169952  jupyter-lab
6068812      2023-10-09 07:35  karths  2994535  2994535  python
6097050      2023-10-09 07:40  karths  2994535  2994535  python
804699       2023-10-09 07:40  karths  2169952  2169952  jupyter-lab
6125268      2023-10-09 07:45  karths  2994535  2994535  python
804699       2023-10-09 07:45  karths  2169952  2169952  jupyter-lab
19576        2023-10-09 07:50  karths  2345804  2345804  python
6931011      2023-10-09 07:50  karths  2169952  2169952  jupyter-lab
47821        2023-10-09 07:55  karths  2345804  2345804  python
6931011      2023-10-09 07:55  karths  2169952  2169952  jupyter-lab
76451        2023-10-09 08:00  karths  2345804  2345804  python
6931011      2023-10-09 08:00  karths  2169952  2169952  jupyter-lab
```
Observe how, if job 2994535 - which ends right at the critical moment - is rolled into job 2169952, we get almost the exact right number for the latter one.  So that's probably what's going on here, something that was seen as an independent job and had been running for a long time is accounted for in the parent.  This is consistent with the hypothesis on this bug, but with a little more complexity.  
And finally, a simple test case.

First run sonar in the background:
```
while true; do
    target/debug/sonar ps --batchless --exclude-system-jobs \
        --exclude-commands kworker,gsd,Isolated,snap,xdg,Utility,evol,gvfs,at-,gnome,Xw,systemd,ibus,pipewire,firefox,WebEx,tracker,dbus,RDD,gdm,Socket,goa,keepass,ssh,pipewire,\(sd,fork,gjs,pulse,Privileged,dconf,update \
        >> sonar-out.txt; sleep 60;
done
```

Then run a program that sucks up a lot of CPU info, here a matrix multiply that takes about 100s on my laptop, run six times:
```
for i in 1 2 3 4 5 6 ; do ../uio-sandbox/mmul/mmul ; done
```

Then look at the sonar output, here pid 8766 is the parent shell of the latter loop above:
```
v=0.10.0,time=2024-04-10T13:54:06+0200,host=bling,user=larstha,cmd=bash,job=8766,pid=8766,cpukib=2996,rssanonkib=2688,cputime_sec=145
v=0.10.0,time=2024-04-10T13:55:06+0200,host=bling,user=larstha,cmd=bash,job=8766,pid=8766,cpukib=2996,rssanonkib=2688,cputime_sec=145
v=0.10.0,time=2024-04-10T13:56:06+0200,host=bling,user=larstha,cmd=bash,job=8766,pid=8766,cpukib=2996,rssanonkib=2688,cputime_sec=145
v=0.10.0,time=2024-04-10T13:57:06+0200,host=bling,user=larstha,cmd=bash,job=8766,pid=8766,cpukib=2996,rssanonkib=2688,cputime_sec=238
v=0.10.0,time=2024-04-10T13:58:06+0200,host=bling,user=larstha,cmd=bash,job=8766,pid=8766,cpukib=2996,rssanonkib=2688,cputime_sec=333
v=0.10.0,time=2024-04-10T13:59:06+0200,host=bling,user=larstha,cmd=bash,job=8766,pid=8766,cpukib=2996,rssanonkib=2688,cputime_sec=333
v=0.10.0,time=2024-04-10T14:00:06+0200,host=bling,user=larstha,cmd=bash,job=8766,pid=8766,cpukib=2996,rssanonkib=2688,cputime_sec=427
v=0.10.0,time=2024-04-10T14:01:06+0200,host=bling,user=larstha,cmd=bash,job=8766,pid=8766,cpukib=2996,rssanonkib=2688,cputime_sec=522
v=0.10.0,time=2024-04-10T14:02:06+0200,host=bling,user=larstha,cmd=bash,job=8766,pid=8766,cpukib=2996,rssanonkib=2688,cputime_sec=522
v=0.10.0,time=2024-04-10T14:03:06+0200,host=bling,user=larstha,cmd=bash,job=8766,pid=8766,cpukib=2996,rssanonkib=2688,cputime_sec=629
v=0.10.0,time=2024-04-10T14:04:06+0200,host=bling,user=larstha,cmd=bash,job=8766,pid=8766,cpukib=2996,rssanonkib=2688,cputime_sec=629
v=0.10.0,time=2024-04-10T14:05:06+0200,host=bling,user=larstha,cmd=bash,job=8766,pid=8766,cpukib=2996,rssanonkib=2688,cputime_sec=741
```
Even though nothing was going on in that shell while it was running the loop, its CPU time as recorded by Sonar increases by about 100s every time one of the matrix multiplies completes.

This is more complicated than I thought.  We have both *process groups* (collections of processes) and *sessions* (collections of process groups).  A session leader is also - at least normally - a process group leader.  But there will be multiple process groups under a session in many cases, and I think the concept we're looking for with `--batchless` is related to process groups, not sessions.  A new process group is created for a new pipeline created from the shell.  The root of the job may more appropriately be the process group leader if that is not also a session leader.  But this is still somewhat mysterious.  And it may actually be not quite clean - after all we're trying to reconstruct an idea (job ID) from a setting in which it does not exist.

Reading a bunch of background documentation, going back esp to 4.3BSD, it is really the case that a "process group" is the same as a "job".  Indeed [this](https://en.wikipedia.org/wiki/Job_control_(Unix)) cites the POSIX definition of a "job" as "A set of processes, comprising a shell pipeline, and any processes descended from it, that are all in the same process group."

What this means is probably that sonar should be tweaked a little to not use "session" but "process group".  The changes to handle this will be very localized and only affect `--batchless` runs anyway.

Fix patch:
diff --git a/.zenodo.json b/.zenodo.json
index f8f0a1a..d955a28 100644
--- a/.zenodo.json
+++ b/.zenodo.json
@@ -25,5 +25,5 @@
     "id": "GPL-3.0"
   },
   "title": "sonar: Tool to profile usage of HPC resources by regularly probing processes using ps.",
-  "version": "0.6.2"
+  "version": "0.11.0"
 }
diff --git a/Cargo.lock b/Cargo.lock
index 2b56725..ebb94f9 100644
--- a/Cargo.lock
+++ b/Cargo.lock
@@ -10,7 +10,7 @@ checksum = "9c198f91728a82281a64e1f4f9eeb25d82cb32a5de251c6bd1b5154d63a8e7bd"
 
 [[package]]
 name = "sonar"
-version = "0.10.0"
+version = "0.11.0"
 dependencies = [
  "libc",
  "subprocess",
diff --git a/Cargo.toml b/Cargo.toml
index 3bfdd69..ca62476 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "sonar"
-version = "0.10.0"
+version = "0.11.0"
 edition = "2021"
 
 # See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
diff --git a/README.md b/README.md
index 658de0b..5198c6b 100644
--- a/README.md
+++ b/README.md
@@ -50,9 +50,18 @@ affect the output nevertheless, ie, most changes not covered by changes to the m
 
 These rules are new with v0.8.0.
 
+### Changes in v0.11.x
+
+**Better `ps` data**.  More data points. (v0.11.0)
+
 ### Changes in v0.10.x
 
-**Less output**.  Removed the `cores` and `memtotalkib` fields, as they are supplied by `sonar sysinfo`.
+**Less output**.  Removed the `cores` and `memtotalkib` fields, as they are supplied by `sonar
+sysinfo`. (v0.10.0)
+
+**Batchless job ID**.  The meaning of the `job` field for systems without a batch queue (`sonar ps
+--batchless`) has changed from being the pid of the process below the session leader to being the
+more conventional process group id.  In most situations this won't make a difference. (v0.10.1)
 
 ### Changes in v0.9.x
 
@@ -170,6 +179,12 @@ v=0.7.0,time=2023-08-10T11:09:41+02:00,host=somehost,cores=8,user=someone,job=0,
 v=0.7.0,time=2023-08-10T11:09:41+02:00,host=somehost,cores=8,user=someone,job=0,cmd=slack,cpu%=3.9,cpukib=716924,gpus=none,gpu%=0,gpumem%=0,gpukib=0,cputime_sec=266
 ```
 
+### Version 0.11.0 `ps` output format
+
+Version 0.11.0 adds one field:
+
+`ppid` (optional, default "0"): The parent process ID of the job, a positive integer.
+
 ### Version 0.10.0 `ps` output format
 
 The fields `cores` and `memtotalkib` were removed, as they were unused by all clients and are
@@ -252,6 +267,8 @@ ID.  Multiple records with the job ID 0 should never be merged into a single job
 (see `rolledup` below) this has value zero.  Otherwise, this record represents one process and so
 the field holds the process ID.
 
+`ppid` (optional, default "0"): The parent process ID of the job, a positive integer.
+
 `cpu%` (optional, default "0"): The running average CPU percentage over the true lifetime of the
 process (ie computed independently of the sonar log), a nonnegative floating-point number.  100.0
 corresponds to "one full core's worth of computation".
diff --git a/src/batchless.rs b/src/batchless.rs
index 420e7a7..019afc8 100644
--- a/src/batchless.rs
+++ b/src/batchless.rs
@@ -1,22 +1,7 @@
-// jobs::JobManager for systems without a job queue.
+// jobs::JobManager for systems without a batch job queue.
 //
-// In this system, the job ID of a process is found by walking the tree of jobs from the PID until
-// we reach a process that is directly below a session leader, and then taking that process's PID as
-// the job ID.  (The session leader is a process whose session PID is its own PID.)  If the process
-// we're given is a session leader then we take its own PID to be the job ID.  Most other things end
-// up at the init process or at "the system", which is OK - their own PIDs are their job IDs.
-//
-// There's a possibility that the job ID will be reused during the lifetime of the system, confusing
-// our statistics.  On Linux, the PIDs wrap around at about 4e6, and on a busy system this happens
-// in a matter of days.  However, the job ID is not used in isolation (at the moment), but always
-// with the user and the command name, so the reuse problem is not huge.
-//
-// There's also a challenge with this scheme in that, since the output is keyed on program name as
-// well as on user name and job ID, multiple output lines are going to have the same user name and
-// job ID in a tree of *different* processes (ie where subprocesses of the root process in the job
-// exec something with a different name).  This is not wrong but it is something that the consumer
-// must take into account.  For example, in assessing the resources for a job, the resources for all
-// the different programs for the job must be taken into account.
+// Since 4.3BSD it's been the case that "job" === "a process group", and POSIX defines it thus.
+// Hence the process group ID of a process is its job ID, in the absence of other information.
 
 use crate::jobs;
 #[cfg(test)]
@@ -24,72 +9,21 @@ use crate::jobs::JobManager;
 use crate::procfs;
 use std::collections::HashMap;
 
-pub struct BatchlessJobManager {
-    // Process tables can be large and searching them sequentially for every lookup will be slow, so
-    // add a cache.  Various structures could work.  Here a hashmap maps PID -> (session, PPID).
-    cache: HashMap<usize, (usize, usize)>,
-}
+pub struct BatchlessJobManager {}
 
 impl BatchlessJobManager {
     pub fn new() -> BatchlessJobManager {
-        BatchlessJobManager {
-            cache: HashMap::new(),
-        }
-    }
-}
-
-impl BatchlessJobManager {
-    fn lookup(&mut self, processes: &[procfs::Process], want_pid: usize) -> Option<(usize, usize)> {
-        let probe = self.cache.get(&want_pid);
-        if probe.is_some() {
-            return probe.copied();
-        }
-
-        for procfs::Process {
-            pid, ppid, session, ..
-        } in processes
-        {
-            if *pid == want_pid {
-                let entry = (*session, *ppid);
-                self.cache.insert(want_pid, entry);
-                return Some(entry);
-            }
-        }
-
-        None
+        BatchlessJobManager {}
     }
 }
 
 impl jobs::JobManager for BatchlessJobManager {
-    fn job_id_from_pid(&mut self, mut proc_pid: usize, processes: &[procfs::Process]) -> usize {
-        let mut probe = self.lookup(processes, proc_pid);
-        if probe.is_none() {
+    fn job_id_from_pid(&mut self, proc_pid: usize, processes: &HashMap<usize, procfs::Process>) -> usize {
+        if let Some(p) = processes.get(&proc_pid) {
+            p.pgrp
+        } else {
             // Lost process is job 0
             0
-        } else {
-            loop {
-                let (proc_session, parent_pid) = probe.unwrap();
-                if proc_session == 0 {
-                    // System process is its own job
-                    break proc_session;
-                }
-                if proc_session == proc_pid {
-                    // Session leader is its own job
-                    break proc_session;
-                }
-                let probe_parent = self.lookup(processes, parent_pid);
-                if probe_parent.is_none() {
-                    // Orphaned subprocess is its own job
-                    break proc_session;
-                }
-                let (parent_session, _) = probe_parent.unwrap();
-                if parent_pid == parent_session {
-                    // Parent process is session leader, so this process is the job root
-                    break proc_pid;
-                }
-                proc_pid = parent_pid;
-                probe = probe_parent;
-            }
         }
     }
 }
@@ -98,357 +32,134 @@ impl jobs::JobManager for BatchlessJobManager {
 fn test_batchless_jobs() {
     let mut jm = BatchlessJobManager::new();
     let procs = parsed_full_test_output();
-    assert!(jm.job_id_from_pid(82554, &procs) == 82329); // firefox subprocess -> firefox, b/c firefox is below session leader
-    assert!(jm.job_id_from_pid(82329, &procs) == 82329); // firefox -> firefox
-    assert!(jm.job_id_from_pid(1, &procs) == 1); // init
-    assert!(jm.job_id_from_pid(1805, &procs) == 1805); // sd-pam -> sd-pam, b/c 1804 is session leader
-    assert!(jm.job_id_from_pid(232, &procs) == 0); // session 0
-    assert!(jm.job_id_from_pid(74536, &procs) == 74536); // shell
-    assert!(jm.job_id_from_pid(2305, &procs) == 2225); // ibus-extension- -> ibus-daemon, b/c ibus-daemon is below session leader
+    assert!(jm.job_id_from_pid(205415, &procs) == 205408);
     assert!(jm.job_id_from_pid(200, &procs) == 0); // lost process
-    assert!(jm.job_id_from_pid(80199, &procs) == 1823); // lost parent process
 }
 
-// Generated from an old PS output and subsequently munged.
+// More data than we need right now, but oh well.
+// ps -x -h -o pid,ppid,pgrp,cmd | awk '{ print "(" $1 ", " $2 ", " $3 ", " "\"" $4 "\")," }'
 #[cfg(test)]
-fn parsed_full_test_output() -> Vec<procfs::Process> {
+fn parsed_full_test_output() -> HashMap<usize, procfs::Process> {
     vec![
-        (1, "systemd", 0, 1),
-        (2, "kthreadd", 0, 0),
-        (3, "rcu_gp", 2, 0),
-        (4, "rcu_par_gp", 2, 0),
-        (5, "slub_flushwq", 2, 0),
-        (6, "netns", 2, 0),
-        (8, "kworker/0:0H-events_highpri", 2, 0),
-        (10, "mm_percpu_wq", 2, 0),
-        (11, "rcu_tasks_kthread", 2, 0),
-        (12, "rcu_tasks_rude_kthread", 2, 0),
-        (13, "rcu_tasks_trace_kthread", 2, 0),
-        (14, "ksoftirqd/0", 2, 0),
-        (15, "rcu_preempt", 2, 0),
-        (16, "migration/0", 2, 0),
-        (17, "idle_inject/0", 2, 0),
-        (19, "cpuhp/0", 2, 0),
-        (20, "cpuhp/1", 2, 0),
-        (21, "idle_inject/1", 2, 0),
-        (22, "migration/1", 2, 0),
-        (23, "ksoftirqd/1", 2, 0),
-        (25, "kworker/1:0H-events_highpri", 2, 0),
-        (26, "cpuhp/2", 2, 0),
-        (27, "idle_inject/2", 2, 0),
-        (28, "migration/2", 2, 0),
-        (29, "ksoftirqd/2", 2, 0),
-        (31, "kworker/2:0H-events_highpri", 2, 0),
-        (32, "cpuhp/3", 2, 0),
-        (33, "idle_inject/3", 2, 0),
-        (34, "migration/3", 2, 0),
-        (35, "ksoftirqd/3", 2, 0),
-        (37, "kworker/3:0H-events_highpri", 2, 0),
-        (38, "cpuhp/4", 2, 0),
-        (39, "idle_inject/4", 2, 0),
-        (40, "migration/4", 2, 0),
-        (41, "ksoftirqd/4", 2, 0),
-        (43, "kworker/4:0H-kblockd", 2, 0),
-        (44, "cpuhp/5", 2, 0),
-        (45, "idle_inject/5", 2, 0),
-        (46, "migration/5", 2, 0),
-        (47, "ksoftirqd/5", 2, 0),
-        (49, "kworker/5:0H-events_highpri", 2, 0),
-        (50, "cpuhp/6", 2, 0),
-        (51, "idle_inject/6", 2, 0),
-        (52, "migration/6", 2, 0),
-        (53, "ksoftirqd/6", 2, 0),
-        (55, "kworker/6:0H-events_highpri", 2, 0),
-        (56, "cpuhp/7", 2, 0),
-        (57, "idle_inject/7", 2, 0),
-        (58, "migration/7", 2, 0),
-        (59, "ksoftirqd/7", 2, 0),
-        (61, "kworker/7:0H-events_highpri", 2, 0),
-        (62, "kdevtmpfs", 2, 0),
-        (63, "inet_frag_wq", 2, 0),
-        (64, "kauditd", 2, 0),
-        (65, "khungtaskd", 2, 0),
-        (67, "oom_reaper", 2, 0),
-        (69, "writeback", 2, 0),
-        (70, "kcompactd0", 2, 0),
-        (71, "ksmd", 2, 0),
-        (72, "khugepaged", 2, 0),
-        (73, "kintegrityd", 2, 0),
-        (74, "kblockd", 2, 0),
-        (75, "blkcg_punt_bio", 2, 0),
-        (78, "tpm_dev_wq", 2, 0),
-        (79, "ata_sff", 2, 0),
-        (81, "md", 2, 0),
-        (82, "edac-poller", 2, 0),
-        (83, "devfreq_wq", 2, 0),
-        (84, "watchdogd", 2, 0),
-        (85, "kworker/0:1H-acpi_thermal_pm", 2, 0),
-        (86, "kswapd0", 2, 0),
-        (87, "ecryptfs-kthread", 2, 0),
-        (93, "kthrotld", 2, 0),
-        (98, "irq/124-pciehp", 2, 0),
-        (99, "irq/125-pciehp", 2, 0),
-        (104, "acpi_thermal_pm", 2, 0),
-        (105, "xenbus_probe", 2, 0),
-        (107, "vfio-irqfd-clea", 2, 0),
-        (108, "mld", 2, 0),
-        (109, "kworker/5:1H-kblockd", 2, 0),
-        (110, "ipv6_addrconf", 2, 0),
-        (115, "kstrp", 2, 0),
-        (121, "zswap-shrink", 2, 0),
-        (170, "charger_manager", 2, 0),
-        (208, "kworker/7:1H-events_highpri", 2, 0),
-        (229, "kworker/3:1H-events_highpri", 2, 0),
-        (231, "nvme-wq", 2, 0),
-        (232, "nvme-reset-wq", 2, 0),
-        (233, "nvme-delete-wq", 2, 0),
-        (238, "irq/173-SYNA30B7:00", 2, 0),
-        (239, "kworker/2:1H-events_highpri", 2, 0),
-        (243, "irq/174-WACF4233:00", 2, 0),
-        (267, "jbd2/nvme0n1p2-8", 2, 0),
-        (268, "ext4-rsv-conver", 2, 0),
-        (303, "kworker/6:1H-kblockd", 2, 0),
-        (308, "systemd-journal", 1, 308),
-        (335, "kworker/4:1H-events_highpri", 2, 0),
-        (336, "kworker/1:1H-events_highpri", 2, 0),
-        (339, "systemd-udevd", 1, 339),
-        (469, "cfg80211", 2, 0),
-        (485, "irq/175-iwlwifi:default_queue", 2, 0),
-        (488, "irq/176-iwlwifi:queue_1", 2, 0),
-        (489, "irq/177-iwlwifi:queue_2", 2, 0),
-        (490, "irq/178-iwlwifi:queue_3", 2, 0),
-        (491, "irq/179-iwlwifi:queue_4", 2, 0),
-        (492, "irq/180-iwlwifi:queue_5", 2, 0),
-        (493, "irq/181-iwlwifi:queue_6", 2, 0),
-        (494, "irq/182-iwlwifi:queue_7", 2, 0),
-        (496, "irq/183-iwlwifi:queue_8", 2, 0),
-        (498, "irq/184-iwlwifi:exception", 2, 0),
-        (512, "systemd-oomd", 1, 512),
-        (513, "systemd-resolve", 1, 513),
-        (514, "systemd-timesyn", 1, 514),
-        (535, "cryptd", 2, 0),
-        (581, "accounts-daemon", 1, 581),
-        (584, "acpid", 1, 584),
-        (587, "avahi-daemon", 1, 587),
-        (589, "cron", 1, 589),
-        (590, "dbus-daemon", 1, 590),
-        (592, "NetworkManager", 1, 592),
-        (602, "irqbalance", 1, 602),
-        (616, "networkd-dispat", 1, 616),
-        (617, "polkitd", 1, 617),
-        (618, "power-profiles-", 1, 618),
-        (619, "rsyslogd", 1, 619),
-        (621, "snapd", 1, 621),
-        (626, "switcheroo-cont", 1, 626),
-        (643, "systemd-logind", 1, 643),
-        (654, "thermald", 1, 654),
-        (655, "udisksd", 1, 655),
-        (677, "wpa_supplicant", 1, 677),
-        (687, "avahi-daemon", 587, 587),
-        (719, "ModemManager", 1, 719),
-        (722, "boltd", 1, 722),
-        (751, "unattended-upgr", 1, 751),
-        (757, "gdm3", 1, 757),
-        (761, "iio-sensor-prox", 1, 761),
-        (792, "bluetoothd", 1, 792),
-        (799, "card0-crtc0", 2, 0),
-        (800, "card0-crtc1", 2, 0),
-        (801, "card0-crtc2", 2, 0),
-        (802, "card0-crtc3", 2, 0),
-        (960, "irq/207-AudioDSP", 2, 0),
-        (1079, "rtkit-daemon", 1, 1079),
-        (1088, "upowerd", 1, 1088),
-        (1352, "packagekitd", 1, 1352),
-        (1523, "colord", 1, 1523),
-        (1618, "kerneloops", 1, 1618),
-        (1622, "kerneloops", 1, 1622),
-        (1789, "gdm-session-wor", 757, 757),
-        (1804, "systemd", 1, 1804),
-        (1805, "(sd-pam)", 1804, 1804),
-        (1811, "pipewire", 1804, 1811),
-        (1812, "pipewire-media-", 1804, 1812),
-        (1813, "pulseaudio", 1804, 1813),
-        (1823, "dbus-daemon", 1804, 1823),
-        (1825, "gnome-keyring-d", 1, 1824),
-        (1834, "gvfsd", 1804, 1834),
-        (1840, "gvfsd-fuse", 1804, 1834),
-        (1855, "xdg-document-po", 1804, 1855),
-        (1859, "xdg-permission-", 1804, 1859),
-        (1865, "fusermount3", 1855, 1865),
-        (1884, "tracker-miner-f", 1804, 1884),
-        (1892, "krfcommd", 2, 0),
-        (1894, "gvfs-udisks2-vo", 1804, 1894),
-        (1899, "gvfs-mtp-volume", 1804, 1899),
-        (1903, "gvfs-goa-volume", 1804, 1903),
-        (1907, "goa-daemon", 1804, 1823),
-        (1914, "goa-identity-se", 1804, 1823),
-        (1916, "gvfs-afc-volume", 1804, 1916),
-        (1925, "gvfs-gphoto2-vo", 1804, 1925),
-        (1938, "gdm-wayland-ses", 1789, 1938),
-        (1943, "gnome-session-b", 1938, 1938),
-        (1985, "gnome-session-c", 1804, 1985),
-        (1997, "gnome-session-b", 1804, 1997),
-        (2019, "gnome-shell", 1804, 2019),
-        (2020, "at-spi-bus-laun", 1997, 1997),
-        (2028, "dbus-daemon", 2020, 1997),
-        (2136, "gvfsd-metadata", 1804, 2136),
-        (2144, "gnome-shell-cal", 1804, 1823),
-        (2150, "evolution-sourc", 1804, 2150),
-        (2163, "dconf-service", 1804, 2163),
-        (2168, "evolution-calen", 1804, 2168),
-        (2183, "evolution-addre", 1804, 2183),
-        (2198, "gjs", 1804, 1823),
-        (2200, "at-spi2-registr", 1804, 1997),
-        (2208, "gvfsd-trash", 1834, 1834),
-        (2222, "sh", 1804, 2222),
-        (2223, "gsd-a11y-settin", 1804, 2223),
-        (2225, "ibus-daemon", 2222, 2222),
-        (2226, "gsd-color", 1804, 2226),
-        (2229, "gsd-datetime", 1804, 2229),
-        (2231, "gsd-housekeepin", 1804, 2231),
-        (2232, "gsd-keyboard", 1804, 2232),
-        (2233, "gsd-media-keys", 1804, 2233),
-        (2234, "gsd-power", 1804, 2234),
-        (2236, "gsd-print-notif", 1804, 2236),
-        (2238, "gsd-rfkill", 1804, 2238),
-        (2239, "gsd-screensaver", 1804, 2239),
-        (2240, "gsd-sharing", 1804, 2240),
-        (2241, "gsd-smartcard", 1804, 2241),
-        (2242, "gsd-sound", 1804, 2242),
-        (2243, "gsd-wacom", 1804, 2243),
-        (2303, "ibus-memconf", 2225, 2222),
-        (2305, "ibus-extension-", 2225, 2222),
-        (2308, "ibus-portal", 1804, 1823),
-        (2311, "evolution-alarm", 1997, 1997),
-        (2319, "gsd-disk-utilit", 1997, 1997),
-        (2375, "snap-store", 1804, 1997),
-        (2417, "ibus-engine-sim", 2225, 2222),
-        (2465, "gsd-printer", 1804, 2236),
-        (2520, "xdg-desktop-por", 1804, 2520),
-        (2530, "xdg-desktop-por", 1804, 2530),
-        (2555, "gjs", 1804, 1823),
-        (2573, "xdg-desktop-por", 1804, 2573),
-        (2636, "fwupd", 1, 2636),
-        (2656, "snapd-desktop-i", 1804, 2656),
-        (2734, "snapd-desktop-i", 2656, 2656),
-        (3325, "Xwayland", 2019, 2019),
-        (3344, "gsd-xsettings", 1804, 3344),
-        (3375, "ibus-x11", 1804, 3344),
-        (3884, "snap", 1804, 1823),
-        (5131, "update-notifier", 1997, 1997),
-        (7780, "gvfsd-http", 1834, 1834),
-        (9221, "gnome-terminal-", 1804, 9221),
-        (9239, "bash", 9221, 9239),
-        (11438, "obsidian", 2019, 2019),
-        (11495, "obsidian", 11438, 2019),
-        (11496, "obsidian", 11438, 2019),
-        (11526, "obsidian", 11495, 2019),
-        (11531, "obsidian", 11438, 2019),
-        (11542, "obsidian", 11438, 2019),
-        (11543, "obsidian", 11438, 2019),
-        (12887, "ssh-agent", 1825, 1824),
-        (74536, "bash", 9221, 74536),
-        (80195, "gnome-calendar", 1804, 1823),
-        (80199, "seahorse", 200, 1823),
-        (82329, "firefox", 2019, 2019),
-        (82497, "Socket procfs::Process", 82329, 2019),
-        (82516, "Privileged Cont", 82329, 2019),
-        (82554, "Isolated Web Co", 82329, 2019),
-        (82558, "Isolated Web Co", 82329, 2019),
-        (82562, "Isolated Web Co", 82329, 2019),
-        (82572, "Isolated Web Co", 82329, 2019),
-        (82584, "Isolated Web Co", 82329, 2019),
-        (82605, "Isolated Web Co", 82329, 2019),
-        (82631, "Isolated Web Co", 82329, 2019),
-        (82652, "Isolated Web Co", 82329, 2019),
-        (82680, "Isolated Web Co", 82329, 2019),
-        (82732, "Isolated Web Co", 82329, 2019),
-        (83002, "WebExtensions", 82329, 2019),
-        (83286, "Isolated Web Co", 82329, 2019),
-        (83326, "Isolated Web Co", 82329, 2019),
-        (83332, "RDD procfs::Process", 82329, 2019),
-        (83340, "Utility procfs::Process", 82329, 2019),
-        (83618, "Isolated Web Co", 82329, 2019),
-        (83689, "Isolated Web Co", 82329, 2019),
-        (83925, "Isolated Web Co", 82329, 2019),
-        (84013, "Isolated Web Co", 82329, 2019),
-        (84177, "Isolated Web Co", 82329, 2019),
-        (96883, "Isolated Web Co", 82329, 2019),
-        (97718, "Isolated Web Co", 82329, 2019),
-        (99395, "Isolated Web Co", 82329, 2019),
-        (99587, "Isolated Web Co", 82329, 2019),
-        (103356, "Isolated Web Co", 82329, 2019),
-        (103359, "Isolated Web Co", 82329, 2019),
-        (103470, "file:// Content", 82329, 2019),
-        (104433, "Isolated Web Co", 82329, 2019),
-        (104953, "Isolated Web Co", 82329, 2019),
-        (116260, "Isolated Web Co", 82329, 2019),
-        (116296, "Isolated Web Co", 82329, 2019),
-        (116609, "Isolated Web Co", 82329, 2019),
-        (116645, "Isolated Web Co", 82329, 2019),
-        (116675, "Isolated Web Co", 82329, 2019),
-        (116997, "Isolated Web Co", 82329, 2019),
-        (119104, "Isolated Web Co", 82329, 2019),
-        (119151, "Isolated Web Co", 82329, 2019),
-        (128778, "emacs", 2019, 2019),
-        (132391, "Isolated Web Co", 82329, 2019),
-        (133097, "Isolated Web Co", 82329, 2019),
-        (134154, "Isolated Web Co", 82329, 2019),
-        (135609, "Isolated Web Co", 82329, 2019),
-        (136169, "kworker/u17:1-i915_flip", 2, 0),
-        (140722, "Isolated Web Co", 82329, 2019),
-        (142642, "kworker/u17:0-i915_flip", 2, 0),
-        (144346, "kworker/1:1-events", 2, 0),
-        (144602, "kworker/u16:57-events_unbound", 2, 0),
-        (144609, "kworker/u16:64-events_power_efficient", 2, 0),
-        (144624, "irq/185-mei_me", 2, 0),
-        (144736, "cupsd", 1, 144736),
-        (144754, "cups-browsed", 1, 144754),
-        (145490, "gjs", 2019, 2019),
-        (145716, "kworker/7:2-events", 2, 0),
-        (146289, "kworker/u16:0-events_power_efficient", 2, 0),
-        (146290, "kworker/6:1-events", 2, 0),
-        (146342, "kworker/2:1-events", 2, 0),
-        (146384, "kworker/5:0-events", 2, 0),
-        (146735, "kworker/0:0-events", 2, 0),
-        (146791, "kworker/1:2-events", 2, 0),
-        (147017, "kworker/4:2-events", 2, 0),
-        (147313, "kworker/3:2-events", 2, 0),
-        (147413, "kworker/7:0-mm_percpu_wq", 2, 0),
-        (147421, "kworker/6:2-inet_frag_wq", 2, 0),
-        (147709, "kworker/2:2-events", 2, 0),
-        (147914, "kworker/5:2-events", 2, 0),
-        (147916, "kworker/4:0-events", 2, 0),
-        (147954, "kworker/1:3-mm_percpu_wq", 2, 0),
-        (148064, "kworker/3:0-events", 2, 0),
-        (148065, "kworker/0:2-events", 2, 0),
-        (148141, "kworker/7:1-events", 2, 0),
-        (148142, "kworker/u17:2", 2, 0),
-        (148173, "kworker/6:0-events", 2, 0),
-        (148253, "kworker/2:0", 2, 0),
-        (148259, "Isolated Servic", 82329, 2019),
-        (148284, "kworker/u16:1-events_power_efficient", 2, 0),
-        (148286, "kworker/4:1-events_freezable", 2, 0),
-        (148299, "Web Content", 82329, 2019),
-        (148301, "Web Content", 82329, 2019),
-        (148367, "kworker/3:1-events", 2, 0),
-        (148371, "kworker/5:1-events", 2, 0),
-        (148378, "Web Content", 82329, 2019),
-        (148406, "ps", 9239, 9239),
+        (205060, 1, 205060, "/usr/lib/systemd/systemd"),
+        (205074, 205060, 205060, "(sd-pam)"),
+        (205095, 1, 205094, "/usr/bin/gnome-keyring-daemon"),
+        (205120, 205004, 205120, "/usr/libexec/gdm-wayland-session"),
+        (205129, 205060, 205129, "/usr/bin/dbus-broker-launch"),
+        (205131, 205129, 205129, "dbus-broker"),
+        (205135, 205120, 205120, "/usr/libexec/gnome-session-binary"),
+        (205225, 205060, 205225, "/usr/libexec/gnome-session-ctl"),
+        (205228, 205060, 205228, "/usr/libexec/uresourced"),
+        (205231, 205060, 205231, "/usr/libexec/gnome-session-binary"),
+        (205236, 205060, 205236, "/usr/libexec/gvfsd"),
+        (205245, 205060, 205236, "/usr/libexec/gvfsd-fuse"),
+        (205259, 205060, 205259, "/usr/bin/pipewire"),
+        (205261, 205060, 205261, "/usr/bin/wireplumber"),
+        (205310, 205060, 205310, "/usr/bin/gnome-shell"),
+        (205408, 205060, 205408, "/usr/libexec/at-spi-bus-launcher"),
+        (205414, 205408, 205408, "/usr/bin/dbus-broker-launch"),
+        (205415, 205414, 205408, "dbus-broker"),
+        (205417, 205060, 205417, "/usr/libexec/at-spi2-registryd"),
+        (205426, 205060, 205426, "/usr/libexec/gnome-shell-calendar-server"),
+        (205435, 205060, 205435, "/usr/libexec/xdg-permission-store"),
+        (205440, 205060, 205440, "/usr/libexec/dconf-service"),
+        (205451, 205060, 205451, "/usr/libexec/evolution-source-registry"),
+        (205465, 205060, 205465, "/usr/bin/gjs"),
+        (205470, 205060, 205470, "/usr/bin/ibus-daemon"),
+        (205471, 205060, 205471, "/usr/libexec/gsd-a11y-settings"),
+        (205477, 205060, 205477, "/usr/libexec/gsd-color"),
+        (205479, 205060, 205479, "/usr/libexec/gsd-datetime"),
+        (205487, 205060, 205487, "/usr/libexec/gsd-housekeeping"),
+        (205490, 205060, 205490, "/usr/libexec/gsd-keyboard"),
+        (205495, 205060, 205495, "/usr/libexec/gsd-media-keys"),
+        (205512, 205231, 205231, "/usr/bin/gnome-software"),
+        (205515, 205060, 205515, "/usr/libexec/gsd-power"),
+        (205520, 205060, 205520, "/usr/libexec/gsd-print-notifications"),
+        (205527, 205060, 205527, "/usr/libexec/gsd-rfkill"),
+        (205530, 205060, 205530, "/usr/libexec/gsd-screensaver-proxy"),
+        (205532, 205060, 205532, "/usr/libexec/gsd-sharing"),
+        (205533, 205231, 205231, "/usr/libexec/gsd-disk-utility-notify"),
+        (205545, 205060, 205545, "/usr/libexec/gsd-smartcard"),
+        (205560, 205060, 205560, "/usr/libexec/gsd-sound"),
+        (205567, 205231, 205231, "/usr/libexec/evolution-data-server/evolution-alarm-notify"),
+        (205569, 205060, 205569, "/usr/libexec/gsd-usb-protection"),
+        (205578, 205060, 205578, "/usr/libexec/gsd-wacom"),
+        (205652, 205470, 205470, "/usr/libexec/ibus-dconf"),
+        (205653, 205470, 205470, "/usr/libexec/ibus-extension-gtk3"),
+        (205677, 205060, 205677, "/usr/libexec/ibus-portal"),
+        (205678, 205060, 205678, "/usr/bin/abrt-applet"),
+        (205679, 205060, 205679, "/usr/libexec/goa-daemon"),
+        (205683, 205060, 205683, "/usr/bin/gjs"),
+        (205687, 205060, 205687, "/usr/libexec/evolution-calendar-factory"),
+        (205688, 205060, 205688, "/usr/libexec/gvfs-udisks2-volume-monitor"),
+        (205730, 205060, 205730, "/usr/libexec/gvfs-mtp-volume-monitor"),
+        (205747, 205060, 205747, "/usr/libexec/goa-identity-service"),
+        (205750, 205060, 205750, "/usr/libexec/evolution-addressbook-factory"),
+        (205761, 205060, 205761, "/usr/libexec/gvfs-gphoto2-volume-monitor"),
+        (205778, 205060, 205778, "/usr/libexec/gvfs-goa-volume-monitor"),
+        (205792, 205060, 205792, "/usr/libexec/gvfs-afc-volume-monitor"),
+        (205834, 205060, 205520, "/usr/libexec/gsd-printer"),
+        (205855, 205470, 205470, "/usr/libexec/ibus-engine-simple"),
+        (205868, 205060, 205868, "/usr/bin/pipewire-pulse"),
+        (205898, 205060, 205898, "/usr/libexec/xdg-desktop-portal"),
+        (205916, 205060, 205916, "/usr/libexec/xdg-document-portal"),
+        (205947, 205060, 205947, "/usr/libexec/xdg-desktop-portal-gnome"),
+        (206013, 205060, 206013, "/usr/libexec/xdg-desktop-portal-gtk"),
+        (206065, 205060, 206065, "/usr/libexec/gvfsd-metadata"),
+        (206124, 205310, 205310, "/usr/lib64/firefox/firefox"),
+        (206145, 205060, 206145, "/usr/libexec/cgroupify"),
+        (206230, 206124, 205310, "/usr/lib64/firefox/firefox"),
+        (206252, 206124, 205310, "/usr/lib64/firefox/firefox"),
+        (206278, 206124, 205310, "/usr/lib64/firefox/firefox"),
+        (206327, 206124, 205310, "/usr/lib64/firefox/firefox"),
+        (206331, 206124, 205310, "/usr/lib64/firefox/firefox"),
+        (206334, 206124, 205310, "/usr/lib64/firefox/firefox"),
+        (206346, 206124, 205310, "/usr/lib64/firefox/firefox"),
+        (206353, 206124, 205310, "/usr/lib64/firefox/firefox"),
+        (206374, 206124, 205310, "/usr/lib64/firefox/firefox"),
+        (206396, 206124, 205310, "/usr/lib64/firefox/firefox"),
+        (206421, 206124, 205310, "/usr/lib64/firefox/firefox"),
+        (206442, 206124, 205310, "/usr/lib64/firefox/firefox"),
+        (206619, 206124, 205310, "/usr/lib64/firefox/firefox"),
+        (206624, 206124, 205310, "/usr/lib64/firefox/firefox"),
+        (206641, 206124, 205310, "/usr/lib64/firefox/firefox"),
+        (206785, 206124, 205310, "/usr/lib64/firefox/firefox"),
+        (207040, 206124, 205310, "/usr/lib64/firefox/firefox"),
+        (207088, 205470, 205470, "/usr/bin/python3"),
+        (207209, 205060, 207209, "/usr/libexec/gnome-terminal-server"),
+        (207228, 207209, 207228, "bash"),
+        (207437, 205310, 205310, "/usr/bin/emacs"),
+        (207440, 205310, 205310, "/usr/bin/Xwayland"),
+        (207445, 205060, 207445, "/usr/libexec/gsd-xsettings"),
+        (207463, 205060, 207445, "/usr/libexec/ibus-x11"),
+        (207476, 205310, 205310, "/usr/libexec/mutter-x11-frames"),
+        (207591, 206124, 205310, "/usr/lib64/firefox/firefox"),
+        (208077, 206124, 205310, "/usr/lib64/firefox/firefox"),
+        (208121, 206124, 205310, "/usr/lib64/firefox/firefox"),
+        (211068, 206124, 205310, "/usr/lib64/firefox/firefox"),
+        (211133, 206124, 205310, "/usr/lib64/firefox/firefox"),
+        (211281, 206124, 205310, "/usr/lib64/firefox/firefox"),
+        (211375, 206124, 205310, "/usr/lib64/firefox/firefox"),
+        (211404, 207228, 211404, "ps"),
+        (211405, 207228, 211404, "awk"),
     ]
     .iter()
-    .map(|(pid, command, ppid, session)| procfs::Process {
-        pid: *pid,
-        command: command.to_string(),
-        ppid: *ppid,
-        session: *session,
-        cpu_pct: 0.0,
-        cputime_sec: 0,
-        mem_pct: 0.0,
-        mem_size_kib: 0,
-        rssanon_kib: 0,
-        uid: 0,
-        user: "user".to_string(),
-    })
-    .collect::<Vec<procfs::Process>>()
+    .map(|(pid, ppid, pgrp, command)|
+         (*pid,
+          procfs::Process {
+              pid: *pid,
+              ppid: *ppid,
+              pgrp: *pgrp,
+              command: command.to_string(),
+              // The following are wrong but we don't need them now
+              cpu_pct: 0.0,
+              cputime_sec: 0,
+              mem_pct: 0.0,
+              mem_size_kib: 0,
+              rssanon_kib: 0,
+              uid: 0,
+              user: "user".to_string(),
+              has_children: false,
+          }))
+    .collect::<HashMap<usize, procfs::Process>>()
 }
diff --git a/src/jobs.rs b/src/jobs.rs
index 4d38c1a..76c2bbb 100644
--- a/src/jobs.rs
+++ b/src/jobs.rs
@@ -2,11 +2,12 @@
 // queue (if any) away from the rest of sonar.
 
 use crate::procfs;
+use std::collections::HashMap;
 
 pub trait JobManager {
     // Compute a job ID from a process ID.
     //
-    // There's an assumption here that the process slice is always the same for all lookups
+    // There's an assumption here that the process map is always the same for all lookups
     // performed on a particular instance of JobManager.
-    fn job_id_from_pid(&mut self, pid: usize, processes: &[procfs::Process]) -> usize;
+    fn job_id_from_pid(&mut self, pid: usize, processes: &HashMap<usize, procfs::Process>) -> usize;
 }
diff --git a/src/main.rs b/src/main.rs
index 2388261..5504e3b 100644
--- a/src/main.rs
+++ b/src/main.rs
@@ -17,6 +17,7 @@ mod users;
 mod util;
 
 const TIMEOUT_SECONDS: u64 = 5; // For subprocesses
+const USAGE_ERROR: i32 = 2;     // clap, Python, Go
 
 enum Commands {
     /// Take a snapshot of the currently running processes
@@ -166,6 +167,17 @@ fn command_line() -> Commands {
                         break;
                     }
                 }
+
+                #[cfg(debug_assertions)]
+                let allow_incompatible = std::env::var("SONARTEST_ROLLUP").is_ok();
+
+                #[cfg(not(debug_assertions))]
+                let allow_incompatible = false;
+
+                if rollup && batchless && !allow_incompatible {
+                    eprintln!("--rollup and --batchless are incompatible");
+                    std::process::exit(USAGE_ERROR);
+                }
                 return Commands::PS {
                     batchless,
                     rollup,
@@ -228,7 +240,8 @@ Options for `ps`:
   --batchless
       Synthesize a job ID from the process tree in which a process finds itself
   --rollup
-      Merge process records that have the same job ID and command name
+      Merge process records that have the same job ID and command name (not
+      compatible with --batchless)
   --min-cpu-percent percentage
       Include records for jobs that have on average used at least this
       percentage of CPU, note this is nonmonotonic [default: none]
@@ -250,5 +263,5 @@ Options for `ps`:
 ",
     );
     let _ = out.flush();
-    std::process::exit(if is_error { 2 } else { 0 });
+    std::process::exit(if is_error { USAGE_ERROR } else { 0 });
 }
diff --git a/src/procfs.rs b/src/procfs.rs
index 7974bf7..9975d57 100644
--- a/src/procfs.rs
+++ b/src/procfs.rs
@@ -1,11 +1,13 @@
 /// Collect CPU process information without GPU information, from files in /proc.
 use crate::procfsapi::{self, parse_usize_field};
 
-use std::collections::HashMap;
+use std::collections::{HashMap, HashSet};
 
 #[derive(PartialEq, Debug)]
 pub struct Process {
     pub pid: usize,
+    pub ppid: usize,
+    pub pgrp: usize,
     pub uid: usize,
     pub user: String, // _noinfo_<uid> if name unobtainable
     pub cpu_pct: f64,
@@ -14,8 +16,7 @@ pub struct Process {
     pub mem_size_kib: usize,
     pub rssanon_kib: usize,
     pub command: String,
-    pub ppid: usize,
-    pub session: usize,
+    pub has_children: bool,
 }
 
 /// Read the /proc/meminfo file from the fs and return the value for total installed memory.
@@ -88,8 +89,8 @@ fn i32_field(l: &str) -> Result<i32, String> {
     }
 }
 
-/// Obtain process information via /proc and return a vector of structures with all the information
-/// we need.  In the returned vector, pids uniquely tag the records.
+/// Obtain process information via /proc and return a hashmap of structures with all the information
+/// we need, keyed by pid.  Pids uniquely tag the records.
 ///
 /// This returns Ok(data) on success, otherwise Err(msg).
 ///
@@ -102,7 +103,7 @@ fn i32_field(l: &str) -> Result<i32, String> {
 pub fn get_process_information(
     fs: &dyn procfsapi::ProcfsAPI,
     memtotal_kib: usize,
-) -> Result<Vec<Process>, String> {
+) -> Result<HashMap<usize, Process>, String> {
     // The boot time is the `btime` field of /proc/stat.  It is measured in seconds since epoch.  We
     // need this to compute the process's real time, which we need to compute ps-compatible cpu
     // utilization.
@@ -134,7 +135,8 @@ pub fn get_process_information(
     // Collect remaining system data from /proc/{pid}/stat for the enumerated pids.
 
     let kib_per_page = fs.page_size_in_kib();
-    let mut result = vec![];
+    let mut result = HashMap::<usize, Process>::new();
+    let mut ppids = HashSet::<usize>::new();
     let mut user_table = UserTable::new();
     let clock_ticks_per_sec = fs.clock_ticks_per_sec() as f64;
     if clock_ticks_per_sec == 0.0 {
@@ -148,7 +150,7 @@ pub fn get_process_information(
         let bsdtime_ticks;
         let mut realtime_ticks;
         let ppid;
-        let sess;
+        let pgrp;
         let mut comm;
         let utime_ticks;
         let stime_ticks;
@@ -204,7 +206,7 @@ pub fn get_process_information(
             }
 
             ppid = parse_usize_field(&fields, 1, &line, "stat", pid, "ppid")?;
-            sess = parse_usize_field(&fields, 3, &line, "stat", pid, "sess")?;
+            pgrp = parse_usize_field(&fields, 2, &line, "stat", pid, "pgrp")?;
 
             // Generally we want to record cumulative self+child time.  The child time we read will
             // be for children that have terminated and have been wait()ed for.  The logic is that
@@ -348,8 +350,10 @@ pub fn get_process_information(
             99.9,
         );
 
-        result.push(Process {
+        result.insert(pid, Process {
             pid,
+            ppid,
+            pgrp,
             uid: uid as usize,
             user: user_table.lookup(fs, uid),
             cpu_pct: pcpu_formatted,
@@ -357,10 +361,15 @@ pub fn get_process_information(
             cputime_sec,
             mem_size_kib: size_kib,
             rssanon_kib,
-            ppid,
-            session: sess,
             command: comm,
+            has_children: false,
         });
+        ppids.insert(ppid);
+    }
+
+    // Mark the processes that have children.
+    for (_, p) in result.iter_mut() {
+        p.has_children = ppids.contains(&p.pid);
     }
 
     Ok(result)
@@ -506,15 +515,16 @@ DirectMap1G:    11534336 kB
 
     let fs = procfsapi::MockFS::new(files, pids, users, now);
     let memtotal_kib = get_memtotal_kib(&fs).unwrap();
-    let info = get_process_information(&fs, memtotal_kib).unwrap();
+    let mut info = get_process_information(&fs, memtotal_kib).unwrap();
     assert!(info.len() == 1);
-    let p = &info[0];
+    let mut xs = info.drain();
+    let p = xs.next().expect("Something").1;
     assert!(p.pid == 4018); // from enumeration of /proc
     assert!(p.uid == 1000); // ditto
     assert!(p.user == "zappa"); // from getent
     assert!(p.command == "firefox"); // field(/proc/4018/stat, 2)
     assert!(p.ppid == 2190); // field(/proc/4018/stat, 4)
-    assert!(p.session == 2189); // field(/proc/4018/stat, 6)
+    assert!(p.pgrp == 2189); // field(/proc/4018/stat, 5)
 
     let now_time = now as f64;
     let now_ticks = now_time * ticks_per_sec;
@@ -572,15 +582,21 @@ pub fn procfs_dead_and_undead_test() {
 
     let fs = procfsapi::MockFS::new(files, pids, users, procfsapi::unix_now());
     let memtotal_kib = get_memtotal_kib(&fs).unwrap();
-    let info = get_process_information(&fs, memtotal_kib).unwrap();
+    let mut info = get_process_information(&fs, memtotal_kib).unwrap();
 
     // 4020 should be dropped - it's dead
     assert!(info.len() == 2);
 
-    assert!(info[0].pid == 4018);
-    assert!(info[0].command == "firefox");
-    assert!(info[1].pid == 4019);
-    assert!(info[1].command == "firefox <defunct>");
+    let mut xs = info.drain();
+    let mut p = xs.next().expect("Something").1;
+    let mut q = xs.next().expect("Something else").1;
+    if p.pid > q.pid {
+        (p, q) = (q, p);
+    }
+    assert!(p.pid == 4018);
+    assert!(p.command == "firefox");
+    assert!(q.pid == 4019);
+    assert!(q.command == "firefox <defunct>");
 }
 
 #[test]
diff --git a/src/ps.rs b/src/ps.rs
index 6e8a7fa..2a5f72f 100644
--- a/src/ps.rs
+++ b/src/ps.rs
@@ -66,8 +66,10 @@ struct ProcInfo<'a> {
     _uid: usize,
     command: &'a str,
     pid: Pid,
+    ppid: Pid,
     rolledup: usize,
     is_system_job: bool,
+    has_children: bool,
     job_id: usize,
     cpu_percentage: f64,
     cputime_sec: usize,
@@ -108,6 +110,8 @@ fn add_proc_info<'a, F>(
     uid: usize,
     command: &'a str,
     pid: Pid,
+    ppid: Pid,
+    has_children: bool,
     cpu_percentage: f64,
     cputime_sec: usize,
     mem_percentage: f64,
@@ -133,14 +137,18 @@ fn add_proc_info<'a, F>(
             e.gpu_percentage += gpu_percentage;
             e.gpu_mem_percentage += gpu_mem_percentage;
             e.gpu_mem_size_kib += gpu_mem_size_kib;
+            assert!(has_children == e.has_children);
+            assert!(ppid == e.ppid);
         })
         .or_insert(ProcInfo {
             user,
             _uid: uid,
             command,
             pid,
+            ppid,
             rolledup: 0,
             is_system_job: uid < 1000,
+            has_children,
             job_id: lookup_job_by_pid(pid),
             cpu_percentage,
             cputime_sec,
@@ -292,13 +300,13 @@ fn do_create_snapshot(
 
     // The table of users is needed to get GPU information, see comments at UserTable.
     let mut user_by_pid = UserTable::new();
-    for proc in pprocinfo_output {
+    for (_, proc) in pprocinfo_output {
         user_by_pid.insert(proc.pid, (&proc.user, proc.uid));
     }
 
     let mut lookup_job_by_pid = |pid: Pid| jobs.job_id_from_pid(pid, pprocinfo_output);
 
-    for proc in pprocinfo_output {
+    for (_, proc) in pprocinfo_output {
         add_proc_info(
             &mut proc_by_pid,
             &mut lookup_job_by_pid,
@@ -306,6 +314,8 @@ fn do_create_snapshot(
             proc.uid,
             &proc.command,
             proc.pid,
+            proc.ppid,
+            proc.has_children,
             proc.cpu_pct,
             proc.cputime_sec,
             proc.mem_pct,
@@ -337,6 +347,12 @@ fn do_create_snapshot(
         }
         Ok(ref nvidia_output) => {
             for proc in nvidia_output {
+                let (ppid, has_children) =
+                    if let Some(process) = pprocinfo_output.get(&proc.pid) {
+                        (process.ppid, process.has_children)
+                    } else {
+                        (1, true)
+                    };
                 add_proc_info(
                     &mut proc_by_pid,
                     &mut lookup_job_by_pid,
@@ -344,6 +360,8 @@ fn do_create_snapshot(
                     proc.uid,
                     &proc.command,
                     proc.pid,
+                    ppid,
+                    has_children,
                     0.0, // cpu_percentage
                     0,   // cputime_sec
                     0.0, // mem_percentage
@@ -372,6 +390,12 @@ fn do_create_snapshot(
         }
         Ok(ref amd_output) => {
             for proc in amd_output {
+                let (ppid, has_children) =
+                    if let Some(process) = pprocinfo_output.get(&proc.pid) {
+                        (process.ppid, process.has_children)
+                    } else {
+                        (1, true)
+                    };
                 add_proc_info(
                     &mut proc_by_pid,
                     &mut lookup_job_by_pid,
@@ -379,6 +403,8 @@ fn do_create_snapshot(
                     proc.uid,
                     &proc.command,
                     proc.pid,
+                    ppid,
+                    has_children,
                     0.0, // cpu_percentage
                     0,   // cputime_sec
                     0.0, // mem_percentage
@@ -421,17 +447,25 @@ fn do_create_snapshot(
     };
 
     let mut candidates = if opts.rollup {
-        // This is a little complicated because jobs with job_id 0 cannot be rolled up.
+        // This is a little complicated because processes with job_id 0 or processes that have
+        // subprocesses cannot be rolled up, nor can we roll up processes with different ppid.
+        //
+        // The reason we cannot roll up processes with job_id 0 is that we don't know that they are
+        // related at all - 0 means "no information".
+        //
+        // The reason we cannot roll up processes with children or processes with different ppids is
+        // that this would break subsequent processing - it would make it impossible to build a
+        // sensible process tree from the sample data.
         //
         // - There is an array `rolledup` of ProcInfo nodes that represent rolled-up data
         //
-        // - When the job ID of a job in `proc_by_pid` is zero, the entry in `rolledup` is a copy of
-        //   that job; these jobs cannot be rolled up (this is why it's complicated)
+        // - When the job ID of a process in `proc_by_pid` is zero, or a process has children, the
+        //   entry in `rolledup` is a copy of that job
         //
-        // - Otherwise, the entry in `rolledup` represent rolled-up information for a (job, command)
-        //   pair
+        // - Otherwise, the entry in `rolledup` represent rolled-up information for a
+        //   (jobid,ppid,command) triple
         //
-        // - There is a hash table `index` that maps the (job, command) pair to the entry in
+        // - There is a hash table `index` that maps the (jobid,ppid,command) triple to the entry in
         //   `rolledup`, if any
         //
         // - When we're done rolling up, we print the `rolledup` table.
@@ -441,12 +475,12 @@ fn do_create_snapshot(
         // is probably the right thing.
 
         let mut rolledup = vec![];
-        let mut index = HashMap::<(JobID, &str), usize>::new();
+        let mut index = HashMap::<(JobID, Pid, &str), usize>::new();
         for proc_info in proc_by_pid.values() {
-            if proc_info.job_id == 0 {
+            if proc_info.job_id == 0 || proc_info.has_children {
                 rolledup.push(proc_info.clone());
             } else {
-                let key = (proc_info.job_id, proc_info.command);
+                let key = (proc_info.job_id, proc_info.ppid, proc_info.command);
                 if let Some(x) = index.get(&key) {
                     let p = &mut rolledup[*x];
                     p.cpu_percentage += proc_info.cpu_percentage;
@@ -504,8 +538,10 @@ fn do_create_snapshot(
             _uid: 0,
             command: "_heartbeat_",
             pid: 0,
+            ppid: 0,
             rolledup: 0,
             is_system_job: true,
+            has_children: false,
             job_id: 0,
             cpu_percentage: 0.0,
             cputime_sec: 0,
@@ -614,8 +650,14 @@ fn print_record(
         fields.push(format!("job={}", proc_info.job_id));
     }
     if proc_info.rolledup == 0 && proc_info.pid != 0 {
+        // pid must be 0 for rolledup > 0 as there is no guarantee that there is any fixed
+        // representative pid for a rolled-up set of processes: the set can change from run to run,
+        // and sonar has no history.
         fields.push(format!("pid={}", proc_info.pid));
     }
+    if proc_info.ppid != 0 {
+        fields.push(format!("ppid={}", proc_info.ppid));
+    }
     if proc_info.cpu_percentage != 0.0 {
         fields.push(format!("cpu%={}", three_places(proc_info.cpu_percentage)));
     }
diff --git a/src/slurm.rs b/src/slurm.rs
index e773758..c542117 100644
--- a/src/slurm.rs
+++ b/src/slurm.rs
@@ -5,11 +5,12 @@ use crate::procfs;
 
 use std::fs::File;
 use std::io::{BufRead, BufReader};
+use std::collections::HashMap;
 
 pub struct SlurmJobManager {}
 
 impl jobs::JobManager for SlurmJobManager {
-    fn job_id_from_pid(&mut self, pid: usize, _processes: &[procfs::Process]) -> usize {
+    fn job_id_from_pid(&mut self, pid: usize, _processes: &HashMap<usize, procfs::Process>) -> usize {
         let slurm_job_id = get_slurm_job_id(pid).unwrap_or_default();
         slurm_job_id.trim().parse::<usize>().unwrap_or_default()
     }


Test patch:
diff --git a/tests/.gitignore b/tests/.gitignore
new file mode 100644
index 0000000..c3b6806
--- /dev/null
+++ b/tests/.gitignore
@@ -0,0 +1,4 @@
+rollup
+rollup2
+rollupchild
+rollupchild2
diff --git a/tests/Makefile b/tests/Makefile
new file mode 100644
index 0000000..8fdae68
--- /dev/null
+++ b/tests/Makefile
@@ -0,0 +1,7 @@
+CFLAGS=-Wall -std=c89
+.PHONY: all
+
+all: rollup rollup2 rollupchild rollupchild2
+
+rollupchild2: rollupchild
+	cp rollupchild rollupchild2
diff --git a/tests/rollup.c b/tests/rollup.c
new file mode 100644
index 0000000..192ea7c
--- /dev/null
+++ b/tests/rollup.c
@@ -0,0 +1,64 @@
+/* Run this with SONARTEST_ROLLUP=1 and --rollup --batchless.
+   If you grep the sonar output for ',cmd=rollup,' there should be 23 lines.
+   Of those, there should be eight that have ',rolledup=1' and none that have any other rollup fields.
+
+   (This code is -std=c89, try to keep it that way.)
+*/
+
+#include <errno.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <unistd.h>
+#include <sys/types.h>
+#include <sys/wait.h>
+
+int main(int argc, char** argv) {
+    long depth;
+    pid_t c1, c2;
+    if (argc < 2) {
+        fprintf(stderr, "Usage: %s depth\n", argv[0]);
+        exit(1);
+    }
+    errno = 0;
+    depth = strtol(argv[1], NULL, 10);
+    if (errno != 0 || depth < 0 || depth > 10) {
+        fprintf(stderr, "Bad depth\n");
+        exit(1);
+    }
+again:
+    /* in Parent */
+    c1 = fork();
+    if (c1 == -1) {
+        perror("Forking child");
+        exit(1);
+    }
+    if (c1 > 0) {
+        /* in Parent */
+        c2 = fork();
+        if (c2 == -1) {
+            perror("Forking child");
+            exit(1);
+        }
+        if (c2 > 0) {
+            /* in Parent */
+            /* printf("Waiting %d\n", getpid()); */
+            wait(NULL);
+            wait(NULL);
+        } else {
+            /* in C2 */
+            if (depth-- > 0) {
+                goto again;
+            }
+            /* printf("Sleeping %d\n", getpid()); */
+            sleep(10);
+        }
+    } else {
+        /* in C1 */
+        if (depth-- > 0) {
+            goto again;
+        }
+        /* printf("Sleeping %d\n", getpid()); */
+        sleep(10);
+    }
+    return 0;
+}
diff --git a/tests/rollup.sh b/tests/rollup.sh
new file mode 100755
index 0000000..7733444
--- /dev/null
+++ b/tests/rollup.sh
@@ -0,0 +1,36 @@
+#!/bin/bash
+#
+# Test these aspects of the process rollup algorithm:
+#  - only leaf processes are rolled up
+#  - only siblings of the same parent are rolled up
+#
+# To do this on a non-slurm system we run --rollup --batchless with an override to allow that.
+#
+# This requires a (probably) 1.6x or later Rust/Cargo toolchain to build Sonar and `make` + any C89
+# or later C compiler to build the C code.
+
+set -e
+
+( cd .. ; cargo build )
+make --quiet
+
+echo "This takes about 10s"
+./rollup 3 &
+sleep 3
+output=$(SONARTEST_ROLLUP=1 ../target/debug/sonar ps --rollup --batchless --exclude-system-jobs)
+matches=$(grep ,cmd=rollup, <<< $output)
+rolled=$(grep ,rolledup=1 <<< $matches)
+rolled2=$(grep ,rolledup= <<< $matches)
+if [[ $(wc -l <<< $matches) != 23 ]]; then
+    echo "Bad number of matching lines"
+    exit 1
+fi
+if [[ $(wc -l <<< $rolled) != 8 ]]; then
+    echo "Bad number of rolled-up lines with value 1"
+    exit 1
+fi
+if [[ $(wc -l <<< $rolled2) != 8 ]]; then
+    echo "Bad number of rolled-up lines - some have a value other than 1"
+    exit 1
+fi
+echo Success
diff --git a/tests/rollup2.c b/tests/rollup2.c
new file mode 100644
index 0000000..97e71d3
--- /dev/null
+++ b/tests/rollup2.c
@@ -0,0 +1,44 @@
+/* Run this with SONARTEST_ROLLUP=1 and --rollup --batchless.
+
+   This will fork off a 9 child processes (rollupchild x 5 and rollupchild2 x 4) that are the
+   same except for the name, all of which will wait 10s.  Sonar, run meanwhile, should rollup
+   the children with the same name only.  Since the rollup field represents n-1 processes,
+   the count for rollupchild should be 4 and for rollupchild2 should be 3.
+
+   (This code is -std=c89, try to keep it that way.)
+*/
+
+#include <errno.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <unistd.h>
+#include <sys/types.h>
+#include <sys/wait.h>
+
+#define TYPE1 5
+#define TYPE2 4
+
+int main(int argc, char** argv) {
+    int i;
+    for ( i=0 ; i < TYPE1+TYPE2 ; i++ ) {
+        switch (fork()) {
+          case -1:
+            perror("Forking child");
+            exit(1);
+          case 0:
+            if (i < TYPE1) {
+                execl("rollupchild", "rollupchild", NULL);
+                fprintf(stderr, "Failed to exec child\n");
+                exit(1);
+            } else {
+                execl("rollupchild2", "rollupchild2", NULL);
+                fprintf(stderr, "Failed to exec child 2\n");
+                exit(1);
+            }
+        }
+    }
+    for ( i=0 ; i < TYPE1+TYPE2 ; i++ ) {
+        wait(NULL);
+    }
+    return 0;
+}
diff --git a/tests/rollup2.sh b/tests/rollup2.sh
new file mode 100755
index 0000000..87111dc
--- /dev/null
+++ b/tests/rollup2.sh
@@ -0,0 +1,23 @@
+#!/bin/bash
+#
+# Test these aspects of the process rollup algorithm:
+#  - only siblings with the same name are rolled up
+#
+# To do this on a non-slurm system we run --rollup --batchless with an override to allow that.
+#
+# This requires a (probably) 1.6x or later Rust/Cargo toolchain to build Sonar and `make` + any C89
+# or later C compiler to build the C code.
+
+set -e
+
+( cd .. ; cargo build )
+make --quiet
+
+echo "This takes about 10s"
+./rollup2 3 &
+sleep 3
+output=$(SONARTEST_ROLLUP=1 ../target/debug/sonar ps --rollup --batchless --exclude-system-jobs)
+# Grep will exit with code 1 if no lines are matched
+matches1=$(grep -E ',cmd=rollupchild,.*,rolledup=4' <<< $output)
+matches2=$(grep -E ',cmd=rollupchild2,.*,rolledup=3' <<< $output)
+echo Success
diff --git a/tests/rollupchild.c b/tests/rollupchild.c
new file mode 100644
index 0000000..0739da3
--- /dev/null
+++ b/tests/rollupchild.c
@@ -0,0 +1,8 @@
+/* See rollup2.c */
+
+#include <unistd.h>
+
+int main(int argc, char** argv) {
+    sleep(10);
+    return 0;
+}