Repository: lkrg-org/lkrg
Pull request #377: Move the CPU metadata dumping under a new config read lock
Author: login: solardiz
Created: 2025-04-19T20:20:21Z | Merged: 2025-04-20T14:11:36Z

and acquire the lock for writing on lkrg.msr_validate changes, which affect whether MSRs are included in the computed hash value.

Fixes #370

### How Has This Been Tested?

I am testing with:

```shell
while :; do sysctl lkrg.msr_validate=1; sysctl lkrg.msr_validate=0; done
```

I also trigger some coredumps in a loop, as this was causing softlocks with my previous attempts at fixing this issue (when I tried acquiring `p_db_lock`).

Linked issue #370: Spurious "Hash of CPU metadata has changed unexpectedly" on msr_validate change
While testing the changes currently in #369, but probably unrelated to them, I triggered the below on Rocky Linux 9.5 in a VM with 4 CPUs (as seen via our remote logging:

```
2025-04-04T00:36:18Z LKRG: STATE: Changing 'profile_validate' from 3 (Heavy) to 4 (Paranoid)
2025-04-04T00:36:26Z LKRG: STATE: Changing 'profile_validate' from 4 (Paranoid) to 2 (Balanced)
2025-04-04T00:36:26Z LKRG: ALERT: DETECT: CPU: Hash of CPU metadata has changed unexpectedly
2025-04-04T00:36:26Z LKRG: ALERT: DETECT: Kernel: 1 checksums changed unexpectedly
2025-04-04T00:36:26Z LKRG: ALERT: BLOCK: Kernel: 1 checksums changed unexpectedly
2025-04-04T00:36:26Z Kernel panic - not syncing: Kernel: 1 checksums changed unexpectedly
2025-04-04T00:36:26Z CPU: 0 PID: 1037682 Comm: kworker/u256:4 Kdump: loaded Tainted: G           OE     -------  ---  5.14.0-503.22.1.el9_5.x86_64 #1
2025-04-04T00:36:26Z Hardware name: oVirt RHEL/RHEL-AV, BIOS 1.16.0-4.module_el8+603+e0ca2c01 04/01/2014
2025-04-04T00:36:26Z Workqueue: events_unbound p_check_integrity [lkrg]
2025-04-04T00:36:26Z Call Trace:
2025-04-04T00:36:26Z  <TASK>
2025-04-04T00:36:26Z  dump_stack_lvl+0x34/0x48
2025-04-04T00:36:26Z  panic+0x107/0x2bb
2025-04-04T00:36:26Z  p_check_integrity.cold+0xa57/0xc49 [lkrg]
2025-04-04T00:36:26Z  ? srso_return_thunk+0x5/0x5f
2025-04-04T00:36:26Z  ? __schedule+0x231/0x550
2025-04-04T00:36:26Z  ? try_to_wake_up+0x69/0x530
2025-04-04T00:36:26Z  process_one_work+0x197/0x380
2025-04-04T00:36:26Z  worker_thread+0x2fe/0x410
2025-04-04T00:36:26Z  ? __pfx_worker_thread+0x10/0x10
2025-04-04T00:36:26Z  kthread+0xe0/0x100
2025-04-04T00:36:26Z  ? __pfx_kthread+0x10/0x10
2025-04-04T00:36:26Z  ret_from_fork+0x2c/0x50
2025-04-04T00:36:26Z  </TASK>
```

I was running `sysctl` commands manually, so it wasn't a stress-test. However, to reproduce this again, I had to run `sysctl` in a shell loop for quite a while until this happened:

```
2025-04-04T00:41:56Z LKRG: ALIVE: LKRG initialized successfully
2025-04-04T00:41:56Z OOM killer enabled.
2025-04-04T00:41:56Z Restarting tasks ... done.
2025-04-04T00:42:20Z LKRG: STATE: Changing 'profile_validate' from 3 (Heavy) to 2 (Balanced)
2025-04-04T00:42:28Z LKRG: STATE: Changing 'profile_validate' from 2 (Balanced) to 1 (Light)
2025-04-04T00:42:32Z LKRG: STATE: Changing 'profile_validate' from 1 (Light) to 0 (Disabled)
2025-04-04T00:42:47Z LKRG: STATE: Changing 'profile_validate' from 0 (Disabled) to 4 (Paranoid)
2025-04-04T00:42:53Z LKRG: STATE: Changing 'profile_validate' from 4 (Paranoid) to 3 (Heavy)
2025-04-04T00:42:58Z LKRG: STATE: Changing 'profile_validate' from 3 (Heavy) to 2 (Balanced)
2025-04-04T00:43:00Z LKRG: STATE: Changing 'profile_validate' from 2 (Balanced) to 1 (Light)
2025-04-04T00:43:02Z LKRG: STATE: Changing 'profile_validate' from 1 (Light) to 0 (Disabled)
2025-04-04T00:43:06Z LKRG: STATE: Changing 'profile_validate' from 0 (Disabled) to 4 (Paranoid)
2025-04-04T00:43:11Z LKRG: STATE: Changing 'profile_validate' from 4 (Paranoid) to 0 (Disabled)
2025-04-04T00:43:17Z LKRG: STATE: Changing 'profile_validate' from 0 (Disabled) to 4 (Paranoid)
2025-04-04T00:43:22Z LKRG: STATE: Changing 'profile_validate' from 4 (Paranoid) to 1 (Light)
2025-04-04T00:43:26Z LKRG: STATE: Changing 'profile_validate' from 1 (Light) to 4 (Paranoid)
2025-04-04T00:43:27Z LKRG: STATE: Changing 'profile_validate' from 4 (Paranoid) to 3 (Heavy)
2025-04-04T00:43:31Z LKRG: STATE: Changing 'profile_validate' from 3 (Heavy) to 4 (Paranoid)
2025-04-04T00:43:32Z LKRG: STATE: Changing 'profile_validate' from 4 (Paranoid) to 2 (Balanced)
2025-04-04T00:44:23Z LKRG: STATE: Changing 'profile_validate' from 2 (Balanced) to 4 (Paranoid)
2025-04-04T00:44:24Z LKRG: STATE: Changing 'profile_validate' from 4 (Paranoid) to 2 (Balanced)
2025-04-04T00:44:25Z LKRG: STATE: Changing 'profile_validate' from 2 (Balanced) to 4 (Paranoid)
2025-04-04T00:44:25Z LKRG: STATE: Changing 'profile_validate' from 4 (Paranoid) to 2 (Balanced)
2025-04-04T00:44:26Z LKRG: STATE: Changing 'profile_validate' from 2 (Balanced) to 4 (Paranoid)
2025-04-04T00:44:27Z LKRG: STATE: Changing 'profile_validate' from 4 (Paranoid) to 2 (Balanced)
2025-04-04T00:44:27Z LKRG: STATE: Changing 'profile_validate' from 2 (Balanced) to 4 (Paranoid)
2025-04-04T00:44:28Z LKRG: STATE: Changing 'profile_validate' from 4 (Paranoid) to 2 (Balanced)
2025-04-04T00:44:34Z LKRG: STATE: Changing 'profile_validate' from 2 (Balanced) to 3 (Heavy)
2025-04-04T00:44:35Z LKRG: STATE: Changing 'profile_validate' from 3 (Heavy) to 4 (Paranoid)
2025-04-04T00:44:36Z LKRG: STATE: Changing 'profile_validate' from 4 (Paranoid) to 2 (Balanced)
2025-04-04T00:44:38Z LKRG: STATE: Changing 'profile_validate' from 2 (Balanced) to 4 (Paranoid)
2025-04-04T00:44:40Z LKRG: STATE: Changing 'profile_validate' from 4 (Paranoid) to 2 (Balanced)
2025-04-04T00:45:03Z LKRG: STATE: Changing 'profile_validate' from 2 (Balanced) to 4 (Paranoid)
2025-04-04T00:45:03Z LKRG: STATE: Changing 'profile_validate' from 4 (Paranoid) to 2 (Balanced)
2025-04-04T00:45:03Z LKRG: STATE: Changing 'profile_validate' from 2 (Balanced) to 4 (Paranoid)
2025-04-04T00:45:03Z LKRG: STATE: Changing 'profile_validate' from 4 (Paranoid) to 2 (Balanced)
2025-04-04T00:45:03Z LKRG: STATE: Changing 'profile_validate' from 2 (Balanced) to 4 (Paranoid)
2025-04-04T00:45:03Z LKRG: STATE: Changing 'profile_validate' from 4 (Paranoid) to 2 (Balanced)
2025-04-04T00:45:03Z LKRG: STATE: Changing 'profile_validate' from 2 (Balanced) to 4 (Paranoid)
2025-04-04T00:45:03Z LKRG: STATE: Changing 'profile_validate' from 4 (Paranoid) to 2 (Balanced)
2025-04-04T00:45:03Z LKRG: STATE: Changing 'profile_validate' from 2 (Balanced) to 4 (Paranoid)
2025-04-04T00:45:03Z LKRG: STATE: Changing 'profile_validate' from 4 (Paranoid) to 2 (Balanced)
2025-04-04T00:45:03Z LKRG: STATE: Changing 'profile_validate' from 2 (Balanced) to 4 (Paranoid)
2025-04-04T00:45:03Z LKRG: STATE: Changing 'profile_validate' from 4 (Paranoid) to 2 (Balanced)
2025-04-04T00:45:03Z LKRG: STATE: Changing 'profile_validate' from 2 (Balanced) to 4 (Paranoid)
2025-04-04T00:45:04Z LKRG: STATE: Changing 'profile_validate' from 4 (Paranoid) to 2 (Balanced)
2025-04-04T00:45:04Z LKRG: STATE: Changing 'profile_validate' from 2 (Balanced) to 4 (Paranoid)
2025-04-04T00:45:04Z LKRG: STATE: Changing 'profile_validate' from 4 (Paranoid) to 2 (Balanced)
2025-04-04T00:45:04Z LKRG: STATE: Changing 'profile_validate' from 2 (Balanced) to 4 (Paranoid)
2025-04-04T00:45:04Z LKRG: STATE: Changing 'profile_validate' from 4 (Paranoid) to 2 (Balanced)
2025-04-04T00:45:04Z LKRG: STATE: Changing 'profile_validate' from 2 (Balanced) to 4 (Paranoid)
[... lots of lines omitted ...]
2025-04-04T01:16:00Z LKRG: STATE: Changing 'profile_validate' from 2 (Balanced) to 4 (Paranoid)
2025-04-04T01:16:08Z LKRG: STATE: Changing 'profile_validate' from 4 (Paranoid) to 2 (Balanced)
2025-04-04T01:16:09Z LKRG: STATE: Changing 'profile_validate' from 2 (Balanced) to 4 (Paranoid)
2025-04-04T01:16:17Z LKRG: STATE: Changing 'profile_validate' from 4 (Paranoid) to 2 (Balanced)
2025-04-04T01:16:18Z LKRG: STATE: Changing 'profile_validate' from 2 (Balanced) to 4 (Paranoid)
2025-04-04T01:16:26Z LKRG: STATE: Changing 'profile_validate' from 4 (Paranoid) to 2 (Balanced)
2025-04-04T01:16:27Z LKRG: STATE: Changing 'profile_validate' from 2 (Balanced) to 4 (Paranoid)
2025-04-04T01:16:35Z LKRG: STATE: Changing 'profile_validate' from 4 (Paranoid) to 2 (Balanced)
2025-04-04T01:16:36Z LKRG: STATE: Changing 'profile_validate' from 2 (Balanced) to 4 (Paranoid)
2025-04-04T01:16:44Z LKRG: STATE: Changing 'profile_validate' from 4 (Paranoid) to 2 (Balanced)
2025-04-04T01:16:45Z LKRG: STATE: Changing 'profile_validate' from 2 (Balanced) to 4 (Paranoid)
2025-04-04T01:16:53Z LKRG: STATE: Changing 'profile_validate' from 4 (Paranoid) to 2 (Balanced)
2025-04-04T01:16:54Z LKRG: STATE: Changing 'profile_validate' from 2 (Balanced) to 4 (Paranoid)
2025-04-04T01:17:02Z LKRG: STATE: Changing 'profile_validate' from 4 (Paranoid) to 2 (Balanced)
2025-04-04T01:17:03Z LKRG: STATE: Changing 'profile_validate' from 2 (Balanced) to 4 (Paranoid)
2025-04-04T01:17:11Z LKRG: STATE: Changing 'profile_validate' from 4 (Paranoid) to 2 (Balanced)
2025-04-04T01:17:12Z LKRG: STATE: Changing 'profile_validate' from 2 (Balanced) to 4 (Paranoid)
2025-04-04T01:17:12Z LKRG: ALERT: DETECT: CPU: Hash of CPU metadata has changed unexpectedly
2025-04-04T01:17:12Z LKRG: ALERT: DETECT: Kernel: 1 checksums changed unexpectedly
2025-04-04T01:17:12Z LKRG: ALERT: BLOCK: Kernel: 1 checksums changed unexpectedly
2025-04-04T01:17:12Z Kernel panic - not syncing: Kernel: 1 checksums changed unexpectedly
2025-04-04T01:17:12Z CPU: 2 PID: 2951 Comm: kworker/u256:0 Kdump: loaded Tainted: G           OE     -------  ---  5.14.0-503.22.1.el9_5.x86_64 #1
2025-04-04T01:17:12Z Hardware name: oVirt RHEL/RHEL-AV, BIOS 1.16.0-4.module_el8+603+e0ca2c01 04/01/2014
2025-04-04T01:17:12Z Workqueue: events_unbound p_check_integrity [lkrg]
2025-04-04T01:17:12Z Call Trace:
2025-04-04T01:17:12Z  <TASK>
2025-04-04T01:17:12Z  dump_stack_lvl+0x34/0x48
2025-04-04T01:17:12Z  panic+0x107/0x2bb
2025-04-04T01:17:12Z  p_check_integrity.cold+0xa57/0xc49 [lkrg]
2025-04-04T01:17:12Z  process_one_work+0x197/0x380
2025-04-04T01:17:12Z  worker_thread+0x2fe/0x410
```

Note that the first time this happened right after transition from 4 to 2, and the second from 2 to 4.

I have no luck (so far at least) reproducing this with transitions between 1 and 2 (running in a loop for many hours by now):

```
2025-04-05T01:18:00Z LKRG: STATE: Changing 'profile_validate' from 1 (Light) to 2 (Balanced)
2025-04-05T01:18:01Z LKRG: STATE: Changing 'profile_validate' from 2 (Balanced) to 1 (Light)
2025-04-05T01:18:02Z LKRG: STATE: Changing 'profile_validate' from 1 (Light) to 2 (Balanced)
2025-04-05T01:18:03Z LKRG: STATE: Changing 'profile_validate' from 2 (Balanced) to 1 (Light)
2025-04-05T01:18:04Z LKRG: STATE: Changing 'profile_validate' from 1 (Light) to 2 (Balanced)
2025-04-05T01:18:05Z LKRG: STATE: Changing 'profile_validate' from 2 (Balanced) to 1 (Light)
2025-04-05T01:18:06Z LKRG: STATE: Changing 'profile_validate' from 1 (Light) to 2 (Balanced)
2025-04-05T01:18:07Z LKRG: STATE: Changing 'profile_validate' from 2 (Balanced) to 1 (Light)
```
Linked issue comments: user: login: solardiz
created_at: 2025-04-05T01:47:40Z
body: I guess the issue is with `msr_validate` transitions, which occur between profile 4 and any other. I'm now trying to reproduce it with that sysctl. However, while this is running I also looked at the code and I think I see the bug already: in `p_check_integrity`, we first call `hash_from_CPU_data` and only then acquire `p_db_lock`. I think these should be in the other order.

To fix this, we need to decide whether we want to move the lock acquisition up in code, or the `hash_from_CPU_data` to below the existing place of lock acquisition. This isn't a simple swap of these two because there's other work we currently do between them.

@Adam-pi3 Do you have any comments (maybe a reason why `hash_from_CPU_data` was done outside of that lock?) and/or any preference for how to fix this best?
author_association: CONTRIBUTOR
user: login: solardiz
created_at: 2025-04-05T01:51:52Z
body: OK, this command:

```
while :; do sysctl lkrg.trigger=1; sysctl lkrg.msr_validate=0; sysctl lkrg.trigger=1; sysctl lkrg.msr_validate=1; done
```

triggered the issue in ~10 seconds.
author_association: CONTRIBUTOR
user: login: Adam-pi3
created_at: 2025-04-08T18:44:44Z
body: @solardiz I think we should find which MSR is changed during the mode switch. However, couple of notes:

1. It is by design to NOT acquire `p_db_lock` because the CPU metadata which we collect suppose to be static and never changed after CPU initialization. In case of x86 it's IDT table and MSR registers - those are collected per each ONLINE core.
2. We also collect information about how many CPUs are available and at witch state
3. If CPU is dynamically changing the state (e.g., from being OFF to ON or other way around) we have notifier registered to dynamically take this into account and rebuilt the DB

In short, we want to keep `hash_from_CPU_data outside` of `p_db_lock` and find out which MSR (and why) is changing the value. Maybe we should not track it?
author_association: COLLABORATOR
user: login: solardiz
created_at: 2025-04-08T19:32:38Z
body: @Adam-pi3 There's no reason to think that any MSR is changing value on mode switch. Rather, depending on mode we're including MSR values in the hash or we don't, which obviously changes the hash value.
author_association: CONTRIBUTOR
user: login: Adam-pi3
created_at: 2025-04-08T22:54:22Z
body: Ah yea - simple overlook :) I think there are 2 ways to go:
1. move the CPU metadata dumping under `p_db_lock` but I think it is overkill
2. introduce a new different lock just for `msr_validation` feature which could be acquired/dropped between CPU metadata dumping and sysctl msr configuration option
author_association: COLLABORATOR
user: login: solardiz
created_at: 2025-04-08T23:01:57Z
body: > * move the CPU metadata dumping under `p_db_lock` but I think it is overkill
> 
> * introduce a new different lock just for `msr_validation` feature which could be acquired/dropped between CPU metadata dumping and sysctl msr configuration option

Yes, I had thought of these too. I don't see the first option as overkill - to me, in terms of added complexity the second option is overkill, because CPU metadata dumping is quick, so it's not a big deal that we hold the lock for a tiny bit longer.

As I wrote, actually "we need to decide whether we want to move the lock acquisition up in code, or the hash_from_CPU_data to below the existing place of lock acquisition. This isn't a simple swap of these two because there's other work we currently do between them." Any preference?
author_association: CONTRIBUTOR
user: login: solardiz
created_at: 2025-04-12T02:13:36Z
body: > while :; do sysctl lkrg.trigger=1; sysctl lkrg.msr_validate=0; sysctl lkrg.trigger=1; sysctl lkrg.msr_validate=1; done

So I wanted to reproduce this on another system (for testing a possible fix then), but instead triggered something else:

```
[2025-04-12 04:06:32] [ 9454.563372] LKRG: STATE: Disabling 'msr_validate' 
[2025-04-12 04:06:32] [ 9454.565508] LKRG: STATE: Enabling 'msr_validate' 
[2025-04-12 04:06:57] [ 9479.429698] watchdog: BUG: soft lockup - CPU#1 stuck for 26s! [sysctl:4847]
```

and the system became unresponsive.

There's `proc_dointvec_minmax` in the backtrace, so I guess it's issue mentioned/suspected in https://github.com/lkrg-org/lkrg/issues/204#issuecomment-1186626846 so now we know it's a real one seen not only in debugging kernel builds.
author_association: CONTRIBUTOR
user: login: solardiz
created_at: 2025-04-16T01:43:27Z
body: > move the CPU metadata dumping under `p_db_lock`

This isn't so easy. My first attempt resulted in two of our CI jobs (in my personal GitHub fork) failing like this:

```
[   26.369001] WARNING: CPU: 0 PID: 291 at kernel/smp.c:424 smp_call_function_many+0xab/0x270
```

I think this has to do with us also acquiring `p_read_cpu_lock`, which we previously never did inside `p_text_section_lock`.
author_association: CONTRIBUTOR
user: login: solardiz
created_at: 2025-04-16T02:17:42Z
body: @Adam-pi3 I don't get a couple of things related to this:
1. Why do we recompute/update `p_db.p_CPU_metadata_hashes` in `p_module_event_notifier`? What does module (un)loading have to do with CPU registers?
2. Why do we `p_text_section_lock();` in `p_cpu_online_action` and `p_cpu_dead_action`? What do changes in the number of CPUs have to do with text section contents? Related: on integrity timer, we compute the CPU metadata hash (for comparison) outside of the text section locks, which feels inconsistent with us taking those locks on CPU count changes. I hope `p_text_section_lock();` in `p_cpu_online_action` and `p_cpu_dead_action` is simply unneeded, as otherwise us not taking those locks when checking is a bug that can lead to false positives.
author_association: CONTRIBUTOR
user: login: solardiz
created_at: 2025-04-16T02:43:27Z
body: > ```
> [   26.369001] WARNING: CPU: 0 PID: 291 at kernel/smp.c:424 smp_call_function_many+0xab/0x270
> ```
> 
> I think this has to do with us also acquiring `p_read_cpu_lock`, which we previously never did inside `p_text_section_lock`.

No, looks like `p_text_section_lock` was unrelated to that, but somehow the combination of `p_db_lock` and `p_read_cpu_lock` is problematic in this way, in either order of taking these locks. @Adam-pi3 Any idea why that would be?

> introduce a new different lock just for msr_validation feature

This makes more sense to me now, but I wish I knew why exactly taking `p_db_lock` is so problematic.
author_association: CONTRIBUTOR
user: login: Adam-pi3
created_at: 2025-04-17T04:36:03Z
body: > [@Adam-pi3](https://github.com/Adam-pi3) I don't get a couple of things related to this:
> 
> 1. Why do we recompute/update `p_db.p_CPU_metadata_hashes` in `p_module_event_notifier`? What does module (un)loading have to do with CPU registers?

Some modules may modify MSR registers.

> 2. Why do we `p_text_section_lock();` in `p_cpu_online_action` and `p_cpu_dead_action`? What do changes in the number of CPUs have to do with text section contents? 

Actually a lot. When that even happens, JUMP_LABEL, STATIC KEYS and others kicks in and modify .text section.
 
> Related: on integrity timer, we compute the CPU metadata hash (for comparison) outside of the text section locks, which feels inconsistent with us taking those locks on CPU count changes. I hope `p_text_section_lock();` in `p_cpu_online_action` and `p_cpu_dead_action` is simply unneeded, as otherwise us not taking those locks when checking is a bug that can lead to false positives.

That's by design to not collect metadata under text section lock since those data are static and we don't need this lock.

author_association: COLLABORATOR
user: login: Adam-pi3
created_at: 2025-04-17T05:08:54Z
body: > > ```
> > [   26.369001] WARNING: CPU: 0 PID: 291 at kernel/smp.c:424 smp_call_function_many+0xab/0x270
> > ```
> > 
> > 
> >     
> >   
> > I think this has to do with us also acquiring `p_read_cpu_lock`, which we previously never did inside `p_text_section_lock`.
> 
> No, looks like `p_text_section_lock` was unrelated to that, but somehow the combination of `p_db_lock` and `p_read_cpu_lock` is problematic in this way, in either order of taking these locks. [@Adam-pi3](https://github.com/Adam-pi3) Any idea why that would be?
> 

AFAIR sending IPI calls `get_cpu` internally which is also acquired by `p_read_cpu_lock` (but maybe that's not a problem?). Also, under the hood if any of the JUMP_LABEL/STATIC_KEY/TRACEPOINT/etc lock is taken, it will collide with `p_text_section_lock`. I would need to investigate what's the actual problem. That's just something from the top of my head. Also, the order of the locks in `p_text_section_lock` are carefully chosen to not generate any ABBA.

> > introduce a new different lock just for msr_validation feature
> 
> This makes more sense to me now, but I wish I knew why exactly taking `p_db_lock` is so problematic.

In that specific use-case having a new exclusive MSR related lock is much more desired since `p_db_lock` is used in heavy system verification.

author_association: COLLABORATOR
user: login: solardiz
created_at: 2025-04-19T02:51:38Z
body: > > 1. Why do we recompute/update `p_db.p_CPU_metadata_hashes` in `p_module_event_notifier`? What does module (un)loading have to do with CPU registers?
> 
> Some modules may modify MSR registers.

OK, I suppose it's more likely they do it on (un)loading than at other times.

> > 2. Why do we `p_text_section_lock();` in `p_cpu_online_action` and `p_cpu_dead_action`? What do changes in the number of CPUs have to do with text section contents?
> 
> Actually a lot. When that even happens, JUMP_LABEL, STATIC KEYS and others kicks in and modify .text section.

I think we'd catch JUMP_LABEL via our other hooks. But OK, if you say there's more than that.
author_association: CONTRIBUTOR
user: login: solardiz
created_at: 2025-04-19T20:10:48Z
body: (Part of?) the problem with acquiring `p_db_lock` for this maybe that we (have to?) disable local IRQs when we acquire this lock (use `_irqsave`) from `p_check_integrity`. At least existing acquisition of this lock from there is such, so it's an unsafe change to stop disabling IRQs there, risking potential deadlocks if we're preempted and reach some other place where we use this lock (I don't know if it's possible or not). And I guess `on_each_cpu` may require that we have local IRQs enabled?

I'm trying to solve this by introducing a new rwlock instead, so we'd acquire it only for read in most places (except on sysctl updates), thereby avoiding risk of deadlocks.
author_association: CONTRIBUTOR

Diff:
diff --git a/CHANGES b/CHANGES
index 65c4e74c..40bfce1d 100644
--- a/CHANGES
+++ b/CHANGES
@@ -3,8 +3,10 @@ The following major changes have been made since 0.9.9:
  *) Overhaul locking of per-task shadow data, using finer-grain locks
  *) Improve performance of per-task shadow data lookups by making them lockless
  *) Fix several lethal race conditions involving SECCOMP_FILTER_FLAG_TSYNC
- *) Fix integrity violation misattribution to the wrong task when pint_enforce=0
+ *) Fix integrity violation misattribution to a wrong task when pint_enforce=0
  *) Fix several integrity violation race conditions when pint_enforce=0
+ *) Fix race condition on msr_validate sysctl changes as well as on transitions
+    between profile_validate=4 and others
  *) Support building against (but not yet loading into) Linux 6.15-rc1+
  *) Build and link the userspace logger tools with hardening flags, and pass
     distributions' RPM packaging hardening flags to the compiler and linker
diff --git a/src/modules/comm_channel/p_comm_channel.c b/src/modules/comm_channel/p_comm_channel.c
index 7a09c657..d750884c 100644
--- a/src/modules/comm_channel/p_comm_channel.c
+++ b/src/modules/comm_channel/p_comm_channel.c
@@ -798,13 +798,13 @@ static int p_sysctl_msr_validate(P_STRUCT_CTL_TABLE *p_table, int p_write,
    int p_cpu;
    unsigned int p_tmp;
 
+   write_lock(&p_config_lock);
    p_tmp = P_CTRL(p_msr_validate);
    p_lkrg_open_rw();
    if ( (p_ret = proc_dointvec_minmax(p_table, p_write, p_buffer, p_len, p_pos)) == 0 && p_write) {
       if (P_CTRL(p_msr_validate) && !p_tmp) {
          P_CTRL(p_profile_validate) = 9;
          p_print_log(P_LOG_STATE, "Enabling 'msr_validate'");
-         spin_lock(&p_db_lock);
          memset(p_db.p_CPU_metadata_array,0,sizeof(p_CPU_metadata_hash_mem)*p_db.p_cpu.p_nr_cpu_ids);
          for_each_present_cpu(p_cpu) {
             if (cpu_online(p_cpu)) {
@@ -812,16 +812,14 @@ static int p_sysctl_msr_validate(P_STRUCT_CTL_TABLE *p_table, int p_write,
             }
          }
          p_db.p_CPU_metadata_hashes = hash_from_CPU_data(p_db.p_CPU_metadata_array);
-         spin_unlock(&p_db_lock);
       } else if (p_tmp && !P_CTRL(p_msr_validate)) {
          P_CTRL(p_profile_validate) = 9;
          p_print_log(P_LOG_STATE, "Disabling 'msr_validate'");
-         spin_lock(&p_db_lock);
          p_db.p_CPU_metadata_hashes = hash_from_CPU_data(p_db.p_CPU_metadata_array);
-         spin_unlock(&p_db_lock);
       }
    }
    p_lkrg_close_rw();
+   write_unlock(&p_config_lock);
 
    return p_ret;
 }
@@ -916,10 +914,10 @@ static int p_sysctl_profile_validate(P_STRUCT_CTL_TABLE *p_table, int p_write,
                   P_CTRL(p_umh_validate) = 0;   // Disabled
                   /* msr_validate */
                   if (P_CTRL(p_msr_validate)) {
-                     spin_lock(&p_db_lock);
+                     write_lock(&p_config_lock);
                      P_CTRL(p_msr_validate) = 0; // Disable
                      p_db.p_CPU_metadata_hashes = hash_from_CPU_data(p_db.p_CPU_metadata_array);
-                     spin_unlock(&p_db_lock);
+                     write_unlock(&p_config_lock);
                   }
 #if defined(CONFIG_X86)
                   /* smep_validate */
@@ -942,10 +940,10 @@ static int p_sysctl_profile_validate(P_STRUCT_CTL_TABLE *p_table, int p_write,
                   P_CTRL(p_umh_validate) = 1;   // Allow specific paths
                   /* msr_validate */
                   if (P_CTRL(p_msr_validate)) {
-                     spin_lock(&p_db_lock);
+                     write_lock(&p_config_lock);
                      P_CTRL(p_msr_validate) = 0; // Disable
                      p_db.p_CPU_metadata_hashes = hash_from_CPU_data(p_db.p_CPU_metadata_array);
-                     spin_unlock(&p_db_lock);
+                     write_unlock(&p_config_lock);
                   }
 #if defined(CONFIG_X86)
                   /* smep_validate */
@@ -984,10 +982,10 @@ static int p_sysctl_profile_validate(P_STRUCT_CTL_TABLE *p_table, int p_write,
                   P_CTRL(p_umh_validate) = 1;   // Allow specific paths
                   /* msr_validate */
                   if (P_CTRL(p_msr_validate)) {
-                     spin_lock(&p_db_lock);
+                     write_lock(&p_config_lock);
                      P_CTRL(p_msr_validate) = 0; // Disable
                      p_db.p_CPU_metadata_hashes = hash_from_CPU_data(p_db.p_CPU_metadata_array);
-                     spin_unlock(&p_db_lock);
+                     write_unlock(&p_config_lock);
                   }
 #if defined(CONFIG_X86)
                   /* smep_validate */
@@ -1026,10 +1024,10 @@ static int p_sysctl_profile_validate(P_STRUCT_CTL_TABLE *p_table, int p_write,
                   P_CTRL(p_umh_validate) = 1;   // Allow specific paths
                   /* msr_validate */
                   if (P_CTRL(p_msr_validate)) {
-                     spin_lock(&p_db_lock);
+                     write_lock(&p_config_lock);
                      P_CTRL(p_msr_validate) = 0; // Disable
                      p_db.p_CPU_metadata_hashes = hash_from_CPU_data(p_db.p_CPU_metadata_array);
-                     spin_unlock(&p_db_lock);
+                     write_unlock(&p_config_lock);
                   }
 #if defined(CONFIG_X86)
                   /* smep_validate */
@@ -1068,7 +1066,7 @@ static int p_sysctl_profile_validate(P_STRUCT_CTL_TABLE *p_table, int p_write,
                   P_CTRL(p_umh_validate) = 2;   // Full lock-down
                   /* msr_validate */
                   if (!P_CTRL(p_msr_validate)) {
-                     spin_lock(&p_db_lock);
+                     write_lock(&p_config_lock);
                      P_CTRL(p_msr_validate) = 1; // Enable
                      memset(p_db.p_CPU_metadata_array,0,sizeof(p_CPU_metadata_hash_mem)*p_db.p_cpu.p_nr_cpu_ids);
                      for_each_present_cpu(p_cpu) {
@@ -1077,7 +1075,7 @@ static int p_sysctl_profile_validate(P_STRUCT_CTL_TABLE *p_table, int p_write,
                         }
                      }
                      p_db.p_CPU_metadata_hashes = hash_from_CPU_data(p_db.p_CPU_metadata_array);
-                     spin_unlock(&p_db_lock);
+                     write_unlock(&p_config_lock);
                   }
 #if defined(CONFIG_X86)
                   /* smep_validate */
diff --git a/src/modules/database/CPU.c b/src/modules/database/CPU.c
index 2c350ed0..5b9b7eb5 100644
--- a/src/modules/database/CPU.c
+++ b/src/modules/database/CPU.c
@@ -168,6 +168,7 @@ int p_cpu_online_action(unsigned int p_cpu) {
 
    p_text_section_lock();
    spin_lock(&p_db_lock);
+   read_lock(&p_config_lock);
 
    smp_call_function_single(p_cpu,p_dump_CPU_metadata,p_db.p_CPU_metadata_array,true);
 
@@ -186,6 +187,7 @@ int p_cpu_online_action(unsigned int p_cpu) {
       p_cpu_rehash("online");
    }
 
+   read_unlock(&p_config_lock);
    /* God mode off ;) */
 //   spin_unlock_irqrestore(&p_db_lock,p_db_flags);
    spin_unlock(&p_db_lock);
@@ -200,6 +202,7 @@ int p_cpu_dead_action(unsigned int p_cpu) {
 
    p_text_section_lock();
    spin_lock(&p_db_lock);
+   read_lock(&p_config_lock);
 
    p_db.p_CPU_metadata_array[p_cpu].p_cpu_online = P_CPU_OFFLINE;
 
@@ -226,6 +229,7 @@ int p_cpu_dead_action(unsigned int p_cpu) {
       p_cpu_rehash("offline");
    }
 
+   read_unlock(&p_config_lock);
    /* God mode off ;) */
 //   spin_unlock_irqrestore(&p_db_lock,p_db_flags);
    spin_unlock(&p_db_lock);
diff --git a/src/modules/database/arch/x86/p_switch_idt/p_switch_idt.c b/src/modules/database/arch/x86/p_switch_idt/p_switch_idt.c
index 0cf1ead8..6f88d18e 100644
--- a/src/modules/database/arch/x86/p_switch_idt/p_switch_idt.c
+++ b/src/modules/database/arch/x86/p_switch_idt/p_switch_idt.c
@@ -38,6 +38,7 @@ static struct kretprobe p_switch_idt_kretprobe = {
 int p_switch_idt_entry(struct kretprobe_instance *p_ri, struct pt_regs *p_regs) {
 
    spin_lock(&p_db_lock);
+   read_lock(&p_config_lock);
 
    /* A dump_stack() here will give a stack backtrace */
    return 0;
@@ -53,6 +54,7 @@ int p_switch_idt_ret(struct kretprobe_instance *ri, struct pt_regs *p_regs) {
    smp_call_function_single(smp_processor_id(),p_dump_CPU_metadata,p_db.p_CPU_metadata_array,true);
    p_db.p_CPU_metadata_hashes = hash_from_CPU_data(p_db.p_CPU_metadata_array);
 
+   read_unlock(&p_config_lock);
    spin_unlock(&p_db_lock);
 
    return 0;
diff --git a/src/modules/integrity_timer/p_integrity_timer.c b/src/modules/integrity_timer/p_integrity_timer.c
index bbb1b2fe..6dc2f239 100644
--- a/src/modules/integrity_timer/p_integrity_timer.c
+++ b/src/modules/integrity_timer/p_integrity_timer.c
@@ -26,6 +26,7 @@ struct timer_list p_timer;
 unsigned int p_time_stamp = 15; /* timeout in seconds */
 /* God mode variables ;) */
 DEFINE_SPINLOCK(p_db_lock);
+DEFINE_RWLOCK(p_config_lock);
 unsigned long p_db_flags;
 unsigned int p_manual = 0;
 
@@ -178,6 +179,7 @@ void p_check_integrity(struct work_struct *p_work) {
    /*
     * Checking all online CPUs critical data
     */
+   read_lock(&p_config_lock);
    p_read_cpu_lock();
 
 //   for_each_present_cpu(p_tmp) {
@@ -220,7 +222,22 @@ void p_check_integrity(struct work_struct *p_work) {
    /* Now we are safe to disable IRQs on current core */
 
    p_tmp_hash = hash_from_CPU_data(p_tmp_cpus);
+
+   if (p_db.p_CPU_metadata_hashes != p_tmp_hash) {
+      /* I'm hacked! ;( */
+      p_print_log(P_LOG_ALERT, "DETECT: CPU: Hash of CPU metadata has changed unexpectedly");
+#define P_KINT_IF_ACCEPT(old, new) \
+      if (!P_CTRL(p_kint_enforce)) \
+         old = new; \
+      p_hack_check++;
+      P_KINT_IF_ACCEPT(p_db.p_CPU_metadata_hashes, p_tmp_hash)
+   }
+
+   p_print_log(P_LOG_WATCH, "Hash of CPU metadata expected 0x%llx vs. actual 0x%llx",
+      p_db.p_CPU_metadata_hashes, p_tmp_hash);
+
    p_read_cpu_unlock();
+   read_unlock(&p_config_lock);
 
    /* Verify kprobes now */
    if (lkrg_verify_kprobes()) {
@@ -251,19 +268,6 @@ void p_check_integrity(struct work_struct *p_work) {
    spin_lock_irqsave(&p_db_lock,p_db_flags);
 //   spin_lock(&p_db_lock);
 
-   if (p_db.p_CPU_metadata_hashes != p_tmp_hash) {
-      /* I'm hacked! ;( */
-      p_print_log(P_LOG_ALERT, "DETECT: CPU: Hash of CPU metadata has changed unexpectedly");
-#define P_KINT_IF_ACCEPT(old, new) \
-      if (!P_CTRL(p_kint_enforce)) \
-         old = new; \
-      p_hack_check++;
-      P_KINT_IF_ACCEPT(p_db.p_CPU_metadata_hashes, p_tmp_hash)
-   }
-
-   p_print_log(P_LOG_WATCH, "Hash of CPU metadata expected 0x%llx vs. actual 0x%llx",
-      p_db.p_CPU_metadata_hashes, p_tmp_hash);
-
    /*
     * Checking memory block:
     * "___ex_table"
diff --git a/src/modules/integrity_timer/p_integrity_timer.h b/src/modules/integrity_timer/p_integrity_timer.h
index 6f57baff..479a09c8 100644
--- a/src/modules/integrity_timer/p_integrity_timer.h
+++ b/src/modules/integrity_timer/p_integrity_timer.h
@@ -40,6 +40,7 @@ void p_offload_cache_delete(void);
 extern struct timer_list p_timer;
 extern unsigned int p_manual;
 extern spinlock_t p_db_lock;
+extern rwlock_t p_config_lock;
 extern unsigned long p_db_flags;
 extern struct kmem_cache *p_offload_cache;
 
diff --git a/src/modules/kmod/p_kmod_notifier.c b/src/modules/kmod/p_kmod_notifier.c
index 586d6c76..9078c778 100644
--- a/src/modules/kmod/p_kmod_notifier.c
+++ b/src/modules/kmod/p_kmod_notifier.c
@@ -101,10 +101,12 @@ static int p_module_event_notifier(struct notifier_block *p_this, unsigned long
 //   if (p_tmp->state == MODULE_STATE_GOING) { <- Linux kernel bug - might not update state value :(
    if (p_event == MODULE_STATE_GOING) {
 
+      read_lock(&p_config_lock);
       p_read_cpu_lock();
       on_each_cpu(p_dump_CPU_metadata,p_db.p_CPU_metadata_array,true);
       p_db.p_CPU_metadata_hashes = hash_from_CPU_data(p_db.p_CPU_metadata_array);
       p_read_cpu_unlock();
+      read_unlock(&p_config_lock);
 
       /*
        * Now recalculate modules information in database!
@@ -173,10 +175,12 @@ static int p_module_event_notifier(struct notifier_block *p_this, unsigned long
 //      if (p_tmp->state == MODULE_STATE_LIVE) { <- Linux kernel bug - might not update state value :(
       if (p_event == MODULE_STATE_LIVE) {
 
+         read_lock(&p_config_lock);
          p_read_cpu_lock();
          on_each_cpu(p_dump_CPU_metadata,p_db.p_CPU_metadata_array,true);
          p_db.p_CPU_metadata_hashes = hash_from_CPU_data(p_db.p_CPU_metadata_array);
          p_read_cpu_unlock();
+         read_unlock(&p_config_lock);
 
          /*
           * Now recalculate modules information in database! Since blocking module is disabled


Reviews:
user: login: Adam-pi3
body: @solardiz I think locks are wrong
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-20T17:48:34Z
user: login: solardiz
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-20T18:44:47Z

Review comments:
diff_hunk: @@ -220,7 +222,22 @@ void p_check_integrity(struct work_struct *p_work) {
    /* Now we are safe to disable IRQs on current core */
path: src/modules/integrity_timer/p_integrity_timer.c
user: login: Adam-pi3
body: This comment should be just before `spin_lock_irqsave(&p_db_lock,p_db_flags);` - it looks like over the time it was shifted
created_at: 2025-04-20T17:45:34Z
author_association: COLLABORATOR
diff_hunk: @@ -168,6 +168,7 @@ int p_cpu_online_action(unsigned int p_cpu) {
 
    p_text_section_lock();
    spin_lock(&p_db_lock);
+   read_lock(&p_config_lock);
path: src/modules/database/CPU.c
user: login: Adam-pi3
body: Shouldn't it be write_lock? CPU metadata may (and will) change when new CPU is changing the state
created_at: 2025-04-20T17:46:48Z
author_association: COLLABORATOR
diff_hunk: @@ -200,6 +202,7 @@ int p_cpu_dead_action(unsigned int p_cpu) {
 
    p_text_section_lock();
    spin_lock(&p_db_lock);
+   read_lock(&p_config_lock);
path: src/modules/database/CPU.c
user: login: Adam-pi3
body: Same here (write_lock)
created_at: 2025-04-20T17:47:12Z
author_association: COLLABORATOR
diff_hunk: @@ -38,6 +38,7 @@ static struct kretprobe p_switch_idt_kretprobe = {
 int p_switch_idt_entry(struct kretprobe_instance *p_ri, struct pt_regs *p_regs) {
 
    spin_lock(&p_db_lock);
+   read_lock(&p_config_lock);
path: src/modules/database/arch/x86/p_switch_idt/p_switch_idt.c
user: login: Adam-pi3
body: Same here (write_lock)
created_at: 2025-04-20T17:47:34Z
author_association: COLLABORATOR
diff_hunk: @@ -101,10 +101,12 @@ static int p_module_event_notifier(struct notifier_block *p_this, unsigned long
 //   if (p_tmp->state == MODULE_STATE_GOING) { <- Linux kernel bug - might not update state value :(
    if (p_event == MODULE_STATE_GOING) {
 
+      read_lock(&p_config_lock);
path: src/modules/kmod/p_kmod_notifier.c
user: login: Adam-pi3
body: Same here (modules can change MSR) - 'write_lock'
created_at: 2025-04-20T17:48:07Z
author_association: COLLABORATOR
diff_hunk: @@ -173,10 +175,12 @@ static int p_module_event_notifier(struct notifier_block *p_this, unsigned long
 //      if (p_tmp->state == MODULE_STATE_LIVE) { <- Linux kernel bug - might not update state value :(
       if (p_event == MODULE_STATE_LIVE) {
 
+         read_lock(&p_config_lock);
path: src/modules/kmod/p_kmod_notifier.c
user: login: Adam-pi3
body: Same here (modules can change MSR) - 'write_lock'
created_at: 2025-04-20T17:48:11Z
author_association: COLLABORATOR
diff_hunk: @@ -220,7 +222,22 @@ void p_check_integrity(struct work_struct *p_work) {
    /* Now we are safe to disable IRQs on current core */
path: src/modules/integrity_timer/p_integrity_timer.c
user: login: solardiz
body: Yes, this comment originally was here (first LKRG commit in 2017):

```c
   on_each_cpu(p_dump_IDT_MSR_CRx,p_tmp_cpus,true);

   put_online_cpus();

   /*
    * OK, so now get the same information for currently locked core!
    */
//   p_dump_IDT_MSR_CRx(p_tmp_cpus); // no return value

   /* Now we are safe to disable IRQs on current core */
   spin_lock_irqsave(&p_db_lock,p_db_flags);

   p_tmp_hash = hash_from_CPU_data(p_tmp_cpus);
```

Arguably, it's still precisely at the place where it's safe to disable IRQs - but no longer at the place where we actually do that (not to mix acquisition/release of different locks in weird and risky ways). I don't mind moving it to right before the `spin_lock_irqsave` again, or just removing it.
created_at: 2025-04-20T18:44:47Z
author_association: CONTRIBUTOR

PR comments:
user: login: solardiz
created_at: 2025-04-19T20:22:27Z
body: Also testing with:

```shell
while :; do sysctl lkrg.trigger=1; sysctl lkrg.msr_validate=0; sysctl lkrg.trigger=1; sysctl lkrg.msr_validate=1; done
```
author_association: CONTRIBUTOR
user: login: solardiz
created_at: 2025-04-19T20:26:27Z
body: Ended up triggering a variation of #375 after a couple of minutes:

```
[2025-04-19 22:23:22] [ 1144.598167] LKRG: ALERT: DETECT: Task: Namespace escape: nsproxy pointer corruption for pid 66759, name sysctl
 [2025-04-19 22:23:22] [ 1144.598194] BUG: kernel NULL pointer dereference, address: 0000000000000008
```

Other bugs like this make it harder to confirm we have indeed reliably fixed this one bug.
author_association: CONTRIBUTOR
user: login: solardiz
created_at: 2025-04-19T20:39:40Z
body: This passes my tests now.

I think some of the old uses of `p_db_lock` that this patch preserves are now unneeded (and some maybe never were), but they should not hurt (except for performance).
author_association: CONTRIBUTOR
user: login: Adam-pi3
created_at: 2025-04-20T17:48:52Z
body: @solardiz can you not merge changes before the review?
author_association: COLLABORATOR
user: login: solardiz
created_at: 2025-04-20T18:36:44Z
body: @Adam-pi3 Sorry I didn't wait for your review. I did request a review, but I felt it was optional given the testing I already performed and that this would be easy to revise later if we need to.

I think the locks are right: the meaning of the new config lock is that it reflects whether LKRG _configuration_ is being written to, which only happens in the sysctl interface. It is no problem that MSRs may change outside of this lock (or when it's only read-locked).

Also, if we were to `write_lock` it in those other places, we'd have the same problems I had seen with the db spinlock, namely that we either disable IRQs (and then it may be wrong to use `on_each_cpu`) or we risk deadlocks on preemption.
author_association: CONTRIBUTOR
user: login: Adam-pi3
created_at: 2025-04-21T05:50:24Z
body: > I think the locks are right: the meaning of the new config lock is that it reflects whether LKRG configuration is being written to, which only happens in the sysctl interface. It is no problem that MSRs may change outside of this lock (or when it's only read-locked).

Ah... make sense. Thanks!
author_association: COLLABORATOR