Repository: bitcoin/bitcoin (language: cpp)

Issue:
Intermittent failures in interface_usdt_mempool.py
https://cirrus-ci.com/task/5779522121891840

```
[0;34m node0 2023-03-31T11:45:58.812384Z (mocktime: 2023-04-14T11:46:01Z) [http] [httpserver.cpp:257] [http_request_cb] [http] Received a POST request for / from 127.0.0.1:38238 [0m
[0;34m node0 2023-03-31T11:45:58.812543Z (mocktime: 2023-04-14T11:46:01Z) [httpworker.2] [rpc/request.cpp:179] [parse] [rpc] ThreadRPCServer method=getpeerinfo user=__cookie__ [0m
[0;36m test  2023-03-31T11:45:58.814000Z TestFramework (INFO): Hooking into mempool:rejected tracepoint... [0m
[0;36m test  2023-03-31T11:45:59.798000Z TestFramework (INFO): Sending invalid transaction... [0m
[0;36m test  2023-03-31T11:45:59.799000Z TestFramework.p2p (DEBUG): Send message to 127.0.0.1:15944: msg_tx(tx=CTransaction(nVersion=2 vin=[CTxIn(prevout=COutPoint(hash=76b2ca9a8c3ffa3c454db9591dfaf5c4979926154e33a5bef67412ea0eaa16c5 n=0) scriptSig= nSequence=0)] vout=[CTxOut(nValue=49.99968800 scriptPubKey=51202913b252fe537830f843bfdc5fa7d20ba48639a87c86ff837b92d083c55ad7c1)] wit=CTxWitness(CScriptWitness(51,c00000000000000000000000000000000000000000000000000000000000000001)) nLockTime=0)) [0m
[0;36m test  2023-03-31T11:45:59.799000Z TestFramework.p2p (DEBUG): Send message to 127.0.0.1:15944: msg_ping(nonce=00000002) [0m
[0;34m node0 2023-03-31T11:45:59.799697Z (mocktime: 2023-04-14T11:46:01Z) [msghand] [net_processing.cpp:3169] [ProcessMessage] [net] received: tx (133 bytes) peer=0 [0m
[0;34m node0 2023-03-31T11:45:59.800472Z (mocktime: 2023-04-14T11:46:01Z) [msghand] [txmempool.cpp:658] [check] [mempool] Checking mempool with 2 transactions and 2 inputs [0m
[0;34m node0 2023-03-31T11:45:59.800746Z (mocktime: 2023-04-14T11:46:01Z) [msghand] [net_processing.cpp:4174] [ProcessMessage] [mempoolrej] c917a71e43b825756c1d45985fcc87f8c245db0654f5eeb000a92dc2cccb1d6c from peer=0 was not accepted: min relay fee not met, 0 < 104 [0m
[0;34m node0 2023-03-31T11:45:59.800913Z (mocktime: 2023-04-14T11:46:01Z) [msghand] [net_processing.cpp:3169] [ProcessMessage] [net] received: ping (8 bytes) peer=0 [0m
[0;34m node0 2023-03-31T11:45:59.800953Z (mocktime: 2023-04-14T11:46:01Z) [msghand] [net.cpp:2841] [PushMessage] [net] sending pong (8 bytes) peer=0 [0m
[0;36m test  2023-03-31T11:45:59.801000Z TestFramework.p2p (DEBUG): Received message from 127.0.0.1:15944: msg_pong(nonce=00000002) [0m
[0;34m node0 2023-03-31T11:45:59.849969Z (mocktime: 2023-04-14T11:46:01Z) [http] [httpserver.cpp:257] [http_request_cb] [http] Received a POST request for / from 127.0.0.1:38238 [0m
[0;36m test  2023-03-31T11:45:59.850000Z TestFramework (INFO): Polling buffer... [0m
[0;34m node0 2023-03-31T11:45:59.850174Z (mocktime: 2023-04-14T11:46:01Z) [httpworker.3] [rpc/request.cpp:179] [parse] [rpc] ThreadRPCServer method=getrawmempool user=__cookie__ [0m
[0;36m test  2023-03-31T11:45:59.898000Z TestFramework (INFO): Ensuring mempool:rejected event was handled successfully... [0m
[0;36m test  2023-03-31T11:45:59.898000Z TestFramework (ERROR): Assertion failed [0m
[0;36m                                   Traceback (most recent call last):[0m
[0;36m                                     File "/tmp/cirrus-ci-build/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/test_framework/test_framework.py", line 132, in main[0m
[0;36m                                       self.run_test()[0m
[0;36m                                     File "/tmp/cirrus-ci-build/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/interface_usdt_mempool.py", line 339, in run_test[0m
[0;36m                                       self.rejected_test()[0m
[0;36m                                     File "/tmp/cirrus-ci-build/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/interface_usdt_mempool.py", line 323, in rejected_test[0m
[0;36m                                       assert_equal(EXPECTED_REJECTED_EVENTS, handled_rejected_events)[0m
[0;36m                                     File "/tmp/cirrus-ci-build/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/test_framework/util.py", line 56, in assert_equal[0m
[0;36m                                       raise AssertionError("not(%s)" % " == ".join(str(arg) for arg in (thing1, thing2) + args))[0m
[0;36m                                   AssertionError: not(1 == 0)[0m
[0;36m test  2023-03-31T11:45:59.899000Z TestFramework (DEBUG): Closing down network thread [0m
[0;36m test  2023-03-31T11:45:59.949000Z TestFramework (INFO): Stopping nodes [0m
[0;36m test  2023-03-31T11:45:59.950000Z TestFramework.node0 (DEBUG): Stopping node [0m
```


Discussion / hints:
Have seen this a couple times by now.

cc @0xB10C @virtu 
Had a brief look. The `mempoolrej` log message shows that we indeed rejected the transaction but the test didn't see the reject. 

Can't reproduce this locally with about 20 test runs. All pass. Might be a CI timing problem then? Will try playing around with the `bpf.perf_buffer_poll()` timeout.

Will investigate further.
Other CI runs with similar behavior:

- https://cirrus-ci.com/task/6650565413830656?logs=ci#L3531

I've been trying to reproduce this by re-running the CI on master https://github.com/bitcoin/bitcoin/commit/49b87bfe7e2799d25ce709123ecafa872b36e87a: https://cirrus-ci.com/task/4749392670883840

So far all runs passed..
Maybe it reproduces faster with `rr`?
Thanks, didn't know `rr`. Especially the chaos mode (https://robert.ocallahan.org/2016/02/introducing-rr-chaos-mode.html) seems to be helpful here. However,  rr and bpf don't work well together yet. `rr` asserts when running the python USDT interface tests. I've tried to patch `rr` to make this work but haven't been successful yet.
Again here https://cirrus-ci.com/task/5250935128064000?logs=ci#L3275
Gave this another look. No luck with `rr` and haven't been able to reproduce this locally or in the CI.
Maybe it will "fix itself" after https://github.com/bitcoin/bitcoin/pull/27360 ? 
> Maybe it will "fix itself" after #27360 ?

Unfortunately, it didn't (https://github.com/bitcoin/bitcoin/runs/13206824577 from master after #27360 was merged)
master @ e0a70c5b4f2c691e8d6b507c8ce879f0b0424254. https://cirrus-ci.com/task/5247512164958208:
```bash
 node0 2023-05-10T17:29:06.449483Z (mocktime: 2023-05-24T17:29:09Z) [httpworker.1] [rpc/request.cpp:181] [parse] [rpc] ThreadRPCServer method=getrawmempool user=__cookie__ 
 test  2023-05-10T17:29:06.450000Z TestFramework (INFO): Polling buffer... 
 test  2023-05-10T17:29:06.486000Z TestFramework (INFO): Ensuring mempool:rejected event was handled successfully... 
 test  2023-05-10T17:29:06.486000Z TestFramework (ERROR): Assertion failed 
                                   Traceback (most recent call last):
                                     File "/tmp/cirrus-ci-build/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/test_framework/test_framework.py", line 132, in main
                                       self.run_test()
                                     File "/tmp/cirrus-ci-build/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/interface_usdt_mempool.py", line 339, in run_test
                                       self.rejected_test()
                                     File "/tmp/cirrus-ci-build/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/interface_usdt_mempool.py", line 323, in rejected_test
                                       assert_equal(EXPECTED_REJECTED_EVENTS, handled_rejected_events)
                                     File "/tmp/cirrus-ci-build/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/test_framework/util.py", line 55, in assert_equal
                                       raise AssertionError("not(%s)" % " == ".join(str(arg) for arg in (thing1, thing2) + args))
                                   AssertionError: not(1 == 0)
 test  2023-05-10T17:29:06.487000Z TestFramework (DEBUG): Closing down network thread
...
interface_usdt_mempool.py                              | ✖ Failed  | 5 s
```
https://cirrus-ci.com/task/6382990889058304?logs=ci#L3266
I'll give this another look next week.
@0xB10C and I had a look at this today. Here's what we found so far:
1. We believe the issue is caused by [one or more of the assertions](https://github.com/bitcoin/bitcoin/blob/b22408df162a224d94ac54e8443b57ef3fd2ca72/test/functional/interface_usdt_mempool.py#L306-L307) failing in the `handle_rejected_event` callback (although we are not sure why they fail).
2. The test framework's `assert_equal()` communicates failed assertions via exceptions. Unfortunately exceptions raised in bcc callback functions are not properly propagated (`Exception ignored on calling ctypes callback function: <function PerfEventArray._open_perf_buffer.<locals>.raw_cb_ at 0xffff83490e50>`), so they only appear on stderr, not in the logs. As a consequence, the logs only show [that the test case failed](https://github.com/bitcoin/bitcoin/blob/b22408df162a224d94ac54e8443b57ef3fd2ca72/test/functional/interface_usdt_mempool.py#L326), not why it failed.
3. Because (2) is a general problem with functional tests for tracepoints, I'll create a PR that fixes the issue for all of them. The best solution we came up with so far is wrapping all `assert_equals()` in bcc callback functions in `try` blocks and logging any `AssertionError` exceptions using `self.log()`. Once this (or a better solution, preferable one that involves not ignoring exceptions in ctype callback functions, if that's possible) is merged, the logs should tell us why the `mempool:rejected` test fails sporadically.
Since the failure hasn't been observed since https://github.com/bitcoin/bitcoin/pull/27177 was merged, there's a good chance it fixed the issue.

However, it still seems like a good idea to address the issue of missing `check_equal()` exception information in the logs, so I created https://github.com/bitcoin/bitcoin/pull/27831 to make sure all tracepoint tests properly log failed `check_equal()` assertions.
Ok, closing for now. Let's reopen if this is an issue again.
> Since the failure hasn't been observed since https://github.com/bitcoin/bitcoin/pull/27177 was merged, there's a good chance it fixed the issue.

For future reference, adding a bit more details on this what we think was happening here. We couldn't actually verify it as we haven't been able to reproduce this locally and the assertions in the callback aren't logged in the CI logs (fixed in #27831). 

The `mempool:rejected` interface test constructs a zero-fee transaction and sends it to the test-node via P2P. This transaction is policy-rejected. The interface test tests that the txid and the rejection reason are what we expect them to be. Due to `get_utxo()` sometimes using an immature UTXO (see #27177), the transaction was consensus-rejected with `bad-txns-premature-spend-of-coinbase`. The `assert_equal("bad-txns-premature-spend-of-coinbase", "min relay fee not met")` would fail. 
Can it be forced to "reproduce" on master by adding a typo to the error string in assert_equal?
> Can it be forced to "reproduce" on master by adding a typo to the error string in assert_equal?

see https://github.com/0xB10C/bitcoin/commit/4091d621f16b6316eef5551a56c00bf0e2c0e541 and this CI task https://cirrus-ci.com/task/4909169740873728
https://cirrus-ci.com/task/6053265947754496 ? 😢 
https://github.com/bitcoin/bitcoin/pull/27831 improves the logging, probably.
https://cirrus-ci.com/task/5268139265490944
https://cirrus-ci.com/task/4626519738286080?logs=ci#L3262

```
 node0 2023-06-29T19:45:37.525879Z (mocktime: 2023-07-13T19:45:38Z) [httpworker.2] [rpc/request.cpp:181] [parse] [rpc] ThreadRPCServer method=getrawmempool user=__cookie__ 
 test  2023-06-29T19:45:37.526000Z TestFramework (INFO): Polling buffer... 
 test  2023-06-29T19:45:37.527000Z TestFramework (INFO): Ensuring mempool:rejected event was handled successfully... 
 test  2023-06-29T19:45:37.527000Z TestFramework (ERROR): Assertion failed 
                                   Traceback (most recent call last):
                                     File "/tmp/cirrus-ci-build/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/test_framework/test_framework.py", line 131, in main
                                       self.run_test()
                                     File "/tmp/cirrus-ci-build/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/interface_usdt_mempool.py", line 329, in run_test
                                       self.rejected_test()
                                     File "/tmp/cirrus-ci-build/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/interface_usdt_mempool.py", line 310, in rejected_test
                                       assert_equal(event.reason.decode("UTF-8"), "min relay fee not met")
                                     File "/tmp/cirrus-ci-build/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/test_framework/util.py", line 57, in assert_equal
                                       raise AssertionError("not(%s)" % " == ".join(str(arg) for arg in (thing1, thing2) + args))
                                   AssertionError: not( == min relay fee not met)
https://cirrus-ci.com/task/6504094303518720?logs=ci#L3264
```bash
 test  2023-07-03T14:16:38.121000Z TestFramework (INFO): Polling buffer... 
 node0 2023-07-03T14:16:38.121067Z (mocktime: 2023-07-17T14:16:35Z) [http] [httpserver.cpp:255] [http_request_cb] [http] Received a POST request for / from 127.0.0.1:40312 
 node0 2023-07-03T14:16:38.121207Z (mocktime: 2023-07-17T14:16:35Z) [httpworker.2] [rpc/request.cpp:181] [parse] [rpc] ThreadRPCServer method=getrawmempool user=__cookie__ 
 test  2023-07-03T14:16:38.122000Z TestFramework (INFO): Ensuring mempool:rejected event was handled successfully... 
 test  2023-07-03T14:16:38.122000Z TestFramework (ERROR): Assertion failed 
                                   Traceback (most recent call last):
                                     File "/tmp/cirrus-ci-build/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/test_framework/test_framework.py", line 131, in main
                                       self.run_test()
                                     File "/tmp/cirrus-ci-build/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/interface_usdt_mempool.py", line 329, in run_test
                                       self.rejected_test()
                                     File "/tmp/cirrus-ci-build/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/interface_usdt_mempool.py", line 310, in rejected_test
                                       assert_equal(event.reason.decode("UTF-8"), "min relay fee not met")
                                     File "/tmp/cirrus-ci-build/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/test_framework/util.py", line 57, in assert_equal
                                       raise AssertionError("not(%s)" % " == ".join(str(arg) for arg in (thing1, thing2) + args))
                                   AssertionError: not( == min relay fee not met)
 test  2023-07-03T14:16:38.123000Z TestFramework (DEBUG): Closing down network thread 
 test  2023-07-03T14:16:38.173000Z TestFramework (INFO): Stopping nodes 
 test  2023-07-03T14:16:38.173000Z TestFramework.node0 (DEBUG): Stopping node 
```
Maybe remove the sub-test case?
Even with the extra information, I'm somewhat at a loss because I cannot reproduce the problem locally. From what I understand:
- the tracepoint got called, the handler was executed
- the tx hash is correct (so `bpf_usdt_readarg_p(1, ctx, &rejected.hash, HASH_LENGTH)` in `trace_rejected` works as expected)
- reject reason is usually correct; but sometime's it's an empty string, suggesting something went wrong with `bpf_usdt_readarg_p(2, ctx, &rejected.reason, MAX_REJECT_REASON_LENGTH)` in `trace_rejected`. Why this only would only happen somethings, I do not know. If the environment is 100% reproducible, this should not happen.

My first reaction to your suggestion of removing the sub-test case was to first try and change the code so it triggers another rejection reason. But without knowing what's going wrong right now, this feels like poking around in the dark.
Looking at the code, I fail to see how this could even pass once, because you can't assume a pointer created by `std::string{}.c_str()` can be used after the statement at all, no?
Or do the `TRACEx` macros create a copy of the data in-line? If yes, it could make sense to document that in doc/tracing.md#c_str
AFAIU the macros themselves don't copy any data, rather just define a static probe point. Any arguments passed to them just get incorporated into the assembly code. The tracing program (BPF) if loaded into the kernel can run code at the trace points. Here it can choose to, and AFAIU does in our case with BPF, copy the data into a different kernel space map when it hits a tracepoint, which can then be read by the USDT probe in our python test.

So the data does get copied but not by the macro, rather by the BPF program.
Is the tracing program executed before the "full-expression" that encompasses the `std::string{}.c_str()` call has been fully evaluated?

Anything that executes after the expression has been evaluated, will operate on destroyed memory. See https://en.cppreference.com/w/cpp/language/lifetime
I'm probably outside of my wheelhouse now, but as I understand the tracepoint defines a place in the binary where a tracing program gets to insert and run (its own) code, if it's loaded in the kernel. The arguments to the macro are passed directly to the (assembly) code provided by the tracing program _in that expression_ and at that execution point BPF will copy values into it's own map in the kernel if a probe is set up. Other tracing programs or runtimes can choose not to copy data if they don't need/want to.

By using helper function `bpf_usdt_readarg` in our test program we should be able to read (pointers) between kernel and user space safely too. In the recent failure here, in theory, the macro _should_ handle the pointers ok as neither `tx` nor `result` go out of scope before it's called, at which point the copy should be done.

All that said, it's obviously still going wrong somehow...
> neither tx nor result go out of scope before it's called

I don't think the lifetime of `result` is relevant here. `GetRejectReason()` gives birth to a fresh temporary copy of the string, having its own lifetime.

> All that said, it's obviously still going wrong somehow...

Yeah, I am just guessing that this is a lifetime issue. It would be good to rule this out.
Ok, I'll disable the broken test due to lack of progress here. 
I guess we can close this for now, and can reopen if #30559 is reverted?
https://github.com/bitcoin/bitcoin/actions/runs/10214494976/job/28262011108?pr=29641#step:7:9088

```
test  2024-08-02T11:52:52.096000Z TestFramework (INFO): Hooking into mempool:rejected tracepoint... 
 test  2024-08-02T11:52:53.880000Z TestFramework (INFO): Sending invalid transaction... 
 node0 2024-08-02T11:52:53.880644Z (mocktime: 2024-08-16T11:52:53Z) [http] [httpserver.cpp:305] [http_request_cb] [http] Received a POST request for / from 127.0.0.1:46704 
 node0 2024-08-02T11:52:53.880818Z (mocktime: 2024-08-16T11:52:53Z) [httpworker.2] [rpc/request.cpp:232] [parse] [rpc] ThreadRPCServer method=getblockchaininfo user=__cookie__ 
 test  2024-08-02T11:52:53.883000Z TestFramework.p2p (DEBUG): Send message to 127.0.0.1:13268: msg_tx(tx=CTransaction(version=2 vin=[CTxIn(prevout=COutPoint(hash=6e95b1706527bac148b682dad0df924b578a2b92fa24de08d5eb6e846b06c07e n=0) scriptSig= nSequence=0)] vout=[CTxOut(nValue=50.00000000 scriptPubKey=51202913b252fe537830f843bfdc5fa7d20ba48639a87c86ff837b92d083c55ad7c1)] wit=CTxWitness(CScriptWitness(51,c00000000000000000000000000000000000000000000000000000000000000001)) nLockTime=0)) 
 test  2024-08-02T11:52:53.884000Z TestFramework.p2p (DEBUG): Send message to 127.0.0.1:13268: msg_ping(nonce=00000000) 
 test  2024-08-02T11:52:53.884000Z TestFramework.p2p (DEBUG): Send message to 127.0.0.1:13268: msg_ping(nonce=00000002) 
 node0 2024-08-02T11:52:53.886782Z (mocktime: 2024-08-16T11:52:53Z) [msghand] [net_processing.cpp:3710] [ProcessMessage] [net] received: tx (133 bytes) peer=0 
 node0 2024-08-02T11:52:53.887424Z (mocktime: 2024-08-16T11:52:53Z) [msghand] [txmempool.cpp:664] [check] [mempool] Checking mempool with 0 transactions and 0 inputs 
 node0 2024-08-02T11:52:53.887636Z (mocktime: 2024-08-16T11:52:53Z) [msghand] [net_processing.cpp:3209] [ProcessInvalidTx] [mempoolrej] 1ca65439fa59f1a719bc6cfd2dd34fb9c2dc6a8c699ece5b4d5397b084f37f95 (wtxid=15bb5f2afba6c18e6ba825160ebcf8fb735e2755045400439d1242cb6499a4e9) from peer=0 was not accepted: min relay fee not met, 0 < 104 
 node0 2024-08-02T11:52:53.887802Z (mocktime: 2024-08-16T11:52:53Z) [msghand] [net_processing.cpp:3710] [ProcessMessage] [net] received: ping (8 bytes) peer=0 
 node0 2024-08-02T11:52:53.887841Z (mocktime: 2024-08-16T11:52:53Z) [msghand] [net.cpp:3808] [PushMessage] [net] sending pong (8 bytes) peer=0 
 node0 2024-08-02T11:52:53.887991Z (mocktime: 2024-08-16T11:52:53Z) [msghand] [net_processing.cpp:3710] [ProcessMessage] [net] received: ping (8 bytes) peer=0 
 test  2024-08-02T11:52:53.888000Z TestFramework.p2p (DEBUG): Received message from 127.0.0.1:13268: msg_pong(nonce=00000000) 
 test  2024-08-02T11:52:53.888000Z TestFramework.p2p (DEBUG): Received message from 127.0.0.1:13268: msg_pong(nonce=00000002) 
 node0 2024-08-02T11:52:53.888038Z (mocktime: 2024-08-16T11:52:53Z) [msghand] [net.cpp:3808] [PushMessage] [net] sending pong (8 bytes) peer=0 
 node0 2024-08-02T11:52:53.935933Z (mocktime: 2024-08-16T11:52:53Z) [http] [httpserver.cpp:305] [http_request_cb] [http] Received a POST request for / from 127.0.0.1:46704 
 test  2024-08-02T11:52:53.936000Z TestFramework (INFO): Polling buffer... 
 node0 2024-08-02T11:52:53.936089Z (mocktime: 2024-08-16T11:52:53Z) [httpworker.0] [rpc/request.cpp:232] [parse] [rpc] ThreadRPCServer method=getrawmempool user=__cookie__ 
 test  2024-08-02T11:52:53.937000Z TestFramework (INFO): Ensuring mempool:rejected event was handled successfully... 
 test  2024-08-02T11:52:53.937000Z TestFramework (ERROR): Assertion failed 
                                   Traceback (most recent call last):
                                     File "/home/runner/work/_temp/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/test_framework/test_framework.py", line 132, in main
                                       self.run_test()
                                     File "/home/runner/work/_temp/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/interface_usdt_mempool.py", line 318, in run_test
                                       self.rejected_test()
                                     File "/home/runner/work/_temp/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/interface_usdt_mempool.py", line 299, in rejected_test
                                       assert_equal(event.reason.decode("UTF-8"), "min relay fee not met")
                                     File "/home/runner/work/_temp/ci/scratch/build/bitcoin-x86_64-pc-linux-gnu/test/functional/test_framework/util.py", line 75, in assert_equal
                                       raise AssertionError("not(%s)" % " == ".join(str(arg) for arg in (thing1, thing2) + args))
                                   AssertionError: not( == min relay fee not met)
I'm once again trying to (reliably) reproduce this to be able to have a chance at debugging it. My current approach is running the mempool:rejected test 100 times along with the other functional tests (https://github.com/0xB10C/bitcoin/tree/2024-08-27380-intermitting-usdt-mempool-failures). 

However, no luck yet getting it to a point where I can reproduce it reliably. Sometimes [all 100 runs succeed](https://github.com/0xB10C/bitcoin/actions/runs/10255800979/job/28373512626), other times it fails at the [second run](https://github.com/0xB10C/bitcoin/actions/runs/10255800979/job/28375206006#step:7:4704).

Edit:

- Attempt 3 succeed: https://github.com/0xB10C/bitcoin/actions/runs/10255800979/attempts/3
- Attempt 4 succeed: https://github.com/0xB10C/bitcoin/actions/runs/10255800979/attempts/4
- Attempt 5 failed: https://github.com/0xB10C/bitcoin/actions/runs/10255800979/attempts/5 (test iteration 8)
- Attempt 6 succeed: https://github.com/0xB10C/bitcoin/actions/runs/10255800979/attempts/6
- Attempt 7 failed: https://github.com/0xB10C/bitcoin/actions/runs/10255800979/attempts/7 (test iteration 8)
- Attempt 8 failed: https://github.com/0xB10C/bitcoin/actions/runs/10255800979/attempts/8 (test iteration 3)
Given that you can reproduce on 50% of your new action runs, does the following diff work around the bug?


```diff
diff --git a/src/validation.cpp b/src/validation.cpp
index cf4dce3762..d2b6969370 100644
--- a/src/validation.cpp
+++ b/src/validation.cpp
@@ -1873,10 +1873,13 @@ MempoolAcceptResult AcceptToMemoryPool(Chainstate& active_chainstate, const CTra
 
         for (const COutPoint& hashTx : coins_to_uncache)
             active_chainstate.CoinsTip().Uncache(hashTx);
+        std::string copy{result.m_state.GetRejectReason()};
+        auto c_str{copy.c_str()};
         TRACE2(mempool, rejected,
                 tx->GetHash().data(),
-                result.m_state.GetRejectReason().c_str()
+                c_str
         );
+        UninterruptibleSleep(1ms);
     }
     // After we've (potentially) uncached entries, ensure our coins cache is still within its size limits
     BlockValidationState state_dummy;

Thanks! Did a few CI runs of https://github.com/0xB10C/bitcoin/commit/3d806eafa04415791b95ed20525eed2fd54f5b38. Your patch doesn't seem to improve the situation.

- Attempt 1 failed: https://github.com/0xB10C/bitcoin/actions/runs/10348240621/attempts/1 (test iteration 4/100)
- Attempt 2 failed: https://github.com/0xB10C/bitcoin/actions/runs/10348240621/attempts/2 (test iteration 18/100)
- Attempt 3 succeded: https://github.com/0xB10C/bitcoin/actions/runs/10348240621/attempts/3
- Attempt 4 succeded: https://github.com/0xB10C/bitcoin/actions/runs/10348240621/attempts/4
- Attempt 5 failed: https://github.com/0xB10C/bitcoin/actions/runs/10348240621/attempts/5 (test iteration 26/100)
- Attempt 6 succeded: https://github.com/0xB10C/bitcoin/actions/runs/10348240621/attempts/6
- Attempt 7 failed: https://github.com/0xB10C/bitcoin/actions/runs/10348240621/attempts/7 (test iteration 11/100)
- Attempt 8 succeded: https://github.com/0xB10C/bitcoin/actions/runs/10348240621/attempts/8
Maybe copying the string into a static or global variable would help rule out a lifetime issue?

```diff
diff --git a/src/validation.cpp b/src/validation.cpp
index 9126d8e928..683d13e0e5 100644
--- a/src/validation.cpp
+++ b/src/validation.cpp
@@ -1863,6 +1863,8 @@ MempoolAcceptResult AcceptToMemoryPool(Chainstate& active_chainstate, const CTra
     CTxMemPool& pool{*active_chainstate.GetMempool()};
 
     std::vector<COutPoint> coins_to_uncache;
+    static char string_buffer[120];
+
     auto args = MemPoolAccept::ATMPArgs::SingleAccept(chainparams, accept_time, bypass_limits, coins_to_uncache, test_accept);
     MempoolAcceptResult result = MemPoolAccept(pool, active_chainstate).AcceptSingleTransaction(tx, args);
     if (result.m_result_type != MempoolAcceptResult::ResultType::VALID) {
@@ -1873,9 +1875,10 @@ MempoolAcceptResult AcceptToMemoryPool(Chainstate& active_chainstate, const CTra
 
         for (const COutPoint& hashTx : coins_to_uncache)
             active_chainstate.CoinsTip().Uncache(hashTx);
+        strncpy(string_buffer, result.m_state.GetRejectReason().c_str(), 120 - 1);
         TRACE2(mempool, rejected,
                 tx->GetHash().data(),
-                result.m_state.GetRejectReason().c_str()
+                string_buffer
         );
     }
     // After we've (potentially) uncached entries, ensure our coins cache is still within its size limits

Thanks @mzumsande - [all 11 CI runs with your patch succeed](https://github.com/0xB10C/bitcoin/actions/runs/10359982384/job/28712303493). 

This should confirm it's indeed a lifetime issue. 
This is still really confusing to me.

I don't understand tracepoints in depth, but it looks like there is a race between bitcoind continuing to run (and overwriting data after it goes out of scope), and the data-collecting program (bpftrace) reading data from the memory location  the `c_str()` result points to. But if that was the case, it would be a very general problem with tracepoints, and I don't see how it would specifically lead to failures in this exact situation with the `mempool-rejected` tracepoint.

I also don't understand why this specifically happens in CI runs with sanitizers enabled. Naively I would expect that sanitizers make the C++ program run slower (while not affecting the speed of bpftrace), so if there really was a race, I would expect failures to happen less often with sanitizers, not more.
To fix a potential lifetime issue, it is probably helpful to replace `bpf_usdt_readarg_p()` (which isn't even documented) in:

https://github.com/bitcoin/bitcoin/blob/f7144b24be09e7db2173a0b15d73f99a08b98118/test/functional/interface_usdt_mempool.py#L103-L104

with `bpf_usdt_readarg()` followed by [`bpf_probe_read_user()`](https://github.com/iovisor/bcc/blob/master/docs/reference_guide.md#10-bpf_probe_read_user) or a [`bpf_probe_read_user_str()`](https://github.com/iovisor/bcc/blob/master/docs/reference_guide.md#11-bpf_probe_read_user_str). We use `bpf_usdt_readarg_p()` in multiple places.

From the documentation:
> For safety, all user address space memory reads must pass through bpf_probe_read_user().


This is also how it's documented in ["6. USDT probes - bcc reference guide"](https://github.com/iovisor/bcc/blob/master/docs/reference_guide.md#6-usdt-probes) and how it's done in e.g. this example [`mysqld_query.py`](https://github.com/iovisor/bcc/blob/8d85dcfac86bb7402a20bea5ceba373e5e019b6c/examples/tracing/mysqld_query.py#L36-L37).

Happy to give this another spin.
> Happy to give this another spin.

To check that this is still an issue, I ran https://github.com/bitcoin/bitcoin/compare/master...0xB10C:bitcoin:refs/heads/2024-11-27380-intermitting-usdt-mempool-failures in CI [20 times](https://github.com/0xB10C/bitcoin/actions/runs/12040804871). It failed 6 times. I then ran it with https://github.com/0xB10C/bitcoin/commit/8a3eeb48482955e9d0cd26632a03457e35a10568 and it [didn't fail a single time](https://github.com/0xB10C/bitcoin/actions/runs/12196112065/attempts/20). It seems that getting rid of `bpf_usdt_readarg_p` indeed fixes this.

I'll work on a PR to get rid of our `bpf_usdt_readarg_p` usage in all tests.

Fix patch:
diff --git a/contrib/tracing/log_raw_p2p_msgs.py b/contrib/tracing/log_raw_p2p_msgs.py
index 69f457acb1bb0..1c0f03e19df0a 100755
--- a/contrib/tracing/log_raw_p2p_msgs.py
+++ b/contrib/tracing/log_raw_p2p_msgs.py
@@ -78,6 +78,7 @@
 int trace_inbound_message(struct pt_regs *ctx) {
     int idx = 0;
     struct p2p_message *msg = msg_arr.lookup(&idx);
+    void *paddr = NULL, *pconn_type = NULL, *pmsg_type = NULL, *pmsg = NULL;
 
     // lookup() does not return a NULL pointer. However, the BPF verifier
     // requires an explicit check that that the `msg` pointer isn't a NULL
@@ -85,11 +86,15 @@
     if (msg == NULL) return 1;
 
     bpf_usdt_readarg(1, ctx, &msg->peer_id);
-    bpf_usdt_readarg_p(2, ctx, &msg->peer_addr, MAX_PEER_ADDR_LENGTH);
-    bpf_usdt_readarg_p(3, ctx, &msg->peer_conn_type, MAX_PEER_CONN_TYPE_LENGTH);
-    bpf_usdt_readarg_p(4, ctx, &msg->msg_type, MAX_MSG_TYPE_LENGTH);
+    bpf_usdt_readarg(2, ctx, &paddr);
+    bpf_probe_read_user_str(&msg->peer_addr, sizeof(msg->peer_addr), paddr);
+    bpf_usdt_readarg(3, ctx, &pconn_type);
+    bpf_probe_read_user_str(&msg->peer_conn_type, sizeof(msg->peer_conn_type), pconn_type);
+    bpf_usdt_readarg(4, ctx, &pmsg_type);
+    bpf_probe_read_user_str(&msg->msg_type, sizeof(msg->msg_type), pmsg_type);
     bpf_usdt_readarg(5, ctx, &msg->msg_size);
-    bpf_usdt_readarg_p(6, ctx, &msg->msg, _TRACEPOINT_TEST_MIN(msg->msg_size, MAX_MSG_DATA_LENGTH));
+    bpf_usdt_readarg(6, ctx, &pmsg);
+    bpf_probe_read_user(&msg->msg, _TRACEPOINT_TEST_MIN(msg->msg_size, MAX_MSG_DATA_LENGTH), pmsg);
 
     inbound_messages.perf_submit(ctx, msg, sizeof(*msg));
     return 0;
@@ -99,17 +104,23 @@
     int idx = 0;
     struct p2p_message *msg = msg_arr.lookup(&idx);
 
+    void *paddr = NULL, *pconn_type = NULL, *pmsg_type = NULL, *pmsg = NULL;
+
     // lookup() does not return a NULL pointer. However, the BPF verifier
     // requires an explicit check that that the `msg` pointer isn't a NULL
     // pointer. See https://github.com/iovisor/bcc/issues/2595
     if (msg == NULL) return 1;
 
     bpf_usdt_readarg(1, ctx, &msg->peer_id);
-    bpf_usdt_readarg_p(2, ctx, &msg->peer_addr, MAX_PEER_ADDR_LENGTH);
-    bpf_usdt_readarg_p(3, ctx, &msg->peer_conn_type, MAX_PEER_CONN_TYPE_LENGTH);
-    bpf_usdt_readarg_p(4, ctx, &msg->msg_type, MAX_MSG_TYPE_LENGTH);
+    bpf_usdt_readarg(2, ctx, &paddr);
+    bpf_probe_read_user_str(&msg->peer_addr, sizeof(msg->peer_addr), paddr);
+    bpf_usdt_readarg(3, ctx, &pconn_type);
+    bpf_probe_read_user_str(&msg->peer_conn_type, sizeof(msg->peer_conn_type), pconn_type);
+    bpf_usdt_readarg(4, ctx, &pmsg_type);
+    bpf_probe_read_user_str(&msg->msg_type, sizeof(msg->msg_type), pmsg_type);
     bpf_usdt_readarg(5, ctx, &msg->msg_size);
-    bpf_usdt_readarg_p(6, ctx, &msg->msg,  _TRACEPOINT_TEST_MIN(msg->msg_size, MAX_MSG_DATA_LENGTH));
+    bpf_usdt_readarg(6, ctx, &pmsg);
+    bpf_probe_read_user(&msg->msg, _TRACEPOINT_TEST_MIN(msg->msg_size, MAX_MSG_DATA_LENGTH), pmsg);
 
     outbound_messages.perf_submit(ctx, msg, sizeof(*msg));
     return 0;
diff --git a/contrib/tracing/mempool_monitor.py b/contrib/tracing/mempool_monitor.py
index 4492cb7570841..7c184cce46ed5 100755
--- a/contrib/tracing/mempool_monitor.py
+++ b/contrib/tracing/mempool_monitor.py
@@ -65,8 +65,9 @@
 
 int trace_added(struct pt_regs *ctx) {
   struct added_event added = {};
-
-  bpf_usdt_readarg_p(1, ctx, &added.hash, HASH_LENGTH);
+  void *phash = NULL;
+  bpf_usdt_readarg(1, ctx, phash);
+  bpf_probe_read_user(&added.hash, sizeof(added.hash), phash);
   bpf_usdt_readarg(2, ctx, &added.vsize);
   bpf_usdt_readarg(3, ctx, &added.fee);
 
@@ -76,9 +77,11 @@
 
 int trace_removed(struct pt_regs *ctx) {
   struct removed_event removed = {};
-
-  bpf_usdt_readarg_p(1, ctx, &removed.hash, HASH_LENGTH);
-  bpf_usdt_readarg_p(2, ctx, &removed.reason, MAX_REMOVAL_REASON_LENGTH);
+  void *phash = NULL, *preason = NULL;
+  bpf_usdt_readarg(1, ctx, phash);
+  bpf_probe_read_user(&removed.hash, sizeof(removed.hash), phash);
+  bpf_usdt_readarg(1, ctx, preason);
+  bpf_probe_read_user_str(&removed.reason, sizeof(removed.reason), preason);
   bpf_usdt_readarg(3, ctx, &removed.vsize);
   bpf_usdt_readarg(4, ctx, &removed.fee);
   bpf_usdt_readarg(5, ctx, &removed.entry_time);
@@ -89,22 +92,25 @@
 
 int trace_rejected(struct pt_regs *ctx) {
   struct rejected_event rejected = {};
-
-  bpf_usdt_readarg_p(1, ctx, &rejected.hash, HASH_LENGTH);
-  bpf_usdt_readarg_p(2, ctx, &rejected.reason, MAX_REJECT_REASON_LENGTH);
-
+  void *phash = NULL, *preason = NULL;
+  bpf_usdt_readarg(1, ctx, phash);
+  bpf_probe_read_user(&rejected.hash, sizeof(rejected.hash), phash);
+  bpf_usdt_readarg(1, ctx, preason);
+  bpf_probe_read_user_str(&rejected.reason, sizeof(rejected.reason), preason);
   rejected_events.perf_submit(ctx, &rejected, sizeof(rejected));
   return 0;
 }
 
 int trace_replaced(struct pt_regs *ctx) {
   struct replaced_event replaced = {};
-
-  bpf_usdt_readarg_p(1, ctx, &replaced.replaced_hash, HASH_LENGTH);
+  void *phash_replaced = NULL, *phash_replacement = NULL;
+  bpf_usdt_readarg(1, ctx, phash_replaced);
+  bpf_probe_read_user(&replaced.replaced_hash, sizeof(replaced.replaced_hash), phash_replaced);
   bpf_usdt_readarg(2, ctx, &replaced.replaced_vsize);
   bpf_usdt_readarg(3, ctx, &replaced.replaced_fee);
   bpf_usdt_readarg(4, ctx, &replaced.replaced_entry_time);
-  bpf_usdt_readarg_p(5, ctx, &replaced.replacement_hash, HASH_LENGTH);
+  bpf_usdt_readarg(5, ctx, phash_replacement);
+  bpf_probe_read_user(&replaced.replacement_hash, sizeof(replaced.replacement_hash), phash_replacement);
   bpf_usdt_readarg(6, ctx, &replaced.replacement_vsize);
   bpf_usdt_readarg(7, ctx, &replaced.replacement_fee);
 
diff --git a/contrib/tracing/p2p_monitor.py b/contrib/tracing/p2p_monitor.py
index 63a27fc17da6d..78225366d9c86 100755
--- a/contrib/tracing/p2p_monitor.py
+++ b/contrib/tracing/p2p_monitor.py
@@ -47,11 +47,15 @@
 
 int trace_inbound_message(struct pt_regs *ctx) {
     struct p2p_message msg = {};
+    void *paddr = NULL, *pconn_type = NULL, *pmsg_type = NULL;
 
     bpf_usdt_readarg(1, ctx, &msg.peer_id);
-    bpf_usdt_readarg_p(2, ctx, &msg.peer_addr, MAX_PEER_ADDR_LENGTH);
-    bpf_usdt_readarg_p(3, ctx, &msg.peer_conn_type, MAX_PEER_CONN_TYPE_LENGTH);
-    bpf_usdt_readarg_p(4, ctx, &msg.msg_type, MAX_MSG_TYPE_LENGTH);
+    bpf_usdt_readarg(2, ctx, &paddr);
+    bpf_probe_read_user_str(&msg.peer_addr, sizeof(msg.peer_addr), paddr);
+    bpf_usdt_readarg(3, ctx, &pconn_type);
+    bpf_probe_read_user_str(&msg.peer_conn_type, sizeof(msg.peer_conn_type), pconn_type);
+    bpf_usdt_readarg(4, ctx, &pconn_type);
+    bpf_probe_read_user_str(&msg.msg_type, sizeof(msg.msg_type), pmsg_type);
     bpf_usdt_readarg(5, ctx, &msg.msg_size);
 
     inbound_messages.perf_submit(ctx, &msg, sizeof(msg));
@@ -60,11 +64,15 @@
 
 int trace_outbound_message(struct pt_regs *ctx) {
     struct p2p_message msg = {};
+    void *paddr = NULL, *pconn_type = NULL, *pmsg_type = NULL;
 
     bpf_usdt_readarg(1, ctx, &msg.peer_id);
-    bpf_usdt_readarg_p(2, ctx, &msg.peer_addr, MAX_PEER_ADDR_LENGTH);
-    bpf_usdt_readarg_p(3, ctx, &msg.peer_conn_type, MAX_PEER_CONN_TYPE_LENGTH);
-    bpf_usdt_readarg_p(4, ctx, &msg.msg_type, MAX_MSG_TYPE_LENGTH);
+    bpf_usdt_readarg(2, ctx, &paddr);
+    bpf_probe_read_user_str(&msg.peer_addr, sizeof(msg.peer_addr), paddr);
+    bpf_usdt_readarg(3, ctx, &pconn_type);
+    bpf_probe_read_user_str(&msg.peer_conn_type, sizeof(msg.peer_conn_type), pconn_type);
+    bpf_usdt_readarg(4, ctx, &pconn_type);
+    bpf_probe_read_user_str(&msg.msg_type, sizeof(msg.msg_type), pmsg_type);
     bpf_usdt_readarg(5, ctx, &msg.msg_size);
 
     outbound_messages.perf_submit(ctx, &msg, sizeof(msg));


Test patch:
diff --git a/test/functional/interface_usdt_coinselection.py b/test/functional/interface_usdt_coinselection.py
index f684848aedfcd..1f4d6d6fbf1c4 100755
--- a/test/functional/interface_usdt_coinselection.py
+++ b/test/functional/interface_usdt_coinselection.py
@@ -49,10 +49,13 @@
 
 int trace_selected_coins(struct pt_regs *ctx) {
     struct event_data data;
+    void *pwallet_name = NULL, *palgo = NULL;
     __builtin_memset(&data, 0, sizeof(data));
     data.type = 1;
-    bpf_usdt_readarg_p(1, ctx, &data.wallet_name, WALLET_NAME_LENGTH);
-    bpf_usdt_readarg_p(2, ctx, &data.algo, ALGO_NAME_LENGTH);
+    bpf_usdt_readarg(1, ctx, &pwallet_name);
+    bpf_probe_read_user_str(&data.wallet_name, WALLET_NAME_LENGTH, pwallet_name);
+    bpf_usdt_readarg(2, ctx, &palgo);
+    bpf_probe_read_user_str(&data.algo, ALGO_NAME_LENGTH, palgo);
     bpf_usdt_readarg(3, ctx, &data.target);
     bpf_usdt_readarg(4, ctx, &data.waste);
     bpf_usdt_readarg(5, ctx, &data.selected_value);
@@ -62,9 +65,11 @@
 
 int trace_normal_create_tx(struct pt_regs *ctx) {
     struct event_data data;
+    void *pwallet_name = NULL;
     __builtin_memset(&data, 0, sizeof(data));
     data.type = 2;
-    bpf_usdt_readarg_p(1, ctx, &data.wallet_name, WALLET_NAME_LENGTH);
+    bpf_usdt_readarg(1, ctx, &pwallet_name);
+    bpf_probe_read_user_str(&data.wallet_name, WALLET_NAME_LENGTH, pwallet_name);
     bpf_usdt_readarg(2, ctx, &data.success);
     bpf_usdt_readarg(3, ctx, &data.fee);
     bpf_usdt_readarg(4, ctx, &data.change_pos);
@@ -74,18 +79,22 @@
 
 int trace_attempt_aps(struct pt_regs *ctx) {
     struct event_data data;
+    void *pwallet_name = NULL;
     __builtin_memset(&data, 0, sizeof(data));
     data.type = 3;
-    bpf_usdt_readarg_p(1, ctx, &data.wallet_name, WALLET_NAME_LENGTH);
+    bpf_usdt_readarg(1, ctx, &pwallet_name);
+    bpf_probe_read_user_str(&data.wallet_name, WALLET_NAME_LENGTH, pwallet_name);
     coin_selection_events.push(&data, 0);
     return 0;
 }
 
 int trace_aps_create_tx(struct pt_regs *ctx) {
     struct event_data data;
+    void *pwallet_name = NULL;
     __builtin_memset(&data, 0, sizeof(data));
     data.type = 4;
-    bpf_usdt_readarg_p(1, ctx, &data.wallet_name, WALLET_NAME_LENGTH);
+    bpf_usdt_readarg(1, ctx, &pwallet_name);
+    bpf_probe_read_user_str(&data.wallet_name, WALLET_NAME_LENGTH, pwallet_name);
     bpf_usdt_readarg(2, ctx, &data.use_aps);
     bpf_usdt_readarg(3, ctx, &data.success);
     bpf_usdt_readarg(4, ctx, &data.fee);
diff --git a/test/functional/interface_usdt_mempool.py b/test/functional/interface_usdt_mempool.py
index 44882f642ea60..7535954172289 100755
--- a/test/functional/interface_usdt_mempool.py
+++ b/test/functional/interface_usdt_mempool.py
@@ -75,8 +75,9 @@
 
 int trace_added(struct pt_regs *ctx) {
   struct added_event added = {};
-
-  bpf_usdt_readarg_p(1, ctx, &added.hash, HASH_LENGTH);
+  void *phash = NULL;
+  bpf_usdt_readarg(1, ctx, &phash);
+  bpf_probe_read_user(&added.hash, sizeof(added.hash), phash);
   bpf_usdt_readarg(2, ctx, &added.vsize);
   bpf_usdt_readarg(3, ctx, &added.fee);
 
@@ -86,9 +87,11 @@
 
 int trace_removed(struct pt_regs *ctx) {
   struct removed_event removed = {};
-
-  bpf_usdt_readarg_p(1, ctx, &removed.hash, HASH_LENGTH);
-  bpf_usdt_readarg_p(2, ctx, &removed.reason, MAX_REMOVAL_REASON_LENGTH);
+  void *phash = NULL, *preason = NULL;
+  bpf_usdt_readarg(1, ctx, &phash);
+  bpf_probe_read_user(&removed.hash, sizeof(removed.hash), phash);
+  bpf_usdt_readarg(2, ctx, &preason);
+  bpf_probe_read_user_str(&removed.reason, sizeof(removed.reason), preason);
   bpf_usdt_readarg(3, ctx, &removed.vsize);
   bpf_usdt_readarg(4, ctx, &removed.fee);
   bpf_usdt_readarg(5, ctx, &removed.entry_time);
@@ -99,22 +102,25 @@
 
 int trace_rejected(struct pt_regs *ctx) {
   struct rejected_event rejected = {};
-
-  bpf_usdt_readarg_p(1, ctx, &rejected.hash, HASH_LENGTH);
-  bpf_usdt_readarg_p(2, ctx, &rejected.reason, MAX_REJECT_REASON_LENGTH);
-
+  void *phash = NULL, *preason = NULL;
+  bpf_usdt_readarg(1, ctx, &phash);
+  bpf_probe_read_user(&rejected.hash, sizeof(rejected.hash), phash);
+  bpf_usdt_readarg(2, ctx, &preason);
+  bpf_probe_read_user_str(&rejected.reason, sizeof(rejected.reason), preason);
   rejected_events.perf_submit(ctx, &rejected, sizeof(rejected));
   return 0;
 }
 
 int trace_replaced(struct pt_regs *ctx) {
   struct replaced_event replaced = {};
-
-  bpf_usdt_readarg_p(1, ctx, &replaced.replaced_hash, HASH_LENGTH);
+  void *preplaced_hash = NULL, *preplacement_hash = NULL;
+  bpf_usdt_readarg(1, ctx, &preplaced_hash);
+  bpf_probe_read_user(&replaced.replaced_hash, sizeof(replaced.replaced_hash), preplaced_hash);
   bpf_usdt_readarg(2, ctx, &replaced.replaced_vsize);
   bpf_usdt_readarg(3, ctx, &replaced.replaced_fee);
   bpf_usdt_readarg(4, ctx, &replaced.replaced_entry_time);
-  bpf_usdt_readarg_p(5, ctx, &replaced.replacement_hash, HASH_LENGTH);
+  bpf_usdt_readarg(5, ctx, &preplacement_hash);
+  bpf_probe_read_user(&replaced.replacement_hash, sizeof(replaced.replacement_hash), preplacement_hash);
   bpf_usdt_readarg(6, ctx, &replaced.replacement_vsize);
   bpf_usdt_readarg(7, ctx, &replaced.replacement_fee);
   bpf_usdt_readarg(8, ctx, &replaced.replaced_by_transaction);
@@ -314,10 +320,7 @@ def handle_rejected_event(_, data, __):
         assert_equal(1, len(events))
         event = events[0]
         assert_equal(bytes(event.hash)[::-1].hex(), tx["tx"].hash)
-        # The next test is already known to fail, so disable it to avoid
-        # wasting CPU time and developer time. See
-        # https://github.com/bitcoin/bitcoin/issues/27380
-        #assert_equal(event.reason.decode("UTF-8"), "min relay fee not met")
+        assert_equal(event.reason.decode("UTF-8"), "min relay fee not met")
 
         bpf.cleanup()
         self.generate(self.wallet, 1)
diff --git a/test/functional/interface_usdt_net.py b/test/functional/interface_usdt_net.py
index beb2546153028..ef67534e96c72 100755
--- a/test/functional/interface_usdt_net.py
+++ b/test/functional/interface_usdt_net.py
@@ -91,12 +91,17 @@
 BPF_PERF_OUTPUT(inbound_messages);
 int trace_inbound_message(struct pt_regs *ctx) {
     struct p2p_message msg = {};
+    void *paddr = NULL, *pconn_type = NULL, *pmsg_type = NULL, *pmsg = NULL;
     bpf_usdt_readarg(1, ctx, &msg.peer_id);
-    bpf_usdt_readarg_p(2, ctx, &msg.peer_addr, MAX_PEER_ADDR_LENGTH);
-    bpf_usdt_readarg_p(3, ctx, &msg.peer_conn_type, MAX_PEER_CONN_TYPE_LENGTH);
-    bpf_usdt_readarg_p(4, ctx, &msg.msg_type, MAX_MSG_TYPE_LENGTH);
+    bpf_usdt_readarg(2, ctx, &paddr);
+    bpf_probe_read_user_str(&msg.peer_addr, sizeof(msg.peer_addr), paddr);
+    bpf_usdt_readarg(3, ctx, &pconn_type);
+    bpf_probe_read_user_str(&msg.peer_conn_type, sizeof(msg.peer_conn_type), pconn_type);
+    bpf_usdt_readarg(4, ctx, &pmsg_type);
+    bpf_probe_read_user_str(&msg.msg_type, sizeof(msg.msg_type), pmsg_type);
     bpf_usdt_readarg(5, ctx, &msg.msg_size);
-    bpf_usdt_readarg_p(6, ctx, &msg.msg, _TRACEPOINT_TEST_MIN(msg.msg_size, MAX_MSG_DATA_LENGTH));
+    bpf_usdt_readarg(6, ctx, &pmsg);
+    bpf_probe_read_user(&msg.msg, _TRACEPOINT_TEST_MIN(msg.msg_size, MAX_MSG_DATA_LENGTH), pmsg);
     inbound_messages.perf_submit(ctx, &msg, sizeof(msg));
     return 0;
 }
@@ -104,12 +109,18 @@
 BPF_PERF_OUTPUT(outbound_messages);
 int trace_outbound_message(struct pt_regs *ctx) {
     struct p2p_message msg = {};
+    void *paddr = NULL, *pconn_type = NULL, *pmsg_type = NULL, *pmsg = NULL;
     bpf_usdt_readarg(1, ctx, &msg.peer_id);
-    bpf_usdt_readarg_p(2, ctx, &msg.peer_addr, MAX_PEER_ADDR_LENGTH);
-    bpf_usdt_readarg_p(3, ctx, &msg.peer_conn_type, MAX_PEER_CONN_TYPE_LENGTH);
-    bpf_usdt_readarg_p(4, ctx, &msg.msg_type, MAX_MSG_TYPE_LENGTH);
+    bpf_usdt_readarg(1, ctx, &msg.peer_id);
+    bpf_usdt_readarg(2, ctx, &paddr);
+    bpf_probe_read_user_str(&msg.peer_addr, sizeof(msg.peer_addr), paddr);
+    bpf_usdt_readarg(3, ctx, &pconn_type);
+    bpf_probe_read_user_str(&msg.peer_conn_type, sizeof(msg.peer_conn_type), pconn_type);
+    bpf_usdt_readarg(4, ctx, &pmsg_type);
+    bpf_probe_read_user_str(&msg.msg_type, sizeof(msg.msg_type), pmsg_type);
     bpf_usdt_readarg(5, ctx, &msg.msg_size);
-    bpf_usdt_readarg_p(6, ctx, &msg.msg, _TRACEPOINT_TEST_MIN(msg.msg_size, MAX_MSG_DATA_LENGTH));
+    bpf_usdt_readarg(6, ctx, &pmsg);
+    bpf_probe_read_user(&msg.msg, _TRACEPOINT_TEST_MIN(msg.msg_size, MAX_MSG_DATA_LENGTH), pmsg);
     outbound_messages.perf_submit(ctx, &msg, sizeof(msg));
     return 0;
 };
diff --git a/test/functional/interface_usdt_utxocache.py b/test/functional/interface_usdt_utxocache.py
index 1617c580f30a6..12a11409e031b 100755
--- a/test/functional/interface_usdt_utxocache.py
+++ b/test/functional/interface_usdt_utxocache.py
@@ -35,7 +35,9 @@
 BPF_PERF_OUTPUT(utxocache_add);
 int trace_utxocache_add(struct pt_regs *ctx) {
     struct utxocache_change add = {};
-    bpf_usdt_readarg_p(1, ctx, &add.txid, 32);
+    void *ptxid = NULL;
+    bpf_usdt_readarg(1, ctx, &ptxid);
+    bpf_probe_read_user(&add.txid, sizeof(add.txid), ptxid);
     bpf_usdt_readarg(2, ctx, &add.index);
     bpf_usdt_readarg(3, ctx, &add.height);
     bpf_usdt_readarg(4, ctx, &add.value);
@@ -47,7 +49,9 @@
 BPF_PERF_OUTPUT(utxocache_spent);
 int trace_utxocache_spent(struct pt_regs *ctx) {
     struct utxocache_change spent = {};
-    bpf_usdt_readarg_p(1, ctx, &spent.txid, 32);
+    void *ptxid = NULL;
+    bpf_usdt_readarg(1, ctx, &ptxid);
+    bpf_probe_read_user(&spent.txid, sizeof(spent.txid), ptxid);
     bpf_usdt_readarg(2, ctx, &spent.index);
     bpf_usdt_readarg(3, ctx, &spent.height);
     bpf_usdt_readarg(4, ctx, &spent.value);
@@ -59,7 +63,9 @@
 BPF_PERF_OUTPUT(utxocache_uncache);
 int trace_utxocache_uncache(struct pt_regs *ctx) {
     struct utxocache_change uncache = {};
-    bpf_usdt_readarg_p(1, ctx, &uncache.txid, 32);
+    void *ptxid = NULL;
+    bpf_usdt_readarg(1, ctx, &ptxid);
+    bpf_probe_read_user(&uncache.txid, sizeof(uncache.txid), ptxid);
     bpf_usdt_readarg(2, ctx, &uncache.index);
     bpf_usdt_readarg(3, ctx, &uncache.height);
     bpf_usdt_readarg(4, ctx, &uncache.value);
diff --git a/test/functional/interface_usdt_validation.py b/test/functional/interface_usdt_validation.py
index 8a98a452de703..9b2e708d95598 100755
--- a/test/functional/interface_usdt_validation.py
+++ b/test/functional/interface_usdt_validation.py
@@ -39,7 +39,9 @@
 BPF_PERF_OUTPUT(block_connected);
 int trace_block_connected(struct pt_regs *ctx) {
     struct connected_block block = {};
-    bpf_usdt_readarg_p(1, ctx, &block.hash, 32);
+    void *phash = NULL;
+    bpf_usdt_readarg(1, ctx, &phash);
+    bpf_probe_read_user(&block.hash, sizeof(block.hash), phash);
     bpf_usdt_readarg(2, ctx, &block.height);
     bpf_usdt_readarg(3, ctx, &block.transactions);
     bpf_usdt_readarg(4, ctx, &block.inputs);