Repository: bitcoin/bitcoin (language: cpp)

Issue:
bitcoind hangs waiting for `g_requests.empty()`
### Is there an existing issue for this?

- [X] I have searched the existing issues

### Current behaviour

bitcoind hangs 

### Expected behaviour

bitcoind shouldn't hang

I was able to dump the running threads and it was stuck waiting for `g_requests` to be empty. The issue seems to be that this callback may not actually be called in case of error: https://github.com/bitcoin/bitcoin/blob/22139f6e83a2bedd2dad9f280567d2c76c54252f/src/httpserver.cpp#L222-L227

See this issue https://github.com/libevent/libevent/issues/589 for more details. I think what happens is that interrupting `bitcoin-cli` before it has received the reply calls the error callback and then the success callback is never called.

I think this is also a memory leak since `g_requests` won't get cleaned up. Whether it's severe or not probably depends on the caller and what type of requests they are making. This new code was introduced here https://github.com/bitcoin/bitcoin/pull/26742

### Steps to reproduce

I was running a simple shell script in 4 terminal windows for fun:

```
#!/bin/bash

for i in {1..50000}
do
	./src/bitcoin-cli -rpcuser=admin -rpcpassword=pass getrawtransaction 217819875434324c412d5e938f7f3c1670fb60814974cf7958f1fac90ee5afb7
done
```

I was messing around with this script and randomly Ctrl-C'd each terminal window in rapid succession and then when I waited up to a minute and Ctrl-C'd bitcoind, it didn't shutdown. I can reproduce it pretty reliably as well.

Compilation commands:
```
./autogen.sh; ./configure --without-wallet --with-gui=no --disable-zmq; make -j4
```

Run-time options (note that I was able to reproduce this with the default number of rpcthreads as well):
```
./src/bitcoind -txindex -rpcthreads=1 -debug
```

Output of `gcc --version`:
```
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/4.2.1
Apple clang version 11.0.3 (clang-1103.0.32.62)
Target: x86_64-apple-darwin21.6.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
```

Output of `g++ --version`:
```
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/4.2.1
Apple clang version 11.0.3 (clang-1103.0.32.62)
Target: x86_64-apple-darwin21.6.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
```

My libevent version is 2.1.12

### Relevant log output

```
^C2023-05-22T20:56:10Z [http] Interrupting HTTP server
2023-05-22T20:56:10Z [rpc] Interrupting HTTP RPC server
2023-05-22T20:56:10Z [rpc] Interrupting RPC
2023-05-22T20:56:10Z tor: Thread interrupt
2023-05-22T20:56:10Z Shutdown: In progress...
2023-05-22T20:56:10Z [rpc] Stopping HTTP RPC server
2023-05-22T20:56:10Z torcontrol thread exit
2023-05-22T20:56:10Z [http] Unregistering HTTP handler for / (exactmatch 1)
2023-05-22T20:56:10Z addcon thread exit
2023-05-22T20:56:10Z [http] Unregistering HTTP handler for /wallet/ (exactmatch 0)
2023-05-22T20:56:10Z [rpc] Stopping RPC
2023-05-22T20:56:10Z net thread exit
2023-05-22T20:56:10Z [rpc] RPC stopped.
2023-05-22T20:56:10Z [http] Stopping HTTP server
2023-05-22T20:56:10Z [http] Waiting for HTTP worker threads to exit
2023-05-22T20:56:10Z [http] Exited http event loop
2023-05-22T20:56:10Z [http] Waiting for 1 requests to stop HTTP server
```

The important bit is the very last log line. bitcoind was still alive and periodically printed log lines about feeding dynamic environment data into the RNG.

### How did you obtain Bitcoin Core

Compiled from source

### What version of Bitcoin Core are you using?

17acb2782a66f033238340e4fb81009e7f79e97a

### Operating system and version

macOS intel

### Machine specifications

_No response_


Discussion / hints:
I think #27245 which uses `evhttp_connection_set_closecb` is needed to avoid this, but:
- it doesn't seem to be slated for v25.0
- I experience this in one of the libevent versions that #27245 doesn't address
Thanks for reporting! I observed this issue sporadically, but were never able to reproduce it until now. With the help of your description (_"... interrupting bitcoin-cli before it has received the reply ..."_) I found it's actually quite trivial at least on my machine:

1. startup `bitcoind -debug=http` (and any other desired flags; note that the hangup also happens without the debug flag, but this allows us to see where it's hanging)
2. execute `bitcoin-cli waitforblockheight 1234567` (or any other RPC call which doesn't terminate immediately)
3. terminate bitcoin-cli via CTRL-C
4. terminate bitcoind via CTRL-C

The shutdown leads to the following messages and then hangs forever:
```
2023-05-23T13:58:01Z [http] Unregistering HTTP handler for / (exactmatch 1)
2023-05-23T13:58:01Z [http] Unregistering HTTP handler for /wallet/ (exactmatch 0)
2023-05-23T13:58:01Z [http] Stopping HTTP server
2023-05-23T13:58:01Z [http] Waiting for HTTP worker threads to exit
2023-05-23T13:58:01Z [http] Waiting for 1 requests to stop HTTP server
2023-05-23T13:58:01Z [http] Exited http event loop
2023-05-23T13:58:01Z msghand thread exit
2023-05-23T13:58:01Z net thread exit
2023-05-23T13:58:03Z opencon thread exit
```

Tested on master branch (commit 5ef2c1ee7a7416907e0f9135dc0701a88d92a7f6) with default configuration, running OpenBSD 7.3 with libevent 2.1.12 here.
I was able to fix this with my branch here https://github.com/Crypt-iQ/bitcoin/commit/4fd7adb30f584664421b6431bce8ebcbf7ceef2f by adding a `evhttp_connection_set_closecb` callback. I didn't need to modify the logic related to `EV_READ` and no functional test errors are introduced on my machine
> I was able to fix this with my branch here [Crypt-iQ@4fd7adb](https://github.com/Crypt-iQ/bitcoin/commit/4fd7adb30f584664421b6431bce8ebcbf7ceef2f) by adding a `evhttp_connection_set_closecb` callback. I didn't need to modify the logic related to `EV_READ` and no functional test errors are introduced on my machine

Can confirm that this patch fixes the issue on my side. 
Hmmm, curiously enough I was not able to reproduce using either method on master @ aa6cc5bec9 

With @theStack method:

```log
2023-05-26T10:39:07Z [http] Received a POST request for / from 127.0.0.1:50704
^C2023-05-26T10:39:10Z [http] Interrupting HTTP server
2023-05-26T10:39:10Z tor: Thread interrupt
2023-05-26T10:39:10Z torcontrol thread exit
2023-05-26T10:39:10Z addcon thread exit
2023-05-26T10:39:10Z opencon thread exit
2023-05-26T10:39:10Z Shutdown: In progress...
2023-05-26T10:39:10Z [http] Unregistering HTTP handler for / (exactmatch 1)
2023-05-26T10:39:10Z [http] Unregistering HTTP handler for /wallet/ (exactmatch 0)
2023-05-26T10:39:10Z [http] Stopping HTTP server
2023-05-26T10:39:10Z [http] Waiting for HTTP worker threads to exit
2023-05-26T10:39:10Z [http] Waiting for 1 requests to stop HTTP server
2023-05-26T10:39:10Z [http] Waiting for HTTP event thread to exit
2023-05-26T10:39:10Z [http] Exited http event loop
2023-05-26T10:39:10Z [http] Stopped HTTP server
2023-05-26T10:39:10Z loadblk thread exit
2023-05-26T10:39:10Z net thread exit
2023-05-26T10:39:10Z msghand thread exit
2023-05-26T10:39:10Z DumpAnchors: Flush 0 outbound block-relay-only peer addresses to anchors.dat started
2023-05-26T10:39:10Z DumpAnchors: Flush 0 outbound block-relay-only peer addresses to anchors.dat completed (0.00s)
2023-05-26T10:39:10Z scheduler thread exit
2023-05-26T10:39:16Z Shutdown: done
```

With @Crypt-iQ method:

<details>
<summary>Details</summary>

```log
... many lines like this
2023-05-26T10:42:04Z [http] Received a POST request for / from 127.0.0.1:59546
2023-05-26T10:42:04Z [http] Received a POST request for / from 127.0.0.1:59562
2023-05-26T10:42:04Z [http] Received a POST request for / from 127.0.0.1:59566
2023-05-26T10:42:04Z [http] Received a POST request for / from 127.0.0.1:59570
2023-05-26T10:42:04Z [http] Received a POST request for / from 127.0.0.1:59574
2023-05-26T10:42:04Z [http] Received a POST request for / from 127.0.0.1:59580
2023-05-26T10:42:04Z [http] Received a POST request for / from 127.0.0.1:59590
2023-05-26T10:42:04Z [http] Received a POST request for / from 127.0.0.1:59596
2023-05-26T10:42:04Z [http] Received a POST request for / from 127.0.0.1:59606
2023-05-26T10:42:04Z [http] Received a POST request for / from 127.0.0.1:59616
2023-05-26T10:42:04Z [http] Received a POST request for / from 127.0.0.1:59618
2023-05-26T10:42:04Z [http] Received a POST request for / from 127.0.0.1:59636
2023-05-26T10:42:04Z [http] Received a POST request for / from 127.0.0.1:59638
2023-05-26T10:42:04Z [http] Received a POST request for / from 127.0.0.1:59660
2023-05-26T10:42:04Z [http] Received a POST request for / from 127.0.0.1:59658
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:59686
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:59684
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:59704
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:59706
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:59722
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:59734
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:59740
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:59746
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:59760
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:59774
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:59790
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:59792
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:59798
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:59802
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:59804
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:59818
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:59822
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:59828
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:59826
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:59840
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:59854
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:59864
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:59866
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:59870
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:59868
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:59886
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:59900
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:59914
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:59928
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:59944
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:59960
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:59964
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:59968
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:59980
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:59982
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:60002
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:60004
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:60010
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:60008
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:60018
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:60016
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:60038
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:60044
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:60048
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:60054
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:60066
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:60064
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:60080
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:60082
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:60108
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:60110
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:60124
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:60128
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:60130
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:60142
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:60154
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:60158
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:60170
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:60168
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:60186
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:60184
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:60196
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:60198
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:60224
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:60226
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:60242
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:60244
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:60258
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:60256
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:60284
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:60296
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:60298
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:60310
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:60320
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:60336
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:60344
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:60342
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:60362
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:60370
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:60378
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:60382
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:60386
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:60388
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:60400
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:60402
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:60422
2023-05-26T10:42:05Z [http] Received a POST request for / from 127.0.0.1:60430
2023-05-26T10:42:06Z [http] Received a POST request for / from 127.0.0.1:60436
2023-05-26T10:42:06Z [http] Received a POST request for / from 127.0.0.1:60450
2023-05-26T10:42:06Z [http] Received a POST request for / from 127.0.0.1:60462
2023-05-26T10:42:06Z [http] Received a POST request for / from 127.0.0.1:60468
2023-05-26T10:42:06Z [http] Received a POST request for / from 127.0.0.1:60472
2023-05-26T10:42:06Z [http] Received a POST request for / from 127.0.0.1:60470
2023-05-26T10:42:06Z [http] Received a POST request for / from 127.0.0.1:60494
2023-05-26T10:42:06Z [http] Received a POST request for / from 127.0.0.1:60496
2023-05-26T10:42:06Z [http] Received a POST request for / from 127.0.0.1:60508
2023-05-26T10:42:06Z [http] Received a POST request for / from 127.0.0.1:60510
2023-05-26T10:42:06Z [http] Received a POST request for / from 127.0.0.1:60514
2023-05-26T10:42:06Z [http] Received a POST request for / from 127.0.0.1:60522
2023-05-26T10:42:06Z [http] Received a POST request for / from 127.0.0.1:60524
2023-05-26T10:42:06Z [http] Received a POST request for / from 127.0.0.1:60532
2023-05-26T10:42:06Z [http] Received a POST request for / from 127.0.0.1:60544
2023-05-26T10:42:06Z [http] Received a POST request for / from 127.0.0.1:60552
2023-05-26T10:42:06Z [http] Received a POST request for / from 127.0.0.1:60556
2023-05-26T10:42:06Z [http] Received a POST request for / from 127.0.0.1:60560
2023-05-26T10:42:06Z [http] Received a POST request for / from 127.0.0.1:60570
2023-05-26T10:42:06Z [http] Received a POST request for / from 127.0.0.1:60582
2023-05-26T10:42:06Z [http] Received a POST request for / from 127.0.0.1:60592
2023-05-26T10:42:06Z [http] Received a POST request for / from 127.0.0.1:60598
2023-05-26T10:42:06Z [http] Received a POST request for / from 127.0.0.1:60612
2023-05-26T10:42:06Z [http] Received a POST request for / from 127.0.0.1:60614
2023-05-26T10:42:06Z [http] Received a POST request for / from 127.0.0.1:60620
2023-05-26T10:42:06Z [http] Received a POST request for / from 127.0.0.1:60626
2023-05-26T10:42:06Z [http] Received a POST request for / from 127.0.0.1:60628
2023-05-26T10:42:06Z [http] Received a POST request for / from 127.0.0.1:60646
2023-05-26T10:42:06Z [http] Received a POST request for / from 127.0.0.1:60654
2023-05-26T10:42:06Z [http] Received a POST request for / from 127.0.0.1:60652
2023-05-26T10:42:06Z [http] Received a POST request for / from 127.0.0.1:60676
2023-05-26T10:42:06Z [http] Received a POST request for / from 127.0.0.1:60678
2023-05-26T10:42:06Z [http] Received a POST request for / from 127.0.0.1:60680
2023-05-26T10:42:06Z [http] Received a POST request for / from 127.0.0.1:60688
2023-05-26T10:42:06Z [http] Received a POST request for / from 127.0.0.1:60700
2023-05-26T10:42:06Z [http] Received a POST request for / from 127.0.0.1:60706
2023-05-26T10:42:06Z [http] Received a POST request for / from 127.0.0.1:60710
2023-05-26T10:42:06Z [http] Received a POST request for / from 127.0.0.1:60712
2023-05-26T10:42:06Z [http] Received a POST request for / from 127.0.0.1:60724
2023-05-26T10:42:06Z [http] Received a POST request for / from 127.0.0.1:60736
2023-05-26T10:42:06Z [http] Received a POST request for / from 127.0.0.1:60750
2023-05-26T10:42:06Z [http] Received a POST request for / from 127.0.0.1:60756
2023-05-26T10:42:06Z [http] Received a POST request for / from 127.0.0.1:60760
2023-05-26T10:42:06Z [http] Received a POST request for / from 127.0.0.1:60772
2023-05-26T10:42:06Z [http] Received a POST request for / from 127.0.0.1:60776
2023-05-26T10:42:06Z [http] Received a POST request for / from 127.0.0.1:60788
2023-05-26T10:42:06Z [http] Received a POST request for / from 127.0.0.1:60802
2023-05-26T10:42:06Z [http] Received a POST request for / from 127.0.0.1:60814
2023-05-26T10:42:06Z [http] Received a POST request for / from 127.0.0.1:60816
2023-05-26T10:42:06Z [http] Received a POST request for / from 127.0.0.1:60828
2023-05-26T10:42:06Z [http] Received a POST request for / from 127.0.0.1:60830
2023-05-26T10:42:06Z [http] Received a POST request for / from 127.0.0.1:60846
2023-05-26T10:42:06Z [http] Received a POST request for / from 127.0.0.1:60848
2023-05-26T10:42:06Z [http] Received a POST request for / from 127.0.0.1:60854
2023-05-26T10:42:06Z [http] Received a POST request for / from 127.0.0.1:60866
2023-05-26T10:42:07Z [http] Received a POST request for / from 127.0.0.1:60880
2023-05-26T10:42:07Z [http] Received a POST request for / from 127.0.0.1:60892
2023-05-26T10:42:07Z [http] Received a POST request for / from 127.0.0.1:60896
2023-05-26T10:42:07Z [http] Received a POST request for / from 127.0.0.1:60906
2023-05-26T10:42:07Z [http] Received a POST request for / from 127.0.0.1:60918
2023-05-26T10:42:07Z [http] Received a POST request for / from 127.0.0.1:60934
2023-05-26T10:42:07Z [http] Received a POST request for / from 127.0.0.1:60940
2023-05-26T10:42:07Z [http] Received a POST request for / from 127.0.0.1:60950
2023-05-26T10:42:07Z [http] Received a POST request for / from 127.0.0.1:60960
2023-05-26T10:42:07Z [http] Received a POST request for / from 127.0.0.1:60972
2023-05-26T10:42:07Z [http] Received a POST request for / from 127.0.0.1:60988
2023-05-26T10:42:07Z [http] Received a POST request for / from 127.0.0.1:60994
2023-05-26T10:42:07Z [http] Received a POST request for / from 127.0.0.1:32772
2023-05-26T10:42:07Z [http] Received a POST request for / from 127.0.0.1:32774
2023-05-26T10:42:07Z [http] Received a POST request for / from 127.0.0.1:32776
2023-05-26T10:42:07Z [http] Received a POST request for / from 127.0.0.1:32778
2023-05-26T10:42:07Z [http] Received a POST request for / from 127.0.0.1:32788
2023-05-26T10:42:07Z [http] Received a POST request for / from 127.0.0.1:32802
2023-05-26T10:42:07Z [http] Received a POST request for / from 127.0.0.1:32812
2023-05-26T10:42:07Z [http] Received a POST request for / from 127.0.0.1:32828
2023-05-26T10:42:07Z [http] Received a POST request for / from 127.0.0.1:32830
2023-05-26T10:42:07Z [http] Received a POST request for / from 127.0.0.1:32832
2023-05-26T10:42:07Z [http] Received a POST request for / from 127.0.0.1:32846
2023-05-26T10:42:07Z [http] Received a POST request for / from 127.0.0.1:32854
2023-05-26T10:42:07Z [http] Received a POST request for / from 127.0.0.1:32870
2023-05-26T10:42:07Z [http] Received a POST request for / from 127.0.0.1:32872
2023-05-26T10:42:07Z [http] Received a POST request for / from 127.0.0.1:32876
2023-05-26T10:42:07Z [http] Received a POST request for / from 127.0.0.1:32890
2023-05-26T10:42:07Z [http] Received a POST request for / from 127.0.0.1:32892
2023-05-26T10:42:07Z [http] Received a POST request for / from 127.0.0.1:32904
2023-05-26T10:42:07Z [http] Received a POST request for / from 127.0.0.1:32906
2023-05-26T10:42:07Z [http] Received a POST request for / from 127.0.0.1:32912
2023-05-26T10:42:07Z [http] Received a POST request for / from 127.0.0.1:32928
2023-05-26T10:42:07Z [http] Received a POST request for / from 127.0.0.1:32934
^C2023-05-26T10:42:14Z [http] Interrupting HTTP server
2023-05-26T10:42:14Z tor: Thread interrupt
2023-05-26T10:42:14Z torcontrol thread exit
2023-05-26T10:42:14Z opencon thread exit
2023-05-26T10:42:14Z addcon thread exit
2023-05-26T10:42:14Z Shutdown: In progress...
2023-05-26T10:42:14Z [http] Unregistering HTTP handler for / (exactmatch 1)
2023-05-26T10:42:14Z [http] Unregistering HTTP handler for /wallet/ (exactmatch 0)
2023-05-26T10:42:14Z [http] Stopping HTTP server
2023-05-26T10:42:14Z [http] Waiting for HTTP worker threads to exit
2023-05-26T10:42:14Z [http] Exited http event loop
2023-05-26T10:42:14Z [http] Waiting for HTTP event thread to exit
2023-05-26T10:42:14Z [http] Stopped HTTP server
2023-05-26T10:42:14Z net thread exit
2023-05-26T10:42:14Z loadblk thread exit
2023-05-26T10:42:14Z msghand thread exit
2023-05-26T10:42:14Z DumpAnchors: Flush 0 outbound block-relay-only peer addresses to anchors.dat started
2023-05-26T10:42:14Z DumpAnchors: Flush 0 outbound block-relay-only peer addresses to anchors.dat completed (0.00s)
2023-05-26T10:42:14Z scheduler thread exit
2023-05-26T10:42:15Z Shutdown: done
```

</details>
@willcl-ark can you find what libevent version you're using? 
This bug appeared in the [nix-bitcoin](https://github.com/fort-nix/nix-bitcoin/) [test suite](https://github.com/fort-nix/nix-bitcoin/blob/master/test/tests.py). I can confirm that it is fixed by https://github.com/Crypt-iQ/bitcoin/commit/4fd7adb30f584664421b6431bce8ebcbf7ceef2f.


<details>
<summary>Details</summary>

The test is executed inside a QEMU VM and runs `bitcoind` together with a bunch of client services (like `clightning`, `lnd`, `joinmarket`). It then stops all services, including `bitcoind`.
Sometimes `bitcoind` hangs during shutdown:
1. First, it waits for a few minutes at [httpserver.cpp#L469](https://github.com/bitcoin/bitcoin/blob/v25.0/src/httpserver.cpp#L469) (`[http] Waiting for HTTP worker threads to exit`)
2. Then it hangs indefinitely at [httpserver.cpp#L484](https://github.com/bitcoin/bitcoin/blob/v25.0/src/httpserver.cpp#L484) (`[http] Waiting for 1 requests to stop HTTP server`')

`bitcoind` gets shutdown *after* all client services have exited, so there are never any lingering active connections.

This bug is hard to reproduce in the context of the test suite.
I once could consistently trigger it on my desktop system, but never again after rebooting the system.
Currently, I can reproduce it in 80% of all test runs on our [nix-bitcoin server](https://nixbitcoin.org).

[Here's the log output](https://gist.github.com/erikarvstedt/843f6d5fdb545b52f6bd34bc66651c52) of the shutdown sequence, interspersed with gdb backtraces of all threads.

</details>


@erikarvstedt it seems like there's a call to `importmulti` that is outstanding while bitcoind is trying to shut down
@Crypt-iQ, the slow progress of `importmulti` is indeed a separate issue. Maybe it has existed before and was just uncovered by https://github.com/bitcoin/bitcoin/pull/26742. I'll check this later.
I've updated my previous post. 
> @willcl-ark can you find what libevent version you're using?

I am using libevent 2.1.12 (like TheStack)
I get this when compiling with both gcc and clang-16. I am also using libevent 2.1.12. Is there anything unique about your setup @willcl-ark ? Also, did you Ctrl-C the `waitforblockheight` command before trying to stop bitcoind?
As @willcl-ark, tried @theStack's method with no sucess, my libevent version is 2.1.7, I'll try to update it and retry again. Thanks.

@Crypt-iQ I did: Ctrl-C the waitforblockheight command before trying to stop bitcoind.
> As @willcl-ark, tried @theStack's method with no sucess, my libevent version is 2.1.7, I'll try to update it and retry again. Thanks.
> 
> @Crypt-iQ I did: Ctrl-C the waitforblockheight command before trying to stop bitcoind.

Good to know, I'll have to test out that version
> I get this when compiling with both gcc and clang-16. I am also using libevent 2.1.12. Is there anything unique about your setup @willcl-ark ? Also, did you Ctrl-C the `waitforblockheight` command before trying to stop bitcoind?

I've had more reports of this deadlock so come back to try and repro it again, but don't seem to be able to no matter what I try 🤷🏼‍♂️ 

I am on Ubuntu 23.04, also using Clang 16 and libevent 2.1.12-stable-8ubuntu3. It doesn't look like there are any debian patches applied which could affect this...

<details>
<summary>debug log with http and libevent looks normal</summary>

```log
2023-06-28T08:47:52Z [libevent:debug] event_active: 0x55c039f36038 (fd 12), res 2, callback 0x7ff7a7371270
2023-06-28T08:47:52Z [libevent:debug] event_process_active: event: 0x55c039f36038, EV_READ   call 0x7ff7a7371270
2023-06-28T08:47:52Z [libevent:debug] evhttp_get_request_connection: new request from 127.0.0.1:46142 on 44

2023-06-28T08:47:52Z [libevent:debug] Attempting connection to 127.0.0.1:46142

2023-06-28T08:47:52Z [libevent:debug] event_del: 0x7ff7500053f0 (fd -1), callback 0x7ff7a7361f90
2023-06-28T08:47:52Z [libevent:debug] event_del: 0x7ff750005470 (fd -1), callback 0x7ff7a7362740
2023-06-28T08:47:52Z [libevent:debug] event_add: event: 0x7ff750005470 (fd 44),  EV_WRITE   call 0x7ff7a7362740
2023-06-28T08:47:52Z [libevent:debug] Epoll ADD(4) on fd 44 okay. Old events were 0; read change was 0 (none); write change was 1 (add); close change was 0 (none)
2023-06-28T08:47:52Z [libevent:debug] event_add: event: 0x7ff7500053f0 (fd 44), EV_READ    call 0x7ff7a7361f90
2023-06-28T08:47:52Z [libevent:debug] Epoll MOD(5) on fd 44 okay. Old events were 4; read change was 1 (add); write change was 0 (none); close change was 0 (none)
2023-06-28T08:47:52Z [libevent:debug] event_del: 0x7ff750005470 (fd 44), callback 0x7ff7a7362740
2023-06-28T08:47:52Z [libevent:debug] Epoll MOD(1) on fd 44 okay. Old events were 6; read change was 0 (none); write change was 2 (del); close change was 0 (none)
2023-06-28T08:47:52Z [libevent:debug] event_add: event: 0x7ff7500053f0 (fd 44), EV_READ   EV_TIMEOUT call 0x7ff7a7361f90
2023-06-28T08:47:52Z [libevent:debug] event_add: event 0x7ff7500053f0, timeout in 30 seconds 0 useconds, call 0x7ff7a7361f90
2023-06-28T08:47:52Z [libevent:debug] event_del: 0x7ff750005470 (fd 44), callback 0x7ff7a7362740
2023-06-28T08:47:52Z [libevent:debug] event_add: event: 0x7ff7500053f0 (fd 44), EV_READ   EV_TIMEOUT call 0x7ff7a7361f90
2023-06-28T08:47:52Z [libevent:debug] event_add: event 0x7ff7500053f0, timeout in 30 seconds 0 useconds, call 0x7ff7a7361f90
2023-06-28T08:47:52Z [libevent:debug] timeout_next: event: 0x7ff7500053f0, in 30 seconds, 0 useconds
2023-06-28T08:47:52Z [libevent:debug] epoll_dispatch: epoll_wait reports 1
2023-06-28T08:47:52Z [libevent:debug] event_active: 0x7ff7500053f0 (fd 44), res 2, callback 0x7ff7a7361f90
2023-06-28T08:47:52Z [libevent:debug] event_process_active: event: 0x7ff7500053f0, EV_READ   call 0x7ff7a7361f90
2023-06-28T08:47:52Z [libevent:debug] event_add: event: 0x7ff7500053f0 (fd 44), EV_READ   EV_TIMEOUT call 0x7ff7a7361f90
2023-06-28T08:47:52Z [libevent:debug] event_add: event 0x7ff7500053f0, timeout in 70594 seconds 199020 useconds, call 0x7ff7a7361f90
2023-06-28T08:47:52Z [libevent:debug] evhttp_add_header: key: Host val: 127.0.0.1

2023-06-28T08:47:52Z [libevent:debug] evhttp_add_header: key: Connection val: close

2023-06-28T08:47:52Z [libevent:debug] evhttp_add_header: key: Content-Type val: application/json

2023-06-28T08:47:52Z [libevent:debug] evhttp_add_header: key: Authorization val: Basic xxxxxxx

2023-06-28T08:47:52Z [libevent:debug] evhttp_add_header: key: Content-Length val: 58

2023-06-28T08:47:52Z [libevent:debug] evhttp_read_header: checking for post data on 44

2023-06-28T08:47:52Z [libevent:debug] evhttp_get_body_length: bytes to read: 58 (in buffer 58)

2023-06-28T08:47:52Z [libevent:debug] event_del: 0x7ff7500053f0 (fd 44), callback 0x7ff7a7361f90
2023-06-28T08:47:52Z [libevent:debug] Epoll DEL(1) on fd 44 okay. Old events were 2; read change was 2 (del); write change was 0 (none); close change was 0 (none)
2023-06-28T08:47:52Z [libevent:debug] event_del: 0x7ff7500053f0 (fd 44), callback 0x7ff7a7361f90
2023-06-28T08:47:52Z [libevent:debug] event_del: 0x7ff7500053f0 (fd 44), callback 0x7ff7a7361f90
2023-06-28T08:47:52Z [http] Received a POST request for / from 127.0.0.1:46142
^C2023-06-28T08:47:57Z [http] Interrupting HTTP server
2023-06-28T08:47:57Z tor: Thread interrupt
2023-06-28T08:47:57Z [libevent:debug] event_active: 0x55c0458e3250 (fd -1), res 1, callback 0x7ff7a7366100
2023-06-28T08:47:57Z [libevent:debug] epoll_dispatch: epoll_wait reports 1
2023-06-28T08:47:57Z [libevent:debug] event_active: 0x55c0458e51f0 (fd 30), res 2, callback 0x7ff7a736c690
2023-06-28T08:47:57Z [libevent:debug] event_del: 0x55c0458e3250 (fd -1), callback 0x7ff7a7366100
2023-06-28T08:47:57Z [libevent:debug] event_process_active: event: 0x55c0458e3250,    call 0x7ff7a7366100
2023-06-28T08:47:57Z [libevent:debug] event_base_loop: asked to terminate loop.
2023-06-28T08:47:57Z [libevent:debug] event_del: 0x7ff640001080 (fd -1), callback 0x55c03736e022
2023-06-28T08:47:57Z torcontrol thread exit
2023-06-28T08:47:57Z Shutdown: In progress...
2023-06-28T08:47:57Z addcon thread exit
2023-06-28T08:47:57Z opencon thread exit
2023-06-28T08:47:57Z [http] Unregistering HTTP handler for / (exactmatch 1)
2023-06-28T08:47:57Z [http] Unregistering HTTP handler for /wallet/ (exactmatch 0)
2023-06-28T08:47:57Z [http] Stopping HTTP server
2023-06-28T08:47:57Z [http] Waiting for HTTP worker threads to exit
2023-06-28T08:47:57Z [libevent:debug] evhttp_add_header: key: Content-Type val: application/json

2023-06-28T08:47:57Z [libevent:debug] evhttp_add_header: key: Connection val: close

2023-06-28T08:47:57Z [libevent:debug] event_active: 0x7ff730001240 (fd -1), res 0, callback 0x55c03750235f
2023-06-28T08:47:57Z [libevent:debug] epoll_dispatch: epoll_wait reports 1
2023-06-28T08:47:57Z [libevent:debug] event_active: 0x55c039f35bc0 (fd 10), res 2, callback 0x7ff7a736c690
2023-06-28T08:47:57Z [libevent:debug] event_del: 0x7ff730001240 (fd -1), callback 0x55c03750235f
2023-06-28T08:47:57Z [libevent:debug] event_process_active: event: 0x7ff730001240,    call 0x55c03750235f
2023-06-28T08:47:57Z [libevent:debug] evhttp_add_header: key: Date val: Wed, 28 Jun 2023 08:47:57 GMT

2023-06-28T08:47:57Z [libevent:debug] evhttp_add_header: key: Content-Length val: 123

2023-06-28T08:47:57Z [libevent:debug] evhttp_add_header: key: Connection val: close

2023-06-28T08:47:57Z [libevent:debug] evhttp_write_buffer: preparing to write buffer

2023-06-28T08:47:57Z [libevent:debug] event_add: event: 0x7ff7500053f0 (fd 44), EV_READ   EV_TIMEOUT call 0x7ff7a7361f90
2023-06-28T08:47:57Z [libevent:debug] Epoll ADD(1) on fd 44 okay. Old events were 0; read change was 1 (add); write change was 0 (none); close change was 0 (none)
2023-06-28T08:47:57Z [libevent:debug] event_add: event 0x7ff7500053f0, timeout in 30 seconds 0 useconds, call 0x7ff7a7361f90
2023-06-28T08:47:57Z [libevent:debug] event_add: event: 0x7ff750005470 (fd 44),  EV_WRITE  EV_TIMEOUT call 0x7ff7a7362740
2023-06-28T08:47:57Z [libevent:debug] Epoll MOD(5) on fd 44 okay. Old events were 2; read change was 0 (none); write change was 1 (add); close change was 0 (none)
2023-06-28T08:47:57Z [libevent:debug] event_add: event 0x7ff750005470, timeout in 30 seconds 0 useconds, call 0x7ff7a7362740
2023-06-28T08:47:57Z [libevent:debug] event_add: event: 0x7ff7500053f0 (fd 44), EV_READ   EV_TIMEOUT call 0x7ff7a7361f90
2023-06-28T08:47:57Z [libevent:debug] event_add: event 0x7ff7500053f0, timeout in 30 seconds 0 useconds, call 0x7ff7a7361f90
2023-06-28T08:47:57Z [libevent:debug] event_add: event: 0x7ff750005470 (fd 44),  EV_WRITE  EV_TIMEOUT call 0x7ff7a7362740
2023-06-28T08:47:57Z [libevent:debug] event_add: event 0x7ff750005470, timeout in 30 seconds 0 useconds, call 0x7ff7a7362740
2023-06-28T08:47:57Z [libevent:debug] event_del: 0x7ff730001240 (fd -1), callback 0x55c03750235f
2023-06-28T08:47:57Z [libevent:debug] event_process_active: event: 0x55c039f35bc0, EV_READ   call 0x7ff7a736c690
2023-06-28T08:47:57Z [libevent:debug] timeout_next: event: 0x7ff7500053f0, in 30 seconds, 0 useconds
2023-06-28T08:47:57Z [libevent:debug] epoll_dispatch: epoll_wait reports 1
2023-06-28T08:47:57Z [libevent:debug] event_active: 0x7ff750005470 (fd 44), res 4, callback 0x7ff7a7362740
2023-06-28T08:47:57Z [libevent:debug] event_active: 0x7ff7500053f0 (fd 44), res 2, callback 0x7ff7a7361f90
2023-06-28T08:47:57Z [libevent:debug] event_process_active: event: 0x7ff750005470,  EV_WRITE  call 0x7ff7a7362740
2023-06-28T08:47:57Z [libevent:debug] event_add: event: 0x7ff750005470 (fd 44),  EV_WRITE  EV_TIMEOUT call 0x7ff7a7362740
2023-06-28T08:47:57Z [libevent:debug] event_add: event 0x7ff750005470, timeout in 70599 seconds 923008 useconds, call 0x7ff7a7362740
2023-06-28T08:47:57Z [libevent:debug] event_del: 0x55c039f36658 (fd 11), callback 0x7ff7a7371270
2023-06-28T08:47:57Z [libevent:debug] Epoll DEL(1) on fd 11 okay. Old events were 2; read change was 2 (del); write change was 0 (none); close change was 0 (none)
2023-06-28T08:47:57Z [libevent:debug] event_del: 0x7ff750005470 (fd 44), callback 0x7ff7a7362740
2023-06-28T08:47:57Z [libevent:debug] Epoll MOD(1) on fd 44 okay. Old events were 6; read change was 0 (none); write change was 2 (del); close change was 0 (none)
2023-06-28T08:47:57Z [libevent:debug] event_del: 0x55c039f36038 (fd 12), callback 0x7ff7a7371270
2023-06-28T08:47:57Z [libevent:debug] Epoll DEL(1) on fd 12 okay. Old events were 2; read change was 2 (del); write change was 0 (none); close change was 0 (none)
2023-06-28T08:47:57Z [libevent:debug] event_active: 0x55c04518ec30 (fd -1), res 1, callback 0x7ff7a7366100
2023-06-28T08:47:57Z [http] Waiting for HTTP event thread to exit
2023-06-28T08:47:57Z [libevent:debug] event_callback_finalize_many_: 3 events finalizing
2023-06-28T08:47:57Z [libevent:debug] event_del: 0x7ff7500053f0 (fd 44), callback 0x7ff7a7361f90
2023-06-28T08:47:57Z [libevent:debug] Epoll DEL(1) on fd 44 gave Bad file descriptor: DEL was unnecessary.
2023-06-28T08:47:57Z [libevent:debug] event_del: 0x7ff750005470 (fd 44), callback 0x7ff7a7362740
2023-06-28T08:47:57Z [libevent:debug] event_del: 0x55c04518ec30 (fd -1), callback 0x7ff7a7366100
2023-06-28T08:47:57Z [libevent:debug] event_process_active: event: 0x55c04518ec30,    call 0x7ff7a7366100
2023-06-28T08:47:57Z [libevent:debug] event_process_active: event: 0x7ff750005470,  EV_WRITE  call 0x7ff7a735f910
2023-06-28T08:47:57Z [libevent:debug] event_base_loop: no events registered.
2023-06-28T08:47:57Z [http] Exited http event loop
2023-06-28T08:47:57Z [libevent:debug] event_del: 0x55c039f35bc0 (fd 10), callback 0x7ff7a736c690
2023-06-28T08:47:57Z [libevent:debug] Epoll DEL(1) on fd 10 okay. Old events were 2; read change was 2 (del); write change was 0 (none); close change was 0 (none)
2023-06-28T08:47:57Z [libevent:debug] event_base_free_: 0 events freed
2023-06-28T08:47:57Z [http] Stopped HTTP server
2023-06-28T08:47:57Z loadblk thread exit
2023-06-28T08:47:57Z msghand thread exit
2023-06-28T08:47:57Z net thread exit
2023-06-28T08:47:58Z i2paccept thread exit
2023-06-28T08:47:58Z DumpAnchors: Flush 2 outbound block-relay-only peer addresses to anchors.dat started
2023-06-28T08:47:58Z DumpAnchors: Flush 2 outbound block-relay-only peer addresses to anchors.dat completed (0.01s)
2023-06-28T08:47:58Z [libevent:debug] event_del: 0x55c0458e51f0 (fd 30), callback 0x7ff7a736c690
2023-06-28T08:47:58Z [libevent:debug] Epoll DEL(1) on fd 30 okay. Old events were 2; read change was 2 (del); write change was 0 (none); close change was 0 (none)
2023-06-28T08:47:58Z [libevent:debug] event_base_free_: 0 events freed
2023-06-28T08:47:58Z scheduler thread exit
2023-06-28T08:47:58Z Flushed fee estimates to fee_estimates.dat.
2023-06-28T08:47:59Z Shutdown: done
```

</details>

I wonder libevent on my system using `Epoll` might be the reason I can't reproduce (and that I've seen mainly reports of this on MacOS)? AFAIK MacOS uses `kqueue`, and not sure what BSD is using but I might guess also `kqueue`... Perhaps `kqueue` blocks if it can't delete a closed fd or something?

```
2023-06-28T08:47:57Z [libevent:debug] Epoll DEL(1) on fd 44 gave Bad file descriptor: DEL was unnecessary.
# continues with shutdown
...
```
> I was able to fix this with my branch here [Crypt-iQ@4fd7adb](https://github.com/Crypt-iQ/bitcoin/commit/4fd7adb30f584664421b6431bce8ebcbf7ceef2f) by adding a `evhttp_connection_set_closecb` callback.

This fixed the issue for me too, but I just saw a crash caused by an assert() just 4 lines before the modified code:

    [Thread 0x7ffec0f4b700 (LWP 1828755) exited]
    bitcoin-qt: httpserver.cpp:223: http_request_cb(evhttp_request*, void*)::<lambda(evhttp_request*, void*)>: Assertion `n == 1' failed.

    Thread 35 "b-http" received signal SIGABRT, Aborted.
    [Switching to Thread 0x7fff577fe700 (LWP 1762429)]
    __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
    50      ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
    (gdb) where
    #0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
    #1  0x00007ffff6261537 in __GI_abort () at abort.c:79
    #2  0x00007ffff626140f in __assert_fail_base (fmt=0x7ffff63d8688 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x555556ae7ae2 "n == 1", file=0x555556ae79c2 "httpserver.cpp", line=223, function=<optimized out>) at assert.c:92
    #3  0x00007ffff6270662 in __GI___assert_fail (assertion=0x555556ae7ae2 "n == 1", file=0x555556ae79c2 "httpserver.cpp", line=223, function=0x555556ae7a98 "http_request_cb(evhttp_request*, void*)::<lambda(evhttp_request*, void*)>") at assert.c:101
    #4  0x0000555555dab98b in http_request_cb(evhttp_request*, void*)::{lambda(evhttp_request*, void*)#2}::operator()(evhttp_request*, void*) const ()
    #5  0x0000555555dab9e6 in http_request_cb(evhttp_request*, void*)::{lambda(evhttp_request*, void*)#2}::_FUN(evhttp_request*, void*) ()
    #6  0x00007ffff68e4da9 in  () at /usr/lib/x86_64-linux-gnu/libevent-2.1.so.7
    #7  0x00007ffff68c3233 in  () at /usr/lib/x86_64-linux-gnu/libevent-2.1.so.7
    #8  0x00007ffff68c8b4f in  () at /usr/lib/x86_64-linux-gnu/libevent-2.1.so.7
    #9  0x00007ffff68c928f in event_base_loop () at /usr/lib/x86_64-linux-gnu/libevent-2.1.so.7
    #10 0x0000555555dacf1a in ThreadHTTP(event_base*) ()
    #11 0x0000555555dc4f11 in void std::__invoke_impl<void, void (*)(event_base*), event_base*>(std::__invoke_other, void (*&&)(event_base*), event_base*&&) ()
    #12 0x0000555555dc4b98 in std::__invoke_result<void (*)(event_base*), event_base*>::type std::__invoke<void (*)(event_base*), event_base*>(void (*&&)(event_base*), event_base*&&) ()
    #13 0x0000555555dc48d4 in void std::thread::_Invoker<std::tuple<void (*)(event_base*), event_base*> >::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) ()
    #14 0x0000555555dc47d7 in std::thread::_Invoker<std::tuple<void (*)(event_base*), event_base*> >::operator()() ()
    #15 0x0000555555dc460f in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)(event_base*), event_base*> > >::_M_run() ()
    #16 0x00007ffff6663ed0 in  () at /usr/lib/x86_64-linux-gnu/libstdc++.so.6
    #17 0x00007ffff641aea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
    #18 0x00007ffff633aa2f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
    (gdb) 

I can't tell what the value of n is, because I don't compile with debug enabled any more. But I guess it isn't 1!


> [Thread 0x7ffec0f4b700 (LWP 1828755) exited]
> bitcoin-qt: httpserver.cpp:223: http_request_cb(evhttp_request*

Do you happen to know what libevent version you were on?
> Do you happen to know what libevent version you were on?

It looks like version 2.1.12.

```
$ dpkg -l | grep libevent
ii  libev4:amd64                                                1:4.33-1                           amd64        high-performance event loop library modelled after libevent
ii  libevent-2.1-7:amd64                                        2.1.12-stable-1                    amd64        Asynchronous event notification library
ii  libevent-core-2.1-7:amd64                                   2.1.12-stable-1                    amd64        Asynchronous event notification library (core)
ii  libevent-dev                                                2.1.12-stable-1                    amd64        Asynchronous event notification library (development files)
ii  libevent-extra-2.1-7:amd64                                  2.1.12-stable-1                    amd64        Asynchronous event notification library (extra)
ii  libevent-openssl-2.1-7:amd64                                2.1.12-stable-1                    amd64        Asynchronous event notification library (openssl)
ii  libevent-pthreads-2.1-7:amd64                               2.1.12-stable-1                    amd64        Asynchronous event notification library (pthreads)
$ apt policy libevent-dev
libevent-dev:
  Installed: 2.1.12-stable-1
  Candidate: 2.1.12-stable-1
  Version table:
 *** 2.1.12-stable-1 500
        500 http://deb.debian.org/debian bullseye/main amd64 Packages
        100 /var/lib/dpkg/status
$ ldd bitcoin-qt | grep event
	libevent_pthreads-2.1.so.7 => /usr/lib/x86_64-linux-gnu/libevent_pthreads-2.1.so.7 (0x00007f3ceafca000)
	libevent-2.1.so.7 => /usr/lib/x86_64-linux-gnu/libevent-2.1.so.7 (0x00007f3ceaf71000)
	libevent_core-2.1.so.7 => /usr/lib/x86_64-linux-gnu/libevent_core-2.1.so.7 (0x00007f3ce9eaa000)

$ apt-file search libevent-2.1.so.7
libevent-2.1-7: /usr/lib/x86_64-linux-gnu/libevent-2.1.so.7
libevent-2.1-7: /usr/lib/x86_64-linux-gnu/libevent-2.1.so.7.0.1
$ dpkg -l libevent-2.1-7
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                 Version         Architecture Description
+++-====================-===============-============-=======================================
ii  libevent-2.1-7:amd64 2.1.12-stable-1 amd64        Asynchronous event notification library
$ 
```

Fix patch:
diff --git a/src/httpserver.cpp b/src/httpserver.cpp
index a83f4421d75c0..069511563cc82 100644
--- a/src/httpserver.cpp
+++ b/src/httpserver.cpp
@@ -17,6 +17,7 @@
 #include <rpc/protocol.h> // For HTTP status codes
 #include <shutdown.h>
 #include <sync.h>
+#include <util/check.h>
 #include <util/strencodings.h>
 #include <util/threadnames.h>
 #include <util/translation.h>
@@ -28,7 +29,7 @@
 #include <memory>
 #include <optional>
 #include <string>
-#include <unordered_set>
+#include <unordered_map>
 
 #include <sys/types.h>
 #include <sys/stat.h>
@@ -149,10 +150,61 @@ static GlobalMutex g_httppathhandlers_mutex;
 static std::vector<HTTPPathHandler> pathHandlers GUARDED_BY(g_httppathhandlers_mutex);
 //! Bound listening sockets
 static std::vector<evhttp_bound_socket *> boundSockets;
+
+/**
+ * @brief Helps keep track of open `evhttp_connection`s with active `evhttp_requests`
+ *
+ */
+class HTTPRequestTracker
+{
+private:
+    mutable Mutex m_mutex;
+    mutable std::condition_variable m_cv;
+    //! For each connection, keep a counter of how many requests are open
+    std::unordered_map<const evhttp_connection*, size_t> m_tracker GUARDED_BY(m_mutex);
+
+    void RemoveConnectionInternal(const decltype(m_tracker)::iterator it) EXCLUSIVE_LOCKS_REQUIRED(m_mutex)
+    {
+        m_tracker.erase(it);
+        if (m_tracker.empty()) m_cv.notify_all();
+    }
+public:
+    //! Increase request counter for the associated connection by 1
+    void AddRequest(evhttp_request* req) EXCLUSIVE_LOCKS_REQUIRED(!m_mutex)
+    {
+        const evhttp_connection* conn{Assert(evhttp_request_get_connection(Assert(req)))};
+        WITH_LOCK(m_mutex, ++m_tracker[conn]);
+    }
+    //! Decrease request counter for the associated connection by 1, remove connection if counter is 0
+    void RemoveRequest(evhttp_request* req) EXCLUSIVE_LOCKS_REQUIRED(!m_mutex)
+    {
+        const evhttp_connection* conn{Assert(evhttp_request_get_connection(Assert(req)))};
+        LOCK(m_mutex);
+        auto it{m_tracker.find(conn)};
+        if (it != m_tracker.end() && it->second > 0) {
+            if (--(it->second) == 0) RemoveConnectionInternal(it);
+        }
+    }
+    //! Remove a connection entirely
+    void RemoveConnection(const evhttp_connection* conn) EXCLUSIVE_LOCKS_REQUIRED(!m_mutex)
+    {
+        LOCK(m_mutex);
+        auto it{m_tracker.find(Assert(conn))};
+        if (it != m_tracker.end()) RemoveConnectionInternal(it);
+    }
+    size_t CountActiveConnections() const EXCLUSIVE_LOCKS_REQUIRED(!m_mutex)
+    {
+        return WITH_LOCK(m_mutex, return m_tracker.size());
+    }
+    //! Wait until there are no more connections with active requests in the tracker
+    void WaitUntilEmpty() const EXCLUSIVE_LOCKS_REQUIRED(!m_mutex)
+    {
+        WAIT_LOCK(m_mutex, lock);
+        m_cv.wait(lock, [this]() EXCLUSIVE_LOCKS_REQUIRED(m_mutex) { return m_tracker.empty(); });
+    }
+};
 //! Track active requests
-static GlobalMutex g_requests_mutex;
-static std::condition_variable g_requests_cv;
-static std::unordered_set<evhttp_request*> g_requests GUARDED_BY(g_requests_mutex);
+static HTTPRequestTracker g_requests;
 
 /** Check if a network address is allowed to access the HTTP server */
 static bool ClientAllowed(const CNetAddr& netaddr)
@@ -210,14 +262,15 @@ std::string RequestMethodString(HTTPRequest::RequestMethod m)
 /** HTTP request callback */
 static void http_request_cb(struct evhttp_request* req, void* arg)
 {
-    // Track requests and notify when a request is completed.
+    evhttp_connection* conn{evhttp_request_get_connection(req)};
+    // Track active requests
     {
-        WITH_LOCK(g_requests_mutex, g_requests.insert(req));
-        g_requests_cv.notify_all();
+        g_requests.AddRequest(req);
         evhttp_request_set_on_complete_cb(req, [](struct evhttp_request* req, void*) {
-            auto n{WITH_LOCK(g_requests_mutex, return g_requests.erase(req))};
-            assert(n == 1);
-            g_requests_cv.notify_all();
+            g_requests.RemoveRequest(req);
+        }, nullptr);
+        evhttp_connection_set_closecb(conn, [](evhttp_connection* conn, void* arg) {
+            g_requests.RemoveConnection(conn);
         }, nullptr);
     }
 
@@ -225,7 +278,6 @@ static void http_request_cb(struct evhttp_request* req, void* arg)
     // See https://github.com/libevent/libevent/commit/5ff8eb26371c4dc56f384b2de35bea2d87814779
     // and https://github.com/bitcoin/bitcoin/pull/11593.
     if (event_get_version_number() >= 0x02010600 && event_get_version_number() < 0x02010900) {
-        evhttp_connection* conn = evhttp_request_get_connection(req);
         if (conn) {
             bufferevent* bev = evhttp_connection_get_bufferevent(conn);
             if (bev) {
@@ -473,13 +525,10 @@ void StopHTTPServer()
     }
     boundSockets.clear();
     {
-        WAIT_LOCK(g_requests_mutex, lock);
-        if (!g_requests.empty()) {
-            LogPrint(BCLog::HTTP, "Waiting for %d requests to stop HTTP server\n", g_requests.size());
+        if (const auto n_connections{g_requests.CountActiveConnections()}; n_connections != 0) {
+            LogPrint(BCLog::HTTP, "Waiting for %d connections to stop HTTP server\n", n_connections);
         }
-        g_requests_cv.wait(lock, []() EXCLUSIVE_LOCKS_REQUIRED(g_requests_mutex) {
-            return g_requests.empty();
-        });
+        g_requests.WaitUntilEmpty();
     }
     if (eventHTTP) {
         // Schedule a callback to call evhttp_free in the event base thread, so


Test patch: