Repository: valkey-io/valkey (C)

Issue:
[BUG] valkey-server slows down with `spop <key>` and `srandmember <key>` spinning in hashtableScanDefrag after a few hours
After a few hours of uptime, one of our valkey-server processes (Docker `8.1.1-alpine` and `8.1.1` images) starts consuming 100% CPU usage and only sporadically responds to application requests.

It appears that one of our applications continuously running `SPOP <set> <n>` (while another application is continuously filling this same set with new data) is working just fine until a few hours later where CPU usage suddenly increases from the expected 25% to 100% and Valkey is responding to requests only very slowly and only after 30-60+ seconds.

I captured a couple gdb backtraces that show valkey is spending a lot of time in `hashtableScanDefrag` when this issue happens. All of the gdb backtraces I captured are the exact same except for the function argument pointer and cursor values varying:

```
#0  0x000059218e456671 in hashtableScanDefrag (defragfn=0x0, flags=0, privdata=<synthetic pointer>, fn=<optimized out>, cursor=1972326, ht=0x725cbcb93ab0) at /usr/src/valkey/src/hashtable.c:1722
#1  hashtableScanDefrag (flags=<optimized out>, defragfn=<optimized out>, privdata=<optimized out>, fn=<optimized out>, cursor=<optimized out>, ht=<optimized out>) at /usr/src/valkey/src/hashtable.c:1707
#2  hashtableScan (fn=<optimized out>, privdata=<synthetic pointer>, cursor=1972326, ht=0x725cbcb93ab0) at /usr/src/valkey/src/hashtable.c:1686
#3  hashtableScan (privdata=<synthetic pointer>, fn=<optimized out>, cursor=1972326, ht=0x725cbcb93ab0) at /usr/src/valkey/src/hashtable.c:1685
#4  hashtableSampleEntries (ht=<optimized out>, dst=dst@entry=0x7ffe67b59750, count=count@entry=280) at /usr/src/valkey/src/hashtable.c:2031
#5  0x000059218e4571c2 in hashtableFairRandomEntry (ht=<optimized out>, found=0x7ffe67b5a058) at /usr/src/valkey/src/hashtable.c:2007
#6  0x000059218e4cc7e5 in setTypeRandomElement (setobj=setobj@entry=0x725d06994530, str=str@entry=0x7ffe67b5a0c8, len=len@entry=0x7ffe67b5a0c0, llele=llele@entry=0x7ffe67b5a0b8) at /usr/src/valkey/src/t_set.c:424
#7  0x000059218e4cc97d in setTypePopRandom (set=set@entry=0x725d06994530) at /usr/src/valkey/src/t_set.c:463
#8  0x000059218e4ceb61 in spopWithCountCommand (c=0x726078f57700) at /usr/src/valkey/src/t_set.c:853
#9  0x000059218e4633de in call (c=c@entry=0x726078f57700, flags=flags@entry=3) at /usr/src/valkey/src/server.c:3742
#10 0x000059218e464b4c in processCommand (c=0x726078f57700) at /usr/src/valkey/src/server.c:4366
#11 0x000059218e48bdf1 in processCommandAndResetClient (c=0x726078f57700) at /usr/src/valkey/src/networking.c:3118
#12 processInputBuffer (c=c@entry=0x726078f57700) at /usr/src/valkey/src/networking.c:3246
#13 0x000059218e48d928 in readQueryFromClient (conn=<optimized out>) at /usr/src/valkey/src/networking.c:3351
#14 0x000059218e5ad375 in callHandler (handler=<optimized out>, conn=0x726049b26d80) at /usr/src/valkey/src/connhelpers.h:79
#15 connSocketEventHandler (el=<optimized out>, fd=<optimized out>, clientData=0x726049b26d80, mask=<optimized out>) at /usr/src/valkey/src/socket.c:300
#16 0x000059218e44e449 in aeProcessEvents (flags=27, eventLoop=0x72638b840780) at /usr/src/valkey/src/ae.c:486
#17 aeProcessEvents (flags=27, eventLoop=0x72638b840780) at /usr/src/valkey/src/ae.c:411
#18 aeMain (eventLoop=0x72638b840780) at /usr/src/valkey/src/ae.c:543
#19 0x000059218e442a4a in main (argc=<optimized out>, argv=<optimized out>) at /usr/src/valkey/src/server.c:7219
```

Some more information regarding the specific key this is happening to:
```
127.0.0.1:38542> debug object urls:unretrievable
Value at:0x725d06994530 refcount:1 encoding:hashtable serializedlength:1668169541 lru:1044411 lru_seconds_idle:0 lfu_freq:0 lfu_access_time_minutes:4079
(20.73s)
127.0.0.1:38542> debug htstats-key urls:unretrievable full
Hash table 0 stats (main hash table):
 table size: 14680064
 number of entries: 1850367
 top-level buckets: 2097152
 child buckets: 200170
 max chain length: 4
 avg chain length: 0.10
 chain length distribution:
   0: 1958960 (93.41%)
   1: 81106 (3.87%)
   2: 52290 (2.49%)
   3: 4700 (0.22%)
   4: 96 (0.00%)
Hash table 1 stats (rehashing target):
 table size: 29360128
 number of entries: 14611037
 top-level buckets: 4194304
 child buckets: 699571
 max chain length: 3
 avg chain length: 0.17
 chain length distribution:
   0: 3507259 (83.62%)
   1: 674541 (16.08%)
   2: 12482 (0.30%)
   3: 22 (0.00%)
```

config (note that we've also tried with setting `activedefrag no` and `activerehashing no` **during runtime** with no effect to combat this issue, but we're trying to re-run or tests with starting up valkey using `activedefrag no` now):
```
valkey-server
--activedefrag yes
--aof-timestamp-enabled yes
--appendfsync everysec
--appendonly yes
--bind ::
--bind 0.0.0.0
--client-output-buffer-limit normal 1gb 16mb 15
--client-output-buffer-limit pubsub 64mb 8mb 15
--client-output-buffer-limit replica 16gb 1gb 300
--client-query-buffer-limit 512mb
--databases 1
--dual-channel-replication-enabled yes
--enable-debug-command local
--enable-module-command local
--enable-protected-configs local
--hide-user-data-from-log no
--io-threads 8
--log-format logfmt
--loglevel verbose
--log-timestamp-format iso8601
--maxmemory 128gb
--maxmemory-clients 1%
--maxmemory-policy noeviction
--max-new-connections-per-cycle 100
--no-appendfsync-on-rewrite yes
--oom-score-adj relative
--port ${PORT}
--proto-max-bulk-len 512mb
--repl-backlog-size 64mb
--repl-diskless-load swapdb
--repl-ping-replica-period 1
--repl-timeout 300
--requirepass ${PASS}
--save ''
--set-proc-title yes
--proc-title-template "{title} ${PROJECT} {listen-addr} {server-mode}"
--shutdown-on-sigint nosave
--shutdown-on-sigterm nosave
--shutdown-timeout 300
--supervised no
--tcp-backlog 1024
--tcp-keepalive 30
```


Discussion: @JimB123 Would you be able to take a stab at this?
Tested with starting Valkey with `activedefrag no` instead, I observed the process staying at ~30% CPU usage for about 4 hours and then quickly ramp up to 100% within a few (~10) minutes.

The same set now shows this information:
```
127.0.0.1:38542> debug object urls:unretrievable
Value at:0x7b9cbddac3d8 refcount:1 encoding:hashtable serializedlength:998526629 lru:1058857 lru_seconds_idle:0 lfu_freq:0 lfu_access_time_minutes:4136
(14.93s)
127.0.0.1:38542> debug htstats-key urls:unretrievable full
Hash table 0 stats (main hash table):
 table size: 29360128
 number of entries: 10100000
 top-level buckets: 4194304
 child buckets: 386767
 max chain length: 3
 avg chain length: 0.09
 chain length distribution:
   0: 3811245 (90.87%)
   1: 379352 (9.04%)
   2: 3706 (0.09%)
   3: 1 (0.00%)
(1.46s)
127.0.0.1:38542> 
```

When checking with `monitor` I can confirm that it's still the `spop urls:unretrievable 10000` commands that are getting stuck. Do note that `10000` is the upper limit of a variable that our application automatically determines based on the `scard urls:unretrievable` being within a certain window so most of the time that spop count is significantly less than that (typically anywhere between 10 and 200). I've already tried reducing this 10000 limit to 100 and while it does help a little bit, it only delays valkey eventually slowing down so much that it's not accepting connections anymore.
I agree it doesn't seem related to defrag. The function `hashtableScanDefrag()` is used for scanning the hash table, also for defrag but not only (confusing function name?). In this case it is the SPOP command that selects a random element using `hashtableFairRandomEntry()`. This function scans the hash table starting at a random position to sample a number of entries. Then it pick a random one of these. This algorithm is fair but also slower than the naive method to just pick random hashtable bucket and a random element in the bucket.

The htstats look fine. There are no long chains. Most of the entries are in the top-level buckets.

I can see in the first htstats that you captured it while incremental rehashing was ongoing. You can see this by "Hash table 1 stats (rehashing target)" and that the size of table 1 is twice the size of table 0. During incremental rehashing, the server does some more work to rehash the entries, but when it is done, the speed should be up to normal again. Is this what you are seeing when you say the CPU "quickly ramp up to 100% within a few (~10) minutes"? Does it come back to 30% afterwards?
> Does it come back to 30% afterwards?

Once the CPU usage ramps up to 100%, it stays there. At least from what we could tell from waiting for about 5 hours for it to return back to normal. During those 5 hours, CPU usage was pinned at 100% any time we were doing `spop`s and the server stayed unresponsive for any other commands. Only a restart helped bring the CPU usage back down.
Our applications basically do the following:

* Application A adds new data (random URLs from the internet) to a set `urls:unretrievable`, the fill rate here is typically hundreds, sometimes thousands of URLs per second
* Application B tries to keep the `urls:unretrievable` set within a configured cardinality of about 10100000 items and as such repeatedly executes `spop urls:unretrievable <n>` where `n` is the amount of items over the set limit but at most 10000.

This causes a very high churn rate on the items stored in this set but the set never drops below 10100000 items which I assume is what's initially causing this problem to happen.
> [@JimB123](https://github.com/JimB123) Would you be able to take a stab at this?

As Viktor replied, this is NOT related to "active defrag".  The stack trace indicates that this is internal to the hashtable implementation.
SPOP with a count of 10000 is an expensive operation. For each item to pop, it scans the hashtable buckets to samples some hundred items and then picks a random item among these. Repeating this many times is expensive. Did you measure how long the SPOP command takes to execute?

At some point, the set size exceeded 14680064 so it doubled its hashtable size to 29360128 and rehashed, but later the number of items is down to 10100000 again. Now, the hash table is sparse. Sampling the same number of items in a sparse table requires more memory accesses, so it gets more expensive.

I'm wondering how important this fairness is for SPOP with a high item count. We've tuned the sample size to get good random selection fairness based on some statistical method described in the hashtable unit test. But maybe we can consider relaxing the fairness for SPOP with a large number. @madolson WDYT?
> Did you measure how long the SPOP command takes to execute?

`SPOP urls:unretrievable 10000` takes just a few hundred milliseconds to complete, not _minutes_.

> Sampling the same number of items in a sparse table requires more memory accesses, so it gets more expensive.

Still not 300ms vs 4 minutes.

I've also noted this:
> I've already tried reducing this 10000 limit to 100 and while it does help a little bit, it only delays valkey eventually slowing down so much that it's not accepting connections anymore.


I've had a script running in the background executing `debug htstats-key urls:unretrievable full` with a `sleep 60` in between. You can see exactly when Valkey suddenly started slowing down significantly versus it taking less than a full second to execute that same command because of the event loop being blocked for so long. `table size: 14680064` stayed the same since server startup (see below).

Also note that this is basically a performance and stability regression from upgrading from Redis 7 where this entire setup was working fine with Redis server running well for over 2 years uptime and our hope was that by upgrading to Valkey we would be able to gain some performance (which we did!, except that it falls apart after around 4 hours.
```
Tue Apr 29 23:52:02 UTC 2025
Hash table 0 stats (main hash table):
 table size: 14680064
 number of entries: 10100000
 top-level buckets: 2097152
 child buckets: 1040664
 max chain length: 4
 avg chain length: 0.50
 chain length distribution:
   0: 1327209 (63.29%)
   1: 515624 (24.59%)
   2: 238122 (11.35%)
   3: 15992 (0.76%)
   4: 205 (0.01%)
Tue Apr 29 23:53:02 UTC 2025
Hash table 0 stats (main hash table):
 table size: 14680064
 number of entries: 10100000
 top-level buckets: 2097152
 child buckets: 1042322
 max chain length: 4
 avg chain length: 0.50
 chain length distribution:
   0: 1325010 (63.18%)
   1: 518216 (24.71%)
   2: 237873 (11.34%)
   3: 15852 (0.76%)
   4: 201 (0.01%)
Tue Apr 29 23:54:02 UTC 2025
Hash table 0 stats (main hash table):
 table size: 14680064
 number of entries: 10100016
 top-level buckets: 2097152
 child buckets: 1043926
 max chain length: 4
 avg chain length: 0.50
 chain length distribution:
   0: 1322776 (63.07%)
   1: 520908 (24.84%)
   2: 237582 (11.33%)
   3: 15690 (0.75%)
   4: 196 (0.01%)
Tue Apr 29 23:55:02 UTC 2025
Hash table 0 stats (main hash table):
 table size: 14680064
 number of entries: 10100000
 top-level buckets: 2097152
 child buckets: 1045553
 max chain length: 4
 avg chain length: 0.50
 chain length distribution:
   0: 1320500 (62.97%)
   1: 523712 (24.97%)
   2: 237172 (11.31%)
   3: 15575 (0.74%)
   4: 193 (0.01%)
Tue Apr 29 23:56:03 UTC 2025
Hash table 0 stats (main hash table):
 table size: 14680064
 number of entries: 10100000
 top-level buckets: 2097152
 child buckets: 1046958
 max chain length: 4
 avg chain length: 0.50
 chain length distribution:
   0: 1318396 (62.87%)
   1: 526390 (25.10%)
   2: 236722 (11.29%)
   3: 15452 (0.74%)
   4: 192 (0.01%)
Tue Apr 29 23:57:03 UTC 2025
Hash table 0 stats (main hash table):
 table size: 14680064
 number of entries: 10100036
 top-level buckets: 2097152
 child buckets: 1048400
 max chain length: 4
 avg chain length: 0.50
 chain length distribution:
   0: 1316298 (62.77%)
   1: 529003 (25.22%)
   2: 236345 (11.27%)
   3: 15317 (0.73%)
   4: 189 (0.01%)
Tue Apr 29 23:58:04 UTC 2025
Hash table 0 stats (main hash table):
 table size: 14680064
 number of entries: 10100008
 top-level buckets: 2097152
 child buckets: 1049511
 max chain length: 4
 avg chain length: 0.50
 chain length distribution:
   0: 1314700 (62.69%)
   1: 530967 (25.32%)
   2: 236097 (11.26%)
   3: 15202 (0.72%)
   4: 186 (0.01%)
Tue Apr 29 23:59:08 UTC 2025
Hash table 0 stats (main hash table):
 table size: 14680064
 number of entries: 10101809
 top-level buckets: 2097152
 child buckets: 1050200
 max chain length: 4
 avg chain length: 0.50
 chain length distribution:
   0: 1313786 (62.65%)
   1: 532062 (25.37%)
   2: 235957 (11.25%)
   3: 15164 (0.72%)
   4: 183 (0.01%)
Wed Apr 30 00:00:15 UTC 2025
Hash table 0 stats (main hash table):
 table size: 14680064
 number of entries: 10100000
 top-level buckets: 2097152
 child buckets: 1050063
 max chain length: 4
 avg chain length: 0.50
 chain length distribution:
   0: 1313701 (62.64%)
   1: 532330 (25.38%)
   2: 235810 (11.24%)
   3: 15131 (0.72%)
   4: 180 (0.01%)
Wed Apr 30 00:01:38 UTC 2025
Hash table 0 stats (main hash table):
 table size: 14680064
 number of entries: 10100721
 top-level buckets: 2097152
 child buckets: 1050054
 max chain length: 4
 avg chain length: 0.50
 chain length distribution:
   0: 1313623 (62.64%)
   1: 532479 (25.39%)
   2: 235754 (11.24%)
   3: 15117 (0.72%)
   4: 179 (0.01%)
Wed Apr 30 00:02:54 UTC 2025
Hash table 0 stats (main hash table):
 table size: 14680064
 number of entries: 10100614
 top-level buckets: 2097152
 child buckets: 1049980
 max chain length: 4
 avg chain length: 0.50
 chain length distribution:
   0: 1313584 (62.64%)
   1: 532608 (25.40%)
   2: 235687 (11.24%)
   3: 15094 (0.72%)
   4: 179 (0.01%)
```
Just adding one more note that after a while, valkey ends up in such a bad state that even a `srandmember urls:unretrievable 10` (on a server that has been out of service and therefore is completely idle) takes mere seconds to execute:

```
127.0.0.1:38542> srandmember urls:unretrievable 10
 1) "<redacted>"
 2) "<redacted>
 3) "<redacted>"
 4) "<redacted>"
 5) "<redacted>"
 6) "<redacted>"
 7) "<redacted>"
 8) "<redacted>"
 9) "<redacted>"
10) "<redacted>"
(3.50s)
127.0.0.1:38542> srandmember urls:unretrievable 100
1) "<redacted>"
[...]
100) "<redacted>"
(43.57s)
```

So far we have been unable to come up with a minimal reproduction case so we would be very grateful for any tips on how we can provide more debugging information. We've also taken a core dump that I can share with someone if it's helpful.

This is what a perf record flamegraph looks like for a `srandmember urls:unretrievable 200` taking 80 seconds to execute.

![Image](https://github.com/user-attachments/assets/aaf4b560-c388-4d0f-8484-02252a849bd0)

After restarting the server, the flamegraph looks significantly healthier even with the limit upped from 200 to 100000 taking only ~2-3 seconds to execute (needed to put the command in a loop to record a full 60 seconds perf)

![Image](https://github.com/user-attachments/assets/1a3ffdc7-abe5-48ee-8d8d-bb9d8b5ac81c)
@valkey-io/contributors This looks like a serious problem but it seems difficult to reproduce. I hope someone can help track it down.
~~@Fusl Can you try this patch? It looks like a bug in the reservoir sampling used for random element sampling.~~ (I was wrong)

```diff
--- a/src/hashtable.c
+++ b/src/hashtable.c
@@ -909,7 +909,7 @@ static void sampleEntriesScanFn(void *privdata, void *entry) {
         /* More entries than we wanted. This can happen if there are long
          * bucket chains. Replace random entries using reservoir sampling. */
         samples->seen++;
-        unsigned idx = random() % samples->seen;
+        unsigned idx = random() % samples->size;
         if (idx < samples->size) samples->entries[idx] = entry;
     }
 }
```

Actually, no, the `if (idx < samples->size)` on the next line guards that we don't replace an element in this case. This looks like it's the correct implementation of reservoir sampling, so I think I was wrong.

But I have another idea. Stay tuned.

I have found a scenario where the random element selection gets very slow. Maybe this is whats happens to you too. It could be.

When we select a random element in a fair way, we do like explain in https://github.com/valkey-io/valkey/issues/2019#issuecomment-2838811903. To mess with this algorithm as much as possible, I used scan to delete half of the elements. This results in a sparse hashtable. If the random algorithm picks an index in the empty section, it will iterate over the empty buckets until it finds the non-empty buckets. This is very expensive. With halft of the elements deleted in this way, picking a random element got 500 times slower. Deleting 75% of the elements made it 1000 times slower.
@Fusl #2085 fixes the problem described in my previous comment. The important part is the changed lines in hashtableSampleEntries, here: https://github.com/valkey-io/valkey/pull/2085/files#diff-549853f5324abb2829a79115cddc56f326cbb91ee19893fc0fd83cd1600bf75dR2075. Can you try it?
@zuiderkwast I tried with your previous suggestion, unfortunately no change either way and we observed the same slowdowns after around 12 hours. I'm now running with a build using the patch from your PR and will report back with test results.



Fix (patch):
diff --git a/src/hashtable.c b/src/hashtable.c
index 10413f5824..756b7de749 100644
--- a/src/hashtable.c
+++ b/src/hashtable.c
@@ -201,7 +201,7 @@ static_assert(MAX_FILL_PERCENT_SOFT <= MAX_FILL_PERCENT_HARD, "Soft vs hard fill
 
 /* --- Random entry --- */
 
-#define FAIR_RANDOM_SAMPLE_SIZE (ENTRIES_PER_BUCKET * 40)
+#define FAIR_RANDOM_SAMPLE_SIZE (ENTRIES_PER_BUCKET * 10)
 #define WEAK_RANDOM_SAMPLE_SIZE ENTRIES_PER_BUCKET
 
 /* --- Types --- */
@@ -2046,8 +2046,10 @@ int hashtableRandomEntry(hashtable *ht, void **found) {
 /* Points 'found' to a random entry in the hash table and returns 1. Returns 0
  * if the table is empty. This one is more fair than hashtableRandomEntry(). */
 int hashtableFairRandomEntry(hashtable *ht, void **found) {
-    void *samples[FAIR_RANDOM_SAMPLE_SIZE];
-    unsigned count = hashtableSampleEntries(ht, &samples[0], FAIR_RANDOM_SAMPLE_SIZE);
+    /* Sample less if it's very sparse. */
+    size_t num_samples = hashtableSize(ht) >= hashtableBuckets(ht) ? FAIR_RANDOM_SAMPLE_SIZE : WEAK_RANDOM_SAMPLE_SIZE;
+    void *samples[num_samples];
+    unsigned count = hashtableSampleEntries(ht, &samples[0], num_samples);
     if (count == 0) return 0;
     unsigned idx = random() % count;
     *found = samples[idx];
@@ -2069,9 +2071,8 @@ unsigned hashtableSampleEntries(hashtable *ht, void **dst, unsigned count) {
     samples.size = count;
     samples.seen = 0;
     samples.entries = dst;
-    size_t cursor = randomSizeT();
     while (samples.seen < count) {
-        cursor = hashtableScan(ht, cursor, sampleEntriesScanFn, &samples);
+        hashtableScan(ht, randomSizeT(), sampleEntriesScanFn, &samples);
     }
     rehashStepOnReadIfNeeded(ht);
     /* samples.seen is the number of entries scanned. It may be greater than