Repository: valkey-io/valkey (2025-08-27T02:40:43Z)

Issue:
[Flaky test] High latency in active defrag tests
Active defragmention should only run on the order of about 1ms, but we observed cases in tests where the latency metric was reporting single digit ms. We implemented a temporary fix in https://github.com/valkey-io/valkey/pull/2421/files, but we should investigate how to return the latency to < 5ms.


Discussion:
big hash
```
[err]: Active Defrag big hash: standalone in tests/unit/memefficiency.tcl
Expected 12 <= 5 (context: type proc line 18 cmd {assert {$max_latency <= $limit_ms}} proc ::validate_latency level 1)
(Fast fail: test will exit now)
[err]: Active Defrag big hash: standalone in tests/unit/memefficiency.tcl
Expected 18 <= 5 (context: type proc line 18 cmd {assert {$max_latency <= $limit_ms}} proc ::validate_latency level 1)
(Fast fail: test will exit now)
```

big set
```
[err]: Active Defrag big set: standalone in tests/unit/memefficiency.tcl
Expected 16 <= 5 (context: type proc line 18 cmd {assert {$max_latency <= $limit_ms}} proc ::validate_latency level 1)
(Fast fail: test will exit now)

Test error (last server port:23623, log:./tests/tmp/server.152762.13/stdout), test will exit now
[err]: Active Defrag big set: standalone in tests/unit/memefficiency.tcl
Expected 12 <= 5 (context: type proc line 18 cmd {assert {$max_latency <= $limit_ms}} proc ::validate_latency level 1)
```

big zset
```
[err]: Active Defrag big zset: standalone in tests/unit/memefficiency.tcl
Expected 24 <= 5 (context: type proc line 18 cmd {assert {$max_latency <= $limit_ms}} proc ::validate_latency level 1)
(Fast fail: test will exit now)

[err]: Active Defrag big zset: standalone in tests/unit/memefficiency.tcl
Expected 16 <= 5 (context: type proc line 18 cmd {assert {$max_latency <= $limit_ms}} proc ::validate_latency level 1)
(Fast fail: test will exit now)
```
@madolson Please assign this to me.  We're currently looking into it.  I've identified the issue in big list (a real issue, not a test issue).  However I'm surprised that we're suddenly seeing more issues in the other tests.  Are we sure that these tests are still running in isolation?  There should be nothing else running on the test machine while these tests are in progress.
Yes, I made sure the tests were performed in isolation. (I tested on different Ubuntu OS/machines) I am looking into whether draining all `$rd reads` at once in tests might be causing the delay
>... I am looking into whether draining all $rd reads at once in tests might be causing the delay

I don't think that's possible.  Both the reads and the defrag use the main thread and don't happen at the same time.  The latency check is isolated to the function performing defrag.  Latency would be high if the called functions don't exit/return when they are supposed to or if the OS is bogged down, somehow delaying the return of the function.

For the LIST, I see that the function could run over time.  I haven't looked into the other classes yet, just mostly surprised that these issues are all occurring at the same time.
> I don't think that's possible. 

I think the issue that sungmin2 is highlighting is that it's possible the timeouts (#2205, a different test failure than this one) are the result of clients getting disconnected. In the memory efficiency tests we discard the responses, so it's possible one of those might be some type of error that we are just ignoring. It was mentioned in https://github.com/valkey-io/valkey/issues/2398#issuecomment-3164374173 that there might be some buffer limits we are hitting.
ok makes sense then.  Yes, I was a bit concerned when I saw this in the old code, but I probably made it worse.
Note:  On the high latency for lists, I suspect the issue is related to a legacy optimization.  To avoid calling `ustime()` too often, a loop-count mechanism is used.  We only check the time after 128 iterations.  This could easily result in higher latency measurements.

Now that this code is using the monotonic clock, such optimizations are unnecessary.  Putting things in perspective, with rough timings generated off my desktop host, a call to `getMonotonicUs()` takes roughly 19ns (including the loop).  A call to `allocatorShouldDefrag()`, which only checks if a block SHOULD be moved, takes roughly 49ns.  Actually moving a block takes a minimum of 43ns (an 8-byte block).  Given that moving a block might take much longer, we should check the clock any time a block is moved.
You missed the comment which is now out of date: https://github.com/valkey-io/valkey/pull/2553/files#r2317074251. Can you also remove that?
> You missed the comment which is now out of date: https://github.com/valkey-io/valkey/pull/2553/files#r2317074251. Can you also remove that?

Removed in #2574 . Thank you!
Now we're good! Thanks!



Patch:
diff --git a/src/defrag.c b/src/defrag.c
index 5228143bbb..79a8b473d2 100644
--- a/src/defrag.c
+++ b/src/defrag.c
@@ -822,7 +822,7 @@ static doneStatus defragLaterStep(monotime endtime, void *privdata) {
     defragKeysCtx *ctx = privdata;
 
     unsigned int iterations = 0;
-    unsigned long long prev_defragged = server.stat_active_defrag_hits;
+    long long prev_defragged = server.stat_active_defrag_hits;
     unsigned long long prev_scanned = server.stat_active_defrag_scanned;
 
     while (defrag_later && listLength(defrag_later) > 0) {
@@ -847,7 +847,7 @@ static doneStatus defragLaterStep(monotime endtime, void *privdata) {
             listDelNode(defrag_later, head);
         }
 
-        if (++iterations > 16 || server.stat_active_defrag_hits - prev_defragged > 512 ||
+        if (++iterations > 16 || server.stat_active_defrag_hits > prev_defragged ||
             server.stat_active_defrag_scanned - prev_scanned > 64) {
             if (getMonotonicUs() > endtime) break;
             iterations = 0;
@@ -882,7 +882,7 @@ static doneStatus defragStageKvstoreHelper(monotime endtime,
     }
 
     unsigned int iterations = 0;
-    unsigned long long prev_defragged = server.stat_active_defrag_hits;
+    long long prev_defragged = server.stat_active_defrag_hits;
     unsigned long long prev_scanned = server.stat_active_defrag_scanned;
 
     if (state.slot == KVS_SLOT_DEFRAG_LUT) {
@@ -895,7 +895,7 @@ static doneStatus defragStageKvstoreHelper(monotime endtime,
     }
 
     while (true) {
-        if (++iterations > 16 || server.stat_active_defrag_hits - prev_defragged > 512 || server.stat_active_defrag_scanned - prev_scanned > 64) {
+        if (++iterations > 16 || server.stat_active_defrag_hits > prev_defragged || server.stat_active_defrag_scanned - prev_scanned > 64) {
             if (getMonotonicUs() >= endtime) break;
             iterations = 0;
             prev_defragged = server.stat_active_defrag_hits;


Test patch:
diff --git a/tests/unit/memefficiency.tcl b/tests/unit/memefficiency.tcl
index 00508a4656..f7e353a133 100644
--- a/tests/unit/memefficiency.tcl
+++ b/tests/unit/memefficiency.tcl
@@ -183,7 +183,7 @@ run_solo {defrag} {
     proc perform_defrag_test {name args} {
         # This value should be 5 ms, but it was flaky in github runners. The issue 
         # https://github.com/valkey-io/valkey/issues/2444 is tracking if we can raise the limit again.
-        set opts(latency) 40
+        set opts(latency) 5
         set opts(while_defragging) {}
         array set opts $args
         assert {[info exists opts(populate)]}