Repository: redis/redis

Problem statement:
[BUG] Hang in SRANDMEMBER call during CoW fork saving
Our application uses Lua EVAL/EVALSHA to do most heavy lifting of logic that requires atomic database changes.
During normal operation, Redis has a seemingly random chance to hang during the (only) `SRANDMEMBER` call of our Lua code when performing a background save (using either RDB or AOF) where it doesn't recover from even after the background save finishes.
We're unable to determine a solid reproduction case at this point since we can't pinpoint exactly what's causing it yet but across multiple GDB backtraces (below) we _believe_ that we were able to figure out that when Redis is hanging during the command call, it's always stuck in [this while loop](https://github.com/redis/redis/blob/7.0.11/src/t_set.c#L779-L793), although we're not 100% sure about that.

The version of Redis we're using is 7.0.11 running inside Docker with this configuration:
```
activedefrag yes
active-defrag-ignore-bytes 100mb
active-defrag-threshold-lower 10
active-defrag-threshold-upper 50
active-defrag-cycle-min 1
active-defrag-cycle-max 25
active-defrag-max-scan-fields 10000
lazyfree-lazy-eviction yes
lazyfree-lazy-expire yes
lazyfree-lazy-server-del yes
replica-lazy-flush yes
lazyfree-lazy-user-del yes
appendonly yes
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 1gb
client-output-buffer-limit slave 10737418240 1073741824 300
bind 0.0.0.0
port ${PORT}
requirepass ${PASS}
save ''
lua-time-limit 5000
busy-reply-threshold 5000
```

Note that while we have AOF configured here with RDB being disabled, we also tried running this Redis server with AOF disabled and RDB set to snapshot every 5 minutes after 1 change. We also tried explicitly setting `lua-time-limit` and `busy-reply-threshold` in the startup configuration as well as setting `CONFIG SET watchdog-period 1000` during runtime but none of those has produced any usable logs at the time Redis became unresponsive.

Logs indicate that the hangs always happen during a background save:

AOF rewrite:
```
1:M 18 May 2023 12:46:34.929 * Starting automatic rewriting of AOF on 100% growth                                                                                          
1:M 18 May 2023 12:46:34.930 * Creating AOF incr file appendonly.aof.4377.incr.aof on background rewrite
1:M 18 May 2023 12:46:35.164 * Background append only file rewriting started by pid 59
<Redis became unresponsive at 12:47:23>
59:C 18 May 2023 12:47:37.119 * Successfully created the temporary AOF base file temp-rewriteaof-bg-59.aof
59:C 18 May 2023 12:47:37.260 * Fork CoW for AOF rewrite: current 1572 MB, peak 1572 MB, average 818 MB
```

AOF rewrite:
```
1:M 19 May 2023 02:08:38.075 * Starting automatic rewriting of AOF on 100% growth
1:M 19 May 2023 02:08:38.076 * Creating AOF incr file appendonly.aof.4385.incr.aof on background rewrite
1:M 19 May 2023 02:08:38.345 * Background append only file rewriting started by pid 25
<Redis became unresponsive at 02:08:50>
25:C 19 May 2023 02:10:04.373 * Successfully created the temporary AOF base file temp-rewriteaof-bg-25.aof
25:C 19 May 2023 02:10:04.591 * Fork CoW for AOF rewrite: current 605 MB, peak 605 MB, average 329 MB
```

RDB save:
```
1:M 19 May 2023 04:25:55.055 * 1 changes in 300 seconds. Saving...
1:M 19 May 2023 04:25:55.374 * Background saving started by pid 28
<Redis became unresponsive at 04:25:58>
28:C 19 May 2023 04:27:20.861 * DB saved on disk
28:C 19 May 2023 04:27:21.083 * Fork CoW for RDB: current 142 MB, peak 142 MB, average 96 MB
```

```
2023-05-18T13:02:50+00:00
#0  _dictKeyIndex (existing=0x0, hash=4486456289374595091, key=0x7f56906063c1, d=0x7f549517e318) at dict.c:1054
#1  dictAddRaw (d=0x7f549517e318, key=0x7f56906063c1, existing=0x0) at dict.c:337
#2  0x0000558d682d2b01 in dictAdd (d=d@entry=0x7f549517e318, key=key@entry=0x7f56906063c1, val=val@entry=0x0) at dict.c:302
#3  0x0000558d6831885e in srandmemberWithCountCommand (c=0x7f56c7525000) at t_set.c:789
#4  0x0000558d682d94db in call (c=c@entry=0x7f56c7525000, flags=15) at server.c:3385
#5  0x0000558d683a68fd in scriptCall (run_ctx=run_ctx@entry=0x7ffc86dab0f0, err=err@entry=0x7ffc86daaca8) at script.c:570
#6  0x0000558d683a4756 in luaRedisGenericCommand (lua=0x558d69f537f0, raise_error=1) at script_lua.c:936
#7  0x0000558d683b9e88 in luaD_precall ()
#8  0x0000558d683c4110 in luaV_execute ()
#9  0x0000558d683ba525 in luaD_call ()
#10 0x0000558d683b9858 in luaD_rawrunprotected ()
#11 0x0000558d683ba6d0 in luaD_pcall ()
#12 0x0000558d683b79f8 in lua_pcall ()
#13 0x0000558d683a5ddb in luaCallFunction (run_ctx=0x7ffc86dab0f0, lua=0x558d69f537f0, keys=0x7f56c7434668, nkeys=0, args=0x7f56c7434668, nargs=10, debug_enabled=0) at script_lua.c:1678
#14 0x0000558d6834c503 in evalGenericCommand (c=0x7f56aff39a00, evalsha=1) at eval.c:553
#15 0x0000558d682d94db in call (c=c@entry=0x7f56aff39a00, flags=flags@entry=15) at server.c:3385
#16 0x0000558d682dbca5 in processCommand (c=c@entry=0x7f56aff39a00) at server.c:4019
#17 0x0000558d682f36ee in processCommandAndResetClient (c=0x7f56aff39a00) at networking.c:2473
#18 processInputBuffer (c=0x7f56aff39a00) at networking.c:2577
#19 0x0000558d682f65e0 in readQueryFromClient (conn=<optimized out>) at networking.c:2713
#20 0x0000558d6839ebc8 in callHandler (handler=<optimized out>, conn=0x7f552bb06380) at connhelpers.h:79
#21 connSocketEventHandler (el=<optimized out>, fd=<optimized out>, clientData=0x7f552bb06380, mask=<optimized out>) at connection.c:310
#22 0x0000558d682d099a in aeProcessEvents (eventLoop=eventLoop@entry=0x7f56c742b1e0, flags=flags@entry=27) at ae.c:436
#23 0x0000558d682d0d3d in aeMain (eventLoop=0x7f56c742b1e0) at ae.c:496
#24 0x0000558d682cc5f1 in main (argc=<optimized out>, argv=0x7ffc86dab498) at server.c:7168

2023-05-18T13:02:56+00:00
#0  0x00007f56c792531f in random () from target:/lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f56c7925799 in rand () from target:/lib/x86_64-linux-gnu/libc.so.6
#2  0x0000558d682d3764 in dictGetFairRandomKey (d=0x7f552a142cf8) at dict.c:797
#3  0x0000558d683179b9 in setTypeRandomElement (setobj=0x7f563c36d7c0, sdsele=0x7ffc86daab80, llele=0x7ffc86daab88) at t_set.c:213
#4  0x0000558d68318889 in srandmemberWithCountCommand (c=0x7f56c7525000) at t_set.c:780
#5  0x0000558d682d94db in call (c=c@entry=0x7f56c7525000, flags=15) at server.c:3385
#6  0x0000558d683a68fd in scriptCall (run_ctx=run_ctx@entry=0x7ffc86dab0f0, err=err@entry=0x7ffc86daaca8) at script.c:570
#7  0x0000558d683a4756 in luaRedisGenericCommand (lua=0x558d69f537f0, raise_error=1) at script_lua.c:936
#8  0x0000558d683b9e88 in luaD_precall ()
#9  0x0000558d683c4110 in luaV_execute ()
#10 0x0000558d683ba525 in luaD_call ()
#11 0x0000558d683b9858 in luaD_rawrunprotected ()
#12 0x0000558d683ba6d0 in luaD_pcall ()
#13 0x0000558d683b79f8 in lua_pcall ()
#14 0x0000558d683a5ddb in luaCallFunction (run_ctx=0x7ffc86dab0f0, lua=0x558d69f537f0, keys=0x7f56c7434668, nkeys=0, args=0x7f56c7434668, nargs=10, debug_enabled=0) at script_lua.c:1678
#15 0x0000558d6834c503 in evalGenericCommand (c=0x7f56aff39a00, evalsha=1) at eval.c:553
#16 0x0000558d682d94db in call (c=c@entry=0x7f56aff39a00, flags=flags@entry=15) at server.c:3385
#17 0x0000558d682dbca5 in processCommand (c=c@entry=0x7f56aff39a00) at server.c:4019
#18 0x0000558d682f36ee in processCommandAndResetClient (c=0x7f56aff39a00) at networking.c:2473
#19 processInputBuffer (c=0x7f56aff39a00) at networking.c:2577
#20 0x0000558d682f65e0 in readQueryFromClient (conn=<optimized out>) at networking.c:2713
#21 0x0000558d6839ebc8 in callHandler (handler=<optimized out>, conn=0x7f552bb06380) at connhelpers.h:79
#22 connSocketEventHandler (el=<optimized out>, fd=<optimized out>, clientData=0x7f552bb06380, mask=<optimized out>) at connection.c:310
#23 0x0000558d682d099a in aeProcessEvents (eventLoop=eventLoop@entry=0x7f56c742b1e0, flags=flags@entry=27) at ae.c:436
#24 0x0000558d682d0d3d in aeMain (eventLoop=0x7f56c742b1e0) at ae.c:496
#25 0x0000558d682cc5f1 in main (argc=<optimized out>, argv=0x7ffc86dab498) at server.c:7168

2023-05-18T13:03:00+00:00
#0  0x0000558d682d4abe in dictSdsKeyCompare (d=<optimized out>, key1=<optimized out>, key2=<optimized out>) at server.c:244
#1  0x0000558d682d2904 in _dictKeyIndex (existing=0x0, hash=15826992378078849043, key=0x7f56906063c1, d=0x7f549517e318) at dict.c:1055
#2  dictAddRaw (d=0x7f549517e318, key=0x7f56906063c1, existing=0x0) at dict.c:337
#3  0x0000558d682d2b01 in dictAdd (d=d@entry=0x7f549517e318, key=key@entry=0x7f56906063c1, val=val@entry=0x0) at dict.c:302
#4  0x0000558d6831885e in srandmemberWithCountCommand (c=0x7f56c7525000) at t_set.c:789
#5  0x0000558d682d94db in call (c=c@entry=0x7f56c7525000, flags=15) at server.c:3385
#6  0x0000558d683a68fd in scriptCall (run_ctx=run_ctx@entry=0x7ffc86dab0f0, err=err@entry=0x7ffc86daaca8) at script.c:570
#7  0x0000558d683a4756 in luaRedisGenericCommand (lua=0x558d69f537f0, raise_error=1) at script_lua.c:936
#8  0x0000558d683b9e88 in luaD_precall ()
#9  0x0000558d683c4110 in luaV_execute ()
#10 0x0000558d683ba525 in luaD_call ()
#11 0x0000558d683b9858 in luaD_rawrunprotected ()
#12 0x0000558d683ba6d0 in luaD_pcall ()
#13 0x0000558d683b79f8 in lua_pcall ()
#14 0x0000558d683a5ddb in luaCallFunction (run_ctx=0x7ffc86dab0f0, lua=0x558d69f537f0, keys=0x7f56c7434668, nkeys=0, args=0x7f56c7434668, nargs=10, debug_enabled=0) at script_lua.c:1678
#15 0x0000558d6834c503 in evalGenericCommand (c=0x7f56aff39a00, evalsha=1) at eval.c:553
#16 0x0000558d682d94db in call (c=c@entry=0x7f56aff39a00, flags=flags@entry=15) at server.c:3385
#17 0x0000558d682dbca5 in processCommand (c=c@entry=0x7f56aff39a00) at server.c:4019
#18 0x0000558d682f36ee in processCommandAndResetClient (c=0x7f56aff39a00) at networking.c:2473
#19 processInputBuffer (c=0x7f56aff39a00) at networking.c:2577
#20 0x0000558d682f65e0 in readQueryFromClient (conn=<optimized out>) at networking.c:2713
#21 0x0000558d6839ebc8 in callHandler (handler=<optimized out>, conn=0x7f552bb06380) at connhelpers.h:79
#22 connSocketEventHandler (el=<optimized out>, fd=<optimized out>, clientData=0x7f552bb06380, mask=<optimized out>) at connection.c:310
#23 0x0000558d682d099a in aeProcessEvents (eventLoop=eventLoop@entry=0x7f56c742b1e0, flags=flags@entry=27) at ae.c:436
#24 0x0000558d682d0d3d in aeMain (eventLoop=0x7f56c742b1e0) at ae.c:496
#25 0x0000558d682cc5f1 in main (argc=<optimized out>, argv=0x7ffc86dab498) at server.c:7168

2023-05-18T13:03:05+00:00
#0  0x0000558d682e30b9 in sdsfree (s=<optimized out>) at sds.c:196
#1  0x0000558d68318938 in srandmemberWithCountCommand (c=0x7f56c7525000) at t_set.c:792
#2  0x0000558d682d94db in call (c=c@entry=0x7f56c7525000, flags=15) at server.c:3385
#3  0x0000558d683a68fd in scriptCall (run_ctx=run_ctx@entry=0x7ffc86dab0f0, err=err@entry=0x7ffc86daaca8) at script.c:570
#4  0x0000558d683a4756 in luaRedisGenericCommand (lua=0x558d69f537f0, raise_error=1) at script_lua.c:936
#5  0x0000558d683b9e88 in luaD_precall ()
#6  0x0000558d683c4110 in luaV_execute ()
#7  0x0000558d683ba525 in luaD_call ()
#8  0x0000558d683b9858 in luaD_rawrunprotected ()
#9  0x0000558d683ba6d0 in luaD_pcall ()
#10 0x0000558d683b79f8 in lua_pcall ()
#11 0x0000558d683a5ddb in luaCallFunction (run_ctx=0x7ffc86dab0f0, lua=0x558d69f537f0, keys=0x7f56c7434668, nkeys=0, args=0x7f56c7434668, nargs=10, debug_enabled=0) at script_lua.c:1678
#12 0x0000558d6834c503 in evalGenericCommand (c=0x7f56aff39a00, evalsha=1) at eval.c:553
#13 0x0000558d682d94db in call (c=c@entry=0x7f56aff39a00, flags=flags@entry=15) at server.c:3385
#14 0x0000558d682dbca5 in processCommand (c=c@entry=0x7f56aff39a00) at server.c:4019
#15 0x0000558d682f36ee in processCommandAndResetClient (c=0x7f56aff39a00) at networking.c:2473
#16 processInputBuffer (c=0x7f56aff39a00) at networking.c:2577
#17 0x0000558d682f65e0 in readQueryFromClient (conn=<optimized out>) at networking.c:2713
#18 0x0000558d6839ebc8 in callHandler (handler=<optimized out>, conn=0x7f552bb06380) at connhelpers.h:79
#19 connSocketEventHandler (el=<optimized out>, fd=<optimized out>, clientData=0x7f552bb06380, mask=<optimized out>) at connection.c:310
#20 0x0000558d682d099a in aeProcessEvents (eventLoop=eventLoop@entry=0x7f56c742b1e0, flags=flags@entry=27) at ae.c:436
#21 0x0000558d682d0d3d in aeMain (eventLoop=0x7f56c742b1e0) at ae.c:496
#22 0x0000558d682cc5f1 in main (argc=<optimized out>, argv=0x7ffc86dab498) at server.c:7168

2023-05-18T13:03:10+00:00
#0  dictGetSomeKeys (d=d@entry=0x7f552a142cf8, des=0x7ffc86daaac0, des@entry=0x7ffc86daaaa0, count=<optimized out>, count@entry=15) at dict.c:764
#1  0x0000558d682d3759 in dictGetFairRandomKey (d=0x7f552a142cf8) at dict.c:791
#2  0x0000558d683179b9 in setTypeRandomElement (setobj=0x7f563c36d7c0, sdsele=0x7ffc86daab80, llele=0x7ffc86daab88) at t_set.c:213
#3  0x0000558d68318889 in srandmemberWithCountCommand (c=0x7f56c7525000) at t_set.c:780
#4  0x0000558d682d94db in call (c=c@entry=0x7f56c7525000, flags=15) at server.c:3385
#5  0x0000558d683a68fd in scriptCall (run_ctx=run_ctx@entry=0x7ffc86dab0f0, err=err@entry=0x7ffc86daaca8) at script.c:570
#6  0x0000558d683a4756 in luaRedisGenericCommand (lua=0x558d69f537f0, raise_error=1) at script_lua.c:936
#7  0x0000558d683b9e88 in luaD_precall ()
#8  0x0000558d683c4110 in luaV_execute ()
#9  0x0000558d683ba525 in luaD_call ()
#10 0x0000558d683b9858 in luaD_rawrunprotected ()
#11 0x0000558d683ba6d0 in luaD_pcall ()
#12 0x0000558d683b79f8 in lua_pcall ()
#13 0x0000558d683a5ddb in luaCallFunction (run_ctx=0x7ffc86dab0f0, lua=0x558d69f537f0, keys=0x7f56c7434668, nkeys=0, args=0x7f56c7434668, nargs=10, debug_enabled=0) at script_lua.c:1678
#14 0x0000558d6834c503 in evalGenericCommand (c=0x7f56aff39a00, evalsha=1) at eval.c:553
#15 0x0000558d682d94db in call (c=c@entry=0x7f56aff39a00, flags=flags@entry=15) at server.c:3385
#16 0x0000558d682dbca5 in processCommand (c=c@entry=0x7f56aff39a00) at server.c:4019
#17 0x0000558d682f36ee in processCommandAndResetClient (c=0x7f56aff39a00) at networking.c:2473
#18 processInputBuffer (c=0x7f56aff39a00) at networking.c:2577
#19 0x0000558d682f65e0 in readQueryFromClient (conn=<optimized out>) at networking.c:2713
#20 0x0000558d6839ebc8 in callHandler (handler=<optimized out>, conn=0x7f552bb06380) at connhelpers.h:79
#21 connSocketEventHandler (el=<optimized out>, fd=<optimized out>, clientData=0x7f552bb06380, mask=<optimized out>) at connection.c:310
#22 0x0000558d682d099a in aeProcessEvents (eventLoop=eventLoop@entry=0x7f56c742b1e0, flags=flags@entry=27) at ae.c:436
#23 0x0000558d682d0d3d in aeMain (eventLoop=0x7f56c742b1e0) at ae.c:496
#24 0x0000558d682cc5f1 in main (argc=<optimized out>, argv=0x7ffc86dab498) at server.c:7168

2023-05-18T13:03:16+00:00
#0  sz_index2size_lookup (index=1) at include/jemalloc/internal/sz.h:199
#1  sz_index2size (index=1) at include/jemalloc/internal/sz.h:207
#2  arena_salloc (ptr=<optimized out>, tsdn=<optimized out>) at include/jemalloc/internal/arena_inlines_b.h:195
#3  isalloc (ptr=<optimized out>, tsdn=<optimized out>) at include/jemalloc/internal/jemalloc_internal_inlines_c.h:38
#4  je_malloc_usable_size (ptr=ptr@entry=0x7f56906063c0) at src/jemalloc.c:3740
#5  0x0000558d682e69fe in zfree (ptr=0x7f56906063c0) at zmalloc.c:372
#6  zfree (ptr=0x7f56906063c0) at zmalloc.c:364
#7  0x0000558d68318938 in srandmemberWithCountCommand (c=0x7f56c7525000) at t_set.c:792
#8  0x0000558d682d94db in call (c=c@entry=0x7f56c7525000, flags=15) at server.c:3385
#9  0x0000558d683a68fd in scriptCall (run_ctx=run_ctx@entry=0x7ffc86dab0f0, err=err@entry=0x7ffc86daaca8) at script.c:570
#10 0x0000558d683a4756 in luaRedisGenericCommand (lua=0x558d69f537f0, raise_error=1) at script_lua.c:936
#11 0x0000558d683b9e88 in luaD_precall ()
#12 0x0000558d683c4110 in luaV_execute ()
#13 0x0000558d683ba525 in luaD_call ()
#14 0x0000558d683b9858 in luaD_rawrunprotected ()
#15 0x0000558d683ba6d0 in luaD_pcall ()
#16 0x0000558d683b79f8 in lua_pcall ()
#17 0x0000558d683a5ddb in luaCallFunction (run_ctx=0x7ffc86dab0f0, lua=0x558d69f537f0, keys=0x7f56c7434668, nkeys=0, args=0x7f56c7434668, nargs=10, debug_enabled=0) at script_lua.c:1678
#18 0x0000558d6834c503 in evalGenericCommand (c=0x7f56aff39a00, evalsha=1) at eval.c:553
#19 0x0000558d682d94db in call (c=c@entry=0x7f56aff39a00, flags=flags@entry=15) at server.c:3385
#20 0x0000558d682dbca5 in processCommand (c=c@entry=0x7f56aff39a00) at server.c:4019
#21 0x0000558d682f36ee in processCommandAndResetClient (c=0x7f56aff39a00) at networking.c:2473
#22 processInputBuffer (c=0x7f56aff39a00) at networking.c:2577
#23 0x0000558d682f65e0 in readQueryFromClient (conn=<optimized out>) at networking.c:2713
#24 0x0000558d6839ebc8 in callHandler (handler=<optimized out>, conn=0x7f552bb06380) at connhelpers.h:79
#25 connSocketEventHandler (el=<optimized out>, fd=<optimized out>, clientData=0x7f552bb06380, mask=<optimized out>) at connection.c:310
#26 0x0000558d682d099a in aeProcessEvents (eventLoop=eventLoop@entry=0x7f56c742b1e0, flags=flags@entry=27) at ae.c:436
#27 0x0000558d682d0d3d in aeMain (eventLoop=0x7f56c742b1e0) at ae.c:496
#28 0x0000558d682cc5f1 in main (argc=<optimized out>, argv=0x7ffc86dab498) at server.c:7168

2023-05-18T13:03:20+00:00
#0  0x00007f56c7a42a31 in ?? () from target:/lib/x86_64-linux-gnu/libc.so.6
#1  0x0000558d682d4ab2 in dictSdsKeyCompare (d=<optimized out>, key1=<optimized out>, key2=<optimized out>) at server.c:243
#2  0x0000558d682d2904 in _dictKeyIndex (existing=0x0, hash=16039903783278520339, key=0x7f56906063c1, d=0x7f549517e318) at dict.c:1055
#3  dictAddRaw (d=0x7f549517e318, key=0x7f56906063c1, existing=0x0) at dict.c:337
#4  0x0000558d682d2b01 in dictAdd (d=d@entry=0x7f549517e318, key=key@entry=0x7f56906063c1, val=val@entry=0x0) at dict.c:302
#5  0x0000558d6831885e in srandmemberWithCountCommand (c=0x7f56c7525000) at t_set.c:789
#6  0x0000558d682d94db in call (c=c@entry=0x7f56c7525000, flags=15) at server.c:3385
#7  0x0000558d683a68fd in scriptCall (run_ctx=run_ctx@entry=0x7ffc86dab0f0, err=err@entry=0x7ffc86daaca8) at script.c:570
#8  0x0000558d683a4756 in luaRedisGenericCommand (lua=0x558d69f537f0, raise_error=1) at script_lua.c:936
#9  0x0000558d683b9e88 in luaD_precall ()
#10 0x0000558d683c4110 in luaV_execute ()
#11 0x0000558d683ba525 in luaD_call ()
#12 0x0000558d683b9858 in luaD_rawrunprotected ()
#13 0x0000558d683ba6d0 in luaD_pcall ()
#14 0x0000558d683b79f8 in lua_pcall ()
#15 0x0000558d683a5ddb in luaCallFunction (run_ctx=0x7ffc86dab0f0, lua=0x558d69f537f0, keys=0x7f56c7434668, nkeys=0, args=0x7f56c7434668, nargs=10, debug_enabled=0) at script_lua.c:1678
#16 0x0000558d6834c503 in evalGenericCommand (c=0x7f56aff39a00, evalsha=1) at eval.c:553
#17 0x0000558d682d94db in call (c=c@entry=0x7f56aff39a00, flags=flags@entry=15) at server.c:3385
#18 0x0000558d682dbca5 in processCommand (c=c@entry=0x7f56aff39a00) at server.c:4019
#19 0x0000558d682f36ee in processCommandAndResetClient (c=0x7f56aff39a00) at networking.c:2473
#20 processInputBuffer (c=0x7f56aff39a00) at networking.c:2577
#21 0x0000558d682f65e0 in readQueryFromClient (conn=<optimized out>) at networking.c:2713
#22 0x0000558d6839ebc8 in callHandler (handler=<optimized out>, conn=0x7f552bb06380) at connhelpers.h:79
#23 connSocketEventHandler (el=<optimized out>, fd=<optimized out>, clientData=0x7f552bb06380, mask=<optimized out>) at connection.c:310
#24 0x0000558d682d099a in aeProcessEvents (eventLoop=eventLoop@entry=0x7f56c742b1e0, flags=flags@entry=27) at ae.c:436
#25 0x0000558d682d0d3d in aeMain (eventLoop=0x7f56c742b1e0) at ae.c:496
#26 0x0000558d682cc5f1 in main (argc=<optimized out>, argv=0x7ffc86dab498) at server.c:7168

2023-05-18T13:03:24+00:00
#0  dictGetSomeKeys (d=d@entry=0x7f552a142cf8, des=des@entry=0x7ffc86daaaa0, count=<optimized out>, count@entry=15) at dict.c:738
#1  0x0000558d682d3759 in dictGetFairRandomKey (d=0x7f552a142cf8) at dict.c:791
#2  0x0000558d683179b9 in setTypeRandomElement (setobj=0x7f563c36d7c0, sdsele=0x7ffc86daab80, llele=0x7ffc86daab88) at t_set.c:213
#3  0x0000558d68318889 in srandmemberWithCountCommand (c=0x7f56c7525000) at t_set.c:780
#4  0x0000558d682d94db in call (c=c@entry=0x7f56c7525000, flags=15) at server.c:3385
#5  0x0000558d683a68fd in scriptCall (run_ctx=run_ctx@entry=0x7ffc86dab0f0, err=err@entry=0x7ffc86daaca8) at script.c:570
#6  0x0000558d683a4756 in luaRedisGenericCommand (lua=0x558d69f537f0, raise_error=1) at script_lua.c:936
#7  0x0000558d683b9e88 in luaD_precall ()
#8  0x0000558d683c4110 in luaV_execute ()
#9  0x0000558d683ba525 in luaD_call ()
#10 0x0000558d683b9858 in luaD_rawrunprotected ()
#11 0x0000558d683ba6d0 in luaD_pcall ()
#12 0x0000558d683b79f8 in lua_pcall ()
#13 0x0000558d683a5ddb in luaCallFunction (run_ctx=0x7ffc86dab0f0, lua=0x558d69f537f0, keys=0x7f56c7434668, nkeys=0, args=0x7f56c7434668, nargs=10, debug_enabled=0) at script_lua.c:1678
#14 0x0000558d6834c503 in evalGenericCommand (c=0x7f56aff39a00, evalsha=1) at eval.c:553
#15 0x0000558d682d94db in call (c=c@entry=0x7f56aff39a00, flags=flags@entry=15) at server.c:3385
#16 0x0000558d682dbca5 in processCommand (c=c@entry=0x7f56aff39a00) at server.c:4019
#17 0x0000558d682f36ee in processCommandAndResetClient (c=0x7f56aff39a00) at networking.c:2473
#18 processInputBuffer (c=0x7f56aff39a00) at networking.c:2577
#19 0x0000558d682f65e0 in readQueryFromClient (conn=<optimized out>) at networking.c:2713
#20 0x0000558d6839ebc8 in callHandler (handler=<optimized out>, conn=0x7f552bb06380) at connhelpers.h:79
#21 connSocketEventHandler (el=<optimized out>, fd=<optimized out>, clientData=0x7f552bb06380, mask=<optimized out>) at connection.c:310
#22 0x0000558d682d099a in aeProcessEvents (eventLoop=eventLoop@entry=0x7f56c742b1e0, flags=flags@entry=27) at ae.c:436
#23 0x0000558d682d0d3d in aeMain (eventLoop=0x7f56c742b1e0) at ae.c:496
#24 0x0000558d682cc5f1 in main (argc=<optimized out>, argv=0x7ffc86dab498) at server.c:7168

2023-05-18T13:03:29+00:00
#0  dictGetSomeKeys (d=d@entry=0x7f552a142cf8, des=0x7ffc86daab00, des@entry=0x7ffc86daaaa0, count=<optimized out>, count@entry=15) at dict.c:733
#1  0x0000558d682d3759 in dictGetFairRandomKey (d=0x7f552a142cf8) at dict.c:791
#2  0x0000558d683179b9 in setTypeRandomElement (setobj=0x7f563c36d7c0, sdsele=0x7ffc86daab80, llele=0x7ffc86daab88) at t_set.c:213
#3  0x0000558d68318889 in srandmemberWithCountCommand (c=0x7f56c7525000) at t_set.c:780
#4  0x0000558d682d94db in call (c=c@entry=0x7f56c7525000, flags=15) at server.c:3385
#5  0x0000558d683a68fd in scriptCall (run_ctx=run_ctx@entry=0x7ffc86dab0f0, err=err@entry=0x7ffc86daaca8) at script.c:570
#6  0x0000558d683a4756 in luaRedisGenericCommand (lua=0x558d69f537f0, raise_error=1) at script_lua.c:936
#7  0x0000558d683b9e88 in luaD_precall ()
#8  0x0000558d683c4110 in luaV_execute ()
#9  0x0000558d683ba525 in luaD_call ()
#10 0x0000558d683b9858 in luaD_rawrunprotected ()
#11 0x0000558d683ba6d0 in luaD_pcall ()
#12 0x0000558d683b79f8 in lua_pcall ()
#13 0x0000558d683a5ddb in luaCallFunction (run_ctx=0x7ffc86dab0f0, lua=0x558d69f537f0, keys=0x7f56c7434668, nkeys=0, args=0x7f56c7434668, nargs=10, debug_enabled=0) at script_lua.c:1678
#14 0x0000558d6834c503 in evalGenericCommand (c=0x7f56aff39a00, evalsha=1) at eval.c:553
#15 0x0000558d682d94db in call (c=c@entry=0x7f56aff39a00, flags=flags@entry=15) at server.c:3385
#16 0x0000558d682dbca5 in processCommand (c=c@entry=0x7f56aff39a00) at server.c:4019
#17 0x0000558d682f36ee in processCommandAndResetClient (c=0x7f56aff39a00) at networking.c:2473
#18 processInputBuffer (c=0x7f56aff39a00) at networking.c:2577
#19 0x0000558d682f65e0 in readQueryFromClient (conn=<optimized out>) at networking.c:2713
#20 0x0000558d6839ebc8 in callHandler (handler=<optimized out>, conn=0x7f552bb06380) at connhelpers.h:79
#21 connSocketEventHandler (el=<optimized out>, fd=<optimized out>, clientData=0x7f552bb06380, mask=<optimized out>) at connection.c:310
#22 0x0000558d682d099a in aeProcessEvents (eventLoop=eventLoop@entry=0x7f56c742b1e0, flags=flags@entry=27) at ae.c:436
#23 0x0000558d682d0d3d in aeMain (eventLoop=0x7f56c742b1e0) at ae.c:496
#24 0x0000558d682cc5f1 in main (argc=<optimized out>, argv=0x7ffc86dab498) at server.c:7168

2023-05-18T13:03:34+00:00
#0  0x0000558d682d27be in dictTypeExpandAllowed (d=0x7f549517e318) at dict.c:989
#1  _dictExpandIfNeeded (d=0x7f549517e318) at dict.c:1008
#2  _dictKeyIndex (existing=0x0, hash=4334487920869140499, key=0x7f56906063c1, d=0x7f549517e318) at dict.c:1048
#3  dictAddRaw (d=0x7f549517e318, key=0x7f56906063c1, existing=0x0) at dict.c:337
#4  0x0000558d682d2b01 in dictAdd (d=d@entry=0x7f549517e318, key=key@entry=0x7f56906063c1, val=val@entry=0x0) at dict.c:302
#5  0x0000558d6831885e in srandmemberWithCountCommand (c=0x7f56c7525000) at t_set.c:789
#6  0x0000558d682d94db in call (c=c@entry=0x7f56c7525000, flags=15) at server.c:3385
#7  0x0000558d683a68fd in scriptCall (run_ctx=run_ctx@entry=0x7ffc86dab0f0, err=err@entry=0x7ffc86daaca8) at script.c:570
#8  0x0000558d683a4756 in luaRedisGenericCommand (lua=0x558d69f537f0, raise_error=1) at script_lua.c:936
#9  0x0000558d683b9e88 in luaD_precall ()
#10 0x0000558d683c4110 in luaV_execute ()
#11 0x0000558d683ba525 in luaD_call ()
#12 0x0000558d683b9858 in luaD_rawrunprotected ()
#13 0x0000558d683ba6d0 in luaD_pcall ()
#14 0x0000558d683b79f8 in lua_pcall ()
#15 0x0000558d683a5ddb in luaCallFunction (run_ctx=0x7ffc86dab0f0, lua=0x558d69f537f0, keys=0x7f56c7434668, nkeys=0, args=0x7f56c7434668, nargs=10, debug_enabled=0) at script_lua.c:1678
#16 0x0000558d6834c503 in evalGenericCommand (c=0x7f56aff39a00, evalsha=1) at eval.c:553
#17 0x0000558d682d94db in call (c=c@entry=0x7f56aff39a00, flags=flags@entry=15) at server.c:3385
#18 0x0000558d682dbca5 in processCommand (c=c@entry=0x7f56aff39a00) at server.c:4019
#19 0x0000558d682f36ee in processCommandAndResetClient (c=0x7f56aff39a00) at networking.c:2473
#20 processInputBuffer (c=0x7f56aff39a00) at networking.c:2577
#21 0x0000558d682f65e0 in readQueryFromClient (conn=<optimized out>) at networking.c:2713
#22 0x0000558d6839ebc8 in callHandler (handler=<optimized out>, conn=0x7f552bb06380) at connhelpers.h:79
#23 connSocketEventHandler (el=<optimized out>, fd=<optimized out>, clientData=0x7f552bb06380, mask=<optimized out>) at connection.c:310
#24 0x0000558d682d099a in aeProcessEvents (eventLoop=eventLoop@entry=0x7f56c742b1e0, flags=flags@entry=27) at ae.c:436
#25 0x0000558d682d0d3d in aeMain (eventLoop=0x7f56c742b1e0) at ae.c:496
#26 0x0000558d682cc5f1 in main (argc=<optimized out>, argv=0x7ffc86dab498) at server.c:7168

2023-05-18T13:03:38+00:00
#0  0x0000558d682d4a7d in sdslen (s=0x7f56909d2311 "i:blZ7htj") at sds.h:89
#1  dictSdsKeyCompare (d=<optimized out>, key1=0x7f56906063c1, key2=0x7f56909d2311) at server.c:241
#2  0x0000558d682d2904 in _dictKeyIndex (existing=0x0, hash=2834940632465860627, key=0x7f56906063c1, d=0x7f549517e318) at dict.c:1055
#3  dictAddRaw (d=0x7f549517e318, key=0x7f56906063c1, existing=0x0) at dict.c:337
#4  0x0000558d682d2b01 in dictAdd (d=d@entry=0x7f549517e318, key=key@entry=0x7f56906063c1, val=val@entry=0x0) at dict.c:302
#5  0x0000558d6831885e in srandmemberWithCountCommand (c=0x7f56c7525000) at t_set.c:789
#6  0x0000558d682d94db in call (c=c@entry=0x7f56c7525000, flags=15) at server.c:3385
#7  0x0000558d683a68fd in scriptCall (run_ctx=run_ctx@entry=0x7ffc86dab0f0, err=err@entry=0x7ffc86daaca8) at script.c:570
#8  0x0000558d683a4756 in luaRedisGenericCommand (lua=0x558d69f537f0, raise_error=1) at script_lua.c:936
#9  0x0000558d683b9e88 in luaD_precall ()
#10 0x0000558d683c4110 in luaV_execute ()
#11 0x0000558d683ba525 in luaD_call ()
#12 0x0000558d683b9858 in luaD_rawrunprotected ()
#13 0x0000558d683ba6d0 in luaD_pcall ()
#14 0x0000558d683b79f8 in lua_pcall ()
#15 0x0000558d683a5ddb in luaCallFunction (run_ctx=0x7ffc86dab0f0, lua=0x558d69f537f0, keys=0x7f56c7434668, nkeys=0, args=0x7f56c7434668, nargs=10, debug_enabled=0) at script_lua.c:1678
#16 0x0000558d6834c503 in evalGenericCommand (c=0x7f56aff39a00, evalsha=1) at eval.c:553
#17 0x0000558d682d94db in call (c=c@entry=0x7f56aff39a00, flags=flags@entry=15) at server.c:3385
#18 0x0000558d682dbca5 in processCommand (c=c@entry=0x7f56aff39a00) at server.c:4019
#19 0x0000558d682f36ee in processCommandAndResetClient (c=0x7f56aff39a00) at networking.c:2473
#20 processInputBuffer (c=0x7f56aff39a00) at networking.c:2577
#21 0x0000558d682f65e0 in readQueryFromClient (conn=<optimized out>) at networking.c:2713
#22 0x0000558d6839ebc8 in callHandler (handler=<optimized out>, conn=0x7f552bb06380) at connhelpers.h:79
#23 connSocketEventHandler (el=<optimized out>, fd=<optimized out>, clientData=0x7f552bb06380, mask=<optimized out>) at connection.c:310
#24 0x0000558d682d099a in aeProcessEvents (eventLoop=eventLoop@entry=0x7f56c742b1e0, flags=flags@entry=27) at ae.c:436
#25 0x0000558d682d0d3d in aeMain (eventLoop=0x7f56c742b1e0) at ae.c:496
#26 0x0000558d682cc5f1 in main (argc=<optimized out>, argv=0x7ffc86dab498) at server.c:7168

2023-05-18T13:03:44+00:00
#0  dictGetSomeKeys (d=d@entry=0x7f552a142cf8, des=0x7ffc86daaac0, des@entry=0x7ffc86daaaa0, count=<optimized out>, count@entry=15) at dict.c:768
#1  0x0000558d682d3759 in dictGetFairRandomKey (d=0x7f552a142cf8) at dict.c:791
#2  0x0000558d683179b9 in setTypeRandomElement (setobj=0x7f563c36d7c0, sdsele=0x7ffc86daab80, llele=0x7ffc86daab88) at t_set.c:213
#3  0x0000558d68318889 in srandmemberWithCountCommand (c=0x7f56c7525000) at t_set.c:780
#4  0x0000558d682d94db in call (c=c@entry=0x7f56c7525000, flags=15) at server.c:3385
#5  0x0000558d683a68fd in scriptCall (run_ctx=run_ctx@entry=0x7ffc86dab0f0, err=err@entry=0x7ffc86daaca8) at script.c:570
#6  0x0000558d683a4756 in luaRedisGenericCommand (lua=0x558d69f537f0, raise_error=1) at script_lua.c:936
#7  0x0000558d683b9e88 in luaD_precall ()
#8  0x0000558d683c4110 in luaV_execute ()
#9  0x0000558d683ba525 in luaD_call ()
#10 0x0000558d683b9858 in luaD_rawrunprotected ()
#11 0x0000558d683ba6d0 in luaD_pcall ()
#12 0x0000558d683b79f8 in lua_pcall ()
#13 0x0000558d683a5ddb in luaCallFunction (run_ctx=0x7ffc86dab0f0, lua=0x558d69f537f0, keys=0x7f56c7434668, nkeys=0, args=0x7f56c7434668, nargs=10, debug_enabled=0) at script_lua.c:1678
#14 0x0000558d6834c503 in evalGenericCommand (c=0x7f56aff39a00, evalsha=1) at eval.c:553
#15 0x0000558d682d94db in call (c=c@entry=0x7f56aff39a00, flags=flags@entry=15) at server.c:3385
#16 0x0000558d682dbca5 in processCommand (c=c@entry=0x7f56aff39a00) at server.c:4019
#17 0x0000558d682f36ee in processCommandAndResetClient (c=0x7f56aff39a00) at networking.c:2473
#18 processInputBuffer (c=0x7f56aff39a00) at networking.c:2577
#19 0x0000558d682f65e0 in readQueryFromClient (conn=<optimized out>) at networking.c:2713
#20 0x0000558d6839ebc8 in callHandler (handler=<optimized out>, conn=0x7f552bb06380) at connhelpers.h:79
#21 connSocketEventHandler (el=<optimized out>, fd=<optimized out>, clientData=0x7f552bb06380, mask=<optimized out>) at connection.c:310
#22 0x0000558d682d099a in aeProcessEvents (eventLoop=eventLoop@entry=0x7f56c742b1e0, flags=flags@entry=27) at ae.c:436
#23 0x0000558d682d0d3d in aeMain (eventLoop=0x7f56c742b1e0) at ae.c:496
#24 0x0000558d682cc5f1 in main (argc=<optimized out>, argv=0x7ffc86dab498) at server.c:7168

2023-05-18T13:03:51+00:00
#0  0x00007f56c7a42a28 in ?? () from target:/lib/x86_64-linux-gnu/libc.so.6
#1  0x0000558d682d4ab2 in dictSdsKeyCompare (d=<optimized out>, key1=<optimized out>, key2=<optimized out>) at server.c:243
#2  0x0000558d682d2904 in _dictKeyIndex (existing=0x0, hash=1090945667569254371, key=0x7f56906063c1, d=0x7f549517e318) at dict.c:1055
#3  dictAddRaw (d=0x7f549517e318, key=0x7f56906063c1, existing=0x0) at dict.c:337
#4  0x0000558d682d2b01 in dictAdd (d=d@entry=0x7f549517e318, key=key@entry=0x7f56906063c1, val=val@entry=0x0) at dict.c:302
#5  0x0000558d6831885e in srandmemberWithCountCommand (c=0x7f56c7525000) at t_set.c:789
#6  0x0000558d682d94db in call (c=c@entry=0x7f56c7525000, flags=15) at server.c:3385
#7  0x0000558d683a68fd in scriptCall (run_ctx=run_ctx@entry=0x7ffc86dab0f0, err=err@entry=0x7ffc86daaca8) at script.c:570
#8  0x0000558d683a4756 in luaRedisGenericCommand (lua=0x558d69f537f0, raise_error=1) at script_lua.c:936
#9  0x0000558d683b9e88 in luaD_precall ()
#10 0x0000558d683c4110 in luaV_execute ()
#11 0x0000558d683ba525 in luaD_call ()
#12 0x0000558d683b9858 in luaD_rawrunprotected ()
#13 0x0000558d683ba6d0 in luaD_pcall ()
#14 0x0000558d683b79f8 in lua_pcall ()
#15 0x0000558d683a5ddb in luaCallFunction (run_ctx=0x7ffc86dab0f0, lua=0x558d69f537f0, keys=0x7f56c7434668, nkeys=0, args=0x7f56c7434668, nargs=10, debug_enabled=0) at script_lua.c:1678
#16 0x0000558d6834c503 in evalGenericCommand (c=0x7f56aff39a00, evalsha=1) at eval.c:553
#17 0x0000558d682d94db in call (c=c@entry=0x7f56aff39a00, flags=flags@entry=15) at server.c:3385
#18 0x0000558d682dbca5 in processCommand (c=c@entry=0x7f56aff39a00) at server.c:4019
#19 0x0000558d682f36ee in processCommandAndResetClient (c=0x7f56aff39a00) at networking.c:2473
#20 processInputBuffer (c=0x7f56aff39a00) at networking.c:2577
#21 0x0000558d682f65e0 in readQueryFromClient (conn=<optimized out>) at networking.c:2713
#22 0x0000558d6839ebc8 in callHandler (handler=<optimized out>, conn=0x7f552bb06380) at connhelpers.h:79
#23 connSocketEventHandler (el=<optimized out>, fd=<optimized out>, clientData=0x7f552bb06380, mask=<optimized out>) at connection.c:310
#24 0x0000558d682d099a in aeProcessEvents (eventLoop=eventLoop@entry=0x7f56c742b1e0, flags=flags@entry=27) at ae.c:436
#25 0x0000558d682d0d3d in aeMain (eventLoop=0x7f56c742b1e0) at ae.c:496
#26 0x0000558d682cc5f1 in main (argc=<optimized out>, argv=0x7ffc86dab498) at server.c:7168

2023-05-18T13:04:13+00:00
#0  zfree (ptr=0x7f56906063c0) at zmalloc.c:373
#1  zfree (ptr=0x7f56906063c0) at zmalloc.c:364
#2  0x0000558d68318938 in srandmemberWithCountCommand (c=0x7f56c7525000) at t_set.c:792
#3  0x0000558d682d94db in call (c=c@entry=0x7f56c7525000, flags=15) at server.c:3385
#4  0x0000558d683a68fd in scriptCall (run_ctx=run_ctx@entry=0x7ffc86dab0f0, err=err@entry=0x7ffc86daaca8) at script.c:570
#5  0x0000558d683a4756 in luaRedisGenericCommand (lua=0x558d69f537f0, raise_error=1) at script_lua.c:936
#6  0x0000558d683b9e88 in luaD_precall ()
#7  0x0000558d683c4110 in luaV_execute ()
#8  0x0000558d683ba525 in luaD_call ()
#9  0x0000558d683b9858 in luaD_rawrunprotected ()
#10 0x0000558d683ba6d0 in luaD_pcall ()
#11 0x0000558d683b79f8 in lua_pcall ()
#12 0x0000558d683a5ddb in luaCallFunction (run_ctx=0x7ffc86dab0f0, lua=0x558d69f537f0, keys=0x7f56c7434668, nkeys=0, args=0x7f56c7434668, nargs=10, debug_enabled=0) at script_lua.c:1678
#13 0x0000558d6834c503 in evalGenericCommand (c=0x7f56aff39a00, evalsha=1) at eval.c:553
#14 0x0000558d682d94db in call (c=c@entry=0x7f56aff39a00, flags=flags@entry=15) at server.c:3385
#15 0x0000558d682dbca5 in processCommand (c=c@entry=0x7f56aff39a00) at server.c:4019
#16 0x0000558d682f36ee in processCommandAndResetClient (c=0x7f56aff39a00) at networking.c:2473
#17 processInputBuffer (c=0x7f56aff39a00) at networking.c:2577
#18 0x0000558d682f65e0 in readQueryFromClient (conn=<optimized out>) at networking.c:2713
#19 0x0000558d6839ebc8 in callHandler (handler=<optimized out>, conn=0x7f552bb06380) at connhelpers.h:79
#20 connSocketEventHandler (el=<optimized out>, fd=<optimized out>, clientData=0x7f552bb06380, mask=<optimized out>) at connection.c:310
#21 0x0000558d682d099a in aeProcessEvents (eventLoop=eventLoop@entry=0x7f56c742b1e0, flags=flags@entry=27) at ae.c:436
#22 0x0000558d682d0d3d in aeMain (eventLoop=0x7f56c742b1e0) at ae.c:496
#23 0x0000558d682cc5f1 in main (argc=<optimized out>, argv=0x7ffc86dab498) at server.c:7168

2023-05-18T13:04:17+00:00
#0  0x0000558d682d35d9 in dictGetSomeKeys (d=d@entry=0x7f552a142cf8, des=des@entry=0x7ffc86daaaa0, count=<optimized out>, count@entry=15) at dict.c:755
#1  0x0000558d682d3759 in dictGetFairRandomKey (d=0x7f552a142cf8) at dict.c:791
#2  0x0000558d683179b9 in setTypeRandomElement (setobj=0x7f563c36d7c0, sdsele=0x7ffc86daab80, llele=0x7ffc86daab88) at t_set.c:213
#3  0x0000558d68318889 in srandmemberWithCountCommand (c=0x7f56c7525000) at t_set.c:780
#4  0x0000558d682d94db in call (c=c@entry=0x7f56c7525000, flags=15) at server.c:3385
#5  0x0000558d683a68fd in scriptCall (run_ctx=run_ctx@entry=0x7ffc86dab0f0, err=err@entry=0x7ffc86daaca8) at script.c:570
#6  0x0000558d683a4756 in luaRedisGenericCommand (lua=0x558d69f537f0, raise_error=1) at script_lua.c:936
#7  0x0000558d683b9e88 in luaD_precall ()
#8  0x0000558d683c4110 in luaV_execute ()
#9  0x0000558d683ba525 in luaD_call ()
#10 0x0000558d683b9858 in luaD_rawrunprotected ()
#11 0x0000558d683ba6d0 in luaD_pcall ()
#12 0x0000558d683b79f8 in lua_pcall ()
#13 0x0000558d683a5ddb in luaCallFunction (run_ctx=0x7ffc86dab0f0, lua=0x558d69f537f0, keys=0x7f56c7434668, nkeys=0, args=0x7f56c7434668, nargs=10, debug_enabled=0) at script_lua.c:1678
#14 0x0000558d6834c503 in evalGenericCommand (c=0x7f56aff39a00, evalsha=1) at eval.c:553
#15 0x0000558d682d94db in call (c=c@entry=0x7f56aff39a00, flags=flags@entry=15) at server.c:3385
#16 0x0000558d682dbca5 in processCommand (c=c@entry=0x7f56aff39a00) at server.c:4019
#17 0x0000558d682f36ee in processCommandAndResetClient (c=0x7f56aff39a00) at networking.c:2473
#18 processInputBuffer (c=0x7f56aff39a00) at networking.c:2577
#19 0x0000558d682f65e0 in readQueryFromClient (conn=<optimized out>) at networking.c:2713
#20 0x0000558d6839ebc8 in callHandler (handler=<optimized out>, conn=0x7f552bb06380) at connhelpers.h:79
#21 connSocketEventHandler (el=<optimized out>, fd=<optimized out>, clientData=0x7f552bb06380, mask=<optimized out>) at connection.c:310
#22 0x0000558d682d099a in aeProcessEvents (eventLoop=eventLoop@entry=0x7f56c742b1e0, flags=flags@entry=27) at ae.c:436
#23 0x0000558d682d0d3d in aeMain (eventLoop=0x7f56c742b1e0) at ae.c:496
#24 0x0000558d682cc5f1 in main (argc=<optimized out>, argv=0x7ffc86dab498) at server.c:7168

2023-05-18T13:04:23+00:00
#0  0x00007f56c7a42a24 in ?? () from target:/lib/x86_64-linux-gnu/libc.so.6
#1  0x0000558d682d4ab2 in dictSdsKeyCompare (d=<optimized out>, key1=<optimized out>, key2=<optimized out>) at server.c:243
#2  0x0000558d682d2904 in _dictKeyIndex (existing=0x0, hash=17057246290930548707, key=0x7f56906063c1, d=0x7f549517e318) at dict.c:1055
#3  dictAddRaw (d=0x7f549517e318, key=0x7f56906063c1, existing=0x0) at dict.c:337
#4  0x0000558d682d2b01 in dictAdd (d=d@entry=0x7f549517e318, key=key@entry=0x7f56906063c1, val=val@entry=0x0) at dict.c:302
#5  0x0000558d6831885e in srandmemberWithCountCommand (c=0x7f56c7525000) at t_set.c:789
#6  0x0000558d682d94db in call (c=c@entry=0x7f56c7525000, flags=15) at server.c:3385
#7  0x0000558d683a68fd in scriptCall (run_ctx=run_ctx@entry=0x7ffc86dab0f0, err=err@entry=0x7ffc86daaca8) at script.c:570
#8  0x0000558d683a4756 in luaRedisGenericCommand (lua=0x558d69f537f0, raise_error=1) at script_lua.c:936
#9  0x0000558d683b9e88 in luaD_precall ()
#10 0x0000558d683c4110 in luaV_execute ()
#11 0x0000558d683ba525 in luaD_call ()
#12 0x0000558d683b9858 in luaD_rawrunprotected ()
#13 0x0000558d683ba6d0 in luaD_pcall ()
#14 0x0000558d683b79f8 in lua_pcall ()
#15 0x0000558d683a5ddb in luaCallFunction (run_ctx=0x7ffc86dab0f0, lua=0x558d69f537f0, keys=0x7f56c7434668, nkeys=0, args=0x7f56c7434668, nargs=10, debug_enabled=0) at script_lua.c:1678
#16 0x0000558d6834c503 in evalGenericCommand (c=0x7f56aff39a00, evalsha=1) at eval.c:553
#17 0x0000558d682d94db in call (c=c@entry=0x7f56aff39a00, flags=flags@entry=15) at server.c:3385
#18 0x0000558d682dbca5 in processCommand (c=c@entry=0x7f56aff39a00) at server.c:4019
#19 0x0000558d682f36ee in processCommandAndResetClient (c=0x7f56aff39a00) at networking.c:2473
#20 processInputBuffer (c=0x7f56aff39a00) at networking.c:2577
#21 0x0000558d682f65e0 in readQueryFromClient (conn=<optimized out>) at networking.c:2713
#22 0x0000558d6839ebc8 in callHandler (handler=<optimized out>, conn=0x7f552bb06380) at connhelpers.h:79
#23 connSocketEventHandler (el=<optimized out>, fd=<optimized out>, clientData=0x7f552bb06380, mask=<optimized out>) at connection.c:310
#24 0x0000558d682d099a in aeProcessEvents (eventLoop=eventLoop@entry=0x7f56c742b1e0, flags=flags@entry=27) at ae.c:436
#25 0x0000558d682d0d3d in aeMain (eventLoop=0x7f56c742b1e0) at ae.c:496
#26 0x0000558d682cc5f1 in main (argc=<optimized out>, argv=0x7ffc86dab498) at server.c:7168

2023-05-18T13:04:27+00:00
#0  0x00007f56c7925330 in random () from target:/lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f56c7925799 in rand () from target:/lib/x86_64-linux-gnu/libc.so.6
#2  0x0000558d682d3764 in dictGetFairRandomKey (d=0x7f552a142cf8) at dict.c:797
#3  0x0000558d683179b9 in setTypeRandomElement (setobj=0x7f563c36d7c0, sdsele=0x7ffc86daab80, llele=0x7ffc86daab88) at t_set.c:213
#4  0x0000558d68318889 in srandmemberWithCountCommand (c=0x7f56c7525000) at t_set.c:780
#5  0x0000558d682d94db in call (c=c@entry=0x7f56c7525000, flags=15) at server.c:3385
#6  0x0000558d683a68fd in scriptCall (run_ctx=run_ctx@entry=0x7ffc86dab0f0, err=err@entry=0x7ffc86daaca8) at script.c:570
#7  0x0000558d683a4756 in luaRedisGenericCommand (lua=0x558d69f537f0, raise_error=1) at script_lua.c:936
#8  0x0000558d683b9e88 in luaD_precall ()
#9  0x0000558d683c4110 in luaV_execute ()
#10 0x0000558d683ba525 in luaD_call ()
#11 0x0000558d683b9858 in luaD_rawrunprotected ()
#12 0x0000558d683ba6d0 in luaD_pcall ()
#13 0x0000558d683b79f8 in lua_pcall ()
#14 0x0000558d683a5ddb in luaCallFunction (run_ctx=0x7ffc86dab0f0, lua=0x558d69f537f0, keys=0x7f56c7434668, nkeys=0, args=0x7f56c7434668, nargs=10, debug_enabled=0) at script_lua.c:1678
#15 0x0000558d6834c503 in evalGenericCommand (c=0x7f56aff39a00, evalsha=1) at eval.c:553
#16 0x0000558d682d94db in call (c=c@entry=0x7f56aff39a00, flags=flags@entry=15) at server.c:3385
#17 0x0000558d682dbca5 in processCommand (c=c@entry=0x7f56aff39a00) at server.c:4019
#18 0x0000558d682f36ee in processCommandAndResetClient (c=0x7f56aff39a00) at networking.c:2473
#19 processInputBuffer (c=0x7f56aff39a00) at networking.c:2577
#20 0x0000558d682f65e0 in readQueryFromClient (conn=<optimized out>) at networking.c:2713
#21 0x0000558d6839ebc8 in callHandler (handler=<optimized out>, conn=0x7f552bb06380) at connhelpers.h:79
#22 connSocketEventHandler (el=<optimized out>, fd=<optimized out>, clientData=0x7f552bb06380, mask=<optimized out>) at connection.c:310
#23 0x0000558d682d099a in aeProcessEvents (eventLoop=eventLoop@entry=0x7f56c742b1e0, flags=flags@entry=27) at ae.c:436
#24 0x0000558d682d0d3d in aeMain (eventLoop=0x7f56c742b1e0) at ae.c:496
#25 0x0000558d682cc5f1 in main (argc=<optimized out>, argv=0x7ffc86dab498) at server.c:7168

2023-05-18T13:04:32+00:00
#0  0x00007f56c7925712 in random_r () from target:/lib/x86_64-linux-gnu/libc.so.6
#1  0x00007f56c79252dd in random () from target:/lib/x86_64-linux-gnu/libc.so.6
#2  0x00007f56c7925799 in rand () from target:/lib/x86_64-linux-gnu/libc.so.6
#3  0x0000558d682d3764 in dictGetFairRandomKey (d=0x7f552a142cf8) at dict.c:797
#4  0x0000558d683179b9 in setTypeRandomElement (setobj=0x7f563c36d7c0, sdsele=0x7ffc86daab80, llele=0x7ffc86daab88) at t_set.c:213
#5  0x0000558d68318889 in srandmemberWithCountCommand (c=0x7f56c7525000) at t_set.c:780
#6  0x0000558d682d94db in call (c=c@entry=0x7f56c7525000, flags=15) at server.c:3385
#7  0x0000558d683a68fd in scriptCall (run_ctx=run_ctx@entry=0x7ffc86dab0f0, err=err@entry=0x7ffc86daaca8) at script.c:570
#8  0x0000558d683a4756 in luaRedisGenericCommand (lua=0x558d69f537f0, raise_error=1) at script_lua.c:936
#9  0x0000558d683b9e88 in luaD_precall ()
#10 0x0000558d683c4110 in luaV_execute ()
#11 0x0000558d683ba525 in luaD_call ()
#12 0x0000558d683b9858 in luaD_rawrunprotected ()
#13 0x0000558d683ba6d0 in luaD_pcall ()
#14 0x0000558d683b79f8 in lua_pcall ()
#15 0x0000558d683a5ddb in luaCallFunction (run_ctx=0x7ffc86dab0f0, lua=0x558d69f537f0, keys=0x7f56c7434668, nkeys=0, args=0x7f56c7434668, nargs=10, debug_enabled=0) at script_lua.c:1678
#16 0x0000558d6834c503 in evalGenericCommand (c=0x7f56aff39a00, evalsha=1) at eval.c:553
#17 0x0000558d682d94db in call (c=c@entry=0x7f56aff39a00, flags=flags@entry=15) at server.c:3385
#18 0x0000558d682dbca5 in processCommand (c=c@entry=0x7f56aff39a00) at server.c:4019
#19 0x0000558d682f36ee in processCommandAndResetClient (c=0x7f56aff39a00) at networking.c:2473
#20 processInputBuffer (c=0x7f56aff39a00) at networking.c:2577
#21 0x0000558d682f65e0 in readQueryFromClient (conn=<optimized out>) at networking.c:2713
#22 0x0000558d6839ebc8 in callHandler (handler=<optimized out>, conn=0x7f552bb06380) at connhelpers.h:79
#23 connSocketEventHandler (el=<optimized out>, fd=<optimized out>, clientData=0x7f552bb06380, mask=<optimized out>) at connection.c:310
#24 0x0000558d682d099a in aeProcessEvents (eventLoop=eventLoop@entry=0x7f56c742b1e0, flags=flags@entry=27) at ae.c:436
#25 0x0000558d682d0d3d in aeMain (eventLoop=0x7f56c742b1e0) at ae.c:496
#26 0x0000558d682cc5f1 in main (argc=<optimized out>, argv=0x7ffc86dab498) at server.c:7168

2023-05-18T13:21:03+00:00
#0  0x00007f56c7a42a28 in ?? () from target:/lib/x86_64-linux-gnu/libc.so.6
#1  0x0000558d682d4ab2 in dictSdsKeyCompare (d=<optimized out>, key1=<optimized out>, key2=<optimized out>) at server.c:243
#2  0x0000558d682d2904 in _dictKeyIndex (existing=0x0, hash=18240379238795239395, key=0x7f56906063c1, d=0x7f549517e318) at dict.c:1055
#3  dictAddRaw (d=0x7f549517e318, key=0x7f56906063c1, existing=0x0) at dict.c:337
#4  0x0000558d682d2b01 in dictAdd (d=d@entry=0x7f549517e318, key=key@entry=0x7f56906063c1, val=val@entry=0x0) at dict.c:302
#5  0x0000558d6831885e in srandmemberWithCountCommand (c=0x7f56c7525000) at t_set.c:789
#6  0x0000558d682d94db in call (c=c@entry=0x7f56c7525000, flags=15) at server.c:3385
#7  0x0000558d683a68fd in scriptCall (run_ctx=run_ctx@entry=0x7ffc86dab0f0, err=err@entry=0x7ffc86daaca8) at script.c:570
#8  0x0000558d683a4756 in luaRedisGenericCommand (lua=0x558d69f537f0, raise_error=1) at script_lua.c:936
#9  0x0000558d683b9e88 in luaD_precall ()
#10 0x0000558d683c4110 in luaV_execute ()
#11 0x0000558d683ba525 in luaD_call ()
#12 0x0000558d683b9858 in luaD_rawrunprotected ()
#13 0x0000558d683ba6d0 in luaD_pcall ()
#14 0x0000558d683b79f8 in lua_pcall ()
#15 0x0000558d683a5ddb in luaCallFunction (run_ctx=0x7ffc86dab0f0, lua=0x558d69f537f0, keys=0x7f56c7434668, nkeys=0, args=0x7f56c7434668, nargs=10, debug_enabled=0) at script_lua.c:1678
#16 0x0000558d6834c503 in evalGenericCommand (c=0x7f56aff39a00, evalsha=1) at eval.c:553
#17 0x0000558d682d94db in call (c=c@entry=0x7f56aff39a00, flags=flags@entry=15) at server.c:3385
#18 0x0000558d682dbca5 in processCommand (c=c@entry=0x7f56aff39a00) at server.c:4019
#19 0x0000558d682f36ee in processCommandAndResetClient (c=0x7f56aff39a00) at networking.c:2473
#20 processInputBuffer (c=0x7f56aff39a00) at networking.c:2577
#21 0x0000558d682f65e0 in readQueryFromClient (conn=<optimized out>) at networking.c:2713
#22 0x0000558d6839ebc8 in callHandler (handler=<optimized out>, conn=0x7f552bb06380) at connhelpers.h:79
#23 connSocketEventHandler (el=<optimized out>, fd=<optimized out>, clientData=0x7f552bb06380, mask=<optimized out>) at connection.c:310
#24 0x0000558d682d099a in aeProcessEvents (eventLoop=eventLoop@entry=0x7f56c742b1e0, flags=flags@entry=27) at ae.c:436
#25 0x0000558d682d0d3d in aeMain (eventLoop=0x7f56c742b1e0) at ae.c:496
#26 0x0000558d682cc5f1 in main (argc=<optimized out>, argv=0x7ffc86dab498) at server.c:7168

2023-05-18T13:32:52+00:00
#0  extend_to_usable (ptr=ptr@entry=0x7f56906063c0, size=size@entry=16) at zmalloc.c:107
#1  0x0000558d682e6320 in zmalloc_usable (size=size@entry=11, usable=usable@entry=0x7ffc86daab18) at zmalloc.c:162
#2  0x0000558d682e2dc3 in _sdsnewlen (init=0x7f56004947b1, initlen=9, trymalloc=<optimized out>) at sds.c:117
#3  0x0000558d6831884a in srandmemberWithCountCommand (c=0x7f56c7525000) at t_set.c:784
#4  0x0000558d682d94db in call (c=c@entry=0x7f56c7525000, flags=15) at server.c:3385
#5  0x0000558d683a68fd in scriptCall (run_ctx=run_ctx@entry=0x7ffc86dab0f0, err=err@entry=0x7ffc86daaca8) at script.c:570
#6  0x0000558d683a4756 in luaRedisGenericCommand (lua=0x558d69f537f0, raise_error=1) at script_lua.c:936
#7  0x0000558d683b9e88 in luaD_precall ()
#8  0x0000558d683c4110 in luaV_execute ()
#9  0x0000558d683ba525 in luaD_call ()
#10 0x0000558d683b9858 in luaD_rawrunprotected ()
#11 0x0000558d683ba6d0 in luaD_pcall ()
#12 0x0000558d683b79f8 in lua_pcall ()
#13 0x0000558d683a5ddb in luaCallFunction (run_ctx=0x7ffc86dab0f0, lua=0x558d69f537f0, keys=0x7f56c7434668, nkeys=0, args=0x7f56c7434668, nargs=10, debug_enabled=0) at script_lua.c:1678
#14 0x0000558d6834c503 in evalGenericCommand (c=0x7f56aff39a00, evalsha=1) at eval.c:553
#15 0x0000558d682d94db in call (c=c@entry=0x7f56aff39a00, flags=flags@entry=15) at server.c:3385
#16 0x0000558d682dbca5 in processCommand (c=c@entry=0x7f56aff39a00) at server.c:4019
#17 0x0000558d682f36ee in processCommandAndResetClient (c=0x7f56aff39a00) at networking.c:2473
#18 processInputBuffer (c=0x7f56aff39a00) at networking.c:2577
#19 0x0000558d682f65e0 in readQueryFromClient (conn=<optimized out>) at networking.c:2713
#20 0x0000558d6839ebc8 in callHandler (handler=<optimized out>, conn=0x7f552bb06380) at connhelpers.h:79
#21 connSocketEventHandler (el=<optimized out>, fd=<optimized out>, clientData=0x7f552bb06380, mask=<optimized out>) at connection.c:310
#22 0x0000558d682d099a in aeProcessEvents (eventLoop=eventLoop@entry=0x7f56c742b1e0, flags=flags@entry=27) at ae.c:436
#23 0x0000558d682d0d3d in aeMain (eventLoop=0x7f56c742b1e0) at ae.c:496
#24 0x0000558d682cc5f1 in main (argc=<optimized out>, argv=0x7ffc86dab498) at server.c:7168
```

[Here](https://github.com/redis/redis/files/11513911/logs.txt) is also the output of `ltrace -ffs256 -p<pid>` of the hanging Redis process.


Hints:
@Fusl Can you provide the parameters of the `srandmember` command and the length of this key? 
My guess is that the `dictAdd()` is abnormally slow because the rehash of the dictionary is suspended during the rdb saving or aof rewrite.
I can't exactly tell the parameters of the `srandmember` call since most of the decision logic happens during runtime in the Lua script but from a quick `monitor` I can see that we're limiting the call to a max count of 30 at this time, however, the count can be lower than that as well:
```
1684496276.975048 [0 lua] "srandmember" "imgur:todo" "30"
1684496276.980532 [0 lua] "srandmember" "imgur:todo" "10"
1684496276.981337 [0 lua] "srandmember" "imgur:todo" "27"
1684496276.982097 [0 lua] "srandmember" "imgur:todo" "3" 
1684496276.995408 [0 lua] "srandmember" "imgur:todo" "15"
1684496276.996122 [0 lua] "srandmember" "imgur:todo" "7" 
1684496276.996658 [0 lua] "srandmember" "imgur:todo" "11"
1684496276.997268 [0 lua] "srandmember" "imgur:todo" "2" 
1684496277.008314 [0 lua] "srandmember" "imgur:todo" "27"
1684496277.012712 [0 lua] "srandmember" "imgur:todo" "13"
1684496277.036401 [0 lua] "srandmember" "imgur:todo" "30"
1684496277.065458 [0 lua] "srandmember" "imgur:todo" "30"
1684496277.070976 [0 lua] "srandmember" "imgur:todo" "30"
1684496277.071830 [0 lua] "srandmember" "imgur:todo" "30"
1684496277.073522 [0 lua] "srandmember" "imgur:todo" "22"
1684496277.081004 [0 lua] "srandmember" "imgur:todo" "30"
1684496277.082138 [0 lua] "srandmember" "imgur:todo" "1" 
1684496277.093768 [0 lua] "srandmember" "imgur:todo" "29"
1684496277.103692 [0 lua] "srandmember" "imgur:todo" "25"
1684496277.118868 [0 lua] "srandmember" "imgur:todo" "30"
1684496277.125603 [0 lua] "srandmember" "imgur:todo" "18"
1684496277.134662 [0 lua] "srandmember" "imgur:todo" "28"
1684496277.145458 [0 lua] "srandmember" "imgur:todo" "27"
1684496277.149010 [0 lua] "srandmember" "imgur:todo" "12"
1684496277.156520 [0 lua] "srandmember" "imgur:todo" "13"
1684496277.177948 [0 lua] "srandmember" "imgur:todo" "30"
```
The current `scard` of this key is `5747349`. Do note though that the Lua script that executes the `srandmember` also immediately afterwards does a `srem` after making some further decisions on the set member string, so it's quite possible that this set was a little larger than reporting this bug initially, but still well below 10M entries.
@Fusl Thx, Can you also provide the output of `INFO ALL` and `SLOWLOG GET`? 
Maybe also include lua scripts if possible.
[`info all`](https://github.com/redis/redis/files/11516651/info_all.txt)
[`slowlog get`](https://github.com/redis/redis/files/11516649/slowlog_get.txt)
[`script.lua`](https://github.com/redis/redis/files/11516650/project_request.content.eval0.lua.txt)

Note that we're currently running Redis without `activedefrag yes` to see if that's maybe causing issues for us and it's been chugging along since 6 hours again. However, that also doesn't mean a lot since Redis has been happy for a few days before suddenly not being happy anymore and hanging multiple times within an hour after restarting the container.
@sundb your theory is interesting. It means that srandmember isn't hand, it just becomes slower during fork, and then when run repeatedly it looks like it's hand.
This rehashing prevention is something that was changed in one of the recent versions.
Anyway, if that's that, it should be easy to fix, by pre-expanding the dict.
@Fusl let us know if you'll be able to test a patch to prove it.
I can test a patch if given in form of a Docker image.
I see we did call dictExpand first, this is an empty dict, so it will skip the rehash and then finish the expand.
the count is 30, so this dictionary should also be small

https://github.com/redis/redis/blob/7.0.11/src/t_set.c#L774-L794
```
    /* CASE 4: We have a big set compared to the requested number of elements.
     * In this case we can simply get random elements from the set and add
     * to the temporary set, trying to eventually get enough unique elements
     * to reach the specified count. */
    else {
        unsigned long added = 0;
        sds sdsele;

        dictExpand(d, count);
        while (added < count) {
            encoding = setTypeRandomElement(set,&ele,&llele);
            if (encoding == OBJ_ENCODING_INTSET) {
                sdsele = sdsfromlonglong(llele);
            } else {
                sdsele = sdsdup(ele);
            }
            /* Try to add the object to the dictionary. If it already exists
             * free it, otherwise increment the number of objects we have
             * in the result dictionary. */
            if (dictAdd(d,sdsele,NULL) == DICT_OK)
                added++;
            else
                sdsfree(sdsele);
        }
    }
```
From looking through the `logs.txt` ltrace file attached in my initial bug report post it looks like it's getting stuck in this while loop because it keeps missing the `if (dictAdd(d,sdsele,NULL) == DICT_OK)` condition because `setTypeRandomElement` keeps returning the same entries repeatedly, thus ending up in an infinite loop condition:
```
~/Downloads > cat logs.txt | fgrep memcpy | cut -d'"' -f2 | head
i:D8ZgAFd
i:kub6c1U
i:8FOvrOb
i:JOy6XF6
i:kub6c1U
i:3EoJJ9A
i:2j8F8yR
i:8FOvrOb
i:Ca0vivs
i:cjfhC8r
```
```
~/Downloads > cat logs.txt | fgrep memcpy | cut -d'"' -f2 | wc -l
4021
```
```
~/Downloads > cat logs.txt | fgrep memcpy | cut -d'"' -f2 | sort -u | wc -l
29
```
```
~/Downloads > cat logs.txt | fgrep memcpy | cut -d'"' -f2 | sort | uniq -c | sort -nk1,1
     99 i:cjfhC8r
    114 i:0MbpsVS
    118 i:Y50L9kv
    122 i:JOy6XF6
    123 i:iLsWxYn
    123 i:vLH6ekx
    124 i:47TNxEw
    125 i:D8ZgAFd
    126 i:0KJoCZD
    126 i:2j8F8yR
    126 i:THCgaW5
    132 i:TujMofJ
    132 i:dUZXiJy
    133 i:bON8Faw
    136 i:Ca0vivs
    136 i:Spz2BLF
    141 i:7rwtfe7
    141 i:Dtb2VA7
    141 i:KzxS7Zx
    141 i:S99DlMj
    144 i:Bd0HtRi
    144 i:sloc6wM
    150 i:iOLgwqS
    151 i:3EoJJ9A
    151 i:kub6c1U
    152 i:YrguwbM
    154 i:g7rKNjj
    163 i:O1ezsVh
    253 i:8FOvrOb
```
The only reason i can think of that setTypeRandomElement will return the same elements many times if if the dict's hash table is seriously too small (maybe if there are constantly forks, and _dictExpandIfNeeded never happens).

it would be nice if we could print the size of the hash table, but sadly, it looks like the only way to do that is by DEBUG HTSTAT-KEY, but that one is also printing bucket statistics, which will take too long to gather on this key (could cause redis to hang).

maybe we can patch redis to have a light version of HTSTAT, one that just prints the sizes of the hash tables. but it would mean restarting the server and hoping to still be able to reproduce this.
i've run a test to see how long would `DEBUG HTSTAT-KEY` take.
on 5m entries it took me roughly 100ms, and on 15m it took some 400ms.
so maybe that's acceptable for you to try.
@Fusl Did you do a lot of delete operations(srem or spop) on this set?
Yes, this set is in constant movement with millions of items getting added and removed every few minutes. At this very moment, the set is empty because of the project behind it having run out of data to process, so I can't easily reproduce the original issue.
```c
#define HASHTABLE_MIN_FILL        10      /* Minimal hash table fill 10% */
int dictNeedsResize(dict *dict) {
    long long size, used;

    size = dictSlots(dict);
    used = dictSize(dict);
    return (size > DICT_HT_INITIAL_SIZE &&
            (used*100/size < HASHTABLE_MIN_FILL));
}

void test() {
    dict *dict_ret = dictCreate(&BenchmarkDictType);
    dict *dict = dictCreate(&BenchmarkDictType);
    long count = 5000000;
    char buf[4096];
    init_genrand64(1684911460);

    /* Create a dict with 5,000,000 elements */
    dictSetResizeEnabled(DICT_RESIZE_AVOID);
    for (int j = 0; j < count; j++) {
        int retval = dictAdd(dict,stringFromLongLong(j),(void*)j);
        assert(retval == DICT_OK);
    }
    assert((long)dictSize(dict) == count);

    /* Wait for rehashing. */
    while (dictIsRehashing(dict)) {
        dictRehashMilliseconds(dict,100);
    }

    /* Delete to 10,000 elements left. */
    int del = 4990000;
    while (del--) {
        dictEntry *de = dictGetRandomKey(dict);
        char *str = dictGetKey(de);
        assert((dictDelete(dict, str) == DICT_OK));
    }

    /* Restart rehash */
    dictSetResizeEnabled(DICT_RESIZE_ENABLE);
    assert(dictNeedsResize(dict));
    dictResize(dict);
    dictRehashNum(dict, 93900);

    /* Print ht stats */
    dictGetStats(buf,sizeof(buf),dict,1);
    printf("%s\n", buf);
    printf("dict->rehashidx: %d\n", dict->rehashidx);

    /* Disable rehash and get random keys from this dict */
    dictSetResizeEnabled(DICT_RESIZE_AVOID);
    for (int i = 0; i < 1000; i++) {
        dictEntry *de = dictGetFairRandomKey(dict);
        char *str = dictGetKey(de);
        dictAdd(dict_ret, zstrdup(str), NULL);
        // printf("random key: %s\n", str);
    }
    printf("unique dict size: %ld\n", dictSize(dict_ret));

    dictRelease(dict); 
    dictRelease(dict_ret);
}
```

Output:
```sh
Hash table 0 stats (main hash table):
 table size: 1048576
 number of elements: 1066
 different slots: 538
 max chain length: 7
 avg chain length (counted): 1.98
 avg chain length (computed): 1.98
 Chain length distribution:
   0: 1048038 (99.95%)
   1: 249 (0.02%)
   2: 151 (0.01%)
   3: 75 (0.01%)
   4: 37 (0.00%)
   5: 17 (0.00%)
   6: 6 (0.00%)
   7: 3 (0.00%)
Hash table 1 stats (rehashing target):
 table size: 16384
 number of elements: 8934
 different slots: 4172
 max chain length: 15
 avg chain length (counted): 2.14
 avg chain length (computed): 2.14
 Chain length distribution:
   0: 12212 (74.54%)
   1: 1865 (11.38%)
   2: 1108 (6.76%)
   3: 552 (3.37%)
   4: 335 (2.04%)
   5: 163 (0.99%)
   6: 78 (0.48%)
   7: 36 (0.22%)
   8: 14 (0.09%)
   9: 12 (0.07%)
   10: 5 (0.03%)
   11: 1 (0.01%)
   13: 2 (0.01%)
   15: 1 (0.01%)

dict->rehashidx: 943760
unique dict size: 71
```

## Steps:
1) Disable rehash (set dict_can_resize to DICT_RESIZE_AVOID)
2) Create a dict with 5,000,000 elements and rehash it.
3) Delete the elements of this dictionary to 100,000.
4) Enable rehash (set dict_can_resize to DICT_RESIZE_ENABLE)
5) Rehash the dict so that `rehashidx` goes as far as possible to the end of dict[0].
    In the above example, let the `rehashidx` reach 943760, the maximum is 1048576.
6) Get 1000 random keys from this dict.

From the above logs, we can find that we get 1000 random keys from the dict, but only 71 unique keys in the end.

https://github.com/redis/redis/blob/d71478a889935382f3bf8b9a39ab073eba0a856a/src/dict.c#L1079:L1080
The reason is that when the dict shrinks and the `rehashidx` is very close to the size of dict[0], the code will most likely enter the branch above.

https://github.com/redis/redis/blob/d71478a889935382f3bf8b9a39ab073eba0a856a/src/dict.c#L298:L302
This issue is probably introduced by #11692, when `dict_can_resize` is set to `DICT_RESIZE_AVOID`, the above code only considers the expansion case and still allows rehash, but ignores the shrinking case.

Try using the following patch to fix and the example will be fine.
```diff
-    if (dict_can_resize == DICT_RESIZE_FORBID || !dictIsRehashing(d)) return 0;
+    unsigned long s0 = DICTHT_SIZE(d->ht_size_exp[0]);
+    unsigned long s1 = DICTHT_SIZE(d->ht_size_exp[1]);
     if (dict_can_resize == DICT_RESIZE_AVOID && 
-        (DICTHT_SIZE(d->ht_size_exp[1]) / DICTHT_SIZE(d->ht_size_exp[0]) < dict_force
_resize_ratio))
+        ((s1 > s0 && s1 / s0 < dict_force_resize_ratio) ||
+         (s1 < s0 && s0 / s1 < dict_force_resize_ratio)))
```

BTW: I still can't simulate the above result using commands.
Some more background information: The configuration of our application has a high and low watermark where the application tries to keep a set filled with items to satisfy the boundaries of the high and low watermark. That high/low is currently configured as 30M/1M so if the amount of items in the set falls below 1M it will start filling that set with items until it reaches 30M items at which point it will stop filling it again. I wonder if that behavior of our application is something that's just accelerating some kind of race condition (maybe not race condition but I lack the in-depth understanding of what's going on to find a better word for it) to surface as the bug we're seeing.
An accidental reappearance:
I use `srandmember` command to let the `rehashidx` of the hash as close to 131072 as possible.
It means that the speculation in the comment https://github.com/redis/redis/issues/12200#issuecomment-1560754386 is reasonable.

the status of this set key:
current hash `rehashidx`: 130296.
```
127.0.0.1:6379> debug HTSTATS-KEY s full
Hash table 0 stats (main hash table).
 table size: 131072
 number of elements: 9
 different slots: 5
 max chain length: 3
 avg chain length (counted): 1.80
 avg chain length (computed): 1.80
 Chain length distribution.
   0: 131067 (100.00%)
   1: 2 (0.00%)
   2: 2 (0.00%)
   3: 1 (0.00%)
Hash table 1 stats (rehashing target).
 table size: 1024
 number of elements: 970
 different slots: 11
 max chain length: 117
 avg chain length (counted): 88.18
 avg chain length (computed): 88.18
 Chain length distribution.
   0: 1013 (98.93%)
   25: 1 (0.10%)
```

Command and output:
```
127.0.0.1:6379> srandmember s 110
  1) "m:81564"
  2) "m:55787"
  3) "m:63276"
  4) "m:86639"
  5) "m:74482"
  6) "m:41890"
  7) "m:54130"
  8) "m:90807"
  ...
  (2.98s)     <- too long
```


@Fusl Can you use this patch to verify if it will be reproduced again?
```diff
diff --git a/src/dict.c b/src/dict.c
index ebd431d4..c2e471b7 100644
--- a/src/dict.c
+++ b/src/dict.c
@@ -294,9 +294,11 @@ int dictTryExpand(dict *d, unsigned long size) {
  * work it does would be unbound and the function may block for a long time. */
 int dictRehash(dict *d, int n) {
     int empty_visits = n*10; /* Max number of empty buckets to visit. */
-    if (dict_can_resize == DICT_RESIZE_FORBID || !dictIsRehashing(d)) return 0;
+    unsigned long s0 = DICTHT_SIZE(d->ht_size_exp[0]);
+    unsigned long s1 = DICTHT_SIZE(d->ht_size_exp[1]);
     if (dict_can_resize == DICT_RESIZE_AVOID && 
-        (DICTHT_SIZE(d->ht_size_exp[1]) / DICTHT_SIZE(d->ht_size_exp[0]) < dict_force_resize_ratio))
+        ((s1 > s0 && s1 / s0 < dict_force_resize_ratio) ||
+         (s1 < s0 && s0 / s1 < dict_force_resize_ratio)))
     {
         return 0;
     }
```
I can use this patch but I can't get a reproduction case together that will cause this behavior to happen again.
@sundb first, good job reproducing it. i suppose you did it while there is a fork child?
secondly, how exactly did you produce that table?
i see it's down sizing, and the original (bigger) one is nearly empty (was probably already nearly empty before rehashing), but what i don't understand is how come the new (smaller) one has such a long chain.
did you use SSCAN to delete members?

regarding the fix, maybe it would be possible to improve dictGetSomeKeys, but i suppose long chains would still mean we never sample from the end of the chain.

regarding the fix you're proposing, 
1. we should still respect FORBID, we don't want the fork child to do any re-hashing.
2. downsizing happens at x/10, so checking `dict_force_resize_ratio` (5) seems pointless.
> @sundb first, good job reproducing it. i suppose you did it while there is a fork child? secondly, how exactly did you produce that table? i see it's down sizing, and the original (bigger) one is nearly empty (was probably already nearly empty before rehashing), but what i don't understand is how come the new (smaller) one has such a long chain. did you use SSCAN to delete members?

Yes, using `SSCAN` to delete members.
I did this because I wanted the members of dict[0] to be concentrated at the end as much as possible.

Config:
```
enable-debug-command yes
save ""
set-max-listpack-entries 0
rdb-key-save-delay 2147483647
```

Steps:
1) Use python to create a dict just in time to start shrinking and start rehashing.
```py
import redis
import os

def is_rehashing(r):
    resp = r.execute_command("debug", "HTSTATS-KEY", "s", "full")
    return resp.find("rehashing target") != -1

r = redis.Redis(host='localhost', port=6379, decode_responses=True)
r.flushdb()

pipeline = r.pipeline()
for num in range(0,100000):
    pipeline.sadd("s", "m:" + str(num))
pipeline.execute()

while is_rehashing(r):
    r.srandmember("s", 100)

r.bgsave()

print(is_rehashing(r))
pipeline = r.pipeline()
count = 0
cursor = 0
while count < 99000:
    resp = r.sscan("s", cursor)
    cursor = resp[0]
    for m in resp[1]:
        count += 1
        pipeline.srem("s", m)
pipeline.execute()

os.system("pgrep -f redis-rdb-bgsave | xargs kill -9")

while is_rehashing(r) == False:
    r.spop("s")
```

2) Manually execute `srandmember s <count>` to rehash this dict 
    to let its `rehashidx` close to dict[0]'s maxsizemask(131071).
3) `BGSAVE` to stop rehash.

If at this time, the elements of dict[0] are concentrated after `rehashidx`, `srandmember` will have a probability of 0.0078(1024/131071) to randomly find duplicate members all the time.
Theoretically, if there are GETFAIR_NUM_ENTRIES(15) members exactly after the rehashidx, `srandmember` will probably die loop.
> 2. downsizing happens at x/10, so checking `dict_force_resize_ratio` (5) seems pointless.

The resize occurs when (used/size) /10 of dict[0], but dict_force_resize_ratio(5) is between dict_size_0 and dict_size_1, they don't conflict with each other.
In the above example:
before rehash
```
table used: 979
table size: 131072

(131072 / 979) /10 = 13.38, start rehash.
```

start rehash
```
table 1 size: 131072
table 2 size: 1024

(131072 / 1024) = 138, I think this dict still can rehash.
```

Ok. SCAN+DEL is a recipe for trouble (not just for rehashing, but also SCAN, RAND, eviction, etc).
But is that really what @Fusl did?

So before #11692 both the fork child and the parent could keep incremental rehashing during fork, and besides blocking the fork child from doing it, I also prevented the parent from doing it unless we need an extreme upscale, but we should have also allowed down scale. Right?
The FORBID part should be kept, and this fix should be taken to several versions.
> Ok. SCAN+DEL is a recipe for trouble (not just for rehashing, but also SCAN, RAND, eviction, etc). But is that really what @Fusl did?

Because i saw a lot of `SSCAN` commands  in `INFO ALL` posted in https://github.com/redis/redis/issues/12200#issuecomment-1554476660
```
cmdstat_srem:calls=1914081,usec=56042921,usec_per_call=29.28,rejected_calls=0,failed_calls=0
cmdstat_sscan:calls=804510,usec=4211523144,usec_per_call=5234.89,rejected_calls=0,failed_calls=0
cmdstat_spop:calls=76303,usec=93211447,usec_per_call=1221.60,rejected_calls=0,failed_calls=0
```

> 
> So before #11692 both the fork child and the parent could keep incremental rehashing during fork, and besides blocking the fork child from doing it, I also prevented the parent from doing it unless we need an extreme upscale, but we should have also allowed down scale. Right? 

Yes, This is the only possibility I can think of for `dictGetSomeKeys()` to return duplicate data.
Another way to fix it is to optimize dictGetSomeKeys(), so that it randomly the slot from [rehashidx, dict_size_2 + dict_size_1 - rehashidx] instead of [0, maxsizemask] (`unsigned long i = randomULong() & maxsizemask;`), but I'm not sure why antirez didn't do this.
`SSCAN` is part of how we maintain items in a set, yes. We do an `SSCAN`, move that data elsewhere (to offload huge amounts of data to disk) and then `SREM` those entries. Now that I think about it, we only started seeing the behavior of Redis hanging when we enabled said feature and this one is the only one doing a combination of `SSCAN`+`SREM` while the main system is the one running `SRANDMEMBER` as the primary way to get items from it.
> Another way to fix it is to optimize dictGetSomeKeys(), so that it randomly the slot from [rehashidx, dict_size_2 + dict_size_1 - rehashidx] instead of [0, maxsizemask] (`unsigned long i = randomULong() & maxsizemask;`), but I'm not sure why antirez didn't do this.

i think that when we have such a long chain in a bucket, we have no chance to ever pick the items at the end, so this pathological issue isn't gonna be solved by any change to the bucket selection.




> `SSCAN` is part of how we maintain items in a set, yes. We do an `SSCAN`, move that data elsewhere (to offload huge amounts of data to disk) and then `SREM` those entries. Now that I think about it, we only started seeing the behavior of Redis hanging when we enabled said feature and this one is the only one doing a combination of `SSCAN`+`SREM` while the main system is the one running `SRANDMEMBER` as the primary way to get items from it.

@Fusl so did you use this pattern on older versions (before 7.0.8)? or just on the on later ones?
in any case, i advise to avoid this pattern, redis really doesn't like it.
> i advise to avoid this pattern, redis really doesn't like it.

I'm aware of that now but we have to do it this way due to how we store data in sets. Even if it is advised to avoid it, it shouldn't cause Redis to hang forever. A possible workaround for us would be to use negative numbers in srandmember to make it not try to find unique entries and allow duplicates to be returned (we'd be handing duplicates within our application then) but I don't know what the downsides are here internally inside Redis.

> did you use this pattern on older versions

Yes, we used the same pattern before on older versions as well without Redis hanging, although never at the scale that we've done so recently.

Patch:
diff --git a/src/dict.c b/src/dict.c
index 1d9934b4b3c..6760da1540d 100644
--- a/src/dict.c
+++ b/src/dict.c
@@ -294,9 +294,12 @@ int dictTryExpand(dict *d, unsigned long size) {
  * work it does would be unbound and the function may block for a long time. */
 int dictRehash(dict *d, int n) {
     int empty_visits = n*10; /* Max number of empty buckets to visit. */
+    unsigned long s0 = DICTHT_SIZE(d->ht_size_exp[0]);
+    unsigned long s1 = DICTHT_SIZE(d->ht_size_exp[1]);
     if (dict_can_resize == DICT_RESIZE_FORBID || !dictIsRehashing(d)) return 0;
     if (dict_can_resize == DICT_RESIZE_AVOID && 
-        (DICTHT_SIZE(d->ht_size_exp[1]) / DICTHT_SIZE(d->ht_size_exp[0]) < dict_force_resize_ratio))
+        ((s1 > s0 && s1 / s0 < dict_force_resize_ratio) ||
+         (s1 < s0 && s0 / s1 < dict_force_resize_ratio)))
     {
         return 0;
     }
@@ -1095,19 +1098,30 @@ unsigned int dictGetSomeKeys(dict *d, dictEntry **des, unsigned int count) {
             } else {
                 emptylen = 0;
                 while (he) {
-                    /* Collect all the elements of the buckets found non
-                     * empty while iterating. */
-                    *des = he;
-                    des++;
+                    /* Collect all the elements of the buckets found non empty while iterating.
+                     * To avoid the issue of being unable to sample the end of a long chain,
+                     * we utilize the Reservoir Sampling algorithm to optimize the sampling process.
+                     * This means that even when the maximum number of samples has been reached,
+                     * we continue sampling until we reach the end of the chain.
+                     * See https://en.wikipedia.org/wiki/Reservoir_sampling. */
+                    if (stored < count) {
+                        des[stored] = he;
+                    } else {
+                        unsigned long r = randomULong() % (stored + 1);
+                        if (r < count) des[r] = he;
+                    }
+
                     he = dictGetNext(he);
                     stored++;
-                    if (stored == count) return stored;
                 }
+                if (stored >= count) goto end;
             }
         }
         i = (i+1) & maxsizemask;
     }
-    return stored;
+
+end:
+    return stored > count ? count : stored;
 }
 
 
@@ -1510,7 +1524,9 @@ size_t _dictGetStatsHt(char *buf, size_t bufsize, dict *d, int htidx, int full)
 
     if (d->ht_used[htidx] == 0) {
         return snprintf(buf,bufsize,
-            "No stats available for empty dictionaries\n");
+            "Hash table %d stats (%s):\n"
+            "No stats available for empty dictionaries\n",
+            htidx, (htidx == 0) ? "main hash table" : "rehashing target");
     }
 
     if (!full) {


Test patch:
diff --git a/tests/unit/type/set.tcl b/tests/unit/type/set.tcl
index 768e5cfbf82..a22b044a3f5 100644
--- a/tests/unit/type/set.tcl
+++ b/tests/unit/type/set.tcl
@@ -978,6 +978,129 @@ foreach type {single multiple single_multiple} {
         }
     }
 
+    proc is_rehashing {myset} {
+        set htstats [r debug HTSTATS-KEY $myset]
+        return [string match {*rehashing target*} $htstats]
+    }
+
+    proc rem_hash_set_top_N {myset n} {
+        set cursor 0
+        set members {}
+        set enough 0
+        while 1 {
+            set res [r sscan $myset $cursor]
+            set cursor [lindex $res 0]
+            set k [lindex $res 1]
+            foreach m $k {
+                lappend members $m
+                if {[llength $members] >= $n} {
+                    set enough 1
+                    break
+                }
+            }
+            if {$enough || $cursor == 0} {
+                break
+            }
+        }
+        r srem $myset {*}$members
+    }
+
+    test "SRANDMEMBER with a dict containing long chain" {
+        set origin_save [config_get_set save ""]
+        set origin_max_lp [config_get_set set-max-listpack-entries 0]
+        set origin_save_delay [config_get_set rdb-key-save-delay 2147483647]
+
+        # 1) Create a hash set with 100000 members.
+        set members {}
+        for {set i 0} {$i < 100000} {incr i} {
+            lappend members [format "m:%d" $i]
+        }
+        create_set myset $members
+
+        # 2) Wait for the hash set rehashing to finish.
+        while {[is_rehashing myset]} {
+            r srandmember myset 100
+        }
+
+        # 3) Turn off the rehashing of this set, and remove the members to 500.
+        r bgsave
+        rem_hash_set_top_N myset [expr {[r scard myset] - 500}]
+        assert_equal [r scard myset] 500
+
+        # 4) Kill RDB child process to restart rehashing.
+        set pid1 [get_child_pid 0]
+        catch {exec kill -9 $pid1}
+        waitForBgsave r
+
+        # 5) Let the set hash to start rehashing
+        r spop myset 1
+        assert [is_rehashing myset]
+
+        # 6) Verify that when rdb saving is in progress, rehashing will still be performed (because
+        # the ratio is extreme) by waiting for it to finish during an active bgsave.
+        r bgsave
+
+        while {[is_rehashing myset]} {
+            r srandmember myset 1
+        }
+        if {$::verbose} {
+            puts [r debug HTSTATS-KEY myset full]
+        }
+
+        set pid1 [get_child_pid 0]
+        catch {exec kill -9 $pid1}
+        waitForBgsave r
+
+        # 7) Check that eventually, SRANDMEMBER returns all elements.
+        array set allmyset {}
+        foreach ele [r smembers myset] {
+            set allmyset($ele) 1
+        }
+        unset -nocomplain auxset
+        set iterations 1000
+        while {$iterations != 0} {
+            incr iterations -1
+            set res [r srandmember myset -10]
+            foreach ele $res {
+                set auxset($ele) 1
+            }
+            if {[lsort [array names allmyset]] eq
+                [lsort [array names auxset]]} {
+                break;
+            }
+        }
+        assert {$iterations != 0}
+
+        # 8) Remove the members to 30 in order to calculate the value of Chi-Square Distribution,
+        #    otherwise we would need more iterations.
+        rem_hash_set_top_N myset [expr {[r scard myset] - 30}]
+        assert_equal [r scard myset] 30
+        assert {[is_rehashing myset]}
+
+        # Now that we have a hash set with only one long chain bucket.
+        set htstats [r debug HTSTATS-KEY myset full]
+        assert {[regexp {different slots: ([0-9]+)} $htstats - different_slots]}
+        assert {[regexp {max chain length: ([0-9]+)} $htstats - max_chain_length]}
+        assert {$different_slots == 1 && $max_chain_length == 30}
+
+        # 9) Use positive count (PATH 4) to get 10 elements (out of 30) each time.
+        unset -nocomplain allkey
+        set iterations 1000
+        while {$iterations != 0} {
+            incr iterations -1
+            set res [r srandmember myset 10]
+            foreach ele $res {
+                lappend allkey $ele
+            }
+        }
+        # validate even distribution of random sampling (df = 29, 73 means 0.00001 probability)
+        assert_lessthan [chi_square_value $allkey] 73
+
+        r config set save $origin_save
+        r config set set-max-listpack-entries $origin_max_lp
+        r config set rdb-key-save-delay $origin_save_delay
+    } {OK} {needs:debug slow}
+
     proc setup_move {} {
         r del myset3{t} myset4{t}
         create_set myset1{t} {1 a b}