Repository: emqx/emqx
Pull request #15114: fix(ds): Optimize subscription hand-over in beamformer
Author: login: ieQu1
Created: 2025-04-24T20:10:00Z | Merged: 2025-04-28T20:22:19Z

Fixes [EMQX-14179](https://emqx.atlassian.net/browse/EMQX-14179)

Release version: 5.9

## Summary

<!--
Please compose a nontrivial summary in case of significant changes.
* Point out the crucial changes in logic
* Point out the most relevant files and modules for the change
* Provide some reasoning for the decisions taken
-->

## PR Checklist
<!--
Please convert the PR to a draft if any of the following conditions are not met.
-->
- [x] For internal contributor: there is a jira ticket to track this change
- [x] The changes are covered with new or existing tests
- [x] Change log for changes visible by users has been added to `changes/ee/(feat|perf|fix|breaking)-<PR-id>.en.md` files
- [x] Schema changes are backward compatible or intentionally breaking (describe the changes and the reasoning in the summary)

<!--
Please, take in account the following guidelines while working on PR:
* Try to achieve reasonable coverage of the new code
* Add property-based tests for code that performs complex user input validation or implements a complex algorithm
* Create a PR to [emqx-docs](https://github.com/emqx/emqx-docs) if documentation update is required, or make a follow-up jira ticket
* Do not squash large PRs into a single commit, try to keep comprehensive history of incremental changes
* Do not squash any significant amount of review fixes into the previous commits
-->

<!--
## Checklist for CI (.github/workflows) changes
- [ ] If changed package build workflow, pass [this action](https://github.com/emqx/emqx/actions/workflows/build_packages.yaml) (manual trigger)
- [ ] Change log has been added to `changes/` dir for user-facing artifacts update
-->


[EMQX-14179]: https://emqx.atlassian.net/browse/EMQX-14179?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ


Diff:
diff --git a/apps/emqx_ds_backends/test/emqx_ds_backends_SUITE.erl b/apps/emqx_ds_backends/test/emqx_ds_backends_SUITE.erl
index 4fe2a442ff8..1227a4edf2b 100644
--- a/apps/emqx_ds_backends/test/emqx_ds_backends_SUITE.erl
+++ b/apps/emqx_ds_backends/test/emqx_ds_backends_SUITE.erl
@@ -611,8 +611,8 @@ t_sub_shard_down_notify(Config) ->
             Stream = make_stream(Config),
             {ok, It} = emqx_ds:make_iterator(DB, Stream, [<<"t">>], 0),
             {ok, _Handle, MRef} = emqx_ds:subscribe(DB, It, #{max_unacked => 100}),
-            ?assertMatch(ok, emqx_ds:close_db(DB)),
-            ?assertMatch([MRef], collect_down_msgs())
+            ?assertEqual(ok, emqx_ds:close_db(DB)),
+            ?assertEqual([MRef], collect_down_msgs())
         end,
         []
     ).
diff --git a/apps/emqx_ds_builtin_local/src/emqx_ds_builtin_local.erl b/apps/emqx_ds_builtin_local/src/emqx_ds_builtin_local.erl
index 767fba9d556..ac8c2a8d115 100644
--- a/apps/emqx_ds_builtin_local/src/emqx_ds_builtin_local.erl
+++ b/apps/emqx_ds_builtin_local/src/emqx_ds_builtin_local.erl
@@ -394,7 +394,7 @@ update_iterator(ShardId, Iter0 = #{?tag := ?IT, ?enc := StorageIter0}, Key) ->
 
 -spec next(emqx_ds:db(), iterator(), pos_integer()) -> emqx_ds:next_result(iterator()).
 next(DB, Iter, N) ->
-    {ok, Ref} = emqx_ds_lib:with_worker(?MODULE, do_next, [DB, Iter, N]),
+    {ok, _, Ref} = emqx_ds_lib:with_worker(?MODULE, do_next, [DB, Iter, N]),
     receive
         {Ref, Result} ->
             Result
@@ -500,7 +500,7 @@ make_delete_iterator(DB, ?delete_stream(Shard, InnerStream), TopicFilter, StartT
 -spec delete_next(emqx_ds:db(), delete_iterator(), emqx_ds:delete_selector(), pos_integer()) ->
     emqx_ds:delete_next_result(emqx_ds:delete_iterator()).
 delete_next(DB, Iter, Selector, N) ->
-    {ok, Ref} = emqx_ds_lib:with_worker(?MODULE, do_delete_next, [DB, Iter, Selector, N]),
+    {ok, _, Ref} = emqx_ds_lib:with_worker(?MODULE, do_delete_next, [DB, Iter, Selector, N]),
     receive
         {Ref, Result} -> Result
     end.
diff --git a/apps/emqx_durable_storage/include/emqx_ds_metrics.hrl b/apps/emqx_durable_storage/include/emqx_ds_metrics.hrl
index f9666c3274e..a1f63212924 100644
--- a/apps/emqx_durable_storage/include/emqx_ds_metrics.hrl
+++ b/apps/emqx_durable_storage/include/emqx_ds_metrics.hrl
@@ -48,16 +48,19 @@
 %%%% Metrics related to subscription:
 %% Number of active subscriptions:
 -define(DS_SUBS, emqx_ds_subs).
+-define(DS_SUBS_HANDOVER, emqx_ds_subs_handover).
 %% Total number of beams sent by all workers:
 -define(DS_SUBS_BEAMS_SENT_TOTAL, emqx_ds_subs_beams_sent_total).
 %% Measure of "beam coherence": average number of requests fulfilled
 %% by a single beam:
 -define(DS_SUBS_REQUEST_SHARING, emqx_ds_subs_request_sharing).
 %% Number of subscription stuck/unstuck events:
--define(DS_SUBS_STUCK_TOTAL, emqsx_ds_subs_stuck_total).
--define(DS_SUBS_UNSTUCK_TOTAL, emqsx_ds_subs_unstuck_total).
+-define(DS_SUBS_STUCK_TOTAL, emqx_ds_subs_stuck_total).
+-define(DS_SUBS_UNSTUCK_TOTAL, emqx_ds_subs_unstuck_total).
 %% Sliding average of time spent fulfilling requests per worker type (μs):
 -define(DS_SUBS_FULFILL_TIME, emqx_ds_subs_fulfill_time).
+%% Sliding average of time spent processing pending commands such as sub/unsub/fast-forward (μs):
+-define(DS_SUBS_PROCESS_COMMANDS_TIME, emqx_ds_subs_process_commands_time).
 %% Sliding average of time spent scanning the DB (μs):
 -define(DS_SUBS_SCAN_TIME, emqx_ds_subs_scan_time).
 %% Sliding average of time spent splitting the beam:
diff --git a/apps/emqx_durable_storage/src/emqx_ds_beamformer.erl b/apps/emqx_durable_storage/src/emqx_ds_beamformer.erl
index 188101f4a7d..88784543f23 100644
--- a/apps/emqx_durable_storage/src/emqx_ds_beamformer.erl
+++ b/apps/emqx_durable_storage/src/emqx_ds_beamformer.erl
@@ -374,7 +374,7 @@ poll(_Node, _ReturnAddr, _Shard, _Iterator, #{timeout := _Timeout}) ->
 make_subtab(DBShard) ->
     ets:new(emqx_ds_beamformer_sub_tab, [
         set,
-        private,
+        public,
         {keypos, #sub_state.req_id},
         {heir, where(DBShard), ?heir_info}
     ]).
@@ -740,9 +740,12 @@ generation_event(DBShard) ->
     stuck_sub_tab :: ets:tid(),
     cbm :: module(),
     monitor_tab = ets:tid(),
-    %% List of subscription IDs waiting for dispatching to the
-    %% worker:
+    %% List of subscription states waiting for dispatching to the
+    %% workers. Subscription requests can be retained in this queue
+    %% while the workers are recovering from failure:
     pending = [] :: [sub_state()],
+    %% Storage for the ongoing ownership transfers:
+    inflight = gen_server:reqids_new() :: gen_server:request_id_collection(),
     %% Generation cache:
     generations :: #{emqx_ds:slab() => emqx_ds:slab_info()}
 }).
@@ -756,7 +759,11 @@ init([DBShard = {DB, Shard}, CBM]) ->
     process_flag(trap_exit, true),
     logger:update_process_metadata(#{dbshard => DBShard}),
     emqx_ds_builtin_metrics:init_for_beamformer(DB, Shard),
-    gproc_pool:new(emqx_ds_beamformer_catchup:pool(DBShard), hash, [{auto_size, true}]),
+    %% For catchup pool we don't expect high degree of batch sharing.
+    %% It's actually better to distribute subscriptions evenly.
+    gproc_pool:new(emqx_ds_beamformer_catchup:pool(DBShard), round_robin, [{auto_size, true}]),
+    %% RT pool uses hash of the stream to maximize batch sharing. It's
+    %% critical to use hash-based strategy here:
     gproc_pool:new(emqx_ds_beamformer_rt:pool(DBShard), hash, [{auto_size, true}]),
     logger:debug(#{
         msg => started_bf, shard => DBShard, cbm => CBM
@@ -783,8 +790,44 @@ init([DBShard = {DB, Shard}, CBM]) ->
 
 -spec handle_event(gen_statem:event_type(), _Event, state(), d()) ->
     gen_statem:event_handler_result(state()).
+handle_event(ET, Event, State, D0) ->
+    #d{dbshard = DBShard, inflight = Inflight0, pending = Pending} = D0,
+    case gen_server:check_response(Event, Inflight0, true) of
+        {Response, SubState, Inflight} ->
+            %% The event is result of `enqueue':
+            case Response of
+                {reply, ok} ->
+                    %% We've successfully delegated the subscription
+                    %% to the worker.
+                    D = D0#d{inflight = Inflight},
+                    {keep_state, D};
+                {error, {Reason, ServerRef}} ->
+                    %% This can only happen when the worker is down.
+                    %% Give it time to recover and reschedule request
+                    %% later.
+                    %%
+                    %% TODO: currently this will pause dispatching of
+                    %% subscriptions to *all* workers in the shard.
+                    %% Ideally, error handling should be more
+                    %% targeted.
+                    ?tp(
+                        error,
+                        emqx_ds_beamformer_delegate_error,
+                        #{shard => DBShard, worker => ServerRef, error => Reason}
+                    ),
+                    D = D0#d{inflight = Inflight, pending = [SubState | Pending]},
+                    {next_state, ?recovering, D}
+            end;
+        _ ->
+            %% Handle everything else:
+            do_handle_event(ET, Event, State, D0)
+    end.
+
+%% Handle all events that are not async gen_server replies
+-spec do_handle_event(gen_statem:event_type(), _Event, state(), d()) ->
+    gen_statem:event_handler_result(state()).
 %% Handle subscribe call:
-handle_event(
+do_handle_event(
     {call, From},
     #sub_req{client = Client, it = It, opts = Opts, mref = SubRef},
     State,
@@ -850,7 +893,7 @@ handle_event(
             {keep_state_and_data, {reply, From, Error}}
     end;
 %% Handle unsubscribe call:
-handle_event(
+do_handle_event(
     {call, From},
     #unsub_req{id = SubId},
     _State,
@@ -859,7 +902,7 @@ handle_event(
     {Ret, D} = remove_subscription(SubId, D0),
     {keep_state, D, {reply, From, Ret}};
 %% Handle disown request:
-handle_event(
+do_handle_event(
     cast,
     #disown_stuck_req{sub_state = SubState},
     _State,
@@ -879,7 +922,7 @@ handle_event(
     end,
     keep_state_and_data;
 %% Handle wakeup subscription call:
-handle_event(
+do_handle_event(
     {call, From},
     #wakeup_sub_req{id = SubId},
     State,
@@ -902,7 +945,7 @@ handle_event(
             {keep_state_and_data, Reply}
     end;
 %% Handle worker crash:
-handle_event(
+do_handle_event(
     info,
     {'ETS-TRANSFER', ETS, FromPid, ?heir_info},
     _State,
@@ -915,7 +958,7 @@ handle_event(
     ets:delete(ETS),
     {next_state, ?recovering, D};
 %% Handle recoverable errors:
-handle_event(
+do_handle_event(
     {call, From},
     #handle_recoverable_req{sub_states = SubStates},
     _State,
@@ -924,7 +967,7 @@ handle_event(
     ?tp(info, emqx_ds_beamformer_handle_recoverable, #{pid => From}),
     {next_state, ?recovering, D#d{pending = SubStates ++ Pending}};
 %% Handle unknown call:
-handle_event(
+do_handle_event(
     {call, From},
     Call,
     State,
@@ -937,7 +980,7 @@ handle_event(
     ),
     {keep_state_and_data, {reply, From, ?err_unrec({unknown_call, Call})}};
 %% Handle down event:
-handle_event(
+do_handle_event(
     info,
     {'DOWN', MRef, process, _Pid, _Info},
     _State,
@@ -953,21 +996,21 @@ handle_event(
             keep_state_and_data
     end;
 %% Set up state timeouts:
-handle_event(enter, _OldState, ?initializing, _D) ->
+do_handle_event(enter, _OldState, ?initializing, _D) ->
     {keep_state_and_data, {state_timeout, 0, ?init_timeout}};
-handle_event(enter, _OldState, ?busy, _D) ->
+do_handle_event(enter, _OldState, ?busy, _D) ->
     %% In `busy' state we try to hand over subscriptions to the
     %% workers as soon as possible:
     {keep_state_and_data, {state_timeout, 0, ?schedule_timeout}};
-handle_event(enter, _OldState, ?recovering, _D) ->
+do_handle_event(enter, _OldState, ?recovering, _D) ->
     %% In `recovering' state we delay handover to the worker to let
     %% the system stabilize:
     Cooldown = 1000,
     {keep_state_and_data, {state_timeout, Cooldown, ?schedule_timeout}};
-handle_event(enter, _OldState, _NewState, _D) ->
+do_handle_event(enter, _OldState, _NewState, _D) ->
     keep_state_and_data;
 %% Perform initialization:
-handle_event(state_timeout, ?init_timeout, ?initializing, D = #d{dbshard = {DB, _}}) ->
+do_handle_event(state_timeout, ?init_timeout, ?initializing, D = #d{dbshard = {DB, _}}) ->
     try emqx_ds:list_generations_with_lifetimes(DB) of
         Generations ->
             {next_state, ?busy, D#d{generations = Generations}}
@@ -975,20 +1018,15 @@ handle_event(state_timeout, ?init_timeout, ?initializing, D = #d{dbshard = {DB,
         _:_ ->
             {keep_state_and_data, {state_timeout, 1000, ?init_timeout}}
     end;
-handle_event(_Kind, _Event, ?initializing, _D) ->
+do_handle_event(_Kind, _Event, ?initializing, _D) ->
     %% Ignore events until fulliy initialized:
     keep_state_and_data;
 %% Handover subscriptions to the workers:
-handle_event(state_timeout, ?schedule_timeout, _State, D0) ->
-    {Success, D} = do_schedule(D0),
-    NextState =
-        case Success of
-            true -> ?idle;
-            false -> ?recovering
-        end,
-    {next_state, NextState, D};
+do_handle_event(state_timeout, ?schedule_timeout, _State, D0) ->
+    D = do_schedule(D0),
+    {next_state, ?idle, D};
 %% Handle changes to the generations:
-handle_event(
+do_handle_event(
     cast,
     #generation_event{},
     _State,
@@ -1001,12 +1039,11 @@ handle_event(
     %% Notify the RT workers:
     _ = [emqx_ds_beamformer_rt:seal_generation(DBShard, I) || I <- Sealed],
     {keep_state, D#d{generations = Gens}};
-%% Unknown event:
-handle_event(EventType, Event, State, Data) ->
+do_handle_event(EventType, Event, State, D) ->
     ?tp(
         error,
         emqx_ds_beamformer_unknown_event,
-        #{event_type => EventType, state => State, data => Data, event => Event}
+        #{event_type => EventType, state => State, data => D, event => Event}
     ),
     keep_state_and_data.
 
@@ -1233,40 +1270,19 @@ diff_gens(_DBShard, Old, New) ->
         Old
     ).
 
-do_schedule(D = #d{dbshard = DBShard, pending = Pending}) ->
+do_schedule(D = #d{dbshard = DBShard, pending = Pending, inflight = Inflight0}) ->
     Pool = emqx_ds_beamformer_catchup:pool(DBShard),
-    %% Find the appropriate worker for the requests and group the requests:
-    ByWorker = maps:groups_from_list(
-        fun(#sub_state{stream = Stream}) ->
-            gproc_pool:pick_worker(Pool, Stream)
+    %% Distribute new subscriptions from the pending queue to the workers:
+    Inflight = lists:foldl(
+        fun(#sub_state{} = SubState, Acc) ->
+            %% Find the appropriate worker for the request.
+            Worker = gproc_pool:pick_worker(Pool),
+            emqx_ds_beamformer_catchup:enqueue(Worker, SubState, Acc)
         end,
+        Inflight0,
         Pending
     ),
-    %% Dispatch the subscribtions to the workers and collect ones that
-    %%  couldn't be dispatched for later retry:
-    Unmatched = maps:fold(
-        fun(Worker, Reqs, UnmatchedAcc) ->
-            case emqx_ds_beamformer_catchup:enqueue(Worker, Reqs) of
-                ok ->
-                    UnmatchedAcc;
-                Error ->
-                    %% TODO: currently enqueue cannot fail due to
-                    %% infinite timeout. However, sync call with an
-                    %% infinite timeout is a temporary solution. There
-                    %% should be a async ownership transfer protocol
-                    %% that doesn't block the caller.
-                    ?tp(
-                        error,
-                        emqx_ds_beamformer_delegate_error,
-                        #{shard => DBShard, worker => Worker, error => Error}
-                    ),
-                    Reqs ++ UnmatchedAcc
-            end
-        end,
-        [],
-        ByWorker
-    ),
-    {Unmatched =:= [], D#d{pending = Unmatched}}.
+    D#d{pending = [], inflight = Inflight}.
 
 remove_subscription(
     SubId, D = #d{dbshard = DBShard, stuck_sub_tab = SubTab, monitor_tab = MTab, pending = Pending}
@@ -1301,8 +1317,7 @@ send_out(DBShard = {DB, _}, Node, Pack, Destinations) ->
     }),
     case node() of
         Node ->
-            %% TODO: this may block the worker. Introduce a separate
-            %% worker for local fanout?
+            %% TODO: Introduce a separate worker for local fanout?
             emqx_ds_beamsplitter:dispatch_v2(DB, Pack, Destinations, #{});
         _ ->
             emqx_ds_beamsplitter_proto_v2:dispatch(DBShard, Node, DB, Pack, Destinations, #{})
diff --git a/apps/emqx_durable_storage/src/emqx_ds_beamformer_catchup.erl b/apps/emqx_durable_storage/src/emqx_ds_beamformer_catchup.erl
index c4e1a5d537a..6ab87a1c626 100644
--- a/apps/emqx_durable_storage/src/emqx_ds_beamformer_catchup.erl
+++ b/apps/emqx_durable_storage/src/emqx_ds_beamformer_catchup.erl
@@ -9,7 +9,7 @@
 -behaviour(gen_server).
 
 %% API:
--export([start_link/4, pool/1, enqueue/2]).
+-export([start_link/4, pool/1, enqueue/3]).
 
 %% behavior callbacks:
 -export([init/1, handle_call/3, handle_cast/2, handle_info/2, terminate/2]).
@@ -29,7 +29,7 @@
 %% Type declarations
 %%================================================================================
 
--record(enqueue_req, {sub_states}).
+-record(enqueue_req, {sub_state}).
 -define(fulfill_loop, beamformer_fulfill_loop).
 -define(housekeeping_loop, housekeeping_loop).
 
@@ -62,10 +62,13 @@
 %% API functions
 %%================================================================================
 
--spec enqueue(pid(), [emqx_ds_beamformer:sub_state()]) ->
-    ok | {error, unrecoverable, stale} | emqx_ds:error(_).
-enqueue(Worker, SubStates) ->
-    gen_server:call(Worker, #enqueue_req{sub_states = SubStates}, infinity).
+%% Send an asynchronous subscription handover request to the worker.
+%% Incomplete requests are tracked in the reqid collection with labels
+%% equal to the subscription state.
+-spec enqueue(pid(), emqx_ds_beamformer:sub_state(), gen_server:request_id_collection()) ->
+    gen_server:request_id_collection().
+enqueue(Worker, SubState, ReqIdCollection) ->
+    gen_server:send_request(Worker, #enqueue_req{sub_state = SubState}, SubState, ReqIdCollection).
 
 -spec start_link(module(), _Shard, integer(), emqx_ds_beamformer:opts()) -> {ok, pid()}.
 start_link(Mod, ShardId, Name, Opts) ->
@@ -97,8 +100,8 @@ init([CBM, ShardId, Name, _Opts]) ->
     self() ! ?housekeeping_loop,
     {ok, S}.
 
-handle_call(#enqueue_req{sub_states = SubStates}, _From, S) ->
-    do_enqueue(SubStates, S),
+handle_call(#enqueue_req{sub_state = SubState}, _From, S) ->
+    do_enqueue(SubState, S),
     {reply, ok, S};
 handle_call(_Call, _From, S) ->
     {reply, {error, unknown_call}, S}.
@@ -170,17 +173,11 @@ handle_recoverable(#s{sub_tab = SubTab, queue = Queue, shard_id = DBShard}, Subs
         Subscribers
     ).
 
-do_enqueue(SubStates, #s{
+do_enqueue(SubState, #s{
     shard_id = DBShard, sub_tab = SubTab, queue = Queue
 }) ->
-    lists:foreach(
-        fun(SubS) ->
-            queue_push(Queue, SubS),
-            emqx_ds_beamformer:take_ownership(DBShard, SubTab, SubS)
-        end,
-        SubStates
-    ),
-    ensure_fulfill_loop().
+    queue_push(Queue, SubState),
+    emqx_ds_beamformer:take_ownership(DBShard, SubTab, SubState).
 
 fulfill(S0 = #s{queue = Queue, metrics_id = Metrics}) ->
     %% debug_pending(S),
@@ -260,7 +257,7 @@ do_fulfill(
 ) ->
     s().
 move_to_realtime(
-    S = #s{shard_id = DBShard, queue = Queue, pending_handovers = Handovers0},
+    S = #s{shard_id = DBShard, queue = Queue, pending_handovers = Handovers0, metrics_id = Metrics},
     Stream,
     TopicFilter,
     StartKey,
@@ -277,6 +274,7 @@ move_to_realtime(
                 end_key => EndKey,
                 req_id => Req#sub_state.req_id
             }),
+            emqx_ds_builtin_metrics:inc_handover(Metrics),
             %% Remove request the active queue, but keep it in the sub
             %% table for now:
             queue_drop(Queue, Req),
@@ -291,8 +289,9 @@ move_to_realtime(
 handover_complete(
     Reply,
     SubState = #sub_state{req_id = SubRef},
-    S = #s{shard_id = DBShard, sub_tab = SubTab, queue = Queue}
+    S = #s{shard_id = DBShard, sub_tab = SubTab, queue = Queue, metrics_id = Metrics}
 ) ->
+    emqx_ds_builtin_metrics:dec_handover(Metrics),
     case Reply of
         {reply, Result} ->
             ok;
@@ -428,6 +427,7 @@ queue_new() ->
 
 queue_push(Queue, SubState) ->
     ?tp(beamformer_push_catchup, #{req_id => SubState#sub_state.req_id}),
+    ensure_fulfill_loop(),
     ets:insert(Queue, {queue_key(SubState)}).
 
 %% @doc Get all requests that await data with given stream, topic
diff --git a/apps/emqx_durable_storage/src/emqx_ds_beamformer_rt.erl b/apps/emqx_durable_storage/src/emqx_ds_beamformer_rt.erl
index f3d3a6c9534..358673f5d2d 100644
--- a/apps/emqx_durable_storage/src/emqx_ds_beamformer_rt.erl
+++ b/apps/emqx_durable_storage/src/emqx_ds_beamformer_rt.erl
@@ -17,7 +17,7 @@
 -export([init/1, handle_call/3, handle_cast/2, handle_info/2, terminate/2]).
 
 %% internal exports:
--export([]).
+-export([process_stream_event/4]).
 
 -export_type([]).
 
@@ -31,6 +31,18 @@
 %% Type declarations
 %%================================================================================
 
+%% Whenever we receive a notification that a stream has new data, we
+%% add this stream to this table. Events are deduplicated.
+-record(active_streams, {
+    %% Table of streams
+    t :: ets:tid(),
+    %% Last stream that we processed. It is kept to process streams in
+    %% a round robin fashion
+    last_key :: term()
+}).
+
+-type active_streams() :: #active_streams{}.
+
 -record(s, {
     module :: module(),
     metrics_id,
@@ -39,7 +51,14 @@
     name,
     high_watermark :: ets:tid(),
     queue :: emqx_ds_beamformer_waitq:t(),
-    batch_size :: non_neg_integer()
+    batch_size :: non_neg_integer(),
+    worker :: pid() | undefined,
+    worker_ref :: reference() | undefined,
+    %% Table of streams that has new data:
+    active_streams = active_streams_new() :: active_streams(),
+    %% Queue of pending commands (such as subscribe/unsubscribe) saved
+    %% while the temporary worker is active:
+    cmd_queue = [] :: [fun((S) -> S)]
 }).
 
 -define(housekeeping_loop, housekeeping_loop).
@@ -101,46 +120,42 @@ init([CBM, DBShard, Name, _Opts]) ->
         sub_tab = emqx_ds_beamformer:make_subtab(DBShard),
         metrics_id = emqx_ds_beamformer:metrics_id(DBShard, rt),
         name = Name,
-        high_watermark = ets:new(high_watermark, [ordered_set, private]),
+        high_watermark = ets:new(high_watermark, [ordered_set, public]),
         queue = emqx_ds_beamformer_waitq:new(),
         batch_size = emqx_ds_beamformer:cfg_batch_size()
     },
     self() ! ?housekeeping_loop,
     {ok, S}.
 
-handle_call(Req = #sub_state{}, _From, S) ->
-    Reply = do_enqueue(Req, S),
-    {reply, Reply, S};
+handle_call(Req = #sub_state{}, From, S0) ->
+    Fun = fun(S) ->
+        do_enqueue(Req, From, S)
+    end,
+    push_cmd(S0, Fun);
 handle_call(_Call, _From, S) ->
     {reply, {error, unknown_call}, S}.
 
-handle_cast(#seal_event{rank = Rank}, S = #s{queue = Queue}) ->
-    %% Currently generation seal events are treated as stream events,
-    %% for each known stream that has the matching rank:
-    Streams = emqx_ds_beamformer_waitq:streams_of_rank(Rank, Queue),
-    _ = [process_stream_event(false, Stream, S) || Stream <- Streams],
-    {noreply, S};
+handle_cast(#seal_event{rank = Rank}, S0) ->
+    Fun = fun(S = #s{queue = Queue}) ->
+        %% Currently generation seal events are treated as stream events,
+        %% for each known stream that has the matching rank:
+        Streams = emqx_ds_beamformer_waitq:streams_of_rank(Rank, Queue),
+        %% TODO: pass it through the event queue too
+        _ = [process_stream_event(self(), false, Stream, S) || Stream <- Streams],
+        S
+    end,
+    push_cmd(S0, Fun);
 handle_cast(_Cast, S) ->
     {noreply, S}.
 
-handle_info(E = #shard_event{event = Event}, S = #s{shard = Shard, queue = Queue}) ->
-    ?tp(info, beamformer_rt_event, #{event => Event, shard => Shard}),
-    %% Before processing an event, clear the mailbox from the matching
-    %% events to avoid repeatedly hammering the DB:
-    flush_similar_events(E),
-    %% TODO: make a proper event wrapper?
-    Stream = Event,
-    _ =
-        case emqx_ds_beamformer_waitq:has_candidates(Stream, Queue) of
-            true ->
-                process_stream_event(true, Stream, S);
-            false ->
-                %% Even if we don't have any poll requests for the stream,
-                %% it's still necessary to update the high watermark to
-                %% avoid full scans of very old data:
-                update_high_watermark(Stream, S)
-        end,
-    {noreply, S};
+handle_info(E = #shard_event{}, S = #s{shard = Shard, active_streams = Q}) ->
+    ?tp(debug, beamformer_rt_event, #{event => E, shard => Shard}),
+    active_streams_push(Q, E),
+    {noreply, maybe_dispatch_event(S)};
+handle_info({Ref, Result}, S0 = #s{worker_ref = Ref}) ->
+    ok = Result,
+    S = exec_pending_cmds(S0#s{worker = undefined, worker_ref = undefined}),
+    {noreply, maybe_dispatch_event(S)};
 handle_info(
     ?housekeeping_loop, S0 = #s{name = Name, metrics_id = Metrics, queue = Queue}
 ) ->
@@ -155,19 +170,29 @@ handle_info(
     erlang:send_after(emqx_ds_beamformer:cfg_housekeeping_interval(), self(), ?housekeeping_loop),
     {noreply, S};
 handle_info(
-    #unsub_req{id = SubId}, S = #s{sub_tab = SubTab, queue = Queue}
+    #unsub_req{id = SubId}, S0
 ) ->
-    case emqx_ds_beamformer:sub_tab_take(SubTab, SubId) of
-        undefined ->
-            ok;
-        {ok, SubS} ->
-            queue_drop(Queue, SubS)
+    Fun = fun(S = #s{sub_tab = SubTab, queue = Queue}) ->
+        case emqx_ds_beamformer:sub_tab_take(SubTab, SubId) of
+            undefined ->
+                ok;
+            {ok, SubS} ->
+                queue_drop(Queue, SubS)
+        end,
+        S
     end,
-    {noreply, S};
+    push_cmd(S0, Fun);
 handle_info(_Info, S) ->
     {noreply, S}.
 
-terminate(Reason, #s{shard = ShardId, name = Name, sub_tab = SubTab}) ->
+terminate(Reason, #s{shard = ShardId, name = Name, sub_tab = SubTab, worker = Worker}) ->
+    case Worker of
+        undefined ->
+            ok;
+        Pid when is_pid(Pid) ->
+            unlink(Pid),
+            erlang:exit(Pid, shutdown)
+    end,
     Pool = pool(ShardId),
     gproc_pool:disconnect_worker(Pool, Name),
     gproc_pool:remove_worker(Pool, Name),
@@ -192,52 +217,85 @@ do_enqueue(
         rank = Rank,
         client = Client
     },
+    From,
     S = #s{shard = Shard, queue = Queue, module = CBM, sub_tab = SubTab}
 ) ->
-    case high_watermark(Stream, S) of
-        {ok, HighWatermark} ->
-            case emqx_ds_beamformer:fast_forward(CBM, Shard, It0, HighWatermark) of
-                {ok, It} ->
-                    ?tp(beamformer_push_rt, #{
-                        req_id => ReqId, stream => Stream, key => Key, it => It
-                    }),
-                    emqx_ds_beamformer_waitq:insert(Stream, TF, {node(Client), ReqId}, Rank, Queue),
-                    emqx_ds_beamformer:take_ownership(Shard, SubTab, Req),
-                    ok;
-                {error, unrecoverable, has_data} ->
-                    ?tp(info, beamformer_push_rt_downgrade, #{
-                        req_id => ReqId, stream => Stream, key => Key
-                    }),
-                    {error, unrecoverable, stale};
-                Error = {error, _, _} ->
-                    ?tp(
-                        error,
-                        beamformer_rt_cannot_fast_forward,
-                        #{
-                            it => It0,
-                            reason => Error
-                        }
-                    ),
-                    Error
-            end;
-        undefined ->
-            ?tp(
-                warning,
-                beamformer_push_rt_unknown_stream,
-                #{poll_req => Req}
-            ),
-            {error, unrecoverable, unknown_stream}
-    end.
+    Reply =
+        case high_watermark(Stream, S) of
+            {ok, HighWatermark} ->
+                FFRes = emqx_ds_beamformer:fast_forward(CBM, Shard, It0, HighWatermark),
+                case FFRes of
+                    {ok, It} ->
+                        ?tp(beamformer_push_rt, #{
+                            req_id => ReqId, stream => Stream, key => Key, it => It
+                        }),
+                        emqx_ds_beamformer_waitq:insert(
+                            Stream, TF, {node(Client), ReqId}, Rank, Queue
+                        ),
+                        emqx_ds_beamformer:take_ownership(Shard, SubTab, Req),
+                        ok;
+                    {error, unrecoverable, has_data} ->
+                        %% TODO: This piece of code is very
+                        %% problematic, because it can cause an
+                        %% infinite back and forth loop.
+                        %%
+                        %% Alt. 1: Instead of kicking the subscription
+                        %% back we could attempt to simply read the
+                        %% data and deliver it to the client before
+                        %% adding it to the queue. This may, of
+                        %% course, introduce latency to the other
+                        %% clients in the RT queue. But the effect
+                        %% will be temporary, and only happen when a
+                        %% new subscription joins.
+                        %%
+                        %% Assuming that catchup worker has done all
+                        %% the heavy lifting, the impact on the
+                        %% latency should be small under normal
+                        %% circumstances.
+                        %%
+                        %% Alt. 2: Kill the subsription when this
+                        %% happens with a reason indicating overload
+                        %% and so the client re-subscribes.
+                        %%
+                        %% This should spread the load over time, but
+                        %% it's not the perfect solution.
+                        ?tp(info, beamformer_push_rt_downgrade, #{
+                            req_id => ReqId, stream => Stream, key => Key
+                        }),
+                        {error, unrecoverable, stale};
+                    Error = {error, _, _} ->
+                        ?tp(
+                            error,
+                            beamformer_rt_cannot_fast_forward,
+                            #{
+                                it => It0,
+                                reason => Error
+                            }
+                        ),
+                        Error
+                end;
+            undefined ->
+                ?tp(
+                    warning,
+                    beamformer_push_rt_unknown_stream,
+                    #{poll_req => Req}
+                ),
+                {error, unrecoverable, unknown_stream}
+        end,
+    ok = gen_server:reply(From, Reply),
+    S.
 
-process_stream_event(RetryOnEmpty, Stream, S) ->
+process_stream_event(Parent, RetryOnEmpty, Stream, S) ->
     T0 = erlang:monotonic_time(microsecond),
-    do_process_stream_event(RetryOnEmpty, Stream, S),
+    do_process_stream_event(Parent, RetryOnEmpty, Stream, S),
     emqx_ds_builtin_metrics:observe_beamformer_fulfill_time(
         S#s.metrics_id,
         erlang:monotonic_time(microsecond) - T0
-    ).
+    ),
+    ok.
 
 do_process_stream_event(
+    Parent,
     RetryOnEmpty,
     Stream,
     S = #s{
@@ -265,7 +323,7 @@ do_process_stream_event(
             RetryOnEmpty andalso
                 begin
                     ?tp(debug, beamformer_rt_retry_event, #{stream => Stream}),
-                    erlang:send_after(10, self(), stream_event(Stream))
+                    erlang:send_after(10, Parent, stream_event(Stream))
                 end,
             set_high_watermark(Stream, LastKey, S);
         {ok, LastKey, Batch} ->
@@ -282,7 +340,7 @@ do_process_stream_event(
             ),
             process_batch(Stream, LastKey, Batch, S, Beams),
             set_high_watermark(Stream, LastKey, S),
-            do_process_stream_event(false, Stream, S);
+            do_process_stream_event(Parent, false, Stream, S);
         {error, recoverable, _Err} ->
             %% FIXME:
             exit(retry);
@@ -361,13 +419,78 @@ set_high_watermark(Stream, LastSeenKey, #s{high_watermark = Tab}) ->
     ?tp(debug, beamformer_rt_set_high_watermark, #{stream => Stream, key => LastSeenKey}),
     ets:insert(Tab, {Stream, LastSeenKey}).
 
-flush_similar_events(E = #shard_event{}) ->
-    receive
-        E -> flush_similar_events(E)
-    after 0 ->
-        ok
-    end.
-
 -compile({inline, stream_event/1}).
 stream_event(Stream) ->
     #shard_event{event = Stream}.
+
+%% Add a callback to the pending command queue. It will be executed
+%% when the worker process is not running.
+push_cmd(S0 = #s{cmd_queue = Q0, worker_ref = W}, Fun) ->
+    S = S0#s{cmd_queue = [Fun | Q0]},
+    case W of
+        undefined ->
+            {noreply, exec_pending_cmds(S)};
+        _ when is_reference(W) ->
+            {noreply, S}
+    end.
+
+exec_pending_cmds(S0 = #s{cmd_queue = Q}) ->
+    T0 = erlang:monotonic_time(microsecond),
+    S = lists:foldr(
+        fun(Fun, Acc) ->
+            Fun(Acc)
+        end,
+        S0#s{cmd_queue = []},
+        Q
+    ),
+    emqx_ds_builtin_metrics:observer_beamformer_cmds_time(
+        S#s.metrics_id,
+        erlang:monotonic_time(microsecond) - T0
+    ),
+    S.
+
+active_streams_new() ->
+    #active_streams{
+        t = ets:new(evt_queue, [private, ordered_set, {keypos, #shard_event.event}])
+    }.
+
+maybe_dispatch_event(S = #s{worker_ref = W}) when is_reference(W) ->
+    S;
+maybe_dispatch_event(S = #s{active_streams = AS0, worker_ref = undefined, queue = Queue}) ->
+    case active_streams_pop(AS0) of
+        undefined ->
+            S;
+        {ok, Stream, AS} ->
+            case emqx_ds_beamformer_waitq:has_candidates(Stream, Queue) of
+                true ->
+                    {ok, Pid, Ref} = emqx_ds_lib:with_worker(
+                        ?MODULE, process_stream_event, [self(), true, Stream, S]
+                    ),
+                    S#s{worker = Pid, worker_ref = Ref, active_streams = AS};
+                false ->
+                    %% Even if we don't have any poll requests for the stream,
+                    %% it's still necessary to update the high watermark to
+                    %% avoid full scans of very old data:
+                    _ = update_high_watermark(Stream, S),
+                    S#s{active_streams = AS}
+            end
+    end.
+
+active_streams_push(#active_streams{t = T}, Event = #shard_event{}) ->
+    _ = ets:insert_new(T, Event),
+    ok.
+
+active_streams_pop(Q = #active_streams{t = T, last_key = K0}) ->
+    case ets:next(T, K0) of
+        '$end_of_table' ->
+            case ets:first(T) of
+                '$end_of_table' ->
+                    undefined;
+                Key ->
+                    [#shard_event{event = Stream}] = ets:take(T, Key),
+                    {ok, Stream, Q#active_streams{last_key = Key}}
+            end;
+        Key ->
+            [#shard_event{event = Stream}] = ets:take(T, Key),
+            {ok, Stream, Q#active_streams{last_key = Key}}
+    end.
diff --git a/apps/emqx_durable_storage/src/emqx_ds_beamformer_waitq.erl b/apps/emqx_durable_storage/src/emqx_ds_beamformer_waitq.erl
index bf9d37c200d..3aea5978d76 100644
--- a/apps/emqx_durable_storage/src/emqx_ds_beamformer_waitq.erl
+++ b/apps/emqx_durable_storage/src/emqx_ds_beamformer_waitq.erl
@@ -38,7 +38,7 @@
 %%================================================================================
 
 new() ->
-    ets:new(?MODULE, [ordered_set, private]).
+    ets:new(?MODULE, [ordered_set, public]).
 
 insert(Stream, Filter, ID, Rank, Tab) ->
     Key = make_key(Stream, Filter, ID),
diff --git a/apps/emqx_durable_storage/src/emqx_ds_builtin_metrics.erl b/apps/emqx_durable_storage/src/emqx_ds_builtin_metrics.erl
index f59bb43c99d..66f1134546c 100644
--- a/apps/emqx_durable_storage/src/emqx_ds_builtin_metrics.erl
+++ b/apps/emqx_durable_storage/src/emqx_ds_builtin_metrics.erl
@@ -25,10 +25,13 @@
 
     observe_sharing/2,
     set_subs_count/3,
+    inc_handover/1,
+    dec_handover/1,
     inc_beams_sent/2,
     inc_subs_stuck_total/1,
     inc_subs_unstuck_total/1,
     observe_beamformer_fulfill_time/2,
+    observer_beamformer_cmds_time/2,
     observe_beamformer_scan_time/2,
     observe_beamsplitter_fanout_time/2,
 
@@ -94,9 +97,11 @@
 
 -define(BEAMFORMER_METRICS, [
     {gauge, counter, ?DS_SUBS},
+    {counter, counter, ?DS_SUBS_HANDOVER},
     {counter, counter, ?DS_SUBS_BEAMS_SENT_TOTAL},
     {gauge, slide, ?DS_SUBS_REQUEST_SHARING},
     {counter, slide, ?DS_SUBS_FULFILL_TIME},
+    {counter, slide, ?DS_SUBS_PROCESS_COMMANDS_TIME},
     {counter, slide, ?DS_SUBS_SCAN_TIME}
 ]).
 
@@ -200,7 +205,7 @@ observe_buffer_flush_time(Id, FlushTime) ->
 observe_buffer_latency(Id, FlushTime) ->
     ?CATCH(emqx_metrics_worker:observe(?WORKER, Id, ?DS_BUFFER_LATENCY, FlushTime)).
 
--spec observe_store_batch_time(emqx_ds_storage_layer:shard_id(), non_neg_integer()) -> ok.
+-spec observe_store_batch_time({emqx_ds:db(), emqx_ds:shard()}, non_neg_integer()) -> ok.
 observe_store_batch_time({DB, _}, StoreTime) ->
     ?CATCH(emqx_metrics_worker:observe(?WORKER, DB, ?DS_STORE_BATCH_TIME, StoreTime)).
 
@@ -217,6 +222,12 @@ observe_sharing(Id, Sharing) ->
 set_subs_count(Id, Worker, Len) ->
     ?CATCH(emqx_metrics_worker:set_gauge(?WORKER, Id, Worker, ?DS_SUBS, Len)).
 
+inc_handover(Id) ->
+    ?CATCH(emqx_metrics_worker:inc(?WORKER, Id, ?DS_SUBS_HANDOVER)).
+
+dec_handover(Id) ->
+    ?CATCH(emqx_metrics_worker:inc(?WORKER, Id, ?DS_SUBS_HANDOVER, -1)).
+
 inc_beams_sent(Id, N) ->
     ?CATCH(emqx_metrics_worker:inc(?WORKER, Id, ?DS_SUBS_BEAMS_SENT_TOTAL, N)).
 
@@ -230,6 +241,10 @@ inc_subs_unstuck_total(Id) ->
 observe_beamformer_fulfill_time(Id, Time) ->
     ?CATCH(emqx_metrics_worker:observe(?WORKER, Id, ?DS_SUBS_FULFILL_TIME, Time)).
 
+%% @doc Add a sample of elapsed time spent forming and casting a beam
+observer_beamformer_cmds_time(Id, Time) ->
+    ?CATCH(emqx_metrics_worker:observe(?WORKER, Id, ?DS_SUBS_PROCESS_COMMANDS_TIME, Time)).
+
 %% @doc Add a sample of elapsed time spent scanning the DB stream
 observe_beamformer_scan_time(Id, Time) ->
     ?CATCH(emqx_metrics_worker:observe(?WORKER, Id, ?DS_SUBS_SCAN_TIME, Time)).
@@ -237,19 +252,19 @@ observe_beamformer_scan_time(Id, Time) ->
 observe_beamsplitter_fanout_time(DB, Time) ->
     ?CATCH(emqx_metrics_worker:observe(?WORKER, DB, ?DS_SUBS_FANOUT_TIME, Time)).
 
--spec inc_lts_seek_counter(emqx_ds_storage_layer:shard_id(), non_neg_integer()) -> ok.
+-spec inc_lts_seek_counter({emqx_ds:db(), emqx_ds:shard()}, non_neg_integer()) -> ok.
 inc_lts_seek_counter({DB, _}, Inc) ->
     ?CATCH(emqx_metrics_worker:inc(?WORKER, DB, ?DS_BITFIELD_LTS_SEEK_COUNTER, Inc)).
 
--spec inc_lts_next_counter(emqx_ds_storage_layer:shard_id(), non_neg_integer()) -> ok.
+-spec inc_lts_next_counter({emqx_ds:db(), emqx_ds:shard()}, non_neg_integer()) -> ok.
 inc_lts_next_counter({DB, _}, Inc) ->
     ?CATCH(emqx_metrics_worker:inc(?WORKER, DB, ?DS_BITFIELD_LTS_NEXT_COUNTER, Inc)).
 
--spec inc_lts_collision_counter(emqx_ds_storage_layer:shard_id(), non_neg_integer()) -> ok.
+-spec inc_lts_collision_counter({emqx_ds:db(), emqx_ds:shard()}, non_neg_integer()) -> ok.
 inc_lts_collision_counter({DB, _}, Inc) ->
     ?CATCH(emqx_metrics_worker:inc(?WORKER, DB, ?DS_BITFIELD_LTS_COLLISION_COUNTER, Inc)).
 
--spec collect_shard_counter(emqx_ds_storage_layer:shard_id(), atom(), non_neg_integer()) -> ok.
+-spec collect_shard_counter({emqx_ds:db(), emqx_ds:shard()}, atom(), non_neg_integer()) -> ok.
 collect_shard_counter({DB, _}, Key, Inc) ->
     ?CATCH(emqx_metrics_worker:inc(?WORKER, DB, Key, Inc)).
 
diff --git a/apps/emqx_durable_storage/src/emqx_ds_lib.erl b/apps/emqx_durable_storage/src/emqx_ds_lib.erl
index b0d39834370..a17f9c0ba64 100644
--- a/apps/emqx_durable_storage/src/emqx_ds_lib.erl
+++ b/apps/emqx_durable_storage/src/emqx_ds_lib.erl
@@ -23,10 +23,10 @@
 %%================================================================================
 
 %% @doc The caller will receive message of type `{reference(), Result | {error, unrecoverable, map()}'
--spec with_worker(module(), atom(), list()) -> {ok, reference()}.
+-spec with_worker(module(), atom(), list()) -> {ok, pid(), reference()}.
 with_worker(Mod, Function, Args) ->
     ReplyTo = alias([reply]),
-    _ = spawn_opt(
+    Pid = spawn_opt(
         fun() ->
             Result =
                 try
@@ -43,7 +43,7 @@ with_worker(Mod, Function, Args) ->
         end,
         [link, {min_heap_size, 10000}]
     ),
-    {ok, ReplyTo}.
+    {ok, Pid, ReplyTo}.
 
 -spec terminate(module(), _Reason, map()) -> ok.
 terminate(Module, Reason, Misc) when Reason =:= shutdown; Reason =:= normal ->


Reviews:
user: login: keynslug
state: DISMISSED
author_association: CONTRIBUTOR
submitted_at: 2025-04-25T16:52:19Z
user: login: ieQu1
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-25T21:56:03Z
user: login: ieQu1
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-25T22:07:24Z
user: login: keynslug
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-28T18:11:24Z

Review comments:
diff_hunk: @@ -101,46 +120,42 @@ init([CBM, DBShard, Name, _Opts]) ->
         sub_tab = emqx_ds_beamformer:make_subtab(DBShard),
         metrics_id = emqx_ds_beamformer:metrics_id(DBShard, rt),
         name = Name,
-        high_watermark = ets:new(high_watermark, [ordered_set, private]),
+        high_watermark = ets:new(high_watermark, [ordered_set, public]),
         queue = emqx_ds_beamformer_waitq:new(),
         batch_size = emqx_ds_beamformer:cfg_batch_size()
     },
     self() ! ?housekeeping_loop,
     {ok, S}.
 
-handle_call(Req = #sub_state{}, _From, S) ->
-    Reply = do_enqueue(Req, S),
-    {reply, Reply, S};
+handle_call(Req = #sub_state{}, From, S0) ->
+    Fun = fun(S) ->
+        do_enqueue(Req, From, S)
+    end,
+    push_cmd(S0, Fun);
 handle_call(_Call, _From, S) ->
     {reply, {error, unknown_call}, S}.
 
-handle_cast(#seal_event{rank = Rank}, S = #s{queue = Queue}) ->
-    %% Currently generation seal events are treated as stream events,
-    %% for each known stream that has the matching rank:
-    Streams = emqx_ds_beamformer_waitq:streams_of_rank(Rank, Queue),
-    _ = [process_stream_event(false, Stream, S) || Stream <- Streams],
-    {noreply, S};
+handle_cast(#seal_event{rank = Rank}, S0) ->
+    Fun = fun(S = #s{queue = Queue}) ->
+        %% Currently generation seal events are treated as stream events,
+        %% for each known stream that has the matching rank:
+        Streams = emqx_ds_beamformer_waitq:streams_of_rank(Rank, Queue),
+        %% TODO: pass it through the event queue too
+        _ = [process_stream_event(self(), false, Stream, S) || Stream <- Streams],
+        S
+    end,
+    push_cmd(S0, Fun);
 handle_cast(_Cast, S) ->
     {noreply, S}.
 
-handle_info(E = #shard_event{event = Event}, S = #s{shard = Shard, queue = Queue}) ->
-    ?tp(info, beamformer_rt_event, #{event => Event, shard => Shard}),
-    %% Before processing an event, clear the mailbox from the matching
-    %% events to avoid repeatedly hammering the DB:
-    flush_similar_events(E),
-    %% TODO: make a proper event wrapper?
-    Stream = Event,
-    _ =
-        case emqx_ds_beamformer_waitq:has_candidates(Stream, Queue) of
-            true ->
-                process_stream_event(true, Stream, S);
-            false ->
-                %% Even if we don't have any poll requests for the stream,
-                %% it's still necessary to update the high watermark to
-                %% avoid full scans of very old data:
-                update_high_watermark(Stream, S)
-        end,
-    {noreply, S};
+handle_info(E = #shard_event{}, S = #s{shard = Shard, active_streams = Q}) ->
+    ?tp(info, beamformer_rt_event, #{event => E, shard => Shard}),
path: apps/emqx_durable_storage/src/emqx_ds_beamformer_rt.erl
user: login: keynslug
body: Nit. Volume of such events might be too high for `info` level?
created_at: 2025-04-25T16:36:30Z
author_association: CONTRIBUTOR
diff_hunk: @@ -101,46 +120,42 @@ init([CBM, DBShard, Name, _Opts]) ->
         sub_tab = emqx_ds_beamformer:make_subtab(DBShard),
         metrics_id = emqx_ds_beamformer:metrics_id(DBShard, rt),
         name = Name,
-        high_watermark = ets:new(high_watermark, [ordered_set, private]),
+        high_watermark = ets:new(high_watermark, [ordered_set, public]),
         queue = emqx_ds_beamformer_waitq:new(),
         batch_size = emqx_ds_beamformer:cfg_batch_size()
     },
     self() ! ?housekeeping_loop,
     {ok, S}.
 
-handle_call(Req = #sub_state{}, _From, S) ->
-    Reply = do_enqueue(Req, S),
-    {reply, Reply, S};
+handle_call(Req = #sub_state{}, From, S0) ->
+    Fun = fun(S) ->
+        do_enqueue(Req, From, S)
+    end,
+    push_cmd(S0, Fun);
 handle_call(_Call, _From, S) ->
     {reply, {error, unknown_call}, S}.
 
-handle_cast(#seal_event{rank = Rank}, S = #s{queue = Queue}) ->
-    %% Currently generation seal events are treated as stream events,
-    %% for each known stream that has the matching rank:
-    Streams = emqx_ds_beamformer_waitq:streams_of_rank(Rank, Queue),
-    _ = [process_stream_event(false, Stream, S) || Stream <- Streams],
-    {noreply, S};
+handle_cast(#seal_event{rank = Rank}, S0) ->
+    Fun = fun(S = #s{queue = Queue}) ->
+        %% Currently generation seal events are treated as stream events,
+        %% for each known stream that has the matching rank:
+        Streams = emqx_ds_beamformer_waitq:streams_of_rank(Rank, Queue),
+        %% TODO: pass it through the event queue too
+        _ = [process_stream_event(self(), false, Stream, S) || Stream <- Streams],
path: apps/emqx_durable_storage/src/emqx_ds_beamformer_rt.erl
user: login: keynslug
body: Q: That would be pid of a worker, not the beamformer process, wondering if this is intentional.
created_at: 2025-04-25T16:52:06Z
author_association: CONTRIBUTOR
diff_hunk: @@ -101,46 +120,42 @@ init([CBM, DBShard, Name, _Opts]) ->
         sub_tab = emqx_ds_beamformer:make_subtab(DBShard),
         metrics_id = emqx_ds_beamformer:metrics_id(DBShard, rt),
         name = Name,
-        high_watermark = ets:new(high_watermark, [ordered_set, private]),
+        high_watermark = ets:new(high_watermark, [ordered_set, public]),
         queue = emqx_ds_beamformer_waitq:new(),
         batch_size = emqx_ds_beamformer:cfg_batch_size()
     },
     self() ! ?housekeeping_loop,
     {ok, S}.
 
-handle_call(Req = #sub_state{}, _From, S) ->
-    Reply = do_enqueue(Req, S),
-    {reply, Reply, S};
+handle_call(Req = #sub_state{}, From, S0) ->
+    Fun = fun(S) ->
+        do_enqueue(Req, From, S)
+    end,
+    push_cmd(S0, Fun);
 handle_call(_Call, _From, S) ->
     {reply, {error, unknown_call}, S}.
 
-handle_cast(#seal_event{rank = Rank}, S = #s{queue = Queue}) ->
-    %% Currently generation seal events are treated as stream events,
-    %% for each known stream that has the matching rank:
-    Streams = emqx_ds_beamformer_waitq:streams_of_rank(Rank, Queue),
-    _ = [process_stream_event(false, Stream, S) || Stream <- Streams],
-    {noreply, S};
+handle_cast(#seal_event{rank = Rank}, S0) ->
+    Fun = fun(S = #s{queue = Queue}) ->
+        %% Currently generation seal events are treated as stream events,
+        %% for each known stream that has the matching rank:
+        Streams = emqx_ds_beamformer_waitq:streams_of_rank(Rank, Queue),
+        %% TODO: pass it through the event queue too
+        _ = [process_stream_event(self(), false, Stream, S) || Stream <- Streams],
path: apps/emqx_durable_storage/src/emqx_ds_beamformer_rt.erl
user: login: ieQu1
body: This callback is executed in the main process. The worker only ever runs `process_stream_event` function.
created_at: 2025-04-25T21:56:03Z
author_association: MEMBER
diff_hunk: @@ -101,46 +120,42 @@ init([CBM, DBShard, Name, _Opts]) ->
         sub_tab = emqx_ds_beamformer:make_subtab(DBShard),
         metrics_id = emqx_ds_beamformer:metrics_id(DBShard, rt),
         name = Name,
-        high_watermark = ets:new(high_watermark, [ordered_set, private]),
+        high_watermark = ets:new(high_watermark, [ordered_set, public]),
         queue = emqx_ds_beamformer_waitq:new(),
         batch_size = emqx_ds_beamformer:cfg_batch_size()
     },
     self() ! ?housekeeping_loop,
     {ok, S}.
 
-handle_call(Req = #sub_state{}, _From, S) ->
-    Reply = do_enqueue(Req, S),
-    {reply, Reply, S};
+handle_call(Req = #sub_state{}, From, S0) ->
+    Fun = fun(S) ->
+        do_enqueue(Req, From, S)
+    end,
+    push_cmd(S0, Fun);
 handle_call(_Call, _From, S) ->
     {reply, {error, unknown_call}, S}.
 
-handle_cast(#seal_event{rank = Rank}, S = #s{queue = Queue}) ->
-    %% Currently generation seal events are treated as stream events,
-    %% for each known stream that has the matching rank:
-    Streams = emqx_ds_beamformer_waitq:streams_of_rank(Rank, Queue),
-    _ = [process_stream_event(false, Stream, S) || Stream <- Streams],
-    {noreply, S};
+handle_cast(#seal_event{rank = Rank}, S0) ->
+    Fun = fun(S = #s{queue = Queue}) ->
+        %% Currently generation seal events are treated as stream events,
+        %% for each known stream that has the matching rank:
+        Streams = emqx_ds_beamformer_waitq:streams_of_rank(Rank, Queue),
+        %% TODO: pass it through the event queue too
+        _ = [process_stream_event(self(), false, Stream, S) || Stream <- Streams],
+        S
+    end,
+    push_cmd(S0, Fun);
 handle_cast(_Cast, S) ->
     {noreply, S}.
 
-handle_info(E = #shard_event{event = Event}, S = #s{shard = Shard, queue = Queue}) ->
-    ?tp(info, beamformer_rt_event, #{event => Event, shard => Shard}),
-    %% Before processing an event, clear the mailbox from the matching
-    %% events to avoid repeatedly hammering the DB:
-    flush_similar_events(E),
-    %% TODO: make a proper event wrapper?
-    Stream = Event,
-    _ =
-        case emqx_ds_beamformer_waitq:has_candidates(Stream, Queue) of
-            true ->
-                process_stream_event(true, Stream, S);
-            false ->
-                %% Even if we don't have any poll requests for the stream,
-                %% it's still necessary to update the high watermark to
-                %% avoid full scans of very old data:
-                update_high_watermark(Stream, S)
-        end,
-    {noreply, S};
+handle_info(E = #shard_event{}, S = #s{shard = Shard, active_streams = Q}) ->
+    ?tp(info, beamformer_rt_event, #{event => E, shard => Shard}),
path: apps/emqx_durable_storage/src/emqx_ds_beamformer_rt.erl
user: login: ieQu1
body: True, it was a forgotten debug message. Changed the severity to debug.
created_at: 2025-04-25T22:07:24Z
author_association: MEMBER

PR comments:
user: login: codecov-commenter
created_at: 2025-04-24T22:31:42Z
body: ## [Codecov](https://app.codecov.io/gh/emqx/emqx/pull/15114?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=emqx) Report
Attention: Patch coverage is `90.09901%` with `10 lines` in your changes missing coverage. Please review.
> Project coverage is 82.57%. Comparing base [(`d6bc9ad`)](https://app.codecov.io/gh/emqx/emqx/commit/d6bc9ad9c783bf9eb9cfdad68c00f74ca55eed7b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=emqx) to head [(`82c1383`)](https://app.codecov.io/gh/emqx/emqx/commit/82c1383a243833d2b6a3a60a9179b2bbe6da5099?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=emqx).
> Report is 207 commits behind head on release-59.

| [Files with missing lines](https://app.codecov.io/gh/emqx/emqx/pull/15114?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=emqx) | Patch % | Lines |
|---|---|---|
| [...emqx\_durable\_storage/src/emqx\_ds\_beamformer\_rt.erl](https://app.codecov.io/gh/emqx/emqx/pull/15114?src=pr&el=tree&filepath=apps%2Femqx_durable_storage%2Fsrc%2Femqx_ds_beamformer_rt.erl&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=emqx#diff-YXBwcy9lbXF4X2R1cmFibGVfc3RvcmFnZS9zcmMvZW1xeF9kc19iZWFtZm9ybWVyX3J0LmVybA==) | 90.00% | [7 Missing :warning: ](https://app.codecov.io/gh/emqx/emqx/pull/15114?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=emqx) |
| [...ps/emqx\_durable\_storage/src/emqx\_ds\_beamformer.erl](https://app.codecov.io/gh/emqx/emqx/pull/15114?src=pr&el=tree&filepath=apps%2Femqx_durable_storage%2Fsrc%2Femqx_ds_beamformer.erl&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=emqx#diff-YXBwcy9lbXF4X2R1cmFibGVfc3RvcmFnZS9zcmMvZW1xeF9kc19iZWFtZm9ybWVyLmVybA==) | 81.25% | [3 Missing :warning: ](https://app.codecov.io/gh/emqx/emqx/pull/15114?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=emqx) |

<details><summary>Additional details and impacted files</summary>


```diff
@@              Coverage Diff               @@
##           release-59   #15114      +/-   ##
==============================================
+ Coverage       82.55%   82.57%   +0.02%     
==============================================
  Files            1058     1059       +1     
  Lines           75429    75849     +420     
==============================================
+ Hits            62269    62634     +365     
- Misses          13160    13215      +55     
```

</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/emqx/emqx/pull/15114?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=emqx).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=emqx).

<details><summary> :rocket: New features to boost your workflow: </summary>

- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
author_association: NONE
user: login: emqxqa
created_at: 2025-04-28T20:24:39Z
body: [TestExecution](https://emqx.atlassian.net/browse/QA-7209)
author_association: NONE