Repository: emqx/emqx
Pull request #15036: fix(resource): trigger fallback actions when action is an "unhealthy target"
Author: login: thalesmg
Created: 2025-04-14T14:26:56Z | Merged: 2025-04-15T12:02:35Z

Fixes https://emqx.atlassian.net/browse/EMQX-14109

Release version: 5.9.0

## Summary

Previously, when the Action was marked as an "unhealthy target", the request would not even reach the buffering layer, where the fallback action triggering logic lives.

<!--
Please compose a nontrivial summary in case of significant changes.
* Point out the crucial changes in logic
* Point out the most relevant files and modules for the change
* Provide some reasoning for the decisions taken
-->

## PR Checklist
<!--
Please convert the PR to a draft if any of the following conditions are not met.
-->
- [x] For internal contributor: there is a jira ticket to track this change
- [x] The changes are covered with new or existing tests
- [na] ~~Change log for changes visible by users has been added to `changes/ee/(feat|perf|fix|breaking)-<PR-id>.en.md` files~~ unreleased feature

<!--
Please, take in account the following guidelines while working on PR:
* Try to achieve reasonable coverage of the new code
* Add property-based tests for code that performs complex user input validation or implements a complex algorithm
* Create a PR to [emqx-docs](https://github.com/emqx/emqx-docs) if documentation update is required, or make a follow-up jira ticket
* Do not squash large PRs into a single commit, try to keep comprehensive history of incremental changes
* Do not squash any significant amount of review fixes into the previous commits
-->

<!--
## Checklist for CI (.github/workflows) changes
- [ ] If changed package build workflow, pass [this action](https://github.com/emqx/emqx/actions/workflows/build_packages.yaml) (manual trigger)
- [ ] Change log has been added to `changes/` dir for user-facing artifacts update
-->



Diff:
diff --git a/apps/emqx_resource/src/emqx_resource.erl b/apps/emqx_resource/src/emqx_resource.erl
index 7c3c4ee3b67..4ef65163b91 100644
--- a/apps/emqx_resource/src/emqx_resource.erl
+++ b/apps/emqx_resource/src/emqx_resource.erl
@@ -366,33 +366,41 @@ query(ResId, Request) ->
 
 -spec query(resource_id(), Request :: term(), query_opts()) ->
     Result :: term().
-query(ResId, Request, Opts) ->
-    case emqx_resource_manager:get_query_mode_and_last_error(ResId, Opts) of
+query(ResId, Request, QueryOpts) ->
+    %% Note: `ResId` here, in Actions context, is the action resource id.  In other contexts
+    %% (e.g.: authn/authz), this is connector resource id.
+    case emqx_resource_manager:get_query_mode_and_last_error(ResId, QueryOpts) of
         {error, _} = ErrorTuple ->
             ErrorTuple;
         {ok, {_, unhealthy_target}} ->
             emqx_resource_metrics:matched_inc(ResId),
             emqx_resource_metrics:dropped_resource_stopped_inc(ResId),
+            emqx_resource_buffer_worker:unhealthy_target_maybe_trigger_fallback_actions(
+                ResId, Request, QueryOpts
+            ),
             ?RESOURCE_ERROR(unhealthy_target, "unhealthy target");
         {ok, {_, {unhealthy_target, Message}}} ->
             emqx_resource_metrics:matched_inc(ResId),
             emqx_resource_metrics:dropped_resource_stopped_inc(ResId),
+            emqx_resource_buffer_worker:unhealthy_target_maybe_trigger_fallback_actions(
+                ResId, Request, QueryOpts
+            ),
             ?RESOURCE_ERROR(unhealthy_target, Message);
         {ok, {simple_async, _}} ->
             %% TODO(5.1.1): pass Resource instead of ResId to simple APIs
             %% so the buffer worker does not need to lookup the cache again
-            emqx_resource_buffer_worker:simple_async_query(ResId, Request, Opts);
+            emqx_resource_buffer_worker:simple_async_query(ResId, Request, QueryOpts);
         {ok, {simple_sync, _}} ->
             %% TODO(5.1.1): pass Resource instead of ResId to simple APIs
             %% so the buffer worker does not need to lookup the cache again
-            emqx_resource_buffer_worker:simple_sync_query(ResId, Request, Opts);
+            emqx_resource_buffer_worker:simple_sync_query(ResId, Request, QueryOpts);
         {ok, {simple_async_internal_buffer, _}} ->
             %% This is for bridges/connectors that have internal buffering, such
             %% as Kafka and Pulsar producers.
             %% TODO(5.1.1): pass Resource instead of ResId to simple APIs
             %% so the buffer worker does not need to lookup the cache again
             emqx_resource_buffer_worker:simple_async_internal_buffer_query(
-                ResId, Request, Opts
+                ResId, Request, QueryOpts
             );
         {ok, {simple_sync_internal_buffer, _}} ->
             %% This is for bridges/connectors that have internal buffering, such
@@ -400,12 +408,12 @@ query(ResId, Request, Opts) ->
             %% TODO(5.1.1): pass Resource instead of ResId to simple APIs
             %% so the buffer worker does not need to lookup the cache again
             emqx_resource_buffer_worker:simple_sync_internal_buffer_query(
-                ResId, Request, Opts
+                ResId, Request, QueryOpts
             );
         {ok, {sync, _}} ->
-            emqx_resource_buffer_worker:sync_query(ResId, Request, Opts);
+            emqx_resource_buffer_worker:sync_query(ResId, Request, QueryOpts);
         {ok, {async, _}} ->
-            emqx_resource_buffer_worker:async_query(ResId, Request, Opts)
+            emqx_resource_buffer_worker:async_query(ResId, Request, QueryOpts)
     end.
 
 -spec simple_sync_query(resource_id(), Request :: term()) -> Result :: term().
diff --git a/apps/emqx_resource/src/emqx_resource_buffer_worker.erl b/apps/emqx_resource/src/emqx_resource_buffer_worker.erl
index 149efd1dd84..e770f24ed05 100644
--- a/apps/emqx_resource/src/emqx_resource_buffer_worker.erl
+++ b/apps/emqx_resource/src/emqx_resource_buffer_worker.erl
@@ -55,6 +55,9 @@
     handle_async_internal_buffer_reply/2
 ]).
 
+%% Internal exports for `emqx_resource`.
+-export([unhealthy_target_maybe_trigger_fallback_actions/3]).
+
 -export([clear_disk_queue_dir/2]).
 
 -elvis([{elvis_style, dont_repeat_yourself, disable}]).
@@ -2545,6 +2548,27 @@ request_context(QueryOpts) ->
 result_context(Queries) ->
     #{?queries => Queries}.
 
+-doc """
+This an internal export to be used only in `emqx_resource` application in places where
+the request is dropped/fails before it even reaches the buffering layer (here).  An
+example of such situation is when the resource is deemed an "unhealthy target".
+""".
+-spec unhealthy_target_maybe_trigger_fallback_actions(
+    action_resource_id(), request(), query_opts()
+) ->
+    ok.
+unhealthy_target_maybe_trigger_fallback_actions(ConnResId, Req, QueryOpts0) ->
+    QueryOpts1 = ensure_timeout_query_opts(QueryOpts0, sync),
+    QueryOpts = ensure_expire_at(QueryOpts1),
+    ReplyFun = maps:get(async_reply_fun, QueryOpts, undefined),
+    HasBeenSent = false,
+    ExpireAt = maps:get(expire_at, QueryOpts),
+    RequestContext = request_context(QueryOpts),
+    TraceCtx = maps:get(trace_ctx, QueryOpts, undefined),
+    Query = ?QUERY(ReplyFun, Req, HasBeenSent, ExpireAt, RequestContext, TraceCtx),
+    ResultContext = result_context([Query]),
+    maybe_trigger_fallback_actions(ConnResId, ResultContext).
+
 %% Should be called for individual queries, not batches.
 %% `batch_reply_caller_defer_metrics' handles splitting a batch into individual result
 %% contexts.
diff --git a/apps/emqx_resource/test/emqx_connector_demo.erl b/apps/emqx_resource/test/emqx_connector_demo.erl
index fc65b5a67c3..bbf6e228cb8 100644
--- a/apps/emqx_resource/test/emqx_connector_demo.erl
+++ b/apps/emqx_resource/test/emqx_connector_demo.erl
@@ -422,6 +422,8 @@ on_get_status(ConnResId, #{health_check_agent := Agent}) ->
         {notify, Pid, Status} when ?IS_STATUS(Status) ->
             Pid ! {returning_resource_health_check_result, ConnResId, Status},
             Status;
+        {Status, Msg} when ?IS_STATUS(Status) ->
+            {Status, Msg};
         Status when ?IS_STATUS(Status) ->
             Status
     end;
diff --git a/apps/emqx_resource/test/emqx_resource_SUITE.erl b/apps/emqx_resource/test/emqx_resource_SUITE.erl
index ff646fd2301..d0d5539b3ca 100644
--- a/apps/emqx_resource/test/emqx_resource_SUITE.erl
+++ b/apps/emqx_resource/test/emqx_resource_SUITE.erl
@@ -555,6 +555,15 @@ apply_after(Time, Fn) ->
 fallback_actions_basic_setup(ConnName, ChanQueryMode, CallbackMode, ShouldBatch) ->
     fallback_actions_basic_setup(ConnName, ChanQueryMode, CallbackMode, ShouldBatch, _Opts = #{}).
 
+-doc """"
+Options:
+  - `fallback_fn`: the function that will be called by the resource itself to trigger
+    the fallback actions.
+  - `assert_expected_callmode`: if true, asserts that the expected query mode in buffer
+    worker was called.
+  - `adjust_resource_config_fn`: a function that receives the default resource config for
+    this scenario and returns a new resource config.
+"""".
 fallback_actions_basic_setup(ConnName, ChanQueryMode, CallbackMode, ShouldBatch, Opts) ->
     ChanResourceOptsOverride = maps:get(chan_resource_opts_override, Opts, #{}),
     emqx_connector_demo:set_callback_mode(CallbackMode),
@@ -582,12 +591,14 @@ fallback_actions_basic_setup(ConnName, ChanQueryMode, CallbackMode, ShouldBatch,
         end
     end,
     ConnResId = connector_res_id(ConnName),
+    AdjustConnConfigFn = maps:get(adjust_conn_config_fn, Opts, fun(Cfg) -> Cfg end),
+    ConnConfig = AdjustConnConfigFn(#{name => test_resource}),
     {ok, _} =
         create(
             ConnResId,
             ?DEFAULT_RESOURCE_GROUP,
             ?TEST_RESOURCE,
-            #{name => test_resource},
+            ConnConfig,
             #{
                 health_check_interval => 100,
                 start_timeout => 100
@@ -602,14 +613,15 @@ fallback_actions_basic_setup(ConnName, ChanQueryMode, CallbackMode, ShouldBatch,
         ResourceOpts0,
         ChanResourceOptsOverride
     ]),
+    ChanConfig = #{
+        force_query_mode => ChanQueryMode,
+        resource_opts => ChanResourceOpts
+    },
     ok =
         add_channel(
             ConnResId,
             ChanId,
-            #{
-                force_query_mode => ChanQueryMode,
-                resource_opts => ChanResourceOpts
-            }
+            ChanConfig
         ),
     FallbackFn = maps:get(
         fallback_fn,
@@ -4620,6 +4632,101 @@ t_fallback_actions_expired_internal_buffer(Config) when is_list(Config) ->
     ),
     ok.
 
+%% When a resource/channel is an "unhealthy target", messages do not even reach the
+%% buffering layer, where most of the fallback actions logic live.  So we have to deal
+%% with fallback actions outside it.
+t_fallback_actions_unhealthy_target(Config) ->
+    ConnName = <<"fallback_actions_unhealthy_target">>,
+    ChanQueryMode = sync,
+    CallbackMode = always_sync,
+    ShouldBatch = no_batch,
+    ?check_trace(
+        begin
+            AgentState0 = #{
+                resource_health_check => ?status_connected,
+                channel_health_check =>
+                    {?status_disconnected, {unhealthy_target, <<"something is wrong">>}}
+            },
+            {ok, HCAgent} = emqx_utils_agent:start_link(AgentState0),
+            AdjustConnConfigFn = fun(Cfg) ->
+                Cfg#{health_check_agent => HCAgent}
+            end,
+            Opts = #{adjust_conn_config_fn => AdjustConnConfigFn},
+            Ctx0 = fallback_actions_basic_setup(
+                ConnName, ChanQueryMode, CallbackMode, ShouldBatch, Opts
+            ),
+            Ctx = Ctx0#{assert_expected_callmode => false},
+            ConnResId = maps:get(conn_res_id, Ctx),
+            ChanId = maps:get(chan_id, Ctx),
+            ?assertMatch(
+                #{
+                    status := ?status_disconnected,
+                    error := {unhealthy_target, <<"something is wrong">>}
+                },
+                emqx_resource:channel_health_check(ConnResId, ChanId)
+            ),
+            Results1 = fallback_send_many(sync, Ctx),
+            assert_all_queries_triggered_fallbacks(Ctx),
+            ?assertMatch([?RESOURCE_ERROR(unhealthy_target)], Results1),
+
+            _ = emqx_utils_agent:get_and_update(HCAgent, fun(Old) ->
+                {unused, Old#{
+                    channel_health_check :=
+                        {?status_disconnected, unhealthy_target}
+                }}
+            end),
+            ?retry(
+                200,
+                10,
+                ?assertMatch(
+                    #{
+                        status := ?status_disconnected,
+                        error := unhealthy_target
+                    },
+                    emqx_resource:channel_health_check(ConnResId, ChanId)
+                )
+            ),
+            Results2 = fallback_send_many(sync, Ctx),
+            assert_all_queries_triggered_fallbacks(Ctx),
+            ?assertMatch([?RESOURCE_ERROR(unhealthy_target)], Results2),
+
+            _ = emqx_utils_agent:get_and_update(HCAgent, fun(Old) ->
+                {unused, Old#{
+                    %% Not entirely sure if this path is reached in reality, but adding this case
+                    %% to cover the corresponding code path in
+                    %% `emqx_resource_buffer_worker:call_query`.
+                    resource_health_check := {?status_connecting, unhealthy_target},
+
+                    channel_health_check := ?status_connected
+                }}
+            end),
+            ?retry(
+                200,
+                10,
+                ?assertMatch(
+                    %% Connecting, because connector is not connected
+                    #{status := ?status_connecting},
+                    emqx_resource:channel_health_check(ConnResId, ChanId)
+                )
+            ),
+            ?retry(
+                200,
+                10,
+                ?assertMatch(
+                    {ok, ?status_connecting},
+                    emqx_resource:health_check(ConnResId)
+                )
+            ),
+            Results3 = fallback_send_many(sync, Ctx),
+            assert_all_queries_triggered_fallbacks(Ctx),
+            ?assertMatch([{error, {unrecoverable_error, unhealthy_target}}], Results3),
+
+            ok
+        end,
+        []
+    ),
+    ok.
+
 %% Checks that, if one removal from the state fails, it's periodically retried until it
 %% succeeds.  Adding the same `ChannelId' again should cancel such retries.
 t_retry_remove_channel(_Config) ->


Reviews:
user: login: savonarola
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-14T18:17:44Z
user: login: savonarola
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-14T18:17:59Z
user: login: savonarola
state: DISMISSED
author_association: CONTRIBUTOR
submitted_at: 2025-04-14T18:21:10Z
user: login: savonarola
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-14T18:25:37Z

Review comments:
diff_hunk: @@ -2545,6 +2548,27 @@ request_context(QueryOpts) ->
 result_context(Queries) ->
     #{?queries => Queries}.
 
+-doc """
+This an internal export to be used only in `emqx_resourcee` application in places where
path: apps/emqx_resource/src/emqx_resource_buffer_worker.erl
user: login: savonarola
body: ```suggestion
This an internal export to be used only in `emqx_resource` application in places where
```
created_at: 2025-04-14T18:17:44Z
author_association: CONTRIBUTOR
diff_hunk: @@ -2545,6 +2548,27 @@ request_context(QueryOpts) ->
 result_context(Queries) ->
     #{?queries => Queries}.
 
+-doc """
+This an internal export to be used only in `emqx_resourcee` application in places where
+the request is fropped/fails before it even reaches the buffering layer (here).  An
path: apps/emqx_resource/src/emqx_resource_buffer_worker.erl
user: login: savonarola
body: ```suggestion
the request is dropped/fails before it even reaches the buffering layer (here).  An
```
created_at: 2025-04-14T18:17:59Z
author_association: CONTRIBUTOR

PR comments:
user: login: codecov-commenter
created_at: 2025-04-14T16:30:24Z
body: ## [Codecov](https://app.codecov.io/gh/emqx/emqx/pull/15036?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=emqx) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 82.56%. Comparing base [(`d6bc9ad`)](https://app.codecov.io/gh/emqx/emqx/commit/d6bc9ad9c783bf9eb9cfdad68c00f74ca55eed7b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=emqx) to head [(`2ea8b04`)](https://app.codecov.io/gh/emqx/emqx/commit/2ea8b04612a1817bbed0d4aaa9a3441f4629e4c3?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=emqx).
> Report is 67 commits behind head on release-59.

<details><summary>Additional details and impacted files</summary>


```diff
@@              Coverage Diff               @@
##           release-59   #15036      +/-   ##
==============================================
+ Coverage       82.55%   82.56%   +0.01%     
==============================================
  Files            1058     1059       +1     
  Lines           75429    75548     +119     
==============================================
+ Hits            62269    62379     +110     
- Misses          13160    13169       +9     
```

</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/emqx/emqx/pull/15036?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=emqx).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=emqx).

<details><summary> :rocket: New features to boost your workflow: </summary>

- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
author_association: NONE
user: login: emqxqa
created_at: 2025-04-15T12:22:48Z
body: [TestExecution](https://emqx.atlassian.net/browse/QA-7057)
author_association: NONE