Repository: whitfin/cachex
Pull request #407 (2025-04-14 19:36:45)

Populate $callers where user can provide their code
For issue #406 

I investigated the codebase and I see 5 places where we execute "user provided" code.

- [x] `Cachex.fetch` `fallback` (runs inside `Courier`)
- [x] `get_and_update` `updater` (runs inside `Locksmith`)
- things passed on `start_link`
  - [ ] ~commands~
  - [x] hooks
  - [x] warmers

- [ ] Add docs about `$callers` usage

I'll be adding them one by one in this PR.
@whitfin Let me know if you see any problems with the code for the two already implemented cases and if you can think of any other "user provided" funs or modules that could benefit from that.

tomekowal: It seems, it is not necessary to do anything with commands, because they run in the same process as they are called.
tomekowal: Hey! Thanks for your excellent feedback. I'll apply it later this week :)

> Everything else is only going to have internal child process references, which are:
> - Useless to a user, most likely
> - Derivable using the Supervision tree
>
> For example hooks will always just be the Informant PID, warmers will always just be the Incubator PID, etc.

I believe it is still useful.
The main idea behind using `$callers` is that:
- user doesn't need to care about process hierarchy at all
- they can use mocks local to the test process (and in turn have `async: true` on their tests)

Imagine a scenario where I make an expensive API call that returns a list of 1000 items and I am adding it to warmup.
What I want to write is:

```
test "foo" do
  setup_warmaup_request_mock()
  start_cache()
  call_business_logic_that_uses_the_cache()
end
```

If we don't set `$callers` for warmers this test would fail on `start_cache()` because Incubator does not know about the test process. The problem is solvable with explicit allowances, but that forces the user to learn `Cachex` internals.

However, if we have the test process PID in `$callers` list of the Incubator, it works like magic. The user does not even have to know that the warmer runs in another process.
whitfin: Okay, I don't use mocks so I guess that's why this is new to me :) this looks like dark magic to me!

In that case, then yeah, probably add it to both the hooks (Informant) and warmers (Incubator) as well. I don't think anything makes sense, as they're not user facing at all. 
tomekowal: Ready for the next round of feedback :)
tomekowal: Thanks! I've applied most of the feedback. It was really good!

I am only not sure about renaming and turning those helpers into macros, so I voiced my concerns in the comment. I am totally fine with you merging the PR as is and doing final tweaks to your liking after that :)

I am super happy, you were receptive to adding `$callers` handling across the project and helped me find all the cases. I appreciate the time you've spent on it and the time you already invested in this project. Thank you!
whitfin: Thank you again for all your work here @tomekowal!

I'm going to go ahead and merge this, particularly because I feel we seem to have time zone differences that might make this drag and I don't want to sink your time 😅 

This will ship as a v4.1, which I intend to publish as soon as I've run over everything - no point keeping you waiting for it! 

Patch:
diff --git a/docs/overview.md b/docs/overview.md
index 6fb04b6..0ca086d 100644
--- a/docs/overview.md
+++ b/docs/overview.md
@@ -139,3 +139,22 @@ These are just some of the conveniences made available by Cachex's API, but ther
 For further information or examples on supported features and options, please see the Cachex [documentation](https://hexdocs.pm/cachex) where there are several guides on specific features and workflows.
 
 All of the hosted documentation is also available in raw form in the [repository](https://github.com/whitfin/cachex/tree/main/docs).
+
+# Testing
+
+`Cachex` propagates `$callers` similarly to Elixir's `Task`. It allows libaries that use this pattern (e.g. `Req` or `Mox`) to define mocks and stubs in the test process even when the code calling the stub runs in another process. E.g.
+
+```
+test "some test" do
+  # runs in test process
+  Req.Test.stub(MyApp.Weather, fn conn ->
+    Req.Test.json(conn, %{"celsius" => 25.0})
+  end)
+
+  Cachex.fetch(:my_cache, key, fn ->
+    # runs in Cachex process
+    # but it will be able to find the stub
+    MyApp.Weather.get_temperature("Kraków, Poland")
+  end)
+end
+```
diff --git a/lib/cachex/actions/warm.ex b/lib/cachex/actions/warm.ex
index e798898..9e500c9 100644
--- a/lib/cachex/actions/warm.ex
+++ b/lib/cachex/actions/warm.ex
@@ -54,12 +54,14 @@ defmodule Cachex.Actions.Warm do
   end
 
   # Invokes a warmer with blocking enabled.
-  defp call_warmer(warmer(name: name), true),
-    do: GenServer.call(name, :cachex_warmer, :infinity)
+  defp call_warmer(warmer(name: name), true) do
+    GenServer.call(name, {:cachex_warmer, get_callers()}, :infinity)
+  end
 
   # Invokes a warmer with blocking disabled.
-  defp call_warmer(warmer(name: name), _),
-    do: send(name, :cachex_warmer)
+  defp call_warmer(warmer(name: name), _) do
+    send(name, {:cachex_warmer, get_callers()})
+  end
 
   # Converts a task result to a name reference.
   defp extract_name({_, {:ok, name}}),
diff --git a/lib/cachex/hook.ex b/lib/cachex/hook.ex
index 4f10fe0..0a3963f 100644
--- a/lib/cachex/hook.ex
+++ b/lib/cachex/hook.ex
@@ -139,7 +139,9 @@ defmodule Cachex.Hook do
       end
 
       @doc false
-      def handle_info({:cachex_notify, {event, result}}, state) do
+      def handle_info({:cachex_notify, {event, result, callers}}, state) do
+        put_callers(callers)
+
         case timeout() do
           nil ->
             {:ok, new_state} = handle_notify(event, result, state)
diff --git a/lib/cachex/services/courier.ex b/lib/cachex/services/courier.ex
index 04f03fa..215020b 100644
--- a/lib/cachex/services/courier.ex
+++ b/lib/cachex/services/courier.ex
@@ -39,8 +39,13 @@ defmodule Cachex.Services.Courier do
   call which will wait until a result can be loaded.
   """
   @spec dispatch(Cachex.t(), any, (-> any)) :: any
-  def dispatch(cache() = cache, key, task) when is_function(task, 0),
-    do: service_call(cache, :courier, {:dispatch, key, task, local_stack()})
+  def dispatch(cache() = cache, key, task) when is_function(task, 0) do
+    service_call(
+      cache,
+      :courier,
+      {:dispatch, key, task, local_stack(), get_callers()}
+    )
+  end
 
   ####################
   # Server Callbacks #
@@ -64,7 +69,14 @@ defmodule Cachex.Services.Courier do
   # Due to the nature of the async behaviour, this call will return before
   # the task has been completed, and the :notify callback will receive the
   # results from the task after completion (regardless of outcome).
-  def handle_call({:dispatch, key, task, stack}, caller, {cache, tasks} = state) do
+  def handle_call(
+        {:dispatch, key, task, stack, callers},
+        caller,
+        {cache, tasks} = state
+      ) do
+    put_callers(callers)
+    callers = get_callers()
+
     case Map.get(tasks, key) do
       {pid, listeners} ->
         {:noreply, {cache, Map.put(tasks, key, {pid, [caller | listeners]})}}
@@ -78,6 +90,7 @@ defmodule Cachex.Services.Courier do
               spawn_link(fn ->
                 result =
                   try do
+                    put_callers(callers)
                     task.()
                   rescue
                     e ->
diff --git a/lib/cachex/services/informant.ex b/lib/cachex/services/informant.ex
index f8e5d61..96ec241 100644
--- a/lib/cachex/services/informant.ex
+++ b/lib/cachex/services/informant.ex
@@ -67,7 +67,7 @@ defmodule Cachex.Services.Informant do
         # skip notifying service hooks
         if module.type() != :service do
           # define the base payload, regardless of type
-          payload = {:cachex_notify, {action, result}}
+          payload = {:cachex_notify, {action, result, get_callers()}}
 
           # handle async vs. sync
           case module.async?() do
diff --git a/lib/cachex/services/locksmith/queue.ex b/lib/cachex/services/locksmith/queue.ex
index 4c5bd18..375fc69 100644
--- a/lib/cachex/services/locksmith/queue.ex
+++ b/lib/cachex/services/locksmith/queue.ex
@@ -38,8 +38,9 @@ defmodule Cachex.Services.Locksmith.Queue do
   """
   @spec transaction(Cachex.t(), [any], (-> any)) :: any
   def transaction(cache() = cache, keys, func)
-      when is_list(keys) and is_function(func, 0),
-      do: service_call(cache, :locksmith, {:transaction, keys, func})
+      when is_list(keys) and is_function(func, 0) do
+    service_call(cache, :locksmith, {:transaction, keys, func, get_callers()})
+  end
 
   ####################
   # Server Callbacks #
@@ -72,7 +73,8 @@ defmodule Cachex.Services.Locksmith.Queue do
   # locks after execution. The key here is that the locks set on a key will stop
   # other processes from writing them, and force them to queue their writes
   # inside this queue process instead.
-  def handle_call({:transaction, keys, func}, _ctx, cache) do
+  def handle_call({:transaction, keys, func, callers}, _ctx, cache) do
+    put_callers(callers)
     true = lock(cache, keys)
     val = safe_exec(func)
     true = unlock(cache, keys)
diff --git a/lib/cachex/spec.ex b/lib/cachex/spec.ex
index d36c1de..4b3d3ae 100644
--- a/lib/cachex/spec.ex
+++ b/lib/cachex/spec.ex
@@ -470,4 +470,22 @@ defmodule Cachex.Spec do
   @spec wrap(any, atom) :: {atom, any}
   defmacro wrap(value, tag) when is_atom(tag),
     do: quote(do: {unquote(tag), unquote(value)})
+
+  # extracts list of `$callers` from the process dicionary
+  # and adds the current process to the list
+  # the intended use is to save it in a variable
+  # and use it inside spawned code
+  # e.g.
+  #
+  #     callers = get_callers()
+  #     spawn(fn -> put_callers() ... end)
+  #
+  # `$callers` allows mocking libraries like `Mox`
+  # to set mocks in the test process and use them
+  # in other processes that have the test process in `$callers`
+  @doc false
+  def get_callers(), do: [self() | Process.get(:"$callers") || []]
+
+  @doc false
+  def put_callers(callers), do: Process.put(:"$callers", callers)
 end
diff --git a/lib/cachex/warmer.ex b/lib/cachex/warmer.ex
index 0a56732..c5c1949 100644
--- a/lib/cachex/warmer.ex
+++ b/lib/cachex/warmer.ex
@@ -78,8 +78,8 @@ defmodule Cachex.Warmer do
       #
       # This is a blocking binding to `handle_info(:cache_warmer)`. See
       # the documentation of that implementation for more information.
-      def handle_call(:cachex_warmer, _from, state) do
-        {:noreply, new_state} = handle_info(:cachex_warmer, state)
+      def handle_call({:cachex_warmer, callers}, _from, state) do
+        {:noreply, new_state} = handle_info({:cachex_warmer, callers}, state)
         {:reply, :ok, new_state}
       end
 
@@ -91,7 +91,11 @@ defmodule Cachex.Warmer do
       # cache via `Cachex.put_many/3` if returns in a Tuple tagged with the
       # `:ok` atom. If `:ignore` is returned, nothing happens aside from
       # scheduling the next execution of the warming to occur on interval.
-      def handle_info(:cachex_warmer, {cache, state, interval, timer}) do
+      def handle_info(
+            {:cachex_warmer, callers},
+            {cache, state, interval, timer}
+          ) do
+        put_callers(callers)
         # clean our any existing timers
         if timer, do: Process.cancel_timer(timer)
 


Test patch:
diff --git a/test/cachex/actions/fetch_test.exs b/test/cachex/actions/fetch_test.exs
index 494c2d2..59ce00f 100644
--- a/test/cachex/actions/fetch_test.exs
+++ b/test/cachex/actions/fetch_test.exs
@@ -202,4 +202,21 @@ defmodule Cachex.Actions.FetchTest do
     # all should have been called just once
     assert %{1 => 1000} == calls
   end
+
+  test "fallback function has test process in $callers" do
+    test_process = self()
+    callers_reference = make_ref()
+
+    cache = TestUtils.create_cache()
+
+    fallback_fun = fn ->
+      send(test_process, {callers_reference, Process.get(:"$callers")})
+      "value"
+    end
+
+    {:commit, "value"} = Cachex.fetch(cache, "key", fallback_fun)
+
+    assert_receive({^callers_reference, callers})
+    assert test_process in callers
+  end
 end
diff --git a/test/cachex/actions/get_and_update_test.exs b/test/cachex/actions/get_and_update_test.exs
index c410f1e..51f62b8 100644
--- a/test/cachex/actions/get_and_update_test.exs
+++ b/test/cachex/actions/get_and_update_test.exs
@@ -120,4 +120,22 @@ defmodule Cachex.Actions.GetAndUpdateTest do
     assert(get1 == {:ok, "1"})
     assert(get2 == {:ok, "2"})
   end
+
+  test "fallback function has test process in $callers" do
+    test_process = self()
+    callers_reference = make_ref()
+
+    cache = TestUtils.create_cache()
+
+    result =
+      Cachex.get_and_update(cache, "key", fn _ ->
+        send(test_process, {callers_reference, Process.get(:"$callers")})
+        {:commit, "value"}
+      end)
+
+    assert(result == {:commit, "value"})
+
+    assert_receive({^callers_reference, callers})
+    assert test_process in callers
+  end
 end
diff --git a/test/cachex/hook_test.exs b/test/cachex/hook_test.exs
index 9ac3e95..87df62e 100644
--- a/test/cachex/hook_test.exs
+++ b/test/cachex/hook_test.exs
@@ -1,6 +1,26 @@
 defmodule Cachex.HookTest do
   use Cachex.Test.Case
 
+  defmodule Hook.Callers do
+    use Cachex.Hook
+    import Cachex.Spec
+
+    def async?, do: true
+    def actions, do: :all
+    def type, do: :pre
+
+    @doc """
+    Returns a hook definition for a custom execute hook.
+    """
+    def create(name \\ nil),
+      do: hook(module: __MODULE__, args: self(), name: name)
+
+    def handle_notify(_, _, proc) do
+      send(proc, Process.get(:"$callers"))
+      {:noreply, proc}
+    end
+  end
+
   setup_all do
     ForwardHook.bind(
       concat_hook_1: [type: :pre],
@@ -11,6 +31,15 @@ defmodule Cachex.HookTest do
     :ok
   end
 
+  test "$callers in hooks" do
+    test_process = self()
+    callers_hook = Hook.Callers.create()
+    cache = TestUtils.create_cache(hooks: [callers_hook])
+    Cachex.fetch(cache, "key1", fn _ -> "val1" end)
+    assert_receive hook_callers
+    assert test_process in hook_callers
+  end
+
   test "concatenating hooks in a cache" do
     # create a set of 3 hooks to test with
     hook1 = ForwardHook.create(:concat_hook_1)
diff --git a/test/cachex/warmer_test.exs b/test/cachex/warmer_test.exs
index 697a4cd..fc6f185 100644
--- a/test/cachex/warmer_test.exs
+++ b/test/cachex/warmer_test.exs
@@ -1,6 +1,15 @@
 defmodule Cachex.WarmerTest do
   use Cachex.Test.Case
 
+  defmodule Warmer.Callers do
+    use Cachex.Warmer
+
+    def execute(proc) do
+      send(proc, Process.get(:"$callers"))
+      {:ok, []}
+    end
+  end
+
   test "warmers which set basic values" do
     # create a test warmer to pass to the cache
     TestUtils.create_warmer(:basic_warmer, fn _ ->
@@ -148,4 +157,22 @@ defmodule Cachex.WarmerTest do
     assert_receive({{:put_many, [[{1, 1}], []]}, {:ok, true}})
     assert_receive({{:put_many, [[{2, 2}], []]}, {:ok, true}})
   end
+
+  test "warmers populate $callers" do
+    test_process = self()
+
+    TestUtils.create_cache(
+      warmers: [
+        warmer(
+          module: Warmer.Callers,
+          interval: 15000,
+          required: true,
+          state: test_process
+        )
+      ]
+    )
+
+    assert_receive callers
+    assert test_process in callers
+  end
 end