Repository: joblib/joblib
Pull request #1584 (2024-05-18 22:05:05)

Add public method to upfront check if call in cache and valid
Following the reasoning and implementation behind #820, this PR adds a public method to check if a given cached function call is in cache and valid (=meets the cache validation callback function).

codecov[bot]: ## [Codecov](https://app.codecov.io/gh/joblib/joblib/pull/1584?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=joblib) Report
Attention: Patch coverage is `93.75000%` with `1 lines` in your changes are missing coverage. Please review.
> Project coverage is 95.20%. Comparing base [(`dc0cb6d`)](https://app.codecov.io/gh/joblib/joblib/commit/dc0cb6d02dd9f43db76a0583542bc5da0f9d4b03?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=joblib) to head [(`267730b`)](https://app.codecov.io/gh/joblib/joblib/pull/1584?dropdown=coverage&src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=joblib).

| [Files](https://app.codecov.io/gh/joblib/joblib/pull/1584?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=joblib) | Patch % | Lines |
|---|---|---|
| [joblib/memory.py](https://app.codecov.io/gh/joblib/joblib/pull/1584?src=pr&el=tree&filepath=joblib%2Fmemory.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=joblib#diff-am9ibGliL21lbW9yeS5weQ==) | 80.00% | [1 Missing :warning: ](https://app.codecov.io/gh/joblib/joblib/pull/1584?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=joblib) |

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main    #1584      +/-   ##
==========================================
- Coverage   95.23%   95.20%   -0.03%     
==========================================
  Files          45       45              
  Lines        7705     7721      +16     
==========================================
+ Hits         7338     7351      +13     
- Misses        367      370       +3     
```



</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/joblib/joblib/pull/1584?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=joblib).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=joblib).

LudwigStumpp: > ## [Codecov](https://app.codecov.io/gh/joblib/joblib/pull/1584?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=joblib) Report
> 
> All modified and coverable lines are covered by tests ✅
> 
> > Project coverage is 95.22%. Comparing base [(`dc0cb6d`)](https://app.codecov.io/gh/joblib/joblib/commit/dc0cb6d02dd9f43db76a0583542bc5da0f9d4b03?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=joblib) to head [(`35bc127`)](https://app.codecov.io/gh/joblib/joblib/pull/1584?dropdown=coverage&src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=joblib).
> 
> Additional details and impacted files
> 
> ```diff
> @@            Coverage Diff             @@
> ##             main    #1584      +/-   ##
> ==========================================
> - Coverage   95.23%   95.22%   -0.02%     
> ==========================================
>   Files          45       45              
>   Lines        7705     7725      +20     
> ==========================================
> + Hits         7338     7356      +18     
> - Misses        367      369       +2     
> ```
> 
> [☔ View full report in Codecov by Sentry](https://app.codecov.io/gh/joblib/joblib/pull/1584?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=joblib). 📢 Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=joblib).

Anyone has a clue where the increase of +2 misses comes from?
LudwigStumpp: Hello Tom, yeah, I agree with your proposal of instead improving / fixing the already present `check_call_in_cache` and will update accordingly.
LudwigStumpp: @tomMoral I just moved the functionality over to the already existing `check_call_in_cache` method but unfortunately got some errors which to me however seem unrelated to the change I did - however maybe I am missing something. Could you please take a look?
tomMoral: it seems that the failure is unrelated, could you push a every in the change log?
LudwigStumpp: @tomMoral I added a section in the change log. Apart from above docs build error that I still seem to be getting, I would be good to go.

Patch:
diff --git a/CHANGES.rst b/CHANGES.rst
index f187646..81f3112 100644
--- a/CHANGES.rst
+++ b/CHANGES.rst
@@ -8,6 +8,11 @@ In development
   be passed directly to ``Memory.reduce_size``.
   https://github.com/joblib/joblib/pull/1569
 
+- Extend functionality of the ``check_call_in_cache`` method to now also
+  check against cache validity. Before, it would only check for a given call
+  if it is in cache memory.
+  https://github.com/joblib/joblib/pull/1584
+
 Release 1.4.2 -- 2024/05/02
 ---------------------------
 
diff --git a/joblib/memory.py b/joblib/memory.py
index 29e2649..c6c2060 100644
--- a/joblib/memory.py
+++ b/joblib/memory.py
@@ -593,19 +593,23 @@ class MemorizedFunc(Logger):
         return state
 
     def check_call_in_cache(self, *args, **kwargs):
-        """Check if function call is in the memory cache.
+        """Check if the function call is cached and valid for given arguments.
+
+        Does not call the function or do any work besides function inspection
+        and argument hashing.
 
-        Does not call the function or do any work besides func inspection
-        and arg hashing.
+        - Compare the function code with the one from the cached function,
+          asserting if it has changed.
+        - Check if the function call is present in the cache.
+        - Call `cache_validation_callback` for user define cache validation.
 
         Returns
         -------
         is_call_in_cache: bool
-            Whether or not the result of the function has been cached
-            for the input arguments that have been passed.
+            Whether or not the function call is in cache and can be used.
         """
         call_id = (self.func_id, self._get_args_id(*args, **kwargs))
-        return self.store_backend.contains_item(call_id)
+        return self._is_in_cache_and_valid(call_id)
 
     # ------------------------------------------------------------------------
     # Private interface


Test patch:
diff --git a/joblib/test/test_memory.py b/joblib/test/test_memory.py
index 1a6c69c..0b4100e 100644
--- a/joblib/test/test_memory.py
+++ b/joblib/test/test_memory.py
@@ -602,18 +602,31 @@ def test_persistence(tmpdir):
     gp(1)
 
 
-def test_check_call_in_cache(tmpdir):
-    for func in (MemorizedFunc(f, tmpdir.strpath),
-                 Memory(location=tmpdir.strpath, verbose=0).cache(f)):
+@pytest.mark.parametrize("consider_cache_valid", [True, False])
+def test_check_call_in_cache(tmpdir, consider_cache_valid):
+    for func in (
+        MemorizedFunc(
+            f,
+            tmpdir.strpath,
+            cache_validation_callback=lambda _: consider_cache_valid
+        ),
+        Memory(location=tmpdir.strpath, verbose=0).cache(
+            f,
+            cache_validation_callback=lambda _: consider_cache_valid
+        )
+    ):
         result = func.check_call_in_cache(2)
-        assert not result
         assert isinstance(result, bool)
+        assert not result
         assert func(2) == 5
         result = func.check_call_in_cache(2)
-        assert result
         assert isinstance(result, bool)
+        assert result == consider_cache_valid
         func.clear()
 
+    func = NotMemorizedFunc(f)
+    assert not func.check_call_in_cache(2)
+
 
 def test_call_and_shelve(tmpdir):
     # Test MemorizedFunc outputting a reference to cache.