Repository: django-guardian/django-guardian

Issue:
ContentType cache in 3.0.0 causes TransactionTestCases using Object permissions to break
Upgrading to guardian 3.0 in our project [authentik](https://github.com/goauthentik/authentik) caused some tests to fail due to `get_objects_for_user` not returning objects that correctly get permissions assigned to them.

The testcase this fails in is a `TransactionTestCase` instead of `TestCase`. After some digging I noticed that `_get_ct_cached` due to being cached would result in an incorrect permission query, so as a workaround patching `_get_ct_cached` to not be cached resolves this issue: https://github.com/goauthentik/authentik/pull/14453/commits/8e7ea4570415cb3dc31b409f5f8fb2534ebb2988

I'm not sure if this is something that's expected (TransactionTestCase _does_ truncate all tables including ContentType so it makes sense this is broken) or an unintended side-effect. I wasn't able to find anything in the docs/code about special considerations when using TransactionTestCase.


Discussion:
Hi @BeryJu Thanks for the issue report! It seems like something changed about the caching policy in this circumstance (perhaps something like the cache not being populated until the first hit *after* the initial transaction). We'd welcome further investigation on this, or clarified docs!

In the meantime thanks for the workaround (reproduced here in case anyone else has the same issue):
```py
def patched__get_ct_cached(app_label, codename):
    """Caches `ContentType` instances like its `QuerySet` does."""
    return ContentType.objects.get(app_label=app_label, permission__codename=codename)


@patch("guardian.shortcuts._get_ct_cached", patched__get_ct_cached)
class YourTestCase(TransactionTestCase):
     pass
```
Ran into this issue as well, we are using django-tenants, which means that the same model has different ContentType IDs per request (as it changes which db schema is accessed based on the url). With a single worker serving multiple database schemas, simply caching the content type in memory using the lru_cache leads to content type mismatches due to the cache returning the wrong content types. (our symptoms are MixedContentTypeError in the shortcut functions).

A possible fix in our scenario would be using djangos built-in caching which can be tenant-agnostic. I don't think that helps with the above mentioned transaction test case though. There also is the `ctypes` module in guardian that allows customizing a `get_content_type_function`, although that's a different signature that doesn't work for the permission attribute based query done in the `shortcuts` module.
Hello, I spent the last days investigating this.

I bisected the history of `shortcuts.py` and found out my tests started failing (`get_objects_for_user` didn't return the expected objects) from the introduction of content types caching (c1b9d5d029a0923f1d7483e3f25859911627d2d4) and started to raise an exception from 3bf514b049ff2c1cd674e572ae60319d352b276d (I guess it surfaced the problem because of better checking).

It was particularly difficult to debug because running each test separately works fine.  
The problem appears only when executing more than one test in the same run: DB sequences are not reset between tests by default in `TransactionTestCase`, so the object retrieved from the cache [here](https://github.com/django-guardian/django-guardian/blob/ad826236f0ac6c37429a548f28a063872ad9aa55/guardian/shortcuts.py#L550) has the ID from the previous test invocation, which is different from the actual ID in the DB. This in turn makes [this check](https://github.com/django-guardian/django-guardian/blob/ad826236f0ac6c37429a548f28a063872ad9aa55/guardian/shortcuts.py#L567) fail when it should pass.

Some alternatives, each one less desirable than the other:

- manually disable CT caching in the calling code (with @thclark [method](#issuecomment-2879771186))
- clear CT cache after each test:
    ```py
    def setUp(self):
        super().setUp()
        from guardian.shortcuts import _get_ct_cached
        _get_ct_cached.cache_clear()
    ``` 
- enable `TransactionTestCase.reset_sequences` (this slows down the tests negating the advantages of caching in this case, also only works for tests)


Looks like there are some use cases where the cached CT objects become stale because the actual objects in the DB change, without a way to make their cache entries expire.
Very much would welcome a PR to address this, folks, and thanks for the rigorous diagnostics so far!

Unless someone can figure out a better way of caching, I'd suggest the best fix is to:
- [ ] add a pluggable content types cache (much as @felixrindt discusses above)
  - which is definable with a django setting
  - which allows people to implement any caching strategy they like
  - which **by default doesn't cache** the CTs (essentially reverting [c1b9d5](https://github.com/django-guardian/django-guardian/commit/c1b9d5d029a0923f1d7483e3f25859911627d2d4))
- [ ] provide the cache method used in c1b9d5 as an example for how to add your own CT cache
- [ ] add a documentation section to cover this

@sevdog when you implemented the cached content types, were there significant performance benefits for certain use cases in particular?

If nothing else comes up before my next timebox (I spend c. 1 morning per 2 months) then I'll take a closer look but would welcome a PR because it's better for me to be reviewing and jockeying releases than spending my time in the weeds.
In my project there is a double object-level permission query which is performed by every API also multiple times. Thus before the cache was implemented I had thousands of queries for the same 2 content-type every minute, the overall gain was small yet meaningful (because the CT table is small).


I was looking how Django handles this within the contenttype app, to use a similar logic to handle a similar issue (which is not present in django itself, thus I thought it was already handled) and I found that it handled through the `post_migrate` signal:

https://github.com/django/django/blob/d0e4dd5cdd743a5c43c4ccc2c8fa29d3982eaa71/django/contrib/contenttypes/apps.py#L20

Because this signal is called by TransactionTestCase in setup/teardown (according to the configuration of the class).

Just add this in `app.py`:

```diff
diff --git a/guardian/apps.py b/guardian/apps.py
index 25d38a9..6e9f1e2 100644
--- a/guardian/apps.py
+++ b/guardian/apps.py
@@ -1,5 +1,6 @@
 from . import monkey_patch_user, monkey_patch_group
 from django.apps import AppConfig
+from django.db.models.signals import post_migrate
 from guardian.conf import settings
 
 
@@ -8,6 +9,8 @@ class GuardianConfig(AppConfig):
     default_auto_field = 'django.db.models.AutoField'
 
     def ready(self):
+        from .shortcuts import _get_ct_cached
+        post_migrate.connect(_get_ct_cached.cache_clear)
         # Must patch Group here since generic
         # group permission model is definable
         monkey_patch_group()
```

I will try this before talking about removing the cache or using a pluggable mechanism (which Django does not have for CT).
That's really helpful, thanks @sevdog :)

I am working on a fix for this, patch incoming!
@sevdog awesome thank you - is there any chance you could convert to using django's inbuilt caching while you're in there, to address @felixrindt 's issue above please (I think we have two related issue intertwined here)?
Looking at `django-tenants` I see that they manually reset the CT cache upon changing tenant:
https://github.com/django-tenants/django-tenants/blob/813e8b91ceb506a5a07713ad9ca33f3fbad722d7/django_tenants/postgresql_backend/base.py#L84-L106

This can be considered "addressed" by documenting the method which resets the cache here (see the upcoming PR for more discussion).
Folks, I think @sevdog 's PR [here](https://github.com/django-guardian/django-guardian/pull/899) should solve the transaction issues. It'll be released in 3.1 shortly.

@felixrindt I'm not sure whether it'll solve your caching problem or not, but please do raise a specific issue and PR to switch over the cache type if not (refer back to this discussion!).



Patch:
diff --git a/guardian/apps.py b/guardian/apps.py
index 416ced6e..1b6da491 100644
--- a/guardian/apps.py
+++ b/guardian/apps.py
@@ -1,5 +1,6 @@
 from django.apps import AppConfig
 from django.conf import settings
+from django.db.models.signals import post_migrate
 
 from . import monkey_patch_group, monkey_patch_user
 
@@ -9,6 +10,9 @@ class GuardianConfig(AppConfig):
     default_auto_field = "django.db.models.AutoField"
 
     def ready(self):
+        from .shortcuts import clear_ct_cache
+
+        post_migrate.connect(clear_ct_cache)
         if settings.GUARDIAN_MONKEY_PATCH_GROUP:
             monkey_patch_group()
         if settings.GUARDIAN_MONKEY_PATCH_USER:
diff --git a/guardian/shortcuts.py b/guardian/shortcuts.py
index 0babdfab..94b6a344 100644
--- a/guardian/shortcuts.py
+++ b/guardian/shortcuts.py
@@ -47,11 +47,17 @@
 
 
 @lru_cache(None)
-def _get_ct_cached(app_label, codename):
+def _get_ct_cached(app_label: str, codename: str) -> ContentType:
     """Caches `ContentType` instances like its `QuerySet` does."""
     return ContentType.objects.get(app_label=app_label, permission__codename=codename)
 
 
+# kwargs are required to be connected to a django signal
+def clear_ct_cache(**kwargs) -> None:
+    """Helper to clear cache of `_get_ct_cached`"""
+    _get_ct_cached.cache_clear()
+
+
 def _get_first(t):
     """Allow sorting/grouping by pk by returning first in result tuple"""
     return t[0]


Test patch:
diff --git a/guardian/testapp/tests/test_shortcuts.py b/guardian/testapp/tests/test_shortcuts.py
index 0cb6b523..20b2a73d 100644
--- a/guardian/testapp/tests/test_shortcuts.py
+++ b/guardian/testapp/tests/test_shortcuts.py
@@ -6,13 +6,14 @@
 from django.contrib.auth.models import AnonymousUser
 from django.contrib.contenttypes.models import ContentType
 from django.db.models.query import QuerySet
-from django.test import TestCase
+from django.test import TestCase, TransactionTestCase
 
 from guardian.compat import get_user_permission_full_codename
 from guardian.core import ObjectPermissionChecker
 from guardian.exceptions import MixedContentTypeError, MultipleIdentityAndObjectError, NotUserNorGroup, WrongAppError
 from guardian.models import Group, Permission
 from guardian.shortcuts import (
+    _get_ct_cached,
     assign,
     assign_perm,
     get_group_perms,
@@ -1258,3 +1259,37 @@ def test_exception_different_ctypes(self):
         self.assertRaises(
             MixedContentTypeError, get_objects_for_group, self.group1, ["auth.change_permission", "auth.change_group"]
         )
+
+
+class ContentTypeCacheMixin:
+    def test_first_access(self):
+        shortcut_ct = _get_ct_cached("auth", "change_permission")
+        ct = ContentType.objects.get_by_natural_key("auth", "permission")
+        self.assertEqual(ct, shortcut_ct)
+
+        with self.assertNumQueries(0):
+            cached_ct_shortcut = _get_ct_cached("auth", "change_permission")
+            cached_ct = ContentType.objects.get_by_natural_key("auth", "permission")
+        self.assertEqual(cached_ct, cached_ct_shortcut)
+        self.assertIs(ct, cached_ct)
+        self.assertIs(shortcut_ct, cached_ct_shortcut)
+
+    def test_second_access(self):
+        shortcut_ct = _get_ct_cached("auth", "change_permission")
+        ct = ContentType.objects.get_by_natural_key("auth", "permission")
+        self.assertEqual(ct, shortcut_ct)
+
+        with self.assertNumQueries(0):
+            cached_ct_shortcut = _get_ct_cached("auth", "change_permission")
+            cached_ct = ContentType.objects.get_by_natural_key("auth", "permission")
+        self.assertEqual(cached_ct, cached_ct_shortcut)
+        self.assertIs(ct, cached_ct)
+        self.assertIs(shortcut_ct, cached_ct_shortcut)
+
+
+class ContentTypeCacheTestCase(ContentTypeCacheMixin, TestCase):
+    """Test cache against TestCase"""
+
+
+class ContentTypeCacheTransactionTestCase(ContentTypeCacheMixin, TransactionTestCase):
+    """Test cache against TransactionTestCase"""