Repository: typeddjango/django-stubs

Issue:
`Self` typed custom queryset methods incompatible with base queryset type
# Bug report

## What's wrong

Custom queryset methods with a return type of `Self` have a return type incompatible with the base `QuerySet` class.

```python
class LogQuerySet(QuerySet["Log"]):

    def filter_a(self) -> QuerySet["Log"]:
        return self

    def filter_b(self) -> Self:
        return self

LogManager = Manager.from_queryset(LogQuerySet)

class Log(models.Model):
    name = models.CharField(max_length=100)
    objects = LogManager()

a: models.QuerySet[Log] = Log.objects.filter_a()  # OK
b: models.QuerySet[Log] = Log.objects.filter_b()  # Incompatible types in assignment
```
mypy reports:
> Incompatible types in assignment (expression has type "ManagerFromLogQuerySet[Log]", variable has type "_QuerySet[Log, Log]")  [assignment]

## How is that should be

It would be great if the return type was inferred to be a sub-type of `QuerySet` with the correct type, eg, `QuerySet[Log]`.

## System information

- OS:
- `python` version: 3.11.6/3.8.18
- `django` version: 3.2.23
- `mypy` version: 1.6.1
- `django-stubs` version: 4.2.6
- `django-stubs-ext` version: 4.2.5



Discussion:
I've been investigating this. It arises from #1789 , fixing #1788, where it was specified that queryset method return types of `Self` used in `from_queryset()` or `as_manager()` should resolve to the dynamic manager class. The issue is that such a method does actually return a queryset and the dynamic manager class is not a queryset (or at least mypy doesn't think so). Referring back to the example in the description of this issue, I think that `Self` should resolve to `LogQuerySet[Log, Log]`.
I do not think this is realy an error. Since Log.objects is not realy a LogQuerySet[Log, Log]. It is a generated manager by Django that derivates from LogQuerySet, but is not a LogQuerySet. So self should not rersolve a 
LogQuerySet striclty : it has its methods but also Manager methods for example.

But maybe the issue here is more global (cf @flaeppe) that ManagerFromLogQuerySet is not detected has a subclass of LogQuerySet, which I think it is.



Hi @moranabadie, thanks for taking a look. The manager that Django creates in `from_queryset` or `as_manager` is conceptually derived from LogQuerySet but [not by sub-classing the queryset](https://github.com/django/django/blob/main/django/db/models/manager.py#L111-L118). Rather the manager has methods created that wrap the queryset methods. It makes sense (to me anyway) that in this scenario `Self` is resolved with respect to the queryset and not the dynamic manager class.

Perhaps the plugin's current application of `Self` would work for the example I've provided if `ManagerFromLogQuerySet` was a sub-class of `LogQuerySet`. But I don't believe it's correct to say that queryset methods wrapped on the derived manager class return an instance of the derived manager class as managers have methods and attributes that querysets do not.

This stems from the fact that Django dynamically adds queryset methods as attributes of a manager. The mypy plugin simulates that behaviour as well. There's a note hinting towards that happening, which is also slightly relevant:

https://github.com/typeddjango/django-stubs/blob/82c394fbdc1461d0d483162577fac50e9cacd605/django-stubs/db/models/manager.pyi#L37-L38

We probably need to help mypy _resolve_ `Self` (for convenience, usability and to avoid confusion) in the plugin when constructing the dynamic manager (in this case called `ManagerFromLogQuerySet`). As mypy resolves to its current self(the manager), even though the method came from a different class.

You would probably also find the incorrectness by printing out the return type of `.filter_b()` e.g. `print(type(Log.objects.filter_b()))` and see that it's indeed of `LogQuerySet` and not `ManagerFromLogQuerySet`

I happen to know that the plugin strips the first argument (`self`) when adding queryset methods to the initialized manager (mypy does this too, see reveal for `line 13` in playground link below).

Stripping happens here via the `[1:]`

https://github.com/typeddjango/django-stubs/blob/82c394fbdc1461d0d483162577fac50e9cacd605/mypy_django_plugin/transformers/managers.py#L143-L150

If we try to simulate the plugin behaviour manually in playground, we see that mypy emits an error for the `self` argument, which happen to be stripped by the plugin: https://mypy-play.net/?mypy=latest&python=3.11&gist=b2aee3895956ae3e8ceaaefe4837d0ad

Inlining playground contents for clarity

```python
from typing_extensions import Self

class QuerySet:
    def method(self) -> Self:
        return self
    

class Manager:
    method = QuerySet.method
    
    
reveal_type(Manager.method)
reveal_type(Manager().method)
reveal_type(Manager().method())
```

I'll just keep going here, mostly for notes to self. But the issues emitted from the above snippet can instead of stripping be fixed by e.g. annotate every manager method as:

```diff
+from typing import Any, Callable, ClassVar
from typing_extensions import Self

class QuerySet:
    def method(self) -> Self:
        return self
    

class Manager:
-    method = QuerySet.method
+    method: ClassVar[Callable[[Any], QuerySet]] = QuerySet.method
    
    
reveal_type(Manager.method)
reveal_type(Manager().method)
reveal_type(Manager().method())
```

Notes:

- It's only valid to swap out `typing_extensions.Self` or `typing.Self` in the return type
- Instead of stripping the `self` parameter we could annotate it as `Any` on the manager callable (as `Manager` and `QuerySet` don't share any bases)
- We should make sure all methods are `ClassVar`s (I think?)
- We have to be careful to not annotate types that doesn't exist in the module, in case we add annotations, refs:
  - https://github.com/typeddjango/django-stubs/issues/709#issuecomment-1006578356
  - https://github.com/typeddjango/django-stubs/pull/820#issue-1104854693

Resolving patch:
diff --git a/mypy_django_plugin/transformers/managers.py b/mypy_django_plugin/transformers/managers.py
--- a/mypy_django_plugin/transformers/managers.py
+++ b/mypy_django_plugin/transformers/managers.py
@@ -109,19 +109,21 @@ def _process_dynamic_method(
     variables = method_type.variables
     ret_type = method_type.ret_type
 
+    if not is_fallback_queryset:
+        queryset_instance = Instance(queryset_info, manager_instance.args)
+    else:
+        # The fallback queryset inherits _QuerySet, which has two generics
+        # instead of the one exposed on QuerySet. That means that we need
+        # to add the model twice. In real code it's not possible to inherit
+        # from _QuerySet, as it doesn't exist at runtime, so this fix is
+        # only needed for plugin-generated querysets.
+        queryset_instance = Instance(queryset_info, [manager_instance.args[0], manager_instance.args[0]])
+
     # For methods on the manager that return a queryset we need to override the
     # return type to be the actual queryset class, not the base QuerySet that's
     # used by the typing stubs.
     if method_name in MANAGER_METHODS_RETURNING_QUERYSET:
-        if not is_fallback_queryset:
-            ret_type = Instance(queryset_info, manager_instance.args)
-        else:
-            # The fallback queryset inherits _QuerySet, which has two generics
-            # instead of the one exposed on QuerySet. That means that we need
-            # to add the model twice. In real code it's not possible to inherit
-            # from _QuerySet, as it doesn't exist at runtime, so this fix is
-            # only needed for pluign-generated querysets.
-            ret_type = Instance(queryset_info, [manager_instance.args[0], manager_instance.args[0]])
+        ret_type = queryset_instance
         variables = []
     args_types = method_type.arg_types[1:]
     if _has_compatible_type_vars(base_that_has_method):
@@ -138,7 +140,7 @@ def _process_dynamic_method(
             ]
     if base_that_has_method.self_type:
         # Manages -> Self returns
-        ret_type = _replace_type_var(ret_type, base_that_has_method.self_type.fullname, manager_instance)
+        ret_type = _replace_type_var(ret_type, base_that_has_method.self_type.fullname, queryset_instance)
 
     # Drop any 'self' argument as our manager is already initialized
     return method_type.copy_modified(


Test patch:
diff --git a/tests/typecheck/managers/querysets/test_as_manager.yml b/tests/typecheck/managers/querysets/test_as_manager.yml
--- a/tests/typecheck/managers/querysets/test_as_manager.yml
+++ b/tests/typecheck/managers/querysets/test_as_manager.yml
@@ -1,10 +1,11 @@
 -   case: self_return_management
     main: |
-        from myapp.models import MyModel
-        reveal_type(MyModel.objects.example_simple())  # N: Revealed type is "myapp.models.ManagerFromMyQuerySet[myapp.models.MyModel]"
-        reveal_type(MyModel.objects.example_list())  # N: Revealed type is "builtins.list[myapp.models.ManagerFromMyQuerySet[myapp.models.MyModel]]"
+        from myapp.models import MyModel, MyModelWithoutSelf
+        reveal_type(MyModel.objects.example_simple())  # N: Revealed type is "myapp.models.MyQuerySet[myapp.models.MyModel]"
+        reveal_type(MyModel.objects.example_list())  # N: Revealed type is "builtins.list[myapp.models.MyQuerySet[myapp.models.MyModel]]"
         reveal_type(MyModel.objects.example_simple().just_int())  # N: Revealed type is "builtins.int"
-        reveal_type(MyModel.objects.example_dict())  # N: Revealed type is "builtins.dict[builtins.str, myapp.models.ManagerFromMyQuerySet[myapp.models.MyModel]]"
+        reveal_type(MyModel.objects.example_dict())  # N: Revealed type is "builtins.dict[builtins.str, myapp.models.MyQuerySet[myapp.models.MyModel]]"
+        reveal_type(MyModelWithoutSelf.objects.method())  # N: Revealed type is "myapp.models.QuerySetWithoutSelf"
 
     installed_apps:
         - myapp
@@ -26,6 +27,13 @@
 
                 class MyModel(models.Model):
                     objects = MyQuerySet.as_manager()
+
+                class QuerySetWithoutSelf(models.QuerySet["MyModelWithoutSelf"]):
+                    def method(self) -> "QuerySetWithoutSelf":
+                        return self
+
+                class MyModelWithoutSelf(models.Model):
+                    objects = QuerySetWithoutSelf.as_manager()
 -   case: declares_manager_type_like_django
     main: |
         from myapp.models import MyModel
@@ -192,6 +200,8 @@
         reveal_type(MyOtherModel.objects.dummy_override())  # N: Revealed type is "myapp.models.MyOtherModel"
         reveal_type(MyOtherModel.objects.example_mixin(MyOtherModel()))  # N: Revealed type is "myapp.models.MyOtherModel"
         reveal_type(MyOtherModel.objects.example_other_mixin())  # N: Revealed type is "myapp.models.MyOtherModel"
+        reveal_type(MyOtherModel.objects.test_self())  # N: Revealed type is "myapp.models._MyModelQuerySet2[myapp.models.MyOtherModel]"
+        reveal_type(MyOtherModel.objects.test_sub_self())  # N: Revealed type is "myapp.models._MyModelQuerySet2[myapp.models.MyOtherModel]"
     installed_apps:
         - myapp
     files:
@@ -200,6 +210,7 @@
             content: |
               from typing import TypeVar, Generic
               from django.db import models
+              from typing_extensions import Self
 
               T = TypeVar("T", bound=models.Model)
               T_2 = TypeVar("T_2", bound=models.Model)
@@ -215,12 +226,14 @@
                   def override(self) -> T: ...
                   def override2(self) -> T: ...
                   def dummy_override(self) -> int: ...
+                  def test_sub_self(self) -> Self: ...
 
               class _MyModelQuerySet2(SomeMixin, _MyModelQuerySet[T_2]):
                   def example_2(self) -> T_2: ...
                   def override(self) -> T_2: ...
                   def override2(self) -> T_2: ...
                   def dummy_override(self) -> T_2: ...  # type: ignore[override]
+                  def test_self(self) -> Self: ...
 
               class MyModelQuerySet(_MyModelQuerySet2["MyModel"]):
                   def override(self) -> "MyModel": ...
diff --git a/tests/typecheck/managers/querysets/test_from_queryset.yml b/tests/typecheck/managers/querysets/test_from_queryset.yml
--- a/tests/typecheck/managers/querysets/test_from_queryset.yml
+++ b/tests/typecheck/managers/querysets/test_from_queryset.yml
@@ -1,8 +1,12 @@
 -   case: from_queryset_self_return_management
     main: |
-        from myapp.models import MyModel
-        reveal_type(MyModel.objects.example_simple())  # N: Revealed type is "myapp.models.BaseManagerFromModelQuerySet[myapp.models.MyModel]"
-        reveal_type(MyModel.objects.example_list())  # N: Revealed type is "builtins.list[myapp.models.BaseManagerFromModelQuerySet[myapp.models.MyModel]]"
+        from myapp.models import MyModel, MyModelWithoutSelf
+        reveal_type(MyModel.objects.example_simple())  # N: Revealed type is "myapp.models.MyQuerySet[myapp.models.MyModel]"
+        reveal_type(MyModel.objects.example_list())  # N: Revealed type is "builtins.list[myapp.models.MyQuerySet[myapp.models.MyModel]]"
+        reveal_type(MyModel.objects.example_simple().just_int())  # N: Revealed type is "builtins.int"
+        reveal_type(MyModel.objects.example_dict())  # N: Revealed type is "builtins.dict[builtins.str, myapp.models.MyQuerySet[myapp.models.MyModel]]"
+        reveal_type(MyModel.objects.test_custom_manager())  # N: Revealed type is "myapp.models.CustomManagerFromMyQuerySet[myapp.models.MyModel]"
+        reveal_type(MyModelWithoutSelf.objects.method())  # N: Revealed type is "myapp.models.QuerySetWithoutSelf"
     installed_apps:
         - myapp
     files:
@@ -11,16 +15,34 @@
             content: |
                 from django.db import models
                 from django.db.models.manager import BaseManager
+                from typing import List, Dict
                 from typing_extensions import Self
-                from typing import List
 
-                class ModelQuerySet(models.QuerySet):
+                class CustomManager(BaseManager):
+                    def test_custom_manager(self) -> Self: ...
+
+                class BaseQuerySet(models.QuerySet):
+                    def example_dict(self) -> Dict[str, Self]: ...
+
+                class MyQuerySet(BaseQuerySet):
                     def example_simple(self) -> Self: ...
                     def example_list(self) -> List[Self]: ...
-                NewManager = BaseManager.from_queryset(ModelQuerySet)
+                    def just_int(self) -> int: ...
+
+                NewManager = CustomManager.from_queryset(MyQuerySet)
+
                 class MyModel(models.Model):
                     objects = NewManager()
 
+                class QuerySetWithoutSelf(models.QuerySet["MyModelWithoutSelf"]):
+                    def method(self) -> "QuerySetWithoutSelf":
+                        return self
+
+                ManagerWithoutSelf = BaseManager.from_queryset(QuerySetWithoutSelf)
+
+                class MyModelWithoutSelf(models.Model):
+                    objects = ManagerWithoutSelf()
+
 -   case: from_queryset_with_base_manager
     main: |
         from myapp.models import MyModel