Repository: typeddjango/django-stubs

Issue:
Reverse relation is lost for abstract models with custom querysets
## What's wrong

The reverse relation to a concrete subclass of an abstract model that uses a custom queryset is not being populated by `django-stubs`.

A full set of test cases that illustrate the situation are in a commit [here](https://github.com/leamingrad/django-stubs/commit/00b7d75b42b8d757e2e2c5b6f0a1578b22f1db49) that I added on top of the latest master.

Since an example is worth a thousand words:

```python
from django.db import models

class AbstractLogQuerySet(models.QuerySet):
    pass

class Transaction(models.Model):
    pass

class AbstractLog(models.Model):
    objects = AbstractLogQuerySet.as_manager()
    class Meta:
        abstract=True

class TransactionLog(AbstractLog):
    transaction = models.ForeignKey(Transaction, on_delete=models.CASCADE, related_name="logs")

Transaction().logs
# myapp/models:13: error: "Transaction" has no attribute "logs"
```

Interestingly, the problem seems to be specific to querysets being used to make managers:
* An abstract model with the default manager works fine
* An abstract model with a custom manager works fine
* An abstract model with a custom manager and queryset fails as well

I'm happy to help with a fix, but to be honest I have no idea where to start.

## System information

I've seen this on multiple Python/Django/mypy/django-stubs versions but the test cases are written against:

- `python` version: 3.9.2
- `django` version:3.2.4
- `mypy` version: 0.910
- `django-stubs` version: HEAD

(Apologies if this is a duplicate - I looked and there seem to be similar issues but not this exact one)


Discussion:
> I'm happy to help with a fix, but to be honest I have no idea where to start.

Great to hear! I would be happy to help.

1. Start with a failing test somewhere here: https://github.com/typeddjango/django-stubs/tree/master/tests/typecheck/db/models Your example would be a great starting point
2. Then, look at the plugin itself, the execution path you are interested in is:
  - `get_base_class_hook` https://github.com/typeddjango/django-stubs/blob/master/mypy_django_plugin/main.py#L289 ->
  - `transform_model_class` https://github.com/typeddjango/django-stubs/blob/159a0e47904ad5dabf7d1ca4f3b43233ebcbea9f/mypy_django_plugin/main.py#L30 -> 
  - `process_model_class` https://github.com/typeddjango/django-stubs/blob/159a0e47904ad5dabf7d1ca4f3b43233ebcbea9f/mypy_django_plugin/transformers/models.py#L363 ->
  - `AddRelatedManagers` https://github.com/typeddjango/django-stubs/blob/159a0e47904ad5dabf7d1ca4f3b43233ebcbea9f/mypy_django_plugin/transformers/models.py#L237

Ask any questions you have! 👍 
I've managed to make a bit of progress on this, but would appreciate any help as I am a bit stuck.

From what I can tell, the root of the problem is when the `AddRelatedManagers` transformer is dealing with the `TransactionLog` class in the example above, it does not have an `objects` property.

This should be being set by the [`AddManagers` transformer](https://github.com/typeddjango/django-stubs/blob/8da8ab486232713da5bb3532e5f0252d0d4b8b1a/mypy_django_plugin/transformers/models.py#L145), but it fails to do so because it is never able to find the type info for the manager class when the `from_queryset` method is involved (I have switched my examples to use that as it generally seems to be more supported than `.from_manager`).

I have managed to make some progress in the commit [here](https://github.com/leamingrad/django-stubs/commit/0d59246813fd0cba0d8625cf29a5d1cc0623e9f4) by making sure that we always look for generated managers. This makes things work when we change the example above to:

```py
from django.db import models

class AbstractLogQuerySet(models.QuerySet):
    pass

class Transaction(models.Model):
    pass

# Declare the queryset manager outside the model, and use from_queryset
AbstractLogQSManager = models.Manager.from_queryset(AbstractLogQuerySet)

class AbstractLog(models.Model):
    objects = AbstractLogQSManager()
    class Meta:
        abstract=True

class TransactionLog(AbstractLog):
    transaction = models.ForeignKey(Transaction, on_delete=models.CASCADE, related_name="logs")

Transaction().logs
# No output - so everything is fine
```

However, it still does not work when the manager declaration is inline in the `AbstractLog` class. When I move the declaration of `objects` inside `TransactionLog`, everything works as expected, but I guess that is because the `AddManagers` transformation does not need to do anything as the `objects` property is declared explicitly?

I guess my confusion boils down to: "is there a way to make sure that `create_new_manager_class_from_from_queryset_method`" is called when a manager is declared from a queryset within an abstract class rather than globally?
You try to `defer()` if `objects`'s type are not yet analyzed. This might help.

Resolving patch:
diff --git a/mypy_django_plugin/transformers/models.py b/mypy_django_plugin/transformers/models.py
--- a/mypy_django_plugin/transformers/models.py
+++ b/mypy_django_plugin/transformers/models.py
@@ -273,7 +273,8 @@ def run_with_model_cls(self, model_cls: Type[Model]) -> None:
                     related_manager_info = self.lookup_typeinfo_or_incomplete_defn_error(
                         fullnames.RELATED_MANAGER_CLASS
                     )  # noqa: E501
-                    if "objects" not in related_model_info.names:
+                    objects = related_model_info.get("objects")
+                    if not objects:
                         raise helpers.IncompleteDefnException()
                 except helpers.IncompleteDefnException as exc:
                     if not self.api.final_iteration:
@@ -283,7 +284,7 @@ def run_with_model_cls(self, model_cls: Type[Model]) -> None:
 
                 # create new RelatedManager subclass
                 parametrized_related_manager_type = Instance(related_manager_info, [Instance(related_model_info, [])])
-                default_manager_type = related_model_info.names["objects"].type
+                default_manager_type = objects.type
                 if default_manager_type is None:
                     default_manager_type = self.try_generate_related_manager(related_model_cls, related_model_info)
                 if (


Test patch:
diff --git a/tests/typecheck/fields/test_related.yml b/tests/typecheck/fields/test_related.yml
--- a/tests/typecheck/fields/test_related.yml
+++ b/tests/typecheck/fields/test_related.yml
@@ -601,6 +601,37 @@
                     pass
 
 
+-   case: test_foreign_key_from_superclass_inherits_correctly_when_also_inheriting_manager
+    main: |
+        from myapp.models import MyUser, Book, Article, LibraryEntity
+        reveal_type(Book().registered_by_user)  # N: Revealed type is "myapp.models.MyUser*"
+        reveal_type(Article().registered_by_user)  # N: Revealed type is "myapp.models.MyUser*"
+
+        user = MyUser()
+        reveal_type(user.book_set) # N: Revealed type is "django.db.models.manager.RelatedManager[myapp.models.Book]"
+        reveal_type(user.article_set) # N: Revealed type is "django.db.models.manager.RelatedManager[myapp.models.Article]"
+    installed_apps:
+        - myapp
+    files:
+        -   path: myapp/__init__.py
+        -   path: myapp/models.py
+            content: |
+              from django.db import models
+              class MyUser(models.Model):
+                  pass
+              class LibraryEntityQuerySet(models.QuerySet):
+                  pass
+              class LibraryEntity(models.Model):
+                  class Meta:
+                      abstract = True
+                  objects = models.Manager.from_queryset(LibraryEntityQuerySet)()
+                  registered_by_user = models.ForeignKey(MyUser, on_delete=models.CASCADE)
+              class Book(LibraryEntity):
+                  pass
+              class Article(LibraryEntity):
+                  pass
+
+
 -   case: foreign_key_relationship_for_models_with_custom_manager
     main: |
         from myapp.models import Transaction