Repository: nautobot/nautobot

Issue:
[2.1.3] TypeError in certain app list views

### Environment
* Nautobot version (Docker tag too if applicable): 2.1.3

Seen with at least DLM and GC apps, example traceback:

```
nautobot-1  | 15:46:41.534 ERROR   django.request :
nautobot-1  |   Internal Server Error: /plugins/nautobot-device-lifecycle-mgmt/validated-software-device-report/
nautobot-1  | Traceback (most recent call last):
nautobot-1  |   File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 47, in inner
nautobot-1  |     response = get_response(request)
nautobot-1  |                ^^^^^^^^^^^^^^^^^^^^^
nautobot-1  |   File "/usr/local/lib/python3.11/site-packages/django/core/handlers/base.py", line 181, in _get_response
nautobot-1  |     response = wrapped_callback(request, *callback_args, **callback_kwargs)
nautobot-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
nautobot-1  |   File "/usr/local/lib/python3.11/site-packages/django/views/generic/base.py", line 70, in view
nautobot-1  |     return self.dispatch(request, *args, **kwargs)
nautobot-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
nautobot-1  |   File "/usr/local/lib/python3.11/site-packages/nautobot/core/views/mixins.py", line 165, in dispatch
nautobot-1  |     return super().dispatch(request, *args, **kwargs)
nautobot-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
nautobot-1  |   File "/usr/local/lib/python3.11/site-packages/django/views/generic/base.py", line 98, in dispatch
nautobot-1  |     return handler(request, *args, **kwargs)
nautobot-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
nautobot-1  |   File "/usr/local/lib/python3.11/site-packages/nautobot/core/views/generic.py", line 272, in get
nautobot-1  |     table = self.table(self.queryset, user=request.user, order_by=order_by)
nautobot-1  |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
nautobot-1  |   File "/usr/local/lib/python3.11/site-packages/nautobot/core/tables.py", line 149, in __init__
nautobot-1  |     self.data.data = self.data.data.select_related(*select_fields).prefetch_related(*prefetch_fields)
nautobot-1  |                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
nautobot-1  |   File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 1051, in select_related
nautobot-1  |     raise TypeError("Cannot call select_related() after .values() or .values_list()")
nautobot-1  | TypeError: Cannot call select_related() after .values() or .values_list()
nautobot-1  | ERROR:django.request:Internal Server Error: /plugins/nautobot-device-lifecycle-mgmt/validated-software-device-report/
nautobot-1  | Traceback (most recent call last):
nautobot-1  |   File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 47, in inner
nautobot-1  |     response = get_response(request)
nautobot-1  |                ^^^^^^^^^^^^^^^^^^^^^
nautobot-1  |   File "/usr/local/lib/python3.11/site-packages/django/core/handlers/base.py", line 181, in _get_response
nautobot-1  |     response = wrapped_callback(request, *callback_args, **callback_kwargs)
nautobot-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
nautobot-1  |   File "/usr/local/lib/python3.11/site-packages/django/views/generic/base.py", line 70, in view
nautobot-1  |     return self.dispatch(request, *args, **kwargs)
nautobot-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
nautobot-1  |   File "/usr/local/lib/python3.11/site-packages/nautobot/core/views/mixins.py", line 165, in dispatch
nautobot-1  |     return super().dispatch(request, *args, **kwargs)
nautobot-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
nautobot-1  |   File "/usr/local/lib/python3.11/site-packages/django/views/generic/base.py", line 98, in dispatch
nautobot-1  |     return handler(request, *args, **kwargs)
nautobot-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
nautobot-1  |   File "/usr/local/lib/python3.11/site-packages/nautobot/core/views/generic.py", line 272, in get
nautobot-1  |     table = self.table(self.queryset, user=request.user, order_by=order_by)
nautobot-1  |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
nautobot-1  |   File "/usr/local/lib/python3.11/site-packages/nautobot/core/tables.py", line 149, in __init__
nautobot-1  |     self.data.data = self.data.data.select_related(*select_fields).prefetch_related(*prefetch_fields)
nautobot-1  |                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
nautobot-1  |   File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 1051, in select_related
nautobot-1  |     raise TypeError("Cannot call select_related() after .values() or .values_list()")
nautobot-1  | TypeError: Cannot call select_related() after .values() or .values_list()
```

Appears to have been introduced by #5215.



Discussion:

Resolving patch:
diff --git a/nautobot/core/tables.py b/nautobot/core/tables.py
--- a/nautobot/core/tables.py
+++ b/nautobot/core/tables.py
@@ -3,6 +3,7 @@
 from django.contrib.auth.models import AnonymousUser
 from django.contrib.contenttypes.fields import GenericForeignKey
 from django.core.exceptions import FieldDoesNotExist
+from django.db import NotSupportedError
 from django.db.models.fields.related import ForeignKey, RelatedField
 from django.db.models.fields.reverse_related import ManyToOneRel
 from django.urls import reverse
@@ -61,7 +62,6 @@ def __init__(self, *args, user=None, **kwargs):
                 )
             # symmetric relationships are already handled above in the source_type case
 
-        model = getattr(self.Meta, "model", None)
         # Disable ordering on these TreeNode Models Table because TreeNode do not support sorting
         if model and issubclass(model, TreeNode):
             kwargs["orderable"] = False
@@ -106,27 +106,28 @@ def __init__(self, *args, user=None, **kwargs):
 
         # Dynamically update the table's QuerySet to ensure related fields are pre-fetched
         if isinstance(self.data, TableQuerysetData):
+            queryset = self.data.data
             select_fields = []
             prefetch_fields = []
             for column in self.columns:
                 if column.visible:
-                    model = getattr(self.Meta, "model")
+                    column_model = model
                     accessor = column.accessor
                     select_path = []
                     prefetch_path = []
                     for field_name in accessor.split(accessor.SEPARATOR):
                         try:
-                            field = model._meta.get_field(field_name)
+                            field = column_model._meta.get_field(field_name)
                         except FieldDoesNotExist:
                             break
                         if isinstance(field, ForeignKey) and not prefetch_path:
                             # Follow ForeignKeys to the related model via select_related
                             select_path.append(field_name)
-                            model = field.remote_field.model
+                            column_model = field.remote_field.model
                         elif isinstance(field, (RelatedField, ManyToOneRel)) and not select_path:
                             # Follow O2M and M2M relations to the related model via prefetch_related
                             prefetch_path.append(field_name)
-                            model = field.remote_field.model
+                            column_model = field.remote_field.model
                         elif isinstance(field, GenericForeignKey) and not select_path:
                             # Can't prefetch beyond a GenericForeignKey
                             prefetch_path.append(field_name)
@@ -140,13 +141,55 @@ def __init__(self, *args, user=None, **kwargs):
                         select_fields.append("__".join(select_path))
                     elif prefetch_path:
                         prefetch_fields.append("__".join(prefetch_path))
-            logger.debug(
-                "Applying .select_related(%s).prefetch_related(%s) to %s QuerySet",
-                select_fields,
-                prefetch_fields,
-                self.data.data.model.__name__,
-            )
-            self.data.data = self.data.data.select_related(*select_fields).prefetch_related(*prefetch_fields)
+
+            if select_fields:
+                # Django doesn't allow .select_related() on a QuerySet that had .values()/.values_list() applied, or
+                # one that has had union()/intersection()/difference() applied.
+                # We can detect and avoid these cases the same way that Django itself does.
+                if queryset._fields is not None:
+                    logger.debug(
+                        "NOT applying select_related(%s) to %s QuerySet as it includes .values()/.values_list()",
+                        select_fields,
+                        model.__name__,
+                    )
+                elif queryset.query.combinator:
+                    logger.debug(
+                        "NOT applying select_related(%s) to %s QuerySet as it is a combinator query",
+                        select_fields,
+                        model.__name__,
+                    )
+                else:
+                    logger.debug("Applying .select_related(%s) to %s QuerySet", select_fields, model.__name__)
+                    # Belt and suspenders - we should have avoided any error cases above, but be safe anyway:
+                    try:
+                        queryset = queryset.select_related(*select_fields)
+                    except (TypeError, ValueError, NotSupportedError) as exc:
+                        logger.warning(
+                            "Unexpected error when trying to .select_related() on %s QuerySet: %s",
+                            model.__name__,
+                            exc,
+                        )
+
+            if prefetch_fields:
+                if queryset.query.combinator:
+                    logger.debug(
+                        "NOT applying prefetch_related(%s) to %s QuerySet as it is a combinator query",
+                        prefetch_fields,
+                        model.__name__,
+                    )
+                else:
+                    logger.debug("Applying .prefetch_related(%s) to %s QuerySet", prefetch_fields, model.__name__)
+                    # Belt and suspenders - we should have avoided any error cases above, but be safe anyway:
+                    try:
+                        queryset = queryset.prefetch_related(*prefetch_fields)
+                    except (TypeError, ValueError, NotSupportedError) as exc:
+                        logger.warning(
+                            "Unexpected error when trying to .prefetch_related() on %s QuerySet: %s",
+                            model.__name__,
+                            exc,
+                        )
+
+            self.data.data = queryset
 
     @property
     def configurable_columns(self):


Test patch:
diff --git a/nautobot/core/tests/nautobot_config.py b/nautobot/core/tests/nautobot_config.py
--- a/nautobot/core/tests/nautobot_config.py
+++ b/nautobot/core/tests/nautobot_config.py
@@ -37,6 +37,8 @@
     "AWS_S3_REGION_NAME": "us-west-1",
 }
 
+# Use in-memory Constance backend instead of database backend so that settings don't leak between parallel tests.
+CONSTANCE_BACKEND = "constance.backends.memory.MemoryBackend"
 
 # Enable test data factories, as they're a pre-requisite for Nautobot core tests.
 TEST_USE_FACTORIES = True