Repository: nautobot/nautobot

Issue:
Config context causes timeout on devices API endpoint
<!--
    NOTE: IF YOUR ISSUE DOES NOT FOLLOW THIS TEMPLATE, IT WILL BE CLOSED.

    This form is only for reporting reproducible bugs. If you need assistance
    with Nautobot installation, or if you have a general question, please start a
    discussion instead: https://github.com/nautobot/nautobot/discussions

    Please describe the environment in which you are running Nautobot. Be sure
    that you are running an unmodified instance of the latest stable release
    before submitting a bug report, and that any plugins have been disabled.
-->
### Environment
* Python version: 3.6.15
* Nautobot version: 1.2.7

<!--
    Describe in detail the exact steps that someone else can take to reproduce
    this bug using the current stable release of Nautobot. Begin with the
    creation of any necessary database objects and call out every operation
    being performed explicitly. If reporting a bug in the REST API, be sure to
    reconstruct the raw HTTP request(s) being made: Don't rely on a client
    library such as pynautobot.
-->
We have 6300 devices in Nautobot and **no config context defined**. If I try to get all devices with no filters, it times out. I have to run the request several times to get the data to cache and than I am able to get the data in 37 seconds. If I exclude config context, I get the data right away in 25 seconds. I'm using pynautobot with threading, it uses 1000 limit for single request. The timeout on server is 60 seconds.

There seems to be some intensive computing done on getting config context even if there is none defined. This is the load on the server without using the filter, the peak is when I tried to get the data. First is from test server, second from production (a bit more powerful VM). It took 7 tries without threading to get the data from production server.
![image](https://user-images.githubusercontent.com/26114847/157002231-a3dc6ce7-726c-4d2c-9cfb-3eb1c7a67a38.png)
![image](https://user-images.githubusercontent.com/26114847/157004085-2f01a271-9eb5-4d3b-bc29-9ea241f258d1.png)


### Steps to Reproduce
1. Make sure there are no config contexts defined
2. GET all devices from `/api/dcim/devices` endpoint **without** excluding config context
3. GET all devices from `/api/dcim/devices` endpoint **with** excluding config context

<!-- What did you expect to happen? -->
### Expected Behavior
No difference, both take same time and manage to provide same data right away

<!-- What happened instead? -->
### Observed Behavior
Request with config context times out with 502 for several tries before the data gets cached


Discussion:
Here is a query analysis of what happens when you do a Device list view without `?exclude=config_context` which results in `Device.objects.annotate_config_context_data()`:

https://explain.depesz.com/s/P4sG

The query:

```sql
SELECT DISTINCT
    "dcim_device"."id",
    "dcim_device"."created",
    "dcim_device"."last_updated",
    "dcim_device"."_custom_field_data",
    "dcim_device"."status_id",
    "dcim_device"."local_context_data",
    "dcim_device"."local_context_schema_id",
    "dcim_device"."local_context_data_owner_content_type_id",
    "dcim_device"."local_context_data_owner_object_id",
    "dcim_device"."device_type_id",
    "dcim_device"."device_role_id",
    "dcim_device"."tenant_id",
    "dcim_device"."platform_id",
    "dcim_device"."name",
    "dcim_device"."_name",
    "dcim_device"."serial",
    "dcim_device"."asset_tag",
    "dcim_device"."site_id",
    "dcim_device"."rack_id",
    "dcim_device"."position",
    "dcim_device"."face",
    "dcim_device"."primary_ip4_id",
    "dcim_device"."primary_ip6_id",
    "dcim_device"."cluster_id",
    "dcim_device"."virtual_chassis_id",
    "dcim_device"."vc_position",
    "dcim_device"."vc_priority",
    "dcim_device"."comments",
    "dcim_device"."secrets_group_id",
    (
        SELECT
            JSONB_AGG(V0. "data") AS "_data"
        FROM
            "extras_configcontext" V0
        LEFT OUTER JOIN "extras_configcontext_platforms" V1 ON (V0. "id" = V1. "configcontext_id")
    LEFT OUTER JOIN "extras_configcontext_cluster_groups" V3 ON (V0. "id" = V3. "configcontext_id")
    LEFT OUTER JOIN "extras_configcontext_clusters" V5 ON (V0. "id" = V5. "configcontext_id")
    LEFT OUTER JOIN "extras_configcontext_tenant_groups" V7 ON (V0. "id" = V7. "configcontext_id")
    LEFT OUTER JOIN "extras_configcontext_tenants" V9 ON (V0. "id" = V9. "configcontext_id")
    LEFT OUTER JOIN "extras_configcontext_tags" V11 ON (V0. "id" = V11. "configcontext_id")
    LEFT OUTER JOIN "extras_configcontext_roles" V13 ON (V0. "id" = V13. "configcontext_id")
    LEFT OUTER JOIN "extras_configcontext_device_types" V15 ON (V0. "id" = V15. "configcontext_id")
    LEFT OUTER JOIN "extras_configcontext_sites" V17 ON (V0. "id" = V17. "configcontext_id")
    LEFT OUTER JOIN "extras_configcontext_regions" V19 ON (V0. "id" = V19. "configcontext_id")
    LEFT OUTER JOIN "dcim_region" V20 ON (V19. "region_id" = V20. "id")
WHERE ((V1. "platform_id" = "dcim_device"."platform_id"
        OR V1. "platform_id" IS NULL)
    AND (V3. "clustergroup_id" = "virtualization_cluster"."group_id"
        OR V3. "clustergroup_id" IS NULL)
    AND (V5. "cluster_id" = "dcim_device"."cluster_id"
        OR V5. "cluster_id" IS NULL)
    AND (V7. "tenantgroup_id" = "tenancy_tenant"."group_id"
        OR V7. "tenantgroup_id" IS NULL)
    AND (V9. "tenant_id" = "dcim_device"."tenant_id"
        OR V9. "tenant_id" IS NULL)
    AND (V11. "tag_id" IN (
            SELECT
                U0. "tag_id"
            FROM
                "extras_taggeditem" U0
                INNER JOIN "django_content_type" U1 ON (U0. "content_type_id" = U1. "id")
            WHERE (U1. "app_label" = 'dcim'
                AND U1. "model" = 'device'
                AND U0. "object_id" = "dcim_device"."id"))
        OR V11. "tag_id" IS NULL)
    AND V0. "is_active"
    AND (V13. "devicerole_id" = "dcim_device"."device_role_id"
        OR V13. "devicerole_id" IS NULL)
    AND (V15. "devicetype_id" = "dcim_device"."device_type_id"
        OR V15. "devicetype_id" IS NULL)
    AND (V17. "site_id" = "dcim_device"."site_id"
        OR V17. "site_id" IS NULL)
    AND ((V20. "level" <= "dcim_region"."level"
            AND V20. "lft" <= "dcim_region"."lft"
            AND V20. "rght" >= "dcim_region"."rght"
            AND V20. "tree_id" = "dcim_region"."tree_id")
        OR V19. "region_id" IS NULL))) AS "config_context_data"
FROM
    "dcim_device"
    LEFT OUTER JOIN "virtualization_cluster" ON ("dcim_device"."cluster_id" = "virtualization_cluster"."id")
    LEFT OUTER JOIN "tenancy_tenant" ON ("dcim_device"."tenant_id" = "tenancy_tenant"."id")
    INNER JOIN "dcim_site" ON ("dcim_device"."site_id" = "dcim_site"."id")
    LEFT OUTER JOIN "dcim_region" ON ("dcim_site"."region_id" = "dcim_region"."id")
ORDER BY
    "dcim_device"."_name" ASC
LIMIT 21;
```
And this happening on every `Device` instance and is likely where the badness is:

https://github.com/nautobot/nautobot/blob/28e89f77fd5868e55b329a2b75983f8a0315e335/nautobot/extras/models/models.py#L187

Here's the query analysis for that one:

https://explain.depesz.com/s/jsLE

And the query that results PER ROW:

```sql
SELECT DISTINCT
    "extras_configcontext"."data",
    "extras_configcontext"."weight",
    "extras_configcontext"."name"
FROM
    "extras_configcontext"
    LEFT OUTER JOIN "extras_configcontext_regions" ON ("extras_configcontext"."id" = "extras_configcontext_regions"."configcontext_id")
    LEFT OUTER JOIN "extras_configcontext_sites" ON ("extras_configcontext"."id" = "extras_configcontext_sites"."configcontext_id")
    LEFT OUTER JOIN "extras_configcontext_roles" ON ("extras_configcontext"."id" = "extras_configcontext_roles"."configcontext_id")
    LEFT OUTER JOIN "extras_configcontext_device_types" ON ("extras_configcontext"."id" = "extras_configcontext_device_types"."configcontext_id")
    LEFT OUTER JOIN "extras_configcontext_platforms" ON ("extras_configcontext"."id" = "extras_configcontext_platforms"."configcontext_id")
    LEFT OUTER JOIN "extras_configcontext_cluster_groups" ON ("extras_configcontext"."id" = "extras_configcontext_cluster_groups"."configcontext_id")
    LEFT OUTER JOIN "extras_configcontext_clusters" ON ("extras_configcontext"."id" = "extras_configcontext_clusters"."configcontext_id")
    LEFT OUTER JOIN "extras_configcontext_tenant_groups" ON ("extras_configcontext"."id" = "extras_configcontext_tenant_groups"."configcontext_id")
    LEFT OUTER JOIN "extras_configcontext_tenants" ON ("extras_configcontext"."id" = "extras_configcontext_tenants"."configcontext_id")
    LEFT OUTER JOIN "extras_configcontext_tags" ON ("extras_configcontext"."id" = "extras_configcontext_tags"."configcontext_id")
    LEFT OUTER JOIN "extras_tag" ON ("extras_configcontext_tags"."tag_id" = "extras_tag"."id")
WHERE (("extras_configcontext_regions"."region_id" IN (
            SELECT
                U0. "id"
            FROM
                "dcim_region" U0
            WHERE (U0. "lft" <= 66
                AND U0. "rght" >= 67
                AND U0. "tree_id" = 4))
        OR "extras_configcontext_regions"."region_id" IS NULL)
    AND ("extras_configcontext_sites"."site_id" = '77b157aa-e6c0-4a34-838a-a731fc949773'::uuid
        OR "extras_configcontext_sites"."site_id" IS NULL)
    AND ("extras_configcontext_roles"."devicerole_id" = 'fa830a2a-3534-4cb6-a921-78e7c3b60c61'::uuid
        OR "extras_configcontext_roles"."devicerole_id" IS NULL)
    AND ("extras_configcontext_device_types"."devicetype_id" = 'ba9e0196-21b9-4711-901f-13e55ccb6b02'::uuid
        OR "extras_configcontext_device_types"."devicetype_id" IS NULL)
    AND ("extras_configcontext_platforms"."platform_id" = '522f846f-3e9f-4924-a54b-e55b5b4d4722'::uuid
        OR "extras_configcontext_platforms"."platform_id" IS NULL)
    AND ("extras_configcontext_cluster_groups"."clustergroup_id" IS NULL
        OR "extras_configcontext_cluster_groups"."clustergroup_id" IS NULL)
    AND ("extras_configcontext_clusters"."cluster_id" IS NULL
        OR "extras_configcontext_clusters"."cluster_id" IS NULL)
    AND ("extras_configcontext_tenant_groups"."tenantgroup_id" IS NULL
        OR "extras_configcontext_tenant_groups"."tenantgroup_id" IS NULL)
    AND ("extras_configcontext_tenants"."tenant_id" = '707d0c3f-0a50-4f06-bedd-dab3a82f683e'::uuid
        OR "extras_configcontext_tenants"."tenant_id" IS NULL)
    AND ("extras_tag"."slug" IN ( SELECT DISTINCT
                U0. "slug"
            FROM
                "extras_tag" U0
                INNER JOIN "extras_taggeditem" U1 ON (U0. "id" = U1. "tag_id")
                INNER JOIN "django_content_type" U2 ON (U1. "content_type_id" = U2. "id")
            WHERE (U2. "app_label" = 'dcim'
                AND U2. "model" = 'device'
                AND U1. "object_id" = '1dd14c4f-ce6a-450e-a809-f265543289b3'::uuid))
        OR "extras_configcontext_tags"."tag_id" IS NULL)
    AND "extras_configcontext"."is_active")
ORDER BY
    "extras_configcontext"."weight" ASC,
    "extras_configcontext"."name" ASC
LIMIT 21;
```

Resolving patch:
diff --git a/nautobot/extras/models/models.py b/nautobot/extras/models/models.py
--- a/nautobot/extras/models/models.py
+++ b/nautobot/extras/models/models.py
@@ -184,8 +184,15 @@ def get_config_context(self):
         Return the rendered configuration context for a device or VM.
         """
 
-        # always manually query for config contexts
-        config_context_data = ConfigContext.objects.get_for_object(self).values_list("data", flat=True)
+        if not hasattr(self, "config_context_data"):
+            # Annotation not available, so fall back to manually querying for the config context
+            config_context_data = ConfigContext.objects.get_for_object(self).values_list("data", flat=True)
+        else:
+            config_context_data = self.config_context_data or []
+            # Annotation has keys "weight" and "name" (used for ordering) and "data" (the actual config context data)
+            config_context_data = [
+                c["data"] for c in sorted(config_context_data, key=lambda k: (k["weight"], k["name"]))
+            ]
 
         # Compile all config data, overwriting lower-weight values with higher-weight values where a collision occurs
         data = OrderedDict()
diff --git a/nautobot/extras/querysets.py b/nautobot/extras/querysets.py
--- a/nautobot/extras/querysets.py
+++ b/nautobot/extras/querysets.py
@@ -1,4 +1,5 @@
-from django.db.models import Model, OuterRef, Subquery, Q
+from django.db.models import Model, OuterRef, Subquery, Q, F
+from django.db.models.functions import JSONObject
 from django_celery_beat.managers import ExtendedQuerySet
 
 from nautobot.extras.models.tags import TaggedItem
@@ -10,9 +11,6 @@ class ConfigContextQuerySet(RestrictedQuerySet):
     def get_for_object(self, obj):
         """
         Return all applicable ConfigContexts for a given object. Only active ConfigContexts will be included.
-
-        Args:
-          aggregate_data: If True, use the JSONBAgg aggregate function to return only the list of JSON data objects
         """
 
         # `device_role` for Device; `role` for VirtualMachine
@@ -71,8 +69,9 @@ class ConfigContextModelQuerySet(RestrictedQuerySet):
     def annotate_config_context_data(self):
         """
         Attach the subquery annotation to the base queryset.
-        Order By clause in Subquery is not guaranteed to be respected
-        within the aggregated JSON array.
+
+        Order By clause in Subquery is not guaranteed to be respected within the aggregated JSON array, which is why
+        we include "weight" and "name" into the result so that we can sort it within Python to ensure correctness.
         """
         from nautobot.extras.models import ConfigContext
 
@@ -80,7 +79,15 @@ def annotate_config_context_data(self):
             config_context_data=Subquery(
                 ConfigContext.objects.filter(self._get_config_context_filters())
                 .order_by("weight", "name")
-                .annotate(_data=EmptyGroupByJSONBAgg("data"))
+                .annotate(
+                    _data=EmptyGroupByJSONBAgg(
+                        JSONObject(
+                            data=F("data"),
+                            name=F("name"),
+                            weight=F("weight"),
+                        )
+                    )
+                )
                 .values("_data")
             )
         ).distinct()
diff --git a/nautobot/utilities/query_functions.py b/nautobot/utilities/query_functions.py
--- a/nautobot/utilities/query_functions.py
+++ b/nautobot/utilities/query_functions.py
@@ -1,6 +1,5 @@
 from django.db.models import Aggregate, JSONField
 
-from django.contrib.postgres.aggregates.mixins import OrderableAggMixin
 from django.db import NotSupportedError
 from django.db.models import Func
 
@@ -30,10 +29,18 @@ def as_sql(self, compiler, connection, function=None, template=None, arg_joiner=
 
 
 class JSONBAgg(Aggregate):
+    """
+    Like django.contrib.postgres.aggregates.JSONBAgg, but different.
+
+    1. Supports both Postgres (JSONB_AGG) and MySQL (JSON_ARRAYAGG)
+    2. Does not support `ordering` as JSON_ARRAYAGG does not guarantee ordering.
+    """
+
     function = None
     output_field = JSONField()
+    # TODO: Django's JSONBAgg has `allow_distinct=True`, we might want to think about adding that at some point?
 
-    # Borrowed from `django.contrib.postrgres.aggregates.JSONBagg`.
+    # Borrowed from `django.contrib.postgres.aggregates.JSONBagg`.
     def convert_value(self, value, expression, connection):
         if not value:
             return "[]"
@@ -60,16 +67,6 @@ class EmptyGroupByJSONBAgg(JSONBAgg):
     JSONBAgg is a builtin aggregation function which means it includes the use of a GROUP BY clause.
     When used as an annotation for collecting config context data objects, the GROUP BY is
     incorrect. This subclass overrides the Django ORM aggregation control to remove the GROUP BY.
-
-    TODO in Django 3.2 ordering is supported natively on JSONBAgg so we only need to inherit from JSONBAgg.
     """
 
     contains_aggregate = False
-
-
-class OrderableJSONBAgg(OrderableAggMixin, JSONBAgg):
-    """
-    TODO in Django 3.2 ordering is supported natively on JSONBAgg so this is no longer needed.
-    """
-
-    template = "%(function)s(%(distinct)s%(expressions)s %(ordering)s)"


Test patch:
diff --git a/nautobot/extras/tests/test_api.py b/nautobot/extras/tests/test_api.py
--- a/nautobot/extras/tests/test_api.py
+++ b/nautobot/extras/tests/test_api.py
@@ -207,12 +207,24 @@ def test_render_configcontext_for_object(self):
         self.assertEqual(rendered_context["bar"], 456)
         self.assertEqual(rendered_context["baz"], 789)
 
+        # Test API response as well
+        self.add_permissions("dcim.view_device")
+        device_url = reverse("dcim-api:device-detail", kwargs={"pk": device.pk})
+        response = self.client.get(device_url, **self.header)
+        self.assertHttpStatus(response, status.HTTP_200_OK)
+        self.assertIn("config_context", response.data)
+        self.assertEqual(response.data["config_context"], {"foo": 123, "bar": 456, "baz": 789}, response.data)
+
         # Add another context specific to the site
         configcontext4 = ConfigContext(name="Config Context 4", data={"site_data": "ABC"})
         configcontext4.save()
         configcontext4.sites.add(site)
         rendered_context = device.get_config_context()
         self.assertEqual(rendered_context["site_data"], "ABC")
+        response = self.client.get(device_url, **self.header)
+        self.assertHttpStatus(response, status.HTTP_200_OK)
+        self.assertIn("config_context", response.data)
+        self.assertEqual(response.data["config_context"]["site_data"], "ABC", response.data["config_context"])
 
         # Override one of the default contexts
         configcontext5 = ConfigContext(name="Config Context 5", weight=2000, data={"foo": 999})
@@ -220,6 +232,10 @@ def test_render_configcontext_for_object(self):
         configcontext5.sites.add(site)
         rendered_context = device.get_config_context()
         self.assertEqual(rendered_context["foo"], 999)
+        response = self.client.get(device_url, **self.header)
+        self.assertHttpStatus(response, status.HTTP_200_OK)
+        self.assertIn("config_context", response.data)
+        self.assertEqual(response.data["config_context"]["foo"], 999, response.data["config_context"])
 
         # Add a context which does NOT match our device and ensure it does not apply
         site2 = Site.objects.create(name="Site 2", slug="site-2")
@@ -228,6 +244,10 @@ def test_render_configcontext_for_object(self):
         configcontext6.sites.add(site2)
         rendered_context = device.get_config_context()
         self.assertEqual(rendered_context["bar"], 456)
+        response = self.client.get(device_url, **self.header)
+        self.assertHttpStatus(response, status.HTTP_200_OK)
+        self.assertIn("config_context", response.data)
+        self.assertEqual(response.data["config_context"]["bar"], 456, response.data["config_context"])
 
     def test_schema_validation_pass(self):
         """